Skip to content

test(slack): give weak tests real outcome checks - #25661

Merged
iamwhatever merged 1 commit into
mainfrom
test-cleanup/weak-slack2
Oct 29, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
test-cleanup/weak-slack2

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Slack tests that only checked a mock's call record now check what the user or the store actually receives, so a wrong reply, label, stored row or delivery target fails a test.

278 functions in test/test_slack*.py are weak under the strict goal_scan.py weak rule: no check, or only checks on a mock's record (assert_called*, call_args, await_count). This change gives 23 of them a check on a real result that no test checked before.

Why it matters

A test that only proves a mock was called cannot catch a wrong outcome. The first Slack pass showed that rewriting assert_called_once_with as call_args_list == [...] moves the metric without adding a check; this pass adds real outcomes instead.

Not a goal

  • No production code changes (src/ is untouched).
  • No test removals or merges; no new behaviour under test.
  • The 255 Slack tests still weak are left as they are (see the list below).

What changed (motivation → approach → change)

Each of the 23 tests now reads a real result:

Outcome now checked Tests
agent.default_agent written to config.json by the real _set_default_agent (and the cached default it leaves) TestHandleAgent::test_off_resets_default_agent, ::test_known_agent_switches; TestAgentSelect::test_switches_to_resolved_agent, ::test_off_resets_to_default
No config.json written when the switch is refused or empty TestAgentSelect::test_unresolvable_agent_aborts, ::test_empty_selection_is_a_noop, ::test_non_owner_rejected
Allowlist state left on the orchestrator TestAllowlistButtons::test_value_without_display_name_falls_back_to_id (user added), TestHandleAllowlistCmd::test_multi_user_is_refused (nobody added)
Stored transcript rows in a real ConversationLog(base_dir=tmp_path) test_slack_subagent_skips_persistence_for_restricted_thread (both ids), test_slack_subagent_persists_even_when_slack_post_fails, test_slack_subagent_persists_exactly_once_after_timeout_retry, test_taskrunner_parent_injects_without_slack_post
Transcript metadata agent in a real ConversationLog test_auto_reply_records_agent_in_first_turn, test_spawn_command_records_agent_in_first_turn
Return values approval callback returns False when the prompt denies (test_all_slots_trusted_does_not_auto_approve); cron Done delivery returns its message when the store is busy, unreadable or the job is already gone (3 tests); /kirocrew cron remove reply (test_caller_falls_back_to_surface_when_no_user); _notify_nudge_expired returns True (test_cycle_cap_outranks_a_stall)
Module and slot state stop-decline marker armed (test_slack_inline_stop.py), OPTIONS records cleared before the edit (test_expiry_runs_once_even_if_more_turns_follow)

About 160 rewrites from the earlier version of this PR were reverted to main: they read the same reply, post or notice text back from a hand-written recording fake (_Replies, _Inbox, _HookSink, _Bell, _NotificationLog, list-backed append/notify/send_message, MockSlackClient.actions) that the old call_args check already read. A fake that keeps what it was sent is a call-record check, the same rule as #24495, so those tests stay as main has them. The helpers they needed are gone, and the Stop-confirm class is back in test/test_slack_interactions_coverage.py (1,996 lines, under the 2,000-line cap).

Approval, allowlist and owner-gate tests were only strengthened: every existing check stays.

Backwards compatibility

Compatible: test-only change.

Removes nothing: no test is removed, merged, renamed or moved; the collected count for test/test_slack*.py is 3,637 on both base and head, with the same function list.

Tests

Proof: not provable -- test-only change

Weak counts for test/test_slack*.py, branch vs merge-base 138385511a:

Rule Base Head
strict (default) 278 255
--no-inner 296 273
--loose 107 98
--loose --no-inner 125 116

Repo-wide strict count: 1,719 → 1,696.

Planted bugs (source edited, test run with PYTHONDONTWRITEBYTECODE=1 and a fresh PYTHONPYCACHEPREFIX, reverted): 13 of 13 caught.

Planted bug Caught by
agent select writes an unresolved name test_unresolvable_agent_aborts
_set_default_agent stores the name upper-cased test_switches_to_resolved_agent, test_known_agent_switches
agent-select reset writes the selected word TestAgentSelect::test_off_resets_to_default
/kirocrew agent off writes the word test_off_resets_default_agent
allowlist approve does not add the user test_value_without_display_name_falls_back_to_id
/kirocrew allowlist add adds the user test_multi_user_is_refused
subagent completion stores the announce as the reply the three persistence tests
restricted threads persisted both skips_persistence ids
cron Done path returns "done" instead of the message the busy, unreadable and already-removed tests
cron remove reply wording changed test_caller_falls_back_to_surface_when_no_user
_notify_nudge_expired reports not delivered test_cycle_cap_outranks_a_stall
transcript metadata ignores the agent both records_agent_in_first_turn tests
dashboard approval ignores a deny test_all_slots_trusted_does_not_auto_approve

Coverage, touched test files, two runs per side, union of covered lines and branches: base 108,996 lines / 4,861 branches, head 109,175 / 4,930 (+179 lines). No line lost. One branch lost: executors.py 1381→1383, the shutdown of a subprocess pool, which runs only when a test on the same xdist worker happened to create that pool; it is not reached by these tests' assertions.

Stability: every touched file passed three times with -n 4 (632 tests). test/test_flake_pattern_ratchet.py and test/test_wait_discipline_ratchet.py pass; no new timed waits.

Left as they are (255 strict-weak), by reason:

  • Same content read from a recording fake, reverted as call-record checks (about 160; see above).
  • Delegation-only: the outcome is that a seam (_route_message, dispatch_interactive, _handle_slash, _run_chat, a handler spy, SubagentManager, stop_turn/note_stop) receives a call, with no downstream state to read short of an integration test (about 45).
  • Failure-is-swallowed smoke tests whose only outcome is "does not raise" (about 20).
  • Boot/update/process tests whose outcome is a subprocess or service call (test_slack_gateway_update_apply.py, test_slack_gateway_run.py, test_slack_gateway_memory_boot.py, test_slack_gateway_update_wheel.py; about 20).
  • test_remove_all_nothing_deleted_audits_failed: the reply says "Removed 1 cron job(s)" even when nothing was deleted, so pinning the reply would lock in a doubtful message.

Already checked via helper: none found.

Manual verification

N/A — test-only change; the changed tests run in-process with no network or Slack.

Related Issues

Refs #25195

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 29, 2026 05:21
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision f541c655a05beb2f9f4acc789bdaab8e0d3c5404 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f541c655a05beb2f9f4acc789bdaab8e0d3c5404 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)

Design-level review of f541c655a05beb2f9f4acc789bdaab8e0d3c5404 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Most of what the description says this PR does is not in the diff. This revision changes about 30 assertions in 9 files, so the weak-test counts, the new fakes and the file move it reports never happened.

Blockers

The description claims work the diff does not contain
Trigger: The description says "277 → 94" strict-weak tests and gives a planted-bug table. It also names new fakes (_Replies, _HookSink/hook_sink, _Inbox, _Bell, _NotificationLog, _fake_transport.sent), says "MockSlackClient.update_message gains an optional blocks argument", and says "The Stop-confirm class moves … to test/test_slack_interactions_coverage_stop_confirm.py". None of these appear in authentic.patch. The new file does not exist in the tree, and the patch adds no blocks parameter. The stated reason for the move is false too: test/test_slack_interactions_coverage.py is 1,996 lines, under the cap. The PR also measures against merge-base bf47e1598b, which is not this branch's base, 13838551.
Outcome: Most of the push seems to have been lost, or the description was written for another branch. Merging records progress on #25195 that did not ship: about 180 Slack tests stay weak. The compatibility section and the planted-bug proof also describe code that reviewers and later readers cannot find, so neither can be checked.
Fix: push the missing commit content, or rewrite ## What changed, ## Backwards compatibility and ## Tests to match the roughly 30 assertions this diff actually changes, with counts taken against 13838551.
Clears when: every claimed fake, the update_message(blocks=…) change and the stop-confirm move are in the diff, or the description lists only what the diff contains, with weak counts re-measured on this revision.

Notes

  • The assertions the diff does add are sound in shape: a real ConversationLog(base_dir=tmp_path) and a real config.json replace mock call records.

[DESIGN-REVIEWED] f541c65

Description read

[DESCRIPTION-READ] 0f5a430ce4f86fd2a7cd7ce36b380f8d3d08b62f6453b35be1253480c2eb19fe

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f541c655a05beb2f9f4acc789bdaab8e0d3c5404 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f541c65

Verdict parsed from the review's SHA-scoped output markers for commit f541c655a05beb2f9f4acc789bdaab8e0d3c5404.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f541c655a05beb2f9f4acc789bdaab8e0d3c5404 false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the test-cleanup/weak-slack2 branch from bd3df64 to 9c8d2f5 Compare October 29, 2026 05:27
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f541c655a05beb2f9f4acc789bdaab8e0d3c5404 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f541c65

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f541c655a05beb2f9f4acc789bdaab8e0d3c5404 false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the test-cleanup/weak-slack2 branch from 9c8d2f5 to f541c65 Compare October 29, 2026 06:21
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 29, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Design review on f541c655a05beb2f9f4acc789bdaab8e0d3c5404 ("the description claims work the diff does not contain"): fixed in the description. That review read the description of the earlier revision. This revision reverts about 160 rewrites that only read the same text back from recording fakes (call-record checks, same rule as #24495), so the fakes, the update_message(blocks=...) change and the stop-confirm move are gone on purpose. The description now lists only the 23 tests this diff changes, with weak counts, planted bugs and coverage re-measured against merge-base 138385511a.

@iamwhatever
iamwhatever enabled auto-merge (squash) October 29, 2026 07:15
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Edited ## Not a goal at the owner's request: the weak-test count left in Slack files is 255 after the rework, not 94. No other change to the body, and no push.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 29, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Kiro Crew Auto-Pipeline [operator: buluoray]:

Approving at f541c655a0. Test-only; no assertion weakened, no prompt wording pinned, writes go to per-test isolated directories. Skipping the config write in _set_default_agent reddens 4 named agent tests; adding the user in _handle_allowlist_cmd reddens test_multi_user_is_refused. 632 tests pass across the 9 files; CI green. Non-blocking: the two handler first-turn tests no longer check source_thread and source_user.

@iamwhatever
iamwhatever merged commit f22af6c into main Oct 29, 2026
114 checks passed
@iamwhatever
iamwhatever deleted the test-cleanup/weak-slack2 branch October 29, 2026 16:21
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants