Repository navigation
test(slack): give weak tests real outcome checks - #25661
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)Design-level review of Design-Verdict: PASS Most of what the description says this PR does is not in the diff. This revision changes about 30 assertions in 9 files, so the weak-test counts, the new fakes and the file move it reports never happened. BlockersThe description claims work the diff does not contain Notes
[DESIGN-REVIEWED] f541c65 Description read[DESCRIPTION-READ] 0f5a430ce4f86fd2a7cd7ce36b380f8d3d08b62f6453b35be1253480c2eb19fe That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
bd3df64 to
9c8d2f5
Compare
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
9c8d2f5 to
f541c65
Compare
|
Design review on |
|
Edited |
buluoray
left a comment
There was a problem hiding this comment.
🤖 Kiro Crew Auto-Pipeline [operator: buluoray]:
Approving at f541c655a0. Test-only; no assertion weakened, no prompt wording pinned, writes go to per-test isolated directories. Skipping the config write in _set_default_agent reddens 4 named agent tests; adding the user in _handle_allowlist_cmd reddens test_multi_user_is_refused. 632 tests pass across the 9 files; CI green. Non-blocking: the two handler first-turn tests no longer check source_thread and source_user.
Problem / Motivation
Goal: Slack tests that only checked a mock's call record now check what the user or the store actually receives, so a wrong reply, label, stored row or delivery target fails a test.
278 functions in
test/test_slack*.pyare weak under the strictgoal_scan.py weakrule: no check, or only checks on a mock's record (assert_called*,call_args,await_count). This change gives 23 of them a check on a real result that no test checked before.Why it matters
A test that only proves a mock was called cannot catch a wrong outcome. The first Slack pass showed that rewriting
assert_called_once_withascall_args_list == [...]moves the metric without adding a check; this pass adds real outcomes instead.Not a goal
src/is untouched).What changed (motivation → approach → change)
Each of the 23 tests now reads a real result:
agent.default_agentwritten toconfig.jsonby the real_set_default_agent(and the cached default it leaves)TestHandleAgent::test_off_resets_default_agent,::test_known_agent_switches;TestAgentSelect::test_switches_to_resolved_agent,::test_off_resets_to_defaultconfig.jsonwritten when the switch is refused or emptyTestAgentSelect::test_unresolvable_agent_aborts,::test_empty_selection_is_a_noop,::test_non_owner_rejectedTestAllowlistButtons::test_value_without_display_name_falls_back_to_id(user added),TestHandleAllowlistCmd::test_multi_user_is_refused(nobody added)ConversationLog(base_dir=tmp_path)test_slack_subagent_skips_persistence_for_restricted_thread(both ids),test_slack_subagent_persists_even_when_slack_post_fails,test_slack_subagent_persists_exactly_once_after_timeout_retry,test_taskrunner_parent_injects_without_slack_postagentin a realConversationLogtest_auto_reply_records_agent_in_first_turn,test_spawn_command_records_agent_in_first_turnFalsewhen the prompt denies (test_all_slots_trusted_does_not_auto_approve); cron Done delivery returns its message when the store is busy, unreadable or the job is already gone (3 tests);/kirocrew cron removereply (test_caller_falls_back_to_surface_when_no_user);_notify_nudge_expiredreturnsTrue(test_cycle_cap_outranks_a_stall)test_slack_inline_stop.py), OPTIONS records cleared before the edit (test_expiry_runs_once_even_if_more_turns_follow)About 160 rewrites from the earlier version of this PR were reverted to
main: they read the same reply, post or notice text back from a hand-written recording fake (_Replies,_Inbox,_HookSink,_Bell,_NotificationLog, list-backedappend/notify/send_message,MockSlackClient.actions) that the oldcall_argscheck already read. A fake that keeps what it was sent is a call-record check, the same rule as #24495, so those tests stay asmainhas them. The helpers they needed are gone, and the Stop-confirm class is back intest/test_slack_interactions_coverage.py(1,996 lines, under the 2,000-line cap).Approval, allowlist and owner-gate tests were only strengthened: every existing check stays.
Backwards compatibility
Compatible: test-only change.
Removes nothing: no test is removed, merged, renamed or moved; the collected count for
test/test_slack*.pyis 3,637 on both base and head, with the same function list.Tests
Proof: not provable -- test-only change
Weak counts for
test/test_slack*.py, branch vs merge-base138385511a:--no-inner--loose--loose --no-innerRepo-wide strict count: 1,719 → 1,696.
Planted bugs (source edited, test run with
PYTHONDONTWRITEBYTECODE=1and a freshPYTHONPYCACHEPREFIX, reverted): 13 of 13 caught.test_unresolvable_agent_aborts_set_default_agentstores the name upper-casedtest_switches_to_resolved_agent,test_known_agent_switchesTestAgentSelect::test_off_resets_to_default/kirocrew agent offwrites the wordtest_off_resets_default_agenttest_value_without_display_name_falls_back_to_id/kirocrew allowlist addadds the usertest_multi_user_is_refusedskips_persistenceids"done"instead of the messagecron removereply wording changedtest_caller_falls_back_to_surface_when_no_user_notify_nudge_expiredreports not deliveredtest_cycle_cap_outranks_a_stallrecords_agent_in_first_turnteststest_all_slots_trusted_does_not_auto_approveCoverage, touched test files, two runs per side, union of covered lines and branches: base 108,996 lines / 4,861 branches, head 109,175 / 4,930 (+179 lines). No line lost. One branch lost:
executors.py1381→1383, the shutdown of a subprocess pool, which runs only when a test on the same xdist worker happened to create that pool; it is not reached by these tests' assertions.Stability: every touched file passed three times with
-n 4(632 tests).test/test_flake_pattern_ratchet.pyandtest/test_wait_discipline_ratchet.pypass; no new timed waits.Left as they are (255 strict-weak), by reason:
_route_message,dispatch_interactive,_handle_slash,_run_chat, a handler spy,SubagentManager,stop_turn/note_stop) receives a call, with no downstream state to read short of an integration test (about 45).test_slack_gateway_update_apply.py,test_slack_gateway_run.py,test_slack_gateway_memory_boot.py,test_slack_gateway_update_wheel.py; about 20).test_remove_all_nothing_deleted_audits_failed: the reply says "Removed 1 cron job(s)" even when nothing was deleted, so pinning the reply would lock in a doubtful message.Already checked via helper: none found.
Manual verification
N/A — test-only change; the changed tests run in-process with no network or Slack.
Related Issues
Refs #25195
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)