Repository navigation
test(chat): give weak tests real outcome checks - #24650
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change only touches tests, and each mock assertion it drops is replaced by a check of the actual result, so no test now accepts the opposite behaviour. [DESIGN-REVIEWED] 7af1284 Description read[DESCRIPTION-READ] 72ec0d4d626fbde1eefd2c6e441e00657dcd7d94660b35497f4f2dcc11112aca That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
71d05a1 to
7af1284
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (12 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change; chat runner tests gain real outcome assertions.
Problem / Motivation
Goal: Chat tests that only checked a mock call now also check what the user or the next step actually gets, so a broken chat turn fails a test.
46 tests in the chat area (
test/test_chat*.py) passed as long as some mock was called. They did not look at the heartbeat that was sent, the card or tool row the user sees, the flag left on the slot, or the log line that names a refusal. A change that broke those results kept them green.Why it matters
These tests guard the approval ladder, the WakaTime heartbeat scope (no beats for cron, restored or denied work), pending resets and the queue cycle. A test that cannot see the outcome cannot catch a regression in it.
Not a goal
src/is only edited temporarily to plant bugs, then reverted.What changed (motivation → approach → change)
Each strengthened test now checks a real result next to, or instead of, its mock check:
test_chat_runner_coverage.py, 16 tests)heartbeats._schedule_sendcaptured into a list). The realnote_coding_activity/note_coding_cadencerun, so a denied, injected or restored turn must send[], and a counted turn must send exactly oneai codingbeattest_chat_runner_coverage_approval.py, 12 tests)rejected/approved/ no card), and the tool row text (🔧 …for auto-approve,✅ …after a click)test_chat_runner_coverage_pending.py, 7 tests)_consume_pending_reset, the discard flag cleared on the slot, the prefetch arm entry removed, and the expiry log line_queue,_commands,_turn_start, 5 tests)injectrow, the slot's synthesis flags, the/goalreply text, the eager-spawn bail log linereset: truebody, the untouched legacy OAuth banner, the folder warning on the current state logger (a real logger andcaplog)One fake key literal in a touched test is now built from split parts.
Backwards compatibility
Compatible: test-only change.
Removes nothing: no test, fixture or product code is removed; collected test ids in the 12 touched files are identical to the merge-base (649).
Tests
Proof: not provable -- test-only change
Weak-test scan (
goal_scan.py weak), merge-base96c4fef57cvs this branch:--looseThe strict count is the real cut: chat-area hits went from 65 to 19.
Planted bugs, one at a time in
src/, each reverted withgit checkout: 12 caught / 12 tried. They were: cron turn counted as coding, restored turn counted, cadence beat dropped, prefetch arm entry kept, eager-spawn bail log changed, folder warning sent to a fixed logger, refreshed title not applied, voice cache ignored, idle cycle not flushing held notes, discard flag left armed, untimestamped OAuth banner rewritten, auto-approve tool row losing its marker.Coverage of
src/kiro_crewfrom the 12 touched files, two runs per side, unions compared: base 99,580 lines, head 99,571. Head adds 21 lines (chat_titlerefresh,/goalhandler,note_coding_activitybody). It loses 30. 26 of those are the real WakaTime send path (send_heartbeat,resolve_api_key, vault read), which base reached only because background send tasks fired from chat tests and read the vault. Those lines are covered bytest_wakatime_heartbeats.py,test_wakatime_service.pyandtest_secrets_vault.py. The other 4 (chat_runner.py:1172,turn_dispatch.py:631-632) are cancel/exception paths that vary run to run.Determinism: the touched files passed 10 times at
-n 4and once at-n0in reverse file order.test_flake_pattern_ratchet.pyandtest_wait_discipline_ratchet.pypass. The newcaplogchecks filter by logger name, and the arm entry is set withmonkeypatch.setitem.Left as main has them (19, no real outcome to check): the four composition-contract child-process tests (already checked by
_run_child, which asserts),test_a_non_shell_approval_mints_no_shell_witness(the pin cache depends on the hostPATH),test_chat_backfill.py::test_no_slack_client_is_a_noop, and intest_chat_runner_coverage_output.py:test_recorder_failure_is_swallowed,test_zero_duration_skips_the_emit,test_latch_failure_never_raises,test_no_slack_client_is_a_noop,test_unlinked_session_is_a_noop,test_link_from_the_session_store_is_used,test_post_failure_is_swallowed,test_empty_text_is_not_mirrored,test_reply_is_chunked_per_transport_limit,test_transport_failure_never_disrupts_the_turn,test_nothing_is_scheduled_without_text_or_loop,test_widget_and_image_each_schedule_one_task,test_emission_failure_is_swallowed.Manual verification
N/A — test-only change; the planted bugs above show the new checks fail when the behaviour breaks.
Related Issues
N/A — part of the ongoing weak-test cleanup; no tracking issue.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)