Skip to content

test(chat): give weak tests real outcome checks - #24650

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
test-cleanup/weak-chat
Oct 27, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
test-cleanup/weak-chat

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Chat tests that only checked a mock call now also check what the user or the next step actually gets, so a broken chat turn fails a test.

46 tests in the chat area (test/test_chat*.py) passed as long as some mock was called. They did not look at the heartbeat that was sent, the card or tool row the user sees, the flag left on the slot, or the log line that names a refusal. A change that broke those results kept them green.

Why it matters

These tests guard the approval ladder, the WakaTime heartbeat scope (no beats for cron, restored or denied work), pending resets and the queue cycle. A test that cannot see the outcome cannot catch a regression in it.

Not a goal

  • No product code changes. src/ is only edited temporarily to plant bugs, then reverted.
  • No test is removed, merged or renamed.
  • Tests whose only observable result is a mock record (pure "does not raise" swallow tests, transport stubs) stay as main has them; they are listed below.

What changed (motivation → approach → change)

Each strengthened test now checks a real result next to, or instead of, its mock check:

Area Real outcome now checked
WakaTime accounting (test_chat_runner_coverage.py, 16 tests) The heartbeat payloads the turn hands to the sender (heartbeats._schedule_send captured into a list). The real note_coding_activity / note_coding_cadence run, so a denied, injected or restored turn must send [], and a counted turn must send exactly one ai coding beat
Approval ladder (test_chat_runner_coverage_approval.py, 12 tests) The card the user sees and how it resolved (rejected / approved / no card), and the tool row text (🔧 … for auto-approve, ✅ … after a click)
Pending reset and prefetch (test_chat_runner_coverage_pending.py, 7 tests) The return value of _consume_pending_reset, the discard flag cleared on the slot, the prefetch arm entry removed, and the expiry log line
Queue, commands, turn start (_queue, _commands, _turn_start, 5 tests) The queue emptied by a hand-off, held notes flushed into an inject row, the slot's synthesis flags, the /goal reply text, the eager-spawn bail log line
Other chat files (6 tests) The refreshed title on the slot, the cached voice list on a second request, the persisted assistant segment, the route's reset: true body, the untouched legacy OAuth banner, the folder warning on the current state logger (a real logger and caplog)

One fake key literal in a touched test is now built from split parts.

Backwards compatibility

Compatible: test-only change.

Removes nothing: no test, fixture or product code is removed; collected test ids in the 12 touched files are identical to the merge-base (649).

Tests

Proof: not provable -- test-only change

Weak-test scan (goal_scan.py weak), merge-base 96c4fef57c vs this branch:

Rule Merge-base Branch Change
strict (default; mock-record asserts count as weak) 2,295 2,249 −46
--loose 1,646 1,609 −37

The strict count is the real cut: chat-area hits went from 65 to 19.

Planted bugs, one at a time in src/, each reverted with git checkout: 12 caught / 12 tried. They were: cron turn counted as coding, restored turn counted, cadence beat dropped, prefetch arm entry kept, eager-spawn bail log changed, folder warning sent to a fixed logger, refreshed title not applied, voice cache ignored, idle cycle not flushing held notes, discard flag left armed, untimestamped OAuth banner rewritten, auto-approve tool row losing its marker.

Coverage of src/kiro_crew from the 12 touched files, two runs per side, unions compared: base 99,580 lines, head 99,571. Head adds 21 lines (chat_title refresh, /goal handler, note_coding_activity body). It loses 30. 26 of those are the real WakaTime send path (send_heartbeat, resolve_api_key, vault read), which base reached only because background send tasks fired from chat tests and read the vault. Those lines are covered by test_wakatime_heartbeats.py, test_wakatime_service.py and test_secrets_vault.py. The other 4 (chat_runner.py:1172, turn_dispatch.py:631-632) are cancel/exception paths that vary run to run.

Determinism: the touched files passed 10 times at -n 4 and once at -n0 in reverse file order. test_flake_pattern_ratchet.py and test_wait_discipline_ratchet.py pass. The new caplog checks filter by logger name, and the arm entry is set with monkeypatch.setitem.

Left as main has them (19, no real outcome to check): the four composition-contract child-process tests (already checked by _run_child, which asserts), test_a_non_shell_approval_mints_no_shell_witness (the pin cache depends on the host PATH), test_chat_backfill.py::test_no_slack_client_is_a_noop, and in test_chat_runner_coverage_output.py: test_recorder_failure_is_swallowed, test_zero_duration_skips_the_emit, test_latch_failure_never_raises, test_no_slack_client_is_a_noop, test_unlinked_session_is_a_noop, test_link_from_the_session_store_is_used, test_post_failure_is_swallowed, test_empty_text_is_not_mirrored, test_reply_is_chunked_per_transport_limit, test_transport_failure_never_disrupts_the_turn, test_nothing_is_scheduled_without_text_or_loop, test_widget_and_image_each_schedule_one_task, test_emission_failure_is_swallowed.

Manual verification

N/A — test-only change; the planted bugs above show the new checks fail when the behaviour breaks.

Related Issues

N/A — part of the ongoing weak-test cleanup; no tracking issue.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 26, 2026 23:56
@iamwhatever
iamwhatever requested a review from CrysisDeu October 26, 2026 23:56
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 7af128446422d5b09152ded18b6949c9e56f6df0 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 7af128446422d5b09152ded18b6949c9e56f6df0 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 7af128446422d5b09152ded18b6949c9e56f6df0 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only touches tests, and each mock assertion it drops is replaced by a check of the actual result, so no test now accepts the opposite behaviour.

[DESIGN-REVIEWED] 7af1284

Description read

[DESCRIPTION-READ] 72ec0d4d626fbde1eefd2c6e441e00657dcd7d94660b35497f4f2dcc11112aca

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 7af128446422d5b09152ded18b6949c9e56f6df0 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7af1284

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7af128446422d5b09152ded18b6949c9e56f6df0 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 7af128446422d5b09152ded18b6949c9e56f6df0 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7af1284

Verdict parsed from the review's SHA-scoped output markers for commit 7af128446422d5b09152ded18b6949c9e56f6df0.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7af128446422d5b09152ded18b6949c9e56f6df0 false-positive: <reason, 20 to 500 characters>

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (6) is red on the six test/test_slack_team_id_env.py::TestSpawnEnv tests (FileNotFoundError: [WinError 2]). This PR does not touch that file; it is main's red, tracked in #24612 with the fix in #24613 (still open). This PR waits for that fix and will rebase onto it.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@iamwhatever

iamwhatever commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto main to pick up #24609, which fixes the Windows (6) test_slack_team_id_env.py failures. New head 7af1284. The 12 touched test files pass locally (649 passed). CI re-running.

@iamwhatever
iamwhatever force-pushed the test-cleanup/weak-chat branch from 71d05a1 to 7af1284 Compare October 27, 2026 01:39
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 27, 2026 03:04

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (12 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change; chat runner tests gain real outcome assertions.

@KiroCrewCaptain
KiroCrewCaptain merged commit d07affc into main Oct 27, 2026
108 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the test-cleanup/weak-chat branch October 27, 2026 03:27
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants