Repository navigation
test(acp): give weak tests real outcome checks - #24494
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change only edits tests. Every replaced mock-call assertion is swapped for a stricter outcome check, and no product code, test id or pinned decision is removed. [DESIGN-REVIEWED] 9a81401 Description read[DESCRIPTION-READ] c0bc2d9d2fcf9f74c54dd7f784a9cea40f3fb31376fb4f9aebe68b50df63059a That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
aa36f76 to
21db358
Compare
|
Backend Tests (Windows) (4) on 21db358 fails in test/test_telegram_dispatcher.py::TestABusyReplyCarriesItsContext (test_an_injection_reply_on_the_queue_path_audits_exactly_once, test_a_declined_steer_with_an_injection_parent_audits_once_at_the_queue: assert [] == ['telegram_reply_parent']). This PR does not touch that file or its source. The same two tests fail on Linux at main's own merge commit 9d51fb4 (#23730), which added them, and at 20c46e9. No open PR or issue tracks it yet. |
21db358 to
4533cdd
Compare
|
Backend Tests (Windows) (6) red is not from this PR. All 6 failures are in test/test_slack_team_id_env.py::TestSpawnEnv (FileNotFoundError [WinError 2] spawning '/usr/bin/kiro-cli acp'). That file arrived with #22380 (a3ee04f), and main's own CI run on a3ee04f fails the same 6 tests in Backend Tests (Windows) (6): https://github.com/kirodotdev/KiroCrew/actions/runs/40401185256/job/121866397755. This PR does not touch that file, and no open fix PR or issue exists yet. Leaving it for a main-side fix. |
4533cdd to
9a81401
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (20 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change; ACP tests gain real outcome assertions.
Problem / Motivation
Goal: Make 64 weak ACP tests check a real outcome, so a broken shutdown, approval, compact or start path fails a test instead of passing it.
These tests in
test/test_acp*.pyeither checked nothing or only asserted that a mock was called (assert_awaited_once(), often with no args). They passed against code that answered the wrong id, ran teardown steps out of order, kept stale state, or logged nothing.Why it matters
A test that only checks a mock call stays green when the call has the wrong arguments, the wrong order, or the wrong side effects. The ACP layer owns process teardown, tool approval and model pinning, where those slips leak processes or run the wrong model.
Not a goal
What changed (motivation → approach → change)
Each weak test now asserts the outcome its name promises. The kinds of check added:
approve_toolsendscall(43, {... "optionId": "allow"})and nothing elsekillpgcompact()clears a stale_compact_result; a fresh start leavesresumedFalse;reject_toolsets_turn_permission_denied; a rate-limited model poll returns the snapshotapprove_toolreturns the handle's bool;cleanup_session("")never resolves the sessions dirasyncio.sleep(0.1), so "no cancel was sent" cannot pass because the check never ranThe security tests in
test_acp_tool_identity.py(forged shell directive, third-party MCP directive) gain checks only: the applier is still never called, and the transcript shows the untrusted output verbatim instead of an applied outcome.Files touched: 20 test files under
test/, alltest_acp*.py.Backwards compatibility
Compatible: test-only change.
Removes nothing: no test, fixture or product code is removed; one helper in
test_acp_stale_recovery.pyreturns an extra event its four callers use.Tests
Proof: not provable -- test-only change
goal_scan.py weak) on the ACP files: 65 before, 1 after. Repo total: 1,664 on main 2356a1b, 1,600 on this branch.git checkout: subagent destroy before cancel; hook cancel after the kill;touch_activityskips the handle keepalive;compact()keeps the stale result; a fresh start marked resumed; a rate-limited poll returns[];reject_toolskips_turn_permission_denied; the warning moved ahead of the no-id return; pipes closed after the SIGTERM;shutdownskips clearing_process_tree_confirmed_dead; the survivor warning demoted to debug; the initial-effort failure demoted to debug; the failed-start kill called withexpected=True.--cov=kiro_crewon the 20 touched files, union of two runs per side: base 113,344 lines, head 113,345, 0 lines lost. Two early base runs showed 15 lines insel.pyandsession_*_sig.pythat a third base run did not; those are first-run key-bootstrap branches, not ACP code.-n 4; the four timing-sensitive files (test_acp_stale_recovery.py,test_acp_turn_lock.py,test_acp_sandbox_init_classification.py,test_acp_runtime_pre_turn_drain.py) passed 10 times in a row at-n0.test_flake_pattern_ratchet.pyandtest_wait_discipline_ratchet.py: 113 passed. black, isort, flake8,check_file_line_cap.py,check_comment_history.py: clean.Determinism: no new sleep, clock or port. The TOCTOU tests trade a fixed 0.1 s sleep for an event wait under the default lost-run bound. caplog reads filter on
kiro_crew.acp.clientorkiro_crew.providers.acp.Left alone (1):
test_acp_deepseek_backend_vault.py::test_a_provider_key_mapping_the_harness_honours_is_acceptedis a pure "the happy mapping does not raise" check; its refusal twins carry the assertions.Manual verification
N/A — test-only change; the runs above are the evidence.
Related Issues
N/A — part of the ongoing weak-test cleanup; no tracking issue.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.