Skip to content

test(acp): give weak tests real outcome checks - #24494

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
test-cleanup/weak-acp
Oct 27, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
test-cleanup/weak-acp

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make 64 weak ACP tests check a real outcome, so a broken shutdown, approval, compact or start path fails a test instead of passing it.

These tests in test/test_acp*.py either checked nothing or only asserted that a mock was called (assert_awaited_once(), often with no args). They passed against code that answered the wrong id, ran teardown steps out of order, kept stale state, or logged nothing.

Why it matters

A test that only checks a mock call stays green when the call has the wrong arguments, the wrong order, or the wrong side effects. The ACP layer owns process teardown, tool approval and model pinning, where those slips leak processes or run the wrong model.

Not a goal

  • No product code change. Source files were edited only for the temporary planted bugs below, and reverted.
  • No test removed, renamed or merged: the collected test-id count is unchanged.
  • The rest of the weak-test list outside the ACP files.

What changed (motivation → approach → change)

Each weak test now asserts the outcome its name promises. The kinds of check added:

Check Example
Exact call list instead of "was called" approve_tool sends call(43, {... "optionId": "allow"}) and nothing else
Order of teardown steps a subagent shutdown cancels the turn before it destroys the handle; hook tasks stop before the runtime kill; pipes close before killpg
State the call leaves behind compact() clears a stale _compact_result; a fresh start leaves resumed False; reject_tool sets _turn_permission_denied; a rate-limited model poll returns the snapshot
Return value approve_tool returns the handle's bool; cleanup_session("") never resolves the sessions dir
Logs, filtered by logger name a swallowed effort error logs a warning; a retained live child PID is reported; a notification with no id logs nothing
A positive signal before an absence the four TOCTOU watchdog tests now wait for the oracle's second consult instead of asyncio.sleep(0.1), so "no cancel was sent" cannot pass because the check never ran

The security tests in test_acp_tool_identity.py (forged shell directive, third-party MCP directive) gain checks only: the applier is still never called, and the transcript shows the untrusted output verbatim instead of an applied outcome.

Files touched: 20 test files under test/, all test_acp*.py.

Backwards compatibility

Compatible: test-only change.

Removes nothing: no test, fixture or product code is removed; one helper in test_acp_stale_recovery.py returns an extra event its four callers use.

Tests

Proof: not provable -- test-only change

  • Weak-test scan (goal_scan.py weak) on the ACP files: 65 before, 1 after. Repo total: 1,664 on main 2356a1b, 1,600 on this branch.
  • Planted bugs: 13 caught out of 13 tried. Each was a one-line source edit, reverted with git checkout: subagent destroy before cancel; hook cancel after the kill; touch_activity skips the handle keepalive; compact() keeps the stale result; a fresh start marked resumed; a rate-limited poll returns []; reject_tool skips _turn_permission_denied; the warning moved ahead of the no-id return; pipes closed after the SIGTERM; shutdown skips clearing _process_tree_confirmed_dead; the survivor warning demoted to debug; the initial-effort failure demoted to debug; the failed-start kill called with expected=True.
  • Coverage, --cov=kiro_crew on the 20 touched files, union of two runs per side: base 113,344 lines, head 113,345, 0 lines lost. Two early base runs showed 15 lines in sel.py and session_*_sig.py that a third base run did not; those are first-run key-bootstrap branches, not ACP code.
  • Collected ids on the 20 files: 1,078 on base and head.
  • Every touched file passed 3 times at -n 4; the four timing-sensitive files (test_acp_stale_recovery.py, test_acp_turn_lock.py, test_acp_sandbox_init_classification.py, test_acp_runtime_pre_turn_drain.py) passed 10 times in a row at -n0.
  • test_flake_pattern_ratchet.py and test_wait_discipline_ratchet.py: 113 passed. black, isort, flake8, check_file_line_cap.py, check_comment_history.py: clean.

Determinism: no new sleep, clock or port. The TOCTOU tests trade a fixed 0.1 s sleep for an event wait under the default lost-run bound. caplog reads filter on kiro_crew.acp.client or kiro_crew.providers.acp.

Left alone (1): test_acp_deepseek_backend_vault.py::test_a_provider_key_mapping_the_harness_honours_is_accepted is a pure "the happy mapping does not raise" check; its refusal twins carry the assertions.

Manual verification

N/A — test-only change; the runs above are the evidence.

Related Issues

N/A — part of the ongoing weak-test cleanup; no tracking issue.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 9a81401efc93a35503d334f9299125baaa7fd4b9 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 9a81401efc93a35503d334f9299125baaa7fd4b9 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 9a81401efc93a35503d334f9299125baaa7fd4b9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only edits tests. Every replaced mock-call assertion is swapped for a stricter outcome check, and no product code, test id or pinned decision is removed.

[DESIGN-REVIEWED] 9a81401

Description read

[DESCRIPTION-READ] c0bc2d9d2fcf9f74c54dd7f784a9cea40f3fb31376fb4f9aebe68b50df63059a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 9a81401efc93a35503d334f9299125baaa7fd4b9 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9a81401

Verdict parsed from the review's SHA-scoped output markers for commit 9a81401efc93a35503d334f9299125baaa7fd4b9.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9a81401efc93a35503d334f9299125baaa7fd4b9 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 9a81401efc93a35503d334f9299125baaa7fd4b9 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9a81401

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9a81401efc93a35503d334f9299125baaa7fd4b9 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 26, 2026
@iamwhatever
iamwhatever force-pushed the test-cleanup/weak-acp branch from aa36f76 to 21db358 Compare October 26, 2026 18:12
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main 20c46e9 to pick up #24485: the red Backend Tests (3.12, 2) and Windows (2) shards on aa36f76 were the three XPASS(strict) guards in test/test_crew_log_fullscan_guard.py that #24485 un-marked, not this change. Every other shard was green on that head. CI re-running on 21db358.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 26, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (4) on 21db358 fails in test/test_telegram_dispatcher.py::TestABusyReplyCarriesItsContext (test_an_injection_reply_on_the_queue_path_audits_exactly_once, test_a_declined_steer_with_an_injection_parent_audits_once_at_the_queue: assert [] == ['telegram_reply_parent']). This PR does not touch that file or its source. The same two tests fail on Linux at main's own merge commit 9d51fb4 (#23730), which added them, and at 20c46e9. No open PR or issue tracks it yet.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Disposition for the Backend Tests (3.12, 4) and Windows (4) failures in test/test_telegram_dispatcher.py::TestABusyReplyCarriesItsContext: main is red on these two tests too, and this PR does not touch them. The open fix is #24531 (issues #24514, #24529). I will rebase this branch once it merges.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 26, 2026
@iamwhatever
iamwhatever force-pushed the test-cleanup/weak-acp branch from 21db358 to 4533cdd Compare October 26, 2026 21:26
@iamwhatever

iamwhatever commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto main (a3ee04f; new head 4533cdd) to pick up #24531, which fixes the telegram test failures in Backend Tests (3.12, 4), Backend Tests (Windows, 4) and the Coverage Gate. The 20 touched test/test_acp*.py files pass locally (1078 passed, -n 4).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 26, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (6) red is not from this PR. All 6 failures are in test/test_slack_team_id_env.py::TestSpawnEnv (FileNotFoundError [WinError 2] spawning '/usr/bin/kiro-cli acp'). That file arrived with #22380 (a3ee04f), and main's own CI run on a3ee04f fails the same 6 tests in Backend Tests (Windows) (6): https://github.com/kirodotdev/KiroCrew/actions/runs/40401185256/job/121866397755. This PR does not touch that file, and no open fix PR or issue exists yet. Leaving it for a main-side fix.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Disposition for Backend Tests (Windows) (6): the six test_slack_team_id_env.py::TestSpawnEnv failures are main's (issue #24612). The open fix is #24613. I will rebase this branch once it merges.

@iamwhatever
iamwhatever force-pushed the test-cleanup/weak-acp branch from 4533cdd to 9a81401 Compare October 27, 2026 01:19
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main 8a518e1 to pick up #24609, which fixes the Windows (6) test_slack_team_id_env.py failures. New head 9a81401. The 20 touched test_acp files pass locally (1078 passed). CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 27, 2026 03:05

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (20 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change; ACP tests gain real outcome assertions.

@KiroCrewCaptain
KiroCrewCaptain merged commit beaec90 into main Oct 27, 2026
108 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the test-cleanup/weak-acp branch October 27, 2026 03:05
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants