Skip to content

test(acp): give weak tests real outcome checks - #26206

Merged
iamwhatever merged 1 commit into
mainfrom
test-cleanup/weak-acp2
Oct 30, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
test-cleanup/weak-acp2

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Give eight weak ACP tests a check on a real result, so a regression in what the code leaves behind fails them.

The strict weak-test scan (goal_scan.py weak) flags 41 tests in test/test_acp*.py on main after the first pass (#24494). A weak test has no check, or only checks a mock's call record. This second pass sorted all 41 and fixed the ones that have a real result to check.

Why it matters

A test that only checks that a mock was called still passes when the code stops storing, writing or logging what it should. These eight now fail on a planted bug in the code they cover.

Not a goal

  • No change under src/.
  • No rewrite of a mock check into another mock-check form.
  • No change to tests whose only output is a call across an outward edge (JSON-RPC to the harness child, a spawn, a kill, a config write to the child), or to pure "does not raise" tests.
  • No helper renamed or reshaped to move the scan count.

What changed (motivation → approach → change)

Triage of the 41 flagged tests:

Group Count Action
a. fixable with a real result 8 fixed in this PR
b. already checked through a helper the scan misses 4 left as is
c. boundary only: the one output is a call across an outward edge 25 left as is
d. does not raise 4 left as is

Fixes, by kind of result:

Kind Count Tests
Stored state, written file or return value 6 see below
Log line (caplog filtered by logger name kiro_crew.acp.client) 2 see below

Stored-result fixes:

  • test_acp_client_tool_hooks.py::test_swallows_sel_errors: after the SEL write fails, the call id is in _sel_settled_tool_calls, so a later event does not retry it.
  • test_acp_codex_harness.py::test_create_session_feeds_and_persists_once: the spies on the feed and persist are gone. The session/new answer now advertises two fake model ids in a model select; the test reads them back from the real model_registry cache and from the sidecar file the real persist writes under tmp_path. The registry globals are monkeypatched to empty dicts and the sidecar path points at tmp_path, so nothing touches the data home.
  • test_acp_provider.py::test_dead_runtime_respawned_and_create_session_on_new: the provider's client now serves the respawned runtime and its new handle (_client._runtime is new_runtime, _client._handle is new_handle). The mock checks on kill and spawn stay, so this one is partly real.
  • test_acp_dynamic_config.py::test_syncs_levels_to_persistence: the patched persistence calls are gone; the test reads the real global order and value set after _sync_effort_levels(). The conftest autouse fixture _reset_reasoning_effort_globals restores both globals.
  • test_acp_tool_search.py::test_overlay_on_init: the patched overlay method is gone; the constructor writes the real cli.json under tmp_path for kiro, and writes nothing for claude. Param ids are unchanged.
  • test_acp_session_provider.py::test_touch_activity_stamps_its_own_handle: uses a real AcpSessionHandle and checks its _keepalive_ts was stamped.

Log-line fixes (own row; the label reaches only the log line, _stderr_lines stores the text without it):

  • test_acp_spawn_label.py::test_kiro_stderr_keeps_its_existing_prefix
  • test_acp_spawn_label.py::test_adapter_stderr_uses_the_label_its_call_site_resolved

Both read caplog.records filtered on r.name == "kiro_crew.acp.client" instead of patching the module logger. The stderr input changed from the literal \\n text to a real newline, which readline returns and the drain strips.

Already checked through a helper (b, left as is): the four test_acp_stale_recovery.py::test_toctou_* tests. Each waits on await_until(oracle_rechecked, ...), which raises when the watchdog never consults the oracle a second time. Removing the runtime-clock arm of the stale guard fails test_toctou_stale_path_c_runtime_activity_prevents_probe through that helper, so these already check a real result.

Boundary only (c, left as is): in test_acp_session_provider.py, the shutdown, cancel, reject and set_model tests (11); test_acp_provider.py effort push, start priority, KAS require_replay and forced probe (4); test_acp_codex_harness.py::test_the_resolver_is_delegated_not_duplicated (the argv result is already checked by test_the_resolved_entry_is_the_whole_argv); test_acp_copilot_backend.py::test_allow_always_is_answered_as_allow_once; test_acp_runtime_kill.py::test_kill_uses_the_tree_kill_while_the_root_identity_holds; test_acp_client_process_lifecycle.py::test_kill_process_awaits_async_variant_not_sync; test_acp_runtime_pre_turn_drain.py::test_pre_turn_drain_cancels_a_stranded_elicitation_request; the two pi, one goose and one opencode effort writes; test_acp_tool_identity.py::test_automation_provenance_reaches_directive_applier (the output is the hand-off to the directive applier).

Does not raise (d, left as is): test_the_verified_copilot_release_starts, test_the_version_refusal_is_copilots_alone, test_a_kas_runtime_is_not_refused_at_session_start, test_a_provider_key_mapping_the_harness_honours_is_accepted.

test/test_acp_tool_search.py was reformatted with black 26.10.1 and is now black-clean, so its line in .github/black-baseline.txt is removed.

Backwards compatibility

Compatible: test-only change; no source file is touched.

Reader: scripts/check_black_formatting.py:main -- release -- test/test_acp_tool_search.py is black-clean on this head, so the gate passes with its baseline line removed

Tests

Proof: not provable -- test-only change

Strict weak count for test/test_acp*.py, merge-base vs head, same goal_scan.py:

Rule Before After
default (strict, by-name, trusted helpers) 41 33
--no-byname 41 33
--no-trusted 41 33
--no-inner 41 33
--loose 9 7
--loose --no-inner 9 7

Repo-wide strict: 1,475 before, 1,467 after.

Planted bugs, each in the source under test and reverted after: 9 caught out of 9 tried. One per fixed test, two for the codex test (skip the feed; skip the persist). Each new check fails on its plant; for the SEL, respawn and keepalive plants no other test in the file fails.

Collected test ids in the seven touched files: 498 before, 498 after. Every touched file passed 3 runs with -n 4.

Coverage (touched files, union of 2 runs per side): 84,811 lines covered on base, 84,869 on head; 0 lines lost, 58 gained.

Local gates: flake8, isort, black 26.10.1, check_file_line_cap.py, check_comment_history.py, check_black_formatting.py, test_flake_pattern_ratchet.py, test_wait_discipline_ratchet.py, Semgrep 1.136.0 (--config semgrep/ --baseline-commit <merge-base> --error), and a grep of the added lines for the .woke.yml terms.

Manual verification

N/A -- unit coverage sufficient; test-only change.

Related Issues

Refs #25195

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 30, 2026 12:46
@iamwhatever
iamwhatever enabled auto-merge (squash) October 30, 2026 12:46
@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision ed0c2770b42caf62ebcc1401a5d49c03141e5b74 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ed0c2770b42caf62ebcc1401a5d49c03141e5b74 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ed0c2770b42caf62ebcc1401a5d49c03141e5b74 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only touches tests, which now check real results instead of mock calls, plus one line removed from the black baseline.

[DESIGN-REVIEWED] ed0c277

Description read

[DESCRIPTION-READ] ac46db805ea956e1b98648c2a7114a5f9c6885f809b1f557b08d83c647a697d7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ed0c2770b42caf62ebcc1401a5d49c03141e5b74 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ed0c277

Verdict parsed from the review's SHA-scoped output markers for commit ed0c2770b42caf62ebcc1401a5d49c03141e5b74.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ed0c2770b42caf62ebcc1401a5d49c03141e5b74 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ed0c2770b42caf62ebcc1401a5d49c03141e5b74 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ed0c277

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ed0c2770b42caf62ebcc1401a5d49c03141e5b74 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 30, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (8 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: seven ACP test files replace mock-call assertions with checks on real outcomes, and one test file reformatted by black leaves the black baseline; no source file changes.

@iamwhatever
iamwhatever merged commit 13e3ff2 into main Oct 30, 2026
145 of 146 checks passed
@iamwhatever
iamwhatever deleted the test-cleanup/weak-acp2 branch October 30, 2026 14:03
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants