Repository navigation
test(acp): give weak tests real outcome checks - #26206
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change only touches tests, which now check real results instead of mock calls, plus one line removed from the black baseline. [DESIGN-REVIEWED] ed0c277 Description read[DESCRIPTION-READ] ac46db805ea956e1b98648c2a7114a5f9c6885f809b1f557b08d83c647a697d7 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (8 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: seven ACP test files replace mock-call assertions with checks on real outcomes, and one test file reformatted by black leaves the black baseline; no source file changes.
Problem / Motivation
Goal: Give eight weak ACP tests a check on a real result, so a regression in what the code leaves behind fails them.
The strict weak-test scan (
goal_scan.py weak) flags 41 tests intest/test_acp*.pyon main after the first pass (#24494). A weak test has no check, or only checks a mock's call record. This second pass sorted all 41 and fixed the ones that have a real result to check.Why it matters
A test that only checks that a mock was called still passes when the code stops storing, writing or logging what it should. These eight now fail on a planted bug in the code they cover.
Not a goal
src/.What changed (motivation → approach → change)
Triage of the 41 flagged tests:
Fixes, by kind of result:
kiro_crew.acp.client)Stored-result fixes:
test_acp_client_tool_hooks.py::test_swallows_sel_errors: after the SEL write fails, the call id is in_sel_settled_tool_calls, so a later event does not retry it.test_acp_codex_harness.py::test_create_session_feeds_and_persists_once: the spies on the feed and persist are gone. Thesession/newanswer now advertises two fake model ids in amodelselect; the test reads them back from the realmodel_registrycache and from the sidecar file the real persist writes undertmp_path. The registry globals are monkeypatched to empty dicts and the sidecar path points attmp_path, so nothing touches the data home.test_acp_provider.py::test_dead_runtime_respawned_and_create_session_on_new: the provider's client now serves the respawned runtime and its new handle (_client._runtime is new_runtime,_client._handle is new_handle). The mock checks on kill and spawn stay, so this one is partly real.test_acp_dynamic_config.py::test_syncs_levels_to_persistence: the patched persistence calls are gone; the test reads the real global order and value set after_sync_effort_levels(). The conftest autouse fixture_reset_reasoning_effort_globalsrestores both globals.test_acp_tool_search.py::test_overlay_on_init: the patched overlay method is gone; the constructor writes the realcli.jsonundertmp_pathfor kiro, and writes nothing for claude. Param ids are unchanged.test_acp_session_provider.py::test_touch_activity_stamps_its_own_handle: uses a realAcpSessionHandleand checks its_keepalive_tswas stamped.Log-line fixes (own row; the label reaches only the log line,
_stderr_linesstores the text without it):test_acp_spawn_label.py::test_kiro_stderr_keeps_its_existing_prefixtest_acp_spawn_label.py::test_adapter_stderr_uses_the_label_its_call_site_resolvedBoth read
caplog.recordsfiltered onr.name == "kiro_crew.acp.client"instead of patching the module logger. The stderr input changed from the literal\\ntext to a real newline, whichreadlinereturns and the drain strips.Already checked through a helper (b, left as is): the four
test_acp_stale_recovery.py::test_toctou_*tests. Each waits onawait_until(oracle_rechecked, ...), which raises when the watchdog never consults the oracle a second time. Removing the runtime-clock arm of the stale guard failstest_toctou_stale_path_c_runtime_activity_prevents_probethrough that helper, so these already check a real result.Boundary only (c, left as is): in
test_acp_session_provider.py, the shutdown, cancel, reject andset_modeltests (11);test_acp_provider.pyeffort push, start priority, KASrequire_replayand forced probe (4);test_acp_codex_harness.py::test_the_resolver_is_delegated_not_duplicated(the argv result is already checked bytest_the_resolved_entry_is_the_whole_argv);test_acp_copilot_backend.py::test_allow_always_is_answered_as_allow_once;test_acp_runtime_kill.py::test_kill_uses_the_tree_kill_while_the_root_identity_holds;test_acp_client_process_lifecycle.py::test_kill_process_awaits_async_variant_not_sync;test_acp_runtime_pre_turn_drain.py::test_pre_turn_drain_cancels_a_stranded_elicitation_request; the two pi, one goose and one opencode effort writes;test_acp_tool_identity.py::test_automation_provenance_reaches_directive_applier(the output is the hand-off to the directive applier).Does not raise (d, left as is):
test_the_verified_copilot_release_starts,test_the_version_refusal_is_copilots_alone,test_a_kas_runtime_is_not_refused_at_session_start,test_a_provider_key_mapping_the_harness_honours_is_accepted.test/test_acp_tool_search.pywas reformatted with black 26.10.1 and is now black-clean, so its line in.github/black-baseline.txtis removed.Backwards compatibility
Compatible: test-only change; no source file is touched.
Reader: scripts/check_black_formatting.py:main -- release -- test/test_acp_tool_search.py is black-clean on this head, so the gate passes with its baseline line removed
Tests
Proof: not provable -- test-only change
Strict weak count for
test/test_acp*.py, merge-base vs head, samegoal_scan.py:--no-byname--no-trusted--no-inner--loose--loose --no-innerRepo-wide strict: 1,475 before, 1,467 after.
Planted bugs, each in the source under test and reverted after: 9 caught out of 9 tried. One per fixed test, two for the codex test (skip the feed; skip the persist). Each new check fails on its plant; for the SEL, respawn and keepalive plants no other test in the file fails.
Collected test ids in the seven touched files: 498 before, 498 after. Every touched file passed 3 runs with
-n 4.Coverage (touched files, union of 2 runs per side): 84,811 lines covered on base, 84,869 on head; 0 lines lost, 58 gained.
Local gates: flake8, isort, black 26.10.1,
check_file_line_cap.py,check_comment_history.py,check_black_formatting.py,test_flake_pattern_ratchet.py,test_wait_discipline_ratchet.py, Semgrep 1.136.0 (--config semgrep/ --baseline-commit <merge-base> --error), and a grep of the added lines for the.woke.ymlterms.Manual verification
N/A -- unit coverage sufficient; test-only change.
Related Issues
Refs #25195
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.