Skip to content

test(acp): seal the agent-spec self-heal in the runtime cleanup lifecycle fixture - #19093

Merged
iamwhatever merged 1 commit into
mainfrom
fix/runtime-cleanup-load-flake
Oct 11, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/runtime-cleanup-load-flake

Conversation

@CrysisDeu

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The two [load] tests in test/test_runtime_cleanup_regression.py pass on every shard, every time, without a wider timeout.

test_session_start_persists_new_descendants[load] and test_cancelled_descendant_scan_terminates_only_failed_session[load] fail with TimeoutError (the 5 s asyncio.wait_for around rt.load_session) on Backend Tests (Windows) (2). Seen on PRs that touch nothing near this code (#17684, #17784, #18708, #18964, #18894) while main stays green. The [new] twins of both tests never fail.

Why it matters

Every open PR can turn red on a Windows shard for a test it did not touch, and each red costs a rerun and a flake-ledger entry. The test is a lifecycle test against a fake kernel; a timeout there says nothing about the code under test.

Not a goal

  • Changing the 5 s budgets, skipping, retrying or marking the tests.
  • Changing load_session, ensure_agent_materialized or the agent-spec self-heal in production code.
  • Changing the other three tests in the file.

What changed (motivation → approach → change)

The two [load] tests call rt.load_session. Its MCP-array hop reads the agent spec through session_mcp._agent_spec_and_snapshot_for, which calls ensure_agent_materialized("kirocrew") first. The per-test agents dir is always empty, so that call runs the whole rebuild_agent_config() inside the hop: a dozen managed agent spec files, a config migration with a backup, and the sensitive-path anchor caches. That is ~300 ms on Linux and open-ended on a loaded Windows shard; the CI traceback shows the task parked in exactly that to_thread when the 5 s budget expires. The [new] tests pass mcp_servers=[], which skips the hop entirely, so only [load] ever paid for the rebuild.

The lifecycle fixture already seals the other half of that same hop (pooled_session_servers returns []). It now seals this half too: session_mcp.ensure_agent_materialized is stubbed, the same seam test_acp_session_mcp.py, test_acp_client.py and test_acp_mcp_ref_guard.py stub. No kiro-cli runs in this test, so nothing reads the spec that would have been written.

With the stub the [load] hop is ~10 ms and the [load] tests take the same time as their [new] twins.

Backwards compatibility

Compatible: test-only change; no production code is touched.

Removes nothing: one fixture gains one monkeypatch line.

Tests

  • test/test_runtime_cleanup_regression.py: the lifecycle fixture stubs the agent-spec self-heal, so both [load] tests exercise the lifecycle code with the same inputs the [new] tests already use.

Determinism proof (testing-conventions § Proving a determinism fix), Linux, kc-shared-venv, PYTHONPATH=src:

Check Result
20 × -n0, whole file 20/20 green
10 × -n 4 --dist loadgroup --max-worker-restart=2 with test_acp_session_mcp.py and test_acp_client.py as co-tenants see PR comment
Shuffled order (@file, seed in PR comment) see PR comment
Forced condition: a throwaway -p plugin that makes kiro_crew.agent.rebuild_agent_config take 6 s parent f07162523a: exactly the two [load] tests red with TimeoutError, 3 passed; this branch: 5 passed

The forced-condition plugin and the commands are in a PR comment; the plugin is not committed.

Manual verification

N/A — the fixture change is covered by the five tests in the file plus the forced-condition run above. The Windows CI shard is the real-platform check.

Related Issues

Closes #18874
Closes #18676

Pattern harvest

Rule candidate: review-prompt
Pattern: a lifecycle/fake-kernel test fixture that stubs one half of a two-read to_thread hop (the pooled array) but leaves the other half (the agent-spec read with its self-heal) real, so one parametrization pays for a full on-disk rebuild under a per-call budget.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 11:17
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 11, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Determinism proof for head a3dab0e (Linux, kc-shared-venv, PYTHONPATH=src; testing-conventions § Proving a determinism fix).

Where the 5 s went (hop trace, parent f07162523a, [load]):

to_thread _pooled_session_servers_and_ref_spec   0.304s   <- ensure_agent_materialized -> rebuild_agent_config()
to_thread kiro_control_plane_servers             0.029s

Same hop on this branch: 0.008s – 0.030s. The CI tracebacks (runs 38123860920 and 38126323826, Backend Tests (Windows) (2)) park the task in exactly that to_thread when the budget expires; the [new] tests pass mcp_servers=[] and never enter it.

Repeats

for i in $(seq 20); do python -m pytest -n0 -p no:cacheprovider -q test/test_runtime_cleanup_regression.py || break; done
# 20/20 green
for i in $(seq 10); do python -m pytest -n 4 --dist loadgroup --max-worker-restart=2 -p no:cacheprovider -q \
  test/test_runtime_cleanup_regression.py test/test_acp_session_mcp.py test/test_acp_client.py || break; done
# 10/10 green (869 passed per run)

Shuffled order (seed=21361, ids of the file plus test_acp_session_mcp.py via @shuffled.txt): 167 passed, exit 0.

Forced condition — throwaway -p plugin, not committed:

# slow_rebuild.py
import time
from kiro_crew import agent as agent_mod
_real = agent_mod.rebuild_agent_config
def _slow_rebuild(*args, **kwargs):
    time.sleep(6.0)          # a loaded Windows shard at the seam the diagnosis names
    return _real(*args, **kwargs)
agent_mod.rebuild_agent_config = _slow_rebuild
# parent f07162523a (origin/main):
PYTHONPATH=src:/tmp/kc-trace python -m pytest test/test_runtime_cleanup_regression.py -p slow_rebuild -n0 -q
FAILED test/test_runtime_cleanup_regression.py::test_session_start_persists_new_descendants[load]              - TimeoutError
FAILED test/test_runtime_cleanup_regression.py::test_cancelled_descendant_scan_terminates_only_failed_session[load] - TimeoutError
2 failed, 3 passed

# this branch a3dab0e5d932, same plugin:
5 passed

Backend gate floor from the resolved profile: all green locally (scoped backend tests, black, isort, flake8, mypy, every scripts/check_*.py, docs lint, test_flake_pattern_ratchet.py 53 passed). Frontend gates not run: no website/ change. semgrep is not installed on this host; CI's SAST lane is the check for it.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision a3dab0e5d932d88de2a425ebc0fc03f8a68636e0 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a3dab0e5d932d88de2a425ebc0fc03f8a68636e0 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a3dab0e5d932d88de2a425ebc0fc03f8a68636e0 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the Windows [load] shard turns green over repeated CI runs. The PR's local proof comes from Linux only.

[DESIGN-REVIEWED] a3dab0e

Description read

[DESCRIPTION-READ] 3e1f8461c9c014e62347995d2cee16d0e74b4a15be953c02f5dc3dcb9770a13a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a3dab0e5d932d88de2a425ebc0fc03f8a68636e0 — this comment is updated in place on each push.

Review details

I found no defects in this PR's changes, which edit one test fixture. The test patches ensure_agent_materialized in session_mcp. That module imports the function by name and calls it at line 679, so the patch takes effect.

No findings.

[OPUS-REVIEWED] a3dab0e

Verdict parsed from the review's SHA-scoped output markers for commit a3dab0e5d932d88de2a425ebc0fc03f8a68636e0.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a3dab0e5d932d88de2a425ebc0fc03f8a68636e0: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a3dab0e5d932d88de2a425ebc0fc03f8a68636e0 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a3dab0e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a3dab0e5d932d88de2a425ebc0fc03f8a68636e0: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 11, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Check that the Windows [load] shard turns green over repeated CI runs — rebutted (as a gate), evidence recorded

On this head all 8 Backend Tests (Windows) shards passed in CI run 38135032690, including shard (2), the one both ledger issues (#18874, #18676) name.
The platform-independent proof is the forced condition: with rebuild_agent_config delayed 6 s the parent reds on exactly these two tests and this head stays green, so the hop that blew the 5 s budget on Windows is no longer on the [load] path at all.
The remaining [load] cost equals the [new] twins' (0.5 s on Linux, same hops), and the [new] variants have no Windows red on record.
Repeated Windows runs accrue on main after merge; a recurrence would reopen #18874/#18676, which the fix closes.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR fixes the Windows-only TimeoutError flake in the two [load] tests of test/test_runtime_cleanup_regression.py. The lifecycle fixture now stubs session_mcp.ensure_agent_materialized, so load_session's MCP-array hop no longer runs a full rebuild_agent_config() inside the 5 s budget. That rebuild is the root cause: the [new] twins pass mcp_servers=[] and never enter the hop. I confirmed that session_mcp imports the function by name and calls it at line 679, so the patch takes effect. This is the same seam the fixture already seals for pooled_session_servers and that other ACP tests stub, and no kiro-cli in this test reads the spec. The forced-condition proof makes rebuild take 6 s: the parent goes red on exactly the two [load] tests and this head stays green. All 8 Windows shards passed on this head. The Design-lane note has a posted disposition. This is a test-only change with no production code, base is main, checks are green, and there are no change requests. — crew-pr-reviewer (deep re-check)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky on Windows: test_session_start_persists_new_descendants[load] CancelledError Flaky on Windows: test_cancelled_descendant_scan_terminates[load]

2 participants