Repository navigation
test(subagent): end a channel injection attempt whose deadline hit the reload - #35657
Conversation
|
Sibling scope, open question for a maintainer (no change made for it here). This PR fixes the completion injection route only. The same mechanism reaches two other callers on main:
The common cause is Options:
|
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS The description says this PR changes production code, but it doesn't: the diff only touches tests and the spec. The fix it relies on, Notes
[DESIGN-REVIEWED] 38e61eb Description read[DESCRIPTION-READ] 7733a82740970199b44cbc6beffc70123df8a1a00d5ebc89736e6aa0e4773644 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — 🔴 changes requested (blocking)GPT 6.1 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- test/test_ci_backend_lint_side_by_side.py:333 -- Failure cleanup leaves the subprocess tree running
[BLOCK-MERGE] 38e61eb Adjudication (Opus 5.5) — is blocking on each finding proportionate?I didn't adjudicate any blocking findings (the input block was empty). I flagged the one fenced finding, F1. A FLAG does not unblock the merge: F1 keeps blocking until a repository writer posts an F1 (fenced) — LOW in practice, not unbounded. The leak only happens when the test has already failed: 🏷️ Fenced finding(s) machine-flagged as likely edge caseThe security fence keeps these findings blocking regardless of adjudication; the only clearance path is a human override. The annotate-only pass judged the condition combination each one requires extreme, and pre-drafted the override rationale below. A repository writer must independently verify a rationale before posting it — it is machine-authored, and a wrong override on a security-class finding ships exactly the class the fence exists to stop.
Replace False positive or not applicable? A repository writer can comment: |
6c4ac90 to
6c2428f
Compare
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own. Test-and-spec-only change. The production behaviour the goal names is already on main: SessionManager.reset() re-raises the cancellation its end record absorbed (teardown.py:333, teardown.py:481-485; pinned by test_session_reset_end_record_cancel.py), and _inject_with_retry calls reset() inside the attempt, so the deadline is honoured. The PR makes the two reload tests deterministic: a wrapper around asyncio.wait_for runs each attempt under asyncio.timeout(None) and fires it on cue with reschedule(now), which removes the 1.0 s race behind the macOS flake. It also adds a test that holds the reload's crumb hop and fires the deadline inside it. That test fails without the re-raise (the second send would run after the deadline and sent_after_deadline would read [False, True]) and passes with it (attempt 1 times out, attempt 2 claims sess-b, sess-c stays unclaimed). The spec sentence matches the code. The description is stale: it claims a cancelling() check in runtime_charge.py that is not in the diff, and it misstates the expected session order. The code is right, so this goes in the notes. Tier: 2, held for a person: design-doc-gate: A. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
The reload tests in test_subagent_injection_channel_timeout_resend raced a fixed injection deadline against the reload's worker-thread hops; on a loaded macOS runner the deadline landed in the reset's metrics crumb hop and the test hung to its 30s ceiling. The cancel that hop absorbed is now re-raised by SessionManager.reset() (#37984). These tests now fire the attempt deadline on a signal (the parked prompt) instead of a clock, and a new test fires it inside the held crumb hop and checks that no prompt is sent under a deadline that has already fired. The subagent spec names that guarantee. Fixes #31835
6c2428f to
38e61eb
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
This review checked head 38e61eb. The diff changes only test/test_subagent_injection_channel_timeout_resend.py and one sentence in docs/system-specs/modules/subagent.md. The test file passes 7/7. With every executor hop forced 0.25 s or 0.5 s slower, the merge-base copy of the reload tests fails and the head copy passes, which is the #31835 flake shape. A mutation that disables the cancel re-raise in reset() (teardown.py:481) turns the new route-level test red. Every wait is bounded by _HANG_CEILING_S, and the wait_for patch is scoped to a with block. The new spec sentence matches main.
Done well: _AttemptDeadline fires a real asyncio.timeout at a point the test chooses. The real TimeoutError path, reset and retry all still run, and the test no longer races the host clock.
The design is signed off under the standing maintainer sign-off for changes by team members.
Optional:
- The title says
fix(subagent)and the body describes acancelling()check in runtime_charge.py. That check is not in this diff. The production re-raise shipped in #37984, and the only commit istest(subagent): .... Consider retitling totest(subagent): ...and updating "What changed", "Tests" (the test asserts sess-a then sess-b, with sess-c unclaimed) and the Proof line (it names an older head). The frozen sections can stay as they are. - The sibling sweep lists cron_turn.py and handler_runtime/turn.py as unfixed. Both call
sessions.reset(), which already re-raises the absorbed cancel on main, so no follow-up issue is needed. - In the new test, only
sent_after_deadline == [False, False]catches the regression. Consider keeping the "late" message on that assertion alone, so a later cleanup does not drop the one assertion that pins the fix.
Problem / Motivation
Goal: A sub-agent completion injected into a channel parent always ends its attempt at
INJECTION_TIMEOUT, even when the deadline fires while the attempt is re-loading a lost session, andtest_a_timeout_after_a_reload_reads_the_session_that_received_the_completionstops hanging on slow runners.The test hung to its 30 s ceiling on a macOS runner (#31835, run 42289849622). Its log shows the session-not-found reload, then no "attempt 1/2 timed out" warning: the patched 1.0 s attempt deadline never ended the attempt, and the second prompt stayed parked. The same thing can happen in production: a completion whose prompt hits a lost session, with the deadline firing during the reload, runs the reloaded prompt with no deadline.
Why it matters
The test is a recurring red on the macOS shard of unrelated PRs. In the product, the channel route's per-attempt deadline is what stops a hung completion turn from holding the parent's session; losing it leaves that turn bounded only by the 1200 s outer cap.
Not a goal
metrics/sessions.pyabsorbs one during its crumb hand-off on purpose, so post-pop cleanup runs).INJECTION_TIMEOUTor the attempt count.What changed (motivation → approach → change)
sessions.reset()callsrecord_session_ended, whose_unlink_generationsabsorbs aCancelledErrorat its shielded crumb hop so post-pop cleanup runs. Nothing re-raised it, so the caller's spentasyncio.timeoutnever ended the attempt.fix(session): re-raise a cancel that reset's end record absorbed) makesreset()raiseCancelledErrorafter its cleanup. That fix is in the shared code every timeout-bounded caller goes through. This PR used to add a second guard in_inject_with_retry(acancelling()comparison after the reload). After rebasing onto fix(session): re-raise a cancel that reset's end record absorbed #37984 that guard can never fire, because the cancel no longer survivesreset(), so it is dropped and the PR carries no production change._AttemptDeadline: the patchedwait_forruns the attempt under anasyncio.timeout(None)that the parked prompt reschedules to now). test: stop the reload-resend test racing its own injection timeout #37354 had changedtest_a_reload_inside_the_retry_onto_a_new_session_sends_the_full_completionto raiseTimeoutErrordirectly. This PR replaces that with the signalled deadline, so a real cancellation reaches the route again. A new test holds the reload's crumb hop on its worker thread, fires the deadline from inside it, and checks that no prompt is sent under a deadline that has already fired.docs/system-specs/modules/subagent.md§ Prompt-Busy Recovery names thereset()re-raise and the test that pins it.Backwards compatibility
Compatible: test and spec only; no production code changes.
Removes nothing: no key, record, path or public function is removed or tightened.
Tests
test_a_deadline_that_lands_inside_the_reload_still_ends_the_attempt(new): the deadline fires while the reload's crumb hop is in flight. Attempt 1 must end before its reload claims a session, attempt 2 delivers the completion on the one session it claims (sess-athensess-b,sess-cnever built), and no prompt is sent under a fired deadline. Mutation check: withreset()'s re-raise (fix(session): re-raise a cancel that reset's end record absorbed #37984) removed, this test fails.test_a_reload_inside_the_retry_onto_a_new_session_sends_the_full_completionandtest_a_timeout_after_a_reload_reads_the_session_that_received_the_completion: same assertions, deadline fired by the parked prompt instead of a clock.Proof: not provable -- test-only change
Determinism proof (Linux, Python 3.12, head 38e61eb):
-n0, each in ashuforder of the 7 node ids, run as 4 to 8 concurrent streams while 24 busy-loop processes loaded the host: 40/40 green (280/280 tests).test_flake_pattern_ratchet.py.Manual verification
N/A: the defect is a cancellation race, and the forced-condition tests above reproduce it deterministically.
Related Issues
Fixes #31835
Sibling sweep
Swept: git grep -n "acp_error_is_session_not_found(" -- src/kiro_crew, then each hit's reset inside a deadline
Sibling:
src/kiro_crew/slack/gateway_runtime/cron_turn.pyandsrc/kiro_crew/slack/handler_runtime/turn.pycall the samesessions.reset(), so #37984's re-raise covers them; not changed here.Pattern harvest
Rule candidate: review-prompt
Pattern: a coroutine that absorbs CancelledError (to finish its own cleanup) and returns normally leaves every caller's asyncio.timeout/wait_for deadline spent; a caller that keeps working afterwards must compare task.cancelling() and re-raise.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.