Skip to content

test(subagent): end a channel injection attempt whose deadline hit the reload - #35657

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/31835-timeout-reload-flake
Nov 20, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/31835-timeout-reload-flake

Conversation

@dodgeblaster

@dodgeblaster dodgeblaster commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A sub-agent completion injected into a channel parent always ends its attempt at INJECTION_TIMEOUT, even when the deadline fires while the attempt is re-loading a lost session, and test_a_timeout_after_a_reload_reads_the_session_that_received_the_completion stops hanging on slow runners.

The test hung to its 30 s ceiling on a macOS runner (#31835, run 42289849622). Its log shows the session-not-found reload, then no "attempt 1/2 timed out" warning: the patched 1.0 s attempt deadline never ended the attempt, and the second prompt stayed parked. The same thing can happen in production: a completion whose prompt hits a lost session, with the deadline firing during the reload, runs the reloaded prompt with no deadline.

Why it matters

The test is a recurring red on the macOS shard of unrelated PRs. In the product, the channel route's per-attempt deadline is what stops a hung completion turn from holding the parent's session; losing it leaves that turn bounded only by the 1200 s outer cap.

Not a goal

  • Changing how session teardown treats a cancellation in general (metrics/sessions.py absorbs one during its crumb hand-off on purpose, so post-pop cleanup runs).
  • Other callers that reset a session inside a deadline (see Sibling sweep).
  • Changing INJECTION_TIMEOUT or the attempt count.

What changed (motivation → approach → change)

  • Symptom: the attempt deadline was lost when it fired during the in-attempt reload, so the reloaded prompt ran unbounded and the test hung to its 30 s ceiling.
  • Root cause: ending the lost session in sessions.reset() calls record_session_ended, whose _unlink_generations absorbs a CancelledError at its shielded crumb hop so post-pop cleanup runs. Nothing re-raised it, so the caller's spent asyncio.timeout never ended the attempt.
  • Root fix, now on main: fix(session): re-raise a cancel that reset's end record absorbed #37984 (fix(session): re-raise a cancel that reset's end record absorbed) makes reset() raise CancelledError after its cleanup. That fix is in the shared code every timeout-bounded caller goes through. This PR used to add a second guard in _inject_with_retry (a cancelling() comparison after the reload). After rebasing onto fix(session): re-raise a cancel that reset's end record absorbed #37984 that guard can never fire, because the cancel no longer survives reset(), so it is dropped and the PR carries no production change.
  • Tests: the reload tests fired a real fixed deadline against the reload's worker-thread hops, which is the race that hung on the macOS runner. They now fire the attempt deadline on a signal (_AttemptDeadline: the patched wait_for runs the attempt under an asyncio.timeout(None) that the parked prompt reschedules to now). test: stop the reload-resend test racing its own injection timeout #37354 had changed test_a_reload_inside_the_retry_onto_a_new_session_sends_the_full_completion to raise TimeoutError directly. This PR replaces that with the signalled deadline, so a real cancellation reaches the route again. A new test holds the reload's crumb hop on its worker thread, fires the deadline from inside it, and checks that no prompt is sent under a deadline that has already fired.
  • Spec: docs/system-specs/modules/subagent.md § Prompt-Busy Recovery names the reset() re-raise and the test that pins it.

Backwards compatibility

Compatible: test and spec only; no production code changes.

Removes nothing: no key, record, path or public function is removed or tightened.

Tests

  • test_a_deadline_that_lands_inside_the_reload_still_ends_the_attempt (new): the deadline fires while the reload's crumb hop is in flight. Attempt 1 must end before its reload claims a session, attempt 2 delivers the completion on the one session it claims (sess-a then sess-b, sess-c never built), and no prompt is sent under a fired deadline. Mutation check: with reset()'s re-raise (fix(session): re-raise a cancel that reset's end record absorbed #37984) removed, this test fails.
  • test_a_reload_inside_the_retry_onto_a_new_session_sends_the_full_completion and test_a_timeout_after_a_reload_reads_the_session_that_received_the_completion: same assertions, deadline fired by the parked prompt instead of a clock.

Proof: not provable -- test-only change

Determinism proof (Linux, Python 3.12, head 38e61eb):

  • Repeats: 40 runs of the whole file at -n0, each in a shuf order of the 7 node ids, run as 4 to 8 concurrent streams while 24 busy-loop processes loaded the host: 40/40 green (280/280 tests).
  • Timing paths exercised: the deadline firing in the first prompt before the reload, inside the reset's crumb hop on its worker thread, and in the reloaded session's prompt after a reload. Also the route's 2 s pause between attempts, run as written.
  • Gates on the changed files: black, isort, flake8, file line cap, comment-history, brand, test_flake_pattern_ratchet.py.

Manual verification

N/A: the defect is a cancellation race, and the forced-condition tests above reproduce it deterministically.

Related Issues

Fixes #31835

Sibling sweep

Swept: git grep -n "acp_error_is_session_not_found(" -- src/kiro_crew, then each hit's reset inside a deadline
Sibling: src/kiro_crew/slack/gateway_runtime/cron_turn.py and src/kiro_crew/slack/handler_runtime/turn.py call the same sessions.reset(), so #37984's re-raise covers them; not changed here.

Pattern harvest

Rule candidate: review-prompt
Pattern: a coroutine that absorbs CancelledError (to finish its own cleanup) and returns normally leaves every caller's asyncio.timeout/wait_for deadline spent; a caller that keeps working afterwards must compare task.cancelling() and re-raise.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@dodgeblaster

Copy link
Copy Markdown
Contributor Author

Sibling scope, open question for a maintainer (no change made for it here).

This PR fixes the completion injection route only. The same mechanism reaches two other callers on main:

  • slack/gateway_runtime/cron_turn.py (~line 1101): a session-not-found cron retry calls sessions.reset() and then retry(job), inside the job's asyncio.wait_for per-wake deadline (cron_service/service/run_execution.py:281). A deadline that fires during the reset's crumb hand-off is absorbed, and the retried turn runs past the job's budget.
  • slack/handler_runtime/turn.py (~line 1251): a user's Stop (a task cancel) that lands in the same reset hop is absorbed, and the turn carries on.

The common cause is metrics/sessions.py::_unlink_generations absorbing CancelledError without re-delivering it, which is deliberate so post-pop cleanup in destroy/reset finishes (test_a_cancelled_end_lets_its_callers_post_pop_cleanup_run).

Options:

  1. Keep this PR narrow and track the class in a separate issue.
  2. Add the same cancelling() check to the cron retry here.
  3. Re-raise the absorbed cancellation at the end of the session teardown entry points (reset, destroy, ...) once their cleanup has run. That covers every caller, but it changes session lifecycle behaviour that is pinned today.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 38e61ebb842701a419959ee882920553cfbadb96 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 38e61ebb842701a419959ee882920553cfbadb96 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

The description says this PR changes production code, but it doesn't: the diff only touches tests and the spec. The fix it relies on, reset() re-raising, is already on main, so nothing in the design blocks.

Notes

  • Phantom description: "_inject_with_retry records the task's cancelling() count on entry… raises CancelledError if the count grew" and "Sibling: runtime_charge.py -- changed here" have no matching code in the patch. The fix actually comes from main's existing end_cancelled re-raise in session_lifecycle_parts/teardown.py:333/:481. As written, the squash commit would credit the fix to the wrong site. Rewrite "What changed" and the sibling line to match the diff (leave Goal/Why/Not-a-goal alone, since they are frozen). Also consider retitling to test:, because this PR adds no fix.
  • The new spec sentence correctly credits SessionManager.reset(). That contradicts the description's root-cause and change bullets, which is one more reason to reconcile them.
  • The sibling sweep names cron_turn.py and handler_runtime/turn.py as still absorbing the cancellation. Since only reset re-raises today (per the record_session_ended docstring), check whether those paths go through reset. If they don't, file the issue the sweep says has not been filed yet.

[DESIGN-REVIEWED] 38e61eb

Description read

[DESCRIPTION-READ] 7733a82740970199b44cbc6beffc70123df8a1a00d5ebc89736e6aa0e4773644

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 38e61ebb842701a419959ee882920553cfbadb96 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 38e61ebb842701a419959ee882920553cfbadb96 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 38e61eb

Verdict parsed from the review's SHA-scoped output markers for commit 38e61ebb842701a419959ee882920553cfbadb96.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 38e61ebb842701a419959ee882920553cfbadb96 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — 🔴 changes requested (blocking)

GPT 6.1 found at least one blocking issue that must be resolved before merging 38e61ebb842701a419959ee882920553cfbadb96. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- test/test_ci_backend_lint_side_by_side.py:333 -- Failure cleanup leaves the subprocess tree running

proc.kill()
Trigger: A bounded wait or proc.communicate() times out while the mypy stub runs sleep 600; the finally block kills only the outer Bash process.
Outcome: SIGKILL bypasses Bash’s forwarding traps, leaving the wrapper and sleeper running after the failed test, until the sleeper’s ten-minute deadline.
Anchor: no-test-side-effects
Fix: Start the subprocess in an owned process group; unconditionally terminate that group during cleanup, including when its leader has exited, and reap the outer process.

[BLOCK-MERGE] 38e61eb
[GPT-REVIEWED] 38e61eb

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

I didn't adjudicate any blocking findings (the input block was empty). I flagged the one fenced finding, F1. A FLAG does not unblock the merge: F1 keeps blocking until a repository writer posts an /ai-review override.

F1 (fenced) — LOW in practice, not unbounded. The leak only happens when the test has already failed: wait_until or communicate has to time out first (test/test_ci_backend_lint_side_by_side.py:322-332). The process left running is the stub's exec sleep 600 (test/test_ci_backend_lint_side_by_side.py:59), which exits on its own after ten minutes. It touches no files and holds no credentials, and nothing else cleans it up in the meantime. The real fix (an owned process group plus a group kill in finally) is cheap, so a human could accept the risk or ask for the fix.

[ADJUDICATION] 38e61ebb842701a419959ee882920553cfbadb96 total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 38e61ebb842701a419959ee882920553cfbadb96

[ADJUDICATION-FENCED] 38e61ebb842701a419959ee882920553cfbadb96 fenced=1 flagged=1
FLAG F1 test/test_ci_backend_lint_side_by_side.py:333 -- Reachable only after the test has already gone red on a bounded-wait timeout (lines 322-332), and the orphan is the stub's `exec sleep 600` (line 59): it touches nothing outside tmp, holds no credentials, and exits by itself within ten minutes, so the residual is a short, bounded leak on a run that is already failing.
[GPT-ADJUDICATED-FENCED] 38e61ebb842701a419959ee882920553cfbadb96

🏷️ Fenced finding(s) machine-flagged as likely edge case

The security fence keeps these findings blocking regardless of adjudication; the only clearance path is a human override. The annotate-only pass judged the condition combination each one requires extreme, and pre-drafted the override rationale below. A repository writer must independently verify a rationale before posting it — it is machine-authored, and a wrong override on a security-class finding ships exactly the class the fence exists to stop.

  • F1 test/test_ci_backend_lint_side_by_side.py:333 — Reachable only after the test has already gone red on a bounded-wait timeout (lines 322-332), and the orphan is the stub's exec sleep 600 (line 59): it touches nothing outside tmp, holds no credentials, and exits by itself within ten minutes, so the residual is a short, bounded leak on a run that is already failing.

    /ai-review override gpt 38e61ebb842701a419959ee882920553cfbadb96 <class>: Reachable only after the test has already gone red on a bounded-wait timeout (lines 322-332), and the orphan is the stub's exec sleep 600 (line 59): it touches nothing outside tmp, holds no credentials, and exits by itself within ten minutes, so the residual is a short, bounded leak on a run that is already failing.
    

Replace <class> with the class that records what you did. These findings are real, so for an accepted risk use owner-decision (link the decision comment in the reason) or deferred #<N> (a tracked follow-up issue); false-positive only if you verified the finding is wrong. Not not-applicable: that class files the override as a rule mismatch in the audit.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 38e61ebb842701a419959ee882920553cfbadb96 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 12, 2026
@dodgeblaster
dodgeblaster force-pushed the fix/31835-timeout-reload-flake branch from 6c4ac90 to 6c2428f Compare November 12, 2026 05:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@dodgeblaster
dodgeblaster marked this pull request as ready for review November 12, 2026 18:53
@dodgeblaster
dodgeblaster requested review from a team and KiroCrewCaptain as code owners November 12, 2026 18:53
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 12, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own.

Test-and-spec-only change. The production behaviour the goal names is already on main: SessionManager.reset() re-raises the cancellation its end record absorbed (teardown.py:333, teardown.py:481-485; pinned by test_session_reset_end_record_cancel.py), and _inject_with_retry calls reset() inside the attempt, so the deadline is honoured. The PR makes the two reload tests deterministic: a wrapper around asyncio.wait_for runs each attempt under asyncio.timeout(None) and fires it on cue with reschedule(now), which removes the 1.0 s race behind the macOS flake. It also adds a test that holds the reload's crumb hop and fires the deadline inside it. That test fails without the re-raise (the second send would run after the deadline and sent_after_deadline would read [False, True]) and passes with it (attempt 1 times out, attempt 2 claims sess-b, sess-c stays unclaimed). The spec sentence matches the code. The description is stale: it claims a cancelling() check in runtime_charge.py that is not in the diff, and it misstates the expected session order. The code is right, so this goes in the notes.

Tier: 2, held for a person: design-doc-gate: A.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 38e61ebb84 · 150 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

The reload tests in test_subagent_injection_channel_timeout_resend
raced a fixed injection deadline against the reload's worker-thread
hops; on a loaded macOS runner the deadline landed in the reset's
metrics crumb hop and the test hung to its 30s ceiling.

The cancel that hop absorbed is now re-raised by SessionManager.reset()
(#37984). These tests now fire the attempt deadline on a signal (the
parked prompt) instead of a clock, and a new test fires it inside the
held crumb hop and checks that no prompt is sent under a deadline that
has already fired. The subagent spec names that guarantee.

Fixes #31835
@dodgeblaster
dodgeblaster force-pushed the fix/31835-timeout-reload-flake branch from 6c2428f to 38e61eb Compare November 20, 2026 03:54
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 20, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 20, 2026 05:30

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review checked head 38e61eb. The diff changes only test/test_subagent_injection_channel_timeout_resend.py and one sentence in docs/system-specs/modules/subagent.md. The test file passes 7/7. With every executor hop forced 0.25 s or 0.5 s slower, the merge-base copy of the reload tests fails and the head copy passes, which is the #31835 flake shape. A mutation that disables the cancel re-raise in reset() (teardown.py:481) turns the new route-level test red. Every wait is bounded by _HANG_CEILING_S, and the wait_for patch is scoped to a with block. The new spec sentence matches main.

Done well: _AttemptDeadline fires a real asyncio.timeout at a point the test chooses. The real TimeoutError path, reset and retry all still run, and the test no longer races the host clock.

The design is signed off under the standing maintainer sign-off for changes by team members.

Optional:

  • The title says fix(subagent) and the body describes a cancelling() check in runtime_charge.py. That check is not in this diff. The production re-raise shipped in #37984, and the only commit is test(subagent): .... Consider retitling to test(subagent): ... and updating "What changed", "Tests" (the test asserts sess-a then sess-b, with sess-c unclaimed) and the Proof line (it names an older head). The frozen sections can stay as they are.
  • The sibling sweep lists cron_turn.py and handler_runtime/turn.py as unfixed. Both call sessions.reset(), which already re-raises the absorbed cancel on main, so no follow-up issue is needed.
  • In the new test, only sent_after_deadline == [False, False] catches the regression. Consider keeping the "late" message on that assertion alone, so a later cleanup does not drop the one assertion that pins the fix.

@KiroCrewCaptain
KiroCrewCaptain merged commit 2f1d420 into main Nov 20, 2026
100 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/31835-timeout-reload-flake branch November 20, 2026 05:30
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 20, 2026
@dodgeblaster dodgeblaster changed the title fix(subagent): end a channel injection attempt whose deadline hit the reload test(subagent): end a channel injection attempt whose deadline hit the reload Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_timeout_after_a_reload_reads_the_session_that_received_the_completion hits its 30s hang ceiling on a macOS runner

2 participants