Skip to content

fix: leave a busy parent running when a completion delivery times out - #33184

Merged
bolichen97 merged 1 commit into
mainfrom
fix/22157-busy-parent-timeout-reset
Nov 8, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/22157-busy-parent-timeout-reset

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A sub-agent completion that cannot be delivered within 20 minutes no longer kills a parent that is still busy in its own long turn.

When a finished sub-agent's result waits longer than _ON_DONE_TIMEOUT (1200 s) to reach its parent, the delivery is given up and the parent session is reset. A cron job, channel or webhook run whose own turn keeps working for more than 20 minutes after a child finishes is killed mid-turn by that reset: the gateway logs completion injection timed out ... after 1200s, then runtime kill pid=<parent> ... reason=provider shutdown, and the run ends with an error.

Why it matters

Any unattended run that starts a short child and then keeps working (a long build, a code change and a push) loses its live turn 20 minutes after the child ends, and each later child's delivery kills the replacement process the same way. The reset was meant to clear a stuck injection prompt; a busy parent is not one.

Not a goal

What changed (motivation → approach → change)

A parent that is busy in its own turn now keeps running when a completion's delivery times out; a delivery that is itself stuck still gets the parent's session reset.

The timeout arm in subagent_manager/terminal.py reset the parent unconditionally. The cron and channel routes wait for the parent's session lease inside the 1200 s cap, so a long, healthy turn reached the same arm as a stuck injection. At that moment the two differ in who holds the lease: the timeout cancels the route first, and the route's finally cancels its own prompt and releases any lease it held before the arm runs. A turn still holding the lease is then the parent's own turn, or a channel member's lifetime lease. A permit that release handed to another queued delivery names no holder until that delivery runs.

The arm now asks sessions.turn_holder(parent_key). No holder: the session is reset as before, so a stuck provider is cleared and a queued delivery validates onto a fresh one. A holder: the parent is left running and the arm logs that. Either way the delivery is still reported failed through notify_injection_failed. The _ON_DONE_TIMEOUT row and the Two-Level Timeout section of docs/system-specs/modules/subagent.md, and the outer-cap row of docs/architecture/resource-protection.md, state the rule.

Backwards compatibility

Compatible: the reset is declined only while a turn holds the parent's session; every timeout with no holder resets exactly as before.

Removes nothing: no path, record, key or function is taken away; one reset now waits for no turn to be holding the session.

Tests

  • test/test_subagent_injection_timeout_reset.py drives the real SessionManager lease under the real terminal report, with only the provider process mocked.
    • test_a_parent_busy_with_its_own_turn_is_left_running[turn-lease|lifecycle-lease] runs the real cron route and locks in that the parent's session, provider and turn survive the timeout.
    • test_a_stuck_injection_is_reset_while_a_sibling_waits_for_the_parent locks in that a delivery stuck holding the lease is still reset when its release hands the permit to a queued sibling, and that the sibling gets a fresh provider. It passes on main too, which resets unconditionally: it guards the reset this change keeps.
    • test_a_stuck_injection_prompt_is_still_reset runs the same through the real cron route with no sibling.
  • test/test_subagent.py::TestOnDoneTimeout::test_injection_timeout_resets_parent_session: with no turn holding the parent, the arm still resets it.
  • Determinism: 20 repeats at -n0 (80 passed); 10 repeats at -n 4 --dist loadgroup --max-worker-restart=2 beside test_subagent_injection_live_turn.py, test_subagent_injection_session_reload.py and TestOnDoneTimeout, one of which lost every test at setup when a concurrent session on the shared temp root removed its pytest base directory and passed again with a private TMPDIR; one shuffled order (seed 25715, the same 14 tests). All green.

Proof: PROVEN test/test_subagent_injection_timeout_reset.py::test_a_parent_busy_with_its_own_turn_is_left_running[turn-lease], test/test_subagent_injection_timeout_reset.py::test_a_parent_busy_with_its_own_turn_is_left_running[lifecycle-lease] (head d949bdf8dd40)

Manual verification

N/A — unit coverage sufficient: the tests drive the real session lease, cron route and terminal report, and only the kiro-cli process is mocked.

Screenshots / video

N/A — no frontend change.

Related Issues

Closes #22157

Builds on #26275 by @rnoack1.

Sibling sweep

Swept: git grep -nE "sessions.reset(" -- src/kiro_crew/subagent_manager src/kiro_crew/slack/gateway_runtime/subagent_injection.py src/kiro_crew/slack/gateway_runtime/runtime_charge.py
Sibling: src/kiro_crew/slack/gateway_runtime/subagent_injection.py:1207 -- not affected -- every parent reset in this file runs under the route's own claim (the channel attempt timeout at 1207, the cron route's new-session reset at 1340 and its injection timeout at 1392) or already passes skip_if_busy=True (the idle cleanup at 1503)
Sibling: src/kiro_crew/slack/gateway_runtime/runtime_charge.py:293 -- not affected -- the prompt-busy and process-death resets in _inject_with_retry (293, 329, 368) run under the delivery's own claim
Sibling: src/kiro_crew/subagent_manager/terminal.py:1051 -- not affected -- resets the child run's own session during a reap, never the parent's
Sibling: src/kiro_crew/subagent_manager/run_lifecycle.py:176 -- not affected -- resets the child run's own session at its teardown, never the parent's

Pattern harvest

Not generalizable: the defect is one timeout arm resetting a session after its caller had already given up the lease; every other parent reset either runs under the route's own claim or already passes skip_if_busy.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

A sub-agent completion that waited out the 1200 s delivery cap reset
the parent session unconditionally, so a cron, channel or webhook
parent still busy in its own long turn was killed mid-turn. By the
time the timeout arm runs, the cancelled delivery route has released
any lease it held, so a turn that still holds the session is the
parent's own. The arm now resets only when no turn holds the session
(sessions.turn_holder) and logs the reset it declined. A stuck
injection is still reset, including when its release has just handed
the lease to a queued sibling delivery, which then gets a fresh
provider.

Builds on #26275.

Co-authored-by: Robert Noack <rnoack@amazon.com>
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ab8526680c1a2d699f2b526e5f84d7c70aff6ded touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of ab8526680c1a2d699f2b526e5f84d7c70aff6ded — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The one behaviour change targets the cause: the timeout arm reset the parent without checking whether the parent's own turn held the lease. Everything else is a log line and a spec update.

Not checked: whether reset(skip_if_busy=True) could replace the outside-lock turn_holder read. Its semaphore.locked() test would wrongly skip the hand-off case.

What this change ships

Inventory (3 items) — 3 justified

Intent: let an unattended run keep working when a finished child's result cannot reach it within 20 minutes. This is a FIX (#22157).

  1. When a completion delivery times out, the parent stays running if its own turn (or a channel member's lifetime lease) holds the session. A stuck delivery with no holder is still reset. — justified
  2. A gateway info log line records that the parent was left running after the timeout. — justified
  3. The _ON_DONE_TIMEOUT row, the Two-Level Timeout steps and the resource-protection outer-cap row now state the holder exception. — justified

[FIRST-PRINCIPLES-REVIEWED] ab85266

Description read

[DESCRIPTION-READ] 672bca7fb94af5fc9d0d2b0dc3f3368c039fabf923d2ff6e422c00a73aecc5e4

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ab8526680c1a2d699f2b526e5f84d7c70aff6ded — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the reset now depends on who holds the lease, which is the property that actually separates a stuck injection from a busy parent.

Notes

  • A parent turn that is itself wedged while holding the lease is no longer cleared by this arm. Before, the outer-cap reset killed it as a side effect. Say in the spec row which watchdog clears that case now.

[DESIGN-REVIEWED] ab85266

Description read

[DESCRIPTION-READ] dbd7fb9e23511df6038e1d0c42c5eeb75f4fd22c7b33a181485a0613c4e4cbde

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ab8526680c1a2d699f2b526e5f84d7c70aff6ded — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ab85266

Verdict parsed from the review's SHA-scoped output markers for commit ab8526680c1a2d699f2b526e5f84d7c70aff6ded.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ab8526680c1a2d699f2b526e5f84d7c70aff6ded false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ab8526680c1a2d699f2b526e5f84d7c70aff6ded and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ab85266

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ab8526680c1a2d699f2b526e5f84d7c70aff6ded false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 8, 2026 13:44
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 8, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (5 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: a completion injection that times out no longer resets the parent session when the parent's own turn holds it; the reset still runs when no turn holds the session. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/architecture/resource-protection.md, docs/system-specs/modules/subagent.md.

@bolichen97
bolichen97 merged commit f216ae2 into main Nov 8, 2026
98 checks passed
@bolichen97
bolichen97 deleted the fix/22157-busy-parent-timeout-reset branch November 8, 2026 14:08
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 8, 2026
buluoray added a commit that referenced this pull request Nov 8, 2026
…#33184)

A sub-agent completion that waited out the 1200 s delivery cap reset
the parent session unconditionally, so a cron, channel or webhook
parent still busy in its own long turn was killed mid-turn. By the
time the timeout arm runs, the cancelled delivery route has released
any lease it held, so a turn that still holds the session is the
parent's own. The arm now resets only when no turn holds the session
(sessions.turn_holder) and logs the reset it declined. A stuck
injection is still reset, including when its release has just handed
the lease to a queued sibling delivery, which then gets a fresh
provider.

Builds on #26275.

Original author: Bolin Chen
(cherry picked from commit f216ae2)
buluoray added a commit that referenced this pull request Nov 8, 2026
…#33184)

A sub-agent completion that waited out the 1200 s delivery cap reset
the parent session unconditionally, so a cron, channel or webhook
parent still busy in its own long turn was killed mid-turn. By the
time the timeout arm runs, the cancelled delivery route has released
any lease it held, so a turn that still holds the session is the
parent's own. The arm now resets only when no turn holds the session
(sessions.turn_holder) and logs the reset it declined. A stuck
injection is still reset, including when its release has just handed
the lease to a queued sibling delivery, which then gets a fresh
provider.

Builds on #26275.

Original author: Bolin Chen
(cherry picked from commit f216ae2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A completion delivery that times out after 1200 s resets a busy parent, killing its live turn

2 participants