Skip to content

fix(autopilot): cancel a halted stage's subagents instead of orphaning them - #14813

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
jshyeung:fix/autopilot-halt-cancels-stage-children
Sep 29, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
jshyeung:fix/autopilot-halt-cancels-stage-children

Conversation

@jshyeung

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When autopilot halts a stage, the stage's subagents stop too instead of running on as orphans.

When a stage's subagent wait is exhausted, _halt_plan() only switched auto-run off. The stage turn timing out, or failing with an internal error, did the same. In all three cases the stage's running and queued children kept going until their own subagent_timeout_secs (3h by default). They held concurrency slots and did work nobody would collect, and the dashboard felt hung until they were reaped. Only the explicit Cancel button had scoped cancellation.

Why it matters

Every autopilot user whose stage runs out of budget leaks its subagents. With a small subagent_auto_max, a few orphaned stages starve new work for hours.

Not a goal

What changed (motivation → approach → change)

Symptom (orphaned children after a halt) → root cause (the halt paths never called the scoped cancellation that explicit Cancel uses) → change in src/kiro_crew/dashboard/chat_orchestrator.py:

  • New _cancel_exhausted_stage_subagents runs on the wait-exhausted, stage-turn-timeout and stage-turn-error halts. It uses the same order as explicit Cancel: reserve every parent scope first, then cancel_for_boundary(retain_scope=True). That way queued stage work can't be admitted during the task-store write. If the reservation is refused, it falls back to retain_scope=False, as explicit Cancel does.
  • The scope is captured off the still-armed boundary, before _preserve_interrupted_stage().
  • Every manager call is guarded. A failure is logged and counted, and never skips the halt message, the audit event or the stage preservation. CancelledError still propagates.
  • The halt waits at most 60s for its cancel (_HALT_CANCEL_BUDGET_SECS). A live child's reap awaits report delivery, which can take up to _ON_DONE_TIMEOUT (1200s) per parent. A cancel still running after 60s continues on a tracked background task. It is not cancelled, because cutting it would abandon a kill already in progress.
  • The cancel returns an outcome (_HaltCancelOutcome). The halt message is worded from that outcome: stopped, pending, refused, failed, or still cancelling. It only offers "Send Go to continue" when every child was confirmed stopped. Otherwise it tells the user to stop the rest first, on all three halt paths.
  • Auto-run is switched off before the cancel awaits. A plan Cancel that lands during the await suppresses the Go guidance.

Backwards compatibility

Compatible: nothing is rejected, renamed or removed. The three halts now also cancel the stage's children, and their messages carry one or two extra sentences.

Tests

  • test/test_autopilot_halt_cancels_real_manager.py (new) uses a real SubagentManager on the durable task queue. The stage's running child is revoked and reaped. Its queued child is cancelled in the store. Another stage's queued row under the same parent is untouched. No hold is left behind. A second test checks that during the store write the manager refuses to admit the stage's queued rows. It fails when the reservation and hold are removed.
  • test/test_autopilot_stage_completion_handoff.py:
    • wait-exhaustion cancels exactly this stage's scope with the hold on;
    • a raising cancel still halts the plan;
    • reservation and probe failures are counted, not raised;
    • auto-run is off before the cancel awaits;
    • a cancel that never resolves doesn't hold the halt back past the budget.
  • test/test_autopilot_wave_wait_event.py:
    • Go after a wait-exhaustion halt runs Stage 2, not Stage 1 again;
    • the timeout and error halts cancel the stage's children;
    • both warn when a child couldn't be confirmed stopped.
  • Every fix was checked by breaking it, one at a time: removing the cancel, removing the hold, reverting the N1/N3/N4 guards, removing the 60s bound, and dropping the residual warning. Each breakage failed at least one test.
  • Local run: 4378 passed, 3 skipped across every test file touching the orchestrator, stage boundaries or boundary cancellation, plus test/test_subagent.py. The backend gates pass except two host-only issues that fail the same way on clean origin/main: a mypy error on the Linux-only PidfdChildWatcher in cli.py, and two test_cli.py::TestDoctorStt tests.

Manual verification

N/A: the unit tests cover the halt paths end to end through _stage_loop, and one test uses a real manager and task queue. I have not reproduced it live on a running gateway.

Related Issues

Fixes #14715

Possible conflict: open PR #13992 ("reconcile stage timeouts") also edits chat_orchestrator.py.

Pattern harvest

Rule candidate: review-prompt
Pattern: a halt, abort or timeout path that stops the controller but not the work it spawned. Every stop path should call the same scoped cancellation as the explicit-cancel path.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines. Local cross-model review: GPT-5.6-Sol, Opus 4.8 and Opus 5.5. Opus 4.8 reviewed the tree before the 60s budget and the residual warning were added; its one advisory finding is fixed in this commit.
  • Documentation updated (if applicable). N/A
  • No secrets, credentials, or internal references in the diff

@jshyeung
jshyeung requested a review from a team as a code owner September 28, 2026 20:04
@jshyeung
jshyeung requested a review from kyleseaman September 28, 2026 20:04
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 28, 2026
@jshyeung

Copy link
Copy Markdown
Contributor Author

Intent: When autopilot halts a stage (subagent wait exhausted, stage-turn timeout, or stage-turn error), cancel that stage's running and queued subagents instead of leaving them to run until their own timeout, and word the halt message from what the cancel actually did.
Not a goal: changing the reaper, or covering the other halts (subagent-check-failed, auth-required delivery failure) that can still leave children running.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 28, 2026
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of edd498f457fb1e2b1f3ccb6d39b0e81b08cdbfac via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

"Fixes #14715" closes the issue while two halt paths still orphan children — confirm the deferred follow-up is tracked before merge.

[DESIGN-REVIEWED] edd498f

Description read

[DESCRIPTION-READ] bd4853a3abe4a1e8b5fa8e0f07a6a7884574ce21beb5ad01e8d5399d461f2e2d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed edd498f457fb1e2b1f3ccb6d39b0e81b08cdbfac via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] edd498f

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of edd498f457fb1e2b1f3ccb6d39b0e81b08cdbfac via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Two halts still orphan children — "subagent check failed" (chat_orchestrator.py:933,986) and delivery failure (line 897); verify the declared follow-up actually gets filed.

What this change ships

Inventory (10 items) — 10 justified

Intent: FIX — when autopilot halts a stage (#14715), the stage's subagents stop too instead of running on for up to 3h as orphans.

  1. Wait-exhausted halt now cancels the stage's running and queued subagents — justified
  2. Stage-turn-timeout halt now cancels the stage's subagents — justified
  3. Stage-turn-error halt now cancels the stage's subagents — justified
  4. Halt messages now state the real cancel outcome (stopped / refused / failed / pending / still finishing) — justified
  5. "Send Go to continue" guidance is offered only when every child was confirmed stopped; otherwise the halt says to stop them first — justified
  6. A halt waits at most 60s for its cancel; a slower one finishes on a tracked background task (_ON_DONE_TIMEOUT = 1200.0 verified at subagent.py:1019) — justified
  7. A Stop landing during that wait still publishes the notice, audit event and stage preservation — justified
  8. Auto-run switches off before the cancel awaits; a plan Cancel in that window suppresses the Go line — justified
  9. Explicit Cancel's per-parent sweep extracted into shared _cancel_boundary_parents (2 pre-existing consumers reuse it; no second spelling) — justified
  10. autopilot.md documents the halt-cancel behaviour and the pre-existing stage-turn ceiling in the same commit — justified

Checks run: cause-level per the description and diff (the halts never called the scoped cancellation explicit Cancel uses; this wires them to it). Siblings counted via _halt_plan call sites: the two unfixed ones are declared under ## Not a goal — accepted-and-deferred, not a demand. No test or comment pinning the old messages is deleted (results may be incomplete grep: source only). No RFC needed: the shape is a fix an issue pins, and no default a user chose is changed.

[FIRST-PRINCIPLES-REVIEWED] edd498f

Description read

[DESCRIPTION-READ] e2d4c9245b7d4ce0c119bf27e3baebbceaef4d536245914eebc95587e2554ac2

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed edd498f457fb1e2b1f3ccb6d39b0e81b08cdbfac via the fork AI-review pipeline; updated in place on each push.

Review details

Nothing blocks; one advisory — the _HaltCancelOutcome.refused docstring contradicts what the field actually carries.

FINDING — src/kiro_crew/dashboard/chat_orchestrator.py:936 — the added docstring states "refused names parents (or the reservation)", but the only per-parent producer is progress.refused.append(reason) in _cancel_boundary_parents, which appends the cap/durable-queue reason string returned by boundary_cancellation_pending_reason, never parent_key — and the consumer _halt_cancel_notice reads it back as a reason (hold ({reason})), so the struct's documented contract disagrees with both its producer and its consumer and the next reader will format it as a parent key → Fix: reword that docstring line to say refused holds the refusal reasons (cap or durable-queue), not parent keys.

[OPUS-REVIEWED] edd498f

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 28, 2026
@jshyeung
jshyeung force-pushed the fix/autopilot-halt-cancels-stage-children branch from 12333b7 to c4ec787 Compare September 29, 2026 02:24
@jshyeung

Copy link
Copy Markdown
Contributor Author

GPT 5.6 (blocking): real-manager tests leak the SubagentManager -- fixed in c4ec787. Both tests in test/test_autopilot_halt_cancels_real_manager.py now close the manager in a finally block (cancel_all + close), including when an assertion fails partway, and assert the task queue is closed afterwards.

@jshyeung

Copy link
Copy Markdown
Contributor Author

Opus 5 (advisory): Stop during the 60s halt-cancel wait dropped the halt notice -- fixed in c4ec787. _cancel_exhausted_stage_subagents now takes a publish callback holding each halt's notice, SEL audit event and (timeout/error halts) stage preservation, and calls it exactly once on every path. On CancelledError during the bounded wait it publishes with the unfinished outcome (worded as still finishing in the background, no clean Go line) and then re-raises; the tracked background cancel is not cancelled. New parametrized test TestAStopDuringTheHaltCancelKeepsTheNotice covers all three halts; reverting the handler fails all three.

@jshyeung

Copy link
Copy Markdown
Contributor Author

Opus 5 + design review: docs/system-specs/modules/autopilot.md stale -- fixed in c4ec787. Step 6 now documents the stage-turn ceiling halt and the error halt, and a new paragraph under step 7 describes the halt-time cancel: reserve-then-cancel with retained hold and no-hold fallback, guarded manager calls, the 60s bound with background continuation, outcome-worded notice, and when the Go line is offered.

@jshyeung

Copy link
Copy Markdown
Contributor Author

Design review: halt cancel duplicates the explicit-Cancel loop (and suggestion to extract it for future halts) -- done in c4ec787. The per-parent cancel-then-probe sequence is now a single helper, _cancel_boundary_parents, used by explicit Cancel (guarded=False: call shape, probe order and exception propagation unchanged; test_plan_cancel_race.py passes untouched) and by the halt (guarded=True: failures logged and counted). The remaining halts that still leave children running (subagent-check failure, auth-required delivery failure) can reuse it; they stay out of scope here as noted in the PR body.

@jshyeung

Copy link
Copy Markdown
Contributor Author

First Principles: the Send-Go-to-continue wording goes beyond the fix -- keeping it, with reasoning. Before this PR the wait-exhausted halt left the plan in a state where Go silently resumed with the stage's subagents still running. Now that the halt cancels them, what Go does changes (it continues to the next stage with partial results and discards undelivered ones), so the message has to say so or it misleads. It is only shown when every child was confirmed stopped and no plan Cancel landed during the wait, and test_autopilot_wave_wait_event.py pins that Go really runs the next stage.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 29, 2026
@jshyeung

Copy link
Copy Markdown
Contributor Author

GPT 5.6 (advisory) on c4ec787: chat_orchestrator.py:1028 docstring says "up to an hour by default" -- accepted. The default is SUBAGENT_TIMEOUT_SECS = 10800 (constants.py:145), three hours, so the docstring is wrong. It is a docstring-only error with no behaviour change, so I am not pushing a new head (and restarting the full CI and review round) for it alone; it will be corrected to "three hours by default" in the next push on this PR, or at the maintainer request.

@jshyeung

Copy link
Copy Markdown
Contributor Author

GPT 5.6 (advisory) on c4ec787: autopilot.md:353 repeats the 60 s value of _HALT_CANCEL_BUDGET_SECS -- accepted. The spec already names the constant, so the number in parentheses is a duplicate that can drift. Doc-only, no behaviour change; it will be dropped in the next push on this PR rather than in a push of its own.

@jshyeung

Copy link
Copy Markdown
Contributor Author

Design Review (PASS) watch + suggestion on c4ec787: up-to-60s silent window before the halt notice; broadcast an interim "stopping subagents" status -- rebutted as out of scope for this fix. Before this PR there was no window at all only because the halt stopped nothing and left the children running for up to three hours. The 60 s bound (_HALT_CANCEL_BUDGET_SECS) is the worst case, hit only when a live child reap is awaiting its report delivery; the usual cancel settles well inside it. An interim status line is a new user-visible message, which this PR deliberately does not add; it is a reasonable follow-up if maintainers want it.

@jshyeung

Copy link
Copy Markdown
Contributor Author

First Principles (PASS) watch on c4ec787: verify #14715 really reports this orphaning defect -- confirmed. #14715 originally blamed the subagent reaper and the session lock; the correction comment on the issue (#14715 (comment)) retracts that diagnosis and narrows the report to this exact defect: the stage-timeout halt only turned auto-run off and left the stage subagents running. This PR fixes that narrowed report.

…g them

When auto-run halts a stage it will not settle -- the subagent wait is
exhausted, the stage turn hits its ceiling, or the stage turn errors -- the
halt used to only switch auto-run off. The stage's children kept running to
their own subagent_timeout_secs deadline while the plan said it had stopped.

The halt now cancels exactly that stage's scope, in the same order explicit
Cancel uses: reserve every captured parent, then cancel with the hold kept so
the dispatcher cannot start a queued stage row while the durable-queue cancel
is suspended in its store write. Every manager call is guarded, so a failing
cancel or probe is logged and counted and never skips the halt. The halt waits
at most 60s for its cancel; a slower one (a live child's report delivery can
take up to 20 minutes) keeps running on a tracked background task while the
halt is published. The halt message is worded from the real outcome, and a
wait-exhausted halt only offers a clean Go when every child was confirmed
stopped. Auto-run is switched off before the cancel awaits, and a plan Cancel
landing during that await suppresses the Go guidance.
@jshyeung
jshyeung force-pushed the fix/autopilot-halt-cancels-stage-children branch from c4ec787 to edd498f Compare September 29, 2026 03:36
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Sep 29, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Sep 29, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. The cancel scope is captured off the still-armed boundary as (owner, parent keys) and every cancel is cancel_for_boundary(parent, owner), so it cannot reach another stage's children (the real-manager test pins a sibling owner's queued row untouched). The halt is bounded by asyncio.wait(timeout=60) with the task tracked, publish runs exactly once on every path including controller-cancel mid-wait, and the SecurityEvent audit calls are kept. No OS-process kill or data deletion beyond the manager's scoped cancel of queued rows.

Non-blocking: on the timeout and error halt paths the notice can now land up to 60s later while _in_stage_execution holds user messages; and 'Stopped N' can undercount while a reap is still running. The two red checks on the head were GitHub API rate-limit failures (0/15000 remaining), re-run.

@iamwhatever
iamwhatever enabled auto-merge (squash) September 29, 2026 06:31
@iamwhatever
iamwhatever merged commit 7a2471e into kirodotdev:main Sep 29, 2026
103 of 105 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Subagent reap storm starves all sessions: serial inline reap holds session global lock across provider.shutdown()

2 participants