Repository navigation
fix(autopilot): cancel a halted stage's subagents instead of orphaning them - #14813
iamwhatever merged 1 commit into
Conversation
|
Intent: When autopilot halts a stage (subagent wait exhausted, stage-turn timeout, or stage-turn error), cancel that stage's running and queued subagents instead of leaving them to run until their own timeout, and word the halt message from what the cancel actually did. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS "Fixes #14715" closes the issue while two halt paths still orphan children — confirm the deferred follow-up is tracked before merge. [DESIGN-REVIEWED] edd498f Description read[DESCRIPTION-READ] bd4853a3abe4a1e8b5fa8e0f07a6a7884574ce21beb5ad01e8d5399d461f2e2d That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Two halts still orphan children — "subagent check failed" ( What this change shipsInventory (10 items) — 10 justifiedIntent: FIX — when autopilot halts a stage (#14715), the stage's subagents stop too instead of running on for up to 3h as orphans.
Checks run: cause-level per the description and diff (the halts never called the scoped cancellation explicit Cancel uses; this wires them to it). Siblings counted via [FIRST-PRINCIPLES-REVIEWED] edd498f Description read[DESCRIPTION-READ] e2d4c9245b7d4ce0c119bf27e3baebbceaef4d536245914eebc95587e2554ac2 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5 Review (fork) — ✅ no blocking findingsReviewed Review detailsNothing blocks; one advisory — the FINDING — src/kiro_crew/dashboard/chat_orchestrator.py:936 — the added docstring states " [OPUS-REVIEWED] edd498f |
12333b7 to
c4ec787
Compare
|
GPT 5.6 (blocking): real-manager tests leak the SubagentManager -- fixed in c4ec787. Both tests in test/test_autopilot_halt_cancels_real_manager.py now close the manager in a finally block (cancel_all + close), including when an assertion fails partway, and assert the task queue is closed afterwards. |
|
Opus 5 (advisory): Stop during the 60s halt-cancel wait dropped the halt notice -- fixed in c4ec787. _cancel_exhausted_stage_subagents now takes a publish callback holding each halt's notice, SEL audit event and (timeout/error halts) stage preservation, and calls it exactly once on every path. On CancelledError during the bounded wait it publishes with the unfinished outcome (worded as still finishing in the background, no clean Go line) and then re-raises; the tracked background cancel is not cancelled. New parametrized test TestAStopDuringTheHaltCancelKeepsTheNotice covers all three halts; reverting the handler fails all three. |
|
Opus 5 + design review: docs/system-specs/modules/autopilot.md stale -- fixed in c4ec787. Step 6 now documents the stage-turn ceiling halt and the error halt, and a new paragraph under step 7 describes the halt-time cancel: reserve-then-cancel with retained hold and no-hold fallback, guarded manager calls, the 60s bound with background continuation, outcome-worded notice, and when the Go line is offered. |
|
Design review: halt cancel duplicates the explicit-Cancel loop (and suggestion to extract it for future halts) -- done in c4ec787. The per-parent cancel-then-probe sequence is now a single helper, _cancel_boundary_parents, used by explicit Cancel (guarded=False: call shape, probe order and exception propagation unchanged; test_plan_cancel_race.py passes untouched) and by the halt (guarded=True: failures logged and counted). The remaining halts that still leave children running (subagent-check failure, auth-required delivery failure) can reuse it; they stay out of scope here as noted in the PR body. |
|
First Principles: the Send-Go-to-continue wording goes beyond the fix -- keeping it, with reasoning. Before this PR the wait-exhausted halt left the plan in a state where Go silently resumed with the stage's subagents still running. Now that the halt cancels them, what Go does changes (it continues to the next stage with partial results and discards undelivered ones), so the message has to say so or it misleads. It is only shown when every child was confirmed stopped and no plan Cancel landed during the wait, and test_autopilot_wave_wait_event.py pins that Go really runs the next stage. |
|
GPT 5.6 (advisory) on c4ec787: |
|
GPT 5.6 (advisory) on c4ec787: |
|
Design Review (PASS) watch + suggestion on c4ec787: up-to-60s silent window before the halt notice; broadcast an interim "stopping subagents" status -- rebutted as out of scope for this fix. Before this PR there was no window at all only because the halt stopped nothing and left the children running for up to three hours. The 60 s bound ( |
|
First Principles (PASS) watch on c4ec787: verify #14715 really reports this orphaning defect -- confirmed. #14715 originally blamed the subagent reaper and the session lock; the correction comment on the issue (#14715 (comment)) retracts that diagnosis and narrows the report to this exact defect: the stage-timeout halt only turned auto-run off and left the stage subagents running. This PR fixes that narrowed report. |
…g them When auto-run halts a stage it will not settle -- the subagent wait is exhausted, the stage turn hits its ceiling, or the stage turn errors -- the halt used to only switch auto-run off. The stage's children kept running to their own subagent_timeout_secs deadline while the plan said it had stopped. The halt now cancels exactly that stage's scope, in the same order explicit Cancel uses: reserve every captured parent, then cancel with the hold kept so the dispatcher cannot start a queued stage row while the durable-queue cancel is suspended in its store write. Every manager call is guarded, so a failing cancel or probe is logged and counted and never skips the halt. The halt waits at most 60s for its cancel; a slower one (a live child's report delivery can take up to 20 minutes) keeps running on a tracked background task while the halt is published. The halt message is worded from the real outcome, and a wait-exhausted halt only offers a clean Go when every child was confirmed stopped. Auto-run is switched off before the cancel awaits, and a plan Cancel landing during that await suppresses the Go guidance.
c4ec787 to
edd498f
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
Reviewed. The cancel scope is captured off the still-armed boundary as (owner, parent keys) and every cancel is cancel_for_boundary(parent, owner), so it cannot reach another stage's children (the real-manager test pins a sibling owner's queued row untouched). The halt is bounded by asyncio.wait(timeout=60) with the task tracked, publish runs exactly once on every path including controller-cancel mid-wait, and the SecurityEvent audit calls are kept. No OS-process kill or data deletion beyond the manager's scoped cancel of queued rows.
Non-blocking: on the timeout and error halt paths the notice can now land up to 60s later while _in_stage_execution holds user messages; and 'Stopped N' can undercount while a reap is still running. The two red checks on the head were GitHub API rate-limit failures (0/15000 remaining), re-run.
Problem / Motivation
Goal: When autopilot halts a stage, the stage's subagents stop too instead of running on as orphans.
When a stage's subagent wait is exhausted,
_halt_plan()only switched auto-run off. The stage turn timing out, or failing with an internal error, did the same. In all three cases the stage's running and queued children kept going until their ownsubagent_timeout_secs(3h by default). They held concurrency slots and did work nobody would collect, and the dashboard felt hung until they were reaped. Only the explicit Cancel button had scoped cancellation.Why it matters
Every autopilot user whose stage runs out of budget leaks its subagents. With a small
subagent_auto_max, a few orphaned stages starve new work for hours.Not a goal
owner._lock) was refuted:provider.shutdown()runs after the lock is released (session_lifecycle.pylock block 950-1058, shutdown at 1110).What changed (motivation → approach → change)
Symptom (orphaned children after a halt) → root cause (the halt paths never called the scoped cancellation that explicit Cancel uses) → change in
src/kiro_crew/dashboard/chat_orchestrator.py:_cancel_exhausted_stage_subagentsruns on the wait-exhausted, stage-turn-timeout and stage-turn-error halts. It uses the same order as explicit Cancel: reserve every parent scope first, thencancel_for_boundary(retain_scope=True). That way queued stage work can't be admitted during the task-store write. If the reservation is refused, it falls back toretain_scope=False, as explicit Cancel does._preserve_interrupted_stage().CancelledErrorstill propagates._HALT_CANCEL_BUDGET_SECS). A live child's reap awaits report delivery, which can take up to_ON_DONE_TIMEOUT(1200s) per parent. A cancel still running after 60s continues on a tracked background task. It is not cancelled, because cutting it would abandon a kill already in progress._HaltCancelOutcome). The halt message is worded from that outcome: stopped, pending, refused, failed, or still cancelling. It only offers "Send Go to continue" when every child was confirmed stopped. Otherwise it tells the user to stop the rest first, on all three halt paths.Backwards compatibility
Compatible: nothing is rejected, renamed or removed. The three halts now also cancel the stage's children, and their messages carry one or two extra sentences.
Tests
test/test_autopilot_halt_cancels_real_manager.py(new) uses a realSubagentManageron the durable task queue. The stage's running child is revoked and reaped. Its queued child is cancelled in the store. Another stage's queued row under the same parent is untouched. No hold is left behind. A second test checks that during the store write the manager refuses to admit the stage's queued rows. It fails when the reservation and hold are removed.test/test_autopilot_stage_completion_handoff.py:test/test_autopilot_wave_wait_event.py:test/test_subagent.py. The backend gates pass except two host-only issues that fail the same way on cleanorigin/main: a mypy error on the Linux-onlyPidfdChildWatcherincli.py, and twotest_cli.py::TestDoctorStttests.Manual verification
N/A: the unit tests cover the halt paths end to end through
_stage_loop, and one test uses a real manager and task queue. I have not reproduced it live on a running gateway.Related Issues
Fixes #14715
Possible conflict: open PR #13992 ("reconcile stage timeouts") also edits
chat_orchestrator.py.Pattern harvest
Rule candidate: review-prompt
Pattern: a halt, abort or timeout path that stops the controller but not the work it spawned. Every stop path should call the same scoped cancellation as the explicit-cancel path.
Checklist