Repository navigation
fix(subagent): report the stop that killed a run, not the runtime death - #13301
Conversation
|
Intent: A sub-agent run that Kiro Crew itself stops (a user Stop, a parent end, a deadline reap) records and logs THAT stop, never the |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The non-echo exception arm lost its old Watch
Suggestions
[DESIGN-REVIEWED] 8fe901b |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The echo arm re-routes deadline-reap tombstones from Not justified as shipped
What this change shipsInventory (10 items) — 9 justifiedIntent: FIX — a run Kiro Crew itself stopped must record the stop, not the
[FIRST-PRINCIPLES-REVIEWED] 8fe901b |
b904305 to
1dd5b12
Compare
|
|
1dd5b12 to
190d2d4
Compare
|
|
|
|
|
190d2d4 to
bcf5032
Compare
|
|
bcf5032 to
822d098
Compare
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
822d098 to
db0a79d
Compare
|
|
_force_reap tears a dedicated run's session down before it cancels the run task, so the in-flight stream observes its own runtime being killed and raises AcpProcessDied -- "Runtime process died during prompt -- killed (provider shutdown) [returncode=<not reaped>]" -- ahead of the reaper's own record. _run's except arm took that echo at face value: it stored the death text as the run's error, tombstoned it cause="error" and logged "Subagent X failed" at ERROR with a traceback, for a run the user had just pressed Stop on, or that a parent end (an identity-sweep retirement, a slot removal) had deliberately cancelled. Four field reports and one ticket chased that text to the provider, the OOM killer and the leak reaper in turn; the one line naming the parent end sat at INFO, below the gateway log's default level. A reap in flight (_reap_started) now makes the death an echo. The record names who stopped the run (_stop_origin, written by cancel / cancel_for_teardown / _force_reap next to the marker) and carries the reap's own tombstone cause (_reap_reason: user_stop / parent_end / reaped / startup_timeout). A user stop or a parent end stays neutral -- error unset, outcome "stopped", partial output kept -- and a deadline reap fails naming the deadline. One log line replaces the traceback: INFO for a user's own stop, WARNING for a parent end or a reap. The parent-end teardown audit line is WARNING for the same reason. The record stays first-arrival (not info.done), so the reaper's own synthesis is never duplicated. The lost-orphan notice after a gateway restart carries the run's progress and, when the native session file still exists, the spawn_continue handle that resumes it: "no result was captured" had sent parents re-spawning work whose conversation was still on disk.
db0a79d to
8fe901b
Compare
|
…th (kirodotdev#13301) _force_reap tears a dedicated run's session down before it cancels the run task, so the in-flight stream observes its own runtime being killed and raises AcpProcessDied -- "Runtime process died during prompt -- killed (provider shutdown) [returncode=<not reaped>]" -- ahead of the reaper's own record. _run's except arm took that echo at face value: it stored the death text as the run's error, tombstoned it cause="error" and logged "Subagent X failed" at ERROR with a traceback, for a run the user had just pressed Stop on, or that a parent end (an identity-sweep retirement, a slot removal) had deliberately cancelled. Four field reports and one ticket chased that text to the provider, the OOM killer and the leak reaper in turn; the one line naming the parent end sat at INFO, below the gateway log's default level. A reap in flight (_reap_started) now makes the death an echo. The record names who stopped the run (_stop_origin, written by cancel / cancel_for_teardown / _force_reap next to the marker) and carries the reap's own tombstone cause (_reap_reason: user_stop / parent_end / reaped / startup_timeout). A user stop or a parent end stays neutral -- error unset, outcome "stopped", partial output kept -- and a deadline reap fails naming the deadline. One log line replaces the traceback: INFO for a user's own stop, WARNING for a parent end or a reap. The parent-end teardown audit line is WARNING for the same reason. The record stays first-arrival (not info.done), so the reaper's own synthesis is never duplicated. The lost-orphan notice after a gateway restart carries the run's progress and, when the native session file still exists, the spawn_continue handle that resumes it: "no result was captured" had sent parents re-spawning work whose conversation was still on disk. Co-authored-by: w <w@k.local>
Problem / Motivation
Since 0.7.0.7, sub-agents "die at random" with one tombstone text, every time:
cause: error,outcome: stopped, andERROR kiro_crew.subagent: Subagent <id> failedwith a full traceback ingateway.log. Four field reports and one ticket chased that text to the provider, the OOM killer and the leak reaper in turn. None of those killed anything. Every one of those runs was stopped by Kiro Crew itself: a Stop-all press, or a parent end that cancelled the parent's children — on 0.7.0.7+ the identity-retirement sweep does that on every dashboard turn while its baseline stays unset (#12956). The one line that would have said so,parent-end teardown: verb=retire_kiro_identity_sessions …, was logged at INFO, below the gateway log's default level.Why it matters
A deliberate stop that reads as a runtime death sends every reporter, and the maintainers reading their logs, to the wrong subsystem. This week that cost a rollback to a version with its own known leak, a ticket filed against the leak reaper, and hours of log reading that could not find the killer because the killer's only line was invisible at the default log level.
What changed (motivation → approach → change)
AcpProcessDied … killed (provider shutdown)as its error, tombstonecause: error, and an ERROR tracebackstopped by user/parent conversation ended (<verb>)/reaped after Ns (<reason>), tombstone causeuser_stop/parent_end/reaped; a user or parent stop stays neutral (errorunset,outcome: stopped, partial output kept)parent-end teardown: verb=…at INFOstopped by user ⏹for everyuser_stoppedrun, and a stage-boundary cancel is tombstoneduser_stopuser_stop/parent_end/stage_cancel), so neither is credited to the user_force_reaptears a dedicated run's session down BEFORE it cancels the run task. That teardown issessions.reset→provider.shutdown()→runtime.kill(reason="provider shutdown"), so the run's in-flightclient.streamsees its own runtime killed first and raisesAcpProcessDiedinside_run'sexcept Exceptionarm, ahead of the reaper's own record. That arm treated the echo as the run's failure.The arm now reads
_reap_started, the marker the reaper sets before its first await, together withis_runtime_death(exc)— theAcpProcessDiedtest, offered fromagent_sdk/drivers/acp_vocab.pyso application code never names the ACP class. When both hold, the exception is the echo of our own teardown. Every stopper names itself explicitly on the record before the reap — the parent-end teardown writes_reap_reason="parent_end"and the verb, the stage-boundary cancel writesstage_canceland its owner, a barecancelis the user's Stop, and_force_reapfills in its own reason only when none is set — and the arm records those (_stop_origin,_reap_reason) instead of the death text. Nothing is inferred from the origin text, and every writer assigns only while the field is empty, so when a user Stop, a parent end and a stage cancel race, the first stopper keeps the attribution. Neutrality follows that first stopper too (SubagentInfo.stop_is_neutral, read by both the echo arm and_force_reap's own record): a Stop that lands while a deadline reap is already tearing the run down cannot turn the deadline's failure into a neutral stop.tombstone_terminal_state(subagent_manager/admission/types.py) maps the two new causes to CANCELLED likeuser_stop, so boot reconciliation settles such a row instead of recovering a run that was deliberately ended. One log line replaces the traceback — INFO for a user's own stop, WARNING for a parent end or a deadline reap. The record stays first-arrival (if not info.done), so the reaper's own synthesis is never written twice; every other exception under a reap is the run's own fault and keeps the existing path and traceback.cancel's public signature is unchanged: the teardown writes the verb on the record before calling it. The gateway's completion announce (slack/gateway.py) reads the same origin for both its ⏹ status line and the detail sentence that reaches the wave digest (Stopped (parent conversation ended (…)) before completing …rather thanStopped by the user …), and_ReportFailureSnapshotcarries it so a redelivered report says the same thing.docs/system-specs/modules/subagent.mddocuments the reap-echo rule, the audit line's two levels and the tombstone causes.Backwards compatibility
Compatible: no field is removed or renamed. Two new private
SubagentInfofields (_reap_reason,_stop_origininsrc/kiro_crew/subagent.py) are registered inDELIVERY_ROUTING_FIELDSasNOT_DELIVERY_STATE. A run that was tombstonederrorwith the death text is now tombstoneduser_stop/parent_end/stage_cancel/reaped; a stage-boundary cancel that was tombstoneduser_stopis nowstage_cancel.outcomeis unchanged for every case.Tests
test/test_subagent_reap_attribution.py(new, 8 tests) drives the realcancel(),cancel_for_teardown()and_force_reap()against a run whose stream raisesAcpProcessDiedunder the reap: a user stop is neutral and logged below WARNING; a parent end is neutral, tombstonedparent_end, and both the teardown audit line and the run's stop line carry the verb at WARNING; a deadline reap fails naming the deadline. All three also assert no ERROR record and noAcpProcessDiedin the tombstone detail; the fourth pins that an unrelated exception under a reap is still the run's own failure with its traceback; the fifth that a childless parent end logs at INFO; the sixth that a stage-boundary cancel names its own cause and owner before the reap; the seventh that a later parent end or stage cancel does not rewrite a user Stop's attribution; the eighth that a Stop landing during a deadline reap leaves the record failed. Reverting onlyrun.pyturns the first three red.test/test_taskq_reconcile.py::test_a_parent_end_or_stage_cancel_tombstone_settles_the_row_cancelledpins the boot-reconcile mapping; revertingtypes.pyturns it red.test/test_slack_gateway.py::TestSubagentDoneStoppedClassification::test_a_parent_end_stop_is_not_announced_as_the_userspins that a parent-end stop's announce body never says the user stopped it; reverting the detail derivation turns it red.Manual verification
Reproduced on a live gateway before the change: three dedicated-process sub-agents stopped with Stop-all at 09:05:37Z on 2026-09-23 all tombstoned
cause: errorwith theprovider shutdowntext andoutcome: stopped; SEL showsPOST /api/spawn/stop-allthenreaper_force_killfor each. Josh Posey's sharedgateway.logshows the same text on 28 runs, in waves of up to seven one second apart — the sequentialcancel_for_teardownloop — each wave followed by the parent chat's runtime respawn, with noparent-end teardownline anywhere because it sat at INFO. Not re-run on a live gateway after the change (the running gateway must not be restarted on this host); unit coverage drives the same code paths.Screenshots / video
N/A — no user-visible frontend surface changes; the affected text is a tombstone, a log line and an injected completion notice.
Issue link
Related: #12956 (the sweep loop that made the misattributed stops frequent). No open issue for the attribution itself.
Pattern harvest
Rule candidate: review-prompt — when a teardown path kills a process whose consumer is still awaiting it, the consumer's error handler must ask "did WE start this teardown?" (a marker written before the first await of the teardown) before recording the observed death as the consumer's failure. A tombstone or log line that attributes a self-inflicted kill to the process/provider misdirects every reader; a teardown that discards live work must log at the default level or it is invisible in every field report.