Skip to content

fix(subagent): report the stop that killed a run, not the runtime death - #13301

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/subagent-reap-attribution
Sep 24, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/subagent-reap-attribution

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Since 0.7.0.7, sub-agents "die at random" with one tombstone text, every time:

kiro_crew.acp.client.AcpProcessDied: Runtime process died during prompt — killed (provider shutdown) [returncode=<not reaped>] stderr_tail: <none>

cause: error, outcome: stopped, and ERROR kiro_crew.subagent: Subagent <id> failed with a full traceback in gateway.log. Four field reports and one ticket chased that text to the provider, the OOM killer and the leak reaper in turn. None of those killed anything. Every one of those runs was stopped by Kiro Crew itself: a Stop-all press, or a parent end that cancelled the parent's children — on 0.7.0.7+ the identity-retirement sweep does that on every dashboard turn while its baseline stays unset (#12956). The one line that would have said so, parent-end teardown: verb=retire_kiro_identity_sessions …, was logged at INFO, below the gateway log's default level.

Why it matters

A deliberate stop that reads as a runtime death sends every reporter, and the maintainers reading their logs, to the wrong subsystem. This week that cost a rollback to a version with its own known leak, a ticket filed against the leak reaper, and hours of log reading that could not find the killer because the killer's only line was invisible at the default log level.

What changed (motivation → approach → change)

Before After
A run stopped by us dies with AcpProcessDied … killed (provider shutdown) as its error, tombstone cause: error, and an ERROR traceback The record names the stop: stopped by user / parent conversation ended (<verb>) / reaped after Ns (<reason>), tombstone cause user_stop / parent_end / reaped; a user or parent stop stays neutral (error unset, outcome: stopped, partial output kept)
parent-end teardown: verb=… at INFO at WARNING when the snapshot names work to discard, INFO for a childless parent end, so the default log shows what ended the children without a line per closed tab
the completion announce says stopped by user ⏹ for every user_stopped run, and a stage-boundary cancel is tombstoned user_stop the announce carries the record's own stop origin and the tombstone its own cause (user_stop / parent_end / stage_cancel), so neither is credited to the user

_force_reap tears a dedicated run's session down BEFORE it cancels the run task. That teardown is sessions.reset → provider.shutdown() → runtime.kill(reason="provider shutdown"), so the run's in-flight client.stream sees its own runtime killed first and raises AcpProcessDied inside _run's except Exception arm, ahead of the reaper's own record. That arm treated the echo as the run's failure.

The arm now reads _reap_started, the marker the reaper sets before its first await, together with is_runtime_death(exc) — the AcpProcessDied test, offered from agent_sdk/drivers/acp_vocab.py so application code never names the ACP class. When both hold, the exception is the echo of our own teardown. Every stopper names itself explicitly on the record before the reap — the parent-end teardown writes _reap_reason="parent_end" and the verb, the stage-boundary cancel writes stage_cancel and its owner, a bare cancel is the user's Stop, and _force_reap fills in its own reason only when none is set — and the arm records those (_stop_origin, _reap_reason) instead of the death text. Nothing is inferred from the origin text, and every writer assigns only while the field is empty, so when a user Stop, a parent end and a stage cancel race, the first stopper keeps the attribution. Neutrality follows that first stopper too (SubagentInfo.stop_is_neutral, read by both the echo arm and _force_reap's own record): a Stop that lands while a deadline reap is already tearing the run down cannot turn the deadline's failure into a neutral stop. tombstone_terminal_state (subagent_manager/admission/types.py) maps the two new causes to CANCELLED like user_stop, so boot reconciliation settles such a row instead of recovering a run that was deliberately ended. One log line replaces the traceback — INFO for a user's own stop, WARNING for a parent end or a deadline reap. The record stays first-arrival (if not info.done), so the reaper's own synthesis is never written twice; every other exception under a reap is the run's own fault and keeps the existing path and traceback. cancel's public signature is unchanged: the teardown writes the verb on the record before calling it. The gateway's completion announce (slack/gateway.py) reads the same origin for both its ⏹ status line and the detail sentence that reaches the wave digest (Stopped (parent conversation ended (…)) before completing … rather than Stopped by the user …), and _ReportFailureSnapshot carries it so a redelivered report says the same thing.

docs/system-specs/modules/subagent.md documents the reap-echo rule, the audit line's two levels and the tombstone causes.

Backwards compatibility

Compatible: no field is removed or renamed. Two new private SubagentInfo fields (_reap_reason, _stop_origin in src/kiro_crew/subagent.py) are registered in DELIVERY_ROUTING_FIELDS as NOT_DELIVERY_STATE. A run that was tombstoned error with the death text is now tombstoned user_stop / parent_end / stage_cancel / reaped; a stage-boundary cancel that was tombstoned user_stop is now stage_cancel. outcome is unchanged for every case.

Tests

  • test/test_subagent_reap_attribution.py (new, 8 tests) drives the real cancel(), cancel_for_teardown() and _force_reap() against a run whose stream raises AcpProcessDied under the reap: a user stop is neutral and logged below WARNING; a parent end is neutral, tombstoned parent_end, and both the teardown audit line and the run's stop line carry the verb at WARNING; a deadline reap fails naming the deadline. All three also assert no ERROR record and no AcpProcessDied in the tombstone detail; the fourth pins that an unrelated exception under a reap is still the run's own failure with its traceback; the fifth that a childless parent end logs at INFO; the sixth that a stage-boundary cancel names its own cause and owner before the reap; the seventh that a later parent end or stage cancel does not rewrite a user Stop's attribution; the eighth that a Stop landing during a deadline reap leaves the record failed. Reverting only run.py turns the first three red.
  • test/test_taskq_reconcile.py::test_a_parent_end_or_stage_cancel_tombstone_settles_the_row_cancelled pins the boot-reconcile mapping; reverting types.py turns it red.
  • test/test_slack_gateway.py::TestSubagentDoneStoppedClassification::test_a_parent_end_stop_is_not_announced_as_the_users pins that a parent-end stop's announce body never says the user stopped it; reverting the detail derivation turns it red.

Manual verification

Reproduced on a live gateway before the change: three dedicated-process sub-agents stopped with Stop-all at 09:05:37Z on 2026-09-23 all tombstoned cause: error with the provider shutdown text and outcome: stopped; SEL shows POST /api/spawn/stop-all then reaper_force_kill for each. Josh Posey's shared gateway.log shows the same text on 28 runs, in waves of up to seven one second apart — the sequential cancel_for_teardown loop — each wave followed by the parent chat's runtime respawn, with no parent-end teardown line anywhere because it sat at INFO. Not re-run on a live gateway after the change (the running gateway must not be restarted on this host); unit coverage drives the same code paths.

Screenshots / video

N/A — no user-visible frontend surface changes; the affected text is a tombstone, a log line and an injected completion notice.

Issue link

Related: #12956 (the sweep loop that made the misattributed stops frequent). No open issue for the attribution itself.

Pattern harvest

Rule candidate: review-prompt — when a teardown path kills a process whose consumer is still awaiting it, the consumer's error handler must ask "did WE start this teardown?" (a marker written before the first await of the teardown) before recording the observed death as the consumer's failure. A tombstone or log line that attributes a self-inflicted kill to the process/provider misdirects every reader; a teardown that discards live work must log at the default level or it is invisible in every field report.

@bolichen97
bolichen97 requested review from a team and chenmingwei23 September 24, 2026 08:34
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: A sub-agent run that Kiro Crew itself stops (a user Stop, a parent end, a deadline reap) records and logs THAT stop, never the AcpProcessDied … killed (provider shutdown) echo the teardown causes; and a restart orphan whose conversation still exists is told how to resume it.
Not a goal: Changing when or why runs are stopped (the identity sweep, parent-end semantics, reap deadlines), the cancel API, or the tombstone/outcome contract for runs that genuinely fail.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 8fe901bfb6ed14ee146a8ff53db63954b8ce2425 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The non-echo exception arm lost its old not info.reaped guard, so a late run fault can overwrite the reaper's already-written neutral record.

Watch

  • In run.py, the old arm was if not info.reaped: <write error>; the new else: branch writes info.error, info.done = True and a second "error" tombstone unconditionally. If a non-AcpProcessDied exception surfaces after _force_reap has already completed its own first-arrival record (info.done set, neutral tombstone written), this arm overwrites the attribution and double-writes the tombstone — reintroducing, in a race window, exactly the misattribution class the PR fixes. The echo arm carries the if not info.done: first-arrival guard the module doc promises ("the record is still first-arrival"); the else arm no longer does.
    Clears when: the else arm regains a first-arrival guard (if not info.done:) or a test pins why the reaped-then-fault ordering is unreachable.

Suggestions

  • Carry the machine cause (_reap_reason) into _ReportFailureSnapshot and the announce instead of string-comparing _stop_origin == "stopped by user" in slack/gateway.py — the description's own "nothing is inferred from the origin text" rule, applied one layer further out, so a reworded origin can't silently change announce phrasing.

[DESIGN-REVIEWED] 8fe901b

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 8fe901bfb6ed14ee146a8ff53db63954b8ce2425 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8fe901b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8fe901bfb6ed14ee146a8ff53db63954b8ce2425: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 8fe901bfb6ed14ee146a8ff53db63954b8ce2425 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The echo arm re-routes deadline-reap tombstones from error onto reaped/startup_timeout — the two causes tombstone_terminal_state still leaves unmapped.

Not justified as shipped

  • Item 4 — symptom-level (point patch, 2 counted siblings): the new test's own docstring names the harm — "an unmapped cause would leave a deliberately ended run to be recovered on the next boot" — yet the map in src/kiro_crew/subagent_manager/admission/types.py:22-35 gains only parent_end/stage_cancel, while run.py:532 (info._reap_reason or "reaped") now writes reaped/startup_timeout where base wrote error (→FAILED, settled). Grepped the map: reaped and startup_timeout are absent, so the probe returns None and the row falls into boot recovery instead of settling FAILED.
    Clears when: reaped and startup_timeout map to FAILED in the same table, or evidence a None verdict already settles a STARTING/RUNNING row without recovering it.

What this change ships

Inventory (10 items) — 9 justified

Intent: FIX — a run Kiro Crew itself stopped must record the stop, not the AcpProcessDied echo of its own teardown.

  1. A stopped run's tombstone and log name who stopped it, never the runtime death — justified
  2. A parent-end stop stays neutral, tombstoned parent_end — justified
  3. A stage-boundary cancel is tombstoned stage_cancel, no longer user_stop — justified
  4. A deadline reap's record names the deadline; tombstone now reaped/startup_timeout where error was written — symptom-level: those two causes stay unmapped at boot reconcile
  5. parent-end teardown audit line now WARNING when work is discarded, INFO when childless — justified
  6. Completion announce and digest carry the stop's origin instead of always "stopped by user" — justified
  7. A Stop landing during an in-flight deadline reap no longer turns the failure neutral — justified
  8. New driver-vocabulary predicate is_runtime_death (one consumer, mandated by the check_agent_sdk_boundary seam) — justified
  9. Two private record fields (_reap_reason, _stop_origin) carried through the failure snapshot — justified
  10. subagent.md updated in the same commit — justified

[FIRST-PRINCIPLES-REVIEWED] 8fe901b

@bolichen97
bolichen97 force-pushed the fix/subagent-reap-attribution branch from b904305 to 1dd5b12 Compare September 24, 2026 08:53
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Echo arm classifies every exception under a reap span=349273773c9c

fixed in 1dd5b12: the arm now requires is_runtime_death(exc) (the AcpProcessDied test, exported from agent_sdk/drivers/acp_vocab.py so subagent.py adds no ACP import edge) alongside _reap_started; any other exception under a reap falls through to the existing failure path with its traceback.
pinned by test_an_unrelated_exception_under_a_reap_is_still_the_runs_own_failure: a RuntimeError raised while a reap is in flight still records outcome=failed, cause error, and an ERROR record with exc_info.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Thread _stop_origin into the subagent_done announcement span=018687d8b37a

fixed in 1dd5b12: slack/gateway.py's three-way classification uses the record's _stop_origin as the ⏹ status ("stopped by user" only when the user pressed Stop; the parent-end verb or a stage cancel otherwise), and _ReportFailureSnapshot carries the field so a redelivered report says the same thing. The spec's terminal-state contract line is updated to match.

@bolichen97
bolichen97 force-pushed the fix/subagent-reap-attribution branch from 1dd5b12 to 190d2d4 Compare September 24, 2026 09:10
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Audit line unconditional at WARNING span=509b7abe5478

fixed in 190d2d4: cancel_for_teardown_impl logs the audit line at WARNING only when the snapshot names work to discard and at INFO for a childless parent end (audit = logger.warning if snapshot_ids else logger.info); pinned by test_a_childless_parent_end_is_not_a_warning.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • parent_end inferred from a non-empty origin span=3e696676ce00

fixed in 190d2d4: the cause is carried explicitly. The parent-end teardown writes _reap_reason="parent_end", the stage-boundary cancel (_revoke_boundary_owners_impl) writes _reap_reason="stage_cancel" and _stop_origin="stage cancelled (<owner>)", a bare cancel fills user_stop only when nothing is set, and _force_reap keeps a caller-named cause. No inference from the origin text remains; pinned by test_a_stage_cancel_names_itself_before_the_reap.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 7 — the lost-orphan resume hint rides along span=e0dad8bb49b9

fixed in 190d2d4 by subtraction: the resume hint (orphan_resume_hint, its test and its spec paragraph) is removed from this PR and will be opened as its own PR against the same base, so this diff is exactly the stop-attribution job its title names.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • stage-boundary cancellation becomes "stopped by user" / user_stop span=27a1897c7ded

fixed in 190d2d4: _revoke_boundary_owners_impl sets _reap_reason="stage_cancel" and _stop_origin="stage cancelled (<boundary_owner>)" before the reap, and cancel_impl only defaults to user_stop when no cause is set, so the tombstone and the announce name the stage cancel.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • orphan_resume_hint checks only sid.json and imports inside the function span=51047c440b4e

fixed in 190d2d4 by subtraction: orphan_resume_hint is removed from this PR (it is a separate change and will ship in its own PR, where it will mirror SessionMap.get's JSONL viability check and import session_map at module scope as this finding asks).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 24, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-reap-attribution branch from 190d2d4 to bcf5032 Compare September 24, 2026 09:30
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • A later cancellation overwrites the stopper already owning the reap span=d5ac4f1e6ea4

fixed in bcf5032: the parent-end teardown and the stage-boundary revocation assign _reap_reason / _stop_origin only while the field is empty, so the first stopper keeps the attribution and a racing parent end or stage cancel cannot rewrite the audit record; pinned by test_the_first_stopper_keeps_the_attribution.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • tombstone_terminal_state does not map parent_end / stage_cancel span=ff3f0ef4a19b

fixed in bcf5032: both causes map to taskq.CANCELLED like user_stop (they are the same deliberate-stop family, written by the same reap), so boot reconciliation settles the row instead of recovering a run that was ended on purpose; pinned by test_taskq_reconcile.py::test_a_parent_end_or_stage_cancel_tombstone_settles_the_row_cancelled. startup_timeout is pre-existing and unchanged by this PR, so it is left out of scope here.

@bolichen97
bolichen97 force-pushed the fix/subagent-reap-attribution branch from bcf5032 to 822d098 Compare September 24, 2026 10:13
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 8fe901bfb6ed14ee146a8ff53db63954b8ce2425 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8fe901b

Verdict parsed from the review's SHA-scoped output markers for commit 8fe901bfb6ed14ee146a8ff53db63954b8ce2425.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8fe901bfb6ed14ee146a8ff53db63954b8ce2425: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 24, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-reap-attribution branch from 822d098 to db0a79d Compare September 24, 2026 10:52
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Status line reads _stop_origin but the detail sentence still says the user pressed Stop span=afebc2c8ad23

fixed in db0a79d: the user_stopped detail is derived from the same origin -- Stopped by the user … only when the origin is the user's Stop, otherwise Stopped (<origin>) … -- so the wave digest a sibling flushes carries the same attribution as the status line; pinned by test_a_parent_end_stop_is_not_announced_as_the_users.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The echo arm depends on _force_reap's kill-before-cancel order; note it as a tracked follow-up span=1dffd2087019

fixed in db0a79d as a note at the order itself: _force_reap_impl's reset-then-cancel block now states that the order is load-bearing for the reap-echo arm and that reordering would make the arm unreachable rather than wrong (the cancel path records its own stop). No separate issue: the coupling is documented where a future edit would meet it, and the spec's Terminal-State Contract already names the ordering.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Sep 24, 2026
_force_reap tears a dedicated run's session down before it cancels the
run task, so the in-flight stream observes its own runtime being killed
and raises AcpProcessDied -- "Runtime process died during prompt --
killed (provider shutdown) [returncode=<not reaped>]" -- ahead of the
reaper's own record. _run's except arm took that echo at face value: it
stored the death text as the run's error, tombstoned it cause="error"
and logged "Subagent X failed" at ERROR with a traceback, for a run the
user had just pressed Stop on, or that a parent end (an identity-sweep
retirement, a slot removal) had deliberately cancelled. Four field
reports and one ticket chased that text to the provider, the OOM killer
and the leak reaper in turn; the one line naming the parent end sat at
INFO, below the gateway log's default level.

A reap in flight (_reap_started) now makes the death an echo. The record
names who stopped the run (_stop_origin, written by cancel /
cancel_for_teardown / _force_reap next to the marker) and carries the
reap's own tombstone cause (_reap_reason: user_stop / parent_end /
reaped / startup_timeout). A user stop or a parent end stays neutral --
error unset, outcome "stopped", partial output kept -- and a deadline
reap fails naming the deadline. One log line replaces the traceback:
INFO for a user's own stop, WARNING for a parent end or a reap. The
parent-end teardown audit line is WARNING for the same reason. The
record stays first-arrival (not info.done), so the reaper's own
synthesis is never duplicated.

The lost-orphan notice after a gateway restart carries the run's
progress and, when the native session file still exists, the
spawn_continue handle that resumes it: "no result was captured" had
sent parents re-spawning work whose conversation was still on disk.
@bolichen97
bolichen97 force-pushed the fix/subagent-reap-attribution branch from db0a79d to 8fe901b Compare September 24, 2026 11:23
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • A late Stop converts a claimed deadline reap into a neutral stop span=3b96d9d7798e

fixed in 8fe901b: neutrality is derived from the first stopper -- SubagentInfo.stop_is_neutral reads _reap_reason against _NEUTRAL_REAP_REASONS (user_stop / parent_end / stage_cancel) -- in BOTH record writers: the echo arm in _run and _force_reap's own guard-1 record. A non-neutral reap puts user_stopped back to False before recording, so a Stop landing during a deadline reap leaves outcome=failed with the deadline's error and a reaped tombstone; pinned by test_a_late_stop_does_not_turn_a_deadline_reap_neutral.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 24, 2026
@chenmingwei23
chenmingwei23 merged commit 4d4e184 into main Sep 24, 2026
141 of 148 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/subagent-reap-attribution branch September 24, 2026 16:11
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 24, 2026
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…th (kirodotdev#13301)

_force_reap tears a dedicated run's session down before it cancels the
run task, so the in-flight stream observes its own runtime being killed
and raises AcpProcessDied -- "Runtime process died during prompt --
killed (provider shutdown) [returncode=<not reaped>]" -- ahead of the
reaper's own record. _run's except arm took that echo at face value: it
stored the death text as the run's error, tombstoned it cause="error"
and logged "Subagent X failed" at ERROR with a traceback, for a run the
user had just pressed Stop on, or that a parent end (an identity-sweep
retirement, a slot removal) had deliberately cancelled. Four field
reports and one ticket chased that text to the provider, the OOM killer
and the leak reaper in turn; the one line naming the parent end sat at
INFO, below the gateway log's default level.

A reap in flight (_reap_started) now makes the death an echo. The record
names who stopped the run (_stop_origin, written by cancel /
cancel_for_teardown / _force_reap next to the marker) and carries the
reap's own tombstone cause (_reap_reason: user_stop / parent_end /
reaped / startup_timeout). A user stop or a parent end stays neutral --
error unset, outcome "stopped", partial output kept -- and a deadline
reap fails naming the deadline. One log line replaces the traceback:
INFO for a user's own stop, WARNING for a parent end or a reap. The
parent-end teardown audit line is WARNING for the same reason. The
record stays first-arrival (not info.done), so the reaper's own
synthesis is never duplicated.

The lost-orphan notice after a gateway restart carries the run's
progress and, when the native session file still exists, the
spawn_continue handle that resumes it: "no result was captured" had
sent parents re-spawning work whose conversation was still on disk.

Co-authored-by: w <w@k.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants