Repository navigation
fix(subagents): end a leaked process at restart without re-announcing it - #18078
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check this before merge: errored, timed-out, cancelled and reaped runs whose kill failed are now also ended at restart. Before this change they were never ended there. Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: when a subagent finishes but its process survives the kill, the parent hears about it once and the process still ends at the next start. FIX.
[FIRST-PRINCIPLES-REVIEWED] 7e6e13a Description read[DESCRIPTION-READ] c92d92e19a87667ca745cb3df0719007d2bc44dd0e74e5799e684ec7aaf433fd That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 7e6e13a Description read[DESCRIPTION-READ] 41e6666747934202e9d211014d6e743790e7b222342cc29f97007b06009b3945 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings that block the merge; one advisory gap remains in how leaked processes are cleaned up. FINDING — src/kiro_crew/subagent_manager/terminal.py:431 — A completed run whose kill failed ( [OPUS-REVIEWED] 7e6e13a Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
a0f0a4a to
52018c5
Compare
|
Both findings were confirmed against source and are fixed in head GPT 6.1 (blocking, Design (concern), the pruner deletes the only record of the survivor: correct. Design suggestion (also retry from the 60s reaper sweep): not taken in this PR. The per-start retry already covers it, and adding a sweep-time kill is new scope. |
52018c5 to
67f880d
Compare
|
GPT 6.1 (blocking, |
67f880d to
87953dc
Compare
|
This fixes a finished subagent's completion being announced to the parent a second time after restart when its teardown kill had failed. The root cause is addressed: a separate |
iamwhatever
left a comment
There was a problem hiding this comment.
Fixes the root cause. A run with no tombstone used to mean two things, "result not delivered" and "process still to end". Now a separate leaked_process flag means only the second, so the next start ends the survivor without announcing the completion a second time. A queued or digest-held completion still gets no tombstone, so it is still re-delivered. I checked the new kill path (_end_orphan_process_impl). It is the old orphan kill block moved unchanged: lease-table authorize_runtime_kill, teardown barrier, audit row. Wrong-process / pid reuse: the kill is pid-only, gated by _is_orphan_process (/proc/ ctime <= pid_recorded_at+2). A recycled pid always gets a newer /proc inode, so it is refused. That was safe in every case I could build. The PR widens who gets killed at start: errored, timeout, cancel and reaped tombstones whose kill failed were never killed on main. Every one goes through the same gate, so this is a wider reach, not a new hole. Its own 88 tests pass at 87953dc, and GPT/Opus/Design/FP all pass. The gap is in the wrong-kill gate's other direction, where it refuses a genuine process. I measured 14 of 538 live processes on this host whose /proc ctime is more than 2 s after their real start, up to 58 h. On macOS/Windows there is no /proc, so _is_orphan_process is always False. In both cases the survivor is not killed, outcome is None, the flag is CLEARED, and the log says "not running". So the process is left standing for good and the record says it was gone. Requested change: src/kiro_crew/subagent_manager/monitoring.py:598 -- when _is_orphan_process is False for a live pid, return a distinct outcome (e.g. "unverified") instead of None. At :562 log that word instead of "not running", and at :564 keep the flag for it only on Linux (or clear it with a WARNING naming the pid, so the user knows a process may still be running). This is a few lines, and no wider check is needed. A start-id pin like 18042's would be better, but that is a follow-up. On the owner decision in the PR body: Yes, record a completed run whose kill failed as a leaked process that the next start ends without a notice. The parent already holds the outcome, so a second announcement can only cause duplicate action. Risk: medium. — crew-pr-reviewer
|
Thanks — the finding holds, and it is fixed in f42b3c2. Verified in source at 87953dc: Fix (
Test: I added Spec: the Gateway Restart Reconciliation paragraph in Local checks: I agree a start-id pin like #18042's is the better long-term check. I've left it as a follow-up. |
CrysisDeu
left a comment
There was a problem hiding this comment.
Approving at f42b3c2. Read the full diff plus the owning spec; nothing blocking. Non-blocking notes, worth a follow-up or a quick fix before merge:
- [MINOR] src/kiro_crew/subagent_persistence.py:1560 (
list_leaked_processes) + :2134 (prune_stale_tombstones) -- a flagged tombstone whosestate.jsonis missing/corrupt (read_state(d.name) is None) is silently skipped bylist_leaked_processes, so_end_leaked_processes_implnever sees it and never callsclear_leaked_process, while the pruner skips every flagged folder. That folder is retained forever with no log line. Either clear the flag (nothing can be ended without a pid) when state is unreadable, or let the pruner's skip apply only whenstate.jsonis present. Cheap, and it closes the one unbounded-retention path the Design lane's "kill keeps failing" concern does not cover. - [NIT] src/kiro_crew/subagent_manager/monitoring.py:~565 -- the info log reads
"Ended leaked process of subagent %s: ... outcome=%s"forrefusedtoo (lease table says other tenants hold the pid, nothing was signalled). Say "handled"/"reconciled" or branch the wording; the SEL row is right, the log is not. - [NIT] src/kiro_crew/subagent_manager/terminal.py:441-463 -- the new
elif(leaked + acknowledged) callsmark_deliveredbut does not unlink the parent's workspaceagent-<id>.md(_ws_result_path) the way theifbranch above does. Not a regression (main never marked such a run delivered either) but the two "delivered" branches now differ for no stated reason; factor the unlink out or say why it is kept.
|
This fixes the duplicate completion a parent got after restart for a finished run whose teardown kill failed, and it fixes the root cause. A separate |
iamwhatever
left a comment
There was a problem hiding this comment.
Deep re-review of a held PR. It fixes the duplicate completion a parent got after restart when a finished run's teardown kill failed. The fix works at the root: a separate leaked_process tombstone flag now means "process still to end", and a missing tombstone keeps meaning "result not delivered". The restart kill reuses the same _end_orphan_process_impl gate as orphan reconciliation: a pid-alive check, then the _is_orphan_process /proc ctime identity check, then authorize_runtime_kill (lease table), then the teardown barrier. I tried a recycled pid, a shared-runtime pid and a still-live run (agent_id in _agents is skipped). In each case nothing is signalled, or the kill is refused and the flag is cleared, so it cannot kill the wrong process. A queued or digest-held completion writes no tombstone, so it is still re-delivered. The hold reason (process-kill code needs a maintainer) still stands. The PR is MERGEABLE and checks are green, but GitHub still shows CHANGES_REQUESTED. That is iamwhatever's own review on old head 87953dc. Its point (an unverified outcome must not log "not running") is fixed in f42b3c2 and pinned by a test. CrysisDeu approved the current head. No dispositions have been posted for the Design or First Principles CONCERNS on f42b3c2. Requested change: (1) iamwhatever dismisses or re-approves the old CHANGES_REQUESTED. (2) The author posts dispositions for spans e554021cfe1c, ffd534b93228 and 4f0222819055. (3) docs/system-specs/modules/subagent.md: say the restart kill only signals where /proc identity holds (Linux); macOS and Windows log "may still be running" and do not kill. That closes span ffd534b93228 with no code change. Optional: cap retries for a kill that keeps failing (span 4f0222819055). Today the folder is kept and retried on every start, which is bounded by restarts and harmless. On the owner decision: Keep the widening. Errored, timed-out and reaped runs whose kill failed also get the flag (FP span e554021cfe1c). On main those survivors leaked for good, and they go through the same guarded kill. Risk: medium. — crew-pr-reviewer
f42b3c2 to
72a51a5
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: every point from the open reviews is addressed in head iamwhatever, deep re-review on f42b3c2 (5466466804)
CrysisDeu's notes on f42b3c2 (5465658469)
How this was verified
|
72a51a5 to
b1b191f
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1's blocking finding on
How this was verified
|
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: these are dispositions for the advisory CONCERNS on First Principles, Item 3: the flag on errored, timed-out, cancelled and reaped tombstones
First Principles, Item 6: unlinking
Design: a queued leaked completion is announced again
The remaining red check
|
b1b191f to
29cb975
Compare
A completed run whose teardown kill was refused published its completion (naming the survivor) but got no tombstone, so the next start's orphan reconciliation ended the process and then announced the same completion to the parent a second time. The teardown now flags such a run, and once the parent acknowledges the completion (not queued, not digest-held) its `delivered` tombstone carries `leaked_process: true`; every tombstone the run's own arm or the reap writes after a failed kill carries it too. A queued completion keeps no tombstone, so a restart still re-delivers it. The tombstone ends the run, so `list_orphans` skips it; `_end_leaked_processes` reads the flag before the orphan scan, ends the recorded pid through the same guarded kill an orphan gets, sends no notice, and clears the flag unless the kill failed. `write_tombstone` carries the flag across a later outcome-less write, and `prune_stale_tombstones` keeps a flagged folder. Refs #13959
A live pid that _is_orphan_process cannot vouch for (no /proc, or a /proc entry newer than the spawn) is not signalled. _end_orphan_process now returns 'unverified' for it instead of the None that means not running, and the leaked-process pass logs a WARNING naming the pid. The pass also logs refused, failed, killed and already-gone outcomes separately, so a lease-table refusal no longer reads as "Ended". list_leaked_processes returns a flagged folder whose state.json is missing or unreadable as an id-only record, so its flag is dropped with a WARNING instead of keeping the folder past the pruner forever. An acknowledged leaked completion now unlinks the parent's workspace copy, as a clean delivery does. The spec states the restart kill signals only where /proc identity holds (Linux) and that a kill that keeps failing is retried once per start by design.
29cb975 to
7e6e13a
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
This PR fixes the duplicate completion a parent got after restart when a finished run's teardown kill failed. It works at the root by splitting one record into two meanings: a leaked_process tombstone flag now means "process still to end", and a missing tombstone keeps meaning "result not delivered". Every point in our CHANGES_REQUESTED on f42b3c2 is met on 7e6e13a. Dispositions are posted for spans e554021cfe1c, ffd534b93228 and 4f0222819055, and subagent.md says the restart kill signals only where /proc identity holds. CrysisDeu's three notes are fixed with tests: an unreadable state.json drops the flag, each outcome logs in its own words, and both delivered branches unlink agent-<id>.md. GPT's security-class finding is fixed too: the leaked branch in terminal.py now also requires not info._report_undelivered. I tried to break the kill path. _end_orphan_process_impl is the old orphan block moved unchanged (pid alive, _is_orphan_process ctime check, authorize_runtime_kill, teardown barrier, audit row), a live run is skipped via _agents, and an unverifiable pid returns unverified and is never signalled. A forged state.json pid could reach this gate, but plain orphan reconciliation on main already has that exposure, so the PR does not widen it. I also checked Opus's advisory about queued leaked completions: the gateway's owed list only covers info.outcome == "completed" (slack/gateway.py ~9027), and a run whose kill failed carries an error, so no queued mark_delivered hides it. The folder stays an orphan, as the Not-a-goal section says. Checks are green and the base is main. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: A subagent that finished but whose process outlived its kill is reported to the parent once, and its process is still ended at the next gateway start.
When a run completed on its own and its teardown kill was then refused, the parent got the completion (naming the survivor with
; kill failed: …), but the run got no tombstone. The next start's orphan reconciliation found the folder, ended the process, and then announced the same completion to the parent a second time.Why it matters
The parent conversation sees a finished run's outcome twice, with nothing that tells the repeat apart from a real new completion. A parent agent may act on it again.
Not a goal
_claim_finalizeon main already guarantees exactly one terminal report, as the triage comments on the issue found.terminal.py, the "completed and reported before its kill decided" branch). Itsdeliveredtombstone still hides the survivor and the failure is only on the audit row. Covering it means threading the flag throughmark_deliveredand the gateway's queued-delivery ledger, which is its own change.deliveredmark for afailedoutcome, so the folder stays an orphan. The next start announces it again instead of losing it.What changed (motivation → approach → change)
list_orphans, so reconciliation announces it.leaked_process: trueflag on it tells the next start to end the process without sending a notice.run.py(_teardown_run_session): when the kill fails on a completed run, the teardown setsSubagentInfo._leaked_processand still writes no tombstone. The folder stays in orphan reconciliation until the parent acknowledges the completion, so a restart that drops a queued completion re-delivers it. Errored, timeout and cancel tombstones that this path re-writes carryleaked_process=True.terminal.py(_report_terminal): when a delivery is acknowledged (not queued, not digest-held), a flagged run writes thedeliveredtombstone withleaked_process, itsoutcomeand thedetailthat names the kill. In_reap_once, both tombstone writes the reap makes after a failed kill carry the flag.subagent_persistence.py: addedlist_leaked_processes()andclear_leaked_process().mark_deliveredtakesleaked_process,outcomeanddetail.write_tombstonekeeps the flag when a later write records no outcome.prune_stale_tombstoneskeeps a flagged folder until the flag clears.monitoring.py: the orphan kill block (ownership gate, teardown barrier, kill,orphan_reconcile_killaudit) moved unchanged into_end_orphan_process. Before the orphan scan, the new_end_leaked_processescalls it for each flagged tombstone, sends no notice, and clears the flag unless the killfailed. A failed kill keeps the flag so the next start tries again.docs/system-specs/modules/subagent.mdwas updated in the force-stop section and in Gateway Restart Reconciliation.Backwards compatibility
Compatible: tombstones without the flag read as before. The flag is one extra key on a JSON record, and older tombstones never carry it.
The diff hides a path: a completed-but-leaked run's folder now has a tombstone, so
list_orphansno longer returns it.The diff hides a path: once a completed-but-leaked run's delivery is acknowledged, it gets a
deliveredtombstone, solist_orphansno longer returns it. A queued run still returns.Reader: src/kiro_crew/subagent_manager/monitoring.py:_reconcile_orphans_impl -- subagent -- ends that run's process through
_end_leaked_processes_impland stops announcing it again (test_the_next_start_ends_the_survivor_and_does_not_announce_it_again). It still announces a queued run (test_a_completion_still_queued_at_restart_is_delivered_again).Reader: src/kiro_crew/subagent_persistence.py:prune_stale_tombstones -- subagent -- skips a flagged folder (
test_the_pruner_keeps_a_flagged_folder_past_its_window).Reader: src/kiro_crew/subagent_manager/admission/taskq_bridge.py:taskq_artifact_probe -- subagent -- reads the recorded
outcome(failed), which is what the live report published.Reader: src/kiro_crew/subagent_persistence.py:_panel_record -- crew page --
classify_persisted_endingreads that samefailedoutcome.Tests
Changes in
test/test_subagent_force_stop_audit.py:TestALeakedProcessIsEndedWithoutASecondNotice. It drives the real teardown with a refused kill and the run's real terminal report, then a restart's reconciliation over the files on disk. It checks five cases:_notify_orphanis never awaited. The flag is then cleared and the detail is kept.test_a_completed_run_whose_kill_failed_names_the_survivor_and_flags_itandtest_a_completed_run_whose_kill_failed_is_flagged_for_its_reportreplace the two tests that pinned the old "no tombstone ever" behaviour.run.pyfrom main._report_terminal, which reddenstest_a_completion_still_queued_at_restart_is_delivered_again.test_the_pruner_keeps_a_flagged_folder_past_its_window.write_tombstone.pytest -n0:test_subagent_force_stop_audit.py,test_session_cleanup.py,test_subagent_reap_race.py,test_subagent_persistence.py,test_subagent_state_write_model.pyandtest_session.py, 700 passed. flake8, mypy on the changed modules, black and the comment-history gate are clean.Manual verification
N/A. The unit coverage drives the real teardown, the real tombstone files and the real reconciliation. The only stubs are the kill primitives, so this host does not have to own a process the test cannot kill. The Windows
taskkillarm of the moved kill block runs unchanged and was not exercised here.Related Issues
Refs #13959
Pattern harvest
Not generalizable: the defect came from one record (the absence of a tombstone) carrying two meanings, "undelivered result" and "process to end". The fix splits that one record, and no lint can catch this kind of overloading.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.
Decision for the owner
The owner must decide whether a completed run whose kill failed should be recorded as a leaked process: the next start ends it with no notice, instead of treating it as an undelivered orphan that gets announced again. This draft writes that record only after the parent acknowledges the completion, so a queued completion is never lost, and it leaves the reap-after-delivery, undecided-then-succeeded and queued-leak cases as named follow-ups, so it uses
Refsinstead ofCloses.