Skip to content

fix(subagents): end a leaked process at restart without re-announcing it - #18078

Merged
buluoray merged 2 commits into
mainfrom
fix/reap-completed-delivery-13959
Oct 11, 2026
Merged

buluoray merged 2 commits into
mainfrom
fix/reap-completed-delivery-13959

Conversation

@buluoray

@buluoray buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A subagent that finished but whose process outlived its kill is reported to the parent once, and its process is still ended at the next gateway start.

When a run completed on its own and its teardown kill was then refused, the parent got the completion (naming the survivor with ; kill failed: …), but the run got no tombstone. The next start's orphan reconciliation found the folder, ended the process, and then announced the same completion to the parent a second time.

Why it matters

The parent conversation sees a finished run's outcome twice, with nothing that tells the repeat apart from a real new completion. A parent agent may act on it again.

Not a goal

  • The one-report rule for a run that completes under an in-flight reap. _claim_finalize on main already guarantees exactly one terminal report, as the triage comments on the issue found.
  • A run that published its own report BEFORE a reap's kill failed (terminal.py, the "completed and reported before its kill decided" branch). Its delivered tombstone still hides the survivor and the failure is only on the audit row. Covering it means threading the flag through mark_delivered and the gateway's queued-delivery ledger, which is its own change.
  • A teardown whose kill was still undecided when the report's grace ran out and that then SUCCEEDED. That run is still left with no tombstone.
  • A leaked completion that was queued or digest-held. The gateway's ledger owes no delivered mark for a failed outcome, so the folder stays an orphan. The next start announces it again instead of losing it.

What changed (motivation → approach → change)

  • Symptom: a duplicate completion after restart. Root cause: the only way to get a leaked process ended at restart was to leave the folder untombstoned, and an untombstoned folder means "undelivered result" to list_orphans, so reconciliation announces it.
  • Approach: record the survivor as a leaked process, separate from an undelivered result. The tombstone still ends the run. A leaked_process: true flag on it tells the next start to end the process without sending a notice.
  • run.py (_teardown_run_session): when the kill fails on a completed run, the teardown sets SubagentInfo._leaked_process and still writes no tombstone. The folder stays in orphan reconciliation until the parent acknowledges the completion, so a restart that drops a queued completion re-delivers it. Errored, timeout and cancel tombstones that this path re-writes carry leaked_process=True.
  • terminal.py (_report_terminal): when a delivery is acknowledged (not queued, not digest-held), a flagged run writes the delivered tombstone with leaked_process, its outcome and the detail that names the kill. In _reap_once, both tombstone writes the reap makes after a failed kill carry the flag.
  • subagent_persistence.py: added list_leaked_processes() and clear_leaked_process(). mark_delivered takes leaked_process, outcome and detail. write_tombstone keeps the flag when a later write records no outcome. prune_stale_tombstones keeps a flagged folder until the flag clears.
  • monitoring.py: the orphan kill block (ownership gate, teardown barrier, kill, orphan_reconcile_kill audit) moved unchanged into _end_orphan_process. Before the orphan scan, the new _end_leaked_processes calls it for each flagged tombstone, sends no notice, and clears the flag unless the kill failed. A failed kill keeps the flag so the next start tries again.
  • Spec: docs/system-specs/modules/subagent.md was updated in the force-stop section and in Gateway Restart Reconciliation.

Backwards compatibility

Compatible: tombstones without the flag read as before. The flag is one extra key on a JSON record, and older tombstones never carry it.

The diff hides a path: a completed-but-leaked run's folder now has a tombstone, so list_orphans no longer returns it.
The diff hides a path: once a completed-but-leaked run's delivery is acknowledged, it gets a delivered tombstone, so list_orphans no longer returns it. A queued run still returns.
Reader: src/kiro_crew/subagent_manager/monitoring.py:_reconcile_orphans_impl -- subagent -- ends that run's process through _end_leaked_processes_impl and stops announcing it again (test_the_next_start_ends_the_survivor_and_does_not_announce_it_again). It still announces a queued run (test_a_completion_still_queued_at_restart_is_delivered_again).
Reader: src/kiro_crew/subagent_persistence.py:prune_stale_tombstones -- subagent -- skips a flagged folder (test_the_pruner_keeps_a_flagged_folder_past_its_window).
Reader: src/kiro_crew/subagent_manager/admission/taskq_bridge.py:taskq_artifact_probe -- subagent -- reads the recorded outcome (failed), which is what the live report published.
Reader: src/kiro_crew/subagent_persistence.py:_panel_record -- crew page -- classify_persisted_ending reads that same failed outcome.

Tests

Changes in test/test_subagent_force_stop_audit.py:

  • New class TestALeakedProcessIsEndedWithoutASecondNotice. It drives the real teardown with a refused kill and the run's real terminal report, then a restart's reconciliation over the files on disk. It checks five cases:
    • After an acknowledged delivery, the survivor is killed and _notify_orphan is never awaited. The flag is then cleared and the detail is kept.
    • A kill that fails again keeps the flag.
    • A completion that is still queued gets no tombstone and is announced again after a restart.
    • The pruner keeps a flagged folder that is 30 days old.
    • Control: a plain orphan is still announced.
  • test_a_completed_run_whose_kill_failed_names_the_survivor_and_flags_it and test_a_completed_run_whose_kill_failed_is_flagged_for_its_report replace the two tests that pinned the old "no tombstone ever" behaviour.
  • Flag assertions were added to the tests for the errored-run re-write, the reap's refused kill and the reap-echo publish.
  • Mutations, each verified by a test that goes red:
    • Restoring run.py from main.
    • Dropping the queued/digest guard in _report_terminal, which reddens test_a_completion_still_queued_at_restart_is_delivered_again.
    • Disabling the pruner skip, which reddens test_the_pruner_keeps_a_flagged_folder_past_its_window.
    • Clearing the flag after a failed kill.
    • Dropping the carry-forward in write_tombstone.
  • Local runs, all with pytest -n0: test_subagent_force_stop_audit.py, test_session_cleanup.py, test_subagent_reap_race.py, test_subagent_persistence.py, test_subagent_state_write_model.py and test_session.py, 700 passed. flake8, mypy on the changed modules, black and the comment-history gate are clean.

Manual verification

N/A. The unit coverage drives the real teardown, the real tombstone files and the real reconciliation. The only stubs are the kill primitives, so this host does not have to own a process the test cannot kill. The Windows taskkill arm of the moved kill block runs unchanged and was not exercised here.

Related Issues

Refs #13959

Pattern harvest

Not generalizable: the defect came from one record (the absence of a tombstone) carrying two meanings, "undelivered result" and "process to end". The fix splits that one record, and no lint can catch this kind of overloading.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

Decision for the owner

The owner must decide whether a completed run whose kill failed should be recorded as a leaked process: the next start ends it with no notice, instead of treating it as an undelivered orphan that gets announced again. This draft writes that record only after the parent acknowledges the completion, so a queued completion is never lost, and it leaves the reap-after-delivery, undecided-then-succeeded and queued-leak cases as named follow-ups, so it uses Refs instead of Closes.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 7e6e13a22f394b9da3444fd6f9297eae39edbf31 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 7e6e13a22f394b9da3444fd6f9297eae39edbf31 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check this before merge: errored, timed-out, cancelled and reaped runs whose kill failed are now also ended at restart. Before this change they were never ended there.

Not justified as shipped

  • Item 4 — rides along: the Goal is about a run that "finished" and was "reported to the parent once". But run.py:837 and terminal.py:1018,1031 also flag tombstones that were already terminal, so a new restart kill now reaches survivors that base left alone. It removes a real harm, a process left running, and reuses the same guarded kill. The description declares it.

What this change ships

Inventory (8 items) — 7 justified

Intent: when a subagent finishes but its process survives the kill, the parent hears about it once and the process still ends at the next start. FIX.

  1. After a restart, the parent no longer gets the same completion a second time — justified
  2. The next start ends a leaked process without sending a notice, and retries if the kill fails — justified
  3. A completion still queued or digest-held is announced again after a restart, not lost — justified
  4. Survivors of errored, timed-out, cancelled and reaped runs are now ended at restart — rides along
  5. The pruner keeps a flagged folder until its flag clears — justified
  6. An acknowledged leaked completion removes the parent's workspace agent-<id>.md — justified
  7. The orphan kill block moves unchanged into _end_orphan_process, which now has 2 callers — justified
  8. A flagged folder with an unverifiable pid or no state record drops its flag and logs a WARNING — justified

[FIRST-PRINCIPLES-REVIEWED] 7e6e13a

Description read

[DESCRIPTION-READ] c92d92e19a87667ca745cb3df0719007d2bc44dd0e74e5799e684ec7aaf433fd

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 7e6e13a22f394b9da3444fd6f9297eae39edbf31 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 7e6e13a

Description read

[DESCRIPTION-READ] 41e6666747934202e9d211014d6e743790e7b222342cc29f97007b06009b3945

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 7e6e13a22f394b9da3444fd6f9297eae39edbf31 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7e6e13a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7e6e13a22f394b9da3444fd6f9297eae39edbf31: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 7e6e13a22f394b9da3444fd6f9297eae39edbf31 — this comment is updated in place on each push.

Review details

No findings that block the merge; one advisory gap remains in how leaked processes are cleaned up.

FINDING — src/kiro_crew/subagent_manager/terminal.py:431 — A completed run whose kill failed (info._leaked_process) has no tombstone yet, and this branch skips it when the delivery is _delivery_queued or _digest_held. Later the parent's drain records the delivery through settle_queued_delivery_impl (waves.py:465) or settle_delivered_batch (subagent_persistence.py:1419). Both call mark_delivered(agent_id, elapsed=..., credits=...) without leaked_process, and there is no earlier tombstone for write_tombstone to carry the flag from. So the delivered tombstone hides the folder from list_orphans, list_leaked_processes never sees it either, and the leftover process is never ended on these paths. The same leak existed before this PR → Fix: carry the leaked state on SubagentDelivery and have both settle paths pass leaked_process=True, outcome=..., detail=... to mark_delivered. This touches waves.py and settle_delivered_batch, which this PR did not change.

[OPUS-REVIEWED] 7e6e13a

Verdict parsed from the review's SHA-scoped output markers for commit 7e6e13a22f394b9da3444fd6f9297eae39edbf31.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7e6e13a22f394b9da3444fd6f9297eae39edbf31: <one-sentence reason>

@buluoray
buluoray force-pushed the fix/reap-completed-delivery-13959 branch from a0f0a4a to 52018c5 Compare October 8, 2026 08:20
@buluoray

buluoray commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Both findings were confirmed against source and are fixed in head 52018c5698.

GPT 6.1 (blocking, run.py:830), a queued completion is lost on restart: correct. The teardown wrote the leaked_process tombstone before the parent had consumed the completion, and a tombstone takes the run out of list_orphans. The fix is that the teardown no longer writes a tombstone for a completed run. It sets SubagentInfo._leaked_process instead, and only _report_terminal's acknowledged branch (not _digest_held, not _delivery_queued) writes the delivered tombstone with the flag. Any queued or digest-held completion keeps no tombstone, so a restart re-delivers it. This is pinned by test_a_completion_still_queued_at_restart_is_delivered_again: when the _delivery_queued guard is dropped, that test goes red.

Design (concern), the pruner deletes the only record of the survivor: correct. prune_stale_tombstones now skips any tombstone with leaked_process: true, and clear_leaked_process drops the flag once the process is ended. This is pinned by test_the_pruner_keeps_a_flagged_folder_past_its_window: when the skip is removed, that test goes red.

Design suggestion (also retry from the 60s reaper sweep): not taken in this PR. The per-start retry already covers it, and adding a sweep-time kill is new scope.

@buluoray
buluoray force-pushed the fix/reap-completed-delivery-13959 branch from 52018c5 to 67f880d Compare October 8, 2026 08:37
@buluoray

buluoray commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

GPT 6.1 (blocking, test/test_subagent_force_stop_audit.py:3212), clock patch on the shared time module: I checked this and it holds, because subagent_persistence does import time, so the patch moved the whole worker's clock. It is fixed in head 67f880d341. The test no longer patches the clock. Instead it ages the flagged tombstone on disk by rewriting its died field to 30 days ago, and prune_stale_tombstones compares that field (_tombstone_died). If the pruner's leaked_process skip is removed, test_the_pruner_keeps_a_flagged_folder_past_its_window still goes red.

@buluoray
buluoray force-pushed the fix/reap-completed-delivery-13959 branch from 67f880d to 87953dc Compare October 8, 2026 09:01
@buluoray
buluoray marked this pull request as ready for review October 8, 2026 15:40
@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners October 8, 2026 15:40
@buluoray
buluoray requested a review from chenmingwei23 October 8, 2026 15:40
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

This fixes a finished subagent's completion being announced to the parent a second time after restart when its teardown kill had failed. The root cause is addressed: a separate leaked_process tombstone flag tells the next start to end the process without sending a notice. A queued or digest-held completion keeps no tombstone, so it is still re-delivered. I read the diff, the two author dispositions (the GPT queued-completion loss, fixed in 52018c5, and the shared time patch, fixed in 67f880d) and the current-head lanes. GPT, Opus, Design and First Principles are all clean on 87953dc. I tried to break the kill path. _end_leaked_processes goes through the same _end_orphan_process as orphan reconciliation: the _is_orphan_process pid/start-time check, authorize_runtime_kill against the lease table, and the teardown barrier. A reused or shared pid reads as not-running or refused, and the flag is kept only after a kill that really failed. I found no defect. HOLD only because the change sits on the process-kill path, which needs a human maintainer's approval under our rules. — crew-pr-reviewer

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes the root cause. A run with no tombstone used to mean two things, "result not delivered" and "process still to end". Now a separate leaked_process flag means only the second, so the next start ends the survivor without announcing the completion a second time. A queued or digest-held completion still gets no tombstone, so it is still re-delivered. I checked the new kill path (_end_orphan_process_impl). It is the old orphan kill block moved unchanged: lease-table authorize_runtime_kill, teardown barrier, audit row. Wrong-process / pid reuse: the kill is pid-only, gated by _is_orphan_process (/proc/ ctime <= pid_recorded_at+2). A recycled pid always gets a newer /proc inode, so it is refused. That was safe in every case I could build. The PR widens who gets killed at start: errored, timeout, cancel and reaped tombstones whose kill failed were never killed on main. Every one goes through the same gate, so this is a wider reach, not a new hole. Its own 88 tests pass at 87953dc, and GPT/Opus/Design/FP all pass. The gap is in the wrong-kill gate's other direction, where it refuses a genuine process. I measured 14 of 538 live processes on this host whose /proc ctime is more than 2 s after their real start, up to 58 h. On macOS/Windows there is no /proc, so _is_orphan_process is always False. In both cases the survivor is not killed, outcome is None, the flag is CLEARED, and the log says "not running". So the process is left standing for good and the record says it was gone. Requested change: src/kiro_crew/subagent_manager/monitoring.py:598 -- when _is_orphan_process is False for a live pid, return a distinct outcome (e.g. "unverified") instead of None. At :562 log that word instead of "not running", and at :564 keep the flag for it only on Linux (or clear it with a WARNING naming the pid, so the user knows a process may still be running). This is a few lines, and no wider check is needed. A start-id pin like 18042's would be better, but that is a follow-up. On the owner decision in the PR body: Yes, record a completed run whose kill failed as a leaked process that the next start ends without a notice. The parent already holds the outcome, so a second announcement can only cause duplicate action. Risk: medium. — crew-pr-reviewer

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 8, 2026
@buluoray

buluoray commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks — the finding holds, and it is fixed in f42b3c2.

Verified in source at 87953dc: _end_orphan_process_impl returned None both for a dead pid and for a live pid that _is_orphan_process refused (no /proc, or a /proc ctime later than pid_recorded_at + 2). _end_leaked_processes_impl then logged "not running" and cleared the flag. So a process left standing was recorded as gone.

Fix (src/kiro_crew/subagent_manager/monitoring.py):

  • _end_orphan_process_impl returns a distinct unverified outcome for a live pid it cannot vouch for. It still sends no signal and writes no audit row, as before.
  • _end_leaked_processes_impl logs a WARNING for unverified that names the pid and says the process may still be running. Then it clears the flag. I went with your second option rather than keeping the flag on Linux. The ctime check gives the same answer on every later start, so keeping the flag would only repeat the warning and pin the folder against the pruner forever.
  • The orphan-scan caller ignores the return value, so its behaviour is unchanged.

Test: I added test_a_live_pid_it_cannot_verify_is_reported_as_possibly_running. It asserts no kill, no notice, a WARNING naming the pid, and no "not running" line. When I revert the monitoring.py change, it fails with a process left standing was logged as gone.

Spec: the Gateway Restart Reconciliation paragraph in docs/system-specs/modules/subagent.md now describes the unverified outcome.

Local checks: test_subagent_force_stop_audit.py and test_subagent_panel_durable_replay.py pass, 302 tests. black and flake8 are clean. mypy reports only the 3 wecom errors that main already has. The macOS/Windows path is checked against source only; I did not run it on those platforms.

I agree a start-id pin like #18042's is the better long-term check. I've left it as a follow-up.

@buluoray
buluoray requested a review from iamwhatever October 8, 2026 18:45
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 8, 2026
CrysisDeu
CrysisDeu previously approved these changes Oct 9, 2026

@CrysisDeu CrysisDeu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at f42b3c2. Read the full diff plus the owning spec; nothing blocking. Non-blocking notes, worth a follow-up or a quick fix before merge:

  • [MINOR] src/kiro_crew/subagent_persistence.py:1560 (list_leaked_processes) + :2134 (prune_stale_tombstones) -- a flagged tombstone whose state.json is missing/corrupt (read_state(d.name) is None) is silently skipped by list_leaked_processes, so _end_leaked_processes_impl never sees it and never calls clear_leaked_process, while the pruner skips every flagged folder. That folder is retained forever with no log line. Either clear the flag (nothing can be ended without a pid) when state is unreadable, or let the pruner's skip apply only when state.json is present. Cheap, and it closes the one unbounded-retention path the Design lane's "kill keeps failing" concern does not cover.
  • [NIT] src/kiro_crew/subagent_manager/monitoring.py:~565 -- the info log reads "Ended leaked process of subagent %s: ... outcome=%s" for refused too (lease table says other tenants hold the pid, nothing was signalled). Say "handled"/"reconciled" or branch the wording; the SEL row is right, the log is not.
  • [NIT] src/kiro_crew/subagent_manager/terminal.py:441-463 -- the new elif (leaked + acknowledged) calls mark_delivered but does not unlink the parent's workspace agent-<id>.md (_ws_result_path) the way the if branch above does. Not a regression (main never marked such a run delivered either) but the two "delivered" branches now differ for no stated reason; factor the unlink out or say why it is kept.

@buluoray
buluoray enabled auto-merge (squash) October 9, 2026 05:27
@iamwhatever

Copy link
Copy Markdown
Contributor

This fixes the duplicate completion a parent got after restart for a finished run whose teardown kill failed, and it fixes the root cause. A separate leaked_process tombstone flag now means "process still to end" instead of overloading the missing tombstone, which meant "result undelivered". iamwhatever's earlier request is addressed: in f42b3c2, _end_orphan_process_impl defaults to "unverified" when _is_orphan_process is False for a live pid (macOS, Windows, or a /proc ctime newer than the spawn). _end_leaked_processes_impl then logs a WARNING naming the pid and saying it "may still be running" instead of "not running". The new test pins that WARNING, the absence of "not running", no kill and no notify. I checked the other caller (monitoring.py:460), which ignores the return value, so the new outcome string changes nothing there. The kill gate is unchanged: lease-table authorization, teardown barrier and the /proc ctime identity check, and I could not find a path past it with a recycled pid. CrysisDeu approved f42b3c2, and its three non-blocking notes are still open (an unreadable state.json keeps a flagged folder forever, the "Ended" wording for refused, and the missing workspace unlink in terminal.py:441-463). HOLD because this is process-kill code: it widens which tombstoned runs get killed at startup, which is a maintainer call, and Design CONCERNS is still open on the head. — crew-pr-reviewer

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep re-review of a held PR. It fixes the duplicate completion a parent got after restart when a finished run's teardown kill failed. The fix works at the root: a separate leaked_process tombstone flag now means "process still to end", and a missing tombstone keeps meaning "result not delivered". The restart kill reuses the same _end_orphan_process_impl gate as orphan reconciliation: a pid-alive check, then the _is_orphan_process /proc ctime identity check, then authorize_runtime_kill (lease table), then the teardown barrier. I tried a recycled pid, a shared-runtime pid and a still-live run (agent_id in _agents is skipped). In each case nothing is signalled, or the kill is refused and the flag is cleared, so it cannot kill the wrong process. A queued or digest-held completion writes no tombstone, so it is still re-delivered. The hold reason (process-kill code needs a maintainer) still stands. The PR is MERGEABLE and checks are green, but GitHub still shows CHANGES_REQUESTED. That is iamwhatever's own review on old head 87953dc. Its point (an unverified outcome must not log "not running") is fixed in f42b3c2 and pinned by a test. CrysisDeu approved the current head. No dispositions have been posted for the Design or First Principles CONCERNS on f42b3c2. Requested change: (1) iamwhatever dismisses or re-approves the old CHANGES_REQUESTED. (2) The author posts dispositions for spans e554021cfe1c, ffd534b93228 and 4f0222819055. (3) docs/system-specs/modules/subagent.md: say the restart kill only signals where /proc identity holds (Linux); macOS and Windows log "may still be running" and do not kill. That closes span ffd534b93228 with no code change. Optional: cap retries for a kill that keeps failing (span 4f0222819055). Today the folder is kept and retried on every start, which is bounded by restarts and harmless. On the owner decision: Keep the widening. Errored, timed-out and reaped runs whose kill failed also get the flag (FP span e554021cfe1c). On main those survivors leaked for good, and they go through the same guarded kill. Risk: medium. — crew-pr-reviewer

@buluoray
buluoray force-pushed the fix/reap-completed-delivery-13959 branch from f42b3c2 to 72a51a5 Compare October 9, 2026 19:08
@buluoray

buluoray commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: every point from the open reviews is addressed in head 72a51a5061. The fixes are folded into the second commit, so the PR still has two.

iamwhatever, deep re-review on f42b3c2 (5466466804)

  • (2) Dispositions for the open lane spans:
    • First Principles e554021cfe1c (the flag also covers errored, timed-out, cancelled and reaped runs): kept, as the review recommended. On main those survivors already had a tombstone, so nothing at restart ever ended them, and they leaked for good. They now go through the same guarded kill (pid-alive check, /proc identity, authorize_runtime_kill, teardown barrier). So this extends an existing gate to more runs; it is not a new kill path.
    • Design ffd534b93228 (the restart kill is Linux-only): fixed in the spec, with no code change. docs/system-specs/modules/subagent.md (Gateway Restart Reconciliation) now says the restart kill only signals where /proc identity holds, which means Linux. On macOS and Windows the leaked process is never ended at restart, and the log says it may still be running. I did not edit the Goal, because it is frozen.
    • Design 4f0222819055 (a kill that keeps failing keeps its folder): the spec now states this retention is intended. It is one guarded retry per gateway start, so it is bounded by the number of restarts. Dropping the flag would leave a survivor that no record names. I did not add a retry cap.
  • (3) The spec change is the one described just above.
  • (1) The CHANGES_REQUESTED on 87953dc can only be cleared by its reviewer, so I have re-requested your review.

CrysisDeu's notes on f42b3c2 (5465658469)

  • [MINOR] A flagged folder with a missing or corrupt state.json was kept forever. Fixed: list_leaked_processes now returns it as {"id": <folder>}. With no pid there is nothing to end, so the flag is dropped with a WARNING that the record names no pid, and the pruner can then remove the folder. Test: test_a_flagged_folder_with_no_readable_state_drops_its_flag.
  • [NIT] The log said "Ended" for a refused kill. Fixed: each outcome now logs in its own words. killed logs "Ended". refused logs "left running: the lease table refused…". failed is a WARNING saying the next start tries again. A pid that is no longer running logs "already gone". Test: test_a_refused_kill_is_not_logged_as_ended.
  • [NIT] The leaked branch did not unlink the workspace agent-<id>.md. Fixed: both delivered branches in _report_terminal now share one unlink step. Test: test_an_acknowledged_leaked_completion_cleans_the_workspace_copy.

How this was verified

  • Mutations: I reverted each of the three source files to f42b3c2 in turn. Each revert turns its matching new test red, as listed above.
  • Local test run with pytest: test_subagent_force_stop_audit.py, test_subagent_reap_race.py, test_subagent_persistence.py, test_subagent_state_write_model.py, test_subagent_panel_durable_replay.py and test_session_cleanup.py, 530 passed.
  • black, flake8, mypy on the changed modules, and the comment-history gate are clean.
  • Not exercised: a real macOS or Windows host.

@buluoray
buluoray requested a review from iamwhatever October 9, 2026 19:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 9, 2026
@buluoray
buluoray force-pushed the fix/reap-completed-delivery-13959 branch from 72a51a5 to b1b191f Compare October 9, 2026 19:38
@buluoray

buluoray commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1's blocking finding on 72a51a5061 (terminal.py:431, security-class) holds. It is fixed in code in head b1b191f9ca, with no override.

  • Finding: the leaked-process delivered branch in _report_terminal did not check _report_undelivered. A channel or cron parent whose injection ran out of ACP retries returns normally from _on_done. That branch then wrote the delivered tombstone over a completion the parent never received, so a restart would not re-deliver it.
  • Fix: the branch now also requires not info._report_undelivered. Such a run keeps no tombstone, so the next start's orphan reconciliation re-delivers the completion and ends the survivor through the orphan kill.
  • Test: the new test test_a_completion_whose_injection_was_given_up_is_delivered_again goes red when the guard is removed.
  • Spec: the force-stop section of the spec is updated in the same commit.
  • Not changed: the non-leaked branch above it (terminal.py:398) has the same gap, but that gap already exists on main and is outside this PR's goal.

How this was verified

  • Targeted subagent suites: 531 passed.
  • mypy, black, flake8 and the comment-history gate are clean.
  • The PR still has two commits.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 9, 2026
@buluoray

buluoray commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: these are dispositions for the advisory CONCERNS on b1b191f9ca. I am not pushing for them, because GPT 6.1, Opus 5.5 and every CI lane except one known flake are clear on this head.

First Principles, Item 3: the flag on errored, timed-out, cancelled and reaped tombstones

  • Kept. iamwhatever's re-review (5466466804) asked for this.
  • On main, those survivors were never ended at restart.
  • They go through the same guarded path as orphans: /proc identity check, lease-table authorization, teardown barrier.

First Principles, Item 6: unlinking agent-<id>.md on a leaked delivery

  • Kept. CrysisDeu's review (5465658469) asked for this.
  • Before the change, the two "delivered" branches did different things for no reason. Now both run one shared unlink step.

Design: a queued leaked completion is announced again

  • Known and out of scope. It is listed under "Not a goal".
  • It is intentional: a queued completion that has not been consumed yet must be re-delivered rather than lost.
  • Fixing it means threading the flag through the gateway's queued-delivery ledger, which is a separate change.

The remaining red check

@buluoray
buluoray force-pushed the fix/reap-completed-delivery-13959 branch from b1b191f to 29cb975 Compare October 11, 2026 02:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@buluoray

Copy link
Copy Markdown
Contributor Author

Conflict-free rebase onto main (c62495f) to pick up #18644, which fixed the session-tags 17b E2E flake that was this PR's only red on b1b191f. git range-diff shows both commits unchanged (=). No code change.

A completed run whose teardown kill was refused published its completion
(naming the survivor) but got no tombstone, so the next start's orphan
reconciliation ended the process and then announced the same completion
to the parent a second time.

The teardown now flags such a run, and once the parent acknowledges the
completion (not queued, not digest-held) its `delivered` tombstone carries
`leaked_process: true`; every tombstone the run's own arm or the reap
writes after a failed kill carries it too. A queued completion keeps no
tombstone, so a restart still re-delivers it. The tombstone ends the run, so
`list_orphans` skips it; `_end_leaked_processes` reads the flag before the
orphan scan, ends the recorded pid through the same guarded kill an orphan
gets, sends no notice, and clears the flag unless the kill failed.
`write_tombstone` carries the flag across a later outcome-less write, and
`prune_stale_tombstones` keeps a flagged folder.

Refs #13959
A live pid that _is_orphan_process cannot vouch for (no /proc, or a /proc
entry newer than the spawn) is not signalled. _end_orphan_process now
returns 'unverified' for it instead of the None that means not running,
and the leaked-process pass logs a WARNING naming the pid.

The pass also logs refused, failed, killed and already-gone outcomes
separately, so a lease-table refusal no longer reads as "Ended".
list_leaked_processes returns a flagged folder whose state.json is
missing or unreadable as an id-only record, so its flag is dropped with
a WARNING instead of keeping the folder past the pruner forever. An
acknowledged leaked completion now unlinks the parent's workspace copy,
as a clean delivery does. The spec states the restart kill signals only
where /proc identity holds (Linux) and that a kill that keeps failing is
retried once per start by design.
@buluoray
buluoray force-pushed the fix/reap-completed-delivery-13959 branch from 29cb975 to 7e6e13a Compare October 11, 2026 03:26
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR fixes the duplicate completion a parent got after restart when a finished run's teardown kill failed. It works at the root by splitting one record into two meanings: a leaked_process tombstone flag now means "process still to end", and a missing tombstone keeps meaning "result not delivered". Every point in our CHANGES_REQUESTED on f42b3c2 is met on 7e6e13a. Dispositions are posted for spans e554021cfe1c, ffd534b93228 and 4f0222819055, and subagent.md says the restart kill signals only where /proc identity holds. CrysisDeu's three notes are fixed with tests: an unreadable state.json drops the flag, each outcome logs in its own words, and both delivered branches unlink agent-<id>.md. GPT's security-class finding is fixed too: the leaked branch in terminal.py now also requires not info._report_undelivered. I tried to break the kill path. _end_orphan_process_impl is the old orphan block moved unchanged (pid alive, _is_orphan_process ctime check, authorize_runtime_kill, teardown barrier, audit row), a live run is skipped via _agents, and an unverifiable pid returns unverified and is never signalled. A forged state.json pid could reach this gate, but plain orphan reconciliation on main already has that exposure, so the PR does not widen it. I also checked Opus's advisory about queued leaked completions: the gateway's owed list only covers info.outcome == "completed" (slack/gateway.py ~9027), and a run whose kill failed carries an error, so no queued mark_delivered hides it. The folder stays an orphan, as the Not-a-goal section says. Checks are green and the base is main. — crew-pr-reviewer (deep re-check)

@buluoray
buluoray merged commit 7caa233 into main Oct 11, 2026
102 of 103 checks passed
@buluoray
buluoray deleted the fix/reap-completed-delivery-13959 branch October 11, 2026 06:05
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants