Repository navigation
fix(acp): signal a runtime's process group after its root has died - #11720
Conversation
|
Intent: Reach the runtime tree that #11565 cannot: one whose root died before any descendant was snapshotted. The root's pid still names its process group; teardowns now signal that group once a live member vouches for it by identity. Also lands the untrack scope and test pin deferred from #11565. |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Run the #11369 fault-injection recipe live once: the environ-token vouch surviving a real sandboxed launcher's death is the only link the fixture-/proc tests can't prove. [DESIGN-REVIEWED] 74eb4a2 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify on a real pre-5.3 or seccomp-filtered kernel that What this change shipsInventory (10 items) — 10 justifiedIntent: stop leaking a runtime's process group (agent + chat, ~400 MB each) when its root dies before teardown — a FIX, with a field reproduction and tests that fail on base.
Notes from counting: [FIRST-PRINCIPLES-REVIEWED] 74eb4a2 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
9a8d066 to
ee5a326
Compare
|
The one blocking finding on The three red backend/coverage jobs on the previous head were |
|
The blocking finding on
The three red backend/coverage jobs were |
ee5a326 to
52864ff
Compare
|
The blocking finding on
Pinned by |
52864ff to
a45ecc9
Compare
|
All three findings on Opus — the escalation re-resolved the group from the dead root's number. GPT F1 — the vouched group could be replaced before GPT F2 — untracking ignored the child's identity. Pinned by |
a45ecc9 to
f72b4aa
Compare
97f10c1 to
b209f90
Compare
|
Three tests there pin It was also redundant:
Rebased onto |
|
self-added: yes
|
b209f90 to
a4c4fcd
Compare
|
The advisory. Why 24 checks are red, and why no push fixes them. All of them reproduce on
None of those files are in this diff. Across the sampled shards there are zero Reviewer state on |
a4c4fcd to
686e03f
Compare
|
This diff touches neither file, so there is nothing in it to fix for that. The
|
|
|
|
|
|
self-added: yes
|
|
The advisory was a real regression from the round before it.
On the reds. Main has reverted both #11947 (via #12091) and #11960 (via Rebased onto |
|
self-added: yes
|
|
self-added: no
|
|
The root is pinned across the terminate. For the record on that lane: this item was stated as an advisory, then as blocking Design's Watch — the silent Linux leak. The unreached-tree WARNING sat under Tests: the pinned call is asserted with the recorded identity and fails if the Rebased onto |
kill_process_tree is killpg(getpgid(pid)), and getpgid raises once the root has exited. AcpRuntime._kill_inner swallowed that, read the dead root as "already killed", and untracked it — while the launcher's children, the agent and its chat process stayed in the group, never signalled, reparented to init and holding hundreds of MB each. A root that dies a few seconds into its life predates every descendant snapshot, so neither the recorded descendants nor the tracked sweep could find that tree. Field evidence on a second host: 10 such trees, 4 GB, all subagent sessions whose launcher exited ~3 s after spawn, every one absent from both PID files; plain SIGTERM reaped 18 of 18. The root was spawned as a session leader, so its pid IS the group id and the number outlives the process. AcpRuntime._signal_tree treats ProcessLookupError from the tree kill as the start of a second path: session_pid._signal_orphaned_runtime_group finds that group's members and signals THEM, each re-verified by start id at the instant of the signal -- never the group number, which is the dead root's pid and can be handed to a fresh session leader at any moment. A member vouches when _marked_group_members finds it carrying this spawn's KIROCREW_SPAWN_INSTANCE -- a per-spawn token spawn() now puts on the root's environment, inherited by the whole tree -- together with the KIROCREW_SPAWNED marker and a runtime argv identity. The instance is the incarnation pin the marker cannot supply: every runtime is a marked session leader, so a root pid released to a fresh spawn names a group that carries the marker just as well, and a signal aimed by number and marker alone would terminate that fresh runtime's live session. No vouching member means no signal; declining costs a leak the sweep still reports, acting would cost an unrelated process. A vouched SIGTERM is followed by the same grace a live tree gets and a SIGKILL pass aimed only at members the SIGTERM vouched that are still alive under the same start id, since the dead root's wait() returned at once and drove no escalation. The escalation never re-resolves the root's number: getpgid on a recycled root pid succeeds for the fresh runtime holding it. Any other OSError is final: the root is there and we were refused. Linux only — the environ read is — so macOS and Windows keep the missed reap rather than gain a wrong kill. _untrack_child_pids gains parent_pid and both teardown callers pass their root; a record carrying a start token removes only a line whose third field matches it. A child pid is reused like any other number -- both numbers of a pair can be -- and another live runtime can have recorded the same pair under its root since ours exited; removing by number alone took that runtime's line too. _snapshot_descendants checks the root's identity before the walk and again before the write, and only the first check was pinned. A test now holds the identity through the walk and drops it before the write, and asserts on the writer, so deleting the second check fails it.
|
A second partial platform stand-in. An assertion that only described one host. To stop this class here rather than one shard per round, I enumerated every test Rebased onto |
Problem / Motivation
A second host, on 0.7.0.4 with #11565's tracking already backported, still
leaked: 10 live
kiro-cli acp+kiro-cli-chatpairs, PPID 1, ~400 MB each,4.0 GB total, none in either PID file. Every one maps to a subagent session
whose sandbox launcher exited ~3 s after spawn. Plain
SIGTERMreaped 18 of 18within 8 s. The leak is a signal never delivered, not a process that refuses
to die.
The signal path is
killpg(getpgid(root)).getpgidraises once the root hasexited;
_kill_innerswallowed that, sawpid_exists(root)false, took the"killed" branch and untracked it. Nothing in the group was signalled.
Why it matters
This is the tree the snapshot in #11565 cannot cover — a root that died
before its first scan — and the tracked sweep cannot either, since nothing
was recorded. It is the only shape observed on that host. Subagent spawns
happen in batches of 2–3, so it accumulates at the rate of one tree per
retried spawn.
What changed (motivation → approach → change)
A dead root's pid is still its group id, so a reaped root starts a second
teardown path instead of ending the first — and that path aims at members it can
prove are ours, never at the number.
src/kiro_crew/acp/runtime.pyProcessLookupErrorfrom the tree signal swallowed; root untracked_signal_treeresolves the group from the root's number only while the root's live start id matches the one recorded at spawn; a root that is gone or unreadable starts the vouched path: signal the members, then the same grace and aSIGKILLpasssrc/kiro_crew/acp/runtime.pykill_process_tree_pinnedholds the handle that verified the identity across the terminate, so the number cannot be recycled in between;Falsemeans unconfirmed, so nothing is sent and the vouched path runs insteadsrc/kiro_crew/acp/runtime.pyreturncode is Noneaccepted as proof the number was heldsrc/kiro_crew/acp/runtime.pyholds,mismatch(both ids read, they differ),unknown(nothing recorded, or the read failed). All refuse authorization; onlymismatchmay describe the rootsrc/kiro_crew/acp/runtime.py_pgroup_has_member_besides, so it speaks on every platform while the group still holds something and stays quiet once it is emptysrc/kiro_crew/acp/runtime.pywait()did not time out, so one grace and oneSIGKILLper member instead of twosrc/kiro_crew/acp/runtime.pyexceptOSErrorstays final — the root is there and we were refusedsrc/kiro_crew/acp/runtime.pysrc/kiro_crew/session_pid.py_signal_orphaned_runtime_groupsignals each MEMBER_marked_group_membersvouched; never the reusable group numbersrc/kiro_crew/session_pid.py_signal_pid_by_identityopens a pidfd, verifies the identity AFTER the open, thenpidfd_send_signal. A descriptor pins one process, so a recycled number is unreachable through it. A kernel without the pidfd SYSCALL keeps the re-verifiedos.kill, keyed on the errno (ENOSYS/EPERM) rather than on the Python attribute, which exists on every Linux build; a refusal of a call the kernel HAS skips the member insteadsrc/kiro_crew/session_pid.pygroup_vouching_available()states it once, for the reader that acts on it and the one that reports itsrc/kiro_crew/constants.pyspawn()also mints a per-spawnKIROCREW_SPAWN_INSTANCEand keeps it asprocess_instance— the incarnation pin a recycled root pid cannot fakedocs/system-specs/modules/acp-client.md,docs/system-specs/modules/session.mdtest/test_acp_runtime_kill.py,test/test_pid_lifecycle.py,test/test_process_tree.py,test/test_acp_spawn_offload.pytest/fixtures/acp_launch_goldens.json,test/acp_launch_capture.py,test/test_acp_launch_goldens.pyKIROCREW_SPAWN_INSTANCEis a volatile key (<spawn-instance>): that the child RECEIVES it is pinned, its per-spawn value is notWhy vouch instead of aiming at the number. A dead root's pid can be handed
to a fresh session leader, which is a marked session leader too — so a signal
aimed by number and marker alone would terminate that runtime's live session. The
per-spawn instance is the pin the marker cannot supply: it says THIS spawn's
tree. No vouching member means no signal, because declining costs a leak the
sweep reports while acting costs an unrelated process.
Why a handle instead of a number, on both halves. Re-reading an identity and
then acting on the number leaves a window the recycled number can land in.
Narrowing it cannot close it — a pid is a key the kernel may reissue, not a
handle. For a member that handle is a pidfd, so open → verify → signal makes the
check binding for as long as the descriptor is held; the ordering does the work,
so a test pins it by having the fake open change what the identity reader
returns. For the root it is
kill_process_tree_pinned, which keeps the queryhandle that verified the identity open across the terminate — the tree signal is
deferred to an executor and, on Windows, resolved by a separate
taskkill, whichis precisely where an unpinned number can change hands.
Why the identity is recorded earlier. A spawn failing between the process
becoming live and the identity read left a LIVE root with nothing recorded, and a
teardown may only resolve a group while the recorded identity matches — so that
root's whole tree went unsignalled, which is the leak the surrounding guard
exists to reap.
Why the verdict has three values.
get_process_start_idreturnsNonebothfor a failed read and for a pid that is gone, so one boolean reported a cause
nobody had checked. Authorization is unchanged; only what the logs may claim is.
Why the escalation never re-resolves the group.
getpgidon a recycled rootpid succeeds for whoever holds it now, so after the first pass the target set is
fixed: members it vouched, still alive under the same start id.
Why an unreached tree is audible, on every platform. Off Linux the environ
read cannot run at all; on Linux the members can be there and simply not vouch —
a sandbox that scrubbed the token, an unreadable environ, a missing argv
identity. Both leave the tree to the orphan sweep, which is the trade this keeps:
a leak over a signal to a stranger. What was wrong was the silence. The guard is
the group rather than the platform, so the line fires whenever something is still
in the group and stays quiet when it is empty, which is the ordinary teardown.
Backwards compatibility
Compatible: no signature, field, key or format that main accepts is rejected,
renamed or removed.
_untrack_child_pids,_prune_dead_descendantsandAcpClient._reset_statekeep main's signatures and behaviour exactly — anearlier revision of this branch scoped the untrack by parent pid and start
token, and that is withdrawn here as an item belonging to #11565, not to this
defect. One new env key travels to spawned children and one new module-level
predicate is added; both are additive.
Known and deferred, in this diff
SIGKILLpass is awaited raises at the await and leaves it running unawaitedshieldbounds one cancellation; the members were already vouched and SIGTERMed_KILL_TERM_TIMEOUTsleep, never waiting on member livenessAcpClient._kill_processkeeps the same reaped-root windowTests
test/test_acp_runtime_kill.py— a root whose identity mismatches never has itsgroup resolved from its number, even with
returncodeunset, while one whoseidentity holds still does, through the pinned call and with the identity recorded
at spawn; a root whose identity cannot be pinned sends nothing and falls through
to the vouched path. A reaped root signals
SIGTERM, waits the grace andescalates to the members the
SIGTERMvouched, both passes carrying the spawninstance; a teardown cancelled inside the grace still escalates; no vouching
member means no grace and no escalation; a
PermissionErrornever reaches thegroup fallback. The three verdicts are asserted on win32, linux and darwin alike,
including that a failed read does not warn about the root. The leak line is
asserted on a vouching host as well: it speaks when the group still holds members
and names the vouch as the cause, and stays quiet when the group is empty, on
macOS-shaped and Windows-shaped hosts too. A host that cannot vouch still
attempts the vouched path, so the diagnostic can never become routing. A teardown
whose
wait()timed out sends exactly oneSIGTERMand one escalation.test/test_pid_lifecycle.py— delivery is asserted twice over, once per seam(pidfd present, pidfd absent), so both must satisfy the same contract; one test
pins that the identity check happens after the descriptor is open and fails when
the two are swapped; the four errnos are split by meaning, with ENOSYS and EPERM
taking the numeric path and EMFILE and ENOMEM skipping the member, plus the
version skew where only the send reports no syscall.
_signal_orphaned_runtime_groupsignals each vouched member by identity (highestpid first, no group signal), skips a member recycled between vouch and signal,
sends nothing without a vouch or an instance, re-signals on escalation only
members still alive under the same start id, treats a refused or vanished member
as nothing rather than aborting the teardown, refuses pgid 0/1 and our own group.
_marked_group_memberson a fixture/proc: only live, marked,identity-bearing members of the group count, and a marked member without runtime
identity does not vouch. Those two assert on every host — the reader is
platform-gated, so the test pins the platform rather than skipping, which would
have left the vouching rules unasserted exactly where a wrong kill is worst.
test/test_process_tree.py— a root whose identity holds at walk time and isgone before the write causes no write.
test/test_acp_spawn_offload.py— the spawn env carries the instance and itequals
process_instance. Main's own contracts in this file cover the rest: afailed spawn must unwind its live tree, and a raise from the identity read must
still reap the process. Its platform stand-in answers the pinned call the same
way it answers the plain one once the recorded identity matches, so those
assertions are unchanged.
Manual verification
Not verified live: it needs a gateway spawning real sandboxed subagent sessions
with a launcher that dies mid-spawn. The reporter's fault-injection recipe on
#11369 (terminate the launcher, then let the runtime tear down) is the manual
check; the unit tests pin the contract.
Related Issues
no linked issue: nothing here closes on merge. This is a follow-up to #11565,
and the report it answers arrived after #11369 was already closed. Addresses the
post-close report on #11369 (2026-09-18, second host). The
_untrack_child_pidsfinding overridden on #11565 stays deferred there and is not shipped here.
Pattern harvest
Rule candidate: review-prompt
Pattern: one falsey value standing for both "measured false" and "could not
measure".
except (OSError, ProcessLookupError): passaround a signal collapses"the target is gone" with "the target refused us"; a
Nonefrom an identityreader collapses "this is someone else" with "nobody looked"; an
OSErrorfrom acapability call collapses "this kernel lacks it" with "it refused me". The
authorization may be the same for both, but the diagnostic never is.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)