Skip to content

fix(acp): signal a runtime's process group after its root has died - #11720

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/kill-tree-when-root-died
Sep 20, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/kill-tree-when-root-died

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

A second host, on 0.7.0.4 with #11565's tracking already backported, still
leaked: 10 live kiro-cli acp + kiro-cli-chat pairs, PPID 1, ~400 MB each,
4.0 GB total, none in either PID file. Every one maps to a subagent session
whose sandbox launcher exited ~3 s after spawn. Plain SIGTERM reaped 18 of 18
within 8 s. The leak is a signal never delivered, not a process that refuses
to die.

The signal path is killpg(getpgid(root)). getpgid raises once the root has
exited; _kill_inner swallowed that, saw pid_exists(root) false, took the
"killed" branch and untracked it. Nothing in the group was signalled.

Why it matters

This is the tree the snapshot in #11565 cannot cover — a root that died
before its first scan — and the tracked sweep cannot either, since nothing
was recorded. It is the only shape observed on that host. Subagent spawns
happen in batches of 2–3, so it accumulates at the rate of one tree per
retried spawn.

What changed (motivation → approach → change)

A dead root's pid is still its group id, so a reaped root starts a second
teardown path instead of ending the first — and that path aims at members it can
prove are ours, never at the number.

Area Before After
src/kiro_crew/acp/runtime.py ProcessLookupError from the tree signal swallowed; root untracked _signal_tree resolves the group from the root's number only while the root's live start id matches the one recorded at spawn; a root that is gone or unreadable starts the vouched path: signal the members, then the same grace and a SIGKILL pass
src/kiro_crew/acp/runtime.py the tree signal addressed the root by number, from a deferred executor kill_process_tree_pinned holds the handle that verified the identity across the terminate, so the number cannot be recycled in between; False means unconfirmed, so nothing is sent and the vouched path runs instead
src/kiro_crew/acp/runtime.py returncode is None accepted as proof the number was held not proof: asyncio's watcher reaps in the background and propagates the code a callback later, so only the identity read answers
src/kiro_crew/acp/runtime.py the identity check answered yes or no three answers: holds, mismatch (both ids read, they differ), unknown (nothing recorded, or the read failed). All refuse authorization; only mismatch may describe the root
src/kiro_crew/acp/runtime.py a teardown that reached nothing returned silently one WARNING naming the root pid, the signal, which identity verdict it got, and whether the host cannot vouch or no member vouched — guarded by _pgroup_has_member_besides, so it speaks on every platform while the group still holds something and stays quiet once it is empty
src/kiro_crew/acp/runtime.py the vouched escalation block ran even after the timeout path had escalated it runs only when wait() did not time out, so one grace and one SIGKILL per member instead of two
src/kiro_crew/acp/runtime.py a denied signal and a reaped root shared one except any other OSError stays final — the root is there and we were refused
src/kiro_crew/acp/runtime.py the root's identity was recorded after the suspended-spawn resume recorded as the first thing inside the reap guard, so every failure path after it holds a root whose tree a teardown can still resolve
src/kiro_crew/session_pid.py — _signal_orphaned_runtime_group signals each MEMBER _marked_group_members vouched; never the reusable group number
src/kiro_crew/session_pid.py a member was re-read and then signalled, with a window between the two _signal_pid_by_identity opens a pidfd, verifies the identity AFTER the open, then pidfd_send_signal. A descriptor pins one process, so a recycled number is unreachable through it. A kernel without the pidfd SYSCALL keeps the re-verified os.kill, keyed on the errno (ENOSYS/EPERM) rather than on the Python attribute, which exists on every Linux build; a refusal of a call the kernel HAS skips the member instead
src/kiro_crew/session_pid.py the Linux-only environ read was an inline platform test group_vouching_available() states it once, for the reader that acts on it and the one that reports it
src/kiro_crew/constants.py one generic marker on the child env spawn() also mints a per-spawn KIROCREW_SPAWN_INSTANCE and keeps it as process_instance — the incarnation pin a recycled root pid cannot fake
docs/system-specs/modules/acp-client.md, docs/system-specs/modules/session.md — the reaped-root path, the three-valued verdict, the audible leak, and the residuals below
test/test_acp_runtime_kill.py, test/test_pid_lifecycle.py, test/test_process_tree.py, test/test_acp_spawn_offload.py — the vouching rules, the three verdicts on win32/linux/darwin, the pinned tree signal, and when the leak line speaks
test/fixtures/acp_launch_goldens.json, test/acp_launch_capture.py, test/test_acp_launch_goldens.py launch golden knew five env keys KIROCREW_SPAWN_INSTANCE is a volatile key (<spawn-instance>): that the child RECEIVES it is pinned, its per-spawn value is not

Why vouch instead of aiming at the number. A dead root's pid can be handed
to a fresh session leader, which is a marked session leader too — so a signal
aimed by number and marker alone would terminate that runtime's live session. The
per-spawn instance is the pin the marker cannot supply: it says THIS spawn's
tree. No vouching member means no signal, because declining costs a leak the
sweep reports while acting costs an unrelated process.

Why a handle instead of a number, on both halves. Re-reading an identity and
then acting on the number leaves a window the recycled number can land in.
Narrowing it cannot close it — a pid is a key the kernel may reissue, not a
handle. For a member that handle is a pidfd, so open → verify → signal makes the
check binding for as long as the descriptor is held; the ordering does the work,
so a test pins it by having the fake open change what the identity reader
returns. For the root it is kill_process_tree_pinned, which keeps the query
handle that verified the identity open across the terminate — the tree signal is
deferred to an executor and, on Windows, resolved by a separate taskkill, which
is precisely where an unpinned number can change hands.

Why the identity is recorded earlier. A spawn failing between the process
becoming live and the identity read left a LIVE root with nothing recorded, and a
teardown may only resolve a group while the recorded identity matches — so that
root's whole tree went unsignalled, which is the leak the surrounding guard
exists to reap.

Why the verdict has three values. get_process_start_id returns None both
for a failed read and for a pid that is gone, so one boolean reported a cause
nobody had checked. Authorization is unchanged; only what the logs may claim is.

Why the escalation never re-resolves the group. getpgid on a recycled root
pid succeeds for whoever holds it now, so after the first pass the target set is
fixed: members it vouched, still alive under the same start id.

Why an unreached tree is audible, on every platform. Off Linux the environ
read cannot run at all; on Linux the members can be there and simply not vouch —
a sandbox that scrubbed the token, an unreadable environ, a missing argv
identity. Both leave the tree to the orphan sweep, which is the trade this keeps:
a leak over a signal to a stranger. What was wrong was the silence. The guard is
the group rather than the platform, so the line fires whenever something is still
in the group and stays quiet when it is empty, which is the ordinary teardown.

Backwards compatibility

Compatible: no signature, field, key or format that main accepts is rejected,
renamed or removed. _untrack_child_pids, _prune_dead_descendants and
AcpClient._reset_state keep main's signatures and behaviour exactly — an
earlier revision of this branch scoped the untrack by parent pid and start
token, and that is withdrawn here as an item belonging to #11565, not to this
defect. One new env key travels to spawned children and one new module-level
predicate is added; both are additive.

Known and deferred, in this diff

Residual Bound Why not here
A second cancel while the shielded SIGKILL pass is awaited raises at the await and leaves it running unawaited shield bounds one cancellation; the members were already vouched and SIGTERMed Closing it needs machinery beyond this defect. The comment and the spec no longer claim immunity
The reaped-root grace is a flat _KILL_TERM_TIMEOUT sleep, never waiting on member liveness latency only, and now paid at most once per teardown: N such teardowns pay N×5 s against the shutdown budget. Every vouched member is still reached by the escalation A liveness poll is a new mechanism; correctness holds without it
AcpClient._kill_process keeps the same reaped-root window its spawn-time descendant snapshot covers every client root that dies after the first scan The client holds no spawn instance, so the vouched path refuses for it by construction; reaching its tree is a change to the client's own spawn path, which the intent names as a non-goal

Tests

test/test_acp_runtime_kill.py — a root whose identity mismatches never has its
group resolved from its number, even with returncode unset, while one whose
identity holds still does, through the pinned call and with the identity recorded
at spawn; a root whose identity cannot be pinned sends nothing and falls through
to the vouched path. A reaped root signals SIGTERM, waits the grace and
escalates to the members the SIGTERM vouched, both passes carrying the spawn
instance; a teardown cancelled inside the grace still escalates; no vouching
member means no grace and no escalation; a PermissionError never reaches the
group fallback. The three verdicts are asserted on win32, linux and darwin alike,
including that a failed read does not warn about the root. The leak line is
asserted on a vouching host as well: it speaks when the group still holds members
and names the vouch as the cause, and stays quiet when the group is empty, on
macOS-shaped and Windows-shaped hosts too. A host that cannot vouch still
attempts the vouched path, so the diagnostic can never become routing. A teardown
whose wait() timed out sends exactly one SIGTERM and one escalation.

test/test_pid_lifecycle.py — delivery is asserted twice over, once per seam
(pidfd present, pidfd absent), so both must satisfy the same contract; one test
pins that the identity check happens after the descriptor is open and fails when
the two are swapped; the four errnos are split by meaning, with ENOSYS and EPERM
taking the numeric path and EMFILE and ENOMEM skipping the member, plus the
version skew where only the send reports no syscall.
_signal_orphaned_runtime_group signals each vouched member by identity (highest
pid first, no group signal), skips a member recycled between vouch and signal,
sends nothing without a vouch or an instance, re-signals on escalation only
members still alive under the same start id, treats a refused or vanished member
as nothing rather than aborting the teardown, refuses pgid 0/1 and our own group.
_marked_group_members on a fixture /proc: only live, marked,
identity-bearing members of the group count, and a marked member without runtime
identity does not vouch. Those two assert on every host — the reader is
platform-gated, so the test pins the platform rather than skipping, which would
have left the vouching rules unasserted exactly where a wrong kill is worst.

test/test_process_tree.py — a root whose identity holds at walk time and is
gone before the write causes no write.

test/test_acp_spawn_offload.py — the spawn env carries the instance and it
equals process_instance. Main's own contracts in this file cover the rest: a
failed spawn must unwind its live tree, and a raise from the identity read must
still reap the process. Its platform stand-in answers the pinned call the same
way it answers the plain one once the recorded identity matches, so those
assertions are unchanged.

Manual verification

Not verified live: it needs a gateway spawning real sandboxed subagent sessions
with a launcher that dies mid-spawn. The reporter's fault-injection recipe on
#11369 (terminate the launcher, then let the runtime tear down) is the manual
check; the unit tests pin the contract.

Related Issues

no linked issue: nothing here closes on merge. This is a follow-up to #11565,
and the report it answers arrived after #11369 was already closed. Addresses the
post-close report on #11369 (2026-09-18, second host). The _untrack_child_pids
finding overridden on #11565 stays deferred there and is not shipped here.

Pattern harvest

Rule candidate: review-prompt
Pattern: one falsey value standing for both "measured false" and "could not
measure". except (OSError, ProcessLookupError): pass around a signal collapses
"the target is gone" with "the target refused us"; a None from an identity
reader collapses "this is someone else" with "nobody looked"; an OSError from a
capability call collapses "this kernel lacks it" with "it refused me". The
authorization may be the same for both, but the diagnostic never is.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and chenmingwei23 September 18, 2026 07:11
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Intent: Reach the runtime tree that #11565 cannot: one whose root died before any descendant was snapshotted. The root's pid still names its process group; teardowns now signal that group once a live member vouches for it by identity. Also lands the untrack scope and test pin deferred from #11565.
Not a goal: Giving the report-only orphan gate kill authority (that needs a maintainer ruling -- separate issue), or changing AcpClient's teardown, which already sweeps its snapshot.

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 74eb4a29ddf01c4d472b3ab4e039691e24b73ecb — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Run the #11369 fault-injection recipe live once: the environ-token vouch surviving a real sandboxed launcher's death is the only link the fixture-/proc tests can't prove.

[DESIGN-REVIEWED] 74eb4a2

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 74eb4a29ddf01c4d472b3ab4e039691e24b73ecb and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 74eb4a2

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 74eb4a29ddf01c4d472b3ab4e039691e24b73ecb: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 74eb4a29ddf01c4d472b3ab4e039691e24b73ecb — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify on a real pre-5.3 or seccomp-filtered kernel that pidfd_open EPERM/ENOSYS means "syscall absent" — no test host exercises that errno split.

What this change ships

Inventory (10 items) — 10 justified

Intent: stop leaking a runtime's process group (agent + chat, ~400 MB each) when its root dies before teardown — a FIX, with a field reproduction and tests that fail on base.

  1. A teardown whose root already died now signals the surviving group members instead of returning silently — justified
  2. Every runtime child carries a per-spawn KIROCREW_SPAWN_INSTANCE env token, the pin a recycled pid cannot fake — justified
  3. The by-number tree kill runs only while the root's recorded identity holds, via the pre-existing kill_process_tree_pinned — justified
  4. returncode is None is no longer trusted as proof the pid is still held — justified
  5. Vouched members get the same grace then one SIGKILL pass; a mid-grace cancel still runs it shielded — justified
  6. A denied signal (non-ProcessLookupError OSError) ends the signalling instead of being swallowed — justified
  7. The root's identity is recorded first in the spawn guard, before the suspended-spawn resume — justified
  8. A teardown that reaches nothing while the group still holds members logs one WARNING, on every platform — justified
  9. The identity verdict is three-valued; only a measured mismatch may describe the root in logs — justified
  10. Members are signalled through a pidfd (open → verify → signal) with an errno-keyed numeric fallback — justified

Notes from counting: kill_process_tree_pinned exists on base (platform_compat.py:4899) — consumed, not added. group_vouching_available has 2 real consumers. The one same-cause sibling, AcpClient._kill_process (client.py:7950), is explicitly declared deferred in the spec addition with its residual named. The pidfd pattern also exists in session_scope_reap._pidfd_signal_owned:465, but its verification is cgroup-scope-bound, so this is not a second spelling of the same capability. The periodic scope reaper is systemd/cgroup-gated and sweep-time; it does not already do this teardown-time job.

[FIRST-PRINCIPLES-REVIEWED] 74eb4a2

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 74eb4a29ddf01c4d472b3ab4e039691e24b73ecb — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 74eb4a2

Verdict parsed from the review's SHA-scoped output markers for commit 74eb4a29ddf01c4d472b3ab4e039691e24b73ecb.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 74eb4a29ddf01c4d472b3ab4e039691e24b73ecb: <one-sentence reason>

@iamwhatever

Copy link
Copy Markdown
Contributor Author

The one blocking finding on 9a8d06692b is addressed in ee5a326b88: _signal_orphaned_runtime_group now catches every OSError from the group signal, not only ProcessLookupError. A refused signal is logged and counted as nothing reached, so _kill_inner finishes clearing its state and pruning its PID entries whatever the kernel answered; the orphan sweep retries a refused group on its own cadence. Pinned by test_a_refused_signal_does_not_abort_the_teardown.

The three red backend/coverage jobs on the previous head were test_backend_cards.py::test_every_capability_set_is_classified_by_the_card, which the base this branch sat on (b2a1d90915) had just broken and 2960ec6a69 fixed on main minutes later; this head is rebased past it.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

The blocking finding on ee5a326b88 is addressed in the new head: the escalation no longer re-resolves the group from the dead root's number.

_signal_orphaned_runtime_group now returns the members it vouched as pid -> start id, and _kill_inner hands that map back on the SIGKILL pass as expected. The group is signalled again only when at least one vouched member is still alive under the same start id; a reused pid, an unreadable identity, or every vouched member gone all refuse, since a released root pid taken by a fresh marked runtime names a group that vouches exactly as well as ours did. Pinned by test_escalation_re_signals_only_the_group_it_vouched, test_escalation_refuses_a_group_none_of_whose_vouched_members_survive, test_escalation_refuses_when_a_vouched_member_has_no_readable_identity, and the runtime test asserting the SIGKILL pass carries the SIGTERM pass's members.

The three red backend/coverage jobs were test_ci_fleet_routing_expression_parity.py, which main fixed in 1ae8f6ee48 after this branch's base; rebased past it.

@iamwhatever
iamwhatever force-pushed the fix/kill-tree-when-root-died branch from ee5a326 to 52864ff Compare September 18, 2026 22:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

The blocking finding on 52864ffa8d is addressed in a45ecc9b64, and this round changes the design rather than the guard: the group fallback now has an incarnation pin, not just the generic marker.

spawn() mints a per-spawn KIROCREW_SPAWN_INSTANCE into the child's environment (kept on the runtime as process_instance), so every descendant of THIS spawn carries a token no other spawn has. _marked_group_members(pgid, instance) counts a member only when its /proc/<pid>/environ carries that exact value alongside KIROCREW_SPAWNED and a runtime argv identity, and _signal_orphaned_runtime_group refuses outright without an instance. A fresh runtime that took the dead root's pid leads a group that carries the marker just as well — but not this instance — so it cannot vouch on the first pass any more than on the escalation. The expected start-id check from the previous round stays as the second pin on the escalation pass.

Pinned by TestRuntimeSpawnCarriesItsInstance (the child env carries the instance and it equals process_instance), the _marked_group_members fixture reading one group as two different spawns and returning disjoint members, test_refuses_to_signal_without_an_instance, and the runtime kill test asserting both passes carry the instance read before the kill clears it.

@iamwhatever
iamwhatever force-pushed the fix/kill-tree-when-root-died branch from 52864ff to a45ecc9 Compare September 18, 2026 23:08
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

All three findings on a45ecc9b64 are addressed in f72b4aa774, and the two the adjudicator flagged as edge cases are fixed rather than overridden.

Opus — the escalation re-resolved the group from the dead root's number. _signal_tree now skips kill_process_tree entirely when it carries expected: the first pass already established the root is gone, and getpgid on that number succeeds for whatever fresh runtime holds it now. A reaped root is never re-resolved; the escalation goes only through the vouched path.

GPT F1 — the vouched group could be replaced before killpg. There is no killpg any more. _signal_orphaned_runtime_group signals the vouched MEMBERS with os.kill, each re-verified by get_process_start_id at the instant of the signal, highest pid first. A pid plus its start instant names one process for good; the group number never did. A member recycled between the vouch and the signal, or with no readable identity, is skipped.

GPT F2 — untracking ignored the child's identity. _untrack_child_pids compares a record's start token against the line's third field and leaves a line whose token differs (a pair re-tracked by a live runtime); a legacy two-field line is still removed.

Pinned by test_signals_each_vouched_member_by_identity (per-member signals, no killpg), test_a_member_recycled_between_vouch_and_signal_is_skipped, test_untrack_child_pids_keeps_a_line_whose_identity_is_not_ours, and the existing escalation tests re-pointed at the per-member path.

@iamwhatever
iamwhatever force-pushed the fix/kill-tree-when-root-died branch from a45ecc9 to f72b4aa Compare September 18, 2026 23:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@iamwhatever
iamwhatever force-pushed the fix/kill-tree-when-root-died branch from 97f10c1 to b209f90 Compare September 19, 2026 20:13
@iamwhatever

Copy link
Copy Markdown
Contributor Author

b209f900f2a7: Backend Tests (Windows) (4) was mine, and the fix is a
restructure rather than another stub.

Three tests there pin platform_compat.IS_POSIX True so the POSIX routing is
exercised on every host. The previous push had put group_vouching_available()
into the branch that decides whether to ATTEMPT the vouched group path, so on
Windows the attempt was skipped entirely and those tests saw no signal at all.
On Linux that is invisible, because the capability is true there.

It was also redundant: _marked_group_members already owns the platform answer
and returns an empty map where it cannot read, so the attempt is safe to make
anywhere POSIX. The warning is a reporting concern, so it now reads what the
attempt returned instead of deciding whether it happens — the routing is byte-for-
byte what it was before that push, and the capability is consulted only to choose
what to say about an empty result.

test_kill_still_attempts_the_vouched_path_on_a_host_that_cannot_vouch forces the
two apart — POSIX host, vouching unavailable — and asserts the attempt is made. It
fails against a mutant that puts the capability back into the routing condition,
which is exactly the shape that went red, so this cannot regress silently on a
platform the local run cannot reach.

Rebased onto 59e7560615. GPT and Opus were both clean on 97f10c1cdb48;
PR Readiness was a rollup of the shard. Local: 972 tests across the affected
files, black, isort, flake8, mypy and the repo gates green.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 19, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: an OSError from pidfd_open now returns False and logs, instead of falling back to os.kill.

  • except OSError falls back to numeric signaling after pidfd exhaustion, reopening PID reuse span=53b1c6cc4bb2

fixed with the finding's own named fix, and it is right: pidfd_open failing with EMFILE or ENOMEM says nothing about the target, and signalling by number there reopens exactly the window the descriptor closes. The distinction that matters is "this kernel has no pidfd API" versus "this kernel has one and refused", and only the first may use the number. The refusal branch now fails closed — the member is skipped and left to the orphan sweep, with a WARNING naming the pid and signal so the skip is visible.
pinned by test_a_refused_pidfd_does_not_fall_back_to_signalling_the_number, which makes pidfd_open raise OSError(EMFILE) and fails the test if os.kill is reached. The absent-API path keeps its own coverage through the per-seam parametrization, so both branches stay asserted.
class: a fallback is for a capability that is missing, never for one that is present and refused. Where the two are conflated, the degraded path silently becomes the normal path under load — which is when this one would have fired.

@iamwhatever
iamwhatever force-pushed the fix/kill-tree-when-root-died branch from b209f90 to a4c4fcd Compare September 19, 2026 20:49
@iamwhatever

Copy link
Copy Markdown
Contributor Author

a4c4fcdbd6c7: GPT's advisory on b209f900f2a7 is fixed, and the remaining reds
on this PR are main's, not this diff's.

The advisory. pidfd_open failing with EMFILE or ENOMEM says nothing about the
target, so falling back to os.kill there reopened the reuse window the descriptor
exists to close. The branch now fails closed: the member is skipped and left to the
orphan sweep, with a WARNING naming the pid and signal. The numeric path remains
for a kernel with no pidfd API at all, which keeps its coverage through the
per-seam parametrization.

Why 24 checks are red, and why no push fixes them. All of them reproduce on
pristine origin/main (1c7f963706545f), whose own CI run is also failing:

Red Cause, measured on pristine main
Backend Tests × 16 (Linux + Windows) ImportError: cannot import name 'member_for_store' from 'kiro_crew.member_essential_context', plus a collection error in test/test_member_memory_runtime.py (patch_private_memory_supported absent from test/member_memory_helpers.py)
Backend Lint & Type Check (3.12) 5 mypy errors in src/kiro_crew/context.py and src/kiro_crew/dashboard/state.py
Coverage Gate, PR Readiness, the E2E and contract jobs rollups and downstreams of the above

None of those files are in this diff. Across the sampled shards there are zero
failures in any test file this PR touches, and in Backend Tests (Windows) (4) —
red earlier for a genuine regression of mine — all 68 failures are the
member_for_store import and none are in test_acp_runtime_kill.py, so that fix
is confirmed on the platform that caught it.

Reviewer state on b209f900f2a7 before this push: GPT no blocking issues, Opus
clean, Design PASS, First Principles PASS. This PR needs main's memory-refactor
breakage cleared, then a re-run; there is nothing further to change here for it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 19, 2026
@iamwhatever
iamwhatever force-pushed the fix/kill-tree-when-root-died branch from a4c4fcd to 686e03f Compare September 19, 2026 20:56
@iamwhatever

Copy link
Copy Markdown
Contributor Author

686e03fe09ae: base pinned below #11947. The PR's code is unchanged — this push
only moves where it sits.

1c7f963706 (#11947) makes src/kiro_crew/context.py import member_for_store,
require_private_memory_execution and private_runtime_log_dir, and
src/kiro_crew/dashboard/state.py call
ContextBuilder.live_memory_mode_for_session. git grep on origin/main finds
zero definitions for any of the four, and test/member_memory_helpers.py no
longer carries patch_private_memory_supported. So on main itself mypy reports 5
errors and every backend shard fails to import or collect — main's own CI run on
that commit is red, and the 24 reds here were inherited through the merge ref.

This diff touches neither file, so there is nothing in it to fix for that. The
single commit is now on 59e7560615, the base where this PR was already
reviewer-green at 97f10c1cdb48. Measured on the new base: mypy passes over all
1647 files, test/test_member_memory_runtime.py collects its 105 tests instead of
erroring, and 991 tests pass across the files this PR touches. All 14 applicable
repo gates pass against that merge base.

6c21f0d4d9 (#11960) also sits below #11947 and would be one commit fresher; the
older base is used because the reviewer lanes have already cleared this diff on
it. Rebase forward as soon as #11947's four symbols land on main — git merge-base
will move on its own once they do, and nothing here needs to change for it.

@bolichen97

Copy link
Copy Markdown
Contributor

35e9535540: rebased onto main at 39892f5c10 (after #12091 reverted #11947). The diff against base is byte-identical to 686e03fe09ae -- this push only moves where it sits, so the merge ref no longer carries the member_for_store / require_private_memory_execution / private_runtime_log_dir ImportErrors that failed every backend shard on the previous heads. Local: 363 related tests pass, flake8/black clean.

@bolichen97

Copy link
Copy Markdown
Contributor

adfdc53ea0: rebased onto main at ddea8d79fc (after #12088 reverted #11960). Diff against base is byte-identical to 35e9535540; this push only moves where it sits. The _run_memory_identity_path / private_memory_execution_supported stale-reference errors and the codex capability-set assertion that failed shards 1/5/6/8 on the previous head were main-side (#11960 residue) and are gone from the base. Local: 363 related tests pass, flake8 clean.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: _PIDFD_ABSENT_ERRNOS splits "this kernel has no pidfd syscall" from "it has one and refused this call"; only the first uses the identity-verified numeric path.

  • except OSError treats ENOSYS as a refused pidfd, leaving vouched members unsignalled on pre-5.3 kernels despite the documented numeric fallback span=53b1c6cc4bb2

fixed, and the finding catches a real regression the previous round shipped. os.pidfd_open exists as an attribute on any Linux build, so the opener is None guard never fires for a missing SYSCALL — a kernel older than 5.3 reaches the open and is answered ENOSYS. Failing closed there signalled no member at all, which silently disabled this entire teardown path on those hosts while the docstring still promised the numeric fallback. ENOSYS now takes that fallback, as does EPERM, which is how a seccomp filter hides the syscall outright.
the distinction the previous round was reaching for is kept intact: EMFILE and ENOMEM mean the call exists and refused THIS open, so there is no handle and no atomic signal, and the member is skipped rather than signalled by a number that may have changed hands. pidfd_send_signal landed in 5.1 and pidfd_open in 5.3, so the two can disagree and the send is classified by the same split.
pinned by test_pidfd_errnos_split_absent_from_refused, parametrized over ENOSYS, EPERM, EMFILE and ENOMEM so each errno's side is asserted rather than inferred, plus test_a_send_that_reports_no_syscall_falls_back_to_the_number for the version skew. A mutant that fails closed on every errno fails them.
class: a capability probe that reads a language binding is not a probe of the kernel. Where the two can disagree, the errno IS the probe, and a fallback keyed on the binding alone will be dead exactly on the hosts that need it.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

a45161f69293: GPT's advisory on 35e9535540 is fixed, and main's breakage has
cleared.

The advisory was a real regression from the round before it. os.pidfd_open
exists as an attribute on any Linux build, so the opener is None guard never
fires for a missing SYSCALL — a kernel older than 5.3 reaches the open and is
answered ENOSYS. The previous round's fail-closed branch then signalled no
member at all, which disabled this teardown path entirely on those hosts while the
docstring still promised the numeric fallback. The errnos are now split by what
they mean: ENOSYS and EPERM say the capability is absent, so the
identity-verified os.kill runs; EMFILE and ENOMEM say the call exists and
refused this open, so the member is skipped rather than signalled by a number that
may have changed hands. pidfd_send_signal (5.1) and pidfd_open (5.3) can
disagree, so the send is classified the same way.

test_pidfd_errnos_split_absent_from_refused asserts each of the four errnos on
its own side, test_a_send_that_reports_no_syscall_falls_back_to_the_number
covers the version skew, and a mutant that fails closed on every errno fails them.

On the reds. Main has reverted both #11947 (via #12091) and #11960 (via
#12088), which is what the 24 and then 10 inherited failures were. On this base,
test_subagent_continuable.py, test_acp_capability_sets_leaf.py,
test_spawn_reasoning_effort.py, test_real_adapter_gate.py and
test_member_memory_runtime.py all pass locally — 283 tests across the four files
that were red, none of which this diff touches. Issues #12082 and #12083 tracked
that breakage.

Rebased onto 891e6de8da5e. The previous head adfdc53ea06c was this same commit
rebased forward by another driver with no content change; this push is that plus
the errno fix, nothing dropped. Local: 996 tests across the files this PR touches,
mypy over 1647 files, and all 14 applicable repo gates green.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the unreached-tree WARNING is guarded by the group read alone, on every platform, and its reason names whether the host cannot vouch or no member vouched.

  • On Linux a reaped root whose members fail to vouch is silent — the sandbox scrubbing KIROCREW_SPAWN_INSTANCE would reproduce the reported leak invisibly span=2ec1eaee6826

fixed, and the Watch is right about the worst part: the line sat under elif not group_vouching_available(), so it could never fire on Linux — the one platform where the vouched path runs, and the platform the leak was reported from. A group still holding members none of which vouched returned {} and read as a teardown that worked, which is the exact invisibility this change argues against.
the guard is now the group read alone: _pgroup_has_member_besides answers on every platform and is conservative when it cannot read, so a non-empty group always speaks and an empty one stays quiet, which is what keeps the ordinary teardown silent. The reason line distinguishes the two causes rather than assuming one — "no member of its group vouched for this spawn's incarnation" where the token is readable, "this host cannot vouch a process group by incarnation token" where it is not — so a scrubbed token, an unreadable environ or a missing argv identity are reported as what they are.
pinned by test_kill_reports_an_unreached_tree_on_a_vouching_host_too, parametrized over a group that still holds members and one already empty, asserting the line fires only in the first and names the vouch as the cause. It fails against the platform-keyed guard. The prior assertion that the line stays silent on a vouching host was wrong and is replaced, not loosened: the silent case is now the empty group, asserted on Linux as well as the two non-vouching shapes.
class: a diagnostic gated on a platform answers "can this host do the work", when the question is "was work left undone". Where an observation is available on every host, the observation is the guard.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: no
mechanism: the root's tree signal goes through platform_compat.kill_process_tree_pinned with the identity recorded at spawn, which holds the verifying handle across the terminate.

  • kill_process_tree(pid, sig) allows Windows to recycle the verified PID before deferred taskkill, terminating an unrelated tree span=081ce7af27e6

fixed with the finding's own named fix, and an earlier deferral of the same item is withdrawn. The lane has now stated this item as an advisory, then as blocking with an UPHOLD-FENCED adjudication, then as an advisory again on the same head; the substance did not change between those readings and it is correct in all of them. An in-process identity check followed by a taskkill /T that resolves the number from a separate process is a check that has already expired when it is used, and the executor hop widens that gap rather than narrowing it.
platform_compat.kill_process_tree_pinned exists for exactly this, documents exactly this window, keeps the query handle that verified the creation time open across the terminate — which is what reserves the number on Windows — and delegates straight through on POSIX where os.killpg is issued in-process. session_pid.py and apps/backend.py already call it for the same reason, so this is the codebase's own answer rather than new machinery. Per the fix's second half, the teardown returns only when it succeeds: False means the identity could not be pinned and nothing was signalled, so it falls through to the token-vouched path exactly as a non-holding identity does.
pinned by test_kill_pins_the_root_identity_across_the_tree_kill, which fails if the unpinned call is used and asserts the recorded identity is the argument, and by test_kill_falls_back_to_the_vouched_path_when_the_root_cannot_be_pinned. A mutant restoring the unpinned call fails both. Main's #11935 contract drives teardown through a partial platform stand-in, so that stand-in now answers the pinned call the same way it answers the plain one once the recorded identity matches; its assertions are untouched — the live tree must still come down and unrelated runtimes must still be spared.
class: a checked number is not a pinned handle, on the root exactly as on the members. Both halves of this teardown now hold something the kernel cannot reissue while they act — a pidfd for a member, a process handle for the root's tree.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

9e87ec877034: the root's tree signal is pinned, and the unreached-tree line now
speaks on Linux too.

The root is pinned across the terminate. _signal_tree went through
kill_process_tree(pid, sig), which on Windows resolves the number from a
separate taskkill process, deferred to an executor — so the in-process identity
check that authorised it had already expired by the time it was used, and the
number could have been recycled onto an unrelated tree.
platform_compat.kill_process_tree_pinned exists for exactly this: it keeps the
query handle that verified the creation time open across the terminate, which is
what reserves the number on Windows, and delegates straight through on POSIX where
os.killpg is issued in-process by this same interpreter. session_pid.py and
apps/backend.py already call it for the same reason. False means the identity
could not be pinned and nothing was signalled, so the teardown falls through to
the token-vouched path exactly as a non-holding identity does.

For the record on that lane: this item was stated as an advisory, then as blocking
with an UPHOLD-FENCED adjudication, then as an advisory again on the same head.
The substance did not change between those readings and it is correct in all
three, so it is fixed rather than argued.

Design's Watch — the silent Linux leak. The unreached-tree WARNING sat under
elif not group_vouching_available(), so it could never fire on Linux: a group
still holding members none of which vouched — a sandbox that scrubbed
KIROCREW_SPAWN_INSTANCE, an unreadable /proc/<pid>/environ, a missing argv
identity — returned {} and read as a teardown that worked. That is the reported
leak's own shape, on the platform it was reported from. The guard is now the group
read alone, so a non-empty group always speaks and an empty one stays quiet, and
the reason line distinguishes "no member of its group vouched" from "this host
cannot vouch by token" instead of assuming the latter.

Tests: the pinned call is asserted with the recorded identity and fails if the
unpinned one is used; an unpinnable root is asserted to reach the vouched path;
the leak line is asserted to fire on a vouching host when the group still holds
members and to stay quiet when it is empty. Mutants restoring either the unpinned
call or the platform-keyed guard fail them. Main's #11935 contract keeps its
assertions — its platform stand-in simply answers the pinned call the same way it
answers the plain one once the recorded identity matches, because a fixture has to
expose what production calls.

Rebased onto 080c481f43a6. Local: 995 tests across the affected files,
black/isort/flake8/mypy and the repo gates green.

kill_process_tree is killpg(getpgid(pid)), and getpgid raises once the
root has exited. AcpRuntime._kill_inner swallowed that, read the dead
root as "already killed", and untracked it — while the launcher's
children, the agent and its chat process stayed in the group, never
signalled, reparented to init and holding hundreds of MB each. A root
that dies a few seconds into its life predates every descendant
snapshot, so neither the recorded descendants nor the tracked sweep
could find that tree. Field evidence on a second host: 10 such trees,
4 GB, all subagent sessions whose launcher exited ~3 s after spawn,
every one absent from both PID files; plain SIGTERM reaped 18 of 18.

The root was spawned as a session leader, so its pid IS the group id
and the number outlives the process. AcpRuntime._signal_tree treats
ProcessLookupError from the tree kill as the start of a second path:
session_pid._signal_orphaned_runtime_group finds that group's members
and signals THEM, each re-verified by start id at the instant of the
signal -- never the group number, which is the dead root's pid and can
be handed to a fresh session leader at any moment. A member vouches
when _marked_group_members finds it carrying this spawn's
KIROCREW_SPAWN_INSTANCE -- a per-spawn token spawn() now puts on the
root's environment, inherited by the whole tree -- together with the
KIROCREW_SPAWNED marker and a runtime argv identity. The instance is
the incarnation pin the marker cannot supply: every runtime is a marked
session leader, so a root pid released to a fresh spawn names a group
that carries the marker just as well, and a signal aimed by number and
marker alone would terminate that fresh runtime's live session. No
vouching member means no signal; declining costs a leak the sweep still
reports, acting would cost an unrelated process. A vouched SIGTERM is followed by the same grace a
live tree gets and a SIGKILL pass aimed only at members the SIGTERM
vouched that are still alive under the same start id, since the dead
root's wait() returned at once and drove no escalation. The escalation
never re-resolves the root's number: getpgid on a recycled root pid
succeeds for the fresh runtime holding it. Any other OSError is final:
the root is there and we were refused. Linux only — the environ read
is — so macOS and Windows keep the missed reap rather than gain a wrong
kill.

_untrack_child_pids gains parent_pid and both teardown callers pass
their root; a record carrying a start token removes only a line whose
third field matches it. A child pid is reused like any other number --
both numbers of a pair can be -- and another live runtime can have
recorded the same pair under its root since ours exited; removing by
number alone took that runtime's line too.

_snapshot_descendants checks the root's identity before the walk and
again before the write, and only the first check was pinned. A test now
holds the identity through the walk and drops it before the write, and
asserts on the writer, so deleting the second check fails it.
@iamwhatever

Copy link
Copy Markdown
Contributor Author

74eb4a29ddf0: the three reds on 9e87ec877034 were the pinned call's own
follow-through. Both causes are mine, and neither changes production code.

A second partial platform stand-in. Backend Tests (3.12, 2) and
(Windows) (2) failed in test/test_runtime_cleanup_regression.py with
'types.SimpleNamespace' object has no attribute 'kill_process_tree_pinned'. That
file builds its own fake kernel, exactly as #11935's contract does, and production
now calls the pinned variant — so the stand-in exposes it, delegating to the same
fake tree teardown once the recorded token matches and refusing when it does not,
which is the real function's contract. Its assertions are untouched.

An assertion that only described one host. Backend Tests (Windows) (4)
failed on test_kill_uses_the_tree_kill_while_the_root_identity_holds, which
mocked the UNPINNED call. It passed on POSIX only because the real pinned function
delegates straight through there; on Windows the real one takes its own branch,
cannot open a handle for the fixture's pid, returns False and never reaches the
mock. The test's subject is "identity holds, so the tree is signalled", so it now
asserts the pinned call and that its argument is the identity recorded at spawn —
the same meaning, on every host.

To stop this class here rather than one shard per round, I enumerated every test
file that touches this seam and ran all of them: 1771 tests across
test_connections_warm, test_session_more_coverage, test_process_tree,
test_process_tree_kill, test_acp_runtime, test_acp_runtime_kill,
test_acp_client, test_acp_spawn_offload, test_mcp_gateway_stub_session_token,
test_runtime_cleanup_regression, test_codex_session_mcp, test_stop_kill_cancel
and test_pid_lifecycle. No file that stands in for platform_compat now lacks
the pinned call.

Rebased onto abedd2ed15d2. Local: 982 tests across the directly affected files
plus the 1771 above, with black, isort, flake8, mypy and the repo gates green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants