Repository navigation
fix(session): ask who owns a pid before every sweep signals it, and reconcile both ways - #14510
Conversation
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify on a live host that the reconciler's first passes only count — never kill — the browser/MCP populations it predicts dominate the unowned reading. What this change shipsInventory (10 items) — 9 justifiedIntent: FIX — stop record-driven sweeps from killing runtimes other sessions are mid-turn on, and give the two measured leak populations (unowned-alive 20–97, owned-dead 0→23; reproduction in the description) an owner.
Verified against the repo: [FIRST-PRINCIPLES-REVIEWED] b83dfbf |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS No owning spec documents the new reconciler or the per-pid gate semantics the sweeps now carry; session.md's sweep contract is now stale. Watch
Suggestions
[DESIGN-REVIEWED] b83dfbf |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/process_identity.py:880 -- the in-function False positive or not applicable? A repository writer can comment: |
6854c49 to
a06f4d0
Compare
102dfca to
1197d9e
Compare
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNo blocking findings. FINDING — src/kiro_crew/process_identity.py:947 — the teardown barrier is plain set membership ( FINDING — src/kiro_crew/runtime_reconcile.py:499 — [OPUS-REVIEWED] b83dfbf Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
c3833d4 to
9e8e4be
Compare
9e8e4be to
64cc6ee
Compare
6403523 to
7df6ae0
Compare
7df6ae0 to
9546959
Compare
9546959 to
db2936f
Compare
…econcile both ways Six kill paths signal a pid without asking whether a session is still using the process it names: both cleanup sweeps, the agent-scope reaper, the sub-agent orphan reconcile, the sub-agent reset ladder and the cron run teardown. Each carries a shield, and each shield is a union of pid sets gathered before the candidate scan and an event-loop hop before the signal -- so it can be stale by the time the signal lands, and the lease table, the one record that knows a session is mid-turn on that process, was not among the sets consulted. A pid can be shield-clean and leased in the same moment. Every one of those paths now asks authorize_runtime_kill about one pid at the decision point, which is the only place the answer cannot already be stale, and skips what it refuses. An unanswerable gate is itself a refusal: deferring housekeeping one tick costs less than killing a live runtime. A refusal never aborts a pass, so one unanswerable pid does not cost the sweep its other candidates. Two of the six are teardowns, and they need the opposite treatment as well. A reset releases its session's lease inside provider.shutdown(), the last step, so a reset that hung before it -- the only reason the last-resort kill runs -- still holds one. A gate asked then lets the session being destroyed refuse its own kill, and the refusal poisons the pid for the gateway's life. Both release the lease of the session they are destroying first, reaching it through tearing_down and matching on the handle's pid AND start identity, so a key with several teardowns in flight gives up only the lease of the process about to be signalled. What survives is another tenant's lease, which is the only thing that may withhold a signal. This is the rule the allocation path's own failure handler already states: release before the kill, or the cleanup refuses its own teardown. The scope reaper needed the gate moved, not added. Stopping the unit IS the kill there -- systemd terminates the whole cgroup -- so ownership is asked of every current member BEFORE the stop, and one leased member aborts the whole reclaim. A scope with a live tenant is not abandoned, whatever the scope-level verdict concluded before that tenant joined, and the next pass reclaims it once the lease is released. The caller records that refusal distinctly from a stop that was attempted and left members behind. runtime_reconcile.py is new and answers the other half: nothing compares the kernel's own process list against the records, in either direction. Measured on a live instance, 20 processes at rest and 57 to 97 under load run inside this install's agent slice with no record claiming them -- nothing will ever end those, because the sweeps that could are driven by the records they are missing from. The mirror image is measured too: records naming pids that are gone, climbing 0 to 23 over fifteen minutes. It reads both truths and acts on each disagreement. A record whose process is gone is retracted and its holder told once. A record whose pid now names a stranger counts in the dead direction, not the healthy one, and the current holder is never signalled. An unowned process is counted, and killed only when five things hold together: this install's spawn marker read from the kernel's exec-time environ, unowned on the previous pass with the SAME process identity, older than five minutes, the gate's permission, and an identity that still matches in the instant before the signal. At most five kills a pass. Everything else is counted and left alone, because absence from a record is evidence something is unclaimed, not that it is abandoned: the unowned population on a healthy host is mostly processes with real owners no session record describes, and killing on first sight would take a user's browser away and call it a leak fixed. An unreadable registry refuses the whole pass, since that single input makes every live process look unowned. The two counts publish as unowned_alive and owned_dead, the names the process-inventory harness uses for the same reconciliation read from outside the process, and the recycled pid is classified the same way in both.
iamwhatever
left a comment
There was a problem hiding this comment.
Holding approval: the new reaper can kill processes a live session still owns.
The lease gating added to the existing sweeps makes those paths more conservative, and that part looks good. The concern is the new RuntimeReconciler:
- It runs on every cleanup tick (
session_cleanup.py) and has no config switch. - It takes every pid in the agent slice and removes only those that are recorded, leased or claimed (
runtime_reconcile.py~L673-677). It does not skip descendants of a live tracked runtime. - A leftover pid gets
_kill_pid_tree(~L753) when it carries the spawn marker (every descendant inherits it), is older than 300s, was seen unowned on two passes, and its command line contains a harness name as a substring (kiro-cli,claude,goose, ...).
So an agent's own long-running command can be SIGKILLed with its whole tree while the parent session is still working, e.g. a nested kiro-cli chat probe or a pytest test/test_claude_* run. If kiro-cli spawns an untracked kiro-cli-chat worker (not verified), a live runtime could be killed too. The per-pass budget of 5 limits the blast radius but does not make the targets right.
Suggested fix:
- Skip any pid whose ancestry reaches a recorded or active runtime pid.
- Put the kill arm behind a config switch that is off by default (log-only until it is proven).
Problem / Motivation
Goal: every sweep, reaper and orphan-reconcile path asks who owns a pid before signalling it, and a two-way reconciler catches what the records and the kernel disagree about.
Measured on a live instance: 20 processes at rest and 57 to 97 under load run inside this install's agent slice with no record claiming them, and records naming pids that are gone climb from 0 to 23 in fifteen minutes. Nothing compares the kernel's process list against the records in either direction, so both populations grow silently.
Why it matters
A sweep kills a runtime somebody is using: every sweep decides from records gathered an event-loop hop before it signals, and the lease table -- the thing that knows a session is mid-turn -- was not among them. The user sees another chat die while theirs is still typing. And a leaked process is never reclaimed: its memory and file handles stay held until the host is rebooted.
Not a goal
cloud/login.pyrunspkill -f "kiro-cli acp"on an EC2 instance over SSM; the local lease table has no record for any pid there, so a gate on that call would permit every kill.What changed (motivation -> approach -> change)
authorize_runtime_killper pid at the decision point, behind a committed teardown barrierstop_unittook the whole cgroupstate.jsonrecordedruntime_reconcile.py(new)unowned_alive/owned_deadsession_pid._kill_pid_treeexpected_start, re-checked before the descendant read, before the child loop, and before the root signal; each child pinned to its discovery identity and its ancestryThe symptom is a sweep signalling a process another session is using. Each sweep held a shield of live pids gathered before the candidate scan and consulted it an event-loop hop before signalling, and that shield never included the lease table. So every gate-then-kill path now asks
authorize_runtime_killper pid at the decision point, and signals only insidecommit_runtime_teardown/release_runtime_teardown, because a verdict computed on a maintenance thread is a statement about the past while a turn can claim on the event loop in between. A refusal withholds that pid and logs why; it never aborts the pass.The reconciler answers the other half. It reads this install's agent slice from the kernel and the records, and acts on each disagreement: alive but claimed by neither table is counted, and killed only when it carries our spawn marker, has been unowned two passes running, is older than the floor, and the gate allows -- at most five per pass. Owned but dead is retracted from the source that holds it and the lease holder is told. A record whose pid now names a stranger (start identity differs) counts as dead and the current holder is never signalled. An unreadable registry or slice refuses the whole pass, because that one input makes every live process look unowned.
_kill_pid_treegains an opt-in identity pin because a re-check at the call site cannot close the window inside the walk: a candidate that exits during the descendant read frees its number, and a managed runtime that takes it next passes the argv gate. All four existing callers are untouched.Backwards compatibility
Compatible: every new parameter is optional with the prior behaviour as default; the reconciler kills nothing unless four independent conditions hold, and refuses entirely when it cannot read either side.
Tests
test_runtime_reconcile.py(new, 101 tests) pins every guard: two-pass confirmation, spawn marker, age floor, kill budget, gate at the decision point, lease-and-tenancy shield, unreadable registry/slice refusal, unsignalable-is-not-dead, recycled pid counts as dead, each of the six gate-then-kill paths withholding a leased pid, the barrier on every path, the identity pin at all three points, and per-child ancestry.test_cron_reaper.py,test_cron_run_claim.pyandtest_subagent.pyfollow the forwarded seam's new keyword. Twelve neighbouring test files pass 670 before and after. Every guard has a mutation that reddens its named test (84/84); the matrix and its per-guard table are in the review thread.Manual verification
Live probe on this branch against the real tables: a pid holding only a tenancy reads
unowned_alive=0while the claim is held and1with a normal reclaim once it is released.Screenshots / video
N/A -- no user-visible UI change.
Related Issues
Refs #14352, #14482, #14484, #14726
Pattern harvest
Rule candidate: review-prompt -- a teardown path must never consult a permission its own subject holds; release the subject's claim first, then ask. A gate copied from a housekeeping sweep onto a teardown path is this bug, and the two read identically at the call site.
Checklist