Skip to content

fix(session): ask who owns a pid before every sweep signals it, and reconcile both ways - #14510

Merged
iamwhatever merged 1 commit into
mainfrom
feat/reconcile-sweep-kill-paths-to-lease-w4
Sep 28, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
feat/reconcile-sweep-kill-paths-to-lease-w4

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: every sweep, reaper and orphan-reconcile path asks who owns a pid before signalling it, and a two-way reconciler catches what the records and the kernel disagree about.

Measured on a live instance: 20 processes at rest and 57 to 97 under load run inside this install's agent slice with no record claiming them, and records naming pids that are gone climb from 0 to 23 in fifteen minutes. Nothing compares the kernel's process list against the records in either direction, so both populations grow silently.

Why it matters

A sweep kills a runtime somebody is using: every sweep decides from records gathered an event-loop hop before it signals, and the lease table -- the thing that knows a session is mid-turn -- was not among them. The user sees another chat die while theirs is still typing. And a leaked process is never reclaimed: its memory and file handles stay held until the host is rebooted.

Not a goal

  • Remote hosts. cloud/login.py runs pkill -f "kiro-cli acp" on an EC2 instance over SSM; the local lease table has no record for any pid there, so a gate on that call would permit every kill.
  • Gating inside the kill primitives. This change gates their callers at the point each decides a pid is disposable; the bypass ratchet keeps the primitives honest.
  • Publishing the withheld-reason breakdown as counter fields. The reasons are on the SEL audit channel; a field with no production reader was removed on review and is not re-added here.

What changed (motivation -> approach -> change)

path before after
cleanup sweeps (orphan pids, untracked MCPs) shield gathered before the scan, no lease read authorize_runtime_kill per pid at the decision point, behind a committed teardown barrier
agent-scope reaper leases only; stop_unit took the whole cgroup survey reads leases and tenancies; one barrier per member, one late tenant abandons the reclaim
sub-agent orphan reconcile signalled whatever state.json recorded asks the gate; still writes the tombstone on refusal
sub-agent reset ladder, cron run teardown no gate gate, with the subject's own teardown lease released first
runtime_reconcile.py (new) -- kernel pids vs records, both directions, publishes unowned_alive / owned_dead
session_pid._kill_pid_tree argv gate only optional expected_start, re-checked before the descendant read, before the child loop, and before the root signal; each child pinned to its discovery identity and its ancestry

The symptom is a sweep signalling a process another session is using. Each sweep held a shield of live pids gathered before the candidate scan and consulted it an event-loop hop before signalling, and that shield never included the lease table. So every gate-then-kill path now asks authorize_runtime_kill per pid at the decision point, and signals only inside commit_runtime_teardown / release_runtime_teardown, because a verdict computed on a maintenance thread is a statement about the past while a turn can claim on the event loop in between. A refusal withholds that pid and logs why; it never aborts the pass.

The reconciler answers the other half. It reads this install's agent slice from the kernel and the records, and acts on each disagreement: alive but claimed by neither table is counted, and killed only when it carries our spawn marker, has been unowned two passes running, is older than the floor, and the gate allows -- at most five per pass. Owned but dead is retracted from the source that holds it and the lease holder is told. A record whose pid now names a stranger (start identity differs) counts as dead and the current holder is never signalled. An unreadable registry or slice refuses the whole pass, because that one input makes every live process look unowned.

_kill_pid_tree gains an opt-in identity pin because a re-check at the call site cannot close the window inside the walk: a candidate that exits during the descendant read frees its number, and a managed runtime that takes it next passes the argv gate. All four existing callers are untouched.

Backwards compatibility

Compatible: every new parameter is optional with the prior behaviour as default; the reconciler kills nothing unless four independent conditions hold, and refuses entirely when it cannot read either side.

Tests

test_runtime_reconcile.py (new, 101 tests) pins every guard: two-pass confirmation, spawn marker, age floor, kill budget, gate at the decision point, lease-and-tenancy shield, unreadable registry/slice refusal, unsignalable-is-not-dead, recycled pid counts as dead, each of the six gate-then-kill paths withholding a leased pid, the barrier on every path, the identity pin at all three points, and per-child ancestry. test_cron_reaper.py, test_cron_run_claim.py and test_subagent.py follow the forwarded seam's new keyword. Twelve neighbouring test files pass 670 before and after. Every guard has a mutation that reddens its named test (84/84); the matrix and its per-guard table are in the review thread.

Manual verification

Live probe on this branch against the real tables: a pid holding only a tenancy reads unowned_alive=0 while the claim is held and 1 with a normal reclaim once it is released.

Screenshots / video

N/A -- no user-visible UI change.

Related Issues

Refs #14352, #14482, #14484, #14726

Pattern harvest

Rule candidate: review-prompt -- a teardown path must never consult a permission its own subject holds; release the subject's claim first, then ask. A gate copied from a housekeeping sweep onto a teardown path is this bug, and the two read identically at the call site.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@chenmingwei23
chenmingwei23 requested a review from a team as a code owner September 27, 2026 18:46
@chenmingwei23
chenmingwei23 requested review from cruisercohen and removed request for a team September 27, 2026 18:46
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 27, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of b83dfbfb0fa62109545425b77033ad0a16ba4aff — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify on a live host that the reconciler's first passes only count — never kill — the browser/MCP populations it predicts dominate the unowned reading.

What this change ships

Inventory (10 items) — 9 justified

Intent: FIX — stop record-driven sweeps from killing runtimes other sessions are mid-turn on, and give the two measured leak populations (unowned-alive 20–97, owned-dead 0→23; reproduction in the description) an owner.

  1. Six sweep/reaper/teardown paths ask the ownership gate per pid at the decision point; a refusal withholds only that pid — justified
  2. New end-of-tick reconciler compares kernel truth against records both ways and publishes unowned_alive/owned_dead — justified
  3. Its kill arm needs marker + two-pass + age floor + gate, ≤5 per pass — justified
  4. Dead records retracted by verbatim row; the holding session notified once via the existing recycle callback — justified
  5. Every kill decision and kill phase writes a SEL audit row (allowed/refused/killed/failed) — justified
  6. Teardown releases the dying subject's own lease first (popped forwarded through the delegation facades) — justified
  7. Tenancy-epoch barriers close the verdict-to-signal window; the scope reaper surveys both tables pre-stop and aborts on a claim — justified
  8. _kill_pid_tree optionally pins the caller-read identity, re-checked at three points; all existing callers unchanged — justified
  9. Scope-reap ladder reads SIGTERM/SIGKILL via platform_compat (POSIX-identical) — rides along: not the fix; makes the new refusal tests runnable on Windows hosts
  10. black-baseline loses one entry — justified

Verified against the repo: runtime_ownership (gate, tenancy, epochs) predates this PR; the <socket>.backends sidecar the reconciler reads is real (mcp_gateway/manager.py:1593); the two remaining _kill_pid_tree record-driven call sites sit inside kill phases the sweeps now gate; the one ungated kill left (cloud/login.py remote pkill) is the declared non-goal; the test diffs delete no pins.

[FIRST-PRINCIPLES-REVIEWED] b83dfbf

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of b83dfbfb0fa62109545425b77033ad0a16ba4aff — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

No owning spec documents the new reconciler or the per-pid gate semantics the sweeps now carry; session.md's sweep contract is now stale.

Watch

session.md describes the orphan-PID/MCP sweeps as shield-protected ("Parked runtimes stay shielded from the orphan-PID sweep"), and the PR itself states the shield "can be stale... never included the lease table" — the gate-at-decision-point rule, the tenancy barrier, and the reconciler's four-condition kill ladder now exist only in module docstrings and the PR body. The next contributor routed to session.md by AGENTS.md edits a sweep against an invariant set that no longer describes what governs a signal.
Clears when: the same commit updates docs/system-specs/modules/session.md (or adds an owning spec the router names) covering the reconciler, its counters, and the per-pid gate on the six paths.

Suggestions

  • unowned_alive has a healthy-host baseline of 20–97 (the PR's own measurement) dominated by permanently-withheld legit processes, so a real leak hides inside load variance; publishing the withheld-reason breakdown (already tracked per pid for auditing) as counter fields would make the SLI actionable without breaking vocabulary parity with the inventory harness.

[DESIGN-REVIEWED] b83dfbf

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of b83dfbfb0fa62109545425b77033ad0a16ba4aff and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/process_identity.py:880 -- the in-function from kiro_crew.sel import sel violates top-level-imports -> Fix: import the module at file scope and call sel_module.sel(). (origin: validation)
[GPT-REVIEWED] b83dfbf

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b83dfbfb0fa62109545425b77033ad0a16ba4aff: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/runtime-lease-gate-and-release-sites-w3a branch from 6854c49 to a06f4d0 Compare September 27, 2026 19:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/reconcile-sweep-kill-paths-to-lease-w4 branch from 102dfca to 1197d9e Compare September 27, 2026 19:15
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed b83dfbfb0fa62109545425b77033ad0a16ba4aff — this comment is updated in place on each push.

Review details

No blocking findings.

FINDING — src/kiro_crew/process_identity.py:947 — the teardown barrier is plain set membership (self._committed.add(pid) / .discard(pid) in runtime_ownership.RuntimeTenancy), so with six new teardown_barriers holders the first finally's release_runtime_teardown(pid) cancels every other holder's barrier — a sweep holding barriers across await asyncio.to_thread(self._deps.kill_confirmed_and_writeback, ...) loses its barrier to a reaper task that commits and releases the same pid on the freed loop, and claim() then stops raising RuntimeTeardownCommitted for a pid about to be SIGKILLed → Fix: in the (untouched) runtime_ownership.py, make _committed a per-pid holder count the way PidRefcount already does, or give each holder a token release_teardown must present.

FINDING — src/kiro_crew/runtime_reconcile.py:499 — kernel = self._slice_pids() treats instance_slice_pids()'s set() for an unresolvable slice (slice_dir, _why = _instance_scope_dir() discards the reason on macOS, Windows, without cgroup v2 delegation, or on a degraded instance token) as a successful read, so emit_counter("session.runtime_reconcile", ...) publishes unowned_alive=0 where the kernel population was never read — indistinguishable from a host with genuinely no leaked runtimes, and contradicting instance_slice_pids' own docstring claim that the reconciler treats an empty kernel reading as nothing to compare → Fix: surface the _why already computed and have run_once return supported=False, reason=<why> for an unresolvable slice, leaving set() to mean a readable, empty slice.

[OPUS-REVIEWED] b83dfbf

Verdict parsed from the review's SHA-scoped output markers for commit b83dfbfb0fa62109545425b77033ad0a16ba4aff.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b83dfbfb0fa62109545425b77033ad0a16ba4aff: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/reconcile-sweep-kill-paths-to-lease-w4 branch from c3833d4 to 9e8e4be Compare September 27, 2026 21:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/reconcile-sweep-kill-paths-to-lease-w4 branch from 9e8e4be to 64cc6ee Compare September 27, 2026 21:39
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 28, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/reconcile-sweep-kill-paths-to-lease-w4 branch from 6403523 to 7df6ae0 Compare September 28, 2026 05:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/reconcile-sweep-kill-paths-to-lease-w4 branch from 7df6ae0 to 9546959 Compare September 28, 2026 07:32
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/reconcile-sweep-kill-paths-to-lease-w4 branch from 9546959 to db2936f Compare September 28, 2026 08:16
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 28, 2026
…econcile both ways

Six kill paths signal a pid without asking whether a session is still using the
process it names: both cleanup sweeps, the agent-scope reaper, the sub-agent
orphan reconcile, the sub-agent reset ladder and the cron run teardown. Each
carries a shield, and each shield is a union of pid sets gathered before the
candidate scan and an event-loop hop before the signal -- so it can be stale by
the time the signal lands, and the lease table, the one record that knows a
session is mid-turn on that process, was not among the sets consulted. A pid can
be shield-clean and leased in the same moment.

Every one of those paths now asks authorize_runtime_kill about one pid at the
decision point, which is the only place the answer cannot already be stale, and
skips what it refuses. An unanswerable gate is itself a refusal: deferring
housekeeping one tick costs less than killing a live runtime. A refusal never
aborts a pass, so one unanswerable pid does not cost the sweep its other
candidates.

Two of the six are teardowns, and they need the opposite treatment as well. A
reset releases its session's lease inside provider.shutdown(), the last step, so
a reset that hung before it -- the only reason the last-resort kill runs -- still
holds one. A gate asked then lets the session being destroyed refuse its own
kill, and the refusal poisons the pid for the gateway's life. Both release the
lease of the session they are destroying first, reaching it through tearing_down
and matching on the handle's pid AND start identity, so a key with several
teardowns in flight gives up only the lease of the process about to be signalled.
What survives is another tenant's lease, which is the only thing that may
withhold a signal. This is the rule the allocation path's own failure handler
already states: release before the kill, or the cleanup refuses its own teardown.

The scope reaper needed the gate moved, not added. Stopping the unit IS the kill
there -- systemd terminates the whole cgroup -- so ownership is asked of every
current member BEFORE the stop, and one leased member aborts the whole reclaim.
A scope with a live tenant is not abandoned, whatever the scope-level verdict
concluded before that tenant joined, and the next pass reclaims it once the lease
is released. The caller records that refusal distinctly from a stop that was
attempted and left members behind.

runtime_reconcile.py is new and answers the other half: nothing compares the
kernel's own process list against the records, in either direction. Measured on a
live instance, 20 processes at rest and 57 to 97 under load run inside this
install's agent slice with no record claiming them -- nothing will ever end
those, because the sweeps that could are driven by the records they are missing
from. The mirror image is measured too: records naming pids that are gone,
climbing 0 to 23 over fifteen minutes.

It reads both truths and acts on each disagreement. A record whose process is
gone is retracted and its holder told once. A record whose pid now names a
stranger counts in the dead direction, not the healthy one, and the current
holder is never signalled. An unowned process is counted, and killed only when
five things hold together: this install's spawn marker read from the kernel's
exec-time environ, unowned on the previous pass with the SAME process identity,
older than five minutes, the gate's permission, and an identity that still
matches in the instant before the signal. At most five kills a pass. Everything
else is counted and left alone, because absence from a record is evidence
something is unclaimed, not that it is abandoned: the unowned population on a
healthy host is mostly processes with real owners no session record describes,
and killing on first sight would take a user's browser away and call it a leak
fixed. An unreadable registry refuses the whole pass, since that single input
makes every live process look unowned.

The two counts publish as unowned_alive and owned_dead, the names the
process-inventory harness uses for the same reconciliation read from outside the
process, and the recycled pid is classified the same way in both.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Holding approval: the new reaper can kill processes a live session still owns.

The lease gating added to the existing sweeps makes those paths more conservative, and that part looks good. The concern is the new RuntimeReconciler:

  • It runs on every cleanup tick (session_cleanup.py) and has no config switch.
  • It takes every pid in the agent slice and removes only those that are recorded, leased or claimed (runtime_reconcile.py ~L673-677). It does not skip descendants of a live tracked runtime.
  • A leftover pid gets _kill_pid_tree (~L753) when it carries the spawn marker (every descendant inherits it), is older than 300s, was seen unowned on two passes, and its command line contains a harness name as a substring (kiro-cli, claude, goose, ...).

So an agent's own long-running command can be SIGKILLed with its whole tree while the parent session is still working, e.g. a nested kiro-cli chat probe or a pytest test/test_claude_* run. If kiro-cli spawns an untracked kiro-cli-chat worker (not verified), a live runtime could be killed too. The per-pass budget of 5 limits the blast radius but does not make the targets right.

Suggested fix:

  1. Skip any pid whose ancestry reaches a recorded or active runtime pid.
  2. Put the kill arm behind a config switch that is off by default (log-only until it is proven).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants