Skip to content

fix(session): count the pid shields, and say who killed a runtime - #14362

Closed
chenmingwei23 wants to merge 1 commit into
mainfrom
feat/runtime-ownership-kill-chokepoint-w1
Closed

chenmingwei23 wants to merge 1 commit into
mainfrom
feat/runtime-ownership-kill-chokepoint-w1

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A process that two things are using stops being killed by the first one
to finish, and every runtime kill says who ordered it.

Two defects, both from a process having more than one interested party.

A pid shield is torn off by the first holder to leave.
session_pid._PROTECTED_PIDS is a set, and several independent holders shield
one pid -- an app-managed worker pool, the knowledge LLM pool -- each pairing its
own register with its own unregister. The first unregister_protected_pid
removes the shield the others still need, and the periodic orphan sweep then
reaps a live process. SessionRegistryState.starting_pids has the same shape,
and the allocator carries a race budget for starting one session twice at once,
so the first start to finish clears the shield the second is still cold-starting
behind.

A runtime kill says nothing about who ended it. The death line is written
where the process is reaped, so it records that a process died, not who decided
it should. Measured on the base of this branch, with docstrings and comments
stripped so only code counts: session_pid._sync_kill_provider has 148 lines of
code and not one of lease / pool / protected / tenant / owns / owner
appears in any of them; AcpRuntime.kill has 16, same result. An AST scan of
src/kiro_crew finds 66 direct calls to the platform_compat kill primitives
across 27 files, excluding the module that defines them, and every one of them
signals unattributed.

Why it matters

The shields are a live-process kill. When two holders shield one pid, one of them
finishing hands the other's running process to the orphan sweep, which is the
same class of failure as #11789 and #11899 seen from the other side: not a
process that never dies, but one that dies while someone is using it.

The attribution gap is why that is hard to diagnose. A session whose process was
killed under it reports only that the process died, so a teardown that took the
process and a genuine crash look identical in the log, and nothing correlates the
death with the caller that caused it.

Not a goal

  • The lease table and the kill gate. A session should hold a lease rather than
    a pid, so one session's teardown cannot end a process another session is running
    on. That is the project this belongs to, and it cannot land in pieces: a table
    nothing acquires is inert machinery whose refusal can never execute, while a
    gate that does refuse, wired to a provider holding a lease for its lifetime,
    also refuses the force-kill paths -- _sync_kill_provider is reached through
    _dispatch_hard_kill and through the dashboard reset-all fallback, and neither
    releases anything -- so the process it declines to signal leaks instead. The
    table, the gate, the provider's acquire and release, and every kill path
    releasing before it signals are one unit, and land together in a follow-up PR.
  • Lowering the 66-site ceiling. This change measures the unattributed kill
    sites and stops the number growing; routing them is the follow-up's work.
  • The other kill paths' attribution. Only the two functions every kill funnels
    into name themselves here. The ten _dispatch_hard_kill call sites, the sweep,
    the scope reaper, the subagent teardown and the cron reaper keep passing through
    those two, so they are attributed at the funnel rather than individually; naming
    each one is part of the follow-up that also makes them release.

What changed (motivation -> approach -> change)

Piece Where What it does
PidRefcount src/kiro_crew/runtime_ownership.py A pid shield that counts holders and releases on the last one, reading like the set it replaces (in, iteration, len, truthiness, set(...), set algebra).
_PROTECTED_PIDS counts src/kiro_crew/session_pid.py The sweep shield survives one of several holders leaving.
starting_pids counts src/kiro_crew/session_allocation.py, src/kiro_crew/session.py The cold-start shield survives one of two concurrent starts finishing; both files carry the type on their pass-through properties.
note_runtime_kill src/kiro_crew/runtime_ownership.py Records pid, caller and reason at WARNING at the decision point, before any signal.
Both kill paths name themselves src/kiro_crew/session_pid.py, src/kiro_crew/acp/runtime.py _sync_kill_provider and AcpRuntime.kill are what every other kill funnels into.
The ratchet test/test_kill_chokepoint_ratchet.py A ceiling of 66 on unattributed primitive calls, plus pins that both paths pass a caller and a reason.
Behaviour tests test/test_runtime_ownership.py The counted shields and the attribution line.

The shields' root cause is that a set records WHICH pids are shielded and not HOW
MANY holders each has, so discard cannot tell "the last holder left" from "one
of several left". Counting is the whole fix, and it is safe to drop in because
every existing holder already pairs one register with one unregister -- which is
also why no holder changes.

The attribution's root cause is placement rather than content: a line written
where the process is reaped cannot name the caller, because by then the caller has
returned. So the line moves to the decision point, and the two functions every
kill funnels into name themselves. WARNING rather than INFO because the gateway
runs at WARNING, so an INFO line would be absent from the log of every deployment
that has the problem.

What this deliberately does not add is a lease table and a gate that refuses a
kill while a runtime is still leased. Both are wanted -- a session should hold a
lease rather than a pid, so one session's teardown cannot end a process another
session is running on -- and neither stands alone. A table nothing acquires is
inert machinery whose refusal can never execute. A gate that does refuse, wired to
a provider that holds a lease for its lifetime, also refuses the force-kill paths:
_sync_kill_provider is reached through _dispatch_hard_kill and through the
dashboard reset-all fallback, and neither releases anything, so the process the
gate declines to signal leaks instead -- trading a kill-too-early defect for a
never-killed one. The table, the gate, the provider's acquire and release, and
every kill path releasing before it signals are therefore one unit, and land
together in a follow-up PR. This change lands the part that stands alone.

Backwards compatibility

Compatible: PidRefcount is a MutableSet[int] supporting every operation its
holders use, no caller assigns a plain set to either shield, and the counted
behaviour differs from the set only in the case the set got wrong (a second
holder). note_runtime_kill returns nothing and gates nothing, so no kill that
works today is refused or skipped. The _sync_kill_provider and
AcpRuntime.kill signatures are unchanged.

Tests

test/test_runtime_ownership.py covers the counted shields -- a pid stays
shielded until its last holder leaves, a zero count is removed rather than left at
zero, the refcount still reads as a set for the sweep's union, non-pids are
rejected -- and the attribution line: it names pid, caller and reason, it is
WARNING and not INFO, it reads a pid off an object, and an unsignalable target is
still attributed with no pid rather than as pid 1.
test/test_kill_chokepoint_ratchet.py holds the 66-site ceiling, asserts the
baseline is not stale, proves the needles match a call site known to exist, and
pins that both kill paths call the attribution with both a caller and a reason.

Both shield defects are RED on base. Running the two shield assertions against an
unmodified origin/main worktree fails both, with second holder still needs the shield and second start still needs the shield.

Three mutations, each run and reverted:

  • Dropping the attribution call from _sync_kill_provider fails 2 ratchet tests.
  • Adding one direct primitive call elsewhere in the tree fails the ceiling at 67
    against 66, and fails the staleness check with it.
  • Reverting PidRefcount.discard to set semantics fails 3 shield tests.

Also green on this head: test_acp_runtime_kill.py,
test_dashboard_reset_sessions.py, test_agent_lifecycle_cycle.py (the
import-cycle pin on session_pid), test_process_identity_structural.py,
test_pid_sweep_helpers.py, test_llm_pool.py, test_acp_runtime.py,
test_session_cleanup.py, test_session_scope_reap.py -- 775 tests. flake8,
isort, mypy src/kiro_crew/, the agent-SDK boundary gate, the baselined black
gate, the blocking-IO-in-async gate and the comment-history gate all exit 0.

Inherited CI failures (not from this change)

None outstanding. This branch is rebased onto a main tip where they are fixed.
Backend Tests shard 3 on 3.12, Windows and macOS, and the Coverage Gate that
waits on them, were red on main itself (#14385): each of those three shard logs
reported exactly one failure and the same node id,
test/test_review_repair_routing_skill.py::test_slim_skills_do_not_regrow_duplicate_guidance_or_wire_model_ids[prepare-pr-68000],
asserting 68015 <= 68000. main has since shrunk that skill file to 67693 bytes,
under its own 68000-byte cap, and the test passes here.

One number moved for the same reason. The kill-primitive ratchet's baseline is 67,
not 66: main gained a bypass site in pdf_extract.py (a Windows-only tree kill
for a document-parsing subprocess, #14424) between this branch's original base and
the tip it now sits on. An identical scan of a pristine main tip worktree also
reports 67, so this change itself adds none, and a mutation that inserts one more
call still fails the ratchet at 68.

Manual verification

N/A -- unit coverage sufficient. Both defects are reproduced as failing
assertions on unmodified base, and the attribution line is asserted at its own log
level.

Related Issues

Refs #14352

This half ships the counted shields and the attribution line. The ownership gate
that actually closes #14352 is in #14482, which carries the Fixes trailer, so
merging this PR does not close the bug.

Pattern harvest

Rule candidate: review-prompt
Pattern: a shared resource guarded by a set of identifiers rather than a count of
holders, so releasing is indistinguishable from releasing last.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) -- module and function docstrings carry the reasoning; no user-facing doc changes
  • No secrets, credentials, or internal references in the diff

@chenmingwei23
chenmingwei23 requested review from a team and pepmach September 27, 2026 04:25
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 242c706eeb077cf32254d9efde2a94743146007c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 242c706

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 242c706eeb077cf32254d9efde2a94743146007c: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 242c706eeb077cf32254d9efde2a94743146007c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Counted shields invert the leak direction — an unpaired register now shields a recycled pid forever — and two stray pytest artifacts are committed.

Watch

  • With the old set, a stale _PROTECTED_PIDS entry was cleared by any single unregister; PidRefcount makes a leaked reference (a crash between register and unregister, or a re-register that was previously idempotent) permanent for the gateway's lifetime, and nothing purges it — the exact hazard acp/runtime.py's own comment names: "the dead PID lingers in _PROTECTED_PIDS forever and, after PID reuse, could shield a genuinely-orphaned process from the orphan sweep." Consequence: a genuine orphan wearing a reused pid is never reaped until restart.
    Clears when: the periodic sweep drops references for pids that fail its liveness/identity check, or the PR states why no register path can fire twice unpaired.
  • .pytest-home/crew-log/members/rearm-test-9a2c5fe3/.lock and log.jsonl are committed test-run artifacts no description sentence accounts for.
    Clears when: both files are removed from the branch (and the path ignored).

[DESIGN-REVIEWED] 242c706

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🔴 BLOCK

Premise-level review of 242c706eeb077cf32254d9efde2a94743146007c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: BLOCK

Two pytest-run droppings under .pytest-home/ ride along in a fix whose declared work is already complete without them.

Not justified as shipped

  • Item 7 — rides along, undeclared: .pytest-home/crew-log/members/rearm-test-9a2c5fe3/{.lock,log.jsonl} appear nowhere in the description; the log.jsonl names rearm-test, the agent id a pre-existing test (test/test_dashboard_chat.py:9643) creates, so these are a test run's leaked crew-log state, not a fixture — nothing in the repo reads them.
    Subtraction: delete both files (git rm -r .pytest-home).
    Clears when: the two files are removed from the branch.

What this change ships

Inventory (7 items) — 6 justified

Intent: stop a shared process being killed when its first holder finishes, and make every runtime kill log who ordered it — a FIX.

  1. A pid shielded by several holders survives until the last one unregisters — justified
  2. A pid two concurrent session starts shield survives the first start finishing — justified
  3. Kills through _sync_kill_provider and AcpRuntime.kill now log pid, caller, reason at WARNING before signalling — justified
  4. New module runtime_ownership.py (PidRefcount: 3 consumers; note_runtime_kill: 2 consumers, grepped src) — justified
  5. Repo-wide ratchet pinning unattributed kill-primitive calls at exactly 67, with positive control — justified
  6. Behaviour tests for the counted shields and the attribution line — justified
  7. Two committed pytest artifacts under .pytest-home/ — rides along, undeclared: leaked test-run state, zero benefit

Blockers

Rider whose fix is complete without it (trigger b). All four parts read off the diff: the title says fix(session); .pytest-home/crew-log/members/rearm-test-9a2c5fe3/.lock and log.jsonl ride along unmentioned by the description; their zero option costs nobody anything (generated crew-log state from a test run — grep rearm-test finds only the pre-existing test/test_dashboard_chat.py and this file, no fixture consumer); and items 1–6 already remove both reported defects on their own. Subtraction: delete the two files.
Clears when: .pytest-home/ is removed from the branch and the diff no longer adds it.

[FIRST-PRINCIPLES-REVIEWED] 242c706

@chenmingwei23 chenmingwei23 changed the title acp: single runtime kill chokepoint with lease ownership (cap=1) fix(acp): single runtime kill chokepoint with lease ownership (cap=1) Sep 27, 2026
@chenmingwei23 chenmingwei23 changed the title fix(acp): single runtime kill chokepoint with lease ownership (cap=1) fix(session): count the pid shields, and say who killed a runtime Sep 27, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/runtime-ownership-kill-chokepoint-w1 branch 2 times, most recently from eb64cc7 to 5307f52 Compare September 27, 2026 06:22
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — 🔴 changes requested (blocking)

Reviewed 242c706eeb077cf32254d9efde2a94743146007c — this comment is updated in place on each push.

BLOCKING — the PR's own new ratchet test fails on this tree: BYPASS_BASELINE = 67 while the scan it pins counts 68.

[BLOCK-MERGE] 242c706

BLOCKING — test/test_kill_chokepoint_ratchet.py:81

BYPASS_BASELINE = 67

_bypass_sites() over src/kiro_crew/**/*.py minus platform_compat.py yields 88 kill_pid/kill_process_tree/kill_pid_pinned/kill_process_tree_pinned/killpg textual hits, of which 20 are docstring/comment/log-string prose → 68 real ast.Call sites (including the base-branch pdf_extract.py:251 the comment says it already paid for). So assert total == BYPASS_BASELINE fails with "the real count is 68 but BYPASS_BASELINE says 67", and assert total <= BYPASS_BASELINE fails too — every run of this module, red.
Fix: set BYPASS_BASELINE = 68.

BLOCKING — .pytest-home/crew-log/members/rearm-test-9a2c5fe3/log.jsonl:1

+{"type":"member","version":1,"id":"rearm-test","createdAt":1790482290246,"name":"rearm-test"}

The commit adds a test-run crew-log directory (plus an empty .lock) at the repo root; .pytest-home is in no .gitignore, and conftest.py:pytest_sessionstart snapshots the root's immediate children before collection — so once .pytest-home is tracked it is in every future baseline and the repo-root residue guard (_not_ignored(added), conftest.py:4273) can never report anything a test writes under it again, while each run re-appends to the tracked log.jsonl and dirties every developer's tree. This is verbatim the incident no-test-side-effects (blocking) names, and test/**/*.py matches two changed files.
Fix: git rm both .pytest-home/... files from the commit (pure additions; the PR's purpose is unaffected).

FINDING — src/kiro_crew/acp/runtime.py:3377 — note_runtime_kill(self, reason=reason or "runtime kill", caller="AcpRuntime.kill") logs at WARNING unconditionally and never reads expected, so every deliberate teardown (kill(expected=True, ...) at session_lifecycle.py:1857/2290/2308, session_allocation.py:876, acp/session_provider.py:272, connections/warm.py:2014, review_pool.py:511, …) now writes a WARNING — contradicting _kill_inner's own contract ten lines below ("expected=True to log the death at INFO") and burying the one unexpected kill the line exists to surface; repeated kill() on an already-dead runtime writes another, with pid=None before spawn() → Fix: give note_runtime_kill a severity/expected input and pass expected from kill.

[OPUS-REVIEWED] 242c706

Verdict parsed from the review's SHA-scoped output markers for commit 242c706eeb077cf32254d9efde2a94743146007c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 242c706eeb077cf32254d9efde2a94743146007c: <one-sentence reason>

Two defects, both from a process having more than one interested party.

The two pid shields were sets, so the first holder to leave tore the shield off a
process the others were still using. session_pid._PROTECTED_PIDS is registered
with by every app worker pool and by the knowledge LLM pool; the first
unregister exposed the live process to the periodic orphan sweep, which reaped
it. SessionRegistryState.starting_pids has the same shape, and the allocator
carries a race budget for starting one session twice at once, so the first start
to finish cleared the shield the second was still cold-starting behind. Both
become PidRefcount, which counts holders and releases on the last one. They read
like the sets they replace, and every holder already pairs one register with one
unregister, so no holder changes.

A runtime kill was unattributed. The death line is written where the process is
reaped, so it says a process died without saying who decided it should, and a
session whose process was killed under it reports only that the process died.
note_runtime_kill records pid, caller and reason at WARNING at the DECISION
point, before any signal; session_pid._sync_kill_provider and AcpRuntime.kill --
the two paths every other kill funnels into -- each name themselves.

WARNING rather than INFO because the gateway runs at WARNING, so an INFO line
would be missing from the log of every deployment that has the problem.

What this deliberately does NOT add is a lease table and a gate that refuses a
kill while a runtime is still leased. Both are wanted, and neither stands alone:
a table nothing acquires is inert and its refusal can never execute, while a gate
wired to a provider that holds a lease for its lifetime also refuses the
force-kill paths -- _sync_kill_provider is reached through _dispatch_hard_kill and
the dashboard reset-all fallback, and neither releases anything -- so the process
it declines to signal leaks instead. The table, the gate, the provider's acquire
and release, and every kill path releasing before it signals are one unit and
land together.

test_kill_chokepoint_ratchet.py ratchets the 66 direct kill-primitive calls that
are still unattributed, and pins that both kill paths name a caller and a reason.
test_runtime_ownership.py covers the counted shields and the attribution line.

The protected-pid test releases only its own pid rather than clearing the
registry. That registry is process-global, and other tests in the same xdist
worker shield their own pids in it, so clearing it would tear their shields off
and fail them instead of this test.
@chenmingwei23
chenmingwei23 force-pushed the feat/runtime-ownership-kill-chokepoint-w1 branch from 5307f52 to 242c706 Compare September 27, 2026 16:50
@chenmingwei23
chenmingwei23 requested a review from a team as a code owner September 27, 2026 16:50
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Closing in favour of #14482, which carries this change plus the half it cannot stand without.

This PR shipped the two counted pid shields and the attribution line, and deliberately left out the lease table and the ownership gate. Reviewing the pair established that the split was not viable: the table, the gate, the provider's acquire and every kill path's release are one unit. A table nothing acquires is inert machinery whose refusal can never execute, and a gate wired without complete release sites refuses the force-kill paths and leaks the process instead, trading a kill-too-early defect for a never-killed one.

A second reason is verification. A stacked PR based on a feature branch gets none of ci.yml, which filters on base == main, so this work only ever ran the review lanes and none of the test, lint, build or coverage lanes. #14482 is based on main and runs the full set.

Every commit here is an ancestor of #14482, so nothing is lost: the shields, the attribution line and the ratchet are all in that diff, and Fixes #14352 now sits there. The branch is left in place because it is an ancestor of the branch #14482 is opened from.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 27, 2026
@bolichen97
bolichen97 deleted the feat/runtime-ownership-kill-chokepoint-w1 branch November 6, 2026 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nothing asks who owns a runtime before killing it: one session's teardown can kill a process another session is using

1 participant