Repository navigation
fix(session): count the pid shields, and say who killed a runtime - #14362
chenmingwei23 wants to merge 1 commit into
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Counted shields invert the leak direction — an unpaired register now shields a recycled pid forever — and two stray pytest artifacts are committed. Watch
[DESIGN-REVIEWED] 242c706 |
First Principles Review (Fable 5) — 🔴 BLOCKPremise-level review of First-Principles-Verdict: BLOCK Two pytest-run droppings under Not justified as shipped
What this change shipsInventory (7 items) — 6 justifiedIntent: stop a shared process being killed when its first holder finishes, and make every runtime kill log who ordered it — a FIX.
BlockersRider whose fix is complete without it (trigger b). All four parts read off the diff: the title says [FIRST-PRINCIPLES-REVIEWED] 242c706 |
eb64cc7 to
5307f52
Compare
Opus 5 Review — 🔴 changes requested (blocking)Reviewed BLOCKING — the PR's own new ratchet test fails on this tree: [BLOCK-MERGE] 242c706 BLOCKING — test/test_kill_chokepoint_ratchet.py:81
BLOCKING — .pytest-home/crew-log/members/rearm-test-9a2c5fe3/log.jsonl:1 The commit adds a test-run crew-log directory (plus an empty FINDING — src/kiro_crew/acp/runtime.py:3377 — [OPUS-REVIEWED] 242c706 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Two defects, both from a process having more than one interested party. The two pid shields were sets, so the first holder to leave tore the shield off a process the others were still using. session_pid._PROTECTED_PIDS is registered with by every app worker pool and by the knowledge LLM pool; the first unregister exposed the live process to the periodic orphan sweep, which reaped it. SessionRegistryState.starting_pids has the same shape, and the allocator carries a race budget for starting one session twice at once, so the first start to finish cleared the shield the second was still cold-starting behind. Both become PidRefcount, which counts holders and releases on the last one. They read like the sets they replace, and every holder already pairs one register with one unregister, so no holder changes. A runtime kill was unattributed. The death line is written where the process is reaped, so it says a process died without saying who decided it should, and a session whose process was killed under it reports only that the process died. note_runtime_kill records pid, caller and reason at WARNING at the DECISION point, before any signal; session_pid._sync_kill_provider and AcpRuntime.kill -- the two paths every other kill funnels into -- each name themselves. WARNING rather than INFO because the gateway runs at WARNING, so an INFO line would be missing from the log of every deployment that has the problem. What this deliberately does NOT add is a lease table and a gate that refuses a kill while a runtime is still leased. Both are wanted, and neither stands alone: a table nothing acquires is inert and its refusal can never execute, while a gate wired to a provider that holds a lease for its lifetime also refuses the force-kill paths -- _sync_kill_provider is reached through _dispatch_hard_kill and the dashboard reset-all fallback, and neither releases anything -- so the process it declines to signal leaks instead. The table, the gate, the provider's acquire and release, and every kill path releasing before it signals are one unit and land together. test_kill_chokepoint_ratchet.py ratchets the 66 direct kill-primitive calls that are still unattributed, and pins that both kill paths name a caller and a reason. test_runtime_ownership.py covers the counted shields and the attribution line. The protected-pid test releases only its own pid rather than clearing the registry. That registry is process-global, and other tests in the same xdist worker shield their own pids in it, so clearing it would tear their shields off and fail them instead of this test.
5307f52 to
242c706
Compare
|
Closing in favour of #14482, which carries this change plus the half it cannot stand without. This PR shipped the two counted pid shields and the attribution line, and deliberately left out the lease table and the ownership gate. Reviewing the pair established that the split was not viable: the table, the gate, the provider's acquire and every kill path's release are one unit. A table nothing acquires is inert machinery whose refusal can never execute, and a gate wired without complete release sites refuses the force-kill paths and leaks the process instead, trading a kill-too-early defect for a never-killed one. A second reason is verification. A stacked PR based on a feature branch gets none of Every commit here is an ancestor of #14482, so nothing is lost: the shields, the attribution line and the ratchet are all in that diff, and |
Problem / Motivation
Goal: A process that two things are using stops being killed by the first one
to finish, and every runtime kill says who ordered it.
Two defects, both from a process having more than one interested party.
A pid shield is torn off by the first holder to leave.
session_pid._PROTECTED_PIDSis aset, and several independent holders shieldone pid -- an app-managed worker pool, the knowledge LLM pool -- each pairing its
own register with its own unregister. The first
unregister_protected_pidremoves the shield the others still need, and the periodic orphan sweep then
reaps a live process.
SessionRegistryState.starting_pidshas the same shape,and the allocator carries a race budget for starting one session twice at once,
so the first start to finish clears the shield the second is still cold-starting
behind.
A runtime kill says nothing about who ended it. The death line is written
where the process is reaped, so it records that a process died, not who decided
it should. Measured on the base of this branch, with docstrings and comments
stripped so only code counts:
session_pid._sync_kill_providerhas 148 lines ofcode and not one of
lease/pool/protected/tenant/owns/ownerappears in any of them;
AcpRuntime.killhas 16, same result. An AST scan ofsrc/kiro_crewfinds 66 direct calls to theplatform_compatkill primitivesacross 27 files, excluding the module that defines them, and every one of them
signals unattributed.
Why it matters
The shields are a live-process kill. When two holders shield one pid, one of them
finishing hands the other's running process to the orphan sweep, which is the
same class of failure as #11789 and #11899 seen from the other side: not a
process that never dies, but one that dies while someone is using it.
The attribution gap is why that is hard to diagnose. A session whose process was
killed under it reports only that the process died, so a teardown that took the
process and a genuine crash look identical in the log, and nothing correlates the
death with the caller that caused it.
Not a goal
a pid, so one session's teardown cannot end a process another session is running
on. That is the project this belongs to, and it cannot land in pieces: a table
nothing acquires is inert machinery whose refusal can never execute, while a
gate that does refuse, wired to a provider holding a lease for its lifetime,
also refuses the force-kill paths --
_sync_kill_provideris reached through_dispatch_hard_killand through the dashboard reset-all fallback, and neitherreleases anything -- so the process it declines to signal leaks instead. The
table, the gate, the provider's acquire and release, and every kill path
releasing before it signals are one unit, and land together in a follow-up PR.
sites and stops the number growing; routing them is the follow-up's work.
into name themselves here. The ten
_dispatch_hard_killcall sites, the sweep,the scope reaper, the subagent teardown and the cron reaper keep passing through
those two, so they are attributed at the funnel rather than individually; naming
each one is part of the follow-up that also makes them release.
What changed (motivation -> approach -> change)
PidRefcountsrc/kiro_crew/runtime_ownership.pyin, iteration,len, truthiness,set(...), set algebra)._PROTECTED_PIDScountssrc/kiro_crew/session_pid.pystarting_pidscountssrc/kiro_crew/session_allocation.py,src/kiro_crew/session.pynote_runtime_killsrc/kiro_crew/runtime_ownership.pysrc/kiro_crew/session_pid.py,src/kiro_crew/acp/runtime.py_sync_kill_providerandAcpRuntime.killare what every other kill funnels into.test/test_kill_chokepoint_ratchet.pytest/test_runtime_ownership.pyThe shields' root cause is that a set records WHICH pids are shielded and not HOW
MANY holders each has, so
discardcannot tell "the last holder left" from "oneof several left". Counting is the whole fix, and it is safe to drop in because
every existing holder already pairs one register with one unregister -- which is
also why no holder changes.
The attribution's root cause is placement rather than content: a line written
where the process is reaped cannot name the caller, because by then the caller has
returned. So the line moves to the decision point, and the two functions every
kill funnels into name themselves. WARNING rather than INFO because the gateway
runs at WARNING, so an INFO line would be absent from the log of every deployment
that has the problem.
What this deliberately does not add is a lease table and a gate that refuses a
kill while a runtime is still leased. Both are wanted -- a session should hold a
lease rather than a pid, so one session's teardown cannot end a process another
session is running on -- and neither stands alone. A table nothing acquires is
inert machinery whose refusal can never execute. A gate that does refuse, wired to
a provider that holds a lease for its lifetime, also refuses the force-kill paths:
_sync_kill_provideris reached through_dispatch_hard_killand through thedashboard reset-all fallback, and neither releases anything, so the process the
gate declines to signal leaks instead -- trading a kill-too-early defect for a
never-killed one. The table, the gate, the provider's acquire and release, and
every kill path releasing before it signals are therefore one unit, and land
together in a follow-up PR. This change lands the part that stands alone.
Backwards compatibility
Compatible:
PidRefcountis aMutableSet[int]supporting every operation itsholders use, no caller assigns a plain
setto either shield, and the countedbehaviour differs from the set only in the case the set got wrong (a second
holder).
note_runtime_killreturns nothing and gates nothing, so no kill thatworks today is refused or skipped. The
_sync_kill_providerandAcpRuntime.killsignatures are unchanged.Tests
test/test_runtime_ownership.pycovers the counted shields -- a pid staysshielded until its last holder leaves, a zero count is removed rather than left at
zero, the refcount still reads as a set for the sweep's union, non-pids are
rejected -- and the attribution line: it names pid, caller and reason, it is
WARNING and not INFO, it reads a pid off an object, and an unsignalable target is
still attributed with no pid rather than as pid 1.
test/test_kill_chokepoint_ratchet.pyholds the 66-site ceiling, asserts thebaseline is not stale, proves the needles match a call site known to exist, and
pins that both kill paths call the attribution with both a caller and a reason.
Both shield defects are RED on base. Running the two shield assertions against an
unmodified
origin/mainworktree fails both, withsecond holder still needs the shieldandsecond start still needs the shield.Three mutations, each run and reverted:
_sync_kill_providerfails 2 ratchet tests.against 66, and fails the staleness check with it.
PidRefcount.discardto set semantics fails 3 shield tests.Also green on this head:
test_acp_runtime_kill.py,test_dashboard_reset_sessions.py,test_agent_lifecycle_cycle.py(theimport-cycle pin on
session_pid),test_process_identity_structural.py,test_pid_sweep_helpers.py,test_llm_pool.py,test_acp_runtime.py,test_session_cleanup.py,test_session_scope_reap.py-- 775 tests.flake8,isort,mypy src/kiro_crew/, the agent-SDK boundary gate, the baselined blackgate, the blocking-IO-in-async gate and the comment-history gate all exit 0.
Inherited CI failures (not from this change)
None outstanding. This branch is rebased onto a
maintip where they are fixed.Backend Testsshard 3 on 3.12, Windows and macOS, and theCoverage Gatethatwaits on them, were red on
mainitself (#14385): each of those three shard logsreported exactly one failure and the same node id,
test/test_review_repair_routing_skill.py::test_slim_skills_do_not_regrow_duplicate_guidance_or_wire_model_ids[prepare-pr-68000],asserting
68015 <= 68000.mainhas since shrunk that skill file to 67693 bytes,under its own 68000-byte cap, and the test passes here.
One number moved for the same reason. The kill-primitive ratchet's baseline is 67,
not 66:
maingained a bypass site inpdf_extract.py(a Windows-only tree killfor a document-parsing subprocess, #14424) between this branch's original base and
the tip it now sits on. An identical scan of a pristine
maintip worktree alsoreports 67, so this change itself adds none, and a mutation that inserts one more
call still fails the ratchet at 68.
Manual verification
N/A -- unit coverage sufficient. Both defects are reproduced as failing
assertions on unmodified base, and the attribution line is asserted at its own log
level.
Related Issues
Refs #14352
This half ships the counted shields and the attribution line. The ownership gate
that actually closes #14352 is in #14482, which carries the
Fixestrailer, somerging this PR does not close the bug.
Pattern harvest
Rule candidate: review-prompt
Pattern: a shared resource guarded by a set of identifiers rather than a count of
holders, so releasing is indistinguishable from releasing last.
Checklist