Repository navigation
fix(acp): recycle idle shared runtimes; unstarve session-start gate - #12944
bolichen97 merged 1 commit into
Conversation
|
Intent: Stop a long-running gateway from leaking agent runtimes and then losing the ability to start sessions at all — four measured leaks that compound: runtimes never recycled off the reuse path, a session-start gate starved by permits parked on callers that gave up, a monitoring loop spending turns on doomed starts, and per-pid binding records nothing collects. |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS All four fixes rest on one operator host's measurements; verify the retained single collector hold — at the default limit of 2 it still parks half the gate for 300 s. What this change shipsInventory (9 items) — 9 justifiedIntent: stop a long-running gateway from leaking agent runtimes until it can no longer start sessions — a FIX (four measured defects).
[FIRST-PRINCIPLES-REVIEWED] 6f0e8d1 |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the backend tolerates in-flight starts briefly exceeding Suggestions
[DESIGN-REVIEWED] 6f0e8d1 |
GPT 5.6 Review (fork) — ✅ no blocking findings (all downgraded on adjudication)Reviewed Opus 4.8 adjudication downgraded every blocking finding to advisory. Adjudication downgraded all 1 blocking finding(s) to advisory: the remedy each one requires is disproportionate to the harm it prevents. Read them as advice, not as merge conditions. Review detailsBLOCKING -- src/kiro_crew/autonudge.py:4512 -- start-failure state is published before persistence Adjudication (Opus 4.8) — is blocking on each finding proportionate?I have enough to rule. The key facts, all confirmed from the base tree in this run:
Harm in both directions is bounded, visible, and self-correcting: a lost increment just repeats a few doomed cycles (the pre-PR behavior the whole PR reduces); a lost clear stands the loop down at most one cycle early, which is visible ( LOW/MEDIUM recoverable+visible harm vs. a cross-module async-conversion remedy that also breaks pattern parity with the sibling that already ships this exact degradation. [ADJUDICATION] 6f0e8d1 total=1 uphold=0 downgrade=1 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
6707f50 to
9d3382b
Compare
|
self-added: yes
|
|
self-added: yes
|
|
|
|
self-added: yes
|
9d3382b to
d14a099
Compare
|
self-added: yes
|
|
|
|
|
|
Four leaks that compound into one failure: agent runtimes that never get recycled, a session-start gate that starves behind requests whose callers gave up, a monitoring loop that keeps spending cycles on doomed starts, and per-pid binding records nothing collects. - The shared background runtime's staleness ceilings were only evaluated on the reuse path, and its pid is shielded from the orphan sweep, so a runtime nobody reuses is never asked and never reaped. The periodic watchdog hook now asks off the reuse path and parks an idle-and-stale runtime. - A timed-out session/new handed its gate permit to a StartCollector for up to 300s; at the default limit of 2 two of those held the whole gate. One permit is now reserved for a start that has not gone out. - An auto-nudge loop whose cycles never got a session re-armed on its plain interval. It now backs off after three and stands down after five, on recorded evidence, cleared by any landed turn. - Aged member-memory pid binding records whose pid names no live process are collected by the same periodic tick, budgeted per pass.
d14a099 to
6f0e8d1
Compare
…irodotdev#12944) Four leaks that compound into one failure: agent runtimes that never get recycled, a session-start gate that starves behind requests whose callers gave up, a monitoring loop that keeps spending cycles on doomed starts, and per-pid binding records nothing collects. - The shared background runtime's staleness ceilings were only evaluated on the reuse path, and its pid is shielded from the orphan sweep, so a runtime nobody reuses is never asked and never reaped. The periodic watchdog hook now asks off the reuse path and parks an idle-and-stale runtime. - A timed-out session/new handed its gate permit to a StartCollector for up to 300s; at the default limit of 2 two of those held the whole gate. One permit is now reserved for a start that has not gone out. - An auto-nudge loop whose cycles never got a session re-armed on its plain interval. It now backs off after three and stands down after five, on recorded evidence, cleared by any landed turn. - Aged member-memory pid binding records whose pid names no live process are collected by the same periodic tick, budgeted per pass. Co-authored-by: w <w@k.local>
Problem / Motivation
A long-running gateway leaks agent processes, then stops being able to start
sessions at all. Four separate defects, measured together on one operator host:
over the 500 MiB recycle ceiling — against 4 live chat slots.
session/new timed out after 90s (0/10 MCP server(s) reported). Samplingshowed no agent process was ever spawned in that window: the starts were
queued.
member-memory-bindings/pids/, spanning nine days, 24 MBof directory inode.
Why it matters
The four compound. Runtimes that are never recycled eat the memory, which makes
session starts slow. A slow start times out and parks a gate permit for five
minutes, and with two permits total the whole gateway's session starts queue
behind callers that already gave up — so the retries time out too and park more.
A monitoring loop on top of that spends a model turn per cycle to fail the same
way, fifteen times, until its cycle cap happens to run out. Nothing recovers
without a gateway restart, and the binding records survive even that.
What changed (motivation → approach → change)
_is_stale()is only asked on the reuse path, and the shared runtime's pid is shielded from the orphan sweepsrc/kiro_crew/session_background.py,src/kiro_crew/session_cleanup.py,src/kiro_crew/session.pysession/newhands its gate permit to a collector for up to 300 s; two of those hold the whole gatesrc/kiro_crew/acp/runtime.pysrc/kiro_crew/autonudge.py,src/kiro_crew/dashboard/chat_runner.py,src/kiro_crew/acp/client.py,src/kiro_crew/acp/session_handle.pysrc/kiro_crew/member_memory_auth.pyOwning specs move in the same commit:
docs/system-specs/modules/session.md(runtime lifecycle and the periodic sweeps),
docs/system-specs/modules/acp-client.md(the gate's collector reservation),
docs/system-specs/modules/learn-cron-dashboard.md(the new stop reason).
Runtime recycling. Both existing displacement triggers live inside
get_bg_session(), so a runtime that stops being reused is never asked again.reap_idle_stale_bg_runtime()asks under the runtime lock and retires only aruntime with no active or initializing session. It PARKS rather than kills:
get_bg_session()pins the runtime under the lock, releases it, and only thenraises
_session_inits_in_flight, so a kill in that window would surface asAcpRuntimeDeadon an innocent caller. The drain reaper kills it a tick later,once that pinned start has registered or failed. Both probes fail toward keeping
the runtime.
Session-start gate.
StartPermit.hold_for_collector()claims one of at mostlimit - 1collector holds. Denied, the permit is released and the collectorgets none — it still owns its request and still adopts or tears down the late
session, it just stops holding back-pressure it cannot release. At
limit == 1no collector holds a permit.
Loop stand-down. The ACP session-start timeout raise sites now tag their
exception
session_start_failedon both exception families, so the chat runnercan count starved cycles without reading message text. It reports one only for
the loop's own
_directive_self_waketurn; any landed turn clears the streak._timerdefers one wake per failure past three — one per failure, because thestreak only grows on a delivered cycle and deferring every wake would freeze it —
and stands the loop down at five with a user-visible
expiredevent. The cyclecap and wall-clock budget still outrank it.
The report is made from BOTH terminal branches through one
_note_cycle_start_failurehelper. The two ACP exception families share no base:a dedicated-client start raises an
AcpError, while a start on the shared runtimeraises an
AcpRuntimeErrorsubclass and lands in a different handler — and amember slot, the population this was measured on, runs there. Reporting from only
one branch would leave exactly those loops never backing off. The persisted streak
is normalised at the
_load()boundary through the same_repair_numberhelpergate,next_due_tsandidle_secsuse, because that store is agent-writable anda non-int reaching the
>=comparison would stop the automation for good.Binding records. A record is deleted only when it is BOTH older than 24 h AND
its pid names no live process; requiring both is what makes it safe against pid
reuse in either direction. Symlinks are neither followed nor removed, unknown
names are left alone, and each pass is capped at 2000 deletions so a six-figure
backlog drains over passes.
Backwards compatibility
Compatible: no field, route, config key or exception type is removed or
tightened.
NudgeLoop.consecutive_start_failuresis an added field with adefault, so an older store loads unchanged;
session_start_failuresjoins there-armable system-imposed stop reasons, and an unknown reason still fails closed
to preserved.
Tests
test/test_runtime_start_gate_headroom.py— collectors cannot take the lastpermit; a denied permit frees its slot for a new start;
limit == 1parksnothing; a release returns the hold so the ceiling recovers; release stays
idempotent and cannot double-book.
test/test_bg_runtime_idle_recycle.py— idle+stale is parked and the slotfreed; busy is left alone; fresh-and-idle is left alone; either probe raising
preserves the runtime; a slot replaced while the probe awaited is not retired;
a closing manager parks nothing; the hook runs the sweep with an empty drain
list and still reaps drained runtimes.
test/test_autonudge_start_failures.py— the streak records without stopping;a landed turn clears it; a short streak still fires; the deferral is one-shot
so the streak can advance; a recovered-then-re-failed streak pays again; the
delay escalates and is capped by the loop's own interval; five failures stand
the loop down with
expired; the cycle cap still wins; the stand-down isre-armable and revival clears the streak; the streak survives a restart.
test/test_member_pid_binding_prune.py— aged + dead pid is removed; young iskept; aged + live pid is kept; unknown names and symlinks are untouched; the
budget bounds one pass; a missing directory is not an error.
test/test_autonudge_handlers_cov80.py— the new loop field is answered asentitled on the unowned legacy read, in the test written to force that
question.
One red on this branch is inherited from the base, not introduced here:
test/test_crew_log_exc_info_sites.py::test_no_exc_info_site_anywhere_in_kiro_crew_holds_a_crew_log_handlefails on Backend Tests shard 7 (Linux and Windows alike), and reproduces locally
on a clean
origin/mainatf47c56dd9e— this PR's base. Its offenders aresrc/kiro_crew/apps/builtins/issue_radar/backend/crew_store.py_landed_since(line 858) and
_all_landed(line 2468); nothing in this diff touchesissue_radar, so it is deliberately left alone rather than fixed here. It isfixed upstream by #12943, and this branch is rebased onto main@80532c1a1e
which includes it.
Manual verification
N/A — unit coverage sufficient. Each fix is a pure decision function over
injectable state (staleness and session probes, gate counters, a persisted
streak, filesystem metadata), and the field evidence that motivated it came from
a running gateway rather than from a path a manual run could reproduce on demand.
Related Issues
no linked issue: found by inspection of a live gateway's process table and data
home, not filed first.
Pattern harvest
Rule candidate: review-prompt
Pattern: a resource ceiling evaluated only on a cache-hit path, so the object
that stops being reused is the one that never gets bounded.
Checklist