Skip to content

fix(acp): recycle idle shared runtimes; unstarve session-start gate - #12944

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
bolichen97:fix/runtime-recycle-start-gate-starvation
Sep 23, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
bolichen97:fix/runtime-recycle-start-gate-starvation

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

A long-running gateway leaks agent processes, then stops being able to start
sessions at all. Four separate defects, measured together on one operator host:

  • 11 agent runtimes holding 6.0 GB, six of them 14.5 h old and five of those
    over the 500 MiB recycle ceiling — against 4 live chat slots.
  • 15 consecutive auto-nudge cycles on one member slot, every one ending in
    session/new timed out after 90s (0/10 MCP server(s) reported). Sampling
    showed no agent process was ever spawned in that window: the starts were
    queued.
  • 165,975 files in member-memory-bindings/pids/, spanning nine days, 24 MB
    of directory inode.

Why it matters

The four compound. Runtimes that are never recycled eat the memory, which makes
session starts slow. A slow start times out and parks a gate permit for five
minutes, and with two permits total the whole gateway's session starts queue
behind callers that already gave up — so the retries time out too and park more.
A monitoring loop on top of that spends a model turn per cycle to fail the same
way, fifteen times, until its cycle cap happens to run out. Nothing recovers
without a gateway restart, and the binding records survive even that.

What changed (motivation → approach → change)

Symptom Root cause Change Files
Runtimes live forever _is_stale() is only asked on the reuse path, and the shared runtime's pid is shielded from the orphan sweep the periodic watchdog hook asks off the reuse path and parks an idle-and-stale runtime src/kiro_crew/session_background.py, src/kiro_crew/session_cleanup.py, src/kiro_crew/session.py
Session starts queue behind dead callers a timed-out session/new hands its gate permit to a collector for up to 300 s; two of those hold the whole gate one permit is reserved for a start that has not gone out src/kiro_crew/acp/runtime.py
A loop burns cycles on doomed starts nothing counted "my cycle never got a session" back off after three, stand down after five, on recorded evidence, reported from BOTH terminal branches src/kiro_crew/autonudge.py, src/kiro_crew/dashboard/chat_runner.py, src/kiro_crew/acp/client.py, src/kiro_crew/acp/session_handle.py
Binding records pile up nothing writes them and nothing collected them the same periodic tick collects aged records whose pid is dead src/kiro_crew/member_memory_auth.py

Owning specs move in the same commit: docs/system-specs/modules/session.md
(runtime lifecycle and the periodic sweeps), docs/system-specs/modules/acp-client.md
(the gate's collector reservation), docs/system-specs/modules/learn-cron-dashboard.md
(the new stop reason).

Runtime recycling. Both existing displacement triggers live inside
get_bg_session(), so a runtime that stops being reused is never asked again.
reap_idle_stale_bg_runtime() asks under the runtime lock and retires only a
runtime with no active or initializing session. It PARKS rather than kills:
get_bg_session() pins the runtime under the lock, releases it, and only then
raises _session_inits_in_flight, so a kill in that window would surface as
AcpRuntimeDead on an innocent caller. The drain reaper kills it a tick later,
once that pinned start has registered or failed. Both probes fail toward keeping
the runtime.

Session-start gate. StartPermit.hold_for_collector() claims one of at most
limit - 1 collector holds. Denied, the permit is released and the collector
gets none — it still owns its request and still adopts or tears down the late
session, it just stops holding back-pressure it cannot release. At limit == 1
no collector holds a permit.

Loop stand-down. The ACP session-start timeout raise sites now tag their
exception session_start_failed on both exception families, so the chat runner
can count starved cycles without reading message text. It reports one only for
the loop's own _directive_self_wake turn; any landed turn clears the streak.
_timer defers one wake per failure past three — one per failure, because the
streak only grows on a delivered cycle and deferring every wake would freeze it —
and stands the loop down at five with a user-visible expired event. The cycle
cap and wall-clock budget still outrank it.

The report is made from BOTH terminal branches through one
_note_cycle_start_failure helper. The two ACP exception families share no base:
a dedicated-client start raises an AcpError, while a start on the shared runtime
raises an AcpRuntimeError subclass and lands in a different handler — and a
member slot, the population this was measured on, runs there. Reporting from only
one branch would leave exactly those loops never backing off. The persisted streak
is normalised at the _load() boundary through the same _repair_number helper
gate, next_due_ts and idle_secs use, because that store is agent-writable and
a non-int reaching the >= comparison would stop the automation for good.

Binding records. A record is deleted only when it is BOTH older than 24 h AND
its pid names no live process; requiring both is what makes it safe against pid
reuse in either direction. Symlinks are neither followed nor removed, unknown
names are left alone, and each pass is capped at 2000 deletions so a six-figure
backlog drains over passes.

Backwards compatibility

Compatible: no field, route, config key or exception type is removed or
tightened. NudgeLoop.consecutive_start_failures is an added field with a
default, so an older store loads unchanged; session_start_failures joins the
re-armable system-imposed stop reasons, and an unknown reason still fails closed
to preserved.

Tests

  • test/test_runtime_start_gate_headroom.py — collectors cannot take the last
    permit; a denied permit frees its slot for a new start; limit == 1 parks
    nothing; a release returns the hold so the ceiling recovers; release stays
    idempotent and cannot double-book.
  • test/test_bg_runtime_idle_recycle.py — idle+stale is parked and the slot
    freed; busy is left alone; fresh-and-idle is left alone; either probe raising
    preserves the runtime; a slot replaced while the probe awaited is not retired;
    a closing manager parks nothing; the hook runs the sweep with an empty drain
    list and still reaps drained runtimes.
  • test/test_autonudge_start_failures.py — the streak records without stopping;
    a landed turn clears it; a short streak still fires; the deferral is one-shot
    so the streak can advance; a recovered-then-re-failed streak pays again; the
    delay escalates and is capped by the loop's own interval; five failures stand
    the loop down with expired; the cycle cap still wins; the stand-down is
    re-armable and revival clears the streak; the streak survives a restart.
  • test/test_member_pid_binding_prune.py — aged + dead pid is removed; young is
    kept; aged + live pid is kept; unknown names and symlinks are untouched; the
    budget bounds one pass; a missing directory is not an error.
  • test/test_autonudge_handlers_cov80.py — the new loop field is answered as
    entitled on the unowned legacy read, in the test written to force that
    question.

One red on this branch is inherited from the base, not introduced here:
test/test_crew_log_exc_info_sites.py::test_no_exc_info_site_anywhere_in_kiro_crew_holds_a_crew_log_handle
fails on Backend Tests shard 7 (Linux and Windows alike), and reproduces locally
on a clean origin/main at f47c56dd9e — this PR's base. Its offenders are
src/kiro_crew/apps/builtins/issue_radar/backend/crew_store.py _landed_since
(line 858) and _all_landed (line 2468); nothing in this diff touches
issue_radar, so it is deliberately left alone rather than fixed here. It is
fixed upstream by #12943, and this branch is rebased onto main@80532c1a1e
which includes it.

Manual verification

N/A — unit coverage sufficient. Each fix is a pure decision function over
injectable state (staleness and session probes, gate counters, a persisted
streak, filesystem metadata), and the field evidence that motivated it came from
a running gateway rather than from a path a manual run could reproduce on demand.

Related Issues

no linked issue: found by inspection of a live gateway's process table and data
home, not filed first.

Pattern harvest

Rule candidate: review-prompt
Pattern: a resource ceiling evaluated only on a cache-hit path, so the object
that stops being reused is the one that never gets bounded.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and smeyffret September 22, 2026 23:31
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Stop a long-running gateway from leaking agent runtimes and then losing the ability to start sessions at all — four measured leaks that compound: runtimes never recycled off the reuse path, a session-start gate starved by permits parked on callers that gave up, a monitoring loop spending turns on doomed starts, and per-pid binding records nothing collects.
Not a goal: redesigning the recycle ceilings, the gate’s sizing, the adaptive concurrency controller, or the member-memory routing that once wrote those records. No new config knob, no user-visible surface.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 6f0e8d18491b25fb705828d359ce602de35959a8 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

All four fixes rest on one operator host's measurements; verify the retained single collector hold — at the default limit of 2 it still parks half the gate for 300 s.

What this change ships

Inventory (9 items) — 9 justified

Intent: stop a long-running gateway from leaking agent runtimes until it can no longer start sessions — a FIX (four measured defects).

  1. Shared background runtime is now recycled by the periodic watchdog when idle and stale, parked then drain-reaped — justified
  2. A timed-out start's collector can no longer take the gate's last permit; denied, the permit is released with a warning log — justified
  3. Session-start timeouts on both ACP exception families now carry a session_start_failed tag — justified
  4. An auto-nudge loop defers wakes after 3 consecutive start failures and stands down at 5 with a re-armable session_start_failures stop and user-visible expired event — justified
  5. New persisted loop field consecutive_start_failures, normalized at load, cleared by any landed turn and on revival — justified
  6. Legacy member-memory-bindings/pids/ records are periodically deleted when 24 h old AND pid-dead, 2000 per pass (0 writers on base; grep member-memory-bindings) — justified
  7. The watchdog hook now takes the runtime lock every tick, even with nothing parked (empty-list early return narrowed) — justified
  8. Owning specs (session.md, acp-client.md, learn-cron-dashboard.md) updated in the same commit — justified
  9. The legacy structured row answers consecutive_start_failures as an entitled key — justified

[FIRST-PRINCIPLES-REVIEWED] 6f0e8d1

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 6f0e8d18491b25fb705828d359ce602de35959a8 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the backend tolerates in-flight starts briefly exceeding session_start_concurrency — permitless collectors' requests are still processing while the reserved permit admits fresh ones.

Suggestions

  • The permit-headroom trade (bounded concurrency overshoot in exchange for never starving) is documented in acp-client.md but not at the agent.session_start_concurrency config key itself; a one-line note there would keep the knob's promise honest for operators tuning it.

[DESIGN-REVIEWED] 6f0e8d1

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings (all downgraded on adjudication)

Reviewed 6f0e8d18491b25fb705828d359ce602de35959a8 via the fork AI-review pipeline; updated in place on each push.

Opus 4.8 adjudication downgraded every blocking finding to advisory. Adjudication downgraded all 1 blocking finding(s) to advisory: the remedy each one requires is disproportionate to the harm it prevents. Read them as advice, not as merge conditions.

Review details

BLOCKING -- src/kiro_crew/autonudge.py:4512 -- start-failure state is published before persistence
loop.consecutive_start_failures += 1 / loop.consecutive_start_failures = 0 / self._persist_soon()
failure or recovery -> chat runner mutates durable loop state -> process exits before background write -> restart repeats doomed cycles or prematurely stands down
Anchor: persist-before-you-publish
Fix: Stage and await persistence under the service lock before publishing either mutation; await both notification paths from chat_runner.
[BLOCK-MERGE-DOWNGRADED] 6f0e8d1
[GPT-REVIEWED] 6f0e8d1

Adjudication (Opus 4.8) — is blocking on each finding proportionate?

I have enough to rule. The key facts, all confirmed from the base tree in this run:

  • _persist_soon() is the established, documented pattern here — its own docstring states the degradation contract: "A lost write degrades to a fresh full countdown after restart, never a premature or dropped fire" (autonudge.py:4633-4634).
  • The new hooks are modeled exactly on the sibling notify_approval_stalled (autonudge.py:4399-4431), which mutates loop.approval_stalled = True then calls the same fire-and-forget self._persist_soon(). The finding's demanded remedy ("await persistence under the service lock before publishing") would make this one counter diverge from its own sibling for no principled reason.
  • Recovery paths for a lost write: the streak rebuilds on each subsequent delivered failing cycle (notify_cycle_start_failed), any landed turn zeroes it (notify_cycle_landed), a revival clears it (autonudge.py:2827 region, new sibling line to approval_stalled = False), and the stand-down emits expired so the notifier tells the user (_timer, diff line ~5480) and is re-armable.

Harm in both directions is bounded, visible, and self-correcting: a lost increment just repeats a few doomed cycles (the pre-PR behavior the whole PR reduces); a lost clear stands the loop down at most one cycle early, which is visible (expired notification) and re-armable. No credential/privilege/governance surface, no silent corruption, no irreversible loss. Remedy cost is real: converting sync turn-lifecycle hooks to awaited persistence, threading awaits through chat_runner's terminal-error paths, and permanently diverging from the sibling counter's pattern.

LOW/MEDIUM recoverable+visible harm vs. a cross-module async-conversion remedy that also breaks pattern parity with the sibling that already ships this exact degradation.

[ADJUDICATION] 6f0e8d1 total=1 uphold=0 downgrade=1
DOWNGRADE F1 src/kiro_crew/autonudge.py:4512 reason=disproportionate-remedy
[GPT-ADJUDICATED] 6f0e8d1

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 6f0e8d18491b25fb705828d359ce602de35959a8 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6f0e8d1

@bolichen97
bolichen97 force-pushed the fix/runtime-recycle-start-gate-starvation branch from 6707f50 to 9d3382b Compare September 22, 2026 23:48
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: _load() now normalises consecutive_start_failures through _repair_number, like the sibling agent-writable numeric fields.

  • Malformed persisted streak crashes the timer span=1c5c73f18adb — fixed in 9d3382b.

Legitimate and accepted as a defect this PR introduced. The field is compared with >= on every wake and the store is agent-writable, so a persisted string or null raised TypeError inside _timer and the active automation would silently never fire again — surviving every reload.
Fixed at the load boundary, not at the comparison: _load() coerces the field through the same _repair_number helper next_due_ts and idle_secs use, clamped to lo=0.0, and sets _store_dirty so the corrected value is flushed once instead of re-repaired every boot. That is the spelling the adjudication named (config_generation, gate, self_armed are normalised there for exactly this threat).
Pinned by test/test_autonudge_start_failures.py::test_a_malformed_persisted_streak_is_normalised_at_load, parametrized over "3", None, -5, 2.9, NaN and 10**400; each case asserts a non-negative int after reload AND drives _timer so the flagged comparison is actually executed.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the collector_holds_denied counter is deleted; the denial log line carries both counts an operator reads.

  • Zero-consumer refusal counter span=b49afb0f3509 — fixed in 9d3382b.

Correct as raised, and the smaller complete solution is removal rather than a reader. The counter had no runtime consumer, no snapshot and no log line; the only thing that needed it was a spec sentence and one assertion in my own test, which is a counter that exists to be asserted.
_reserve_collector_hold now just refuses. The evidence an operator actually needs on a refusal is already emitted at the refusal site: _collect_late_start logs outcome=gate_permit_returned with collector_holds and collector_hold_ceiling, which says the same thing at the moment it matters instead of as a running total nobody reads.
The spec line in acp-client.md is rewritten to state that, so the doc no longer promises a field that is gone. The remaining assertions cover the behaviour (a denied permit is released and frees its slot), not the tally.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Swept population vs the 11 observed processes span=ae2dfe6497af — rebutted.

The premise is right and the scope reading is the answer, not a gap this round should close. This PR bounds ONE population: the shared background runtime, which had no bound at all off the reuse path. It does not claim the 11 processes are all of that lineage, and the body does not say so — the measurement is the symptom that found the leak, not a projected post-fix count.
A per-session runtime is owned by its session and is already reaped by session teardown and the idle-expiry sweep, so it is not unbounded in the way the shared slot was. The residue the item actually names is the BUSY-parked runtime, and the fix it suggests — a drain deadline — is a decision to kill a runtime with live handles mid-work. That is exactly the tradeoff _detach_bg_runtime_locked was written to refuse (killing mid-turn aborts an in-flight caller unrelated to the recycle), and %d _bg runtimes are parked draining already exists as the signal that displacement is outpacing the drain. Adding that deadline here would be a policy change to co-tenant safety smuggled in under a leak fix.
One runtime per pass is deliberate and not a throughput limit: the slot holds exactly one runtime, so retiring it IS the whole population, and the parked entry is reaped by the same hook on the next tick.
Post-fix field evidence cannot be produced from this branch — the leak is a property of a gateway running for hours — so asking this round for it would block a correct bound on a measurement only a deployed build can take.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Split the legacy pid-binding prune into its own PR span=f961163e78bd — rebutted.

The mechanism argument is fair and the packaging is the maintainer's call, which was made before this branch existed: the four fixes were scoped as one change on purpose, because they were found together on one host and each one's evidence is the others' context.
The revert coupling is real but narrow in the direction that matters. The prune touches member_memory_auth.py plus one added _sweep_member_pid_bindings tick; it shares no state, no lock and no call path with the gate or the backoff, so a revert of this commit loses a GC that was collecting nothing before this PR either — the records accumulate on main today. It does not leave the starvation chain half-reverted, which is the coupling that would justify the split.
Recorded rather than dismissed: if a maintainer prefers the split, the prune lifts out cleanly as member_memory_auth.py + the one session_cleanup.py tick + test/test_member_pid_binding_prune.py + the session.md bullet, with no edit to the other three fixes.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the collector_holds_denied counter is deleted.

  • Zero-consumer refusal counter span=b49afb0f3509 — fixed in 9d3382b.

Accepted on the same grounds this lane raises it: a field whose only consumers were its own definition, its own increment, a spec sentence and one assertion in my test is state that exists to be asserted.
Removed rather than given a reader. The refusal already emits what an operator needs at the moment it matters — _collect_late_start logs outcome=gate_permit_returned with collector_holds and collector_hold_ceiling — so a running total adds a second place for the same fact to be wrong. The spec line that promised the counter is rewritten to say that instead.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@bolichen97
bolichen97 force-pushed the fix/runtime-recycle-start-gate-starvation branch from 9d3382b to d14a099 Compare September 23, 2026 00:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: one _note_cycle_start_failure helper, called from both terminal branches.

  • Shared-runtime start failure bypasses the AcpError-only report span=8023cfe9506b — fixed in d14a099.

Legitimate, and an incomplete fix rather than a style point. Confirmed in the code: the branch I had edited is under except AcpError as exc:, and AcpSessionStartTimeout/AcpRequestTimeout descend from AcpRuntimeError, which chat_runner did not name anywhere — so a shared-runtime start failure landed in the generic terminal handler and was never reported. A member slot is exactly the population that runs on the shared runtime, which is where the 15-cycle incident was measured, so the fix missed its own motivating case.
Extracted _note_cycle_start_failure(slot_key, exc, *, self_wake) and called it from BOTH terminal branches. It keeps the two guards unchanged: the report needs the fire path's _directive_self_wake marker, so a human turn cannot spend an unrelated loop's stand-down budget, and it reads the tag with getattr rather than by exception type, which is why one helper can serve two families that share no base.
Pinned by test_both_terminal_branches_report_a_tagged_start_failure (an AcpError and an AcpRuntimeError instance both report) and test_an_untagged_or_human_turn_reports_nothing (an untagged failure and a non-self-wake turn report nothing).

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Load-time streak repair undeclared in the description span=5251e29ffe65 — fixed in d14a099.

Correct: the repair was real and the body did not carry it, which is the one thing the accounting rule exists to catch. It is not a rider I want dropped — it is the fix for the round-1 blocking finding — so the answer is to declare it, not remove it.
## What changed now states it: the persisted streak is normalised at the _load() boundary through the same _repair_number helper gate, next_due_ts and idle_secs use, because the store is agent-writable and a non-int reaching the >= comparison would stop the automation for good. The same paragraph also declares the two-branch report the GPT lane asked for, so the description matches the whole diff rather than the part of it that had a section already.
The lane's reading that it carries no surface of its own stands: no config key, no route, no field added by the repair itself.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Load-time streak repair undeclared in the description span=5251e29ffe65 — fixed in d14a099.

Accepted on the same grounds. The repair is now named in ## What changed with its reason (agent-writable store, >= on every wake, the sibling fields already normalised at that boundary), so it is declared rather than riding along.
It adds no surface of its own, which is why it belongs in the sentence about the streak instead of a section: a reader who wants to know what this PR gives them does not learn anything from a boundary coercion except that the field cannot wedge the timer.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Elevated backend in-flight session/new ceiling under sustained timeouts span=e21ac5e86acc — rebutted, with the bound stated.

The mechanism is described correctly and the elevated ceiling is accepted deliberately, because it is BOUNDED and the behaviour it replaces is not a lower ceiling but a gateway-wide stall.
The bound, from the two budgets already in the code: new starts are still gated at session_start_concurrency (2), and a start cannot become a collector until it has burned session_start_timeout_secs (90 s floor), while a collector lives at most start_collect_timeout_secs (300 s). Concurrent collectors are therefore at most about limit x collect_timeout / start_budget = 2 x 300/90 ~ 7, so backend in-flight session/new is bounded near limit + 7, not unbounded. Sustained timeouts cannot grow it further, because the gate still rate-limits what enters.
What the release does NOT do is send the backend anything new. The collector's request was already accepted and is already being worked on; the permit only decided whether anyone ELSE may enter. So the old shape did not hold the backend to limit — it held the GATEWAY to zero while the backend worked on limit requests anyway, which is the 15-cycle incident.
Judged as a tradeoff: at most a handful of extra concurrent starts at a backend that is already slow, against every session start on the host queuing behind callers that gave up. I am taking the bounded overshoot. Happy to add the arithmetic above to acp-client.md as a durable statement if a maintainer wants it in the spec rather than the ledger.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Separable pid-binding GC bundled with the starvation fix span=94138780a25a — needs-a-decision.

The separability reading is right and I am not going to argue it away: the prune shares no state, lock or call path with the gate and the backoff. What it does share is provenance — all four defects were found in one pass over one host's process table and data home, and each one's measurement is the others' context.
Putting it to a maintainer rather than filing an issue, because this is a packaging preference only a human can settle and it is cheap either way. The revert coupling is narrower than the item implies: reverting this commit loses a GC that collects nothing on main today, so it does not leave the starvation chain half-reverted.
If you want the split, the prune lifts out as src/kiro_crew/member_memory_auth.py, the one _sweep_member_pid_bindings tick plus its dep in session_cleanup.py/session.py, test/test_member_pid_binding_prune.py, and the session.md bullet — no edit to the other three fixes. Say the word and I will cut it as a follow-up PR against this one's base.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Separable pid-binding GC bundled with the starvation fix span=94138780a25a — rebutted (maintainer decision: keep the bundling).

Settled by the maintainer, who scoped the four fixes as one PR: they are one causal chain from one incident, not four coincidences. Runtimes are never recycled, so memory fills and session starts slow; a slow start parks a gate permit and starves the gate; a member slot then never gets a session; the auto-nudge loop on it spins on cycles that cannot work.
The pid-binding records are the same defect class as the rest, not a stray GC: per-pid lifecycle state that is written and never reclaimed. Splitting it would file the one half of that omission separately from the halves it was found with, and lose the shared evidence that makes each one legible.
Supersedes my earlier needs-a-decision on this span.

Four leaks that compound into one failure: agent runtimes that never get
recycled, a session-start gate that starves behind requests whose callers
gave up, a monitoring loop that keeps spending cycles on doomed starts, and
per-pid binding records nothing collects.

- The shared background runtime's staleness ceilings were only evaluated on
  the reuse path, and its pid is shielded from the orphan sweep, so a runtime
  nobody reuses is never asked and never reaped. The periodic watchdog hook
  now asks off the reuse path and parks an idle-and-stale runtime.
- A timed-out session/new handed its gate permit to a StartCollector for up
  to 300s; at the default limit of 2 two of those held the whole gate. One
  permit is now reserved for a start that has not gone out.
- An auto-nudge loop whose cycles never got a session re-armed on its plain
  interval. It now backs off after three and stands down after five, on
  recorded evidence, cleared by any landed turn.
- Aged member-memory pid binding records whose pid names no live process are
  collected by the same periodic tick, budgeted per pass.
@bolichen97
bolichen97 force-pushed the fix/runtime-recycle-start-gate-starvation branch from d14a099 to 6f0e8d1 Compare September 23, 2026 00:38
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 23, 2026 03:56
@bolichen97
bolichen97 merged commit 29d7bc6 into kirodotdev:main Sep 23, 2026
101 of 106 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 23, 2026
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…irodotdev#12944)

Four leaks that compound into one failure: agent runtimes that never get
recycled, a session-start gate that starves behind requests whose callers
gave up, a monitoring loop that keeps spending cycles on doomed starts, and
per-pid binding records nothing collects.

- The shared background runtime's staleness ceilings were only evaluated on
  the reuse path, and its pid is shielded from the orphan sweep, so a runtime
  nobody reuses is never asked and never reaped. The periodic watchdog hook
  now asks off the reuse path and parks an idle-and-stale runtime.
- A timed-out session/new handed its gate permit to a StartCollector for up
  to 300s; at the default limit of 2 two of those held the whole gate. One
  permit is now reserved for a start that has not gone out.
- An auto-nudge loop whose cycles never got a session re-armed on its plain
  interval. It now backs off after three and stands down after five, on
  recorded evidence, cleared by any landed turn.
- Aged member-memory pid binding records whose pid names no live process are
  collected by the same periodic tick, budgeted per pass.

Co-authored-by: w <w@k.local>
@bolichen97
bolichen97 deleted the fix/runtime-recycle-start-gate-starvation branch November 20, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants