Skip to content

fix: spare live-account holders from the identity sweep so it can finish - #14609

Merged
iamwhatever merged 1 commit into
mainfrom
fix/identity-sweep-livelock
Sep 28, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/identity-sweep-livelock

Conversation

@CrysisDeu

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Long-running subagents stop dying as "provider shutdown" on a busy gateway, and the account-identity sweep finishes instead of re-running on every turn.

On a gateway with many live chat sessions and spawn_run subagents in flight, every subagent is killed a few minutes into its run:

kiro_crew.acp.client.AcpProcessDied: Runtime process died during prompt — killed (provider shutdown) [returncode=<not reaped>]

44 kills in ~5 h on one host, all with the same signature, none from OOM, timeout or a model error (194 GB free throughout). Children of one parent chat always die in the same second, when any chat sends a turn; the parent's resume pointer is cleared to discarded_sid at the same moment; idle chats get a fresh kiro-cli pid roughly every two minutes. No account change happened.

Why it matters

Every subagent-backed workflow on a busy host is unusable: PR babysits, research fan-outs and reviews die mid-run and the parent sees only an opaque provider shutdown traceback. Each idle chat is also recycled into a fresh native conversation on every turn anyone takes, for nothing. The trigger fires at INFO, so operators have no signal that the safety sweep is the cause.

Not a goal

  • Changing when the sweep is triggered (baseline, boot seed, interim latch) — fix: seed identity sessions-baseline at boot to stop the unset-baseline sweep loop #12956 already seeds the baseline at boot; this PR is about what a running sweep retires.
  • Retiring a busy holder, or a parent's children, differently when the holder really is on the previous account — that path is unchanged.
  • Closing the documented residual inside a single spawn's bracket read (needs the child to report its own identity).
  • Raising the log level of the sweep's trigger line.

What changed (motivation → approach → change)

Symptom → root cause. retire_kiro_identity_sessions retired every idle kiro-backed session and companion runtime, and reported complete=True only when none of them was busy, nothing was mid-start and no kiro-backed runtime survived. The comment said so explicitly: "with no attempt to spare one that registered after the pending change … registration order does not carry [the identity]". On a gateway with a dozen live chats that precondition is never satisfied, so the baseline never advances (note_sessions_reconciled is only called on a complete sweep), the pending fingerprint stays set, and the turn gate re-sweeps on every turn from every chat. Each sweep recycles every idle session — and a parent chat that ended its turn with spawn_run children still running is idle by the semaphore.locked() test, so its retirement runs _cancel_parent_children(... verb="retire_kiro_identity_sessions") and the children die as provider shutdown.

Approach. Registration order does not carry the identity a session authenticated under, but the spawn-identity stamp (spawn_identity, recorded by every spawn site since the stamp gate landed) does. A holder whose stamp equals the live fingerprint has provably authenticated as the live account; retiring it buys nothing and costs its children, and counting it against completeness is exactly what kept the sweep from ever finishing.

Change.

  • kiro_prerequisite.py: spawn_identity_of(holder) (the provider-or-runtime stamp lookup the flag gate already did inline) and spawned_under(holder, live) — nonempty stamp equal to a nonempty live fingerprint, whole. Deliberately stricter than identity_stamp_mismatch: a mismatch is proven component-wise because a read can only lose components; a spare must not be granted on a partial match. Empty live spares nothing.
  • session_lifecycle.py: the sweep's session loop skips spawned_under(sess.provider, fingerprint) holders — not retired, not flagged, not counted as skipped. The two unfiltered runtime reapers take live= and apply the same spare to the reap and to their completeness post-conditions (registered survivors, parked drainers). Filtered (mismatch-gate) mode is untouched. The flag gate reuses spawn_identity_of.
  • session.py: the two facade methods forward live=.
  • docs/system-specs/modules/session.md: new Live-account spare paragraph in the identity-sweep section.

Backwards compatibility

Compatible: the sweep's signature is unchanged; _retire_kiro_subagent_runtimes / _retire_kiro_bg_runtime gain a keyword-only live="" whose default is the previous behaviour. Unstamped holders, differently-stamped holders, and every sweep run with an empty fingerprint (the sign-out path in kas_login.py calls retire() bare on purpose) behave exactly as before.

Tests

test/test_external_logout_detection.py::TestRetirementCoverage, seven new cases:

  • an idle session stamped with the live account is spared while unstamped and other-stamped siblings are retired, and the sweep is complete;
  • a busy live-stamped session does not block completion and does not leave a pending fingerprint (the livelock);
  • a demuxed session is spared through its shared runtime's stamp;
  • the spare needs the whole fingerprint (CLI-only stamp vs CLI+vault live → retired);
  • an empty live fingerprint spares nothing (a stamped holder is retired);
  • a live-stamped subagent runtime is spared even when busy while an unstamped sibling is reaped, sweep complete;
  • a live-stamped background runtime is spared even when busy, sweep complete.

Existing sweep tests call retire_kiro_identity_sessions() with no fingerprint and are unaffected.

Manual verification

N/A — unit coverage sufficient: the change is a predicate on in-memory state the existing fakes model directly. The field symptom (44 provider shutdown kills, same-second sibling deaths, discarded_sid on the parent) is documented in #14605; the host that produced it runs 0.7.1rc2, which predates both #12956 and the stamp gate, so a live re-test of this head waits for the next insider build.

Related Issues

Fixes #14605. Builds on #12956 (boot-seeded baseline) and the spawn-identity stamp gate.

Pattern harvest

Rule candidate: review-prompt
Pattern: a "retire until quiescent" sweep whose completeness requires every holder to be idle at once — on a busy host that is a livelock, and each pass must be checked for collateral on holders the pass cannot prove guilty.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 8d875b98e94890499ae4afc339119eb1fe52d6e8 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/session_lifecycle.py:1575 -- an A→B→A sweep hits "continue" without clearing the busy A-stamped session’s retirement flag, so its next claim evicts the now-live provider -> Fix: clear retire_on_identity_change before continuing.
[GPT-REVIEWED] 8d875b9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8d875b98e94890499ae4afc339119eb1fe52d6e8: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 28, 2026
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 8d875b98e94890499ae4afc339119eb1fe52d6e8 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Livelock silently returns wherever a spawn-stamp read fails, and no field host has yet exercised the stamp gate this fix depends on.

Watch

The spare only works for stamped holders; the docstring itself admits a stamp can be empty because "the spawn-time read failed, disagreed with its pre-spawn read, or the stamping is unwired." One busy unstamped live-account session then keeps every sweep incomplete — the per-turn resweep and provider shutdown kills of #14605 return in full, and by the PR's own "Why it matters" the trigger fires only at INFO, so operators again get no signal that the sweep is the cause. The description also concedes the reproducing host "predates both #12956 and the stamp gate," so end-to-end resolution is unverified against the field symptom.
Clears when: an incomplete sweep surfaces (WARNING log or counter) the unstamped kiro-backed holders it waited on, and/or the next insider build confirms sweeps complete on the busy host from #14605.

[DESIGN-REVIEWED] 8d875b9

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 8d875b98e94890499ae4afc339119eb1fe52d6e8 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify on a stamp-gated build: the field host (0.7.1rc2) predates the stamp gate, so the spare's premise — holders carry spawn_identity — is untested in the field.

Not justified as shipped

  • Item 4 — undeclared: the description's change list says session.py only "forward[s] live=", but the diff also adds the expected= compare-and-pop pin to release_subagent_runtime in session_allocation.py and changes its return from None to bool. It is load-bearing for the fix (without it a mid-sweep respawn's live-stamped replacement is popped by key and killed), tested by three of the ten new tests, and doc'd — but the description never names it.

What this change ships

Inventory (5 items) — 4 justified

Intent: FIX — stop the identity sweep from killing live-account subagents and from re-running every turn on a busy gateway (#14605).

  1. Idle sessions stamped with the live account are no longer retired or flagged by the identity sweep — justified
  2. Busy live-stamped holders no longer block sweep completion, so the baseline advances and per-turn re-sweeps stop — justified
  3. Companion and background runtimes with a live stamp are spared by the reapers and their completeness post-conditions (live=) — justified
  4. Runtime release is pinned to the runtime the sweep inspected (expected=), so a mid-sweep replacement survives — undeclared: absent from the description's change list
  5. session.md gains the Live-account spare paragraph in the same commit — justified

Evidence checked: the rewritten pin ("no attempt to spare … registration order does not carry it") is quoted by the description itself with the reason its premise fell (the spawn stamp now carries the identity). Stamp sites counted: stamp_spawn_identity called at 7 spawn sites across session_pool.py, session_background.py, session_allocation.py; an unstamped holder fails safe (retired as before). release_subagent_runtime callers: 10 in src, all unaffected by the default expected=None.

[FIRST-PRINCIPLES-REVIEWED] 8d875b9

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 8d875b98e94890499ae4afc339119eb1fe52d6e8 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8d875b9

Verdict parsed from the review's SHA-scoped output markers for commit 8d875b98e94890499ae4afc339119eb1fe52d6e8.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8d875b98e94890499ae4afc339119eb1fe52d6e8: <one-sentence reason>

@CrysisDeu
CrysisDeu force-pushed the fix/identity-sweep-livelock branch from 6ba398e to 44100a5 Compare September 28, 2026 03:57
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
The identity sweep retired every idle kiro-backed session and companion
runtime, and reported itself complete only when none of them was busy.
On a gateway with a dozen live chats that never happens, so the baseline
never advanced, every turn from every chat re-swept, and each sweep
recycled every idle session. A parent chat that ended its turn with
spawn_run children still running is idle by the semaphore test, so its
retirement cancelled the children: every subagent on the host died as
"AcpProcessDied ... killed (provider shutdown)" a few minutes in, on
every turn any chat took, with no account change having happened.

The spawn-identity stamp already records which account each child
authenticated under. The sweep now spares a session or runtime whose
stamp EQUALS the live fingerprint (spawned_under): not retired, not
flagged, not counted against completeness. Unstamped or differently
stamped holders are retired as before, and an empty live fingerprint
(unreadable store; the sign-out path) spares nothing, so the fail-safe
retire-everything sweep is unchanged where the stamp cannot vouch.

Fixes #14605
@CrysisDeu
CrysisDeu force-pushed the fix/identity-sweep-livelock branch from 44100a5 to 8d875b9 Compare September 28, 2026 04:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
@iamwhatever
iamwhatever merged commit 9a40810 into main Sep 28, 2026
100 of 101 checks passed
@iamwhatever
iamwhatever deleted the fix/identity-sweep-livelock branch September 28, 2026 05:32

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff and ran test/test_external_logout_detection.py locally on head 8d875b9: 170 passed. The spare is exact-equality on a nonempty live fingerprint, empty live spares nothing, and the expected= pin on release_subagent_runtime closes the respawn race. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

identity sweep never completes on a busy host and kills every idle parent's subagents on every turn

3 participants