Skip to content

fix: seed identity sessions-baseline at boot to stop the unset-baseline sweep loop - #12956

Merged
bolichen97 merged 1 commit into
mainfrom
fix/identity-sweep-baseline
Sep 24, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/identity-sweep-baseline

Conversation

@helenastafford

@helenastafford helenastafford commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Seed the identity sessions-baseline at gateway boot to stop the unset-baseline sweep loop

Problem

The identity-retirement sweep (_retire_sessions_on_identity_change) compares the live auth-store fingerprint to a sessions-baseline that starts unset, and unset deliberately reads as "changed" (fail-safe). The baseline only advances after a fully quiescent sweep pass — nothing busy, nothing mid-start, no subagent-runtime lock held. On a live gateway that precondition is unsatisfiable: retiring idle sessions makes dashboard tabs eagerly respawn them, and those in-flight starts keep every subsequent pass incomplete. The result is an infinite retire→respawn loop that re-arms ~1 minute after every boot and kills healthy shared runtimes mid-prompt — long-running subagents (20–60 min) die deterministically while short ones slip between sweeps.

No identity ever actually changes. identity_fingerprint already allowlists out all rotating fields (access_token, refresh_token, expires_at), and kiro-cli whoami is stable across the loop. Captured end-to-end at INFO level on a live gateway (2026-09-22): repeated Kiro identity changed; retired N session(s) (incomplete, will retry next turn) with zero fingerprint change and zero store errors, retiring the same session five times in a loop.

Prior art

Fix

Seed the sessions-baseline at gateway boot, before any kiro-backed child can spawn:

  • kiro_prerequisite.py: new seed_sessions_baseline() adopts the on-disk account as the baseline. Refusals are all fail-safe toward the existing behavior (unset baseline → sweep): assume-ready/test mode, baseline already recorded, unreadable store, and a bounded store read (asyncio.wait_for, 5s) so a hung SQLite cannot stall boot. The read runs off-loop via asyncio.to_thread and does not scale with user data.
  • dashboard/server.py: both gateway start sites call the seed at boot, before warm_up() and before the socket serves.

Because every child necessarily postdates the seed, the never-completing unset-baseline boot sweep becomes redundant. A real account change still compares unequal and sweeps exactly as before — the genuine-logout credential guarantee from #5904 is fully preserved, which is why this shape survives the busy-exemption objection recorded there.

A→B→A round trips: closed for every observed switch (interim-identity latch)

Addressing the GPT review's security finding: a bare baseline comparison cannot see a round trip (seed A → switch to B, a child spawns under B → switch back to A compares equal). Rather than bolting the per-child allocation stamp onto this PR (it would rewrite the sweep's documented retire-everything invariant across the most contended session files), the fix latches the observation at the fingerprint-read choke point:

  • Every fresh read already flows through current_identity_fingerprint — status polls (every few seconds per open dashboard tab), every turn gate, readiness probes. If any of them observes a readable, different signed-in account than the baseline, a sticky flag arms and identity_changed_since_sessions reports changed until a sweep completes, whatever the store says by then. The existing retire-everything sweep then recycles the interim children; note_sessions_reconciled clears the flag.
  • Loop-reintroduction guard (deliberate refusals): the latch never arms on component loss — an unreadable CLI store, a vanished vault component — because those are indistinguishable from transient read failures, and a sticky flag armed by a blip would force retire-until-complete sweeps on a healthy host, re-creating the exact pathology this PR removes. Real sign-outs stay covered by the ordinary (non-sticky) comparison. Components that appear or change latch: reads lose components under failure, they do not gain them.
  • Stamp-accuracy races fail safe by construction: the latch only ever triggers the full sweep, never selects victims, so a wrong observation costs one extra recycle of an idle session — never a spared stale child.

Per-child spawn-identity stamp (closes the unobserved round trip): every kiro-backed provider now records the account the store held as its process started (read_spawn_identity → stamp_spawn_identity, wired at all five spawn sites: cold start, shared run runtime, warm-pool fill, background create/recycle — first-stamp-wins so a pool claim keeps its fill-time record). The turn gate compares those records against its fresh read (flag_identity_stamp_mismatches) before the unchanged early-return, so a child from an A→B→A round trip that no read ever observed is still retired from its own record — the one case a gateway-wide baseline is inherently blind to. Loop-safe by construction: flag-only (the existing retire_on_identity_change eviction path recycles it; nothing global or sticky), component-proven mismatches only (a component counts only when nonempty on both sides — reads lose components under failure, never gain or alter them — so a blip at spawn or at the gate can never flag a healthy child), and an unstamped child (unwired entrypoint, failed read) keeps exactly the pre-stamping protections. The gateway-wide baseline test still pins that the baseline stays honestly blind to the unobserved round trip; the child is covered per-record.

Spawn-window hardening (review round 4): every spawn is now bracketed by a pre-start and post-start identity read, and the stamp is recorded only when both agree (pre_spawn_identity → provider.start() → stamp_spawn_identity). A single post-start read could label a child that authenticated as B with the account the store switched back to; disagreement now refuses the stamp (fail-safe: unstamped children keep the pre-stamping protections), and either read observing an interim account feeds the latch through the ordinary read path. The turn gate's mismatch pass also covers the companion runtime registries (_bg_runtime, _subagent_runtimes): those runtimes carry the same spawn stamp but never appear in _sessions, so a proven-mismatch runtime — busy or idle alike — is displaced: popped from the claimable slot (a replacement spawns under the live account) and parked on a drain list, where a later pass ends it only once it is idle past a park grace. Same-pass kills are never issued: the idle probe races a claim the runtime was just handed, which is also why create_session now opens its init scope before the admission gate — a gate-queued claim already reads as initializing.

Spawn-window hardening (review round 6): every stamp read now runs under the start-to-registration PID shield (_starting_pids, unioned into the orphan sweep's active set): the read is a bounded suspension point after start() published the PID but before any registry the sweep consults can see the child, and on Windows the sweep has no age grace — a tick landing in that await killed the healthy child. All six spawn sites shield before the read and release once registration is visible (or on teardown), pinned by a behavioral capture test and a source scan.

Reconcile and eviction hardening (review round 7): interim observations are generation-scoped: every qualifying read bumps a counter (even while the latch is armed), the turn gate captures it right after its initiating read, and note_sessions_reconciled keeps the latch armed when any observation postdates that capture — so a store flap observed while the sweep ran can never be erased by the sweep's own completion, and the next turn covers it with one more sweep. The stamp gate now also mirrors the sweep's clear_sid over flagged sessions (the eviction re-entry otherwise session/loaded the flagged account's conversation, and close_all's deliberate skip preserved the stale pointer across restart), the companion drain reap removes only entries it actually killed instead of rebuilding the list across its kill await (a rebuild dropped a concurrently parked runtime from its only remaining reference), and docs/system-specs/modules/session.md gains the owning "Account-identity retirement" section per the same-commit spec rule.

Remaining residual: a switch and return that both complete strictly between the two bracket reads of one spawn, with the child sampling the interim — closing it fully needs the child to report the identity it actually loaded (a protocol change; documented on stamp_spawn_identity as the named follow-up).

Validation

  • Red/green against the real sweep (stash-toggling only src/): pre-fix, the new regression class fails 9/10 — the one pre-fix pass, test_an_unseeded_busy_gateway_re_sweeps_every_turn, reproduces the production loop through the real _retire_sessions_on_identity_change (5 turns → 5 sweeps → 5 healthy children recycled, never complete). Post-fix: 10/10, including "a switch/sign-out after seeding is still detected" and a source-scan asserting both start sites seed (an upstream refactor that drops one fails loudly).
  • Full test_external_logout_detection.py: 118/118 (117 existing + 1 new hung-store timeout test). Adjacent suites (session, KAS login/host-auth, service boundaries, session-duration metrics): 385 pass.
  • Three startup-test fixtures needed their KiroPrerequisiteService mock upgraded to AsyncMock for the new boot await (included).
  • Live pod boot on the fixed code logs Identity baseline seeded at startup; boot sweep skipped and serves healthy; a fresh unset-baseline boot on unfixed code re-arms the loop within a minute.
  • Latch red/green: the 3 security-property tests (round trip observed by a poll still sweeps; a complete sweep clears the latch and turns go quiet; a vault identity round trip latches) fail without the latch and pass with it; the 4 refusal/invariant tests (CLI-store blip never latches, vanished vault component never latches, unobserved round trip pinned as residual, stable identity never latches) pass on both sides, as refusals must. Full file now 125/125; affected suites 204/204.
  • Stamp red/green: all 11 TestSpawnIdentityStamp tests fail pre-fix and pass post-fix (verified by stash-toggling only src/), including the end-to-end unobserved-round-trip flag, the healthy-host/blip/unstamped refusals, first-stamp-wins, the shared-runtime fallback, and source-scans pinning all five spawn sites plus both startup wirings. Full file now 138/138; adjacent suites 477 + 151 (pool, background lifetime, start gate, cleanup) pass.
  • flake8 / mypy / black / isort clean on changed files.

Pattern harvest

Rule candidate: review-prompt — flag any safety sweep whose corrective action re-creates its own trigger condition (here: retire → eager respawn → next pass incomplete), i.e. a completion precondition that requires global quiescence on a system whose remediation prevents quiescence.

A fail-safe default ("unset means changed") combined with a completion precondition that the fail-safe's own remediation invalidates (retire → eager respawn → next pass incomplete) turns a one-shot safety sweep into a permanent loop. When a safety check's trigger state can be re-created by its own corrective action, the trigger must be resolvable without requiring global quiescence — here, by establishing the baseline before any actor exists that could block reconciliation.

@helenastafford
helenastafford requested review from a team and patrigao September 23, 2026 00:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings (all downgraded on adjudication)

GPT 5.6 flagged blocking issues on acde797ba3f6f526b7ec90ee580421e81377dabe; Opus 5 adjudication downgraded every one of them to advisory. Adjudication downgraded all 1 blocking finding(s) to advisory: the remedy each one requires is disproportionate to the harm it prevents. Read them as advice, not as merge conditions.

This comment is updated in place on each push.

Review details

BLOCKING -- src/kiro_crew/session_lifecycle.py:1262 -- synchronous persistent-store mutation runs on the gateway loop

owner._session_map.clear_sid(key)
Identity mismatch -> turn gate -> contended session-map lock blocks the event loop and freezes gateway work.
Anchor: no-sync-store-call-from-a-coroutine
Fix: call clear_sid via await asyncio.to_thread(...).
[BLOCK-MERGE-DOWNGRADED] acde797
[GPT-REVIEWED] acde797

Adjudication (Opus 5) — is blocking on each finding proportionate?

LOW — the claimed mechanism (loop stalls on store IO) is not present: clear_sid mutates the in-memory dict under _MAP_LOCK (session_map.py:1113-1115, guard at :254) and calls _save, which detects a running loop (:579-584) and defers the write to a debounced task that does the disk write via asyncio.to_thread (:646-651); no sqlite connection or busy_timeout — the mechanism AUTOSDE.yaml:643-646 names — is involved, so the anchor's own allowed clause ("the DB phase already goes through the writer or an *_async seam", AUTOSDE.yaml:670) covers this site.
Self-correction if a flush is lost: _take_pending_snapshot re-owes the dirty mark and the task loops until it observes a clean map (session_map.py:606-611, :628-631, :652-657), so a trailing mutation is never dropped.
Real-fix cost exceeds that: wrapping this call in await asyncio.to_thread(...) inserts a suspension point into post-flag cleanup that docs/system-specs/modules/session.md:2495 says must not sit behind a skippable await, and diverges from the six sibling on-loop clear_sid sites (session_lifecycle.py:889,1401,1955, session_compaction.py:685, session_allocation.py:1877,2023) whose comments already reason about this exact seam — permanent inconsistency for every future reader, against a stall that the deferred-write path already removed.

[ADJUDICATION] acde797ba3f6f526b7ec90ee580421e81377dabe total=1 uphold=0 downgrade=1
DOWNGRADE F1 src/kiro_crew/session_lifecycle.py:1262 reason=disproportionate-remedy
[GPT-ADJUDICATED] acde797ba3f6f526b7ec90ee580421e81377dabe

False positive or not applicable? A repository writer can comment:
/ai-review override gpt acde797ba3f6f526b7ec90ee580421e81377dabe: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed acde797ba3f6f526b7ec90ee580421e81377dabe — this comment is updated in place on each push.

Review details

No blocking violations; two advisory findings on the new identity-park and per-turn bg-lock paths.

FINDING — src/kiro_crew/session_lifecycle.py:1565 — owner._draining_subagent_runtimes.append(displaced) parks a live wrong-account kiro-cli child on a list whose only reaper is the loop at 1517 inside _retire_kiro_subagent_runtimes, reachable solely from the per-turn stamp gate (1274) and the turn-driven sweep (1434), while session.py:2278 unions that list into _collect_active_pids and shields its pid from the orphan sweep — so after the last turn of a chat the displaced child survives, unreapable, until close_all, whereas the analogous _draining_bg_runtimes is drained by the periodic CleanupHook("bg_drain_reap", …) → Fix: register an equivalent periodic hook for _draining_subagent_runtimes in session_cleanup.py alongside _bg_drain_reap_hook (untouched file, hence advisory).

FINDING — src/kiro_crew/session_lifecycle.py:1623 — async with owner._bg_runtime_lock: is entered before any cheap check, and flag_identity_stamp_mismatches → _retire_kiro_bg_runtime(should_retire=_mismatched, …) now runs on every turn (chat_runner.py:3889-3891, ahead of the if not changed: early return), so a turn that coincides with a background-runtime cold start waits out the whole of get_bg_session's lock hold (session_background.py:603-791 spans pre_spawn_identity ≤5s, await replacement.spawn(), and stamp_spawn_identity ≤5s) with no timeout on the best-effort gate → Fix: in the should_retire is not None path, return before acquiring the lock when owner._bg_runtime is absent/non-kiro and owner._draining_bg_runtimes is empty.

[OPUS-REVIEWED] acde797

Verdict parsed from the review's SHA-scoped output markers for commit acde797ba3f6f526b7ec90ee580421e81377dabe.

False positive or not applicable? A repository writer can comment:
/ai-review override fable acde797ba3f6f526b7ec90ee580421e81377dabe: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of acde797ba3f6f526b7ec90ee580421e81377dabe — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Two-thirds of the diff — stamps, parks, drain lists — chases a round trip "NO read ever observed", asserted only by a review pass, and concedes it stays open.

Not justified as shipped

  • Item 4 — oversized rider: seven stamp_spawn_identity call sites (counted: 3 in session_allocation.py, 3 in session_background.py, 1 in session_pool.py), seven new kiro_prerequisite symbols, a new drain list and a 30s park grace exist for the unobserved A→B→A round trip, whose only stated provenance is "Addressing the GPT review's security finding" — while stamp_spawn_identity's own docstring concedes a switch-and-return "strictly between the two reads... can still stamp the wrong account" and names the real close (child reports the identity it loaded, a protocol change). The blindness it narrows exists identically on base once a baseline is set; the seed plus latch already cover every observed switch.
    Subtraction: defer the stamp/displacement layer (stamp_spawn_identity, pre_spawn_identity, identity_stamp_mismatch, flag_identity_stamp_mismatches, _draining_subagent_runtimes, IDENTITY_PARK_GRACE_SECS, the seven call sites) to the named protocol-level follow-up; ship seed + latch.
    Clears when: a linked issue or reproduction of a wrong-account child from an unobserved round trip, or the layer is deferred to the protocol-level fix.

What this change ships

Inventory (7 items) — 3 justified

Intent: stop the boot-time retire→respawn loop that kills healthy long-running children when no identity ever changed — a FIX.

  1. Gateway no longer sweeps/retires healthy sessions every turn after boot (baseline seeded at startup) — justified
  2. Boot waits at most 5s on the store read; a hung or unreadable store keeps the old fail-safe sweep — justified
  3. An observed account switch-and-back now sticks and forces sweeps until one completes (interim latch + generation) — rides along: closes a base-era hole the boot loop never involved, but the harm is the Meeting agents retired mid-init by dashboard chat 'Kiro identity changed' reconcile — meeting wedges status:active, never transcribes #5904 credential boundary
  4. Every kiro spawn is double-read and stamped; the turn gate retires sessions and displaces/parks runtimes whose stamp provably differs — rides along, oversized (see above)
  5. A claim queued behind the admission gate now counts its runtime busy, for all backends — rides along: needed only by item 4's displacement pass
  6. Freshly spawned PIDs are shielded from the orphan sweep across the stamp read; parked runtimes count as active and die at shutdown — rides along: exists only to make item 4's stamp read safe
  7. session.md documents the identity-retirement machinery — justified

[FIRST-PRINCIPLES-REVIEWED] acde797

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of acde797ba3f6f526b7ec90ee580421e81377dabe — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

A store whose fingerprint flaps between two nonempty values re-arms the sticky latch on every poll, reviving the retire/respawn churn this PR removes.

Watch

  • The latch's loop guard rests on "reads lose components under failure, they do not gain or alter them" (_maybe_latch_interim_identity). That holds for local read failures but is asserted, not established, for the stale-alternating reads the PR itself anticipates ("stalled network filesystem"): an A/B nonempty flap qualifies as component change, every status poll bumps the generation, note_sessions_reconciled keeps the latch armed, and the gateway sweeps every turn indefinitely — idle-retire → eager respawn churn, the original pathology on a new trigger, with no log distinguishing it from a real switch.
    Clears when: a churn bound or explicit diagnostic exists for a persistently re-arming latch (e.g. a test or counter showing N consecutive latch-forced sweeps surfaces as a warning naming the flapping store).
  • Opening _session_inits_in_flight before the admission gate (acp/runtime.py) widens "initializing" to gate-queued claims for every consumer of has_active_or_initializing_sessions — backend-switch drains and bg reaps (session_background.py:386,448,533), not just the new displacement pass — so under a saturated gate, drains stay parked and the identity sweep reads incomplete longer, delaying baseline advance and latch clearance. Direction is fail-safe, but only the displacement consumer is pinned.
    Clears when: a test (or author confirmation) shows a saturated start-gate queue cannot hold the identity sweep incomplete beyond the queue's own drain.

Suggestions

  • Log a distinct once-per-interval diagnostic when the latch re-arms immediately after a completed sweep — cheap, and it makes the flap scenario above operator-visible instead of silent churn.

[DESIGN-REVIEWED] acde797

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@helenastafford
helenastafford force-pushed the fix/identity-sweep-baseline branch from 802270e to 4e255d0 Compare September 23, 2026 01:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@helenastafford

Copy link
Copy Markdown
Contributor Author

Addressed in 4e255d0b9 — the A→B→A hole is now closed for every observed switch, via an interim-identity latch at the fingerprint-read choke point rather than the per-child stamp:

  • Any fresh fingerprint read (status polls every few seconds per open dashboard tab, every turn gate, readiness probes) that observes a readable, different signed-in account than the baseline arms a sticky flag; identity_changed_since_sessions then reports changed until a sweep completes, even after the store switches back. The eager-spawn attack path in the finding implies an attached dashboard, whose polls sample the interim window.
  • The latch only ever triggers the existing retire-everything sweep, never selects victims — so observation races fail toward one extra recycle, never toward a spared stale child. It deliberately refuses to arm on component loss (unreadable CLI store, vanished vault component): those are indistinguishable from transient read failures, and a blip-armed sticky flag would force retire-until-complete sweeps on healthy hosts — re-creating the perpetual recycle loop this PR removes. 7 new tests pin both directions (3 fail without the latch; 4 refusal/invariant tests pass on both sides).
  • Remaining residual, narrowed and test-pinned: a round trip with no fresh read in between (headless gateway, no dashboard, no turns). Closing that fully still requires the per-child allocation stamp — kept as the documented follow-up because it must rewrite the sweep's retire-everything invariant across the session allocation/lifecycle files, which is disproportionate blast radius for this fix.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging 4e255d0b94f6ddcc56d4b38f31780cbe1b3c25a5. Adjudication produced no [GPT-ADJUDICATED] marker for 4e255d0: the adjudication stage FAILED TO RUN (its raw output is in the Adjudication details block -- typically an API or setup error), so no proportionality review took place and GPT's blocking verdict stands unreviewed.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/kiro_prerequisite.py:2750 -- Vault-only account round trips bypass session retirement
if baseline is None or not cli or fingerprint == baseline:
Vault A without CLI identity -> poll observes vault B but skips the latch -> eager spawn authenticates as B -> vault returns to A -> next turn reuses the B child.
Anchor: residual/security
Fix: Allow nonempty vault fingerprint changes to latch when both CLI components are absent.
[BLOCK-MERGE] 4e255d0
[GPT-REVIEWED] 4e255d0
False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4e255d0b94f6ddcc56d4b38f31780cbe1b3c25a5: <one-sentence reason>

@helenastafford
helenastafford force-pushed the fix/identity-sweep-baseline branch from 4e255d0 to d4b0732 Compare September 23, 2026 01:31
@helenastafford

Copy link
Copy Markdown
Contributor Author

Fixed in d4b0732e7 — the finding is legitimate: the blanket not cli refusal made the latch inert on vault-only hosts, so a vault A→B→A round trip there left B-authenticated children alive.

The fix keeps the blip-refusal invariant intact while closing the hole: under an absent CLI component, the latch now arms when the vault component independently proves a change — a nonempty vault that appears or differs from the baseline's cannot be a transient read failure (reads lose components under failure; they do not gain or alter them). An empty vault under an absent CLI remains refused (indistinguishable from a blip), so the sticky-latch-on-a-blip loop hazard is not reintroduced.

Red/green verified:

  • test_a_vault_only_round_trip_latches_with_no_cli_store — fails on the previous head's logic, passes now
  • test_a_vault_only_blip_never_latches — passes on both sides, pinning the refusal

127/127 logout-detection, 206 across affected suites, lint clean.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for d4b0732e7cc227fdc777256102924091a370f8ad; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d4b0732e7cc227fdc777256102924091a370f8ad: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@helenastafford
helenastafford force-pushed the fix/identity-sweep-baseline branch from d4b0732 to 64c3240 Compare September 23, 2026 02:40
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 23, 2026
@helenastafford
helenastafford force-pushed the fix/identity-sweep-baseline branch from 7647425 to 1ae410a Compare September 23, 2026 20:29
@helenastafford

Copy link
Copy Markdown
Contributor Author

The blocking finding on 76474251 (busy mismatched companion runtimes stay claimable) is fixed in 1ae410a01 — displacement, exactly as the finding proposed.

The finding was correct: the registry pass skipped a BUSY wrong-account runtime, but owner._subagent_runtimes and the _bg slot are precisely what get_subagent_runtime / background acquisition hand to NEW demuxed sessions, so "busy, catch it later" left the wrong-account process serving fresh sessions for its whole drain.

What changed — mismatched busy runtimes are now displaced out of the claimable slots and parked to drain:

  • Subagent runtimes (session_lifecycle.py, reaper predicate path): a proven-mismatch busy runtime is popped from the registry under its per-parent spawn lock (the lock is dropped with it, so an in-flight waiter re-checks the canonical map — the same dance release_subagent_runtime does, minus the process end) and parked on a new _draining_subagent_runtimes list. The next acquisition finds no entry and spawns a replacement under the live account. A reap at the top of the same reaper (which the per-turn gate and the sweep both call) ends a parked runtime only once it answers idle; probes fail toward busy, preserving work.
  • Background runtime: the predicate path now calls the existing _detach_bg_runtime_locked — the same primitive backend-switch displacement uses — freeing the _bg slot immediately and parking the busy runtime on _draining_bg_runtimes, where the existing drain reap already ends it once its handles finish.
  • No new process-end paths: displacement never ends a process; the only ends are the existing reap paths, gated on has_active_or_initializing_sessions() returning false. The parent-end ratchet's exempt set is unchanged, and its justifying fact (only idle processes are ever ended by this method) still holds — the docstring is updated to say so explicitly.
  • Lifecycle closure: parked runtimes are shielded from the orphan sweep (their PIDs join the active-set union) and torn down at close_all; the unfiltered identity sweep counts a parked wrong-account runtime as incompleteness, so a real account change still retries until the drain finishes.

Validation: 2 new tests red/green verified by stash-toggling only src/ — test_a_busy_mismatched_runtime_is_displaced_never_killed (fails pre-fix with the runtime still claimable and nothing flagged, the exact finding) and test_a_parked_displaced_runtime_is_reaped_once_it_drains (the park is a drain, not a leak). The prior busy-survival invariant is folded into the displacement test: never ended mid-turn, and a busy runtime with a MATCHING stamp is untouched. 152/152 logout-detection, 923 adjacent session-layer tests (session, cleanup, pool, background lifetime, idle recycle, start gate, boundaries, subagent, pid/reap suites), flake8/mypy/black/isort clean.

@helenastafford
helenastafford force-pushed the fix/identity-sweep-baseline branch 2 times, most recently from 309f98b to c1c6b7c Compare September 23, 2026 20:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@helenastafford
helenastafford force-pushed the fix/identity-sweep-baseline branch from c1c6b7c to e7fa74a Compare September 23, 2026 21:48
@helenastafford

Copy link
Copy Markdown
Contributor Author

All three blocking findings on c1c6b7c71 are addressed in e7fa74ab6 — F2 and F3 (upheld by the fence annotation) are fixed in code; F1 (machine-flagged as the documented edge case) remains the named protocol follow-up.

F2 — post-start identity read leaves the child unshielded from Windows orphan cleanup: fixed, at every site. The named remedy is implemented — the PID shield is hoisted above the stamp await — and the audit extended it beyond the flagged cold-start site: all six stamp sites (cold start, both companion-runtime spawns, both background spawns, the warm-pool fill) now add the child's PID to the facade's start-to-registration guard (_starting_pids, already unioned into the sweep's active set) before the read suspends, and discard it once real registration is visible or on teardown. A new shared spawn_pid helper extracts the PID for both provider and raw-runtime shapes. Pool-claimed providers keep their claim-to-registration shield (they leave _pool_pids at claim). Guards: a behavioral test capturing the shield's contents during the stamp read, plus a source scan pinning _starting_pids.add( ahead of every stamp await, so a refactor that drops one fails loudly.

F3 — mismatch retirement can end a companion runtime being claimed: fixed with both halves of the named remedy.

  1. Filtered retirement now detaches and parks, never same-pass: a proven-mismatch runtime — busy or idle alike — is displaced out of the claimable slot (registry pop / _detach_bg_runtime_locked(park_only=True), whose docstring already names this exact claim window) and parked on the drain list. The drain reap ends it only on a later pass, only once a park grace (30s) has elapsed, and only when its busy probe still answers idle.
  2. The idle probe no longer lies during the gate queue: create_session opens its init scope before the admission gate rather than after, so a claim queued behind the gate — the unbounded window the adjudication named — already reads as initializing to has_active_or_initializing_sessions, the predicate every displacement and recycle decision asks. A gate failure or cancellation in the wait closes the scope on the way out (two balance tests pin this). The handout-to-entry gap is synchronous on the event loop, and the remaining pre-gate awaits inside create_session are sub-second and bounded — two orders of magnitude inside the park grace.

F1 — agreeing store reads can mislabel a child's account: unchanged, deliberately. The fence annotation flagged it as the documented residual: it needs two full account changes strictly inside one multi-second spawn window with the child sampling the interim, a wrong stamp at that combination is exactly as inert as no stamp, and the closure (the child reports the identity it actually loaded) is a harness protocol change already carried as the named follow-up. The pre-drafted override rationale is in the review comment for a repository writer to verify and post.

Validation: 7 of the new/updated tests red/green verified by stash-toggling only src/ (same-pass park violations, the unshielded stamp read, the gate-queued claim reading idle); 176 across the regression file and start-gate suite, ~1,500 across the adjacent session-layer suites (session, pool, acp runtime, PID-sweep helpers, bg idle recycle, worker agent), comment-history gate enforcing-mode clean against the merge base, mypy/black/isort/flake8 clean. Single squashed commit rebased onto current main.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@helenastafford
helenastafford force-pushed the fix/identity-sweep-baseline branch from e7fa74a to acde797 Compare September 23, 2026 22:37
@helenastafford

Copy link
Copy Markdown
Contributor Author

All findings from both lanes are fixed in acde797b — the GPT finding on e7fa74ab plus the Opus verdict's three findings from c1c6b7c7 (that code was unchanged on the current head, so they were fixed proactively rather than waiting for the lane's re-run).

GPT F1 — reconciliation erasing a newer observation: fixed with a generation scope, as suggested. Every qualifying interim observation now bumps identity_observation_generation — including observations landing while the latch is already armed, since the previous already-armed early-return would have carried a mid-sweep observation on the pre-sweep generation. The turn gate captures the generation immediately after its identity_changed_since_sessions read and hands it back via note_sessions_reconciled(live, observations_before=...); the reconcile keeps the latch armed when any observation postdates that capture, so the next turn runs one more sweep that covers it. The capture is per-call, not instance state, so two overlapping turn gates cannot overwrite each other's baseline. Loop safety: the ordinary reconcile (no observation during the sweep) still clears the latch — pinned by a dedicated quiet-turns test — and a persistent third account converges in one extra sweep because observations equal to the advanced baseline stop qualifying.

Opus BLOCKING — flagged sessions keeping their resume sid: fixed exactly as suggested. flag_identity_stamp_mismatches now mirrors the sweep's clear_sid over its flagged keys, right after the lock block (clear_sid persists to disk and must not run under the registry lock). Without it, the eviction's get_or_create re-entry read the stale resume_sid and the replacement child issued session/load on the flagged account's conversation — and close_all's deliberate skip of retire-flagged sessions preserved the pointer across restart.

Opus FINDING — drain-list rebuild across the kill await: fixed as suggested. The reap now iterates a snapshot but removes only the entries it actually killed, so a runtime a concurrent turn parks during the await is never dropped from the only list still referencing it. (The bg twin keeps its rebuild: both its parks and its reap hold _bg_runtime_lock, so no concurrent mutation can interleave.)

Opus FINDING — spec not updated same-commit: fixed. docs/system-specs/modules/session.md now carries an "Account-identity retirement" section documenting the boot-seeded baseline, the generation-scoped interim latch, the spawn-identity stamps, the per-turn stamp gate with sid clearing, and the park/drain displacement machinery.

Red/green: 5 of the 6 new tests fail on pre-fix source (git stash toggle of src/ only) with the exact finding signatures; the sixth is the loop-safety invariant. 160/160 on the regression file, 1,154 across adjacent session suites, comment-history gate green in enforcing mode against the merge-base, mypy/flake8/black/isort clean.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 24, 2026 06:53

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving over the GPT BLOCKING, on the code rather than on the Opus downgrade.

The finding says clear_sid does synchronous persistence on the event loop. It does not: clear_sid mutates an in-memory dict under a threading.RLock and calls _save(), which on a running loop only sets _dirty and schedules _flush_async; the disk write happens in asyncio.to_thread with no lock held across the await. No sqlite, no busy_timeout, nothing that can stall the loop.

There are also six sibling on-loop clear_sid call sites (session_map.py 889/1401/1955, session_compaction.py:685, session_allocation.py 1877/2023); wrapping this one in to_thread would make it the odd one out for no gain. Every rejected branch of the boot-time seed falls back to today's unset-baseline behaviour, and the latch only makes retire stricter, so the direction is fail-safe throughout.

@bolichen97
bolichen97 merged commit c0dbcd7 into main Sep 24, 2026
99 of 109 checks passed
@bolichen97
bolichen97 deleted the fix/identity-sweep-baseline branch September 24, 2026 07:35
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 24, 2026
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…t-baseline sweep loop (kirodotdev#12956)

Co-authored-by: Helena Stafford <helenars@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants