Repository navigation
fix: seed identity sessions-baseline at boot to stop the unset-baseline sweep loop - #12956
Conversation
GPT 5.6 Review — ✅ no blocking findings (all downgraded on adjudication)GPT 5.6 flagged blocking issues on This comment is updated in place on each push. Review detailsBLOCKING -- src/kiro_crew/session_lifecycle.py:1262 -- synchronous persistent-store mutation runs on the gateway loop
Adjudication (Opus 5) — is blocking on each finding proportionate?LOW — the claimed mechanism (loop stalls on store IO) is not present: False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNo blocking violations; two advisory findings on the new identity-park and per-turn bg-lock paths. FINDING — src/kiro_crew/session_lifecycle.py:1565 — FINDING — src/kiro_crew/session_lifecycle.py:1623 — [OPUS-REVIEWED] acde797 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Two-thirds of the diff — stamps, parks, drain lists — chases a round trip "NO read ever observed", asserted only by a review pass, and concedes it stays open. Not justified as shipped
What this change shipsInventory (7 items) — 3 justifiedIntent: stop the boot-time retire→respawn loop that kills healthy long-running children when no identity ever changed — a FIX.
[FIRST-PRINCIPLES-REVIEWED] acde797 |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS A store whose fingerprint flaps between two nonempty values re-arms the sticky latch on every poll, reviving the retire/respawn churn this PR removes. Watch
Suggestions
[DESIGN-REVIEWED] acde797 |
802270e to
4e255d0
Compare
|
Addressed in
|
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/kiro_prerequisite.py:2750 -- Vault-only account round trips bypass session retirement |
4e255d0 to
d4b0732
Compare
|
Fixed in The fix keeps the blip-refusal invariant intact while closing the hole: under an absent CLI component, the latch now arms when the vault component independently proves a change — a nonempty vault that appears or differs from the baseline's cannot be a transient read failure (reads lose components under failure; they do not gain or alter them). An empty vault under an absent CLI remains refused (indistinguishable from a blip), so the sticky-latch-on-a-blip loop hazard is not reintroduced. Red/green verified:
127/127 logout-detection, 206 across affected suites, lint clean. |
GPT 5.6 Review —
|
d4b0732 to
64c3240
Compare
7647425 to
1ae410a
Compare
|
The blocking finding on The finding was correct: the registry pass skipped a BUSY wrong-account runtime, but What changed — mismatched busy runtimes are now displaced out of the claimable slots and parked to drain:
Validation: 2 new tests red/green verified by stash-toggling only |
309f98b to
c1c6b7c
Compare
c1c6b7c to
e7fa74a
Compare
|
All three blocking findings on F2 — post-start identity read leaves the child unshielded from Windows orphan cleanup: fixed, at every site. The named remedy is implemented — the PID shield is hoisted above the stamp await — and the audit extended it beyond the flagged cold-start site: all six stamp sites (cold start, both companion-runtime spawns, both background spawns, the warm-pool fill) now add the child's PID to the facade's start-to-registration guard ( F3 — mismatch retirement can end a companion runtime being claimed: fixed with both halves of the named remedy.
F1 — agreeing store reads can mislabel a child's account: unchanged, deliberately. The fence annotation flagged it as the documented residual: it needs two full account changes strictly inside one multi-second spawn window with the child sampling the interim, a wrong stamp at that combination is exactly as inert as no stamp, and the closure (the child reports the identity it actually loaded) is a harness protocol change already carried as the named follow-up. The pre-drafted override rationale is in the review comment for a repository writer to verify and post. Validation: 7 of the new/updated tests red/green verified by stash-toggling only |
…t-baseline sweep loop
e7fa74a to
acde797
Compare
|
All findings from both lanes are fixed in GPT F1 — reconciliation erasing a newer observation: fixed with a generation scope, as suggested. Every qualifying interim observation now bumps Opus BLOCKING — flagged sessions keeping their resume sid: fixed exactly as suggested. Opus FINDING — drain-list rebuild across the kill await: fixed as suggested. The reap now iterates a snapshot but removes only the entries it actually killed, so a runtime a concurrent turn parks during the await is never dropped from the only list still referencing it. (The bg twin keeps its rebuild: both its parks and its reap hold Opus FINDING — spec not updated same-commit: fixed. Red/green: 5 of the 6 new tests fail on pre-fix source ( |
bolichen97
left a comment
There was a problem hiding this comment.
Approving over the GPT BLOCKING, on the code rather than on the Opus downgrade.
The finding says clear_sid does synchronous persistence on the event loop. It does not: clear_sid mutates an in-memory dict under a threading.RLock and calls _save(), which on a running loop only sets _dirty and schedules _flush_async; the disk write happens in asyncio.to_thread with no lock held across the await. No sqlite, no busy_timeout, nothing that can stall the loop.
There are also six sibling on-loop clear_sid call sites (session_map.py 889/1401/1955, session_compaction.py:685, session_allocation.py 1877/2023); wrapping this one in to_thread would make it the odd one out for no gain. Every rejected branch of the boot-time seed falls back to today's unset-baseline behaviour, and the latch only makes retire stricter, so the direction is fail-safe throughout.
…t-baseline sweep loop (kirodotdev#12956) Co-authored-by: Helena Stafford <helenars@amazon.com>
Seed the identity sessions-baseline at gateway boot to stop the unset-baseline sweep loop
Problem
The identity-retirement sweep (
_retire_sessions_on_identity_change) compares the live auth-store fingerprint to a sessions-baseline that starts unset, and unset deliberately reads as "changed" (fail-safe). The baseline only advances after a fully quiescent sweep pass — nothing busy, nothing mid-start, no subagent-runtime lock held. On a live gateway that precondition is unsatisfiable: retiring idle sessions makes dashboard tabs eagerly respawn them, and those in-flight starts keep every subsequent pass incomplete. The result is an infinite retire→respawn loop that re-arms ~1 minute after every boot and kills healthy shared runtimes mid-prompt — long-running subagents (20–60 min) die deterministically while short ones slip between sweeps.No identity ever actually changes.
identity_fingerprintalready allowlists out all rotating fields (access_token,refresh_token,expires_at), andkiro-cli whoamiis stable across the loop. Captured end-to-end at INFO level on a live gateway (2026-09-22): repeatedKiro identity changed; retired N session(s) (incomplete, will retry next turn)with zero fingerprint change and zero store errors, retiring the same session five times in a loop.Prior art
SessionCleanup._expire_idlevs. this sweep) — complementary, no overlap.Fix
Seed the sessions-baseline at gateway boot, before any kiro-backed child can spawn:
kiro_prerequisite.py: newseed_sessions_baseline()adopts the on-disk account as the baseline. Refusals are all fail-safe toward the existing behavior (unset baseline → sweep): assume-ready/test mode, baseline already recorded, unreadable store, and a bounded store read (asyncio.wait_for, 5s) so a hung SQLite cannot stall boot. The read runs off-loop viaasyncio.to_threadand does not scale with user data.dashboard/server.py: both gateway start sites call the seed at boot, beforewarm_up()and before the socket serves.Because every child necessarily postdates the seed, the never-completing unset-baseline boot sweep becomes redundant. A real account change still compares unequal and sweeps exactly as before — the genuine-logout credential guarantee from #5904 is fully preserved, which is why this shape survives the busy-exemption objection recorded there.
A→B→A round trips: closed for every observed switch (interim-identity latch)
Addressing the GPT review's security finding: a bare baseline comparison cannot see a round trip (seed A → switch to B, a child spawns under B → switch back to A compares equal). Rather than bolting the per-child allocation stamp onto this PR (it would rewrite the sweep's documented retire-everything invariant across the most contended session files), the fix latches the observation at the fingerprint-read choke point:
current_identity_fingerprint— status polls (every few seconds per open dashboard tab), every turn gate, readiness probes. If any of them observes a readable, different signed-in account than the baseline, a sticky flag arms andidentity_changed_since_sessionsreports changed until a sweep completes, whatever the store says by then. The existing retire-everything sweep then recycles the interim children;note_sessions_reconciledclears the flag.Per-child spawn-identity stamp (closes the unobserved round trip): every kiro-backed provider now records the account the store held as its process started (
read_spawn_identity→stamp_spawn_identity, wired at all five spawn sites: cold start, shared run runtime, warm-pool fill, background create/recycle — first-stamp-wins so a pool claim keeps its fill-time record). The turn gate compares those records against its fresh read (flag_identity_stamp_mismatches) before the unchanged early-return, so a child from an A→B→A round trip that no read ever observed is still retired from its own record — the one case a gateway-wide baseline is inherently blind to. Loop-safe by construction: flag-only (the existingretire_on_identity_changeeviction path recycles it; nothing global or sticky), component-proven mismatches only (a component counts only when nonempty on both sides — reads lose components under failure, never gain or alter them — so a blip at spawn or at the gate can never flag a healthy child), and an unstamped child (unwired entrypoint, failed read) keeps exactly the pre-stamping protections. The gateway-wide baseline test still pins that the baseline stays honestly blind to the unobserved round trip; the child is covered per-record.Spawn-window hardening (review round 4): every spawn is now bracketed by a pre-start and post-start identity read, and the stamp is recorded only when both agree (
pre_spawn_identity→provider.start()→stamp_spawn_identity). A single post-start read could label a child that authenticated as B with the account the store switched back to; disagreement now refuses the stamp (fail-safe: unstamped children keep the pre-stamping protections), and either read observing an interim account feeds the latch through the ordinary read path. The turn gate's mismatch pass also covers the companion runtime registries (_bg_runtime,_subagent_runtimes): those runtimes carry the same spawn stamp but never appear in_sessions, so a proven-mismatch runtime — busy or idle alike — is displaced: popped from the claimable slot (a replacement spawns under the live account) and parked on a drain list, where a later pass ends it only once it is idle past a park grace. Same-pass kills are never issued: the idle probe races a claim the runtime was just handed, which is also whycreate_sessionnow opens its init scope before the admission gate — a gate-queued claim already reads as initializing.Spawn-window hardening (review round 6): every stamp read now runs under the start-to-registration PID shield (
_starting_pids, unioned into the orphan sweep's active set): the read is a bounded suspension point afterstart()published the PID but before any registry the sweep consults can see the child, and on Windows the sweep has no age grace — a tick landing in that await killed the healthy child. All six spawn sites shield before the read and release once registration is visible (or on teardown), pinned by a behavioral capture test and a source scan.Reconcile and eviction hardening (review round 7): interim observations are generation-scoped: every qualifying read bumps a counter (even while the latch is armed), the turn gate captures it right after its initiating read, and
note_sessions_reconciledkeeps the latch armed when any observation postdates that capture — so a store flap observed while the sweep ran can never be erased by the sweep's own completion, and the next turn covers it with one more sweep. The stamp gate now also mirrors the sweep'sclear_sidover flagged sessions (the eviction re-entry otherwisesession/loaded the flagged account's conversation, andclose_all's deliberate skip preserved the stale pointer across restart), the companion drain reap removes only entries it actually killed instead of rebuilding the list across its kill await (a rebuild dropped a concurrently parked runtime from its only remaining reference), anddocs/system-specs/modules/session.mdgains the owning "Account-identity retirement" section per the same-commit spec rule.Remaining residual: a switch and return that both complete strictly between the two bracket reads of one spawn, with the child sampling the interim — closing it fully needs the child to report the identity it actually loaded (a protocol change; documented on
stamp_spawn_identityas the named follow-up).Validation
src/): pre-fix, the new regression class fails 9/10 — the one pre-fix pass,test_an_unseeded_busy_gateway_re_sweeps_every_turn, reproduces the production loop through the real_retire_sessions_on_identity_change(5 turns → 5 sweeps → 5 healthy children recycled, never complete). Post-fix: 10/10, including "a switch/sign-out after seeding is still detected" and a source-scan asserting both start sites seed (an upstream refactor that drops one fails loudly).test_external_logout_detection.py: 118/118 (117 existing + 1 new hung-store timeout test). Adjacent suites (session, KAS login/host-auth, service boundaries, session-duration metrics): 385 pass.KiroPrerequisiteServicemock upgraded toAsyncMockfor the new bootawait(included).Identity baseline seeded at startup; boot sweep skippedand serves healthy; a fresh unset-baseline boot on unfixed code re-arms the loop within a minute.TestSpawnIdentityStamptests fail pre-fix and pass post-fix (verified by stash-toggling onlysrc/), including the end-to-end unobserved-round-trip flag, the healthy-host/blip/unstamped refusals, first-stamp-wins, the shared-runtime fallback, and source-scans pinning all five spawn sites plus both startup wirings. Full file now 138/138; adjacent suites 477 + 151 (pool, background lifetime, start gate, cleanup) pass.Pattern harvest
Rule candidate: review-prompt — flag any safety sweep whose corrective action re-creates its own trigger condition (here: retire → eager respawn → next pass incomplete), i.e. a completion precondition that requires global quiescence on a system whose remediation prevents quiescence.
A fail-safe default ("unset means changed") combined with a completion precondition that the fail-safe's own remediation invalidates (retire → eager respawn → next pass incomplete) turns a one-shot safety sweep into a permanent loop. When a safety check's trigger state can be re-created by its own corrective action, the trigger must be resolvable without requiring global quiescence — here, by establishing the baseline before any actor exists that could block reconciliation.