Repository navigation
fix(session): spare an idle parent with running subagents from the identity sweep (#17360) - #18055
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (6 items) — 4 justifiedIntent: keep a user's running
[FIRST-PRINCIPLES-REVIEWED] 4ea98cd Description read[DESCRIPTION-READ] e726b293082e2b903106ab6569ca0d86964ad4a424c84236b7250ad1746bc52a That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The fix adds a two-phase teardown-fence protocol and seven new Watch
[DESIGN-REVIEWED] 4ea98cd Description read[DESCRIPTION-READ] 1354860af24a724df037e1abd58691a5e8dc3019dc72ef9dabca6f731ee68931 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
018b763 to
6d7b740
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
This PR stops the account-identity sweep from retiring an idle parent whose spawn_run children are still running. It adds a side-effect-free has_live_or_queued_children probe and defers that parent the same way a busy one is deferred. That fixes the "provider shutdown" kills in #17360, but the identity sweep is also the sign-out path, and the deferral now applies there too. dashboard/handlers/kas_login.py:296 (_retire_runtimes_after_sign_out) calls retire_kiro_identity_sessions() with no fingerprint after a Crew sign-out. At session_lifecycle.py:2098 the new if self._parent_has_live_children(key): ... continue then spares the parent, and its children keep running on the signed-out account's in-memory credential until they finish. Before this PR they were cancelled. That is a real identity/credential regression. The Design lane raised it on head 6d7b740 and it has no author disposition. The same lane notes docs/system-specs/modules/session.md § Account-identity retirement does not describe the new (subagents) deferral. Suggested fix: add a keyword such as spare_children: bool = False to retire_kiro_identity_sessions. Pass True only from the per-turn chat_runner._retire_sessions_on_identity_change sweep, and keep sign-out on the cancel path. Add a test pinning that sign-out still cancels an idle parent's children, and update session.md. The First Principles duplicate-filter point (has_live_or_queued_children_impl copying snapshot_teardown_children_impl's predicates) is worth folding into one shared selection while you are there. Otherwise the probe is pure, and the new tests prove the spare and the childless-retire cases. — crew-pr-reviewer
6d7b740 to
ed15711
Compare
|
Thanks — this is a real regression and it's fixed in The identity sweep is also the sign-out path:
Tests added/updated in
Design lane: First Principles note (duplicate selection): the pure probe mirrors the live/queued selection the teardown snapshot uses. I kept them as two methods rather than folding them into one shared selection: the probe must stay side-effect-free, and the snapshot takes the live-and-queued run ids plus the parked/undelivered handling for teardown, so a single shared helper would either leak the teardown's selection shape into the pure read or re-introduce the coupling this fix removed. Happy to fold them in a follow-up if you'd prefer one selection with a |
ed15711 to
eec91fd
Compare
|
All three review lanes converge on the same real finding, and it's fixed in The finding (GPT 6.1, Opus 5.5, and First Principles item 5): an external Fix: the per-turn caller now passes First Principles blocker (deleted pin): I restored Tests added/changed:
Docs: First Principles item 2 (duplicate live/queued selection): I kept the pure probe and the teardown snapshot as two methods. The probe must stay side-effect-free, while the snapshot also arms the delivery gate and handles parked/undelivered children for teardown; a single shared helper would either leak the teardown's selection shape into the pure read or re-introduce the coupling this change removed. Happy to fold them in a follow-up with a |
5ad5904 to
2c204ce
Compare
|
Addressed in
A new test asserts that a turn starting during the read defers (it fails without the semaphore re-check). mypy, flake8, black and the docs lint are clean, and the related session/overload/logout/session-duration suites pass. Rebased onto current green |
|
Both points fixed in the staged head (will land on the next push).
No design decision is being overridden here — both were real issues in the rework, so they are fixed rather than argued. |
2c204ce to
e525f18
Compare
|
Fixed in A child admitted during the off-loop store read is no longer cancelled. A new test drives exactly this: a child admitted into the fence during the store read defers the parent, and it fails without the re-check. This head also folds the two First Principles fixes (the dead |
|
Both fixed in
A new test drives the tentative fence directly: a recorded admission is visible to mypy, flake8 and black are clean, and the related session/overload/logout/session-duration/queue-depth suites pass. |
|
Both findings are fixed in F1 — the final child re-check and the committed-fence open are now atomic. The sweep's last decision went through a new F2 — a cancelled sweep no longer leaks a provider or leaves a false New tests: |
|
All three items are addressed in Item 3 — duplicate store read removed. Item 7 — Item 4 — the PR description now matches shipped behavior. The
|
|
Both findings are fixed in F1 — cancellation no longer skips a committed retirement's teardown. The committed retirements are no longer drained in a loop after the permit barrier; they are collected into F2 — the end record is bound to its own retirement, under the lock. Each committed retirement records its end with New test: |
|
Fixed in Both the synchronously-retired parents (
Each per-key teardown is individually New test: |
|
This PR stops the account-identity sweep from retiring an idle parent whose spawn_run children are still running, which was the cause of the "provider shutdown" kills in #17360. iamwhatever's earlier request is addressed on head b8f0623. |
iamwhatever
left a comment
There was a problem hiding this comment.
This stops the per-turn identity sweep from cancelling an idle parent's live spawn_run children. Our earlier change request is met on b8f0623. Sign-out still cancels (spare_children defaults to False and is pinned by test_signout_retires_an_idle_parent_with_live_children), an empty or partially dropped credential still cancels (identity_component_dropped), and session.md documents the deferral. I tried to break the fence protocol (an admission between open and commit, a successor registering during the off-loop read, a turn taking the semaphore mid-read) and each path defers or commits safely. Remaining defect: a deferred parent leaves the sweep incomplete for as long as its children run. Every chat turn then re-enters retire_kiro_identity_sessions, which holds owner._start_sem.drain() (all cold-start permits) across await self._parent_has_sparable_child(key), an off-loop task-store read, for each deferred parent (session_lifecycle.py, the maybe_spare loop). On a host with an hours-long subagent after an account switch, every cold start on every chat waits behind that drain every turn. Before this PR the sweep finished once. Suggested fix, as the Design lane proposes: under spare_children, retire the parent the way _evict_stale_session (session_allocation.py ~1178) does. Shut the provider down without _cancel_parent_children, so the sweep completes and the fence, probe and commit protocol (seven new _ChildTeardownHandler methods) can be removed. If you keep the current design, bound and document the deferral cost in session.md. Either way, rewrite the description: it still describes a one-predicate design with "no new dependency" and lists empty fingerprints as a non-goal. — crew-pr-reviewer (deep re-check)
|
Both points from the 19:57Z review are answered on Point 1 — the per-turn drain cost of an incomplete deferral. Confirmed real: a deferred parent keeps the sweep incomplete, the pending fingerprint is not cleared, and so the per-turn gate re-runs I am keeping the current design rather than taking the suggested rework to retire via On why not the Point 2 — the description. Rewritten to match the shipped diff. It no longer describes a single pure predicate or claims "no new dependency": it now describes the teardown-fence decision, the off-loop |
iamwhatever
left a comment
There was a problem hiding this comment.
This PR stops the per-turn identity sweep from cancelling the live spawn_run children of an idle parent. It does that by deciding spare-or-retire at the async teardown, under a tentative admission fence, and the root cause in #17360 is fixed. Point 1 of our 19:57Z review is met on c905358. _parent_has_sparable_child returns early on the synchronous in-memory check, so a long-running live child never reaches the store read inside _start_sem.drain(). Only a durable-only row pays for that read, and session.md now documents the cost. Point 2, the description, has been rewritten. I tried to break the fence by admitting a child between open and promote, by registering a successor during the read, and by taking the semaphore mid-read. Each case defers or commits safely, because commit_identity_sweep_retire promotes the same set under _teardown_fence_lock and the snapshot follows it with no await. One real defect remains in the new cancellation-safe drain, and it is the Opus finding on this head, which has no disposition yet. In the method-level finally of retire_kiro_identity_sessions (session_lifecycle.py ~2405-2422), except asyncio.CancelledError: raise leaves the for key, provider in (*doomed, *committed_teardown) loop after the first entry. Every later entry is already unregistered and recorded as ended, but it never gets shutdown(), _cancel_parent_children or release_subagent_runtime. Those children keep running on the old credential, which is exactly what this finally says it prevents. Fix: record the CancelledError, finish draining each remaining entry with every step shielded, re-raise after the loop, and add a test that cancels during the first of two shutdowns. Also, session.md (~697) says the probe is a limit=1 read, but taskq_pending_ids_for_async calls store.list_pending with no limit (the GPT finding). Either pass a limit or correct the doc. — crew-pr-reviewer (deep re-check)
|
Re: review 2026-10-11 — both points addressed; current head is 1. CancelledError in the shutdown drain abandoned later entries. Fixed. In the method-level 2. session.md said the probe is a Rebase note. The rebase onto current Local gate green on the committed head: |
…entity sweep (#17360) The account-identity sweep retires every idle kiro-backed session, and a parent that ended its turn with spawn_run children still running is idle by the semaphore test. So its retirement cancelled those children ("provider shutdown") on every turn any chat took while the sweep stayed incomplete. Before retiring an idle session, snapshot its children with the existing lock-held _snapshot_parent_children and, if any are live, defer the parent with retire_on_identity_change (the same treatment a mid-turn session gets) and record it in identity_sweep_waiting_on. The children then outlive the turn and the parent is recycled only once it is genuinely idle. A live-account holder is already spared earlier, so this only waits on the right-account parent's own children.
…sion-safe The per-turn identity sweep retired an idle parent that still had running spawn_run children, cancelling them as a provider shutdown every turn. Sparing such a parent needs to know whether it has any child -- live, queued, or held only by the task store -- and that answer kept racing: a sequence of preflight read-then-decide designs each left a window (a sync read freezing the loop, a failed read, a busy parent unlocking mid-await, a stale baseline, a stale negative probe) where the decision disagreed with the admission path. The decision moves to the point of action. There is no preflight probe. Under the registry lock a spare_children candidate is NOT retired; its teardown fence is opened (the same _teardown_fence_lock the admission path records a new child into, before its store write) and it is left registered -- nothing irreversible happens for it under the lock. At the async teardown, off the loop, the sweep asks whether the parent still has a child: live or queued in memory, admitted into the fence since it opened, or a pre-existing durable row in the store. If so -- or if the store read fails -- it defers, cancels nothing, and the next sweep re-checks. Only a childless parent is retired, under a re-taken lock that leaves a successor registered under the key alone. The one synchronous exception stays: a parent whose OWN spawn stamp lost a credential component (identity_component_dropped vs spawn_identity_of) is retired and its children cancelled, the fail-safe for an external logout. The sync preflight probe, has_pending_store_children on the probe path, and the probed-key bookkeeping are removed (net simpler). chat_runner keeps the baseline identity_component_dropped check for the simple logout. Tests: durable-only child, failed read, spawn-stamp drop (cancels), whole spawn stamp (spares), a child admitted into the fence during the teardown, and a pre-existing durable child each as a regression.
|
CI follow-up — new head The branch was 67 commits behind
Fixed both in the test doubles only — Local gate green on the committed head: |
iamwhatever
left a comment
There was a problem hiding this comment.
This PR stops the per-turn account-identity sweep from cancelling the live spawn_run children of an idle parent, which was the root cause of the "provider shutdown" kills in #17360. It decides spare-or-retire at the async teardown under a tentative admission fence. Both points of our 06:03Z change request are met on 4ea98cd. In the method-level finally of retire_kiro_identity_sessions (session_lifecycle.py), the drain loop now records the first CancelledError, keeps shutting down, cancelling children and releasing the runtime for every later entry (each step shielded), and re-raises after the loop. test_identity_retire_cancel_during_first_drain_still_drains_second pins that: A's shutdown raises, and B is still shut down with its children drained. session.md no longer claims a limit=1 probe. The author's reason for keeping the full read is sound, because taskq_pending_ids_for_async also serves the stop-all cascade. I compared the PR-only diff at the old and new heads: the rebase added only test-double fixes in test/test_cron_identity_gate.py. spare_children=True is still passed only from chat_runner._retire_sessions_on_identity_change, and only when live is non-empty and identity_component_dropped(live) is false. So sign-out, external logout and partial credential loss all still cancel. In earlier rounds I tried to break the fence (an admission between open and promote, a successor registering during the off-loop read, a turn taking the semaphore mid-read), and each case deferred or committed safely. This round I tried a cancel during the first of two drains, which the new loop handles. Checks are green, and this approval replaces our earlier CHANGES_REQUESTED. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: Stop the account-identity sweep from killing a session's running sub-agents when that session is only idle because it is waiting for them.
With more than one dashboard session active, background sub-agents started with
spawn_runare killed mid-prompt and surface asAcpProcessDied ... killed (provider shutdown), even though the account never changed. The identity sweep retires every idle kiro-backed session — and a parent that ended its turn withspawn_runchildren still running is idle by the semaphore test. Retiring it tears those children down.Why it matters
A parent waiting on a
spawn_runcompletion is exactly the kind of session the sweep classifies as idle, so a routine action in any other chat (a message, a new chat, a monitor cycle, a cron/digest turn) retires it and cancels its children. While any session stays busy the sweep never completes, so this repeats on every later turn from any chat. Users lose in-flight sub-agent work and read the genericprovider shutdownas model or provider instability.Why this change
Reported by #17360: a user's
spawn_runsub-agents are cancelled asprovider shutdownon turns taken in other chats, with no account change. The need is common on any gateway running more than one dashboard chat with background sub-agents — exactly the fan-out the sub-agent tools exist for.Who else is affected: the identity sweep is also the account-switch and (via
_retire_runtimes_after_sign_out) the sign-out path, so the deferral had to be scoped so it never spares a parent whose account is actually gone. It does not: the deferral is opt-in (spare_children) and applies only to the per-turn identity-change sweep, and only when the live credential is whole; sign-out, an external logout, and a partial credential loss all keep the cancel-the-children behaviour. No default, UI, or output changes for users who never hit the bug — a healthy single-account host never defers.Reviewer note: this changes behaviour on the per-turn identity-change sweep only — an idle parent with still-running children is now deferred (recycled later, once childless or on its own next turn) instead of retired-and-cancelled on an unrelated turn. Sign-out and logout behaviour is unchanged.
Not a goal
semaphore.locked()) or a live-account (spawned_under) session is handled — those spares are untouched.kiro-cli logout, including one that leaves the Crew vault populated — is NOT a defer: the parent and its children are retired and cancelled, the same as sign-out, because the credential the children loaded is gone. Only a whole switch between two real accounts defers a parent with children.session.md).What changed
The fix is a deferral decided at the ASYNC teardown of the per-turn sweep, under a teardown fence, so a child that arrives while the decision runs is seen rather than raced.
retire_kiro_identity_sessions(src/kiro_crew/session_lifecycle.py) now takesspare_children: bool = False:spare_childrencandidate (an idle parent whose stamp does not match the live account) is NOT retired. Its teardown fence is opened — the same_teardown_fence_lockthe admission path records a new child into before its store write — and the session is left registered, so nothing irreversible happens for it under the lock. The one exception decided synchronously is a parent whose OWN spawn stamp lost a credential component (identity_component_dropped): that is the identity its children loaded, so it is retired and its children cancelled — the fail-safe for a partial logout the baseline missed._parent_has_sparable_child: a child live or queued in memory, admitted into the fence since it opened, or a pre-existing durable-only row in the task store (has_pending_store_children, alimit=1existence read). Any child — or a store read that FAILS — defers the parent (fail-safe), drops its fence, and leaves the sweep incomplete so the next sweep re-checks.commit_identity_sweep_retire) so a concurrent/api/spawnis carried into the cancel'saccepted_sincerather than cancelled. A successor registered under the key during the read is left alone; a turn that took the semaphore during the read defers.finally, each per-key teardownasyncio.shielded, so the chat closing mid-sweep cannot leave an already-unregistered parent's children running on a logged-out credential.The
chat_runnerper-turn caller (_retire_sessions_on_identity_change) passesspare_children = bool(live) and not service.identity_component_dropped(live): an empty live fingerprint (external logout / unreadable store) and a dropped baseline component both keep the cancel path. Sign-out (_retire_runtimes_after_sign_out) sweeps with no fingerprint and the defaultspare_children=False, so it keeps cancelling.The deferral uses a pure probe, never the teardown
snapshot_teardown_children(which arms the delivery gate, clearspending_followupsand cancels follow-up watchers) — calling that on a spared parent would drop its children's completion delivery.New public surface:
has_live_or_queued_children,has_pending_store_children,open_teardown_fence,has_fenced_children,release_teardown_fence,commit_identity_sweep_retire,snapshot_teardown_children_onlyonSubagentManager(impls insubagent_manager/cancellation.py);identity_component_droppedon the prerequisite service (kiro_prerequisite.py);spare_childrenonretire_kiro_identity_sessions.Cost of an incomplete deferral
A deferred parent keeps the sweep incomplete, so the per-turn gate re-runs the sweep on the next turn from any chat and re-checks that parent, inside the start-permit barrier (
_start_sem.drain()), awaiting the off-loop store read while the barrier is held. The deferral is intended to be short-lived relative to aspawn_runchild, is NOT timeout-bounded, and a long-running child keeps the parent deferred (and re-checked each turn) for its whole life. This is accepted because the alternative is cancelling the children on every turn, not deferring on every turn; the per-check cost is kept small (synchronous in-memory/fence checks, alimit=1store read on the writer thread). Documented indocs/system-specs/modules/session.md§ Account-identity retirement.Backwards compatibility
Removes nothing: no field, kind, key, validator, or public function is removed, renamed, or tightened. All of the new surface above is additive. The only behavioural change is on the per-turn identity-change sweep: an idle parent with live children is deferred instead of retired-and-cancelled, reaching the same end-state a mid-turn session already does. Sign-out, external logout, and partial-credential-loss paths are unchanged (they cancel).
Tests
In
test/test_session.py:test_identity_retire_spares_an_idle_parent_with_children,test_identity_retire_spares_a_parent_with_only_a_store_child,test_identity_retire_defers_on_a_pre_existing_durable_child: live/queued and durable-only children defer the parent.test_identity_retire_defers_when_the_store_read_fails: a failed store read defers (fail-safe).test_identity_retire_defers_on_a_child_admitted_during_the_teardown,test_identity_retire_defers_a_child_admitted_during_the_read,test_identity_retire_defers_a_turn_that_starts_during_the_read: the fence/semaphore re-checks defer on a race.test_identity_retire_cancels_when_a_spawn_stamp_source_dropped/test_identity_retire_still_spares_when_the_spawn_stamp_is_whole: the spawn-stamp drop gate.test_signout_retires_an_idle_parent_with_live_children: sign-out (defaultspare_children=False) still cancels.test_identity_retire_still_retires_an_idle_childless_parent,test_a_failing_retire_shutdown_warns_and_leaves_the_key: childless retire and its failing-shutdown-warning contract.test_identity_retire_cancel_mid_teardown_still_drains_committed/..._stamp_dropped: cancelling the sweep mid-teardown still drains a committed / stamp-dropped parent.test_has_live_or_queued_children_is_a_pure_live_queued_probe: the in-memory probe is pure.In
test/test_overload_integration_glue.py:test_has_pending_store_children_sees_a_durable_only_waiter,test_identity_sweep_fence_records_but_does_not_suppress_delivery,test_commit_identity_sweep_retire_is_atomic_against_admission.In
test/test_external_logout_detection.py:test_identity_component_dropped_detects_a_partial_logout,test_identity_component_added_is_not_a_drop,test_identity_component_dropped_is_false_with_no_baseline.In
test/metrics/test_session_duration.py: the retire path records onlyEND_REASON_RETIREDacross its two commit points.Manual verification
N/A — unit coverage sufficient. The repro drives the sweep through the public
retire_kiro_identity_sessionsentry point with a child-teardown handler, which is the user-visible path (the sweep runs at the start of every turn).Screenshots / video
N/A — no user-visible frontend surface is touched; this is a backend session-lifecycle change.
Related Issues
Fixes #17360
Pattern harvest
Rule candidate: review-prompt
Pattern: a read used for a decision must be side-effect-free; and when the decision is acted on later, the read goes stale — move the decision to the point of action (here, under the fence lock at the async teardown) rather than probing earlier and committing on the strength of a read a race could have invalidated.