Skip to content

fix(session): spare an idle parent with running subagents from the identity sweep (#17360) - #18055

Merged
iamwhatever merged 2 commits into
mainfrom
fix/17360-spare-idle-parent-with-children
Oct 11, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
fix/17360-spare-idle-parent-with-children

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop the account-identity sweep from killing a session's running sub-agents when that session is only idle because it is waiting for them.

With more than one dashboard session active, background sub-agents started with spawn_run are killed mid-prompt and surface as AcpProcessDied ... killed (provider shutdown), even though the account never changed. The identity sweep retires every idle kiro-backed session — and a parent that ended its turn with spawn_run children still running is idle by the semaphore test. Retiring it tears those children down.

Why it matters

A parent waiting on a spawn_run completion is exactly the kind of session the sweep classifies as idle, so a routine action in any other chat (a message, a new chat, a monitor cycle, a cron/digest turn) retires it and cancels its children. While any session stays busy the sweep never completes, so this repeats on every later turn from any chat. Users lose in-flight sub-agent work and read the generic provider shutdown as model or provider instability.

Why this change

Reported by #17360: a user's spawn_run sub-agents are cancelled as provider shutdown on turns taken in other chats, with no account change. The need is common on any gateway running more than one dashboard chat with background sub-agents — exactly the fan-out the sub-agent tools exist for.

Who else is affected: the identity sweep is also the account-switch and (via _retire_runtimes_after_sign_out) the sign-out path, so the deferral had to be scoped so it never spares a parent whose account is actually gone. It does not: the deferral is opt-in (spare_children) and applies only to the per-turn identity-change sweep, and only when the live credential is whole; sign-out, an external logout, and a partial credential loss all keep the cancel-the-children behaviour. No default, UI, or output changes for users who never hit the bug — a healthy single-account host never defers.

Reviewer note: this changes behaviour on the per-turn identity-change sweep only — an idle parent with still-running children is now deferred (recycled later, once childless or on its own next turn) instead of retired-and-cancelled on an unrelated turn. Sign-out and logout behaviour is unchanged.

Not a goal

  • Changing how a mid-turn (semaphore.locked()) or a live-account (spawned_under) session is handled — those spares are untouched.
  • Sparing a parent whose account is gone. An empty or partial live fingerprint — an external kiro-cli logout, including one that leaves the Crew vault populated — is NOT a defer: the parent and its children are retired and cancelled, the same as sign-out, because the credential the children loaded is gone. Only a whole switch between two real accounts defers a parent with children.
  • Bounding a deferral by a timeout. A parent with a long-running child stays deferred (and re-checked each turn) until the child finishes or the parent takes a turn; the sweep does not force the account change for that key while children run. The per-turn cost of that is documented (see "Cost of an incomplete deferral" below and in session.md).

What changed

The fix is a deferral decided at the ASYNC teardown of the per-turn sweep, under a teardown fence, so a child that arrives while the decision runs is seen rather than raced. retire_kiro_identity_sessions (src/kiro_crew/session_lifecycle.py) now takes spare_children: bool = False:

  • Under the registry lock, a spare_children candidate (an idle parent whose stamp does not match the live account) is NOT retired. Its teardown fence is opened — the same _teardown_fence_lock the admission path records a new child into before its store write — and the session is left registered, so nothing irreversible happens for it under the lock. The one exception decided synchronously is a parent whose OWN spawn stamp lost a credential component (identity_component_dropped): that is the identity its children loaded, so it is retired and its children cancelled — the fail-safe for a partial logout the baseline missed.
  • Off the loop (on the writer thread, still inside the start-permit barrier so no successor can publish a sid), the sweep asks _parent_has_sparable_child: a child live or queued in memory, admitted into the fence since it opened, or a pre-existing durable-only row in the task store (has_pending_store_children, a limit=1 existence read). Any child — or a store read that FAILS — defers the parent (fail-safe), drops its fence, and leaves the sweep incomplete so the next sweep re-checks.
  • Commit happens only for a childless parent, under a re-taken registry lock that re-checks the session identity, the semaphore, and the in-memory/fence children, then atomically promotes the tentative fence to the committed teardown fence (commit_identity_sweep_retire) so a concurrent /api/spawn is carried into the cancel's accepted_since rather than cancelled. A successor registered under the key during the read is left alone; a turn that took the semaphore during the read defers.
  • Teardown drain of the committed retirements (and the synchronously stamp-dropped ones) runs in a cancellation-safe, lock-free method-level finally, each per-key teardown asyncio.shielded, so the chat closing mid-sweep cannot leave an already-unregistered parent's children running on a logged-out credential.

The chat_runner per-turn caller (_retire_sessions_on_identity_change) passes spare_children = bool(live) and not service.identity_component_dropped(live): an empty live fingerprint (external logout / unreadable store) and a dropped baseline component both keep the cancel path. Sign-out (_retire_runtimes_after_sign_out) sweeps with no fingerprint and the default spare_children=False, so it keeps cancelling.

The deferral uses a pure probe, never the teardown snapshot_teardown_children (which arms the delivery gate, clears pending_followups and cancels follow-up watchers) — calling that on a spared parent would drop its children's completion delivery.

New public surface: has_live_or_queued_children, has_pending_store_children, open_teardown_fence, has_fenced_children, release_teardown_fence, commit_identity_sweep_retire, snapshot_teardown_children_only on SubagentManager (impls in subagent_manager/cancellation.py); identity_component_dropped on the prerequisite service (kiro_prerequisite.py); spare_children on retire_kiro_identity_sessions.

Cost of an incomplete deferral

A deferred parent keeps the sweep incomplete, so the per-turn gate re-runs the sweep on the next turn from any chat and re-checks that parent, inside the start-permit barrier (_start_sem.drain()), awaiting the off-loop store read while the barrier is held. The deferral is intended to be short-lived relative to a spawn_run child, is NOT timeout-bounded, and a long-running child keeps the parent deferred (and re-checked each turn) for its whole life. This is accepted because the alternative is cancelling the children on every turn, not deferring on every turn; the per-check cost is kept small (synchronous in-memory/fence checks, a limit=1 store read on the writer thread). Documented in docs/system-specs/modules/session.md § Account-identity retirement.

Backwards compatibility

Removes nothing: no field, kind, key, validator, or public function is removed, renamed, or tightened. All of the new surface above is additive. The only behavioural change is on the per-turn identity-change sweep: an idle parent with live children is deferred instead of retired-and-cancelled, reaching the same end-state a mid-turn session already does. Sign-out, external logout, and partial-credential-loss paths are unchanged (they cancel).

Tests

In test/test_session.py:

  • test_identity_retire_spares_an_idle_parent_with_children, test_identity_retire_spares_a_parent_with_only_a_store_child, test_identity_retire_defers_on_a_pre_existing_durable_child: live/queued and durable-only children defer the parent.
  • test_identity_retire_defers_when_the_store_read_fails: a failed store read defers (fail-safe).
  • test_identity_retire_defers_on_a_child_admitted_during_the_teardown, test_identity_retire_defers_a_child_admitted_during_the_read, test_identity_retire_defers_a_turn_that_starts_during_the_read: the fence/semaphore re-checks defer on a race.
  • test_identity_retire_cancels_when_a_spawn_stamp_source_dropped / test_identity_retire_still_spares_when_the_spawn_stamp_is_whole: the spawn-stamp drop gate.
  • test_signout_retires_an_idle_parent_with_live_children: sign-out (default spare_children=False) still cancels.
  • test_identity_retire_still_retires_an_idle_childless_parent, test_a_failing_retire_shutdown_warns_and_leaves_the_key: childless retire and its failing-shutdown-warning contract.
  • test_identity_retire_cancel_mid_teardown_still_drains_committed / ..._stamp_dropped: cancelling the sweep mid-teardown still drains a committed / stamp-dropped parent.
  • test_has_live_or_queued_children_is_a_pure_live_queued_probe: the in-memory probe is pure.

In test/test_overload_integration_glue.py: test_has_pending_store_children_sees_a_durable_only_waiter, test_identity_sweep_fence_records_but_does_not_suppress_delivery, test_commit_identity_sweep_retire_is_atomic_against_admission.

In test/test_external_logout_detection.py: test_identity_component_dropped_detects_a_partial_logout, test_identity_component_added_is_not_a_drop, test_identity_component_dropped_is_false_with_no_baseline.

In test/metrics/test_session_duration.py: the retire path records only END_REASON_RETIRED across its two commit points.

Manual verification

N/A — unit coverage sufficient. The repro drives the sweep through the public retire_kiro_identity_sessions entry point with a child-teardown handler, which is the user-visible path (the sweep runs at the start of every turn).

Screenshots / video

N/A — no user-visible frontend surface is touched; this is a backend session-lifecycle change.

Related Issues

Fixes #17360

Pattern harvest

Rule candidate: review-prompt
Pattern: a read used for a decision must be side-effect-free; and when the decision is acted on later, the read goes stale — move the decision to the point of action (here, under the fence lock at the async teardown) rather than probing earlier and committing on the strength of a read a race could have invalidated.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 4ea98cd02e022048c039d70bf3345f0a45694f2c touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 4ea98cd02e022048c039d70bf3345f0a45694f2c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

has_fenced_children does the same job as the fence check inside commit_identity_sweep_retire, and the description never says why the parent failed spawned_under when "the account never changed".

Not justified as shipped

  • Item 3 — duplicate of CancellationCoordinator.commit_identity_sweep_retire_inner (subagent_manager/cancellation.py): before committing, its if current: return False already defers on any child the tentative fence recorded, under the same _teardown_fence_lock. The earlier has_fenced_children check in _parent_has_sparable_child only saves one limit=1 store read.
    Subtraction: delete has_fenced_children / has_fenced_children_impl and its _ChildTeardownHandler Protocol entry; the commit check already gives the defer.
    Clears when: the methods are deleted, or a test shows a fenced child that the commit check misses.
  • Item 1 — symptom-level: the description says children die "even though the account never changed". On base, spawned_under already spares a parent whose stamp equals the live fingerprint, so the parent in identity-change sweep force-kills running subagents (reaper_force_kill -> provider shutdown) #17360 must have been unstamped, or a read must have reported a false change. The PR defers the cancel. It does not say which of those two causes started the sweep, or that the cause is left in place.
    Clears when: the description names which cause fired in identity-change sweep force-kills running subagents (reaper_force_kill -> provider shutdown) #17360 and states that this fix sits below it.

What this change ships

Inventory (6 items) — 4 justified

Intent: keep a user's running spawn_run sub-agents alive when a turn in another chat triggers the identity sweep — a FIX.

  1. An idle parent with running children is deferred, not retired, on the per-turn sweep — symptom-level
  2. A partial logout (a credential source dropped from the baseline or from the parent's own spawn stamp) still cancels the children — justified
  3. A child admitted after the fence opens is seen by a pre-commit fence check — duplicate of commit_identity_sweep_retire_inner
  4. A tentative admission fence is atomically promoted to the committed fence when a childless parent is retired — justified
  5. Retirement teardown, including the existing sign-out retirements, now survives the sweep being cancelled — rides along
  6. A deferred parent keeps the sweep incomplete, so it is re-checked on every turn inside the start barrier — justified

[FIRST-PRINCIPLES-REVIEWED] 4ea98cd

Description read

[DESCRIPTION-READ] e726b293082e2b903106ab6569ca0d86964ad4a424c84236b7250ad1746bc52a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 4ea98cd02e022048c039d70bf3345f0a45694f2c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The fix adds a two-phase teardown-fence protocol and seven new SubagentManager methods, which may be more than the bug needs, and deferral has no time bound.

Watch

  • Proportionality. The bug is that an idle parent with running children gets retired. The fix answers it with a tentative fence opened under the registry lock, a probe off the event loop, then a re-lock and a commit_identity_sweep_retire promotion. That adds open_teardown_fence, has_fenced_children, release_teardown_fence, commit_identity_sweep_retire, snapshot_teardown_children_only and two probes, plus a shielded drain in a method-level finally. retire_kiro_identity_sessions goes from about 140 to about 400 lines of interleaved lock and cancel handling. Every later change to the sweep or to spawn admission has to keep this ordering right. There is a simpler option: under the lock, synchronously treat has_live_or_queued_children(key) like semaphore.locked(). That means flag the session, skip it, and mark the sweep incomplete. The fence and store read would then only be worth keeping if durable-only or window-held children are a real cause of identity-change sweep force-kills running subagents (reaper_force_kill -> provider shutdown) #17360. The description never shows that they are.
    Clears when: the PR shows (a trace or a test reproducing #17360) that the in-memory check alone still cancels a real child, or the design shrinks to that synchronous spare.
  • Unbounded deferral holds a gateway-wide barrier on every turn. The description says: "NOT timeout-bounded… awaiting the off-loop store read while the barrier is held." A parent with an hours-long child keeps the sweep incomplete. So every turn, from any chat, drains all cold-start permits while the task store is read. If the writer thread is slow, every new session start on the gateway stalls behind that read. The account change also stays unreconciled for that key for the child's whole life.
    Clears when: the barrier is released before the store probe (re-checking under the re-taken lock), or the deferral gets a bound after which the cancel path resumes.

[DESIGN-REVIEWED] 4ea98cd

Description read

[DESCRIPTION-READ] 1354860af24a724df037e1abd58691a5e8dc3019dc72ef9dabca6f731ee68931

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 4ea98cd02e022048c039d70bf3345f0a45694f2c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4ea98cd

Verdict parsed from the review's SHA-scoped output markers for commit 4ea98cd02e022048c039d70bf3345f0a45694f2c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4ea98cd02e022048c039d70bf3345f0a45694f2c: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 4ea98cd02e022048c039d70bf3345f0a45694f2c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4ea98cd

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4ea98cd02e022048c039d70bf3345f0a45694f2c: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the fix/17360-spare-idle-parent-with-children branch from 018b763 to 6d7b740 Compare October 8, 2026 09:29
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@chenmingwei23
chenmingwei23 marked this pull request as ready for review October 8, 2026 15:29
@chenmingwei23
chenmingwei23 requested review from a team and KiroCrewCaptain as code owners October 8, 2026 15:29
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 8, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR stops the account-identity sweep from retiring an idle parent whose spawn_run children are still running. It adds a side-effect-free has_live_or_queued_children probe and defers that parent the same way a busy one is deferred. That fixes the "provider shutdown" kills in #17360, but the identity sweep is also the sign-out path, and the deferral now applies there too. dashboard/handlers/kas_login.py:296 (_retire_runtimes_after_sign_out) calls retire_kiro_identity_sessions() with no fingerprint after a Crew sign-out. At session_lifecycle.py:2098 the new if self._parent_has_live_children(key): ... continue then spares the parent, and its children keep running on the signed-out account's in-memory credential until they finish. Before this PR they were cancelled. That is a real identity/credential regression. The Design lane raised it on head 6d7b740 and it has no author disposition. The same lane notes docs/system-specs/modules/session.md § Account-identity retirement does not describe the new (subagents) deferral. Suggested fix: add a keyword such as spare_children: bool = False to retire_kiro_identity_sessions. Pass True only from the per-turn chat_runner._retire_sessions_on_identity_change sweep, and keep sign-out on the cancel path. Add a test pinning that sign-out still cancels an idle parent's children, and update session.md. The First Principles duplicate-filter point (has_live_or_queued_children_impl copying snapshot_teardown_children_impl's predicates) is worth folding into one shared selection while you are there. Otherwise the probe is pure, and the new tests prove the spare and the childless-retire cases. — crew-pr-reviewer

@chenmingwei23
chenmingwei23 force-pushed the fix/17360-spare-idle-parent-with-children branch from 6d7b740 to ed15711 Compare October 8, 2026 18:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 8, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Thanks — this is a real regression and it's fixed in ed157112ab, not overridden.

The identity sweep is also the sign-out path: _retire_runtimes_after_sign_out calls retire_kiro_identity_sessions() with no fingerprint, and the deferral I added would have spared a parent with live children there too, leaving those children running on the signed-out account's in-memory credential. That is the fail-unsafe direction, so I scoped the deferral off the sign-out path:

  • retire_kiro_identity_sessions now takes spare_children: bool = False.
  • Only the per-turn identity-change sweep (chat_runner._retire_sessions_on_identity_change) passes spare_children=True, because that path fires on every chat turn and cancelling live children each turn is the behaviour being fixed.
  • Sign-out keeps the default False, so the parent and its children are retired — the account is gone, so they must not keep running.

Tests added/updated in test/test_session.py and test/test_external_logout_detection.py:

  • test_signout_retires_an_idle_parent_with_live_children: the default path retires an idle parent with a live child and cancels it (sign-out behaviour).
  • test_identity_retire_spares_an_idle_parent_with_children: the per-turn path (spare_children=True) still defers.
  • test_returning_to_the_baseline_account_still_sweeps now also asserts the per-turn caller passes spare_children=True.

Design lane: docs/system-specs/modules/session.md § Account-identity retirement now has a "Live-children deferral" paragraph describing the (subagents) deferral, the side-effect-free has_live_or_queued_children probe, the eviction-without-cancel path that lets children's completions reach the successor, and that it is scoped to the per-turn path while sign-out keeps the cancel behaviour.

First Principles note (duplicate selection): the pure probe mirrors the live/queued selection the teardown snapshot uses. I kept them as two methods rather than folding them into one shared selection: the probe must stay side-effect-free, and the snapshot takes the live-and-queued run ids plus the parked/undelivered handling for teardown, so a single shared helper would either leak the teardown's selection shape into the pure read or re-introduce the coupling this fix removed. Happy to fold them in a follow-up if you'd prefer one selection with a collect_ids flag.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/17360-spare-idle-parent-with-children branch from ed15711 to eec91fd Compare October 8, 2026 18:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

All three review lanes converge on the same real finding, and it's fixed in eec91fdac7 — not overridden.

The finding (GPT 6.1, Opus 5.5, and First Principles item 5): an external kiro-cli logout never goes through _retire_runtimes_after_sign_out, so the per-turn sweep is the only path that catches it. There live == "", but my previous head passed spare_children=True unconditionally, so an idle parent with live children was deferred and its children kept running on the signed-out account's in-memory credential. That is the fail-unsafe direction.

Fix: the per-turn caller now passes spare_children=bool(live) (chat_runner.py). An empty live fingerprint — an external logout or an unreadable store — keeps the existing retirement-and-cancellation path, exactly as the sign-out path does. Only a proven switch between two real accounts (non-empty live) defers. This narrows the deferral to the case #17360 is actually about and keeps the fail-safe shape: sign-out and external-logout both cancel.

First Principles blocker (deleted pin): I restored retire_kiro_identity_sessions to the test_a_failing_provider_shutdown_still_ends_the_children parametrize list, running on its default spare_children=False path. The deferral is gated behind spare_children, so that path retires an idle parent even with a live child and the test asserts the child is still cancelled when the shutdown raises. The childless retire-shutdown-warning contract keeps its own dedicated test.

Tests added/changed:

  • test_an_external_logout_cancels_children_on_the_per_turn_path (new): the per-turn path with an empty live fingerprint passes spare_children=False and cancels.
  • test_a_failing_provider_shutdown_still_ends_the_children: retire_kiro_identity_sessions restored to the parametrize list on the default path.
  • test_returning_to_the_baseline_account_still_sweeps keeps asserting the A→B (non-empty live) path passes spare_children=True.

Docs: docs/system-specs/modules/session.md § Account-identity retirement now states the deferral is scoped to a proven A→B switch, and that the per-turn caller passes bool(live) so an external logout keeps the cancel path.

First Principles item 2 (duplicate live/queued selection): I kept the pure probe and the teardown snapshot as two methods. The probe must stay side-effect-free, while the snapshot also arms the delivery gate and handles parked/undelivered children for teardown; a single shared helper would either leak the teardown's selection shape into the pure read or re-introduce the coupling this change removed. Happy to fold them in a follow-up with a collect_ids flag if preferred.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 10, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/17360-spare-idle-parent-with-children branch from 5ad5904 to 2c204ce Compare October 10, 2026 10:17
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Addressed in 2c204ce3c5. The teardown-decision rework landed with four real issues; all are fixed.

  • A turn starting during the off-loop read is no longer shut down. Selection and the pop were one lock hold before this rework; the off-loop read reopened that gap. The commit block now re-checks the semaphore under the re-taken lock: if a turn took it during the read, the parent is deferred (flagged, sid cleared, recorded (busy)), not retired, so provider.shutdown() never lands on a live turn.
  • The defer branch now clears the old-account sid. A deferred parent is evicted and cold-started on its next turn; without the sid drop it would reload the previous account's conversation (the signed-thinking rejection the sweep's sid drop exists to prevent). It is cleared inside the permit barrier, the same place the retired keys clear it.
  • The fence release is identity-guarded. open_teardown_fence returns the set it created, and the defer path releases only when the stored fence IS that object — so a destroy/reset of the same parent that opened a new fence during the read keeps its own, and its post-shutdown cancel still sweeps the store.
  • The double end-record is gone. The retire path now records END_REASON_RETIRED once, for the keys retired under the lock and the spare candidates committed at the teardown together — a single awaited set, still inside the permit barrier. (This was the Windows shard-5 failure.)
  • The spec and docstring now describe the design: docs/system-specs/modules/session.md and the retire_kiro_identity_sessions docstring cover the teardown decision and both dropped-component conditions (the chat_runner baseline check and the per-holder spawn-stamp check).

A new test asserts that a turn starting during the read defers (it fails without the semaphore re-check). mypy, flake8, black and the docs lint are clean, and the related session/overload/logout/session-duration suites pass. Rebased onto current green main.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 10, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Both points fixed in the staged head (will land on the next push).

  • Item 9 — _parent_has_live_children has zero callers. Correct: the teardown rework routes the decision through _parent_has_sparable_child, leaving that method dead. Deleted it.
  • Item 8 — the retired_keys end-record moved out of the selection lock. Correct, and restored. record_sessions_ended(retired_keys, …) is back inside the selection lock hold, with its original invariant note (recorded as one set under the same hold as the removals, so the awaited unlink cannot be cancelled between keys). The spare-children candidates committed at the async teardown record their own ends there, under their own lock hold — two commit points, each recording under its lock, rather than one deferred batch. The telemetry invariant test now asserts the retire path records only END_REASON_RETIRED (and nothing else) across those two points, instead of asserting a single call-site.

No design decision is being overridden here — both were real issues in the rework, so they are fixed rather than argued.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/17360-spare-idle-parent-with-children branch from 2c204ce to e525f18 Compare October 10, 2026 10:49
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Fixed in e525f18386.

A child admitted during the off-loop store read is no longer cancelled. _parent_has_sparable_child could read childless and then an admission for the parent land during the await (registering a live run, or recording a row in the fence before its store write). The commit block now re-reads the in-memory and fence child checks under its own lock, after the await, and defers when a child appeared — the same way it already defers on a successor claim or a turn taking the semaphore. The checks are synchronous and cheap, and the store half an admission writes is also recorded in the fence first, so the fence re-check covers a store-only admission too.

A new test drives exactly this: a child admitted into the fence during the store read defers the parent, and it fails without the re-check.

This head also folds the two First Principles fixes (the dead _parent_has_live_children deleted; the retired_keys end-record restored under the selection lock). mypy, flake8, black and docs lint are clean, and the related session/overload/logout/session-duration suites pass.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 10, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Both fixed in 41ed4e5ccd.

  • Tentative fence no longer suppresses a spared parent's delivery. The identity sweep opened its fence before deciding retire-vs-defer, which made accepted_before_open_teardown treat an expiring child's row as "accepted for a conversation that ended" and drop its report — even when the parent was then spared. The sweep now opens a separate tentative fence (_identity_sweep_fences): the admission path records into it (so a concurrent /api/spawn is still seen by the retire-vs-defer re-check), but accepted_before_open_teardown ignores it. Only a committed teardown fence suppresses delivery, which happens when retirement actually commits (through snapshot_teardown_children). On defer, the tentative fence is dropped and nothing was suppressed.
  • The retirement end-record ratchet is pinned, not loosened. The invariant test asserts found == ["END_REASON_RETIRED", "END_REASON_RETIRED"] again — the two commit points (keys retired under the selection lock, and spare candidates committed at the teardown), exact and in order, so a third recorder or a dropped one trips it.

A new test drives the tentative fence directly: a recorded admission is visible to has_fenced_children but a non-fenced row still delivers (accepted_before_open_teardown is False) while only the tentative fence is open, and a committed fence does suppress. It fails against the previous (committed-fence) behaviour.

mypy, flake8 and black are clean, and the related session/overload/logout/session-duration/queue-depth suites pass.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Both findings are fixed in c3f1755e06.

F1 — the final child re-check and the committed-fence open are now atomic. The sweep's last decision went through a new commit_identity_sweep_retire(key, fence) primitive that runs under the same _teardown_fence_lock the admission path records a new child under. In one hold it either (a) sees the sweep's tentative fence recorded a child since it opened and returns False so the caller defers, or (b) finds it empty and promotes that same set object to be the key's committed teardown fence. Because it is the same object, a row an /api/spawn recorded concurrently is already in it and is carried into accepted_since — so the cancel spares it instead of sweeping it. There is no longer a window where a child lands only in the tentative fence and is then cancelled by the store sweep.

F2 — a cancelled sweep no longer leaks a provider or leaves a false crashed crumb. The maybe_spare loop is wrapped in try/finally: the finally records the end of every retirement already committed in this loop (so an unregistered session always has its end recorded) and releases every tentative fence the loop has not yet resolved. The committed providers are shut down in the existing teardown loop, each under its own guard.

New tests: test_commit_identity_sweep_retire_is_atomic_against_admission (defer when a child was admitted into the tentative fence; promote-and-carry when empty) and the existing tentative-fence delivery test. Related suites stay green.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

All three items are addressed in c3f1755e06.

Item 3 — duplicate store read removed. has_pending_store_children_async is deleted from the admission bridge; the manager's has_pending_store_children now calls taskq_pending_ids_for_async(parent_session_key, include_window=True) directly, so there is a single reading of a parent's store-only waiting children.

Item 7 — note_teardown_snapshot returns None again. The fence-returning variant had no caller, so it is reverted to its original signature.

Item 4 — the PR description now matches shipped behavior. The ## Not a goal bullet is rewritten: an empty or partial live fingerprint is the fail-safe cancel path, not a defer — the parent and its children are retired and cancelled, the same as sign-out, because the account the children loaded is gone. Only a whole switch between two real accounts defers a parent that still has children.

black/flake8/mypy are clean and the related suites pass.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Both findings are fixed in 0430a822b9.

F1 — cancellation no longer skips a committed retirement's teardown. The committed retirements are no longer drained in a loop after the permit barrier; they are collected into committed_teardown and drained inside an always-run finally on the selection loop. Each per-key teardown — provider.shutdown(), the child cancel, and the runtime release — is wrapped in asyncio.shield, so a cancellation of the sweep (the initiating chat closing) lets each committed parent finish tearing down before the CancelledError propagates. The error is re-raised out of the finally, never swallowed. A parent that was unregistered under the lock can no longer be left with its provider alive and its children running under the logged-out credential.

F2 — the end record is bound to its own retirement, under the lock. Each committed retirement records its end with record_sessions_ended([key], ...) inside the same owner._lock hold that removes the session, before any later candidate's off-loop store probe can suspend the sweep. A same-key successor that registers through open_task_session during a later probe can no longer have its live-start entry consumed or its crash crumb deleted by a batched recorder that ran after the await.

New test: test_identity_retire_cancel_mid_teardown_still_drains_committed cancels the sweep while a second candidate's store probe is suspended and asserts the first, already-committed parent is fully torn down (provider shut down, children drained). It fails if the committed teardown is moved back out of the finally. Related suites pass.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Fixed in b8f0623ff3.

Both the synchronously-retired parents (doomed) and the spare parents committed at the teardown (committed_teardown) now drain in one cleanup, and that cleanup satisfies two constraints at once:

  • Cancellation-safe. The async with on the sweep lock is wrapped in a method-level try, and the drain runs in its finally. So a cancellation of the sweep (the initiating chat closing) still shuts every already-unregistered parent down, cancels its children and releases its runtime before the CancelledError propagates. The error is re-raised, never swallowed, and a guard makes the drain run exactly once whether the sweep returns or is cancelled.
  • Outside the sweep lock. The finally runs after the async with has released the lock, so a slow child shutdown here does not hold the sweep lock against a following account change that needs to acquire it.

Each per-key teardown is individually asyncio.shielded. A stamp-dropped parent (a partial logout) can no longer be left with its children running on the logged-out credential when the sweep is cancelled mid-teardown.

New test: test_identity_retire_cancel_mid_teardown_still_drains_stamp_dropped cancels the sweep while a spare candidate's store probe is suspended and asserts the stamp-dropped parent is fully torn down. It fails if that parent's teardown is moved back out of the shared cleanup. The existing fence-recapture test (which requires the drain to stay outside the sweep lock) still passes.

@iamwhatever

Copy link
Copy Markdown
Contributor

This PR stops the account-identity sweep from retiring an idle parent whose spawn_run children are still running, which was the cause of the "provider shutdown" kills in #17360. iamwhatever's earlier request is addressed on head b8f0623. retire_kiro_identity_sessions takes spare_children: bool = False, and only chat_runner._retire_sessions_on_identity_change passes True (and only when the live fingerprint is whole and no identity component dropped). The sign-out path _retire_runtimes_after_sign_out keeps the cancel behaviour, and that is pinned by test_signout_retires_an_idle_parent_with_live_children. docs/system-specs/modules/session.md documents the deferral. I am still holding it. The change touches credential/identity retirement and subagent cancellation, and it has grown to about 740 production lines, including a tentative-fence protocol, an off-loop store probe and partial-credential-loss detection in session_lifecycle.py and subagent_manager/cancellation.py. The Design and First Principles lanes both flag (CONCERNS) that the description still describes the earlier one-predicate design and does not declare these mechanisms. Also, iamwhatever's CHANGES_REQUESTED review at 6d7b740 is still formally open. Checks are green, and GPT/Opus report no blocking findings. Next steps: rewrite the body to cover the whole current diff, have a maintainer review the identity-path additions, and have iamwhatever dismiss or re-review. — crew-pr-reviewer

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This stops the per-turn identity sweep from cancelling an idle parent's live spawn_run children. Our earlier change request is met on b8f0623. Sign-out still cancels (spare_children defaults to False and is pinned by test_signout_retires_an_idle_parent_with_live_children), an empty or partially dropped credential still cancels (identity_component_dropped), and session.md documents the deferral. I tried to break the fence protocol (an admission between open and commit, a successor registering during the off-loop read, a turn taking the semaphore mid-read) and each path defers or commits safely. Remaining defect: a deferred parent leaves the sweep incomplete for as long as its children run. Every chat turn then re-enters retire_kiro_identity_sessions, which holds owner._start_sem.drain() (all cold-start permits) across await self._parent_has_sparable_child(key), an off-loop task-store read, for each deferred parent (session_lifecycle.py, the maybe_spare loop). On a host with an hours-long subagent after an account switch, every cold start on every chat waits behind that drain every turn. Before this PR the sweep finished once. Suggested fix, as the Design lane proposes: under spare_children, retire the parent the way _evict_stale_session (session_allocation.py ~1178) does. Shut the provider down without _cancel_parent_children, so the sweep completes and the fence, probe and commit protocol (seven new _ChildTeardownHandler methods) can be removed. If you keep the current design, bound and document the deferral cost in session.md. Either way, rewrite the description: it still describes a one-predicate design with "no new dependency" and lists empty fingerprints as a non-goal. — crew-pr-reviewer (deep re-check)

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Both points from the 19:57Z review are answered on 22864def55.

Point 1 — the per-turn drain cost of an incomplete deferral. Confirmed real: a deferred parent keeps the sweep incomplete, the pending fingerprint is not cleared, and so the per-turn gate re-runs retire_kiro_identity_sessions on the next turn from any chat and re-checks that parent. That re-check is inside owner._start_sem.drain() and awaits the off-loop _parent_has_sparable_child read while the barrier is held (session_lifecycle.py ~2104 opens the drain, ~2272 awaits the read), so each cold start waits behind it until the candidates resolve. On a host holding an hours-long child the deferral is not timeout-bounded, so this cost recurs each turn for the child's life.

I am keeping the current design rather than taking the suggested rework to retire via _evict_stale_session and remove the fence/probe/commit machinery, and I have bounded and documented the cost instead (docs/system-specs/modules/session.md § Account-identity retirement, new "Cost of an incomplete deferral" paragraph, in 22864def55). The documented bound: the deferral is intended to be short-lived relative to a spawn_run child and clears the moment the parent is childless or takes a turn; it is not timeout-bounded; the per-check cost is kept small (synchronous in-memory/fence checks and a limit=1 store read on the writer thread, with a childless or turn-taking parent dropping out of the candidate set); and it is accepted because the alternative it replaces is cancelling the children on every turn, not deferring on every turn.

On why not the _evict_stale_session-style retire-without-cancel as the whole fix: that path is what lets a deferred parent's children survive its eviction, and the PR already relies on it. But it decides retire-vs-spare the same way the sweep must — on whether a child still exists — so routing the sweep through it does not remove the child existence check; it moves where the check is read. The reason the decision was pulled to the point of action under _teardown_fence_lock is that every earlier shape that read child-presence before the commit (an in-lock read that froze the gateway on the store's busy wait, then an off-loop preflight probe read before the lock and acted on later) left a read-vs-act window where a child admitted in between was cancelled. Deciding at the teardown under the fence closes that window, which is why the fence and the atomic commit_identity_sweep_retire are there; removing them reopens it. If a maintainer prefers the eviction-style rework despite that, I will take it as a follow-up with the race coverage carried over.

Point 2 — the description. Rewritten to match the shipped diff. It no longer describes a single pure predicate or claims "no new dependency": it now describes the teardown-fence decision, the off-loop has_pending_store_children store probe, the synchronous spawn-stamp drop gate and the chat_runner baseline gate, the cancellation-safe shielded teardown drain, and the full new public surface (has_live_or_queued_children, has_pending_store_children, open_teardown_fence, has_fenced_children, release_teardown_fence, commit_identity_sweep_retire, snapshot_teardown_children_only, identity_component_dropped, and spare_children). The Not-a-goal and Tests sections are corrected to the tests that exist on this head, and a "Cost of an incomplete deferral" section mirrors the spec. The empty/partial-fingerprint case is stated as the fail-safe cancel path, not a non-goal.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR stops the per-turn identity sweep from cancelling the live spawn_run children of an idle parent. It does that by deciding spare-or-retire at the async teardown, under a tentative admission fence, and the root cause in #17360 is fixed. Point 1 of our 19:57Z review is met on c905358. _parent_has_sparable_child returns early on the synchronous in-memory check, so a long-running live child never reaches the store read inside _start_sem.drain(). Only a durable-only row pays for that read, and session.md now documents the cost. Point 2, the description, has been rewritten. I tried to break the fence by admitting a child between open and promote, by registering a successor during the read, and by taking the semaphore mid-read. Each case defers or commits safely, because commit_identity_sweep_retire promotes the same set under _teardown_fence_lock and the snapshot follows it with no await. One real defect remains in the new cancellation-safe drain, and it is the Opus finding on this head, which has no disposition yet. In the method-level finally of retire_kiro_identity_sessions (session_lifecycle.py ~2405-2422), except asyncio.CancelledError: raise leaves the for key, provider in (*doomed, *committed_teardown) loop after the first entry. Every later entry is already unregistered and recorded as ended, but it never gets shutdown(), _cancel_parent_children or release_subagent_runtime. Those children keep running on the old credential, which is exactly what this finally says it prevents. Fix: record the CancelledError, finish draining each remaining entry with every step shielded, re-raise after the loop, and add a test that cancels during the first of two shutdowns. Also, session.md (~697) says the probe is a limit=1 read, but taskq_pending_ids_for_async calls store.list_pending with no limit (the GPT finding). Either pass a limit or correct the doc. — crew-pr-reviewer (deep re-check)

@chenmingwei23

chenmingwei23 commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Re: review 2026-10-11 — both points addressed; current head is 4ea98cd02e (the branch was rebased onto current main, so the earlier 744fd6b299 is superseded).

1. CancelledError in the shutdown drain abandoned later entries. Fixed. In the method-level finally of retire_kiro_identity_sessions, the drain loop over (*doomed, *committed_teardown) caught asyncio.CancelledError and re-raised immediately, so a cancel landing on the first entry's shutdown left every later entry un-shut-down even though each was already unregistered and recorded ended. The loop now records the first CancelledError, keeps draining each remaining entry (every step is still shielded), and re-raises it after the loop — so a cancel still propagates while every committed retirement is fully torn down. Added test_identity_retire_cancel_during_first_drain_still_drains_second: two committed candidates, the first's shutdown raises CancelledError, and the test asserts the second is still shut down and the children drained while CancelledError still propagates. It fails before the fix (the second provider is never shut down) and passes after.

2. session.md said the probe is a limit=1 read. Fixed by correcting the doc rather than the code. has_pending_store_children routes through taskq_pending_ids_for_async(..., include_window=True), which is the same reading the teardown's own cancel uses and must return the full row set for the stop-all cascade callers in cancellation.py; passing limit=1 there would truncate those callers, so it is not a safe change. The probe only tests existence (bool(rows)), so the doc now says the store read runs on the writer thread, lists the parent's pending rows, and the probe answers on the first row — no limit=1 claim.

Rebase note. The rebase onto current main surfaced test/test_cron_identity_gate.py (added to main after this branch's base). Its test doubles predate the spare_children parameter and the identity_component_dropped method this PR introduces, so its _retire stub rejected the new keyword (TypeError, swallowed by the gate's except) and its prerequisite double lacked the method — four cases went red. Fixed in the test doubles only (the stub forwards the keyword; the double answers identity_component_dropped → False for a clean account switch). No production behaviour changed: the gate still calls retire before acquire, and the real service already has the method.

Local gate green on the committed head: flake8 src/kiro_crew test, mypy on the changed modules, black --check, and the full drain/retire-path suite (test_session.py, test_cron_identity_gate.py, and every other file referencing the retire gate — 876+ cases) all pass; CI runs the cross-surface set.

kirocrew-agent added 2 commits October 11, 2026 12:57
…entity sweep (#17360)

The account-identity sweep retires every idle kiro-backed session, and a
parent that ended its turn with spawn_run children still running is idle
by the semaphore test. So its retirement cancelled those children
("provider shutdown") on every turn any chat took while the sweep stayed
incomplete.

Before retiring an idle session, snapshot its children with the existing
lock-held _snapshot_parent_children and, if any are live, defer the
parent with retire_on_identity_change (the same treatment a mid-turn
session gets) and record it in identity_sweep_waiting_on. The children
then outlive the turn and the parent is recycled only once it is
genuinely idle. A live-account holder is already spared earlier, so this
only waits on the right-account parent's own children.
…sion-safe

The per-turn identity sweep retired an idle parent that still had running
spawn_run children, cancelling them as a provider shutdown every turn. Sparing
such a parent needs to know whether it has any child -- live, queued, or held
only by the task store -- and that answer kept racing: a sequence of preflight
read-then-decide designs each left a window (a sync read freezing the loop, a
failed read, a busy parent unlocking mid-await, a stale baseline, a stale
negative probe) where the decision disagreed with the admission path.

The decision moves to the point of action. There is no preflight probe. Under
the registry lock a spare_children candidate is NOT retired; its teardown fence
is opened (the same _teardown_fence_lock the admission path records a new child
into, before its store write) and it is left registered -- nothing irreversible
happens for it under the lock. At the async teardown, off the loop, the sweep
asks whether the parent still has a child: live or queued in memory, admitted
into the fence since it opened, or a pre-existing durable row in the store. If
so -- or if the store read fails -- it defers, cancels nothing, and the next
sweep re-checks. Only a childless parent is retired, under a re-taken lock that
leaves a successor registered under the key alone.

The one synchronous exception stays: a parent whose OWN spawn stamp lost a
credential component (identity_component_dropped vs spawn_identity_of) is retired
and its children cancelled, the fail-safe for an external logout.

The sync preflight probe, has_pending_store_children on the probe path, and the
probed-key bookkeeping are removed (net simpler). chat_runner keeps the baseline
identity_component_dropped check for the simple logout.

Tests: durable-only child, failed read, spawn-stamp drop (cancels), whole spawn
stamp (spares), a child admitted into the fence during the teardown, and a
pre-existing durable child each as a regression.
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

CI follow-up — new head 4ea98cd02e.

The branch was 67 commits behind main, so I rebased it onto the current tip (clean, no conflicts) and batched that into the push. The rebase surfaced a test that was added to main after this branch's base, test/test_cron_identity_gate.py (from the cron account-check change). Four of its cases went red because its test doubles predate the spare_children parameter this branch adds to the identity-retire gate:

  • The gate now calls retire_kiro_identity_sessions(fingerprint=live, spare_children=...), but the test's _retire stub only accepted fingerprint, so the extra keyword raised TypeError — swallowed by the gate's except, which is why retire never showed up in the recorded call order.
  • The gate also reads service.identity_component_dropped(live) to tell a clean account switch from a partial credential loss, and the test's prerequisite double did not define that method.

Fixed both in the test doubles only — _retire now accepts the forwarded keywords, and the prerequisite double answers identity_component_dropped (False for these clean-switch cases). No production code changed for this; the real service already has the method. All 9 cases in that file pass locally, along with the other files that stub or call the retire gate.

Local gate green on the committed head: flake8 src/kiro_crew test, mypy on the six changed modules, black --check, and the directly-testing backend suites (test_session.py, test_cron_identity_gate.py, and the files referencing the retire gate) all pass; CI runs the cross-surface set.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR stops the per-turn account-identity sweep from cancelling the live spawn_run children of an idle parent, which was the root cause of the "provider shutdown" kills in #17360. It decides spare-or-retire at the async teardown under a tentative admission fence. Both points of our 06:03Z change request are met on 4ea98cd. In the method-level finally of retire_kiro_identity_sessions (session_lifecycle.py), the drain loop now records the first CancelledError, keeps shutting down, cancelling children and releasing the runtime for every later entry (each step shielded), and re-raises after the loop. test_identity_retire_cancel_during_first_drain_still_drains_second pins that: A's shutdown raises, and B is still shut down with its children drained. session.md no longer claims a limit=1 probe. The author's reason for keeping the full read is sound, because taskq_pending_ids_for_async also serves the stop-all cascade. I compared the PR-only diff at the old and new heads: the rebase added only test-double fixes in test/test_cron_identity_gate.py. spare_children=True is still passed only from chat_runner._retire_sessions_on_identity_change, and only when live is non-empty and identity_component_dropped(live) is false. So sign-out, external logout and partial credential loss all still cancel. In earlier rounds I tried to break the fence (an admission between open and promote, a successor registering during the off-loop read, a turn taking the semaphore mid-read), and each case deferred or committed safely. This round I tried a cancel during the first of two drains, which the new loop handles. Checks are green, and this approval replaces our earlier CHANGES_REQUESTED. — crew-pr-reviewer (deep re-check)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

identity-change sweep force-kills running subagents (reaper_force_kill -> provider shutdown)

2 participants