Skip to content

fix(chat): let a cancelled background start finish its session teardown - #36664

Merged
bolichen97 merged 1 commit into
mainfrom
fix/35669-crew-switch-lease
Nov 17, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/35669-crew-switch-lease

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A chat is no longer locked out with "This session is loaded in another program" by an agent process that a cancelled background start left running.

After a chat's background start was cancelled at the wrong moment, every later send on that chat failed with This session is loaded in another program, kirocrew (pid N), which owns its execution. The process holding the chat's session was one the gateway had already dropped from its registry, so no reset, sweep or agent switch could reach it. Only a gateway restart or a manual kill freed the chat. The report in #35669 hit it right after an Ultra Mode (Crew Mode) toggle on an existing chat.

Why it matters

The chat is unusable until the person finds and kills the stray process or restarts the gateway, and nothing on screen says why. Any chat can hit it: a background start (a focus prefetch or an eager spawn) is cancelled by the next signal for its chat (a focus or blur frame, another start), and moving between chats or changing a chat's agent or model sends exactly those.

Not a goal

  • Changing when a background start is scheduled, cancelled or admitted (fix: keep a resume prefetch whose runtime start has begun #33898 is about which focus cancels may land).
  • Making SessionManager.remove and SessionManager.reset themselves finish when cancelled, for every caller.
  • Finding and stopping a lease-holding process that is already orphaned.
  • Pinning down the exact click sequence of the report: this fixes the leak reproduced below, which ends in the same refusal; the issue stays open for the reporter to confirm.

What changed (motivation → approach → change)

Symptom → cause. A background start of a chat runs in one task, and every later signal for the chat cancels it. Three teardowns inside that task take the session out of the registry first and only then stop its process: the two guard arms of _spawn_admitted_prefetch (the slot vanished, or its bindings changed while the agent was starting) and the deferred project reset it applies first (_consume_pending_reset). A cancel that lands between those two steps abandons the teardown. The agent process keeps running, still holding the chat's resumed session and kiro-cli's execution lease on it, and nothing in the gateway knows it exists. The next start of that chat loads the same session and every send is refused.

Change. Those three teardowns run under asyncio.shield(...), the way the prefetch TTL (_prefetch_ttl) and the over-cap eviction (_evict_prefetches_beyond) in dashboard/chat_turn/eager_spawn.py already wrap theirs. The teardown runs to completion while the cancel still ends the task. A cancelled deferred reset in dashboard/chat_turn/session_reset.py leaves its flag armed, as any failed consume does, so the next consume applies it.

The permit goes with the session. A teardown that runs to its end shuts a provider down, and a real claim can be parked on that session's turn permit. So the two guard arms keep the permit until sessions.remove pops the session, which releases it in the same lock hold as the pop: the parked claim wakes, finds the session gone, and starts its own with the slot's current bindings. Each path that keeps the session releases the permit itself. No new helper, flag or retry.

Backwards compatibility

Compatible: the same sessions are torn down on the same paths. A cancel arriving mid-teardown no longer stops it halfway, and a claim parked on a stale session's permit is served a session carrying the slot's current bindings instead of the one the teardown is about to kill.

Removes nothing: no path, record, key, validator or function is removed or narrowed; an already-started teardown is only allowed to finish.

Tests

  • test/test_eager_spawn.py::TestEagerSpawn::test_a_cancel_mid_teardown_still_ends_the_process[vanished|bindings|pending_reset] drives _eager_spawn into each of the three teardowns with a fake whose teardown pops, then waits on an event before its shutdown step. It cancels the task while the teardown waits, checks the task ended cancelled, releases the event, and checks the shutdown step ran. With the shields reverted each case fails with assert {'shut_down': False} == {'shut_down': True}.
  • test/test_eager_spawn.py::TestEagerSpawn::test_a_claim_parked_on_a_stale_session_starts_its_own[vanished|bindings] runs a real SessionManager: the background start creates the session, a real claim parks on its permit, the guard arm fires, and the test checks the claim was served a different provider, the stale provider was shut down, and the registry holds the claim's session. With the permit released ahead of the teardown both cases fail at assert provider is not stale["provider"].
  • The two guard-arm tests assert the task makes no release call of its own.

Proof: PROVEN test/test_eager_spawn.py::TestEagerSpawn::test_removes_session_when_slot_deleted_mid_handshake, test/test_eager_spawn.py::TestEagerSpawn::test_removes_session_when_bindings_change_mid_handshake, test/test_eager_spawn.py::TestEagerSpawn::test_a_cancel_mid_teardown_still_ends_the_process[vanished], test/test_eager_spawn.py::TestEagerSpawn::test_a_cancel_mid_teardown_still_ends_the_process[bindings], test/test_eager_spawn.py::TestEagerSpawn::test_a_cancel_mid_teardown_still_ends_the_process[pending_reset] (head f7052d4ac990)

prove.py ran on the verification host at commit f7052d4ac990, this change on base c4c95ce148; --per-hunk exits 0 there too and names no uncaught hunk. This head carries the same change plus one paragraph of _eager_spawn's docstring, with no executable line in it, on a newer main. On that base the verification host ran the related backend tests (1,258 passed), the test files naming _eager_spawn or _spawn_admitted_prefetch (2,919 passed; test_eager_spawn.py alone 53), the session-reset tests (765 passed), the tests of main's neighbouring changes to reset, app sessions and the session memory plan (#37984, #37804, #36872; 406 passed), mypy, semgrep and the flake-pattern ratchet, all green.

Determinism: the five new cases passed 20 runs out of 20 at -n0. A shuffled order of test_eager_spawn.py, test_eager_spawn_shutdown.py, test_eager_spawn_resume_prefetch.py and test_chat_runner_coverage_pending.py (133 ids, seed 35669, -n0 from an @file) passed, and those four files passed again at -n 4 --dist loadgroup --max-worker-restart=2. Every wait is an event-bounded await_until; nothing sleeps.

Manual verification

On Linux (Ubuntu 24.04.5 LTS, x86_64), against the real gateway (kiro_crew.testing.harness.spawn_feature_gateway, isolated KIROCREW_HOME, OS-assigned loopback port, Ultra Mode preview on). kiro-cli cannot create a session there without a sign-in, so the gateway ran against a stand-in kiro-cli: the in-tree fake_acp_backend plus kiro-cli's per-session execution lease, which refuses a second process with kiro-cli's own message. Sequence: send on a plain chat, a Crew Mode switch, a focus frame that starts the chat's resume prefetch (the stand-in holds its session/load reply 3 s), a model pick landing mid-start so the guard arm fires, the dashboard's blur frame cancelling the prefetch, then a second send. One throwaway sitecustomize.py (not committed) slowed a single await inside remove (record_session_ended, after the pop, before the shutdown) by 4 s, so a WebSocket frame can land in a gap that is otherwise a few milliseconds wide. No product logic was changed for the run.

Before, on the base:

INFO kiro_crew.dashboard.chat_runner: Eager spawn: slot chat-1-... bindings changed mid-handshake, removing session
   (no "Removed session" line follows: the blur cancelled the removal after the pop)
lease holder alive: pid 823808
send 'ping two' (200)
  reply: [{"role": "error", "content": "❌ Prompt error: {'code': -32603, 'message': 'This session is loaded in another program, kirocrew (pid 823808), which owns its execution.'}"}]

After, on this change, the same run:

INFO kiro_crew.dashboard.chat_runner: Eager spawn: slot chat-1-... bindings changed mid-handshake, removing session
INFO kiro_crew.session: Removed session (map preserved): dashboard:chat-1-...
lease holder alive: NO
send 'ping two' (200)
  reply: [{"role": "assistant", "content": "pong from the fake ACP backend"}]
fake runtimes left after gateway stop: (none)

A plain Crew Mode switch with no slowed await is answered on both, with no refusal.

Related Issues

Part of #35669

Sibling sweep

Swept: git grep -nE "sessions.(remove|reset|remove_if_unclaimed)(" -- src/kiro_crew/dashboard/
Sibling: src/kiro_crew/dashboard/chat_turn/eager_spawn.py -- changed here
Sibling: src/kiro_crew/dashboard/chat_turn/session_reset.py -- changed here
Sibling: src/kiro_crew/dashboard/chat_turn/app_agent_binding.py:121 -- not affected -- turn admission in the turn task, which no chat signal cancels
Sibling: src/kiro_crew/dashboard/chat_handlers.py:993 -- not affected -- switch and reload request handlers, which the server does not cancel
Sibling: src/kiro_crew/dashboard/chat_api/slot_lifecycle.py -- not affected -- the slot close handler (:1208) and the slot cleanup sweep (:1760), which no chat signal cancels
Sibling: src/kiro_crew/dashboard/handlers/session_environments.py -- not affected -- both removals (:731, :840) run in request handlers, which the server does not cancel
Sibling: src/kiro_crew/dashboard/handlers/taskrunner.py:1499 -- not affected -- request handler, which the server does not cancel
Sibling: src/kiro_crew/dashboard/handlers/hooks.py:1320 -- not affected -- request handler, which the server does not cancel

In eager_spawn.py the two guard-arm removals change; the over-cap eviction (:341) and the prefetch TTL (:1225) already shield theirs, and both reach the session through remove_if_unclaimed, which refuses under the lock while a permit is held, so a claimant that won one keeps its session. In session_reset.py the deferred project reset (:389) changes; it holds no permit of its own and reset(skip_if_busy=True) declines under the lock while one is held. Its conversation discard (:480) is never reached from the background-start task, which passes no allow_discard: only a turn cancelled by a slot delete reaches it, a different trigger. Its end-of-turn reset (:628) is turn cleanup that no chat signal cancels.

Pattern harvest

Rule candidate: review-prompt
Pattern: a session teardown that pops the registry entry and then awaits the shutdown (sessions.remove, reset, remove_if_unclaimed) leaks the process it popped when it is awaited unshielded in a task that a later signal cancels, and hands a parked claimant the session it is about to kill when the turn permit is released ahead of it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 501ae7037dc4a2e910c0e0aafe534e5f5e892849 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 501ae7037dc4a2e910c0e0aafe534e5f5e892849 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Every item is tied to the reproduced leak (#35669), and each one fixes the mechanism, not the symptom. The cause they leave alone, that remove/reset can be cancelled partway through, is listed under Not a goal.

Notes

  • Items 1–2 — point patch with counted unfixed siblings: I grepped await (state\.)?sessions\.(remove|reset)\( and got 45 call sites in 21 files, and only these 3 are shielded. Any of the others that runs in a task something can cancel can hit the same gap: the session is popped, and the cancel lands before its process is shut down. The description defers this ("Making SessionManager.remove and SessionManager.reset themselves finish when cancelled, for every caller"), and that larger fix is accepted as deferred. Subtraction: once remove/reset can no longer be cancelled partway through, drop the three call-site asyncio.shield wrappers.

What this change ships

Inventory (5 items) — 5 justified

Intent: stop a chat getting locked out by an agent process that a cancelled background start left running. FIX.

  1. A cancelled background start's stale-session removal (slot vanished or bindings changed) now finishes and stops the agent process. — justified
  2. A cancelled deferred project reset inside a background start now finishes, and its flag stays armed. — justified
  3. A turn waiting on a stale background session now gets a fresh session with the current bindings, because remove frees the permit in the same lock hold as the pop. — justified
  4. The test pin "Semaphore still released before teardown" is rewritten to "no release of its own". The description declares this, and the new test fails on the old order. — justified
  5. The _eager_spawn docstring now says who gives the permit back on each path. — justified

[FIRST-PRINCIPLES-REVIEWED] 501ae70

Description read

[DESCRIPTION-READ] 7fe75d97983e00b40b77efeeb77769dfafada2add011a74074551cb298f4fccf

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 501ae7037dc4a2e910c0e0aafe534e5f5e892849 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: shielding the three pop-then-shutdown teardowns follows the pattern _prefetch_ttl and _evict_prefetches_beyond already use. The real-SessionManager test shows that leaving the permit for remove hands a parked claim a new session and does not deadlock.

Notes

[DESIGN-REVIEWED] 501ae70

Description read

[DESCRIPTION-READ] b5ae33e37485d5979544bfbca0449c20bb6f7f4ffa7ef1f5605a83b9a0d337ac

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 501ae7037dc4a2e910c0e0aafe534e5f5e892849 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 501ae70

Verdict parsed from the review's SHA-scoped output markers for commit 501ae7037dc4a2e910c0e0aafe534e5f5e892849.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 501ae7037dc4a2e910c0e0aafe534e5f5e892849 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 501ae7037dc4a2e910c0e0aafe534e5f5e892849 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 501ae70

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 501ae7037dc4a2e910c0e0aafe534e5f5e892849 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 14:51
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Request changes

Why: the decider found 2 problems and estimates an 80% chance that this change introduces a defect, at or above the 80% level where it asks for changes.

I'm requesting changes: this head does not ship one of the two fixes the PR describes. The shields are in: the three pop-then-shutdown teardowns are now wrapped in asyncio.shield, and the new test covers that. The second fix is missing. The description says the guard arms keep the turn permit until sessions.remove pops the session, and lists tests for that. In the code, sessions.release still runs before both guard arms (eager_spawn.py:1050), the parked-claim tests are absent, and the existing test still asserts release-before-teardown. So a real send queued on the permit can be handed the stale session, and then the shielded remove shuts it down. That is the hazard the GPT lane flagged; I checked it against the code. The proof in the description was run on a different commit (f7052d4ac990) than this head (e2992d3).

Problems found:

  • The description's section "The permit goes with the session" says the two guard arms keep the turn permit until sessions.remove pops the session, and that each path that keeps the session releases the permit itself. The code at this head does not do that: the permit is still released before the st (src/kiro_crew/dashboard/chat_turn/eager_spawn.py, medium)
  • The description says the head has tests that a parked claim is served a fresh provider, and that the guard-arm tests assert the task makes no release call of its own. Neither is in the diff, and the existing test still asserts the old release-before-teardown order. The proof in the description was (test/test_eager_spawn.py, medium)

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit e2992d3c58 · 73 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

A chat could refuse every send with "This session is loaded in another
program, kirocrew (pid N), which owns its execution." after a background
start of that chat was cancelled mid-teardown. Three teardowns inside the
background-start task pop the session from the registry first and stop its
process only afterwards: the two stale arms of _spawn_admitted_prefetch and
the deferred project reset it applies first. A cancel landing between the
two steps abandoned the teardown, so the agent process kept running and
kept kiro-cli's execution lease on the chat's session while the gateway no
longer knew it existed.

Those three teardowns now run under asyncio.shield, as the prefetch TTL and
the over-cap eviction in the same file already do, so the cancel still ends
the task while the teardown completes. A cancelled deferred reset leaves its
flag armed, as any failed consume does, so the next consume applies it.

The two stale arms also keep the session's turn permit until remove() pops
the session, which releases it in the same lock hold: a claim parked on that
permit now wakes to a gone session and starts its own, instead of taking the
stale one the teardown is about to kill.

Part of #35669
@bolichen97
bolichen97 force-pushed the fix/35669-crew-switch-lease branch from e2992d3 to 501ae70 Compare November 17, 2026 11:50
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/dashboard/chat_turn/eager_spawn.py, test/test_eager_spawn.py and test/test_mate_conductor_parity.py since this head went green (c22557c -> df31a4c); CI re-running.

The red Backend Tests (3.12, 3) and Backend Tests (Windows) (3) on the previous head were main's, not this PR's: test/test_mate_conductor_parity.py::test_every_shared_rule_has_an_ultra_counterpart_or_a_named_gap failed with shared rules with no Ultra Mode counterpart: ['goal_plan'], and #36652 fixed it on main after this PR's base. This head sits on a base that carries that fix. Main's changes next to this PR's lines (the early history read of #37595, the app-slot early return of #37804, and reset re-raising an absorbed cancel in #37984) stay as main wrote them; the patch itself is unchanged. Coverage Gate was red only because it fails closed on the backend shard, and Sensitive Change Review was red only because its run hit the API rate limit reading the PR.

[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the two stale guard arms keep the session's turn permit until sessions.remove pops the session and releases it in that same lock hold.

  • Keep the permit until stale-session removal pops the session (span=26cc9b6e03a3)

fixed in 501ae70: the two stale arms no longer release the permit before the teardown. sessions.release is now called only on the paths that keep the session (another creator won, and the kept-session path). On the two arms that remove it, the permit is left to remove, which calls retire_turn_permit in the SAME lock hold as the registry pop, before any await.
Why that closes it: a claim parked on the permit can no longer be handed the stale session at all. It wakes only after the pop, fails the self._sessions.get(key) is session recheck, gives its own permit back and re-enters get_or_create, so it cold-starts with the slot's current bindings instead of having its provider shut down under it.
Both removals named are covered, not just the second: the same change applies to the vanished-slot arm and the bindings-changed arm.
Pinned by test_a_claim_parked_on_a_stale_session_starts_its_own[vanished|bindings], which drives a real SessionManager and a real parked claim. With the permit released ahead of the teardown both cases fail at assert provider is not stale["provider"].

[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The recommendation was right about e2992d3c5897: the description was published ahead of the push that carries it. The new head 501ae7037dc4 contains what the description says. The two guard arms no longer call sessions.release before the teardown, and remove gives the permit back in the lock hold that pops the session. test_a_claim_parked_on_a_stale_session_starts_its_own[vanished|bindings] runs a real SessionManager with a parked claim. The two guard-arm tests assert release is not called. This PR's change and the proof commit differ by one docstring paragraph, with no executable line in it; main's newer code next to it (#37595, #37804, #37984) is unchanged by this PR.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 17, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This head (501ae70) puts asyncio.shield around the two stale-arm sessions.remove calls and the deferred sessions.reset, so a teardown that has started still ends its process after a cancel. It also moves sessions.release onto the paths that keep the session. We checked the permit path against the real SessionManager: remove retires the permit and wakes a parked claim while it still holds the lock, before any await. We also confirmed that each changed line has a test that fails when the line is reverted. PR Readiness is green on this head.
The fix adds no new mechanism. It reuses the shield pattern this module already applies, and test_a_claim_parked_on_a_stale_session_starts_its_own tests the parked-claim hazard with a real claim waiting on the semaphore.
Optional:

  • The Proof line names f7052d4ac990, which is not on GitHub. Rerun prove.py on 501ae70, and add the two test_a_claim_parked_on_a_stale_session_starts_its_own ids to the list.
  • In the pending_reset case, add assert slot._pending_reset_history_key == "dashboard:t1" after the cancel, so a test covers "the flag stays armed".
  • A cancelled consume can still finish its reset and leave the flag armed, so the next consume restarts a session that is already correct. This is safe as documented. A done-callback on the inner task could clear it in a follow-up.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Nov 17, 2026
@bolichen97
bolichen97 merged commit 3de505c into main Nov 17, 2026
100 checks passed
@bolichen97
bolichen97 deleted the fix/35669-crew-switch-lease branch November 17, 2026 13:48
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants