Repository navigation
fix(chat): let a cancelled background start finish its session teardown - #36664
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Every item is tied to the reproduced leak (#35669), and each one fixes the mechanism, not the symptom. The cause they leave alone, that Notes
What this change shipsInventory (5 items) — 5 justifiedIntent: stop a chat getting locked out by an agent process that a cancelled background start left running. FIX.
[FIRST-PRINCIPLES-REVIEWED] 501ae70 Description read[DESCRIPTION-READ] 7fe75d97983e00b40b77efeeb77769dfafada2add011a74074551cb298f4fccf That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: shielding the three pop-then-shutdown teardowns follows the pattern Notes
[DESIGN-REVIEWED] 501ae70 Description read[DESCRIPTION-READ] b5ae33e37485d5979544bfbca0449c20bb6f7f4ffa7ef1f5605a83b9a0d337ac That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: Request changesWhy: the decider found 2 problems and estimates an 80% chance that this change introduces a defect, at or above the 80% level where it asks for changes. I'm requesting changes: this head does not ship one of the two fixes the PR describes. The shields are in: the three pop-then-shutdown teardowns are now wrapped in Problems found:
An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
A chat could refuse every send with "This session is loaded in another program, kirocrew (pid N), which owns its execution." after a background start of that chat was cancelled mid-teardown. Three teardowns inside the background-start task pop the session from the registry first and stop its process only afterwards: the two stale arms of _spawn_admitted_prefetch and the deferred project reset it applies first. A cancel landing between the two steps abandoned the teardown, so the agent process kept running and kept kiro-cli's execution lease on the chat's session while the gateway no longer knew it existed. Those three teardowns now run under asyncio.shield, as the prefetch TTL and the over-cap eviction in the same file already do, so the cancel still ends the task while the teardown completes. A cancelled deferred reset leaves its flag armed, as any failed consume does, so the next consume applies it. The two stale arms also keep the session's turn permit until remove() pops the session, which releases it in the same lock hold: a claim parked on that permit now wakes to a gone session and starts its own, instead of taking the stale one the teardown is about to kill. Part of #35669
e2992d3 to
501ae70
Compare
|
Rebased: main moved in src/kiro_crew/dashboard/chat_turn/eager_spawn.py, test/test_eager_spawn.py and test/test_mate_conductor_parity.py since this head went green (c22557c -> df31a4c); CI re-running. The red [operator: bolichen97] |
|
self-added: yes
[operator: bolichen97] |
|
[operator: bolichen97] The recommendation was right about |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
This head (501ae70) puts asyncio.shield around the two stale-arm sessions.remove calls and the deferred sessions.reset, so a teardown that has started still ends its process after a cancel. It also moves sessions.release onto the paths that keep the session. We checked the permit path against the real SessionManager: remove retires the permit and wakes a parked claim while it still holds the lock, before any await. We also confirmed that each changed line has a test that fails when the line is reverted. PR Readiness is green on this head.
The fix adds no new mechanism. It reuses the shield pattern this module already applies, and test_a_claim_parked_on_a_stale_session_starts_its_own tests the parked-claim hazard with a real claim waiting on the semaphore.
Optional:
- The Proof line names f7052d4ac990, which is not on GitHub. Rerun
prove.pyon 501ae70, and add the twotest_a_claim_parked_on_a_stale_session_starts_its_ownids to the list. - In the
pending_resetcase, addassert slot._pending_reset_history_key == "dashboard:t1"after the cancel, so a test covers "the flag stays armed". - A cancelled consume can still finish its reset and leave the flag armed, so the next consume restarts a session that is already correct. This is safe as documented. A done-callback on the inner task could clear it in a follow-up.
Problem / Motivation
Goal: A chat is no longer locked out with "This session is loaded in another program" by an agent process that a cancelled background start left running.
After a chat's background start was cancelled at the wrong moment, every later send on that chat failed with
This session is loaded in another program, kirocrew (pid N), which owns its execution.The process holding the chat's session was one the gateway had already dropped from its registry, so no reset, sweep or agent switch could reach it. Only a gateway restart or a manual kill freed the chat. The report in #35669 hit it right after an Ultra Mode (Crew Mode) toggle on an existing chat.Why it matters
The chat is unusable until the person finds and kills the stray process or restarts the gateway, and nothing on screen says why. Any chat can hit it: a background start (a focus prefetch or an eager spawn) is cancelled by the next signal for its chat (a focus or blur frame, another start), and moving between chats or changing a chat's agent or model sends exactly those.
Not a goal
SessionManager.removeandSessionManager.resetthemselves finish when cancelled, for every caller.What changed (motivation → approach → change)
Symptom → cause. A background start of a chat runs in one task, and every later signal for the chat cancels it. Three teardowns inside that task take the session out of the registry first and only then stop its process: the two guard arms of
_spawn_admitted_prefetch(the slot vanished, or its bindings changed while the agent was starting) and the deferred project reset it applies first (_consume_pending_reset). A cancel that lands between those two steps abandons the teardown. The agent process keeps running, still holding the chat's resumed session and kiro-cli's execution lease on it, and nothing in the gateway knows it exists. The next start of that chat loads the same session and every send is refused.Change. Those three teardowns run under
asyncio.shield(...), the way the prefetch TTL (_prefetch_ttl) and the over-cap eviction (_evict_prefetches_beyond) indashboard/chat_turn/eager_spawn.pyalready wrap theirs. The teardown runs to completion while the cancel still ends the task. A cancelled deferred reset indashboard/chat_turn/session_reset.pyleaves its flag armed, as any failed consume does, so the next consume applies it.The permit goes with the session. A teardown that runs to its end shuts a provider down, and a real claim can be parked on that session's turn permit. So the two guard arms keep the permit until
sessions.removepops the session, which releases it in the same lock hold as the pop: the parked claim wakes, finds the session gone, and starts its own with the slot's current bindings. Each path that keeps the session releases the permit itself. No new helper, flag or retry.Backwards compatibility
Compatible: the same sessions are torn down on the same paths. A cancel arriving mid-teardown no longer stops it halfway, and a claim parked on a stale session's permit is served a session carrying the slot's current bindings instead of the one the teardown is about to kill.
Removes nothing: no path, record, key, validator or function is removed or narrowed; an already-started teardown is only allowed to finish.
Tests
test/test_eager_spawn.py::TestEagerSpawn::test_a_cancel_mid_teardown_still_ends_the_process[vanished|bindings|pending_reset]drives_eager_spawninto each of the three teardowns with a fake whose teardown pops, then waits on an event before its shutdown step. It cancels the task while the teardown waits, checks the task ended cancelled, releases the event, and checks the shutdown step ran. With the shields reverted each case fails withassert {'shut_down': False} == {'shut_down': True}.test/test_eager_spawn.py::TestEagerSpawn::test_a_claim_parked_on_a_stale_session_starts_its_own[vanished|bindings]runs a realSessionManager: the background start creates the session, a real claim parks on its permit, the guard arm fires, and the test checks the claim was served a different provider, the stale provider was shut down, and the registry holds the claim's session. With the permit released ahead of the teardown both cases fail atassert provider is not stale["provider"].releasecall of its own.Proof: PROVEN test/test_eager_spawn.py::TestEagerSpawn::test_removes_session_when_slot_deleted_mid_handshake, test/test_eager_spawn.py::TestEagerSpawn::test_removes_session_when_bindings_change_mid_handshake, test/test_eager_spawn.py::TestEagerSpawn::test_a_cancel_mid_teardown_still_ends_the_process[vanished], test/test_eager_spawn.py::TestEagerSpawn::test_a_cancel_mid_teardown_still_ends_the_process[bindings], test/test_eager_spawn.py::TestEagerSpawn::test_a_cancel_mid_teardown_still_ends_the_process[pending_reset] (head f7052d4ac990)
prove.pyran on the verification host at commitf7052d4ac990, this change on basec4c95ce148;--per-hunkexits 0 there too and names no uncaught hunk. This head carries the same change plus one paragraph of_eager_spawn's docstring, with no executable line in it, on a newermain. On that base the verification host ran the related backend tests (1,258 passed), the test files naming_eager_spawnor_spawn_admitted_prefetch(2,919 passed;test_eager_spawn.pyalone 53), the session-reset tests (765 passed), the tests of main's neighbouring changes toreset, app sessions and the session memory plan (#37984, #37804, #36872; 406 passed), mypy, semgrep and the flake-pattern ratchet, all green.Determinism: the five new cases passed 20 runs out of 20 at
-n0. A shuffled order oftest_eager_spawn.py,test_eager_spawn_shutdown.py,test_eager_spawn_resume_prefetch.pyandtest_chat_runner_coverage_pending.py(133 ids, seed 35669,-n0from an@file) passed, and those four files passed again at-n 4 --dist loadgroup --max-worker-restart=2. Every wait is an event-boundedawait_until; nothing sleeps.Manual verification
On Linux (Ubuntu 24.04.5 LTS, x86_64), against the real gateway (
kiro_crew.testing.harness.spawn_feature_gateway, isolatedKIROCREW_HOME, OS-assigned loopback port, Ultra Mode preview on). kiro-cli cannot create a session there without a sign-in, so the gateway ran against a stand-in kiro-cli: the in-treefake_acp_backendplus kiro-cli's per-session execution lease, which refuses a second process with kiro-cli's own message. Sequence: send on a plain chat, a Crew Mode switch, a focus frame that starts the chat's resume prefetch (the stand-in holds itssession/loadreply 3 s), a model pick landing mid-start so the guard arm fires, the dashboard's blur frame cancelling the prefetch, then a second send. One throwawaysitecustomize.py(not committed) slowed a single await insideremove(record_session_ended, after the pop, before the shutdown) by 4 s, so a WebSocket frame can land in a gap that is otherwise a few milliseconds wide. No product logic was changed for the run.Before, on the base:
After, on this change, the same run:
A plain Crew Mode switch with no slowed await is answered on both, with no refusal.
Related Issues
Part of #35669
Sibling sweep
Swept: git grep -nE "sessions.(remove|reset|remove_if_unclaimed)(" -- src/kiro_crew/dashboard/
Sibling: src/kiro_crew/dashboard/chat_turn/eager_spawn.py -- changed here
Sibling: src/kiro_crew/dashboard/chat_turn/session_reset.py -- changed here
Sibling: src/kiro_crew/dashboard/chat_turn/app_agent_binding.py:121 -- not affected -- turn admission in the turn task, which no chat signal cancels
Sibling: src/kiro_crew/dashboard/chat_handlers.py:993 -- not affected -- switch and reload request handlers, which the server does not cancel
Sibling: src/kiro_crew/dashboard/chat_api/slot_lifecycle.py -- not affected -- the slot close handler (:1208) and the slot cleanup sweep (:1760), which no chat signal cancels
Sibling: src/kiro_crew/dashboard/handlers/session_environments.py -- not affected -- both removals (:731, :840) run in request handlers, which the server does not cancel
Sibling: src/kiro_crew/dashboard/handlers/taskrunner.py:1499 -- not affected -- request handler, which the server does not cancel
Sibling: src/kiro_crew/dashboard/handlers/hooks.py:1320 -- not affected -- request handler, which the server does not cancel
In
eager_spawn.pythe two guard-arm removals change; the over-cap eviction (:341) and the prefetch TTL (:1225) already shield theirs, and both reach the session throughremove_if_unclaimed, which refuses under the lock while a permit is held, so a claimant that won one keeps its session. Insession_reset.pythe deferred project reset (:389) changes; it holds no permit of its own andreset(skip_if_busy=True)declines under the lock while one is held. Its conversation discard (:480) is never reached from the background-start task, which passes noallow_discard: only a turn cancelled by a slot delete reaches it, a different trigger. Its end-of-turn reset (:628) is turn cleanup that no chat signal cancels.Pattern harvest
Rule candidate: review-prompt
Pattern: a session teardown that pops the registry entry and then awaits the shutdown (
sessions.remove,reset,remove_if_unclaimed) leaks the process it popped when it is awaited unshielded in a task that a later signal cancels, and hands a parked claimant the session it is about to kill when the turn permit is released ahead of it.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)