Repository navigation
fix(crew-mode): keep the turn a pause owes when its write fails or memory prepares - #32524
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Both fixes target the root cause: a failed write is kept and retried, and memory readiness is checked before the record drops. Each change can be reverted on its own, and nothing is taken away. Notes
[DESIGN-REVIEWED] ccd50f7 Description read[DESCRIPTION-READ] 0b3b747b841c904fde140afe0b2341715629a9c1167175bcee8beb17df4e6bee That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing to check. What this change shipsInventory (6 items) — 6 justifiedIntent: when a person presses Resume all, every worker that Pause all stopped should carry on, even if the pause-file write failed or memory is still loading after a restart. This is a FIX (#30183, with tests that fail on base).
[FIRST-PRINCIPLES-REVIEWED] ccd50f7 Description read[DESCRIPTION-READ] 7f2d3ac54f56b146150ea069364a383ff42f8fe92499b9a098f59dcafaa2fcf8 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
8b6be59 to
19d99e7
Compare
|
|
[operator: bolichen97] The four Backend Tests reds (shards 5 and 6 on Linux and Windows) and the Coverage Gate are main's: |
19d99e7 to
ccd50f7
Compare
|
[operator: bolichen97] Rebased onto main 18ec128 at ccd50f7 to pick up #32606, which fixes main's Backend shard 5 and 6 reds. The PR diff is unchanged (range-diff Local review before the push: Opus found no blocking findings. Local GPT raised one Medium: in the size-cap case, the in-memory map stays over the 256 KiB cap, so a later write that does not shrink it enough (for example Resume all on another crew) fails where main would have succeeded. It needs a pause file at the cap. A fix that keeps the refused entries out of the stored map is drafted, but each round of it opened a further rollback edge case, so it is not in this head. It stays a follow-up unless a maintainer wants it here. |
|
[operator: bolichen97] Local review finding on ccd50f7 (size-cap case: the in-memory map stays over the 256 KiB cap, so a later Resume all on another crew can fail) is tracked in #32819 as a follow-up. A fix needs more than this PR's scope: a first attempt showed three edge cases, listed in the issue. This head keeps the change as reviewed in CI. |
Problem / Motivation
Goal: A turn that Pause all ended is still continued by Resume all, even when its pause-file write fails or memory is still being prepared after a restart.
Two rare paths drop a turn that Pause all owes:
crew_pause.owe_turn). If that file write fails (a disk error, or the 256 KiB pause-file cap), nothing records the turn. Resume all then treats the worker as idle and never continues it.held, so the pause is dropped. When memory is ready, the parked drain can drop that card as stale, and it sends no Continue. The worker sits idle and is no longer paused.Why it matters
Both are rare: one needs a write failure during a pause, the other a restart, pending memory preparation and a channel-linked worker at once. When they happen, the worker's stopped work is silently not picked up again, and the person has to notice the idle worker and press Continue.
Not a goal
What changed (motivation → approach → change)
crew_paused409, record unchangedowe_turninsrc/kiro_crew/crew_pause.pynow catches a refused write, publishes the obligation in memory and marks the records unsaved. Every pause write stores the whole map, and the gate's next refusal retries while they are unsaved, so the first write that succeeds stores it. A restart before then forgets it, as main always did; the code notes that limit.memory_ready_for_drainindashboard/handlers/crew_pause.pywaits the same bounded grace a turn's admission waits (wait_for_memory_preparation), and only while memory is still preparing; before startup publishes its task, a preparing startup counts as not ready. Resume all calls it before it drops the pause record, when the record has stopped members. The member Continue indashboard/chat_api/continue_turn.pycalls it before it releases a stopped member, queued card or not (a Continue with no card queues its own recovery entry, which parks the same way). Both wait while the stored record still owes the member its hand-back, and change nothing when memory is still not ready. Once they go on, a card never parks, so it starts or drops while the hand-back can still Continue it.Backwards compatibility
Compatible: routes and response shapes stay the same. Resume all can answer one new refusal,
503 memory_preparing, only while memory is still preparing after a restart; the Board already shows any refused Resume as "could not resume".Removes nothing: no route, record field, outcome or function is taken away.
Tests
test/test_crew_pause.py::test_a_turn_owed_when_its_write_is_refused_is_still_continued(disk full, size cap): the owed-turn write fails, Resume all still finds the worker owed, and once the disk recovers the next refusal stores it so a restart reads it back.test/test_crew_pause.py::test_resume_while_memory_prepares_never_parks_a_stopped_workers_card: memory not ready within the grace, or its task not yet published while startup prepares → no drain, no Continue, and the stored record is unchanged across a restart; ready within the grace → the card is sent as before.test/test_crew_pause.py::test_a_members_continue_while_memory_prepares_keeps_it_paused(with and without a queued card): a member's own Continue neither drains nor continues, and the member stays on the record.Determinism: the new tests passed 10 repeats in a row and once in reversed order; they use no sleeps (a future resolved with
call_soon, a zero grace).Proof: not provable -- other: reverting every production hunk breaks collection, because the autouse fixture resets the new
_unsavedflag; run by hand on origin/main without that line, 4 of the 5 first cases fail (OSError: disk full,OSError: ... would exceed,200 == 503,200 == 409) and the ready case passes.Manual verification
N/A - unit coverage sufficient: both paths need a failed disk write or a restart mid-preparation, which the tests force directly.
Related Issues
Fixes #30183
The other items of #30183 are already on main: the fork recheck and the Edit & Resend recheck in #29928, the extra Continue and the re-hold in #31001, and the member Continue lock in #31717.
Sibling sweep
Swept: git grep -n "owe_turn|drain_held_queue" -- src/
Sibling: src/kiro_crew/dashboard/chat_turn/turn_dispatch_gates.py -- not affected -- calls owe_turn, which no longer raises for a refused write
Sibling: src/kiro_crew/dashboard/chat_api/chat_send.py -- not affected -- its compaction-hold path calls the same owe_turn
Sibling: src/kiro_crew/dashboard/chat_api/continue_turn.py -- changed here
Pattern harvest
Not generalizable: both are hand-back obligations specific to the pause record, which no other feature keeps.
Checklist