Skip to content

fix(crew-mode): keep the turn a pause owes when its write fails or memory prepares - #32524

Merged
bolichen97 merged 1 commit into
mainfrom
fix/pause-deferred-30183
Nov 8, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/pause-deferred-30183

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A turn that Pause all ended is still continued by Resume all, even when its pause-file write fails or memory is still being prepared after a restart.

Two rare paths drop a turn that Pause all owes:

  • The dispatch gate ends a held turn and records it as owed (crew_pause.owe_turn). If that file write fails (a disk error, or the 256 KiB pause-file cap), nothing records the turn. Resume all then treats the worker as idle and never continues it.
  • After a restart, Resume all (or a stopped member's own Continue) sends the member's restored card while memory is still being prepared. The drain parks, and the answer is held, so the pause is dropped. When memory is ready, the parked drain can drop that card as stale, and it sends no Continue. The worker sits idle and is no longer paused.

Why it matters

Both are rare: one needs a write failure during a pause, the other a restart, pending memory preparation and a channel-linked worker at once. When they happen, the worker's stopped work is silently not picked up again, and the person has to notice the idle worker and press Continue.

Not a goal

  • Removing the per-entrypoint pause checks. Each one does something the dispatch gate cannot: the send refuses before anything is recorded, the queue paths keep a message queued instead of starting and ending a turn, and a patrol wake is refused as busy so it spends no cycle.
  • The Ultra Mode default-agent decision, which is a maintainer call.
  • Any change to what Pause all, Resume all or a member's Continue does when the write succeeds and memory is ready.

What changed (motivation → approach → change)

Path Before After
Owed-turn write fails obligation lost, worker idle after Resume all kept in memory, marked unsaved, retried; Resume all continues it
Resume all, memory preparing card parks, pause dropped, card may drop with no Continue waits a turn's grace first; still not ready → 503, pause left as stored
Member's own Continue, memory preparing same as above same wait first; still not ready → existing crew_paused 409, record unchanged

owe_turn in src/kiro_crew/crew_pause.py now catches a refused write, publishes the obligation in memory and marks the records unsaved. Every pause write stores the whole map, and the gate's next refusal retries while they are unsaved, so the first write that succeeds stores it. A restart before then forgets it, as main always did; the code notes that limit.

memory_ready_for_drain in dashboard/handlers/crew_pause.py waits the same bounded grace a turn's admission waits (wait_for_memory_preparation), and only while memory is still preparing; before startup publishes its task, a preparing startup counts as not ready. Resume all calls it before it drops the pause record, when the record has stopped members. The member Continue in dashboard/chat_api/continue_turn.py calls it before it releases a stopped member, queued card or not (a Continue with no card queues its own recovery entry, which parks the same way). Both wait while the stored record still owes the member its hand-back, and change nothing when memory is still not ready. Once they go on, a card never parks, so it starts or drops while the hand-back can still Continue it.

Backwards compatibility

Compatible: routes and response shapes stay the same. Resume all can answer one new refusal, 503 memory_preparing, only while memory is still preparing after a restart; the Board already shows any refused Resume as "could not resume".

Removes nothing: no route, record field, outcome or function is taken away.

Tests

  • test/test_crew_pause.py::test_a_turn_owed_when_its_write_is_refused_is_still_continued (disk full, size cap): the owed-turn write fails, Resume all still finds the worker owed, and once the disk recovers the next refusal stores it so a restart reads it back.
  • test/test_crew_pause.py::test_resume_while_memory_prepares_never_parks_a_stopped_workers_card: memory not ready within the grace, or its task not yet published while startup prepares → no drain, no Continue, and the stored record is unchanged across a restart; ready within the grace → the card is sent as before.
  • test/test_crew_pause.py::test_a_members_continue_while_memory_prepares_keeps_it_paused (with and without a queued card): a member's own Continue neither drains nor continues, and the member stays on the record.

Determinism: the new tests passed 10 repeats in a row and once in reversed order; they use no sleeps (a future resolved with call_soon, a zero grace).

Proof: not provable -- other: reverting every production hunk breaks collection, because the autouse fixture resets the new _unsaved flag; run by hand on origin/main without that line, 4 of the 5 first cases fail (OSError: disk full, OSError: ... would exceed, 200 == 503, 200 == 409) and the ready case passes.

Manual verification

N/A - unit coverage sufficient: both paths need a failed disk write or a restart mid-preparation, which the tests force directly.

Related Issues

Fixes #30183

The other items of #30183 are already on main: the fork recheck and the Edit & Resend recheck in #29928, the extra Continue and the re-hold in #31001, and the member Continue lock in #31717.

Sibling sweep

Swept: git grep -n "owe_turn|drain_held_queue" -- src/
Sibling: src/kiro_crew/dashboard/chat_turn/turn_dispatch_gates.py -- not affected -- calls owe_turn, which no longer raises for a refused write
Sibling: src/kiro_crew/dashboard/chat_api/chat_send.py -- not affected -- its compaction-hold path calls the same owe_turn
Sibling: src/kiro_crew/dashboard/chat_api/continue_turn.py -- changed here

Pattern harvest

Not generalizable: both are hand-back obligations specific to the pause record, which no other feature keeps.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ccd50f7ce82e1fe6e8f9556af65597a50fa4142d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ccd50f7ce82e1fe6e8f9556af65597a50fa4142d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Both fixes target the root cause: a failed write is kept and retried, and memory readiness is checked before the record drops. Each change can be reverted on its own, and nothing is taken away.

Notes

  • Resume all and a member's Continue now wait up to MEMORY_ADMISSION_WAIT_SECONDS (30 s) while holding _root_lock. During that wait, a Pause all on the same root blocks and the HTTP request hangs with no reply. A UI that keeps showing a spinner during the wait would cover this.
  • After a refused write, a successful retry only happens when this gate or another pause write runs again. If no gate fires again before a restart, the owed turn is lost. That is the limit the description states, and it is acceptable.

[DESIGN-REVIEWED] ccd50f7

Description read

[DESCRIPTION-READ] 0b3b747b841c904fde140afe0b2341715629a9c1167175bcee8beb17df4e6bee

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of ccd50f7ce82e1fe6e8f9556af65597a50fa4142d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (6 items) — 6 justified

Intent: when a person presses Resume all, every worker that Pause all stopped should carry on, even if the pause-file write failed or memory is still loading after a restart. This is a FIX (#30183, with tests that fail on base).

  1. A turn owed after Pause all ends it is no longer lost when the pause-file write fails or hits the size cap. It is kept in memory and saved on the next write. — justified
  2. Resume all can now answer 503 memory_preparing and leave the pause untouched while memory is loading and the pause has stopped members. — justified
  3. A stopped member's own Continue now answers the existing crew_paused 409 while memory is loading, and the member stays paused. — justified
  4. Resume all and a member's Continue first wait the same short grace a turn waits for memory. — justified
  5. New helper memory_ready_for_drain in dashboard/handlers/crew_pause.py, used by 2 callers (_resume and api_chat_slot_continue). — justified
  6. Code comments now say a card that waits is held back by sub-agents or an open editor, not by memory loading. — justified

[FIRST-PRINCIPLES-REVIEWED] ccd50f7

Description read

[DESCRIPTION-READ] 7f2d3ac54f56b146150ea069364a383ff42f8fe92499b9a098f59dcafaa2fcf8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ccd50f7ce82e1fe6e8f9556af65597a50fa4142d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ccd50f7

Verdict parsed from the review's SHA-scoped output markers for commit ccd50f7ce82e1fe6e8f9556af65597a50fa4142d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ccd50f7ce82e1fe6e8f9556af65597a50fa4142d false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ccd50f7ce82e1fe6e8f9556af65597a50fa4142d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ccd50f7

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ccd50f7ce82e1fe6e8f9556af65597a50fa4142d false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 7, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 7, 2026 19:14
@bolichen97
bolichen97 force-pushed the fix/pause-deferred-30183 branch from 8b6be59 to 19d99e7 Compare November 7, 2026 19:48
@bolichen97

bolichen97 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=0bafbcd1c6fe An unpublished startup task releases the pause before memory is ready

fixed in 19d99e7: memory_ready_for_drain returns not memory_startup_preparing() when no task is published, so Resume all answers 503 and a member's Continue answers crew_paused with the stored record unchanged.
test_resume_while_memory_prepares_never_parks_a_stopped_workers_card[task_unpublished] fails without the change (200 == 503) and passes with it.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The four Backend Tests reds (shards 5 and 6 on Linux and Windows) and the Coverage Gate are main's: test_dashboard_server_composition_contract.py::test_the_lazy_imports_stay_inside_the_functions_that_need_them (136 == 135) and test_held_note_turn_starts.py (nudge_dashboard_fire.py) fail the same way on this PR's base b3be506 without this change. #32606 fixes both; this PR rebases onto it once it merges.

@bolichen97
bolichen97 force-pushed the fix/pause-deferred-30183 branch from 19d99e7 to ccd50f7 Compare November 7, 2026 23:11
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Rebased onto main 18ec128 at ccd50f7 to pick up #32606, which fixes main's Backend shard 5 and 6 reds. The PR diff is unchanged (range-diff =).

Local review before the push: Opus found no blocking findings. Local GPT raised one Medium: in the size-cap case, the in-memory map stays over the 256 KiB cap, so a later write that does not shrink it enough (for example Resume all on another crew) fails where main would have succeeded. It needs a pause file at the cap. A fix that keeps the refused entries out of the stored map is drafted, but each round of it opened a further rollback edge case, so it is not in this head. It stays a follow-up unless a maintainer wants it here.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 8, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Local review finding on ccd50f7 (size-cap case: the in-memory map stays over the 256 KiB cap, so a later Resume all on another crew can fail) is tracked in #32819 as a follow-up. A fix needs more than this PR's scope: a first attempt showed three edge cases, listed in the issue. This head keeps the change as reviewed in CI.

@bolichen97
bolichen97 merged commit 50c6c11 into main Nov 8, 2026
97 of 104 checks passed
@bolichen97
bolichen97 deleted the fix/pause-deferred-30183 branch November 8, 2026 01:46
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ultra Mode pause: deferred race and tidy-up findings from #29928

2 participants