Skip to content

fix(crew-mode): keep a cap-refused owed turn out of the stored pause map - #35526

Open
dodgeblaster wants to merge 2 commits into
mainfrom
fix/32819-pause-cap-owed
Open

dodgeblaster wants to merge 2 commits into
mainfrom
fix/32819-pause-cap-owed

Conversation

@dodgeblaster

@dodgeblaster dodgeblaster commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When the pause file is at its 256 KiB size cap, Resume all and a member's own Continue still work while an owed turn the file refused is kept in memory.

crew_pause.owe_turn keeps an owed turn in memory when the pause-file write is refused, so Resume all still continues it (#32524). When the refusal came from the size cap, that entry stayed in the map every later write stores, so the map stayed over the cap. Resume all on a different, smaller crew, or a member's own Continue (release), then failed with OSError and that crew stayed paused.

Why it matters

Only reachable with a pause file already at the cap (thousands of member keys across paused crews), but the failure is a crew that will not resume until some other write happens to free enough bytes.

Not a goal

  • Storing a cap-refused owed turn somewhere else so it survives a gateway restart (a restart before a write stores it still forgets it, as before).
  • Raising or changing the 256 KiB cap.
  • Changing the disk-full path beyond what the shared write path does already (the next write still retries storing the turn).

What changed (motivation → approach → change)

Symptom: a write that shrinks the pause file failed at the cap. Cause: the cap-refused owed turn lived inside the stored map, so every write tried to store it again.

Approach: keep those owed turns beside the stored records, never inside them, and never for an entry the file already stores. Readers see both. Each write first tries to store both, and when the cap refuses that, it stores the records alone, so a write that shrinks the file always fits. The issue's three edge cases:

  1. A write that removes another crew succeeds while a memory-only turn exists.
  2. A member's own Continue takes its memory-only turns with it, so they cannot shadow a later Pause all's fresh stop of that member.
  3. A refused Continue (rehold) puts a memory-only turn back in memory, including when the release had ended the whole record. A refused Resume all already stores the record's memory-only turns as resuming, so the hand-back write holds them again.

release and resume can be undone, so they store memory-only turns only when the records their undo writes back would fit with them too. When the cap refused a turn, the room they free stays free for the undo; without this, a Continue that freed room for another member's memory-only turn could leave its own rehold over the cap. A turn only the disk refused still fits, so the next write stores it, as on main.

Backwards compatibility

Compatible: the file format is unchanged, and release() adds a memory_only key to its result only when it took a memory-only turn. continue_turn.py passes that result straight back to rehold() and reads only fields.

Removes nothing: no field, key, function or stored record is removed; memory-only turns are added beside the stored map.

Tests

test/test_crew_pause.py adds:

  • test_a_write_that_shrinks_the_file_fits_while_an_owed_turn_is_kept_in_memory: Resume all on a smaller crew and a member's Continue both succeed at the cap (edge case 1).
  • test_resume_all_continues_an_owed_turn_kept_in_memory: the memory-only turn is continued and stored as resuming.
  • test_a_released_owed_turn_does_not_strip_a_later_pauses_fresh_stop (edge case 2).
  • test_a_refused_continue_puts_back_an_owed_turn_kept_in_memory[record kept / record ended] (edge case 3, rehold).
  • test_a_refused_resume_holds_an_owed_turn_kept_in_memory_again (edge case 3, the Resume all hand-back).
  • test_an_owed_turn_kept_in_memory_is_stored_once_the_file_has_room.
  • test_a_refused_continue_fits_back_beside_another_owed_turn_kept_in_memory: a Continue that frees room keeps it for its own undo.
  • test_an_owed_turn_the_disk_refused_is_stored_by_the_next_continue_or_resume[continue / resume_all]: a turn the disk refused is stored by the next write even when that write can be undone.
  • test_a_stop_a_later_pause_stored_stays_stored_through_a_refused_continue: a stop a later Pause all stored is never treated as memory-only, so a refused Continue keeps it in the file.

Proof: PROVEN test/test_crew_pause.py::test_an_owed_turn_kept_in_memory_is_stored_once_the_file_has_room, test/test_crew_pause.py::test_a_refused_continue_fits_back_beside_another_owed_turn_kept_in_memory (head dc23b9e)

I also tested each mechanism with its own mutant. Dropping the records-only fallback fails 5 of the new tests. Leaving a released member's memory-only turns behind fails the fresh-stop test. Storing a reheld memory-only turn fails the rehold test. Letting release and resume always store memory-only turns fails 2 tests; never storing them there fails both disk-refused tests. Keeping a memory-only entry the file already stores fails the stored-stop test on its final assertion (['w1'] == ['w1', 'w2']). Determinism: the new tests pass in 5 repeated runs, and the whole file (95 tests) passes in a seeded shuffled order.

Nearby inputs to the same write path: pause, settle_stopped, keep_pending and forget_resuming operate on the stored records and are covered by the existing 84 tests, all passing unchanged. The disk-full variant of owe_turn is covered by the existing test_a_turn_owed_when_its_write_is_refused_is_still_continued[disk_full].

Manual verification

N/A: the defect lives in the pause store's in-memory and file state, which the unit tests drive directly with a reduced cap.

Related Issues

Fixes #32819

Sibling sweep

Swept: git grep -n "MAX_FILE_BYTES" -- src/kiro_crew/crew_pause.py src/kiro_crew/dashboard/
Sibling: src/kiro_crew/crew_pause.py:410 -- not affected -- a refused hand-back is reported as pause_file_unsaved and stays owed in the file's resuming, so nothing is lost

Pattern harvest

Not generalizable: specific to one store that keeps a cap-refused entry in memory, which no other capped store in the repo does.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 020960a7560fd66ca63bc3c19e0bb6a228e14095 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix works at the cause level: cap-refused owed turns now live beside the stored map instead of inside it. The _spilled undo-reserve is the one item whose need rests on a hypothetical interleaving.

Notes

  • Item 5 — oversized: _spilled plus rollback=True exist for "a later write took room an undo was counting on (another crew's Continue landing while this one's is decided)". That interleaving has no report in the description or Resume all can fail when the pause file is at its size cap and an owed turn is kept in memory #32819. Subtraction: remove _spilled, _present, _without and the rollback branch in _commit. Let undo= reservation alone protect rehold. Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (6 items) — 5 justified

Intent: when the pause file is at its size cap, Resume all and a member's own Continue still unpause crews while the refused owed turn is kept. FIX.

  1. At the cap, Resume all on another crew and a member's Continue succeed instead of raising OSError — justified
  2. Resume all continues a memory-only owed turn and stores it as resuming — justified
  3. A member's Continue takes its memory-only turns along, so a later Pause all's fresh stop survives — justified
  4. A refused Continue (rehold) puts a memory-only turn back in memory — justified
  5. A write that can be undone keeps room free for its undo, and the undo can push a stored once-refused turn back to memory — oversized
  6. release() result gains an optional memory_only key — justified

[FIRST-PRINCIPLES-REVIEWED] 020960a

Description read

[DESCRIPTION-READ] 6803ebf7d69bd582555626eb12333cb029a1f8da1c78d7b0072aa6a066f6df46

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 020960a7560fd66ca63bc3c19e0bb6a228e14095 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 020960a

Verdict parsed from the review's SHA-scoped output markers for commit 020960a7560fd66ca63bc3c19e0bb6a228e14095.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 020960a7560fd66ca63bc3c19e0bb6a228e14095 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — 🔴 changes requested (blocking)

GPT 6.1 found at least one blocking issue that must be resolved before merging 020960a7560fd66ca63bc3c19e0bb6a228e14095. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/crew_pause.py:262 -- Rollback deletes an already persisted owed turn
trimmed = _without(records, spilled)
Trigger: At the cap, B has a memory-only owed turn. A’s last-member Continue releases its record; while A awaits its subagent probe, C’s Continue persists B’s obligation in the freed space. A is then refused, reaching rehold() → _commit(rollback=True).
Outcome: The rollback deletes B’s durable stopped and owed entries. A restart before another write loses that obligation, so Resume all skips B’s interrupted turn when its queue is empty.
Anchor: residual/crash-data-loss-corruption
Fix: Reserve outstanding release-rollback capacity across writers rather than evicting obligations that have become durable.
Sites: src/kiro_crew/crew_pause.py:639; src/kiro_crew/dashboard/chat_api/continue_turn.py:189; src/kiro_crew/dashboard/handlers/crew_pause.py:380

[BLOCK-MERGE] 020960a
[GPT-REVIEWED] 020960a

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 (fenced). The harm is a lost resume obligation. B's interrupted turn is not continued automatically by Resume all; it is visible, and the user can recover it by hand by pressing Continue on B. Nothing is exposed and no ceiling is affected.

Every condition the finding needs is in the code:

  • The pause file has to be at the 256 KiB cap (crew_pause.py:55), and B's owed turn has to be one the file refused.
  • A's Continue has to be refused after the release, which calls rehold (continue_turn.py:189). That runs _commit(rollback=True) (crew_pause.py:639).
  • C's write has to land inside A's probe window. Only then does the trim at crew_pause.py:260-264 move B's now-stored entry back to memory only.
  • The process has to restart before the next write. Any later _commit stores memory-only turns again once they fit (crew_pause.py:250-254).

The trim is the documented trade-off. Without it, the refused undo raises (crew_pause.py:265-268), and A's obligation is lost even from memory.

[ADJUDICATION] 020960a total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 020960a

[ADJUDICATION-FENCED] 020960a fenced=1 flagged=1
FLAG F1 src/kiro_crew/crew_pause.py:262 -- The loss needs a pause file at its 256 KiB cap, three crews' Continues interleaved inside one subagent-probe window, and a restart before any later write re-stores the memory-only turn (crew_pause.py:250); even then the only consequence is one interrupted turn the user must continue by hand, and the alternative (raising at :265) would drop A's obligation instead.
[GPT-ADJUDICATED-FENCED] 020960a

🏷️ Fenced finding(s) machine-flagged as likely edge case

The security fence keeps these findings blocking regardless of adjudication; the only clearance path is a human override. The annotate-only pass judged the condition combination each one requires extreme, and pre-drafted the override rationale below. A repository writer must independently verify a rationale before posting it — it is machine-authored, and a wrong override on a security-class finding ships exactly the class the fence exists to stop.

  • F1 src/kiro_crew/crew_pause.py:262 — The loss needs a pause file at its 256 KiB cap, three crews' Continues interleaved inside one subagent-probe window, and a restart before any later write re-stores the memory-only turn (crew_pause.py:250); even then the only consequence is one interrupted turn the user must continue by hand, and the alternative (raising at :265) would drop A's obligation instead.

    /ai-review override gpt 020960a7560fd66ca63bc3c19e0bb6a228e14095 <class>: The loss needs a pause file at its 256 KiB cap, three crews' Continues interleaved inside one subagent-probe window, and a restart before any later write re-stores the memory-only turn (crew_pause.py:250); even then the only consequence is one interrupted turn the user must continue by hand, and the alternative (raising at :265) would drop A's obligation instead.
    

Replace <class> with the class that records what you did. These findings are real, so for an accepted risk use owner-decision (link the decision comment in the reason) or deferred #<N> (a tracked follow-up issue); false-positive only if you verified the finding is wrong. Not not-applicable: that class files the override as a rule mismatch in the audit.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 020960a7560fd66ca63bc3c19e0bb6a228e14095 false-positive: <reason, 20 to 500 characters>

@dodgeblaster
dodgeblaster force-pushed the fix/32819-pause-cap-owed branch from 1f1bca7 to 869a306 Compare November 11, 2026 21:52
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 020960a7560fd66ca63bc3c19e0bb6a228e14095 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 020960a7560fd66ca63bc3c19e0bb6a228e14095 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Keeping cap-refused owed turns beside the stored map fixes the cause: a write that shrinks the file now always fits. Nothing blocks.

Notes

  • The pause store now spreads its state across four globals (_records, _stored, _memory_only, _spilled), and _commit takes undo/rollback modes, all for an edge case you only hit at the size cap. The tests listed only partly cover how these interact under concurrent Continues. If this keeps growing, one memory-only ledger that _commit alone reconciles would be easier to reason about.
  • _unsaved no longer marks owed turns, only forget_resuming hand-backs, so the retry path for refused owed turns now runs only through _memory_only. The comment at crew_pause.py:75 already says so, so a later reader shouldn't need anything more.

[DESIGN-REVIEWED] 020960a

Description read

[DESCRIPTION-READ] ec39ba989619423ab8979dc03c954c84ab84af5d93d871e91b98564f7bc4523e

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@dodgeblaster
dodgeblaster force-pushed the fix/32819-pause-cap-owed branch from 869a306 to 97ce546 Compare November 11, 2026 22:06
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 11, 2026
@dodgeblaster
dodgeblaster force-pushed the fix/32819-pause-cap-owed branch from 97ce546 to 5751aa6 Compare November 12, 2026 02:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@dodgeblaster
dodgeblaster force-pushed the fix/32819-pause-cap-owed branch from 5751aa6 to 442efa3 Compare November 12, 2026 02:43
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 12, 2026
@dodgeblaster
dodgeblaster force-pushed the fix/32819-pause-cap-owed branch from 442efa3 to dc23b9e Compare November 12, 2026 05:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 12, 2026
@iamwhatever
iamwhatever force-pushed the fix/32819-pause-cap-owed branch from dc23b9e to 3b0886b Compare November 17, 2026 22:44
@iamwhatever
iamwhatever marked this pull request as ready for review November 17, 2026 22:44
@iamwhatever

Copy link
Copy Markdown
Contributor

Stale-PR cleanup: rebased on main and addressed the GPT 6.1 finding from dc23b9e in 3b0886b. A write the size cap would refuse now first moves owed turns that were memory only (and a later write stored) back to memory only, so crew A's refused Continue can always rehold even after crew C's Continue used the room. Readers still see those turns. New test: test_a_refused_continue_fits_back_after_another_crews_continue_stored_an_owed_turn (fails with the OSError before the fix). test/test_crew_pause.py and the other pause/continue tests pass locally (512 tests). Marked ready for review.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 17, 2026 22:44
@iamwhatever
iamwhatever requested a review from Zedmor November 17, 2026 22:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 17, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Stale-PR cleanup: my fix round (3b0886b) for the earlier GPT finding drew two new GPT findings in the same code, so the cleanup stops here rather than start a second round. Suggested direction for @dodgeblaster: (1) only let the rollback write move a stored turn back to memory: pass a flag from rehold() into _commit() and keep every other writer refusing an over-cap write as before, so pause(), owe_turn() and keep_pending() never drop a stored obligation; (2) in _commit(), record spills from the incoming, pruned memory_only map (keep a copy before it is merged), not from the old global _memory_only, so a turn that rehold() itself stores is tracked too. Both findings are security-class, so after the fix a maintainer may also need to judge the GPT verdict. If you prefer, revert 3b0886b and ask a maintainer for the override that the first review pre-drafted.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 17, 2026
An owed turn the pause file's 256 KiB cap refused stayed in the map
every later write stores, so a write that shrinks the file (Resume all
on another crew, a member's own Continue) still hit the cap and failed,
leaving that crew paused.

Keep those owed turns beside the stored records, in memory only: readers
see both, each write first tries to store both, and a write the cap
refuses with them stores the records alone. Release and rehold carry a
member's memory-only turns, so a refused Continue puts them back as they
were, and a released one cannot shadow a later Pause all's fresh stop.

Fixes #32819
@dodgeblaster
dodgeblaster force-pushed the fix/32819-pause-cap-owed branch from 3b0886b to 91a7a3b Compare November 19, 2026 21:49
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

Fix round for the two GPT 6.1 findings on 3b0886b, now in 91a7a3b (rebased on main):

  • F1 (a growing write drops a stored owed turn): only rehold() passes rollback=True to _commit(), so only an undo may move a once-refused owed turn the file now stores back to memory only. pause(), owe_turn(), keep_pending() and forget_resuming() refuse an over-cap write as before and keep every stored obligation. New test test_a_pause_at_the_cap_never_drops_an_owed_turn_a_later_write_stored fails on 3b0886b (DID NOT RAISE).
  • F2 (a reheld memory-only turn escapes spill tracking): _commit() now also tracks spills from the incoming, pruned memory_only map, so a turn rehold() itself stores can be moved back for a later undo. New test test_a_refused_continue_fits_back_after_another_refused_continue_stored_its_owed_turn fails on 3b0886b with the OSError.

test/test_crew_pause.py (101 tests) and the local gate's related backend tests (648) pass.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026
…fused Continue needs the room

GPT 6.1 review on dc23b9e: crew A's Continue freed room kept for its undo,
then crew C's Continue stored crew B's memory-only owed turn in that room, so
A's rehold passed the cap and raised. A write the cap would refuse now first
moves owed turns that were memory only, and a later write stored, back to
memory only; readers still see them.
@dodgeblaster
dodgeblaster force-pushed the fix/32819-pause-cap-owed branch from 91a7a3b to 020960a Compare November 19, 2026 22:22
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

@bolichen97 I saw you picked up #32819 on Nov 15 and are building on this PR, so I'd like your call before we take the next step.

Where it stands: head 020960a756 fixes both earlier GPT 6.1 findings, adds two regression tests that fail on the old head, and passes all 101 tests in test_crew_pause.py. GPT 6.1 now blocks on a third variant of the same spill design. Crew A's Continue is refused after crew C's Continue has already stored crew B's owed turn in the room A freed. A's rehold then moves B's turn back to memory only. If the gateway restarts before the next write, B's turn is lost and the user has to continue it by hand. The adjudicator rated this an extreme edge, but it is security-fenced, so AI Review Hold stays red.

Options:

  1. Override. A maintainer posts /ai-review override with an owner-decision reason. The trade-off: if the alternative is to raise an error, A's own turn is lost instead.
  2. Redesign. Reserve room for pending undo across all writers. That needs a new "Continue finished" signal.
  3. You take it from here in your own follow-up, and I stop pushing to this branch.

Which do you prefer? I won't push or post an override until you reply.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resume all can fail when the pause file is at its size cap and an owed turn is kept in memory

2 participants