Skip to content

fix(crew-mode): serialize a member's own continue with resume all - #31717

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/30183-rest-continue-lock
Nov 7, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/30183-rest-continue-lock

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A crew member's own Continue that is refused while Resume all runs at the same moment no longer leaves that member idle without its continuation.

During a Pause all, a member's own Continue first takes the member off the pause record, then asks for the Continue, and puts the member back on the record if the Continue is refused (its stopped turn is still tearing down). Resume all could run in between: it read a record that no longer named the member, dropped it, and treated the member as not stopped. The refused Continue then found no record to put the member back on, so after teardown the member sat idle and nothing continued it.

Why it matters

It takes two overlapping human actions during a teardown, so it is rare. When it happens the stopped turn's work is silently not picked up again: the person has to notice the idle member and press Continue a second time.

Not a goal

What changed (motivation → approach → change)

Pause all and Resume all already run one at a time per root under one lock (_root_lock in dashboard/handlers/crew_pause.py). A member's own Continue did not take it, so its release, its Continue and its rollback on refusal could interleave with a Resume all.

The Continue endpoint (dashboard/chat_api/continue_turn.py) now takes that same lock, for each root whose record names the member, before it releases the member, and holds it through the Continue and the rollback. A Resume all pressed meanwhile waits, then reads the record with the member back on it and continues the member itself. A Continue that waited behind a Resume all which already handed the member back (a turn started meanwhile, and no newer pause holds it) dispatches nothing more, as Resume all already does for a stopped member. roots_naming in src/kiro_crew/crew_pause.py lists those roots from the in-memory record. A Continue on a session no pause holds takes no lock and runs as before.

Backwards compatibility

Compatible: the routes, their responses and every refusal code stay the same; a member's Continue and a Resume all on the same crew only stop running at the same time.

Removes nothing: no route, record field, outcome or function is taken away.

Tests

  • test/test_crew_pause.py::test_a_resume_all_during_a_members_refused_continue_still_hands_it_back: Resume all is pressed while a held member's own Continue is in flight, and that Continue is refused. Resume all waits on the root lock until the refusal has put the member back, then continues it.
  • test/test_crew_pause.py::test_a_members_continue_that_waited_behind_resume_all_starts_no_second_turn: a member's Continue waits on the lock while Resume all hands the member back; once the lock opens it starts no second turn.
  • test/test_crew_pause.py::test_a_members_continue_after_resume_and_a_new_pause_still_continues_it: the same wait, but a newer Pause all stops the member again before the lock opens; the Continue still releases and continues it.

Proof: PROVEN test/test_crew_pause.py::test_a_resume_all_during_a_members_refused_continue_still_hands_it_back, test/test_crew_pause.py::test_a_members_continue_that_waited_behind_resume_all_starts_no_second_turn, test/test_crew_pause.py::test_a_members_continue_after_resume_and_a_new_pause_still_continues_it (head c19fe55c3669)

Determinism: 20 runs at -n0 and 10 at -n 4 --dist loadgroup --max-worker-restart=2 beside test/test_queue_hold_gate.py and test/test_slot_running_readers.py, all green; test/test_crew_pause.py in a shuffled order (seed 20473), 70 passed. The two newer tests each ran 20/20 at -n0, and the whole file green after it. All three tests wait on a task parked on the lock (until_parked), not on a clock.

Manual verification

N/A — unit coverage sufficient: the race needs a Resume all click to land inside a refused Continue, which cannot be timed by hand; the test drives exactly that order.

Related Issues

Part of #30183

Sibling sweep

Swept: git grep -nE "crew_pause.(release|rehold|owe_turn)" -- src/
Sibling: src/kiro_crew/dashboard/chat_api/continue_turn.py -- changed here
Sibling: src/kiro_crew/dashboard/chat_turn/turn_dispatch_gates.py -- not affected -- it only adds an obligation, and a record Resume all already dropped is covered by its resuming set
Sibling: src/kiro_crew/dashboard/chat_api/chat_send.py -- not affected -- same owe_turn call as the dispatch gate, with no release or rollback

Pattern harvest

Not generalizable: the pause record has one release-and-rollback pair, and the lock that orders it against Resume all belongs to this feature alone.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 6, 2026 19:37
@bolichen97
bolichen97 requested a review from Zedmor November 6, 2026 19:37
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 6, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 6, 2026 19:43
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 906f66dcd148be151a85743b1980f5dd35e1227f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 906f66dcd148be151a85743b1980f5dd35e1227f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The fix reuses the root lock that Pause all and Resume all already hold, so a member's Continue and its rollback can't interleave with a Resume all, and it adds no new surface.

Notes

  • The Continue now holds _root_lock until continue_slot_turn returns, and that call can wait on the slot lock and the child probe. While it waits, a Pause all on the same root also waits. Resume all already waits on teardowns under this lock, so this matches existing behaviour.

[DESIGN-REVIEWED] 906f66d

Description read

[DESCRIPTION-READ] a24b34922f628bb341a1a546b314535d8f51dbc690a2f3fc7a02d0ea8c2ce2ef

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 906f66dcd148be151a85743b1980f5dd35e1227f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The lock fixes the cause: a release-then-rollback pair that ran outside the lock Pause all and Resume all already share. rehold deliberately does not recreate a record that Resume all dropped.

What this change ships

Inventory (5 items) — 5 justified

Intent: when a crew member's own Continue is refused while Resume all runs at the same moment, the member still gets its turn back afterwards. FIX.

  1. While its crew is paused, a member's own Continue now waits for any Pause all or Resume all running on that root before it acts — justified
  2. A Continue that waited behind a Resume all which already handed the member back now starts no second turn — justified
  3. A refused Continue puts the member back on the pause record before a waiting Resume all reads it, so Resume all continues the member — justified
  4. New crew_pause.roots_naming helper, with one caller in continue_turn.py — justified
  5. The comment on _ROOT_LOCKS now names the member's Continue as a third holder of the lock — justified

[FIRST-PRINCIPLES-REVIEWED] 906f66d

Description read

[DESCRIPTION-READ] 6048b28df2b6ea4949a3fc01fe9aae7235a042ae1290fa0714774c9c7dfaf5d9

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The red Backend Tests shards on 1dfd2dc are main's, not this change's:

The GPT and Opus lanes finished this head with no findings but could not post their verdicts (API rate limit). This PR waits for #31803, then rebases onto main and re-runs CI.

@bolichen97
bolichen97 force-pushed the fix/30183-rest-continue-lock branch from 1dfd2dc to 08d617d Compare November 7, 2026 01:59
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/ and src/kiro_crew/dashboard/ (same-directory neighbours of the four files this PR touches; none of the four changed) since this head went green (a31eccd -> 42f45e3); CI re-running.

The new base carries the fixes for both main-red shards (#31625, #31803). The diff is unchanged (same patch-id as 1dfd2dc); the related backend tests (822) and the two formerly red main tests pass locally on 08d617d.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 906f66dcd148be151a85743b1980f5dd35e1227f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 906f66d

Verdict parsed from the review's SHA-scoped output markers for commit 906f66dcd148be151a85743b1980f5dd35e1227f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 906f66dcd148be151a85743b1980f5dd35e1227f false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 906f66dcd148be151a85743b1980f5dd35e1227f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 906f66d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 906f66dcd148be151a85743b1980f5dd35e1227f false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 7, 2026
A held crew member's own Continue released it from the pause record,
asked for the Continue, and put it back when the Continue was refused.
A Resume all running in between dropped a record that no longer named
the member, so the refused Continue had no record to restore and the
member sat idle without its continuation.

The Continue now takes the per-root lock Pause all and Resume all
already take, for every root whose record names the member, around
the release, the Continue and the rollback.

Part of #30183
@bolichen97
bolichen97 force-pushed the fix/30183-rest-continue-lock branch from 08d617d to 906f66d Compare November 7, 2026 04:46
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: member Continue waits on the root lock behind Resume all
resolution: narrowed

  • span=afb76f4f83e9 Waiting behind Resume all can dispatch a duplicate continuation

fixed in 906f66d: the Continue reads the member's _turns_started before it waits on the root locks. If a turn started meanwhile and no pause holds the member any more, the Resume all it waited behind already handed it back, so it returns ok and dispatches nothing. This is the same counter check Resume all's own hand-back uses.
A newer Pause all that holds the member again before the lock opens still gets the full release, Continue and refusal rollback.
Red/green: test_a_members_continue_that_waited_behind_resume_all_starts_no_second_turn and test_a_members_continue_after_resume_and_a_new_pause_still_continues_it each fail without the check they pin and pass with it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 7, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for the on-call. Review lanes pass on 906f66d, no conflict, no open change request.

@KiroCrewCaptain
KiroCrewCaptain merged commit 74b8552 into main Nov 7, 2026
97 of 99 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants