Skip to content

fix: refuse to retry a failed run whose conversation was continued - #15710

Closed
pepmach wants to merge 1 commit into
mainfrom
fix/subagent-relaunch-after-continue
Closed

pepmach wants to merge 1 commit into
mainfrom
fix/subagent-relaunch-after-continue

Conversation

@pepmach

@pepmach pepmach commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A failed subagent whose work a spawn_continue already carried on can no longer be re-spawned from the dashboard as a second copy of the same task.

A spawn_run member (a073575e) hit turn_limit:100 and ended failed. Its parent resumed it with spawn_continue (run 058e9ff0, conversation subagent:a073575e), and that continuation finished the task. The failed card stayed on the panel, so the dashboard kept offering "Retry failed (1)". Each press of that control re-spawned the ORIGINAL prompt as a fresh run (f9fadd37, 80176b3b, a468d083, 5bb45caa): original task text, no conversation key, a new run id, the same worktree, four times over five hours. The parent never asked for them and only noticed via spawn_list; one of the copies committed changes the parent never asked for.

This is the gap the PR closes: POST /api/spawn/{agent_id}/retry accepted a failed run as retryable work even though a continuation had adopted its conversation and taken the work forward.

Why it matters

Every such retry is a second writer on work that is already done or in progress, in the same worktree as the continuation — duplicate commits, decisions the owner had reserved made twice, and nothing in the panel saying the two runs are the same task. Anyone who continues a failed run and later presses the retry control on its still-"failed" card is hit.

Not a goal

  • Rendering a continued run differently on the panel (a superseded status, excluding it from the "Retry failed (N)" count). That needs frontend code; this PR is backend-only and the card still reads failed. Listed under Pattern harvest.
  • A durable superseded_by field in state.json. The retry route answers 404 for a run the live manager no longer holds, so a durable mark would guard nothing this route can reach, and it would add a state.json write site to the pinned census.
  • Changing what a retry of a continuation run does (it re-spawns the follow-up prompt without the conversation). Listed under Pattern harvest.
  • The state.json status: running left behind by a turn-limited run (the tombstone is the terminal record). Listed under Pattern harvest.

What changed (motivation → approach → change)

Root cause. Every automatic path that starts a run from a stored record keeps the row's id or sets a conversation_key. The one path that mints a new id with an empty conversation key is api_spawn_retry (src/kiro_crew/dashboard/handlers/messaging.py), behind the dashboard's "Retry failed (N)" control: it re-spawns old._raw_task in the old cwd, and its only gates were "terminal" and "outcome is failed". A continued run stays failed, so it stayed retryable.

Fix, part one — the mark. The route refuses, with a typed 409 (code: superseded_by_continuation, continued_by: <run id>), a failed run whose conversation a continuation adopted, and starts nothing. The adoption is recorded on the ORIGINAL's live record, SubagentInfo.superseded_by (src/kiro_crew/subagent.py), when a continuation's admission settles with a live or queued run (any terminal answer leaves it empty), and read by SubagentManager.continuation_of, with a registry fallback (any run or unstarted queue entry whose conversation_key names the original). A failed run nobody continued is retried as before.

Fix, part two — the lock and the reservation. The mark is one read on each side, and each side awaits between that read and its spawn (the route: the execution record, the agent warm; the continuation: spawn_async's store accept). So a Retry click and a spawn_continue (or the follow-up watcher's dispatch) on one run could both be admitted into one worktree. Admission is now serialized per conversation (ContinuationCoordinator, src/kiro_crew/subagent_manager/continuation.py):

  • Ordering contract. One asyncio.Lock per conversation id (_conversation_admission_locks, records counted in and out), with ONE entry — the async with handle conversation_admission(conv_id) — is held by the retry route and by continue_conversation_async from the adoption check through the spawn and the publish, so a retry and a continuation of one run are admitted in turn and the second reads the first's outcome, never the window between a check and a spawn. The handle's entry acquires and reads the retry hold's durable row inside its own rollback, so a claimant cancelled anywhere in the entry (the follow-up watcher under a parent teardown or a stage boundary) leaves the lock free for the next one.
  • Reservation. Under the lock, before the awaited spawn, the adoption is reserved as a claim (_conversation_admissions, reserve_conversation_admission) and settled on every exit (settle_conversation_admission, try/finally): an admitted continuation publishes the mark; any terminal answer (a typed refusal, or a row cancelled before it started) rolls the claim back; an admitted retry holds the claim while its run is in flight — live, queued or being dispatched on the loop (_run_in_flight), else by its durable row (_observe_retry_hold, read off-loop), which keeps a store-only row outside the dispatch window held and lapses the hold for a terminal or absent row. The claim is what a reader without the lock sees: the sync continue_conversation entry is refused at once, continuation_of answers pending for a continuation whose spawn has not returned yet, and a second retry while a retry's run is in flight gets 409 retry_in_flight.

Spec: docs/system-specs/modules/subagent.md updated in the same commit.

Backwards compatibility

Compatible: the route's new answers are 409s for cases that previously started a duplicate run (a continued run; a run whose continuation or retry is being admitted or whose retry is still running); every other retry, and every other caller of SubagentInfo, is unchanged. A continuation refused by the claim carries the conversation_busy prefix every caller already branches on. superseded_by is a new defaulted field, so existing records and callers read as before. The claim is in-memory and empty after a restart, like the mark.

Tests

test/test_subagent_retry_after_continue.py, run alone with -n0.

Round 1 (the mark), red on origin/main (3c9302b45b at the time), green after the fix:

_ TestRetryRouteAfterAContinuation.test_a_failed_run_whose_conversation_was_continued_is_refused_and_nothing_starts _
E   AssertionError: {'id': 'fresh0000deadbeef', 'retried_from': 'orig0000deadbeef', 'status': 'spawned'}
E   assert 200 == 409
_ TestRetryRouteAfterAContinuation.test_the_mark_outlives_the_continuation_in_the_registry _
E   assert 200 == 409
_ TestAdoptionMark.test_an_accepted_continuation_marks_the_original_superseded _
E   AttributeError: 'SubagentInfo' object has no attribute 'superseded_by'
_ TestAdoptionMark.test_the_registry_scan_answers_for_an_unmarked_continuation _
E   AttributeError: continuation_of. Did you mean: '_continuation'?
5 failed, 1 passed

Round 2 (the lock and the reservation), red on the round-1 head 7bb3c54d5e — a retry and a continuation driven concurrently both started a run — green after the fix. The interleavings are driven by injected gates on the documented awaits (asyncio.Events on a patched warm_project_agents_for_spawn and on a recording spawn_async), never by sleeps:

_ TestRetryAndContinuationAdmissionRace.test_retry_first_exactly_one_run_starts_and_the_continuation_is_refused _
>       assert [s["task"] for s in started] == [TASK]
E       AssertionError: assert ['carry on', ...rget, commit'] == ['implement t...rget, commit']
E         Left contains one more item: 'implement the change in the worktree, run the verify target, commit'
_ TestRetryAndContinuationAdmissionRace.test_continuation_first_exactly_one_run_starts_and_the_retry_is_refused _
>       assert [s["task"] for s in started] == ["carry on"]
E       AssertionError: assert ['carry on', ...rget, commit'] == ['carry on']
E         Left contains one more item: 'implement the change in the worktree, run the verify target, commit'
_ TestRetryAndContinuationAdmissionRace.test_a_refused_continuation_rolls_the_reservation_back_under_the_lock _
E           AttributeError: _conversation_admissions
_ TestRetryAndContinuationAdmissionRace.test_the_reservation_is_what_a_reader_without_the_lock_sees _
E           AttributeError: _conversation_admissions
4 failed, 11 deselected

Round 3 (GPT 5.6's two blocking findings on 343377c3ae, both real), red on that head — a store-only retry row let a continuation in beside it; a run cancelled before it started was recorded as an adoption — green after the fix:

_ TestRetryHoldsTheConversation.test_a_store_only_retry_row_keeps_the_hold_until_the_row_ends _
>           assert child is not None and child.done
E           AssertionError: assert (SubagentInfo(id='fresh0000deadbeef', task='carry on', ... done=False, ...)
_ TestRetryHoldsTheConversation.test_a_run_cancelled_before_it_started_is_a_refusal_not_an_adoption _
>       assert original.superseded_by == ""
E       AssertionError: assert 'stopped0deadbeef' == ''
2 failed, 15 deselected

Round 4 (the lock's single entry; GPT 5.6's blocking finding on 5d22fab457, real), red on that head — a claimant cancelled during the stale-hold observation left the lock held — green after the fix:

_ TestAdmissionLockLifecycle.test_a_claimant_cancelled_during_the_hold_observation_leaves_the_lock_free _
>           assert manager._conversation_admission_locks == {}
E           AssertionError: assert {'orig0000dea...d]>, users=1)} == {}
E             
E             Left contains 1 more item:
E             {'orig0000deadbeef': _ConversationAdmissionLock(lock=<asyncio.locks.Lock object at 0x7f87… [locked]>,
E                                                             users=1)}
E             Use -v to get more diff
1 failed (the two other lifecycle tests: one pins the counted-out waiter through the new handle, which the old head lacks; one passes on both heads)

What each test locks in:

  • the route refuses a failed run whose conversation a finished continuation adopted, with the typed code and the continuation's id, and launches nothing — no spawn is asked for and no run folder appears (real run folders: the original's turn_limit tombstone, the continuation's delivered tombstone);
  • a failed run nobody continued is still retried with its original task and cwd and no conversation key (the refusal is exactly as wide as adoption);
  • the mark on the original answers even after the continuation left the registry;
  • an accepted continuation (a real continue_conversation run) marks the original; a refused one does not;
  • the registry scan answers for an unmarked continuation, live, finished or queued, and skips resume entries;
  • retry first: parked at its agent warm after its gate read, a continuation arriving waits on the lock and is then refused with conversation_busy naming the retry's run, no mark lands, the retry's run is the only one spawned (200), and the lock record is gone when both are done;
  • continuation first: parked inside spawn_async with no child in any registry and no mark, a retry arriving waits on the lock and then gets the 409 naming the continuation (continued_by is its real id), the continuation's run is the only one spawned and is marked once its accept returns;
  • a continuation refused by its spawn rolls the reservation back under the lock, so the retry that waited is admitted;
  • what a reader without the lock sees while either side is parked mid-admission: the sync continue_conversation entry is refused at once (a retry of run … is being admitted / a continuation of run … is being admitted), and continuation_of answers pending for a continuation whose spawn has not returned yet;
  • a live retry refuses a continuation from both continue_conversation entries and a second retry (retry_in_flight, naming the run), and the hold lapses with the run — finished, popped from the registry after finishing, or cancelled out of the queue — while a queued or pump-popped row still holds it;
  • a refused retry (capacity, a typed spawn refusal, a failed record read) and a refused continuation (conversation_gone, a store refusal) leave no claim and no mark behind, and the other side is admitted afterwards;
  • a retry row the loop cannot see but the store holds non-terminal (queued, outside the dispatch window) keeps the hold from both continue entries and against a second retry, an unreadable store keeps it, a terminal (cancelled) or absent row lapses it;
  • a spawn answer that is terminal without an error (done, user_stopped: a row cancelled before it started) is a refusal on both sides — no mark, no hold;
  • the lock's lifecycle: a claimant cancelled while the entry reads the retry hold's durable row (the follow-up watcher under a parent teardown) leaves the lock map empty and the next retry admitted within a bounded wait; a claimant cancelled while waiting for the lock counts itself out of the record; a body that raises (the agent warm) releases the lock and rolls the claim back.

Existing retry tests whose manager doubles predate continuation_of model it as "nobody continued this run", and now also answer the claim as the route's (reserve_conversation_admission → None; the lock as the async with handle conversation_admission, which a MagicMock already is; the SimpleNamespace double gets the two claim methods and a no-op handle): test_api_server.py, test_handlers_messaging_coverage.py, test_slack_gateway.py, test_spawn_reasoning_effort.py, test_subagent_scale.py, test_subagent_execution_capture_async.py — one line each, no behaviour asserted by them changed.

Manual verification

N/A — unit coverage sufficient: the route and both continuation entries are driven end to end in-process with real run records and deterministic interleavings, and the only user-visible effects are typed 409s the existing chip error path already surfaces ("retry failed" notice) and a conversation_busy refusal every continuation caller already handles. Nothing was started, stopped or relaunched on a live gateway.

Mechanical evidence this PR rests on (rounds 2 and 3, each head after its amend): scripts/check_black_formatting.py, isort --check-only, flake8 and black --check on every touched file, mypy src/kiro_crew (clean), scripts/docs-lint.sh, COMMENT_HISTORY_BASE_REF=origin/main check_comment_history.py, HARNESS_BASE_REF=origin/main check_harness_parity.py, test_error_code_contract.py, test_subagent_component_globals.py; targeted suites with -n0: test_subagent_retry_after_continue.py (13), the retry tests of the six double files, and the continuation set (test_continuable_followups.py, test_member_continuation_identity.py, test_spawn_followup.py, test_subagent_context_group_plumbing.py, test_subagent_continuable.py, test_subagent_coverage.py, test_subagent_provenance_write.py, test_subagent_dependency_mark.py, test_native_subagent_boundary.py, test_subagent_state_write_model.py, test_subagent_turn_resilience.py: 625 passed); scripts/local-gate.py --base origin/main backend related set (635 files), run twice: on the claim-only tree 47,984 passed, 2 failed — test_subagent_scale.py::TestBatchIdentity::test_stage_boundary_owner_survives_every_run_reconstruction_site was this diff's (the retry reconstruction site moved into _spawn_retry_admitted; the pin now names that function) and test_autonudge_refactor_contract.py::test_each_owner_imports_alone_without_the_facade is a local-environment artifact (the shared virtualenv's editable install predates kiro_crew.autonudge_service; it fails identically on an untouched main checkout); on the lock tree 47,997 passed, 1 failed — that same artifact. Frontend related set (run_scoped_tests.py --surface frontend, 391 files, no website/ file in the diff): a first run had the rebase below land under it and failed exactly the two files main's merge train changed (settingsRegistry.test.ts, MembersPage.test.tsx; both pass in isolation on the rebased tree); re-run whole on the rebased tree, 392 files, 12,126 tests, all passed. The commit was then rebased onto the fresh main tip (d8fa6a527e, the heal that fixed the reds below; merge-tree clean, patch content byte-identical across the replay) and the static gates (whole-tree flake8, mypy clean on 2,007 files) and the targeted suites (1,157 passed, plus the six double files' retry tests) re-run on the rebased tree. Round 4 (the lock's single entry), on the amended head: red-first probe of the lifecycle tests against 5d22fab457 in a detached worktree (above); on the restructured tree, -n0: test_subagent_retry_after_continue.py (18), test_subagent_component_globals.py, the retry tests of the six double files and test_subagent_execution_capture_async.py (98 passed together), the continuation set plus test_error_code_contract.py (629 passed); mypy src/kiro_crew clean (2,007 files); black (pinned 26.3.1) and the ratchet, isort --check-only, flake8, scripts/docs-lint.sh, check_comment_history.py, check_harness_parity.py; push_guard.py --base main --require-single-on-base SAFE on the rebased head. The full related set was not re-run this round (the host ran under a fleet memory gate); the round-2 run covered the same files and no new file enters the diff.

Main at this head's base (fecca3ea56): its latest completed ci.yml run is red on Backend Tests (3.12, 4) with two open flakes — test_autonudge_refactor_contract.py::…::test_every_store_write_goes_through_write_state (#15676) and test_posix_lock_ceiling.py::…::test_a_brief_in_process_holder_is_waited_out (#15683) — and Coverage Gate downstream of that shard; neither touches a file in this diff, and this head's own run is independent of it. CI on the previous head (5d22fab457, base d8fa6a527e): every build, backend shard (Linux and Windows) and coverage gate green; the only reds were the GPT 5.6 lane (this round's finding) and PR Readiness downstream of it. The local GPT / Opus / First Principles review lanes were not run for this round either: the push has a fixed slot and the lanes would not finish before it; the server lanes — where every round's finding came from — re-run on this head and are the review.

Round 4 note. GPT 5.6's BLOCKING finding on 5d22fab457 (continuation.py:394, a cancellation during the stale-hold observation leaked the acquired lock) is real and was the third judged head on which the lane's continuation.py span printed, so this round is a restructure rather than another patch, ruled per the prepare-pr retrospective: the lock's two public methods (acquire_conversation_admission / release_conversation_admission) and the caller-side try/finally discipline are replaced by ONE entry, the async with handle conversation_admission(conv_id) (_ConversationAdmission), whose entry counts in, acquires and performs the observation inside its own rollback and whose exit releases — the lock can no longer be held outside a scope that releases it, by construction. continue_conversation_async_impl is the former _continue_locked_async_impl body under that async with (read it with -w); the dead facade delegate _continue_locked_async (First Principles item 8) is deleted. Kept unchanged: the mark, the claim and its rollback, the terminal-answer rule, the admitted retry's hold and its durable-row observers. Two scope questions the lanes raise are left for the maintainer rather than folded in: whether the reverse direction (a continuation refused while a retry runs, and the retry_in_flight 409 — First Principles items 4 and 5) is intended scope or the Goal is amended, and whether an adoption means a continuation ACCEPTED or STARTED (Opus: a continuation cancelled while still queued leaves superseded_by naming a run that never ran).

Round 3 note. GPT 5.6's two BLOCKING findings on 343377c3ae are both real and fixed red-first (tests above): the hold's lapse moved out of the claim read into observers that consult the run's durable row — _observe_retry_hold_async on the store's writer thread inside acquire_conversation_admission for both event-loop paths, _observe_retry_hold on the calling thread for the sync entry — so a store-only retry row keeps the hold until it is terminal or gone (F1); and settle_conversation_admission / _note_adoption treat every terminal answer as a refusal, including the done, user_stopped record spawn returns for a row cancelled before it started (F2). The two advisories are done as work: the _acquire_admission duck-type is deleted (the route takes the manager's lock directly; the six doubles carry the stub), which also removes the function-local import inspect. Opus's advisory (the fence is keyed on the run's own id, so a retry of a failed continuation child beside a later continuation is not covered) is true and is the frozen Not-a-goal under Pattern harvest; disposition pending the owner.

Round 2 note. GPT 5.6's BLOCKING finding on 7bb3c54d5e (continuation.py:546, "Retry can race continuation admission") is a defect in this PR's own code path and is fixed here red-first: the race tests above fail on that head and pass on this one; the lock is the per-conversation serialization it asked for, held across both paths' awaits, and the reservation taken before the awaited spawn is rolled back on refusal. The route's body past its gates moved into _spawn_retry_admitted so the claim's try/finally wraps exactly the awaits and the spawn; the 409 builder is _superseded_by_continuation; continue_conversation_async_impl is the same body indented under the claim's try (read it with -w). Declared limit: a row the task store keeps OUTSIDE the dispatch window is beyond _run_in_flight exactly as it is beyond _conversation_busy, so the hold then lapses early rather than, in the other failure, outliving a run nothing will ever observe finishing. Two consequences of the mechanism that the finding did not name are in the diff and pinned: a second retry while a retry's run is live gets 409 retry_in_flight, and a second continuation racing the first's admission gets conversation_busy from the claim rather than from a registry it cannot see yet.

Screenshots / video

N/A — no frontend surface is touched; the diff is backend, tests and spec.

Related Issues

Fixes #15615

Pattern harvest

Rule candidate: review-prompt
Pattern: "a durable record that outlives its run gets replayed as new work" — a terminal record stays on a surface that offers to re-run it, with nothing on the record saying its work moved on. Siblings with the same shape, not fixed here:

  • the panel card and the "Retry failed (N)" count still read a continued run as failed; a superseded rendering (frontend) would remove the invitation at the source;
  • a retry of a continuation run re-spawns the follow-up prompt WITHOUT its conversation_key, i.e. context-free;
  • a turn-limited run's state.json keeps status: running (only the tombstone records the ending), which is what made the reporter read the record as resumable;
  • the task-queue window refill re-dispatches any claimable row under its original params; it is guarded by taskq_excluded_ids and the terminal states, and it is not what fired here, but it is the same shape one guard away.

Second pattern: "a gate that is one read, followed by awaits, then the action" — _conversation_busy is the same shape for two continuations (the first child is in no registry while spawn_async awaits its accept), and a store-only row outside the dispatch window is invisible to both that gate and _run_in_flight.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@pepmach
pepmach requested a review from a team as a code owner October 1, 2026 03:52
@pepmach
pepmach requested a review from cruisercohen October 1, 2026 03:52
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 6e2ad1c8c0e83787399bf23a2b97d13faf390cd8 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the dashboard route stays the only retry entry point — the reserve→spawn→settle-in-finally ordering lives in the handler, not the manager.

Suggestions

  • The fence's ordering contract (lock held, claim before first await, settle on every exit) is now implemented twice — continue_conversation_async_impl and api_spawn_retry — so a future retry caller (an MCP tool, a Slack control) can silently skip it; a manager-owned admit_retry(conv_id, spawn: Callable) wrapper that owns the lock/claim/settle and takes the route's spawn as a callable would make the fence impossible to bypass without also changing the layering.

[DESIGN-REVIEWED] 6e2ad1c

Description read

[DESCRIPTION-READ] 6084b6fe2682dcda25a16b7259682bbbe72e2aa9fa478a1316020fc8e785b2d6

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 6e2ad1c8c0e83787399bf23a2b97d13faf390cd8 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the restart window: the mark and claim are in-memory only, so if the registry rebuild re-holds a continued original, its failed card turns retryable again.

Not justified as shipped

  • Item 4 — rides along: a second retry during a retry's run now gets 409 retry_in_flight; the frozen Goal covers only continued runs, though the refusal is forced once the claim exists (an admitted retry's hold must answer the gate something).

What this change ships

Inventory (8 items) — 7 justified

Intent: FIX — stop the dashboard's "Retry failed" control from re-spawning a failed run whose conversation a spawn_continue already carried forward (reproduced incident, four duplicate runs; new test red on base).

  1. Retry of a continued failed run answers typed 409 naming the continuation, starts nothing — justified
  2. The adoption is recorded on the original (superseded_by) and read by continuation_of with a registry fallback — justified (_conversation_busy covers only live runs; grep: 1 re-spawn site, _raw_task or, messaging.py:1768)
  3. Retry and continuation admission serialized per conversation; the loser reads the winner's outcome, not the check-to-spawn window — justified
  4. Second retry while a retry's run is in flight gets 409 retry_in_flight — rides along (declared, forced by the claim mechanism)
  5. Continuation during an in-flight retry refused conversation_busy naming the retry run — justified
  6. An admission in flight is visible lock-lessly: sync entry refused at once, continuation_of answers pending — justified
  7. A retry's hold lapses by reading the run's durable store row, no end-of-run hook — justified
  8. subagent.md spec section in the same commit — justified (mandated)

[FIRST-PRINCIPLES-REVIEWED] 6e2ad1c

Description read

[DESCRIPTION-READ] 7a21d81367c8366cd39bc8f3686b1d372f0422d535891d613b1630387d656f43

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 6e2ad1c8c0e83787399bf23a2b97d13faf390cd8 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 6e2ad1c

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 6e2ad1c8c0e83787399bf23a2b97d13faf390cd8: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 6e2ad1c8c0e83787399bf23a2b97d13faf390cd8 — this comment is updated in place on each push.

Review details

Nothing blocks; the new adoption fence is keyed by the retried run's own id, so a failed continuation's card stays retryable after its conversation has moved on.

FINDING — src/kiro_crew/dashboard/handlers/messaging.py:1657 — continuation_of(agent_id) resolves adoption by the retried run's OWN id, but every continuation is spawned with conversation_key = "subagent:<FIRST run id>" (api_spawn_continue hands back "conversation": conv_id, and _continue_prelude_impl builds conv_key = f"subagent:{conv_id}"), so for a continuation C1 that itself ends failed (turn_limit again — the reported shape) C1.superseded_by is never set by _note_adoption_impl (it only marks conv_id) and the _agents scan for "subagent:C1" matches nothing; one click of "Retry failed (2)" (ActivityViewer.tsx:969 retries every error card) therefore 409s on the original and admits the retry of C1, re-spawning C1._raw_task with no conversation_key into C1.cwd — the same context-free second writer in the same worktree the guard exists to refuse — and the lock/claim keyed on C1 likewise does not fence a concurrent continuation claiming ORIG → Fix: resolve the conversation in the retry gate (old.conversation_key or f"subagent:{old.id}") and have continuation_of answer for any run on that conversation other than the one being retried, the same normalisation _conversation_busy_impl already does.

[OPUS-REVIEWED] 6e2ad1c

Verdict parsed from the review's SHA-scoped output markers for commit 6e2ad1c8c0e83787399bf23a2b97d13faf390cd8.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 6e2ad1c8c0e83787399bf23a2b97d13faf390cd8: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@pepmach
pepmach force-pushed the fix/subagent-relaunch-after-continue branch 2 times, most recently from 7bb3c54 to 343377c Compare October 1, 2026 07:50
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026
@pepmach
pepmach force-pushed the fix/subagent-relaunch-after-continue branch from 343377c to 5d22fab Compare October 1, 2026 09:50
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026
POST /api/spawn/{id}/retry re-spawns a failed run's original prompt with
no conversation key. For a run that spawn_continue has already taken
forward that is a second writer on the same work: a spawn_run member
that hit turn_limit:100 stayed "failed" on the panel after its
continuation had finished the task, and the dashboard's "Retry failed"
control re-spawned it four times over five hours, one click per run,
all in one worktree (#15615).

The route now answers a typed 409 (superseded_by_continuation, naming
the run that carries the work on) for a failed run whose conversation a
continuation adopted, and starts nothing. The adoption is recorded on
the original's own record (SubagentInfo.superseded_by) when a
continuation is accepted, so a continuation dismissed from the panel
still counts, and continuation_of falls back to the registry for a
continuation that reached it without the mark. A failed run nobody
continued is retried exactly as before.

The mark alone is one read on each side, and both sides await between
that read and their own spawn (the route reads the execution record and
warms the project agents; a continuation's spawn_async awaits the store
accept before the child is in any registry), so a Retry click and a
spawn_continue, or the follow-up watcher's dispatch, on one run could
both be admitted. Retry and continuation admission are now serialized
per conversation: one asyncio.Lock per conversation id, held by the
retry route and by the async continuation entry from the adoption check
through the spawn and the publish of the mark, so the second claimant
waits and reads the first's outcome -- the retry answers the typed 409
naming the admitted continuation, the continuation a conversation_busy
naming the retry run in flight. Under the lock the adoption is reserved
as an in-memory claim (_conversation_admissions) BEFORE the awaited
spawn and rolled back when the spawn answers anything terminal -- a
typed refusal, or a row cancelled before it started (done and
user_stopped with no error); the claim is what a reader without the
lock sees: the sync continue entry is refused at once, continuation_of
answers "pending" for a continuation whose spawn has not returned yet,
an admitted retry holds the conversation while its run is in flight (a
second retry gets 409 retry_in_flight). A retry run the loop cannot see
has either ended or waits in the task store outside the dispatch
window, so the hold is read against the run's durable row and lapses
only for a terminal or absent row. The lock has one entry, an async-with
handle that counts the user in, acquires and reads that row inside its
own rollback, so a claimant cancelled anywhere in the entry (the
follow-up watcher under a parent teardown or a stage boundary) leaves
the lock free for the next one instead of held with no scope to
release it.

Fixes #15615
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 1, 2026
@pepmach
pepmach force-pushed the fix/subagent-relaunch-after-continue branch from 5d22fab to 6e2ad1c Compare October 1, 2026 11:50
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 1, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Read-only review note: this and #15962 close the same incident and both rewrite api_spawn_retry and both continuation entries, so a textual conflict is certain. #15962 also refuses a second retry of the same failed run, while #15710 allows one once the first retry has ended, at roughly three times the size. The fix here is sound (409 superseded_by_continuation / retry_in_flight under a per-conversation lock, tests race both orderings), but owners need to pick one. If this one stays: superseded_by is never cleared, so an admitted-then-failed continuation leaves the original unretryable while the 409 tells the user to retry the original; spawn_continue on the original returns conversation_busy for as long as a retry runs; and settle_conversation_admission_impl has a bare assert run is not None in production code. Please state the first two as tradeoffs.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 2, 2026
@pepmach

pepmach commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favour of #15962 -- thanks @iamwhatever for the read and for landing the fix. Both PRs close the same incident (#15615) by rewriting api_spawn_retry and the two continuation entries, and #15962 does it with the stricter rule: a failed run is retried at most once and never after a resume, where this PR still allowed a retry once the first one had ended. Same incident, tighter contract, a third of the size; one of them had to go and it is this one.

Two things this branch carried that #15962 does not, kept here in case a maintainer ever wants them: the per-conversation admission lock that lets a retry and a spawn_continue race from either side and start exactly one run, with the loser getting a typed 409 (superseded_by_continuation or retry_in_flight) instead of a second process; and the tests that race both orderings (retry first, continuation first) plus the one that refuses a continuation from both entries while a retry is live. The branch stays at 6e2ad1c8c0 for reference.

Not pursued here, for the record: superseded_by was never cleared, so an admitted-then-failed continuation left the original unretryable while the 409 told the user to retry it; spawn_continue on the original answered conversation_busy for as long as a retry ran; and settle_conversation_admission_impl carried a bare assert run is not None in production code. All three die with this branch.

@pepmach pepmach closed this Oct 2, 2026
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 2, 2026
@bolichen97
bolichen97 deleted the fix/subagent-relaunch-after-continue branch November 6, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Subagent that hit its turn limit was re-run four times, hours after its spawn_continue finished

2 participants