Repository navigation
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the dashboard route stays the only retry entry point — the reserve→spawn→settle-in-finally ordering lives in the handler, not the manager. Suggestions
[DESIGN-REVIEWED] 6e2ad1c Description read[DESCRIPTION-READ] 6084b6fe2682dcda25a16b7259682bbbe72e2aa9fa478a1316020fc8e785b2d6 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the restart window: the mark and claim are in-memory only, so if the registry rebuild re-holds a continued original, its failed card turns retryable again. Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: FIX — stop the dashboard's "Retry failed" control from re-spawning a failed run whose conversation a
[FIRST-PRINCIPLES-REVIEWED] 6e2ad1c Description read[DESCRIPTION-READ] 7a21d81367c8366cd39bc8f3686b1d372f0422d535891d613b1630387d656f43 That sha256 names the PR description this verdict read. Recompute it with |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNothing blocks; the new adoption fence is keyed by the retried run's own id, so a failed continuation's card stays retryable after its conversation has moved on. FINDING — src/kiro_crew/dashboard/handlers/messaging.py:1657 — [OPUS-REVIEWED] 6e2ad1c Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
7bb3c54 to
343377c
Compare
343377c to
5d22fab
Compare
POST /api/spawn/{id}/retry re-spawns a failed run's original prompt with
no conversation key. For a run that spawn_continue has already taken
forward that is a second writer on the same work: a spawn_run member
that hit turn_limit:100 stayed "failed" on the panel after its
continuation had finished the task, and the dashboard's "Retry failed"
control re-spawned it four times over five hours, one click per run,
all in one worktree (#15615).
The route now answers a typed 409 (superseded_by_continuation, naming
the run that carries the work on) for a failed run whose conversation a
continuation adopted, and starts nothing. The adoption is recorded on
the original's own record (SubagentInfo.superseded_by) when a
continuation is accepted, so a continuation dismissed from the panel
still counts, and continuation_of falls back to the registry for a
continuation that reached it without the mark. A failed run nobody
continued is retried exactly as before.
The mark alone is one read on each side, and both sides await between
that read and their own spawn (the route reads the execution record and
warms the project agents; a continuation's spawn_async awaits the store
accept before the child is in any registry), so a Retry click and a
spawn_continue, or the follow-up watcher's dispatch, on one run could
both be admitted. Retry and continuation admission are now serialized
per conversation: one asyncio.Lock per conversation id, held by the
retry route and by the async continuation entry from the adoption check
through the spawn and the publish of the mark, so the second claimant
waits and reads the first's outcome -- the retry answers the typed 409
naming the admitted continuation, the continuation a conversation_busy
naming the retry run in flight. Under the lock the adoption is reserved
as an in-memory claim (_conversation_admissions) BEFORE the awaited
spawn and rolled back when the spawn answers anything terminal -- a
typed refusal, or a row cancelled before it started (done and
user_stopped with no error); the claim is what a reader without the
lock sees: the sync continue entry is refused at once, continuation_of
answers "pending" for a continuation whose spawn has not returned yet,
an admitted retry holds the conversation while its run is in flight (a
second retry gets 409 retry_in_flight). A retry run the loop cannot see
has either ended or waits in the task store outside the dispatch
window, so the hold is read against the run's durable row and lapses
only for a terminal or absent row. The lock has one entry, an async-with
handle that counts the user in, acquires and reads that row inside its
own rollback, so a claimant cancelled anywhere in the entry (the
follow-up watcher under a parent teardown or a stage boundary) leaves
the lock free for the next one instead of held with no scope to
release it.
Fixes #15615
5d22fab to
6e2ad1c
Compare
|
Read-only review note: this and #15962 close the same incident and both rewrite api_spawn_retry and both continuation entries, so a textual conflict is certain. #15962 also refuses a second retry of the same failed run, while #15710 allows one once the first retry has ended, at roughly three times the size. The fix here is sound (409 superseded_by_continuation / retry_in_flight under a per-conversation lock, tests race both orderings), but owners need to pick one. If this one stays: superseded_by is never cleared, so an admitted-then-failed continuation leaves the original unretryable while the 409 tells the user to retry the original; spawn_continue on the original returns conversation_busy for as long as a retry runs; and settle_conversation_admission_impl has a bare assert run is not None in production code. Please state the first two as tradeoffs. |
|
Closing in favour of #15962 -- thanks @iamwhatever for the read and for landing the fix. Both PRs close the same incident (#15615) by rewriting Two things this branch carried that #15962 does not, kept here in case a maintainer ever wants them: the per-conversation admission lock that lets a retry and a Not pursued here, for the record: |
Problem / Motivation
Goal: A failed subagent whose work a
spawn_continuealready carried on can no longer be re-spawned from the dashboard as a second copy of the same task.A
spawn_runmember (a073575e) hitturn_limit:100and endedfailed. Its parent resumed it withspawn_continue(run058e9ff0, conversationsubagent:a073575e), and that continuation finished the task. The failed card stayed on the panel, so the dashboard kept offering "Retry failed (1)". Each press of that control re-spawned the ORIGINAL prompt as a fresh run (f9fadd37,80176b3b,a468d083,5bb45caa): original task text, no conversation key, a new run id, the same worktree, four times over five hours. The parent never asked for them and only noticed viaspawn_list; one of the copies committed changes the parent never asked for.This is the gap the PR closes:
POST /api/spawn/{agent_id}/retryaccepted a failed run as retryable work even though a continuation had adopted its conversation and taken the work forward.Why it matters
Every such retry is a second writer on work that is already done or in progress, in the same worktree as the continuation — duplicate commits, decisions the owner had reserved made twice, and nothing in the panel saying the two runs are the same task. Anyone who continues a failed run and later presses the retry control on its still-"failed" card is hit.
Not a goal
supersededstatus, excluding it from the "Retry failed (N)" count). That needs frontend code; this PR is backend-only and the card still readsfailed. Listed under Pattern harvest.superseded_byfield instate.json. The retry route answers 404 for a run the live manager no longer holds, so a durable mark would guard nothing this route can reach, and it would add astate.jsonwrite site to the pinned census.state.jsonstatus: runningleft behind by a turn-limited run (the tombstone is the terminal record). Listed under Pattern harvest.What changed (motivation → approach → change)
Root cause. Every automatic path that starts a run from a stored record keeps the row's id or sets a
conversation_key. The one path that mints a new id with an empty conversation key isapi_spawn_retry(src/kiro_crew/dashboard/handlers/messaging.py), behind the dashboard's "Retry failed (N)" control: it re-spawnsold._raw_taskin the oldcwd, and its only gates were "terminal" and "outcome is failed". A continued run staysfailed, so it stayed retryable.Fix, part one — the mark. The route refuses, with a typed 409 (
code: superseded_by_continuation,continued_by: <run id>), a failed run whose conversation a continuation adopted, and starts nothing. The adoption is recorded on the ORIGINAL's live record,SubagentInfo.superseded_by(src/kiro_crew/subagent.py), when a continuation's admission settles with a live or queued run (any terminal answer leaves it empty), and read bySubagentManager.continuation_of, with a registry fallback (any run or unstarted queue entry whoseconversation_keynames the original). A failed run nobody continued is retried as before.Fix, part two — the lock and the reservation. The mark is one read on each side, and each side awaits between that read and its spawn (the route: the execution record, the agent warm; the continuation:
spawn_async's store accept). So a Retry click and aspawn_continue(or the follow-up watcher's dispatch) on one run could both be admitted into one worktree. Admission is now serialized per conversation (ContinuationCoordinator,src/kiro_crew/subagent_manager/continuation.py):asyncio.Lockper conversation id (_conversation_admission_locks, records counted in and out), with ONE entry — theasync withhandleconversation_admission(conv_id)— is held by the retry route and bycontinue_conversation_asyncfrom the adoption check through the spawn and the publish, so a retry and a continuation of one run are admitted in turn and the second reads the first's outcome, never the window between a check and a spawn. The handle's entry acquires and reads the retry hold's durable row inside its own rollback, so a claimant cancelled anywhere in the entry (the follow-up watcher under a parent teardown or a stage boundary) leaves the lock free for the next one._conversation_admissions,reserve_conversation_admission) and settled on every exit (settle_conversation_admission,try/finally): an admitted continuation publishes the mark; any terminal answer (a typed refusal, or a row cancelled before it started) rolls the claim back; an admitted retry holds the claim while its run is in flight — live, queued or being dispatched on the loop (_run_in_flight), else by its durable row (_observe_retry_hold, read off-loop), which keeps a store-only row outside the dispatch window held and lapses the hold for a terminal or absent row. The claim is what a reader without the lock sees: the synccontinue_conversationentry is refused at once,continuation_ofanswerspendingfor a continuation whose spawn has not returned yet, and a second retry while a retry's run is in flight gets 409retry_in_flight.Spec:
docs/system-specs/modules/subagent.mdupdated in the same commit.Backwards compatibility
Compatible: the route's new answers are 409s for cases that previously started a duplicate run (a continued run; a run whose continuation or retry is being admitted or whose retry is still running); every other retry, and every other caller of
SubagentInfo, is unchanged. A continuation refused by the claim carries theconversation_busyprefix every caller already branches on.superseded_byis a new defaulted field, so existing records and callers read as before. The claim is in-memory and empty after a restart, like the mark.Tests
test/test_subagent_retry_after_continue.py, run alone with-n0.Round 1 (the mark), red on
origin/main(3c9302b45bat the time), green after the fix:Round 2 (the lock and the reservation), red on the round-1 head
7bb3c54d5e— a retry and a continuation driven concurrently both started a run — green after the fix. The interleavings are driven by injected gates on the documented awaits (asyncio.Events on a patchedwarm_project_agents_for_spawnand on a recordingspawn_async), never by sleeps:Round 3 (GPT 5.6's two blocking findings on
343377c3ae, both real), red on that head — a store-only retry row let a continuation in beside it; a run cancelled before it started was recorded as an adoption — green after the fix:Round 4 (the lock's single entry; GPT 5.6's blocking finding on
5d22fab457, real), red on that head — a claimant cancelled during the stale-hold observation left the lock held — green after the fix:What each test locks in:
turn_limittombstone, the continuation'sdeliveredtombstone);continue_conversationrun) marks the original; a refused one does not;conversation_busynaming the retry's run, no mark lands, the retry's run is the only one spawned (200), and the lock record is gone when both are done;spawn_asyncwith no child in any registry and no mark, a retry arriving waits on the lock and then gets the 409 naming the continuation (continued_byis its real id), the continuation's run is the only one spawned and is marked once its accept returns;continue_conversationentry is refused at once (a retry of run … is being admitted/a continuation of run … is being admitted), andcontinuation_ofanswerspendingfor a continuation whose spawn has not returned yet;continue_conversationentries and a second retry (retry_in_flight, naming the run), and the hold lapses with the run — finished, popped from the registry after finishing, or cancelled out of the queue — while a queued or pump-popped row still holds it;conversation_gone, a store refusal) leave no claim and no mark behind, and the other side is admitted afterwards;queued, outside the dispatch window) keeps the hold from both continue entries and against a second retry, an unreadable store keeps it, a terminal (cancelled) or absent row lapses it;done,user_stopped: a row cancelled before it started) is a refusal on both sides — no mark, no hold;Existing retry tests whose manager doubles predate
continuation_ofmodel it as "nobody continued this run", and now also answer the claim as the route's (reserve_conversation_admission→None; the lock as theasync withhandleconversation_admission, which aMagicMockalready is; theSimpleNamespacedouble gets the two claim methods and a no-op handle):test_api_server.py,test_handlers_messaging_coverage.py,test_slack_gateway.py,test_spawn_reasoning_effort.py,test_subagent_scale.py,test_subagent_execution_capture_async.py— one line each, no behaviour asserted by them changed.Manual verification
N/A — unit coverage sufficient: the route and both continuation entries are driven end to end in-process with real run records and deterministic interleavings, and the only user-visible effects are typed 409s the existing chip error path already surfaces ("retry failed" notice) and a
conversation_busyrefusal every continuation caller already handles. Nothing was started, stopped or relaunched on a live gateway.Mechanical evidence this PR rests on (rounds 2 and 3, each head after its amend):
scripts/check_black_formatting.py,isort --check-only,flake8andblack --checkon every touched file,mypy src/kiro_crew(clean),scripts/docs-lint.sh,COMMENT_HISTORY_BASE_REF=origin/main check_comment_history.py,HARNESS_BASE_REF=origin/main check_harness_parity.py,test_error_code_contract.py,test_subagent_component_globals.py; targeted suites with-n0:test_subagent_retry_after_continue.py(13), the retry tests of the six double files, and the continuation set (test_continuable_followups.py,test_member_continuation_identity.py,test_spawn_followup.py,test_subagent_context_group_plumbing.py,test_subagent_continuable.py,test_subagent_coverage.py,test_subagent_provenance_write.py,test_subagent_dependency_mark.py,test_native_subagent_boundary.py,test_subagent_state_write_model.py,test_subagent_turn_resilience.py: 625 passed);scripts/local-gate.py --base origin/mainbackend related set (635 files), run twice: on the claim-only tree 47,984 passed, 2 failed —test_subagent_scale.py::TestBatchIdentity::test_stage_boundary_owner_survives_every_run_reconstruction_sitewas this diff's (the retry reconstruction site moved into_spawn_retry_admitted; the pin now names that function) andtest_autonudge_refactor_contract.py::test_each_owner_imports_alone_without_the_facadeis a local-environment artifact (the shared virtualenv's editable install predateskiro_crew.autonudge_service; it fails identically on an untouched main checkout); on the lock tree 47,997 passed, 1 failed — that same artifact. Frontend related set (run_scoped_tests.py --surface frontend, 391 files, nowebsite/file in the diff): a first run had the rebase below land under it and failed exactly the two files main's merge train changed (settingsRegistry.test.ts,MembersPage.test.tsx; both pass in isolation on the rebased tree); re-run whole on the rebased tree, 392 files, 12,126 tests, all passed. The commit was then rebased onto the fresh main tip (d8fa6a527e, the heal that fixed the reds below; merge-tree clean, patch content byte-identical across the replay) and the static gates (whole-treeflake8,mypyclean on 2,007 files) and the targeted suites (1,157 passed, plus the six double files' retry tests) re-run on the rebased tree. Round 4 (the lock's single entry), on the amended head: red-first probe of the lifecycle tests against5d22fab457in a detached worktree (above); on the restructured tree,-n0:test_subagent_retry_after_continue.py(18),test_subagent_component_globals.py, the retry tests of the six double files andtest_subagent_execution_capture_async.py(98 passed together), the continuation set plustest_error_code_contract.py(629 passed);mypy src/kiro_crewclean (2,007 files); black (pinned 26.3.1) and the ratchet,isort --check-only,flake8,scripts/docs-lint.sh,check_comment_history.py,check_harness_parity.py;push_guard.py --base main --require-single-on-baseSAFE on the rebased head. The full related set was not re-run this round (the host ran under a fleet memory gate); the round-2 run covered the same files and no new file enters the diff.Main at this head's base (
fecca3ea56): its latest completedci.ymlrun is red on Backend Tests (3.12, 4) with two open flakes —test_autonudge_refactor_contract.py::…::test_every_store_write_goes_through_write_state(#15676) andtest_posix_lock_ceiling.py::…::test_a_brief_in_process_holder_is_waited_out(#15683) — and Coverage Gate downstream of that shard; neither touches a file in this diff, and this head's own run is independent of it. CI on the previous head (5d22fab457, based8fa6a527e): every build, backend shard (Linux and Windows) and coverage gate green; the only reds were the GPT 5.6 lane (this round's finding) and PR Readiness downstream of it. The local GPT / Opus / First Principles review lanes were not run for this round either: the push has a fixed slot and the lanes would not finish before it; the server lanes — where every round's finding came from — re-run on this head and are the review.Round 4 note. GPT 5.6's BLOCKING finding on
5d22fab457(continuation.py:394, a cancellation during the stale-hold observation leaked the acquired lock) is real and was the third judged head on which the lane'scontinuation.pyspan printed, so this round is a restructure rather than another patch, ruled per the prepare-pr retrospective: the lock's two public methods (acquire_conversation_admission/release_conversation_admission) and the caller-sidetry/finallydiscipline are replaced by ONE entry, theasync withhandleconversation_admission(conv_id)(_ConversationAdmission), whose entry counts in, acquires and performs the observation inside its own rollback and whose exit releases — the lock can no longer be held outside a scope that releases it, by construction.continue_conversation_async_implis the former_continue_locked_async_implbody under thatasync with(read it with-w); the dead facade delegate_continue_locked_async(First Principles item 8) is deleted. Kept unchanged: the mark, the claim and its rollback, the terminal-answer rule, the admitted retry's hold and its durable-row observers. Two scope questions the lanes raise are left for the maintainer rather than folded in: whether the reverse direction (a continuation refused while a retry runs, and theretry_in_flight409 — First Principles items 4 and 5) is intended scope or the Goal is amended, and whether an adoption means a continuation ACCEPTED or STARTED (Opus: a continuation cancelled while still queued leavessuperseded_bynaming a run that never ran).Round 3 note. GPT 5.6's two BLOCKING findings on
343377c3aeare both real and fixed red-first (tests above): the hold's lapse moved out of the claim read into observers that consult the run's durable row —_observe_retry_hold_asyncon the store's writer thread insideacquire_conversation_admissionfor both event-loop paths,_observe_retry_holdon the calling thread for the sync entry — so a store-only retry row keeps the hold until it is terminal or gone (F1); andsettle_conversation_admission/_note_adoptiontreat every terminal answer as a refusal, including thedone, user_stoppedrecordspawnreturns for a row cancelled before it started (F2). The two advisories are done as work: the_acquire_admissionduck-type is deleted (the route takes the manager's lock directly; the six doubles carry the stub), which also removes the function-localimport inspect. Opus's advisory (the fence is keyed on the run's own id, so a retry of a failed continuation child beside a later continuation is not covered) is true and is the frozen Not-a-goal under Pattern harvest; disposition pending the owner.Round 2 note. GPT 5.6's BLOCKING finding on
7bb3c54d5e(continuation.py:546, "Retry can race continuation admission") is a defect in this PR's own code path and is fixed here red-first: the race tests above fail on that head and pass on this one; the lock is the per-conversation serialization it asked for, held across both paths' awaits, and the reservation taken before the awaited spawn is rolled back on refusal. The route's body past its gates moved into_spawn_retry_admittedso the claim'stry/finallywraps exactly the awaits and the spawn; the 409 builder is_superseded_by_continuation;continue_conversation_async_implis the same body indented under the claim'stry(read it with-w). Declared limit: a row the task store keeps OUTSIDE the dispatch window is beyond_run_in_flightexactly as it is beyond_conversation_busy, so the hold then lapses early rather than, in the other failure, outliving a run nothing will ever observe finishing. Two consequences of the mechanism that the finding did not name are in the diff and pinned: a second retry while a retry's run is live gets 409retry_in_flight, and a second continuation racing the first's admission getsconversation_busyfrom the claim rather than from a registry it cannot see yet.Screenshots / video
N/A — no frontend surface is touched; the diff is backend, tests and spec.
Related Issues
Fixes #15615
Pattern harvest
Rule candidate: review-prompt
Pattern: "a durable record that outlives its run gets replayed as new work" — a terminal record stays on a surface that offers to re-run it, with nothing on the record saying its work moved on. Siblings with the same shape, not fixed here:
failed; asupersededrendering (frontend) would remove the invitation at the source;conversation_key, i.e. context-free;state.jsonkeepsstatus: running(only the tombstone records the ending), which is what made the reporter read the record as resumable;taskq_excluded_idsand the terminal states, and it is not what fired here, but it is the same shape one guard away.Second pattern: "a gate that is one read, followed by awaits, then the action" —
_conversation_busyis the same shape for two continuations (the first child is in no registry whilespawn_asyncawaits its accept), and a store-only row outside the dispatch window is invisible to both that gate and_run_in_flight.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement