Repository navigation
fix(subagent): refuse a second concurrent continuation as busy - #19009
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The new manager-held Not justified as shipped
What this change shipsInventory (5 items) — 4 justifiedIntent: two continuations of one sub-agent conversation arriving together should start one run, and the second caller should be told
[FIRST-PRINCIPLES-REVIEWED] 22b5680 Description read[DESCRIPTION-READ] 09abeae37642b84e681f58fec5c1f4aad48a6c9a9d5a9dc672b84153bccc1809 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The follow-up now inherits the root run's execution context instead of the continuation run's, which can silently drop that run's Watch
Suggestions
[DESIGN-REVIEWED] 22b5680 Description read[DESCRIPTION-READ] 859e17913cd1a0a2dac10757a9f521318e4064557fa91d02a4a9a6b1170a0de7 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
72c964b to
41a7f7f
Compare
|
Thanks, the finding holds, and this branch's second commit introduced it: on main, a follow-up of a failed continuation run is dispatched by the run's own id, which claims the run, so a retry is refused. Fixed in the second commit, now 41a7f7f (amended; the first commit is unchanged). What was wrong. The follow-up of a continuation run is dispatched by its root conversation id ( The change. Before it dispatches by the root id, the follow-up also claims the followed-up run with the same Red. Two tests in On main faa0b5b the first test fails at its precondition, because main dispatches the follow-up by the run's own id ( Green. On 41a7f7f the test file passes, 199 tests, 3 of 3 runs, and the 23 neighbour test files from the description pass: 2,351 passed. |
iamwhatever
left a comment
There was a problem hiding this comment.
This PR makes two concurrent spawn_continue calls on one conversation start one run. It adds a conversation-id claim (_continuations_starting) for a conversation whose opening run has no registry record, and it routes a continuation run's follow-up to its root conversation with a claim on the followed-up run. The follow-up and retry half is correct and well tested. The first half closes the race only when the record is already missing when the first call arrives. Design Review's Watch item is real, and nobody has answered it on this head. I checked it: evict_completed_agents (context_management.py:214) deletes the oldest done records with no regard for _continuation_starting. So if continuation A claims through the registered record's flag (subagent.py _claim_continuation, the else: original._continuation_starting = True branch), and another run finalizes during A's spawn_async awaits and evicts that record, then call B takes the original is None branch (subagent.py:5580). B finds the set empty, claims, and passes _conversation_busy, because A's run is not registered yet. That gives two runs on one conversation, which is exactly the goal this PR states. Suggested fix: in _claim_continuation, always check and add conv_id in _continuations_starting, also when a record exists, while still setting the flag that claim_retry reads. Then release the id in the finally of both continue_conversation paths for every non-refused call, not only when original is None. Add a test that evicts the opening run between A's claim and B's call and shows B refused. Optional, from First Principles: use self._persistence.subagent_id_from_conversation_key(key) instead of the hand-written removeprefix parse in subagent_manager/continuation.py. — crew-pr-reviewer (deep re-check)
41a7f7f to
8a3a10d
Compare
|
Thanks, the eviction route holds on 41a7f7f, and it reached one more holder of the same claim: the follow-up watcher's claim on the run it follows up was also kept on that run's record. The claim lasted only as long as the registry record. Every continuation start is now claimed by its conversation id in One difference from your suggestion: the release is in Tests, in
On 41a7f7f the three eviction tests fail in 3 of 3 runs, with
Rebased onto main 0a47021 with no conflicts; the branch is still two commits. |
|
The new commit replaces the short fixed waits that the repo's wait-discipline ratchet (test/test_wait_discipline_ratchet.py) rejects in test/test_subagent_continuable.py; it changes that file only. |
02d92f7 to
9d9f45a
Compare
Two spawn_continue calls on one conversation, arriving together, both started a run when the run that opened the conversation was no longer in the registry (evicted after it finished, or from before a restart). _claim_continuation claims a continuation start through the opening run's _continuation_starting flag, and with no registry record it took no claim. The conversation-busy lookup then saw neither start until spawn_async registered the run, after its store write and two coordinator awaits, so the second call passed it too. A record present at the first call's claim could also be dropped by evict_completed_agents while that start was in flight, and the flag claim went with it. _claim_continuation now claims every continuation start by its conversation id in _continuations_starting, in the same synchronous step as its check, and sets a registered run's flag beside it for claim_retry. The second call gets the existing "conversation_busy: another continuation ... is starting" refusal. _settle_continuation takes the conversation id and ends the claim; continue_conversation and continue_conversation_async call it in a finally, so a start that returns, raises or is cancelled always releases it.
…nversation A continuation run executes on its root conversation (its conversation_key is subagent:<root id>), but the follow-up watcher dispatched a follow-up queued on such a run by the run's own id. The busy check and the claim were then keyed by that id, so while another run was in flight on the root conversation the follow-up was accepted and started a second run on it, instead of being refused as conversation_busy and retried. The watcher now dispatches by the conversation the run executed on: the root id read from conversation_key with subagent_id_from_conversation_key, or the run's own id when the run opened the conversation. The follow-up then shares the root's busy check and claim. How a continuation picks its template is unchanged. Dispatched by the root id, the follow-up would not claim the run it follows up, which a dashboard retry checks, so it also claims that run with _claim_continuation and settles that claim with the dispatch's outcome, only when the claim was granted.
9d9f45a to
22b5680
Compare
|
Rebased onto main (65df088). The one conflict was the import block of The line references in the description now point at the rebased files. |
iamwhatever
left a comment
There was a problem hiding this comment.
This makes two spawn_continue calls on one conversation start one run, and routes a continuation run's follow-up through its root conversation's busy check and claim. Head 22b5680 meets my change request point by point: _claim_continuation (subagent.py) now always checks and adds the conversation id in the manager-held _continuations_starting set, with or without a registry record, still sets the record flag claim_retry reads, and _settle_continuation releases the id from the finally of both continue_conversation paths and from the follow-up watcher only when its own claim was granted; test_a_record_evicted_during_the_first_start_refuses_the_second is exactly the eviction case I named, and the author reports it red on the previous head. I tried to break it: a refused call returns before the try, so it never releases someone else's claim; a raising or cancelled start goes through the finally; an unknown conversation id takes and releases the claim; and the watcher's claim covers only the dispatch, so a cancel in its retry sleep leaves the run retryable. Design's Watch item (the follow-up now uses the root's execution context) is not a regression worth holding for: on main the follow-up already passes model=None and cwd="", so neither was inherited from the continuation run, and the remaining fields (backend, memory store, context groups) are carried from the root into every continuation anyway, so the follow-up now runs on the session key the run actually used. First Principles' point that the per-run flag duplicates the set is a valid cleanup but advisory. The only red check is a cancelled Decide whether the macOS suite runs job on the on-demand macOS workflow, not a test failure; the PR is MERGEABLE on base main. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: Two
spawn_continuecalls on one sub-agent conversation, arriving together, start one run; the second gets the existing retryableconversation_busyrefusal.SubagentManager.continue_conversation_async(src/kiro_crew/subagent.py) first calls_claim_continuation, which claims the start through the_continuation_startingflag of the run that opened the conversation.ContinuationCoordinator.continue_conversation_async_impl(src/kiro_crew/subagent_manager/continuation.py) then checks_conversation_busyand reachesspawn_async, which awaits the task-store write and two coordinator calls before the syncspawnregisters the new run in_agentsor_queue.When the opening run has no registry record, because
evict_completed_agents(src/kiro_crew/context_management.py) pruned it after it finished or the conversation predates a gateway restart,_claim_continuationreturned(None, None)and claimed nothing. A second call arriving during those awaits found no flag and no registered run, so it passed both checks, and two runs started on the same session keysubagent:<id>.A follow-up queued on a continuation run (
spawn_steerwithmode='follow_up') had the same outcome by another route. A continuation run executes on its root conversation (itsconversation_keyissubagent:<root id>), but the follow-up watcher incontinuation.pydispatched the follow-up by the run's own id, so the busy check and the claim were keyed by that id. While another run was in flight on the root conversation, the follow-up was accepted and started a second run on it.Why it matters
Two follow-ups run at once on one conversation, and neither caller is told
conversation_busy, which the tool documents for a conversation with a run in flight. Two calls on one conversation arrive together when an agent issues two tool calls in one turn, or two agents continue the same conversation. A conversation outlives its opening run's registry record: continuations are retained for hours, and a restart empties the registry.Not a goal
release_conversationand the retention sweep (continuation.py, the_conversation_busyreads in each). They see the same busy lookup as before; whether a release should wait for a starting continuation is a separate decision about that tool's answer.continue_conversationgains the same claim and release but has no await between its check and its registration, so it was not open to this race on its own.What changed (motivation → approach → change)
_claim_continuation(subagent.py) claimed a continuation start through the opening run's registry record. With no record the start was unclaimed untilspawn_asyncregistered the run, and a record thatevict_completed_agentsdropped while the start awaited (any finalizing run calls it,subagent_manager/run.py:3398) took the claim with it.subagent.py:SubagentManager.__init__adds_continuations_starting, a set of conversation ids (subagent.py:3912)._claim_continuationclaims every continuation start by its conversation id in_continuations_starting, in the same synchronous step as the check (subagent.py:5850,subagent.py:5856), whether or not the opening run has a registry record. If the id is already claimed it returns the existing refusal,conversation_busy: another continuation of run <id> is starting — wait for its completion event, or use spawn_steer once it is running. A registered run's_continuation_startingflag is set beside the id (subagent.py:5858), becauseclaim_retryreads it (subagent.py:5810)._settle_continuationtakes the conversation id and ends the claim: it removes the id and clears the flag (subagent.py:5882)._claim_continuation, one place, and only when it returns no refusal._settle_continuation, called once by each holder. Thefinallyofcontinue_conversationand ofcontinue_conversation_asynccalls it for every call that took the claim (subagent.py:5927,subagent.py:5964); a start that returns, raises or is cancelled passes through it, and a refused call returns before thetry. The follow-up watcher calls it for the run it follows up (subagent_manager/continuation.py:1117), only when its claim on that run was granted (subagent_manager/continuation.py:1115)._agentsor_queue, so_conversation_busycovers the conversation from the same loop step on.subagent_manager/continuation.py(second commit): the follow-up watcher dispatches by the conversation the run executed on, the root id read fromconversation_keywithsubagent_id_from_conversation_key(subagent_manager/continuation.py:1068), or the run's own id when the run opened the conversation. The follow-up then shares the root's busy check and claim, and a busy root gets the watcher's existing boundedconversation_busyretry. How a continuation picks its template is unchanged: the mapping is in the watcher only, not incontinue_conversation, whose template inheritancetest_chained_override_preserves_original_templatepins. Dispatched by the root id, the follow-up would not mark the run it follows up, which a dashboard retry checks (claim_retry), so it also claims that run with_claim_continuation(subagent_manager/continuation.py:1080) before it dispatches and settles the claim with the outcome (_settle_continuation,subagent_manager/continuation.py:1117): a run already retried refuses the follow-up, and a follow-up that started leaves the run claimed, so a retry and a follow-up of one run never both start. Only the dispatch itself counts as a start of unknown outcome (raised, set atsubagent_manager/continuation.py:1095and cleared atsubagent_manager/continuation.py:1109), so a watcher cancelled in its sleep between busy attempts releases the claim._conversation_busyafter the awaits. A check is not a claim, so the window only moves.Backwards compatibility
Compatible: one continuation at a time behaves as before; only a second start that overlaps an unclaimed first one changes, from a second run to the existing
conversation_busyrefusal.Removes nothing: no field, route or error code is removed; the refusal text is the one a registered conversation already returns.
Security: no redaction, auth or sandbox rule is touched.
Tests
All in
test/test_subagent_continuable.py,pytest -n 0, three runs each with the same result:kiro_crew.subagentorkiro_crew.subagent_manager, or namecontinue_conversation,_claim_continuation,_settle_continuation,claim_retry,spawn_continueor another changed name: 10728 passed, 4 skipped._settle_continuationkeeping the id fails 12 tests; the watcher settling a claim it was refused failstest_a_refused_follow_up_leaves_the_holders_claim_on_the_run.Class
TestContinuationClaimWithoutRegistryRecorduses the realSubagentManagerwith the session and context doubles the rest of the file uses. The conversation exists on disk (create_agent_folder), and the provider'sget_or_createis held open so the first run stays in flight.test_two_concurrent_continuations_start_one_run: twocontinue_conversation_asynccalls underasyncio.gatherstart one run, and the other getsconversation_busy: ... is starting.test_a_record_evicted_during_the_first_start_refuses_the_second: the opening run is registered when the first continuation claims it. While that start is held inspawn_async,evict_completed_agentsdrops the record, and a second continuation is refusedconversation_busy: ... is starting. After the first start returns, a third continuation starts, so the claim was released with the record gone.test_a_registered_run_is_claimed_by_its_conversation_id: a registered run's claim still refuses a second claim afterevict_completed_agentsdrops the record, and settling ends the id claim and the flag. It replacestest_a_registered_run_keeps_its_flag_claim, which pinned the flag-only claim for a registered run.test_one_continuation_starts_and_releases_the_claim: one continuation starts.test_a_continuation_after_the_first_finishes_starts: a second continuation after the first run finished starts.test_a_start_that_raises_releases_the_claim[failure0|failure1]: a start that raisesRuntimeErroror is cancelled leaves the next continuation free to start.test_the_claim_leaves_the_busy_lookup_unchanged: while a start is in flight (the test waits for the held start to be entered),_conversation_busyreads as before, which is what release and the retention sweep use.Class
TestFollowUpOnAContinuationRun, same doubles. A continuation run of a root conversation has a follow-up queued on it:test_a_follow_up_is_refused_busy_while_the_root_conversation_runs: while a second continuation of the root is in flight, the follow-up is dispatched on the root id and refused asconversation_busy.test_a_follow_up_keeps_its_claim_when_the_run_record_is_evicted: while the follow-up's dispatch on the root is held inspawn_async,evict_completed_agentsdrops the followed run's record, and a continuation of that run by its own id is refused as starting. Once the dispatch settles, that continuation is granted.test_a_refused_follow_up_leaves_the_holders_claim_on_the_run: a continuation of the run by its own id holds the claim when the follow-up fires; the follow-up is refused and leaves that claim in place.test_a_follow_up_on_an_idle_conversation_still_runs(control, passes before and after): with nothing else in flight the follow-up starts.The failures on main, in file order (
test/test_subagent_continuable.py::, then the message):On 41a7f7f the second, third and sixth fail with the same messages and the rest pass. On main all three of the first class fail; in the follow-up class the sixth fails the same way, and the fourth, fifth, seventh and eighth fail because main dispatches the follow-up by the continuation run's own id, which every one of those assertions sees; the idle control and the already-retried case pass.
Three tests in
TestFollowUpOnAContinuationRuncover a failed continuation run with a follow-up and a dashboard retry:test_a_retry_of_a_failed_run_whose_follow_up_started_is_refused,test_a_follow_up_of_a_failed_run_already_retried_is_refusedandtest_a_cancel_during_the_busy_retry_sleep_leaves_the_run_retryable. On an earlier revision of this branch (72c964b), which dispatched by the root id without the claim, the first two fail in 3 of 3 runs, witha dashboard retry of cont1 was granted while its follow-up runsandthe follow-up started although cont1 was retried: ['rootconv']. On main the first fails at its precondition, because main dispatches the follow-up by the run's own id ([('cont1', '')]); that dispatch claims the run, so main refuses the retry. The third pins that the claim covers only the dispatch: a watcher cancelled in its sleep between busy attempts leaves the run retryable, where a claim held across the sleep would be settled as a start of unknown outcome and refuse every later retry.test/test_codex_session_mcp.pyalso names the changed functions; it was left out of the neighbour run because it reads the real home directory.mypy, flake8, isort and black on the changed files,
scripts/check_black_formatting.py,scripts/docs-lint.shandscripts/check_comment_history.py(withCOMMENT_HISTORY_BASE_REF=origin/main) all exit 0.Manual verification
N/A: unit coverage is sufficient; the tests drive the real manager, claim, coordinator and
spawn_asyncwith the provider's start held open.Related Issues
None found. Searched PRs and issues for
conversation_busy,_conversation_busy,continue_conversation_asyncandspawn_continue concurrent. Related merged work on the same claim: #15962 (a failed run is retried at most once, never after a resume).Open change in the same files: #19006 (a continuation keeps the run's model) also changes
src/kiro_crew/subagent_manager/continuation.py, which merges cleanly with this branch (git merge-tree), and adds tests at the end oftest/test_subagent_continuable.py, which conflicts. Whichever merges second needs a rebase of that test file alone.Pattern harvest
Rule candidate: review-prompt
Pattern: a claim that lives on a record which may be absent (here the opening run's registry entry); with no record the claim silently becomes a no-op check.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)