Skip to content

fix(subagent): refuse a second concurrent continuation as busy - #19009

Merged
javenciu merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/concurrent-continue-is-refused-busy
Oct 13, 2026
Merged

javenciu merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/concurrent-continue-is-refused-busy

Conversation

@javenciu

@javenciu javenciu commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Two spawn_continue calls on one sub-agent conversation, arriving together, start one run; the second gets the existing retryable conversation_busy refusal.

SubagentManager.continue_conversation_async (src/kiro_crew/subagent.py) first calls _claim_continuation, which claims the start through the _continuation_starting flag of the run that opened the conversation. ContinuationCoordinator.continue_conversation_async_impl (src/kiro_crew/subagent_manager/continuation.py) then checks _conversation_busy and reaches spawn_async, which awaits the task-store write and two coordinator calls before the sync spawn registers the new run in _agents or _queue.

When the opening run has no registry record, because evict_completed_agents (src/kiro_crew/context_management.py) pruned it after it finished or the conversation predates a gateway restart, _claim_continuation returned (None, None) and claimed nothing. A second call arriving during those awaits found no flag and no registered run, so it passed both checks, and two runs started on the same session key subagent:<id>.

A follow-up queued on a continuation run (spawn_steer with mode='follow_up') had the same outcome by another route. A continuation run executes on its root conversation (its conversation_key is subagent:<root id>), but the follow-up watcher in continuation.py dispatched the follow-up by the run's own id, so the busy check and the claim were keyed by that id. While another run was in flight on the root conversation, the follow-up was accepted and started a second run on it.

Why it matters

Two follow-ups run at once on one conversation, and neither caller is told conversation_busy, which the tool documents for a conversation with a run in flight. Two calls on one conversation arrive together when an agent issues two tool calls in one turn, or two agents continue the same conversation. A conversation outlives its opening run's registry record: continuations are retained for hours, and a restart empties the registry.

Not a goal

  • release_conversation and the retention sweep (continuation.py, the _conversation_busy reads in each). They see the same busy lookup as before; whether a release should wait for a starting continuation is a separate decision about that tool's answer.
  • The sync continue_conversation gains the same claim and release but has no await between its check and its registration, so it was not open to this race on its own.

What changed (motivation → approach → change)

  • Root cause: _claim_continuation (subagent.py) claimed a continuation start through the opening run's registry record. With no record the start was unclaimed until spawn_async registered the run, and a record that evict_completed_agents dropped while the start awaited (any finalizing run calls it, subagent_manager/run.py:3398) took the claim with it.
  • Change, subagent.py:
    • SubagentManager.__init__ adds _continuations_starting, a set of conversation ids (subagent.py:3912).
    • _claim_continuation claims every continuation start by its conversation id in _continuations_starting, in the same synchronous step as the check (subagent.py:5850, subagent.py:5856), whether or not the opening run has a registry record. If the id is already claimed it returns the existing refusal, conversation_busy: another continuation of run <id> is starting — wait for its completion event, or use spawn_steer once it is running. A registered run's _continuation_starting flag is set beside the id (subagent.py:5858), because claim_retry reads it (subagent.py:5810).
    • _settle_continuation takes the conversation id and ends the claim: it removes the id and clears the flag (subagent.py:5882).
  • Every path that sets or clears the claim:
    • Set: _claim_continuation, one place, and only when it returns no refusal.
    • Clear: _settle_continuation, called once by each holder. The finally of continue_conversation and of continue_conversation_async calls it for every call that took the claim (subagent.py:5927, subagent.py:5964); a start that returns, raises or is cancelled passes through it, and a refused call returns before the try. The follow-up watcher calls it for the run it follows up (subagent_manager/continuation.py:1117), only when its claim on that run was granted (subagent_manager/continuation.py:1115).
    • The id set lives on the manager, so evicting the record during a start leaves the claim in place.
    • Once a start returns, its run is in _agents or _queue, so _conversation_busy covers the conversation from the same loop step on.
  • Change, subagent_manager/continuation.py (second commit): the follow-up watcher dispatches by the conversation the run executed on, the root id read from conversation_key with subagent_id_from_conversation_key (subagent_manager/continuation.py:1068), or the run's own id when the run opened the conversation. The follow-up then shares the root's busy check and claim, and a busy root gets the watcher's existing bounded conversation_busy retry. How a continuation picks its template is unchanged: the mapping is in the watcher only, not in continue_conversation, whose template inheritance test_chained_override_preserves_original_template pins. Dispatched by the root id, the follow-up would not mark the run it follows up, which a dashboard retry checks (claim_retry), so it also claims that run with _claim_continuation (subagent_manager/continuation.py:1080) before it dispatches and settles the claim with the outcome (_settle_continuation, subagent_manager/continuation.py:1117): a run already retried refuses the follow-up, and a follow-up that started leaves the run claimed, so a retry and a follow-up of one run never both start. Only the dispatch itself counts as a start of unknown outcome (raised, set at subagent_manager/continuation.py:1095 and cleared at subagent_manager/continuation.py:1109), so a watcher cancelled in its sleep between busy attempts releases the claim.
  • Rejected: re-checking _conversation_busy after the awaits. A check is not a claim, so the window only moves.

Backwards compatibility

Compatible: one continuation at a time behaves as before; only a second start that overlaps an unclaimed first one changes, from a second run to the existing conversation_busy refusal.
Removes nothing: no field, route or error code is removed; the refusal text is the one a registered conversation already returns.
Security: no redaction, auth or sandbox rule is touched.

Tests

All in test/test_subagent_continuable.py, pytest -n 0, three runs each with the same result:

  • Red on main 65df088 (this test file copied into a clean checkout), the two classes below: 8 failed, 7 passed.
  • Red on this branch's previous head 41a7f7f (same copy): 3 failed, 12 passed, the three eviction tests below.
  • Green on this branch: the whole file, 202 passed in 3 of 3 runs. The 149 test files that import kiro_crew.subagent or kiro_crew.subagent_manager, or name continue_conversation, _claim_continuation, _settle_continuation, claim_retry, spawn_continue or another changed name: 10728 passed, 4 skipped.
  • Mutations: _settle_continuation keeping the id fails 12 tests; the watcher settling a claim it was refused fails test_a_refused_follow_up_leaves_the_holders_claim_on_the_run.

Class TestContinuationClaimWithoutRegistryRecord uses the real SubagentManager with the session and context doubles the rest of the file uses. The conversation exists on disk (create_agent_folder), and the provider's get_or_create is held open so the first run stays in flight.

  • test_two_concurrent_continuations_start_one_run: two continue_conversation_async calls under asyncio.gather start one run, and the other gets conversation_busy: ... is starting.
  • test_a_record_evicted_during_the_first_start_refuses_the_second: the opening run is registered when the first continuation claims it. While that start is held in spawn_async, evict_completed_agents drops the record, and a second continuation is refused conversation_busy: ... is starting. After the first start returns, a third continuation starts, so the claim was released with the record gone.
  • test_a_registered_run_is_claimed_by_its_conversation_id: a registered run's claim still refuses a second claim after evict_completed_agents drops the record, and settling ends the id claim and the flag. It replaces test_a_registered_run_keeps_its_flag_claim, which pinned the flag-only claim for a registered run.
  • Controls, each passing before and after the change:
    • test_one_continuation_starts_and_releases_the_claim: one continuation starts.
    • test_a_continuation_after_the_first_finishes_starts: a second continuation after the first run finished starts.
    • test_a_start_that_raises_releases_the_claim[failure0|failure1]: a start that raises RuntimeError or is cancelled leaves the next continuation free to start.
    • test_the_claim_leaves_the_busy_lookup_unchanged: while a start is in flight (the test waits for the held start to be entered), _conversation_busy reads as before, which is what release and the retention sweep use.

Class TestFollowUpOnAContinuationRun, same doubles. A continuation run of a root conversation has a follow-up queued on it:

  • test_a_follow_up_is_refused_busy_while_the_root_conversation_runs: while a second continuation of the root is in flight, the follow-up is dispatched on the root id and refused as conversation_busy.
  • test_a_follow_up_keeps_its_claim_when_the_run_record_is_evicted: while the follow-up's dispatch on the root is held in spawn_async, evict_completed_agents drops the followed run's record, and a continuation of that run by its own id is refused as starting. Once the dispatch settles, that continuation is granted.
  • test_a_refused_follow_up_leaves_the_holders_claim_on_the_run: a continuation of the run by its own id holds the claim when the follow-up fires; the follow-up is refused and leaves that claim in place.
  • test_a_follow_up_on_an_idle_conversation_still_runs (control, passes before and after): with nothing else in flight the follow-up starts.

The failures on main, in file order (test/test_subagent_continuable.py::, then the message):

TestContinuationClaimWithoutRegistryRecord::test_two_concurrent_continuations_start_one_run - started 2 runs on one conversation
TestContinuationClaimWithoutRegistryRecord::test_a_registered_run_is_claimed_by_its_conversation_id - a second continuation was granted after the claimed record was evicted
TestContinuationClaimWithoutRegistryRecord::test_a_record_evicted_during_the_first_start_refuses_the_second - a second run started on one conversation: run <id>
TestFollowUpOnAContinuationRun::test_a_follow_up_is_refused_busy_while_the_root_conversation_runs - [('cont1', '')]
TestFollowUpOnAContinuationRun::test_a_retry_of_a_failed_run_whose_follow_up_started_is_refused - [('cont1', '')]
TestFollowUpOnAContinuationRun::test_a_follow_up_keeps_its_claim_when_the_run_record_is_evicted - a continuation of cont1 was granted while its follow-up starts
TestFollowUpOnAContinuationRun::test_a_refused_follow_up_leaves_the_holders_claim_on_the_run - [('cont1', 'conversation_busy: another continuation of run cont1 is starting ...'), ...]
TestFollowUpOnAContinuationRun::test_a_cancel_during_the_busy_retry_sleep_leaves_the_run_retryable - [('cont1', '')]

On 41a7f7f the second, third and sixth fail with the same messages and the rest pass. On main all three of the first class fail; in the follow-up class the sixth fails the same way, and the fourth, fifth, seventh and eighth fail because main dispatches the follow-up by the continuation run's own id, which every one of those assertions sees; the idle control and the already-retried case pass.

Three tests in TestFollowUpOnAContinuationRun cover a failed continuation run with a follow-up and a dashboard retry: test_a_retry_of_a_failed_run_whose_follow_up_started_is_refused, test_a_follow_up_of_a_failed_run_already_retried_is_refused and test_a_cancel_during_the_busy_retry_sleep_leaves_the_run_retryable. On an earlier revision of this branch (72c964b), which dispatched by the root id without the claim, the first two fail in 3 of 3 runs, with a dashboard retry of cont1 was granted while its follow-up runs and the follow-up started although cont1 was retried: ['rootconv']. On main the first fails at its precondition, because main dispatches the follow-up by the run's own id ([('cont1', '')]); that dispatch claims the run, so main refuses the retry. The third pins that the claim covers only the dispatch: a watcher cancelled in its sleep between busy attempts leaves the run retryable, where a claim held across the sleep would be settled as a start of unknown outcome and refuse every later retry.
test/test_codex_session_mcp.py also names the changed functions; it was left out of the neighbour run because it reads the real home directory.
mypy, flake8, isort and black on the changed files, scripts/check_black_formatting.py, scripts/docs-lint.sh and scripts/check_comment_history.py (with COMMENT_HISTORY_BASE_REF=origin/main) all exit 0.

Manual verification

N/A: unit coverage is sufficient; the tests drive the real manager, claim, coordinator and spawn_async with the provider's start held open.

Related Issues

None found. Searched PRs and issues for conversation_busy, _conversation_busy, continue_conversation_async and spawn_continue concurrent. Related merged work on the same claim: #15962 (a failed run is retried at most once, never after a resume).

Open change in the same files: #19006 (a continuation keeps the run's model) also changes src/kiro_crew/subagent_manager/continuation.py, which merges cleanly with this branch (git merge-tree), and adds tests at the end of test/test_subagent_continuable.py, which conflicts. Whichever merges second needs a rebase of that test file alone.

Pattern harvest

Rule candidate: review-prompt
Pattern: a claim that lives on a record which may be absent (here the opening run's registry entry); with no record the claim silently becomes a no-op check.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 02:05
@javenciu
javenciu requested a review from buluoray October 11, 2026 02:05
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 22b56806e029060b2b7305c650a39bf18c4f2053 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 22b56806e029060b2b7305c650a39bf18c4f2053 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The new manager-held _continuations_starting set makes the per-run _continuation_starting flag redundant, yet the PR keeps both and writes them in step.

Not justified as shipped

  • Item 3 — duplicate of SubagentManager._continuations_starting (subagent.py): the diff keeps writing original._continuation_starting = True beside self._continuations_starting.add(conv_id), and the description gives the reason as "because claim_retry reads it". Grepping _continuation_starting in src/ finds one reader, claim_retry (subagent.py:5802). That method is on the same manager that owns the set, so failed.id in self._continuations_starting answers the same question. Every write now has to keep both copies in step.
    Subtraction: delete the SubagentInfo._continuation_starting field and its set and clear lines in _claim_continuation and _settle_continuation, and have claim_retry test failed.id in self._continuations_starting.
    Clears when: the field is deleted and claim_retry reads the set, or a second reader that cannot reach the manager is named.

What this change ships

Inventory (5 items) — 4 justified

Intent: two continuations of one sub-agent conversation arriving together should start one run, and the second caller should be told conversation_busy. This is a FIX.

  1. A second concurrent spawn_continue on a conversation whose opening run was evicted, or predates a restart, now gets conversation_busy instead of starting a second run — justified
  2. A follow-up queued on a continuation run is now dispatched against the root conversation, so it gets the root's busy refusal and bounded retry — justified
  3. The per-run _continuation_starting flag is still written beside the new conversation-id set — duplicate of SubagentManager._continuations_starting
  4. A follow-up also claims the run it follows up, so a dashboard retry and a follow-up of that run never both start — justified
  5. The claim is now released on every exit path (finally), including a cancel during the sleep between watcher retries — justified

[FIRST-PRINCIPLES-REVIEWED] 22b5680

Description read

[DESCRIPTION-READ] 09abeae37642b84e681f58fec5c1f4aad48a6c9a9d5a9dc672b84153bccc1809

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of 22b56806e029060b2b7305c650a39bf18c4f2053 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The follow-up now inherits the root run's execution context instead of the continuation run's, which can silently drop that run's model/cwd overrides. The description doesn't mention this.

Watch

  • The follow-up's execution context changes. The hunk conversation_id = ...subagent_id_from_conversation_key(info.conversation_key ...) or info.id makes the watcher call continue_conversation_async(conversation_id, ...). continue_conversation_async_impl then reads self._manager._agents.get(conv_id).execution_context, and when that is missing it falls back to read_run_execution(conv_id). Both are keyed by the root id now, so a follow-up on a spawn_continue run that set its own model or cwd falls back to the root's. The description only says "How a continuation picks its template is unchanged" and doesn't mention the model, the working directory or the memory mode.
    Clears when: the description states which execution context a follow-up on a continuation run gets, as intended behaviour, and a test pins it. Or the watcher passes the followed run's model/cwd through explicitly, the way it already passes agent=info.agent.

Suggestions

  • Calling spawn_continue directly with a continuation run's id still keys the busy check and claim by that id, not the root's. The PR maps to the root in the follow-up watcher only. A one-line note in the description, or a short follow-up issue, would keep it from being mistaken for covered.

[DESIGN-REVIEWED] 22b5680

Description read

[DESCRIPTION-READ] 859e17913cd1a0a2dac10757a9f521318e4064557fa91d02a4a9a6b1170a0de7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 22b56806e029060b2b7305c650a39bf18c4f2053 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 22b5680

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 22b56806e029060b2b7305c650a39bf18c4f2053 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 22b5680

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 11, 2026
@javenciu
javenciu force-pushed the fix/concurrent-continue-is-refused-busy branch from 72c964b to 41a7f7f Compare October 11, 2026 03:52
@javenciu

Copy link
Copy Markdown
Contributor Author

Thanks, the finding holds, and this branch's second commit introduced it: on main, a follow-up of a failed continuation run is dispatched by the run's own id, which claims the run, so a retry is refused. Fixed in the second commit, now 41a7f7f (amended; the first commit is unchanged).

What was wrong. The follow-up of a continuation run is dispatched by its root conversation id (subagent_manager/continuation.py:1041), so it claims the root and never marks the run it follows up. claim_retry checks only that run's own claim and its own conversation (subagent.py:5544, subagent.py:5549), so after the follow-up started, a dashboard retry of the failed run was granted and the task ran twice. In the other order, a retry first, the follow-up did not see the retry's claim and started as well.

The change. Before it dispatches by the root id, the follow-up also claims the followed-up run with the same _claim_continuation a continuation by the run's own id uses (subagent_manager/continuation.py:1051). A run a retry already claimed refuses the follow-up, which is reported as a failed follow-up. After the dispatch the claim is settled with its outcome (_settle_continuation, subagent_manager/continuation.py:1086), so a follow-up that started leaves the run claimed by its successor, and one that did not start releases it. Only the dispatch itself counts as a start of unknown outcome (raised, set at subagent_manager/continuation.py:1066 and cleared at subagent_manager/continuation.py:1080), so a watcher cancelled in its sleep between busy attempts, as a parent teardown does, releases the claim instead of leaving the run refused for every later retry.

Red. Two tests in test/test_subagent_continuable.py, on the class's existing harness: test_a_retry_of_a_failed_run_whose_follow_up_started_is_refused and test_a_follow_up_of_a_failed_run_already_retried_is_refused. Against 72c964b, 3 of 3 runs:

E           AssertionError: a dashboard retry of cont1 was granted while its follow-up runs
E           AssertionError: the follow-up started although cont1 was retried: ['rootconv']

On main faa0b5b the first test fails at its precondition, because main dispatches the follow-up by the run's own id ([('cont1', '')]); that dispatch claims the run, so main refuses the retry. A third test, test_a_cancel_during_the_busy_retry_sleep_leaves_the_run_retryable, pins that the claim covers only the dispatch: with a second continuation of the root in flight, the follow-up is refused busy, its watcher is cancelled in the sleep before the next attempt, and a dashboard retry of the run is still granted.

Green. On 41a7f7f the test file passes, 199 tests, 3 of 3 runs, and the 23 neighbour test files from the description pass: 2,351 passed.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR makes two concurrent spawn_continue calls on one conversation start one run. It adds a conversation-id claim (_continuations_starting) for a conversation whose opening run has no registry record, and it routes a continuation run's follow-up to its root conversation with a claim on the followed-up run. The follow-up and retry half is correct and well tested. The first half closes the race only when the record is already missing when the first call arrives. Design Review's Watch item is real, and nobody has answered it on this head. I checked it: evict_completed_agents (context_management.py:214) deletes the oldest done records with no regard for _continuation_starting. So if continuation A claims through the registered record's flag (subagent.py _claim_continuation, the else: original._continuation_starting = True branch), and another run finalizes during A's spawn_async awaits and evicts that record, then call B takes the original is None branch (subagent.py:5580). B finds the set empty, claims, and passes _conversation_busy, because A's run is not registered yet. That gives two runs on one conversation, which is exactly the goal this PR states. Suggested fix: in _claim_continuation, always check and add conv_id in _continuations_starting, also when a record exists, while still setting the flag that claim_retry reads. Then release the id in the finally of both continue_conversation paths for every non-refused call, not only when original is None. Add a test that evicts the opening run between A's claim and B's call and shows B refused. Optional, from First Principles: use self._persistence.subagent_id_from_conversation_key(key) instead of the hand-written removeprefix parse in subagent_manager/continuation.py. — crew-pr-reviewer (deep re-check)

@javenciu
javenciu force-pushed the fix/concurrent-continue-is-refused-busy branch from 41a7f7f to 8a3a10d Compare October 12, 2026 09:10
@javenciu

Copy link
Copy Markdown
Contributor Author

Thanks, the eviction route holds on 41a7f7f, and it reached one more holder of the same claim: the follow-up watcher's claim on the run it follows up was also kept on that run's record.

The claim lasted only as long as the registry record. Every continuation start is now claimed by its conversation id in _continuations_starting, with or without a record (subagent.py:5721, subagent.py:5727). A registered run's _continuation_starting flag is still set beside the id (subagent.py:5729), because claim_retry reads it (subagent.py:5681).

One difference from your suggestion: the release is in _settle_continuation, which now takes the conversation id (subagent.py:5753). Both continue_conversation paths call it in their finally for every call that took the claim (subagent.py:5798, subagent.py:5835). A release only in those finally blocks would leak the watcher's claim, because the watcher calls _claim_continuation itself (subagent_manager/continuation.py:1071) and never passes through them. The watcher settles its own claim (subagent_manager/continuation.py:1108), and only a claim it was granted (subagent_manager/continuation.py:1106).

Tests, in test/test_subagent_continuable.py:

  • test_a_record_evicted_during_the_first_start_refuses_the_second is your scenario. The first start claims a registered run and is held in spawn_async, evict_completed_agents drops the record, and the second continuation is refused as starting. A third continuation, after the first returns, starts.
  • test_a_registered_run_is_claimed_by_its_conversation_id replaces test_a_registered_run_keeps_its_flag_claim, which pinned the flag-only claim.
  • test_a_follow_up_keeps_its_claim_when_the_run_record_is_evicted: the watcher's claim on the followed run survives eviction of that run's record.
  • test_a_refused_follow_up_leaves_the_holders_claim_on_the_run: a refused watcher claim leaves the holder's claim in place.

On 41a7f7f the three eviction tests fail in 3 of 3 runs, with a second continuation was granted after the claimed record was evicted, a second run started on one conversation: run <id> and a continuation of cont1 was granted while its follow-up starts. On the new head the file passes, 202 tests in 3 of 3 runs, and the 137 test files that import the two changed modules or name the changed functions pass (9670 passed, 4 skipped). Removing the release from _settle_continuation fails 12 tests, and letting the watcher settle a refused claim fails the last test above.

subagent_id_from_conversation_key. Done: the watcher reads the root id with self._persistence.subagent_id_from_conversation_key (subagent_manager/continuation.py:1059).

Rebased onto main 0a47021 with no conflicts; the branch is still two commits.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

The new commit replaces the short fixed waits that the repo's wait-discipline ratchet (test/test_wait_discipline_ratchet.py) rejects in test/test_subagent_continuable.py; it changes that file only.
The file counted 84 short fixed waits and counts 64 again, its baseline (nineteen 5 s or 10 s ceilings raised to 30 s, the _until poll moved to async_wait_until); the ratchet passes on the head merged with main (c4c51aa), and the file passes 3 of 3 runs (202 tests each).

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 12, 2026
@javenciu
javenciu force-pushed the fix/concurrent-continue-is-refused-busy branch from 02d92f7 to 9d9f45a Compare October 12, 2026 14:51
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
Two spawn_continue calls on one conversation, arriving together, both
started a run when the run that opened the conversation was no longer
in the registry (evicted after it finished, or from before a restart).
_claim_continuation claims a continuation start through the opening
run's _continuation_starting flag, and with no registry record it took
no claim. The conversation-busy lookup then saw neither start until
spawn_async registered the run, after its store write and two
coordinator awaits, so the second call passed it too. A record present
at the first call's claim could also be dropped by
evict_completed_agents while that start was in flight, and the flag
claim went with it.

_claim_continuation now claims every continuation start by its
conversation id in _continuations_starting, in the same synchronous
step as its check, and sets a registered run's flag beside it for
claim_retry. The second call gets the existing "conversation_busy:
another continuation ... is starting" refusal. _settle_continuation
takes the conversation id and ends the claim; continue_conversation
and continue_conversation_async call it in a finally, so a start that
returns, raises or is cancelled always releases it.
…nversation

A continuation run executes on its root conversation (its
conversation_key is subagent:<root id>), but the follow-up watcher
dispatched a follow-up queued on such a run by the run's own id. The
busy check and the claim were then keyed by that id, so while another
run was in flight on the root conversation the follow-up was accepted
and started a second run on it, instead of being refused as
conversation_busy and retried.

The watcher now dispatches by the conversation the run executed on:
the root id read from conversation_key with
subagent_id_from_conversation_key, or the run's own id when the run
opened the conversation. The follow-up then shares the root's busy
check and claim. How a continuation picks its template is unchanged.
Dispatched by the root id, the follow-up would not claim the run it
follows up, which a dashboard retry checks, so it also claims that run
with _claim_continuation and settles that claim with the dispatch's
outcome, only when the claim was granted.
@javenciu
javenciu force-pushed the fix/concurrent-continue-is-refused-busy branch from 9d9f45a to 22b5680 Compare October 12, 2026 20:24
@javenciu

Copy link
Copy Markdown
Contributor Author

Rebased onto main (65df088). The one conflict was the import block of test/test_subagent_continuable.py: main now imports await_until next to async_wait_until (#19334), so I kept main's line. The new tests are unchanged and still fail on main and pass on this branch.

The line references in the description now point at the rebased files.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@javenciu
javenciu enabled auto-merge (squash) October 12, 2026 23:19
@javenciu
javenciu requested a review from iamwhatever October 12, 2026 23:19

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes two spawn_continue calls on one conversation start one run, and routes a continuation run's follow-up through its root conversation's busy check and claim. Head 22b5680 meets my change request point by point: _claim_continuation (subagent.py) now always checks and adds the conversation id in the manager-held _continuations_starting set, with or without a registry record, still sets the record flag claim_retry reads, and _settle_continuation releases the id from the finally of both continue_conversation paths and from the follow-up watcher only when its own claim was granted; test_a_record_evicted_during_the_first_start_refuses_the_second is exactly the eviction case I named, and the author reports it red on the previous head. I tried to break it: a refused call returns before the try, so it never releases someone else's claim; a raising or cancelled start goes through the finally; an unknown conversation id takes and releases the claim; and the watcher's claim covers only the dispatch, so a cancel in its retry sleep leaves the run retryable. Design's Watch item (the follow-up now uses the root's execution context) is not a regression worth holding for: on main the follow-up already passes model=None and cwd="", so neither was inherited from the continuation run, and the remaining fields (backend, memory store, context groups) are carried from the root into every continuation anyway, so the follow-up now runs on the session key the run actually used. First Principles' point that the per-run flag duplicates the set is a valid cleanup but advisory. The only red check is a cancelled Decide whether the macOS suite runs job on the on-demand macOS workflow, not a test failure; the PR is MERGEABLE on base main. — crew-pr-reviewer (deep re-check)

@javenciu
javenciu merged commit defaa0b into kirodotdev:main Oct 13, 2026
104 of 105 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants