Skip to content

fix(subagent): decide a claimed row's start with one store compare-and-set - #20319

Merged
bolichen97 merged 1 commit into
mainfrom
fix/subagent-claim-start-cas-16685
Oct 17, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/subagent-claim-start-cas-16685

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A subagent row the task store has cancelled never starts, however the cancel reaches the store.

The off-loop pump claims a queued row on the store's writer thread, then re-reads it before registering the run. A read is not a write. A cancel that reaches the row through the store alone, with no record on the event loop (an orphan cancel, a reconcile), can commit after that re-read answers and before the pump's coroutine resumes. The pump then registers a run over a cancelled row. The run finishes in memory, every store write it makes is refused as stale_result, and nothing stops it.

Why it matters

Work the user, or the system, cancelled keeps running and spending a slot and a model turn, and the store's record of it says it was cancelled. docs/system-specs/modules/taskq.md documented the window and nothing closed it. The start of a durable row is the one place where loop state and store state must agree, and a read cannot make them agree.

Not a goal

  • Changing what Stop all counts or reports for rows it finds queued or running.
  • Changing the inline (same-thread) pump, which claims and registers in one step and has no such window.
  • Changing how the boot reconciler treats admitted or starting rows.
  • Any new cancel path or a change to TaskStore's API.

What changed (motivation → approach → change)

The claim/start boundary needed a write the store refuses when the row moved, not a read the pump trusts. TaskStore.transition already tests the state and the generation in the same UPDATE it writes, and every cancel bumps the generation. So the claimer now takes the start itself.

  • claim_and_start (pump.py): after the claim, check the loop for a stop recorded while the claim was in flight (_stopped_while_claimed, unchanged), then write admitted -> starting under the claim's generation through store.run(taskq_claim_start). It answers taken, refused, or raises TaskStoreUnavailable (retain the claim, as before). The re-read taskq_claim_still_current and the claimer's _batched_stops join (_batched_stop_of) are gone: a cancel that lands first refuses the write; one that lands later meets a starting row that taskq_cancel_queued's unstarted-row predicate refuses.
  • Between the write landing and the pump's resumption a loop callback can run. A single cancel there finds a starting row with no record. _starting_claims (subagent_parts/manager.py, filed by claim_and_start, read by starting_claim_of) holds the start's answer; cancel_impl (cancellation.py) joins it and then stops the run the pump registered.
  • A taken start the claimer does not register would leave a starting row with no run under it, which boot reconcile reads as a run that started. A re-entry that raises, a re-entry that returns without registering (gateway admission closed meanwhile), and a claimer cancelled while it awaits the shielded start write all release it with one generation-fenced starting -> retry_wait (_release_taken_start, taskq_bridge.py; _release_or_retain_taken_start, pump.py), so the accepted row is claimable again; a joined cancel gets its answer only after that release. A process killed between the start write and registration leaves starting for the boot reconcile, which settles it by side-effect class.
  • A release the store cannot take retains the claim in _retained_claims, like a start-write outage, cancelled claimer or not. taskq_claim_start accepts this claim's own starting row, and taskq_cancel_queued(retained=True) lets a cancel or Stop all end it, so the retry registers, releases, or is refused. The release write is shielded: cancelling the claimer cannot drop it. Only a release outage during shutdown leaves the row to the boot reconcile. A cancel under the claim await is unchanged from base, deferred to Subagent claim: release the admitted row when the claimer is cancelled under its claim await #20391.
  • Stop all's running sweep also takes this parent's rows whose start is in flight (_starting_claim_parents, starting_claim_ids_for): a start that landed before Stop all's pending read is neither an unstarted row nor yet in _agents, so the sweep stops it through cancel, which joins the start.
  • The waits.py cascades, a cancelled parent's (taskq_cancel_children_of[_async]) and a failed sibling's (_cancel_live_or_row), cancel rows they do not count live unconditionally (cancel_tree). _live_agent_ids now adds _starting_claims, so both hand an in-flight start to cancel, which joins it and reaps the run.
  • gate.py's posted admitted -> starting mark stays for the inline pump; for an off-loop claim advance finds the row already starting and writes nothing.
  • Specs: taskq.md (cancel-vs-dispatch, the claim/start split, re-entry raise) and subagent.md (spawn step 5, Stop all, pinned tests) describe the compare-and-set instead of the re-read.

Backwards compatibility

Compatible: no store schema, API, state, or transition changes; admitted -> starting is the edge the posted mark already wrote, now written one step earlier by the off-loop claimer.

Behaviour change (maintainer decision requested on the First Principles thread): a Stop all whose batched cancel reaches the writer after the claimer's start write was submitted now finds the row starting, so the run registers and the same Stop all reaps it as running, counted (1, 0); with the join removed, base's (0, 1) (counted as queued, never started) holds only for the other order. No row is double counted.

Removes nothing: taskq_claim_still_current and _batched_stop_of were internal methods of SpawnAdmissionCoordinator with no callers outside claim_and_start and its tests.

Tests

The claim/start and Stop all tests below live in test/test_taskq_claim_start.py, split out of test/test_taskq_admission_integration.py to keep both under the 2,000-line cap (whole definitions moved, imports from the old file for _manager and _STOP_PARENT; spec pins in subagent.md follow the move). test/test_taskq_bridge_bindings.py adds the three new _TaskqBridgeMixin members (_release_taken_start, _retained_claim_ids_for, _retained_starting_ids) and the new taskq_cancel_queued[_each] signatures to its frozen inventory.

test/test_taskq_claim_start.py:

  • test_a_store_only_cancel_between_claim_and_start_refuses_the_start (kept): a store-only cancel between the claim and the start write refuses the start, now as a stale_result for starting.
  • test_a_store_only_cancel_after_the_start_was_taken_cannot_end_an_unstarted_row (new): after the start write landed and before registration, a cancel carrying the unstarted-row predicate is refused, the run registers at its generation, and its next write is accepted.
  • test_a_single_stop_between_the_taken_start_and_registration_reaps_the_run (new): a single cancel in that gap joins _starting_claims and reaps the registered run (True, reaped, row cancelled, slot back).
  • test_a_retained_claim_stop_all_cancelled_is_refused_on_retry (new, the pin asked for on the issue): a claim retained across a store outage, cancelled by Stop all, is refused by retry_retained_claims -> claim_and_start(retained_generation=...) without asking the store.
  • test_a_claimer_cancelled_under_its_start_write_releases_the_row (new): the claimer's task cancelled with the start write in flight leaves a retry_wait, unleased row, no pending start and the slot back, and the next pump pass claims and starts it under a new generation.
  • test_stop_all_reaching_a_start_in_flight_after_its_pending_read_stops_the_run (new): Stop all's pending read sees the row starting and skips it; the running sweep still stops the run it registers, (1, 0), row cancelled; red without the sweep change.
  • test_a_reentry_refused_by_closed_admission_releases_the_start_to_retry_wait (new): admission closing after the start write landed leaves a retry_wait row with no failed transition and the slot back; red when the finally fails every unregistered taken start.
  • test_a_release_outage_retains_the_claim_and_the_retry_registers_it (new, one and two outages): the claim is retained with one reservation, taskq_claim_start accepts only the claim's own starting row, and the retry registers the run at the same generation; red when a release outage is only logged.
  • test_a_cancelled_claimer_retains_a_release_outage_unless_shutting_down (new, two cases): the cancel propagates; a live manager retains and the retry registers, a shutting-down one leaves the row.
  • test_a_retained_starting_claim_is_stopped_and_refused_on_retry (new, single_cancel, stop_all): a claim retained across a release outage, row starting, is cancelled and its retry refused; red without retained=.
  • test_a_release_queued_behind_another_writer_job_survives_the_claimers_cancel (new, four cases): the claimer cancelled while its release waits behind another writer job still lands the release, then re-raises the cancel.
  • test_a_cancelled_claimer_retains_a_start_outage_unless_shutting_down (new, two cases): the claimer cancelled while its start write fails with the store unavailable propagates the cancel; a live manager retains the admitted claim and the retry registers it, a shutting-down one retains nothing; red without the retain.
  • test_a_parent_cancel_cascade_reaps_a_child_whose_start_is_in_flight (new, two cases: parent_cancelled, sibling_failed): each cascade leaves a starting row with its start in flight to the start, and the scheduled cancel reaps the run it registers; both red without the waits.py change.
  • test_stop_all_between_claim_and_start_keeps_the_row_stopped (kept, two cases): Stop all before the start write still ends the row stopped as a queued row. The former third case placed Stop all after the re-read; a Stop all after the start write finds a starting row and reaps the run, which test_a_row_registered_during_stop_alls_read_is_reaped_not_replaced (kept, asserts starting) pins.
  • test_a_claim_resumed_between_stop_alls_post_and_answer_is_reaped_once (rewritten from ..._refuses_the_start): Stop all posting its batch after the start write counts the run once, as running, and the row's only cancel is the reap's own.
  • test_a_queued_stop_record_installed_during_the_release_does_not_keep_the_reservation (new): a queued-stop record installed while the claimer awaits the release of an unregistered start does not count as a registration; the reservation goes back and the pending start answers False. claim_and_start now decides registered right after the re-entry, before the release awaits. Red without that.
  • test_a_stop_of_a_start_in_flight_reads_the_store_off_the_loop (new): a cancel of a row whose start is in flight (the path Stop all's sweep and the waits.py cascades take) runs its store phase through taskq_cancel_queued_async on the writer thread, and still reaps the run. Red with the sync call on the loop.
  • test_a_reentry_that_raises_after_registering_keeps_the_live_run (new): a re-entry that registers the run and then raises keeps the run live, its row starting and its slot; no release is written and the pending start answers True. Red when the except arm released every taken start.
  • test_claim_revalidation_outage_retains_generation_and_slot_until_retry, test_a_claim_of_a_row_the_store_never_saw_still_starts: gate on taskq_claim_start.

test/test_taskq_unconditional_cancel_callers.py (new): names every runtime caller of an unconditional task-store cancel (TaskStore.cancel without only_from, WaitLedger.cancel_tree) with why it cannot land on a starting row the claimer is about to register (boot-only, or routed through _live_agent_ids), and fails on a new one; red with a probe caller added.

test/test_overload_integration_glue.py::test_a_raise_inside_reenter_releases_the_reservation: a re-entry that raises after the start was taken releases the row to retry_wait at the claim's generation, unleased, with no failed transition, and the reservation goes back.

test/test_taskq_reconcile.py::test_a_row_killed_in_starting_with_no_run_settles_by_its_class (new, three classes): a row killed in starting with no run is parked (unknown) or recovered with one lost-owner recovery spent (none, idempotent_key).

The touched tests wait on signals (kiro_crew.testing.wait.await_until / async_wait_until, and the _finished helper for a coroutine the test calls) instead of asyncio.wait_for(..., 10), so both files stay at zero short fixed waits under test/test_wait_discipline_ratchet.py.

Determinism proof (testing-conventions D12), run on the 33 new/changed node ids: see the PR comments posted with the results.

Manual verification

N/A — unit coverage sufficient: every interleaving is pinned with explicit barriers on the real writer-thread pump, and the related backend set (run_scoped_tests.py --surface backend, 1975 files) passed except 22 failures that reproduce identically on a clean origin/main checkout in this environment (path and tool-availability tests unrelated to this change).

Related Issues

Closes #16685

Pattern harvest

Rule candidate: review-prompt
Pattern: a post-write re-read used as a guard where the write it guards must be the compare-and-set itself (read-then-act on a store another writer can move between the read and the act).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners October 14, 2026 13:50
@bolichen97
bolichen97 requested a review from mbriones98 October 14, 2026 13:51
@bolichen97

Copy link
Copy Markdown
Contributor Author

Determinism proof for the new and changed tests (testing-conventions D12), run on head 06426c0 in this environment (Linux, Python 3.12.13).

Node ids proven (13):

test/test_taskq_admission_integration.py::test_claim_revalidation_outage_retains_generation_and_slot_until_retry
test/test_taskq_admission_integration.py::test_stop_all_between_claim_and_start_keeps_the_row_stopped[read_before_claim]
test/test_taskq_admission_integration.py::test_stop_all_between_claim_and_start_keeps_the_row_stopped[after_claim]
test/test_taskq_admission_integration.py::test_a_row_registered_during_stop_alls_read_is_reaped_not_replaced
test/test_taskq_admission_integration.py::test_a_claim_resumed_between_stop_alls_post_and_answer_is_reaped_once
test/test_taskq_admission_integration.py::test_a_store_only_cancel_between_claim_and_start_refuses_the_start
test/test_taskq_admission_integration.py::test_a_store_only_cancel_after_the_start_was_taken_cannot_end_an_unstarted_row
test/test_taskq_admission_integration.py::test_a_single_stop_between_the_taken_start_and_registration_reaps_the_run
test/test_taskq_admission_integration.py::test_a_parent_cancel_cascade_reaps_a_child_whose_start_is_in_flight
test/test_taskq_admission_integration.py::test_a_retained_claim_stop_all_cancelled_is_refused_on_retry
test/test_taskq_admission_integration.py::test_a_claimer_cancelled_under_its_start_write_releases_the_row
test/test_taskq_admission_integration.py::test_a_claim_of_a_row_the_store_never_saw_still_starts
test/test_overload_integration_glue.py::test_a_raise_inside_reenter_releases_the_reservation
Run Result
20 x -n0 -p no:cacheprovider 20/20 pass
10 x -n 4 --dist loadgroup --max-worker-restart=2 with test/test_taskq_nested_propagation.py and test/test_taskq_claim_fencing.py as co-tenants 10/10 pass
Shuffled order, both test files (138 node ids), seed 20737, -n0 pass

Negative controls (red on the parent of the fix, green on it):

  • test_a_parent_cancel_cascade_reaps_a_child_whose_start_is_in_flight with waits.py reverted to main: AssertionError: the row is left to its start (cancelled == starting).
  • test_a_store_only_cancel_between_claim_and_start_refuses_the_start on main pins the same refusal, now asserted as the store's stale_result for starting.

Related backend set (scripts/run_scoped_tests.py --surface backend, 1975 files): 41276 passed, 22 failed; the same 22 fail identically on a clean origin/main checkout in this environment (test_acp_spawn_offload::TestRuntimeProjectedLaunchDocuments, code_review_sage/tests/test_store_ancestor_chain, test_auto_improvement_members, test_ci_pytest_progress, test_cli::TestDoctorStt: home-path and tool-availability dependent, unrelated to this change).

@bolichen97
bolichen97 enabled auto-merge (squash) October 14, 2026 13:51
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 0cc529bd7d8bb40fa0ead4dec77eea04164f2433 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

A second loop-side registry (_starting_claims) now has to be joined by every cancel path, and the change to Stop all's count still has no maintainer decision.

Watch

  • New registry every canceller must remember. The store write now closes the window. But it opens a new one: a row can sit in starting while _agents has no record of it. Each canceller had to be taught about that row separately: cancel_impl, Stop all's running sweep (starting_claim_ids_for), _live_agent_ids for the waits.py cascades, and a retained=True widening of taskq_cancel_queued. A future cancel path that only checks _agents, or calls cancel_tree, would cancel nothing or cancel under a taken start, and the registered run would keep going with no one to stop it. Nothing enforces the join, and the claimer now has 3 release/retain branches.
    Clears when: one helper answers "is this id live or starting in this process" for every canceller, and a test fails if a canceller decides from _agents alone; or the description says why a single helper does not fit.
  • Stop all's report changes. The description says: "a Stop all whose batched cancel reaches the writer after the claimer's start write was submitted now finds the row starting … counted (1, 0)". In that order, a row the user saw as queued now starts, spends a model turn, and is reported as stopped while running. The description asks for a maintainer decision, and none is recorded yet.
    Clears when: a maintainer accepts the (1, 0) count on the thread, or an entry under docs/decisions/ records it.

Suggestions

  • cancel_impl joins the start on the store-refused path. Do that check first, before the synchronous taskq_cancel_queued. That would also let the retained and starting special cases in the unstarted-row predicate go.

[DESIGN-REVIEWED] 0cc529b

Description read

[DESCRIPTION-READ] 57caec2b99e6afce8895a601d7c4879ddf750fe51c9fa6cdfdca2857948deed3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 0cc529bd7d8bb40fa0ead4dec77eea04164f2433 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Keeping a claim alive after a failed release is the PR's largest new surface, and it handles a double store failure the Goal never names.

Not justified as shipped

  • Item 5 — oversized: if the start was taken and the release then fails, the claim is kept for retry. That covers two store outages in a row, not "A subagent row the task store has cancelled never starts". Shutdown already "leaves the logged row to the boot reconcile".
    Subtraction: drop retention when the release fails. Delete retained= on taskq_cancel_queued[_each], _retained_claim_ids_for, _retained_starting_ids, and the own-starting fallback in taskq_claim_start. Leave the row to boot reconcile in every case, as shutdown does.
    Clears when: the mechanism is removed, or a human amends the Goal.
  • Item 8 — rides along: rewritten pin. The base test asserted "(0, 1), stopped once, as the queued row it was". The renamed test now asserts (1, 0). This is declared under "Behaviour change (maintainer decision requested)", and the compare-and-set makes it unavoidable once the start write is queued first.

What this change ships

Inventory (8 items) — 6 justified

Intent: a subagent a cancel reached through the store alone never runs; a FIX.

  1. A store-only cancel committed before the start is now refused by one generation-fenced admitted -> starting write — justified
  2. A single Stop that lands between the start write and registration now waits for the start and then reaps the run — justified
  3. Stop all's running sweep now also stops a row whose start is in flight — justified
  4. The cancel cascades for a cancelled parent and a failed sibling now treat an in-flight start as a live run — justified
  5. If the release write fails, the claim is kept for retry, and Stop all can cancel it — oversized
  6. A start the claimer took but never registered goes back to retry_wait — justified
  7. The re-read and the batched-stop join (_batched_stop_of) are deleted — justified
  8. Stop all whose cancel lands after the start write reports (1, 0), not (0, 1) — rides along

[FIRST-PRINCIPLES-REVIEWED] 0cc529b

Description read

[DESCRIPTION-READ] ce8db1f2d4712c8b82fafbc75ed45175f33291d5a6b75270e9e01f03dfc03009

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 0cc529bd7d8bb40fa0ead4dec77eea04164f2433 — this comment is updated in place on each push.

Review details

No findings that block the merge.

FINDING — src/kiro_crew/subagent_manager/cancellation.py:1589 — the modified stored = self._manager._admission.taskq_cancel_queued(agent_id, retained=...) still runs store.get/store.cancel synchronously inside the async cancel_impl, so a single Stop of a queued or retained-claim row can hold the gateway loop for the store's busy timeout while another connection is using the store (no-sync-store-call-from-a-coroutine). The call existed before this PR; the diff only adds the retained argument and moves the other branch off the loop. → Fix: use await self._manager._admission.taskq_cancel_queued_async(agent_id, retained=...) on this branch too, as the starting branch beside it already does.

[OPUS-REVIEWED] 0cc529b

Verdict parsed from the review's SHA-scoped output markers for commit 0cc529bd7d8bb40fa0ead4dec77eea04164f2433.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 0cc529bd7d8bb40fa0ead4dec77eea04164f2433: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 0cc529bd7d8bb40fa0ead4dec77eea04164f2433 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 0cc529bd7d8bb40fa0ead4dec77eea04164f2433 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0cc529b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0cc529bd7d8bb40fa0ead4dec77eea04164f2433: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 14, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-claim-start-cas-16685 branch from 06426c0 to 6dfccc6 Compare October 14, 2026 14:28
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 14, 2026
@bolichen97

bolichen97 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Recover admitted claims when cancellation skips the start write span=c65eacf3d1b8

fixed in ebf3b78: taskq_release_start posts _release_start_db, which reads the row on the writer thread under the claim's generation and moves starting -> retry_wait (the write landed) or admitted -> queued (the queued job was removed by the cancel), both eligible at once and unleased; a row whose generation moved is left alone.
pinned by test_a_claimer_cancelled_before_its_start_write_ran_requeues_the_row: the start job is queued behind an occupied writer, the claimer is cancelled, the row ends queued and the next pump pass dispatches it; red with the previous starting -> retry_wait write (the row stayed admitted).
self-added: yes
mechanism: read-then-move release on the writer thread, generation-fenced

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Stop all coverage without reading _starting_claims

rebutted: Stop all needs no join. Its batch cancel (taskq_cancel_queued_each) is a writer job carrying the unstarted-row predicate; whichever of it and the start write lands first decides. Cancel first: the generation is bumped and taskq_claim_start is refused (test_stop_all_between_claim_and_start_keeps_the_row_stopped, two cases). Start first: the row is starting, the cancel does not accept it, and the claimer's wakeup is queued on the loop ahead of the batch's answer (one writer, FIFO; one loop, FIFO), so _registered_run sees the run and the running sweep reaps it once (test_a_claim_resumed_between_stop_alls_post_and_answer_is_reaped_once, test_a_row_registered_during_stop_alls_read_is_reaped_not_replaced).
_starting_claims is only for the two readers that act on the loop without a writer hop between the row read and the record read: the single cancel's synchronous taskq_cancel_queued, and the parent cascade's _live_agent_ids snapshot.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The "no window" claim and the unconditional store.cancel sites

fixed in 6dfccc6 (spec only): docs/system-specs/modules/taskq.md now states the claim with its scope. While the pump runs, every cancel of a waiting row (Stop all's batch included) carries the unstarted-row predicate, and the parent cascade this process runs reaches a row with its start in flight through cancel (_live_agent_ids counts _starting_claims). The two unconditional cancels named, reconcile_on_boot (taskq/reconcile.py) and the ledger's one-time rebuild of orphans (taskq/waits.py), run before the pump dispatches anything (ensure_coordinator_async on the first pass), so they never race a live claimer.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Determinism proof re-run on head 6dfccc6 (same tree as 31d8df67 before the rebase onto c50003c9; main moved in nothing this PR touches). The 14 proven node ids are the 13 listed above plus:

test/test_taskq_admission_integration.py::test_a_claimer_cancelled_before_its_start_write_ran_requeues_the_row
Run Result
20 x -n0 -p no:cacheprovider 20/20 pass
10 x -n 4 --dist loadgroup --max-worker-restart=2 with test/test_taskq_nested_propagation.py and test/test_taskq_claim_fencing.py as co-tenants 10/10 pass
Shuffled order, both test files (139 node ids), seed 4305, -n0 pass

Negative control for the new test: with _release_start_db replaced by the previous single starting -> retry_wait transition, it fails with 'admitted' == 'queued' (the row stays admitted).

@bolichen97
bolichen97 force-pushed the fix/subagent-claim-start-cas-16685 branch from 6dfccc6 to ebf3b78 Compare October 14, 2026 14:39
@bolichen97

Copy link
Copy Markdown
Contributor Author

Determinism proof re-run on head ebf3b78 (the touched tests now wait on signals through kiro_crew.testing.wait; test_taskq_admission_integration.py has zero short fixed waits and its wait_discipline_baseline.txt line is dropped). Same 14 node ids as above.

Run Result
20 x -n0 -p no:cacheprovider 20/20 pass
10 x -n 4 --dist loadgroup --max-worker-restart=2 with test/test_taskq_nested_propagation.py and test/test_taskq_claim_fencing.py as co-tenants 10/10 pass
Shuffled order, both test files (139 node ids), seed 25990, -n0 pass

test/test_wait_discipline_ratchet.py and test/test_flake_pattern_ratchet.py pass locally on this head (the former was the red Backend Tests (3.12, 2) / (Windows) (2) on 06426c0a).

The fixed disposition above for span c65eacf3d1b8 refers to the same code as this head: 6dfccc6b and ebf3b781 differ only in the two test files and the baseline line.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 14, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-claim-start-cas-16685 branch from ebf3b78 to b65440a Compare October 14, 2026 15:03
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The window after the compare-and-set is closed only for predicated cancels and the cascade

fixed in b65440a, by the first Clears when branch and one more cascade. docs/system-specs/modules/taskq.md (cancel-vs-dispatch) now names the two unconditional store.cancel callers and when they run: reconcile_on_boot's tombstone settle and WaitLedger.rebuild's orphan cancel both run in open_store at boot, and the rebuild again in the coordinator's first build (ensure_coordinator_async), which the first pump pass and the accept path's re-entry run ahead of their claims (taskq/__init__.py open_store: "and only then may the caller dispatch"; pump.py _drain_queue_pass_impl orders ensure_coordinator_async before the pick). Neither can commit between a claimer's start write and its registration.
The other in-process path that cancelled rows unconditionally, the failed-sibling cancel under fail_parent (_cancel_live_or_row), now reads _live_agent_ids like the parent cascade, so a row with its start in flight goes to cancel; pinned by test_a_parent_cancel_cascade_reaps_a_child_whose_start_is_in_flight[sibling_failed], red without the change.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Whether the admitted -> queued branch of _release_start_db belongs here

rebutted: the branch is a defect fix in code this PR adds, not a ride-along. taskq_release_start exists only because of this PR (the claimer's cancellation path around the new start write), and its first shape wrote starting -> retry_wait alone; the GPT lane's upheld blocking finding on 06426c0a (span c65eacf3d1b8) showed that a start job the cancel removed from the writer leaves the row admitted, where that write is refused and the release strands accepted work until a restart. A release that handles only one of the two states the interrupted start can leave is incomplete; the admitted branch is the missing sibling of the starting one, pinned by test_a_claimer_cancelled_before_its_start_write_ran_requeues_the_row.
That base also stranded an admitted row on a cancel during its re-read is true and not what this branch is for: it makes this PR's own release whole.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Determinism proof re-run on head b65440a. Node ids now 16: the 14 above, with the cascade test split into [parent_cancelled] and [sibling_failed].

Run Result
20 x -n0 -p no:cacheprovider 20/20 pass
10 x -n 4 --dist loadgroup --max-worker-restart=2 with test/test_taskq_nested_propagation.py and test/test_taskq_claim_fencing.py as co-tenants 10/10 pass
Shuffled order, both test files (140 node ids), seed 22862, -n0 pass

Negative control: [sibling_failed] with waits.py reverted to main fails (the row is cancelled under its taken start).

@bolichen97
bolichen97 force-pushed the fix/subagent-claim-start-cas-16685 branch from f44cbd8 to 95695a0 Compare October 17, 2026 03:59
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased f44cbd8 -> 95695a0 onto main 1cb15f6. Behavior unchanged; still one commit.

Main split the files this PR touched, so the same hunks moved with the code:

  • _starting_claims / _starting_claim_parents fields: subagent.py -> subagent_parts/manager.py (refactor(subagent): bind SubagentManager methods from subagent_parts #21278).
  • taskq_pending_ids_for[_async], _retained_claim_ids_for, _retained_starting_ids: taskq_bridge.py -> taskq_bridge_reads.py, bound on the class (refactor(subagent): split taskq_bridge.py into five bound owners #21043). The store helper is called through self, as the bindings test requires.
  • taskq_cancel_queued[_each] and taskq_post_cancel_queued (retained=): -> taskq_bridge_cancel.py.
  • test/test_taskq_bridge_bindings.py: three new member rows and the two new cancel signatures.
  • The claim/start tests moved whole into test/test_taskq_claim_start.py: with them test_taskq_admission_integration.py was 2,556 lines, over the 2,000-line cap. Spec pins in subagent.md follow the move.

Local: the 7 touched/related test files, 469 passed; bindings, line-cap and wait-ratchet tests green; black, isort, flake8, mypy clean. Unrelated reds in test_acp_spawn_offload.py, test_design_tweak_backend.py, test_cli.py and others reproduce the same on clean main on this host.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 17, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-claim-start-cas-16685 branch from 95695a0 to d38515e Compare October 17, 2026 04:25
@bolichen97

bolichen97 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author
  • fixed in d38515e: a queued-stop record can no longer consume an unregistered reservation span=bca84a05f2d3

claim_and_start now sets registered right after the re-entry returns, with no await between, and the finally reads that value instead of _agents membership.
A synthetic queued-stop record installed while the release awaits is no registration: the reservation goes back and the pending start answers False.
Pinned by test_taskq_claim_start.py::test_a_queued_stop_record_installed_during_the_release_does_not_keep_the_reservation, red on 95695a0.
self-added: yes
mechanism: registration decided before the release await

@bolichen97

bolichen97 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author
  • fixed in d38515e: a stop of a start in flight reads the store off the loop span=d5ac4f1e6ea4

cancel_impl checks starting_claim_of first. With a start in flight, both store phases (before and after joining the start) go through taskq_cancel_queued_async on the writer thread.
After the join it reaps a registered run, else cancels the released / retained row as unstarted work; the no-start path keeps main's existing behavior.
Pinned by test_taskq_claim_start.py::test_a_stop_of_a_start_in_flight_reads_the_store_off_the_loop, red on 95695a0.
self-added: yes
mechanism: async store phase for in-flight starts in cancel_impl

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 17, 2026
…d-set

The off-loop claimer re-read its row after the claim and before it
registered. A read is not a write: a store-only cancel that committed after
the re-read answered and before the claimer resumed still let it register a
run over a cancelled row, every write of which was then fenced out.

claim_and_start now takes the start itself: one generation-fenced
`admitted -> starting` transition (taskq_claim_start), awaited on the writer
thread before registration. A cancel that committed first bumped the
generation and the write is refused; one that commits later meets a
`starting` row no unstarted-row cancel accepts, and the registered run is
reaped as the live run it is. The loop record check stays ahead of the
write, which is what refuses a claim retained across a store outage whose
row Stop all cancelled meanwhile. The claimer's join of a Stop all batch is
gone with the re-read it protected.

Between the write landing and the claimer's registration a single cancel
finds a `starting` row with no record; it joins the start's answer
(`_starting_claims`) and then reaps the run instead of answering "nothing
running". The parent-cancel and failed-sibling cascades count such a row as
live and hand it to that cancel instead of the ledger's unconditional
cancel_tree; a test names every unconditional cancel caller. Stop all's running
sweep takes the parent's in-flight starts the same way.

A taken start the claimer does not register is never left `starting`: a
re-entry that raises, a re-entry that returns without registering, and a
claimer cancelled while it awaits the shielded start write all release it
with one fenced `starting -> retry_wait`, so the row is claimable again. A
row killed between the write and registration is settled by boot reconcile
by side-effect class, now pinned by a reconcile test.
A release the store cannot take retains the claim like a start-write
outage, and the start write accepts the claim's own `starting` row so the
retry registers or releases without a second slot; a single Stop or Stop
all ends such a retained `starting` row. The release write is shielded, so
cancelling the claimer cannot drop it, and a start write that fails with
the store unavailable under a cancelled claimer retains the claim unless
the manager is shutting down.

Closes #16685
@bolichen97
bolichen97 force-pushed the fix/subagent-claim-start-cas-16685 branch from d38515e to 0cc529b Compare October 17, 2026 04:47
@bolichen97

bolichen97 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author
  • fixed: a re-entry that raises after registering keeps its live run span=bca84a05f2d3

The except arm now sets registered from the claim and _agents before any await, as the return path does, and releases the taken start only when the run was not registered.
A run registered before the raise keeps its starting row, its task and its slot; the finally consumes the reservation and answers the pending start True.
Pinned by test_taskq_claim_start.py::test_a_reentry_that_raises_after_registering_keeps_the_live_run, red on d38515e (a release was written).
self-added: yes
mechanism: registration decided before the release await, on both exits

@bolichen97

bolichen97 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author

The three loop-side cancellers need different answers: cancel_impl joins one id (starting_claim_of), Stop all's sweep lists one parent's ids (starting_claim_ids_for), and the waits.py cascades ask "live or starting" (_live_agent_ids, which now includes _starting_claims).
A canceller that decides from _agents alone and then cancels in the store hits the CAS: an unstarted-row cancel is refused for a starting row, so it cannot end a taken start under its claimer.
The unconditional store cancels that skip that predicate are listed by test_taskq_unconditional_cancel_callers.py, which fails on a new one.
An audit-backed list of every _agents liveness reader, with a pin, is the deferred work in #20572 (deferred-finding), which waits on this PR.

@bolichen97

bolichen97 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author
  • decided: the maintainer accepted the (1, 0) count span=eee411f334b8

Maintainer ruling on this thread: #20319 (comment) accepts option (a), the order-dependent (1, 0) count, with no code change.

@bolichen97

bolichen97 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author
  • partly fixed in 0cc529b, rest rebutted span=62d4515b06b5

cancel_impl now checks starting_claim_of first; with a start in flight both store phases run through taskq_cancel_queued_async, never the sync call.
The retained and starting cases in the unstarted-row predicate stay: a retained claim has no claimer in flight, so there is no start to join, and its starting row is only cancellable through that predicate.

@bolichen97

bolichen97 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: removing the retain reopens a defect the Goal covers span=4940aeb2f55a

Without it, a taken start whose release write fails leaves a starting row with no run and no claimer; boot reconcile then reads it as a run that started, so a stop or cancel the user made could be lost or the row parked.
The retain reuses the existing start-outage path (_retain_claim, same generation and reservation), adds no new state, and is pinned by test_a_release_outage_retains_the_claim_and_the_retry_registers_it.
self-added: yes
mechanism: retain a claim whose release write failed

@bolichen97

bolichen97 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author
  • decided: the maintainer accepted the (1, 0) count the rewritten pin asserts span=77820213c22a

Maintainer ruling on this thread: #20319 (comment) accepts option (a), the order-dependent (1, 0) count, with no code change; the rewritten pin records that ruling.

@bolichen97

bolichen97 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: the sync store phase for a queued row is main's, unchanged here span=24940bf6b055

cancel_impl on main already calls the sync taskq_cancel_queued for a row with no record; this PR only passes retained=, which adds no store call.
The path this PR added (a start in flight, reached by Stop all's sweep and the cascades) now runs its store phase on the writer thread (0cc529b). Moving main's single-Stop path off the loop is a separate change.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 17, 2026
@bolichen97
bolichen97 merged commit 14a3629 into main Oct 17, 2026
106 of 114 checks passed
@bolichen97
bolichen97 deleted the fix/subagent-claim-start-cas-16685 branch October 17, 2026 06:39
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Subagent claim: make the post-claim check a compare-and-set so a store-only cancel cannot slip past it

2 participants