Repository navigation
fix(subagent): decide a claimed row's start with one store compare-and-set - #20319
Conversation
|
Determinism proof for the new and changed tests (testing-conventions D12), run on head 06426c0 in this environment (Linux, Python 3.12.13). Node ids proven (13):
Negative controls (red on the parent of the fix, green on it):
Related backend set ( |
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS A second loop-side registry ( Watch
Suggestions
[DESIGN-REVIEWED] 0cc529b Description read[DESCRIPTION-READ] 57caec2b99e6afce8895a601d7c4879ddf750fe51c9fa6cdfdca2857948deed3 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Keeping a claim alive after a failed release is the PR's largest new surface, and it handles a double store failure the Goal never names. Not justified as shipped
What this change shipsInventory (8 items) — 6 justifiedIntent: a subagent a cancel reached through the store alone never runs; a FIX.
[FIRST-PRINCIPLES-REVIEWED] 0cc529b Description read[DESCRIPTION-READ] ce8db1f2d4712c8b82fafbc75ed45175f33291d5a6b75270e9e01f03dfc03009 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings that block the merge. FINDING — src/kiro_crew/subagent_manager/cancellation.py:1589 — the modified [OPUS-REVIEWED] 0cc529b Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
06426c0 to
6dfccc6
Compare
|
|
|
|
Determinism proof re-run on head 6dfccc6 (same tree as 31d8df67 before the rebase onto
Negative control for the new test: with |
6dfccc6 to
ebf3b78
Compare
|
Determinism proof re-run on head ebf3b78 (the touched tests now wait on signals through
The |
ebf3b78 to
b65440a
Compare
|
|
|
Determinism proof re-run on head b65440a. Node ids now 16: the 14 above, with the cascade test split into
Negative control: |
f44cbd8 to
95695a0
Compare
|
Rebased f44cbd8 -> 95695a0 onto main 1cb15f6. Behavior unchanged; still one commit. Main split the files this PR touched, so the same hunks moved with the code:
Local: the 7 touched/related test files, 469 passed; bindings, line-cap and wait-ratchet tests green; black, isort, flake8, mypy clean. Unrelated reds in |
95695a0 to
d38515e
Compare
|
|
…d-set The off-loop claimer re-read its row after the claim and before it registered. A read is not a write: a store-only cancel that committed after the re-read answered and before the claimer resumed still let it register a run over a cancelled row, every write of which was then fenced out. claim_and_start now takes the start itself: one generation-fenced `admitted -> starting` transition (taskq_claim_start), awaited on the writer thread before registration. A cancel that committed first bumped the generation and the write is refused; one that commits later meets a `starting` row no unstarted-row cancel accepts, and the registered run is reaped as the live run it is. The loop record check stays ahead of the write, which is what refuses a claim retained across a store outage whose row Stop all cancelled meanwhile. The claimer's join of a Stop all batch is gone with the re-read it protected. Between the write landing and the claimer's registration a single cancel finds a `starting` row with no record; it joins the start's answer (`_starting_claims`) and then reaps the run instead of answering "nothing running". The parent-cancel and failed-sibling cascades count such a row as live and hand it to that cancel instead of the ledger's unconditional cancel_tree; a test names every unconditional cancel caller. Stop all's running sweep takes the parent's in-flight starts the same way. A taken start the claimer does not register is never left `starting`: a re-entry that raises, a re-entry that returns without registering, and a claimer cancelled while it awaits the shielded start write all release it with one fenced `starting -> retry_wait`, so the row is claimable again. A row killed between the write and registration is settled by boot reconcile by side-effect class, now pinned by a reconcile test. A release the store cannot take retains the claim like a start-write outage, and the start write accepts the claim's own `starting` row so the retry registers or releases without a second slot; a single Stop or Stop all ends such a retained `starting` row. The release write is shielded, so cancelling the claimer cannot drop it, and a start write that fails with the store unavailable under a cancelled claimer retains the claim unless the manager is shutting down. Closes #16685
d38515e to
0cc529b
Compare
|
|
|
|
|
|
|
Problem / Motivation
Goal: A subagent row the task store has cancelled never starts, however the cancel reaches the store.
The off-loop pump claims a queued row on the store's writer thread, then re-reads it before registering the run. A read is not a write. A cancel that reaches the row through the store alone, with no record on the event loop (an orphan cancel, a reconcile), can commit after that re-read answers and before the pump's coroutine resumes. The pump then registers a run over a cancelled row. The run finishes in memory, every store write it makes is refused as
stale_result, and nothing stops it.Why it matters
Work the user, or the system, cancelled keeps running and spending a slot and a model turn, and the store's record of it says it was cancelled.
docs/system-specs/modules/taskq.mddocumented the window and nothing closed it. The start of a durable row is the one place where loop state and store state must agree, and a read cannot make them agree.Not a goal
admittedorstartingrows.TaskStore's API.What changed (motivation → approach → change)
The claim/start boundary needed a write the store refuses when the row moved, not a read the pump trusts.
TaskStore.transitionalready tests the state and the generation in the sameUPDATEit writes, and every cancel bumps the generation. So the claimer now takes the start itself.claim_and_start(pump.py): after the claim, check the loop for a stop recorded while the claim was in flight (_stopped_while_claimed, unchanged), then writeadmitted -> startingunder the claim's generation throughstore.run(taskq_claim_start). It answers taken, refused, or raisesTaskStoreUnavailable(retain the claim, as before). The re-readtaskq_claim_still_currentand the claimer's_batched_stopsjoin (_batched_stop_of) are gone: a cancel that lands first refuses the write; one that lands later meets astartingrow thattaskq_cancel_queued's unstarted-row predicate refuses.cancelthere finds astartingrow with no record._starting_claims(subagent_parts/manager.py, filed byclaim_and_start, read bystarting_claim_of) holds the start's answer;cancel_impl(cancellation.py) joins it and then stops the run the pump registered.startingrow with no run under it, which boot reconcile reads as a run that started. A re-entry that raises, a re-entry that returns without registering (gateway admission closed meanwhile), and a claimer cancelled while it awaits the shielded start write all release it with one generation-fencedstarting -> retry_wait(_release_taken_start,taskq_bridge.py;_release_or_retain_taken_start,pump.py), so the accepted row is claimable again; a joined cancel gets its answer only after that release. A process killed between the start write and registration leavesstartingfor the boot reconcile, which settles it by side-effect class._retained_claims, like a start-write outage, cancelled claimer or not.taskq_claim_startaccepts this claim's ownstartingrow, andtaskq_cancel_queued(retained=True)lets a cancel or Stop all end it, so the retry registers, releases, or is refused. The release write is shielded: cancelling the claimer cannot drop it. Only a release outage during shutdown leaves the row to the boot reconcile. A cancel under the claim await is unchanged from base, deferred to Subagent claim: release the admitted row when the claimer is cancelled under its claim await #20391._starting_claim_parents,starting_claim_ids_for): a start that landed before Stop all's pending read is neither an unstarted row nor yet in_agents, so the sweep stops it throughcancel, which joins the start.waits.pycascades, a cancelled parent's (taskq_cancel_children_of[_async]) and a failed sibling's (_cancel_live_or_row), cancel rows they do not count live unconditionally (cancel_tree)._live_agent_idsnow adds_starting_claims, so both hand an in-flight start tocancel, which joins it and reaps the run.gate.py's postedadmitted -> startingmark stays for the inline pump; for an off-loop claimadvancefinds the row alreadystartingand writes nothing.taskq.md(cancel-vs-dispatch, the claim/start split, re-entry raise) andsubagent.md(spawnstep 5, Stop all, pinned tests) describe the compare-and-set instead of the re-read.Backwards compatibility
Compatible: no store schema, API, state, or transition changes;
admitted -> startingis the edge the posted mark already wrote, now written one step earlier by the off-loop claimer.Behaviour change (maintainer decision requested on the First Principles thread): a Stop all whose batched cancel reaches the writer after the claimer's start write was submitted now finds the row
starting, so the run registers and the same Stop all reaps it as running, counted(1, 0); with the join removed, base's(0, 1)(counted as queued, never started) holds only for the other order. No row is double counted.Removes nothing:
taskq_claim_still_currentand_batched_stop_ofwere internal methods ofSpawnAdmissionCoordinatorwith no callers outsideclaim_and_startand its tests.Tests
The claim/start and Stop all tests below live in
test/test_taskq_claim_start.py, split out oftest/test_taskq_admission_integration.pyto keep both under the 2,000-line cap (whole definitions moved, imports from the old file for_managerand_STOP_PARENT; spec pins insubagent.mdfollow the move).test/test_taskq_bridge_bindings.pyadds the three new_TaskqBridgeMixinmembers (_release_taken_start,_retained_claim_ids_for,_retained_starting_ids) and the newtaskq_cancel_queued[_each]signatures to its frozen inventory.test/test_taskq_claim_start.py:test_a_store_only_cancel_between_claim_and_start_refuses_the_start(kept): a store-only cancel between the claim and the start write refuses the start, now as astale_resultforstarting.test_a_store_only_cancel_after_the_start_was_taken_cannot_end_an_unstarted_row(new): after the start write landed and before registration, a cancel carrying the unstarted-row predicate is refused, the run registers at its generation, and its next write is accepted.test_a_single_stop_between_the_taken_start_and_registration_reaps_the_run(new): a singlecancelin that gap joins_starting_claimsand reaps the registered run (True, reaped, row cancelled, slot back).test_a_retained_claim_stop_all_cancelled_is_refused_on_retry(new, the pin asked for on the issue): a claim retained across a store outage, cancelled by Stop all, is refused byretry_retained_claims -> claim_and_start(retained_generation=...)without asking the store.test_a_claimer_cancelled_under_its_start_write_releases_the_row(new): the claimer's task cancelled with the start write in flight leaves aretry_wait, unleased row, no pending start and the slot back, and the next pump pass claims and starts it under a new generation.test_stop_all_reaching_a_start_in_flight_after_its_pending_read_stops_the_run(new): Stop all's pending read sees the rowstartingand skips it; the running sweep still stops the run it registers,(1, 0), rowcancelled; red without the sweep change.test_a_reentry_refused_by_closed_admission_releases_the_start_to_retry_wait(new): admission closing after the start write landed leaves aretry_waitrow with nofailedtransition and the slot back; red when thefinallyfails every unregistered taken start.test_a_release_outage_retains_the_claim_and_the_retry_registers_it(new, one and two outages): the claim is retained with one reservation,taskq_claim_startaccepts only the claim's ownstartingrow, and the retry registers the run at the same generation; red when a release outage is only logged.test_a_cancelled_claimer_retains_a_release_outage_unless_shutting_down(new, two cases): the cancel propagates; a live manager retains and the retry registers, a shutting-down one leaves the row.test_a_retained_starting_claim_is_stopped_and_refused_on_retry(new,single_cancel,stop_all): a claim retained across a release outage, rowstarting, is cancelled and its retry refused; red withoutretained=.test_a_release_queued_behind_another_writer_job_survives_the_claimers_cancel(new, four cases): the claimer cancelled while its release waits behind another writer job still lands the release, then re-raises the cancel.test_a_cancelled_claimer_retains_a_start_outage_unless_shutting_down(new, two cases): the claimer cancelled while its start write fails with the store unavailable propagates the cancel; a live manager retains theadmittedclaim and the retry registers it, a shutting-down one retains nothing; red without the retain.test_a_parent_cancel_cascade_reaps_a_child_whose_start_is_in_flight(new, two cases:parent_cancelled,sibling_failed): each cascade leaves astartingrow with its start in flight to the start, and the scheduledcancelreaps the run it registers; both red without thewaits.pychange.test_stop_all_between_claim_and_start_keeps_the_row_stopped(kept, two cases): Stop all before the start write still ends the row stopped as a queued row. The former third case placed Stop all after the re-read; a Stop all after the start write finds astartingrow and reaps the run, whichtest_a_row_registered_during_stop_alls_read_is_reaped_not_replaced(kept, assertsstarting) pins.test_a_claim_resumed_between_stop_alls_post_and_answer_is_reaped_once(rewritten from..._refuses_the_start): Stop all posting its batch after the start write counts the run once, as running, and the row's only cancel is the reap's own.test_a_queued_stop_record_installed_during_the_release_does_not_keep_the_reservation(new): a queued-stop record installed while the claimer awaits the release of an unregistered start does not count as a registration; the reservation goes back and the pending start answersFalse.claim_and_startnow decidesregisteredright after the re-entry, before the release awaits. Red without that.test_a_stop_of_a_start_in_flight_reads_the_store_off_the_loop(new): acancelof a row whose start is in flight (the path Stop all's sweep and thewaits.pycascades take) runs its store phase throughtaskq_cancel_queued_asyncon the writer thread, and still reaps the run. Red with the sync call on the loop.test_a_reentry_that_raises_after_registering_keeps_the_live_run(new): a re-entry that registers the run and then raises keeps the run live, its rowstartingand its slot; no release is written and the pending start answersTrue. Red when the except arm released every taken start.test_claim_revalidation_outage_retains_generation_and_slot_until_retry,test_a_claim_of_a_row_the_store_never_saw_still_starts: gate ontaskq_claim_start.test/test_taskq_unconditional_cancel_callers.py(new): names every runtime caller of an unconditional task-store cancel (TaskStore.cancelwithoutonly_from,WaitLedger.cancel_tree) with why it cannot land on astartingrow the claimer is about to register (boot-only, or routed through_live_agent_ids), and fails on a new one; red with a probe caller added.test/test_overload_integration_glue.py::test_a_raise_inside_reenter_releases_the_reservation: a re-entry that raises after the start was taken releases the row toretry_waitat the claim's generation, unleased, with nofailedtransition, and the reservation goes back.test/test_taskq_reconcile.py::test_a_row_killed_in_starting_with_no_run_settles_by_its_class(new, three classes): a row killed instartingwith no run is parked (unknown) or recovered with one lost-owner recovery spent (none,idempotent_key).The touched tests wait on signals (
kiro_crew.testing.wait.await_until/async_wait_until, and the_finishedhelper for a coroutine the test calls) instead ofasyncio.wait_for(..., 10), so both files stay at zero short fixed waits undertest/test_wait_discipline_ratchet.py.Determinism proof (testing-conventions D12), run on the 33 new/changed node ids: see the PR comments posted with the results.
Manual verification
N/A — unit coverage sufficient: every interleaving is pinned with explicit barriers on the real writer-thread pump, and the related backend set (
run_scoped_tests.py --surface backend, 1975 files) passed except 22 failures that reproduce identically on a cleanorigin/maincheckout in this environment (path and tool-availability tests unrelated to this change).Related Issues
Closes #16685
Pattern harvest
Rule candidate: review-prompt
Pattern: a post-write re-read used as a guard where the write it guards must be the compare-and-set itself (read-then-act on a store another writer can move between the read and the act).
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)