Repository navigation
perf(subagent): batch bulk-stop store cancels on the writer thread - #16394
Conversation
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Dropping window entries before the batched cancel lands brings in six compensating mechanisms. The PR never weighs them against the await-then-unqueue shape that teardown already uses. Not justified as shipped
What this change shipsInventory (10 items) — 4 justifiedIntent: Stop all on a parent with many queued subagents should no longer freeze the gateway or log a warning for every row — a FIX (perf).
[FIRST-PRINCIPLES-REVIEWED] 90ee1fd Description read[DESCRIPTION-READ] 03ae1fcef1a9083c19d9b941cbaa94abc9d2974d54e5d54003c1267bb0541e83 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that a stuck writer thread can't leave Watch
[DESIGN-REVIEWED] 90ee1fd Description read[DESCRIPTION-READ] 4a90f540210f69e62594406cf690c192b9fcf6ae985bc0d164bcefac5978f749 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
5bc4c0c to
d9f7e59
Compare
|
GPT 6.1 F1 (cancellation.py:1137, "Preserve bulk-stop reports across shutdown") — fixed in part, rebutted in part. Head 9d957c0 (rebased again over #15869; the code these address is unchanged). The finding makes two claims, and I handled them differently. 1. "If application finishes during the drain, its newly spawned reporters also escape the drain's snapshot." Fixed. This was a real defect. 2. "The applier is cancelled at the 30 s drain deadline, so rows cancelled in the store lose their reports." Not widened by this PR, so no code change. To reach that state, shutdown has to begin during a Stop all whose writer job outlives the 30 s drain. Two things make that a non-regression:
Making a cancelled row's report survive a SIGKILL would need a persisted pending-report record. That is new durable state for a pre-existing shutdown residual, which is out of scope for U8. |
|
First-principles CONCERNS, item 5 (the DEBUG downgrade of a same-state refused settle): kept, with the path it serves now named and pinned. Head 9d957c0 (rebased again over #15869; the code these address is unchanged). The review's condition for clearing was: "someone names a path that still settles a row whose cancel already landed". Here is one: a queued stop whose own cancel did not land. The review was right that the comment named a case this PR had removed (a single
A refusal over any other state still logs at WARNING, so the audibility rule in |
d9f7e59 to
9d957c0
Compare
|
GPT 6.1 F1 on 9d957c0 (cancellation.py:1131, "Shutdown permanently drops committed bulk-stop completion reports"): rebutted. Not reachable as described, and not widened relative to main. The finding's mechanism is: "remaining cancels exceed the 30-second drain budget ->
A narrower follow-up remains possible without new durable state: report each row the moment its cancel commits, through a |
|
/ai-review override gpt 9d957c0: the 30 s straggler cancel is unreachable from the gateway, which cuts shutdown at 10 s and force-exits. Reports still pending at that cut-off are lost on main too (and there Stop all froze the loop). This PR drains every report its applier spawns within the same budget. A durable outbox for queued-stop reports is new scope and goes to a follow-up. |
Human judgment recorded@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
buluoray
left a comment
There was a problem hiding this comment.
Verdict: REQUEST CHANGES
Reviewed head 9d957c0210f3c25bdebef30c8d9f322409bf053e. CI on that SHA: 83 success, 12 skipped, 0 failure. Auto-merge is off and no human review has been posted. I did not run the full suite. I ran only test/test_queue_depth_reconcile.py, under the mutations listed below, plus one probe.
Blocking
A single cancel() of a windowed row while Stop all's batched job is on the writer thread produces two subagent_done reports for that row (cancellation.py:1121-1133, _stop_queued, together with _report_stopped_rows at :1160-1170).
_stop_queued now pops the window entries and then suspends on the writer-thread job. While it is suspended, cancel_impl (:1196-1209) can run for one of those rows. The row has no _agents record yet, so cancel_impl calls the synchronous taskq_cancel_queued on the loop. That call lands, because the batch has not reached the row yet. _unqueue finds no window entry and returns stored, and _report_queued_stop(..., row_settled=True) reports the row. When the batch then reaches the row, its cancel returns None. _report_stopped_rows sees entry is not None and stored is None, so it calls _repost_unlanded_cancel (a spurious WARNING plus an extra writer job) and then _report_queued_stop(entry) a second time. _report_queued_stop_impl creates a fresh SubagentInfo on every call, and _claim_finalize_impl keys the claim on that object's _finalized. So the second claim also succeeds: you get a second subagent_done and a second completion injected into the parent, and batch accounting counts that member twice.
This is fail-open: it emits a second terminal response. It is new in this PR. On main, _stop_queued contains no await, so no other stop can run between a row's unqueue and its report. It is reachable in the production default (pump_off_loop = True, admission/__init__.py:73). The inline mode does not hit it, because there the batch runs synchronously.
Repro: I added a test to the PR's own fixtures and confirmed it fails, then deleted it.
mgr = await _manager(monkeypatch, pump_off_loop=True, max_concurrent=1)
with patch.object(SubagentManager, "_run", new=_park):
mgr.spawn("holds the slot", parent_session_key="dash:elsewhere")
rows = [mgr.spawn(f"t{i}", parent_session_key=_PARENT) for i in range(3)]
await _settle(mgr)
events = _record(mgr)
entered, release, _ = _contend_row_cancels(monkeypatch)
stop = asyncio.ensure_future(mgr.cancel_for_parent(_PARENT))
await _until(entered.is_set, "batch on writer thread")
await mgr.cancel(rows[-1].id) # user presses Stop on one queued card
release.set(); await stop; await _settle(mgr)
# subagent_done ids: [c521…, 8174…, 902c…, c521…] <- rows[-1] reported twiceSuggested fix: before reporting a row whose batched cancel did not land, check whether another stop has already reported it. For example, skip the row in _report_stopped_rows (and skip _repost_unlanded_cancel for it) when _agents.get(id) is a queued record that is already _finalized. Alternatively, make _report_queued_stop_impl refuse when an existing queued record for that id is finalized. Add the scenario above as a pinning test. #16390's new guard in _report_queued_stop_impl does not cover this case: it only skips records that are not queued.
Non-blocking follow-ups
- A row whose batched cancel raised loses its window entry.
_stop_queuedpops every entry before it knows any outcome (:1123), and_report_stopped_rowsnever restores the entry of a row whose cancel raised. On main,_unqueueraised before the pop, so that row stayed in the window. The behavior is fail-closed: the store row is stillqueued, and the next refill windows it again once_stopping_parentsis released.test_a_row_that_could_not_be_unqueued_fails_the_stop_and_reaps_nothingasserts the store state but not the window state. Either restore the entry or say in the docstring that the entry is dropped on purpose. - No bound on the request's wait.
await asyncio.shield(applying)waits on a single job that runs N cancels serially, each of which can take up to the store's busy timeout. The loop stays free, but the Stop all request and the writer thread can both be held for N × busy timeout, and every other session's posted settles and claims queue behind that job. The design bot raised the same point. Consider chunking the job, or a per-pass deadline that leaves the remaining rows to the re-post path. - Drain membership test is quadratic.
cancel_all's drain checkst not in pending_reportsagainst a list (cancellation.py:~1391). Use a set. - Store close mid-batch.
TaskStore.close()shuts the executor down withcancel_futures=True, so a queued batch future is cancelled,_applynever reports, and the window entries are already gone. This only happens at shutdown, and the rows stayqueuedon disk. Worth one line in the spec's failure list.
Bot dispositions
- GPT F1 (shutdown drops reports), human override. I checked the override reasoning:
GRACEFUL_SHUTDOWN_SECS = 10(gateway_shutdown_budget.py:6), andgateway.py:13119wraps_shutdown()in that budget, so the 30 s straggler cancel cannot be reached from the gateway. The override holds. - First-principles PASS with a check on the
_stopping_parentsscope. The fence is a counter that thefinallyatcancellation.py:781-785releases on every exit, including failure and cancellation, so a failed pass does not keep its rows out of the window. - Design PASS with a check that one N-row job could starve the writer. That concern is real and does not gate this PR (follow-up 2).
Overlap with #16390 and #16395 (merge order)
- #16394 × #16390: real conflict.
git merge-treereports content conflicts insrc/kiro_crew/subagent_manager/cancellation.py(_stop_queued, which #16390 patches in its old synchronous form and #16394 rewrites) and indocs/system-specs/modules/subagent.md. The conflict is semantic as well as textual. #16390's "registered after the ids were read, skip" check has to move into the new_stop_queuedbeforetaskq_post_cancel_queued, so the batch never posts a cancel for a registered run. #16390's_report_queued_stop_implguard must also be kept, because a run can register between the post and the moment the applier runs. Whichever PR lands second has to do that port and re-run both PRs' tests. #16390'staskq_pending_ids_for_async(include_admitted=True)composes with the batch (allow_admitteddefaults to True), so that part needs no change. - #16394 × #16395: clean merge (
git merge-treeexits clean). #16395 adds aninclude_windowkeyword totaskq_pending_ids_for_async, and #16394 calls that function positionally, so the call still works. #16395 adds no new caller of_stop_queuedor_report_queued_stop. Its parent-end edits sit next to #16394'srow_settledchange at the teardown report (cancellation.py:683) and merge cleanly. Neither order creates a dependency. - For reference, #16390 × #16395 conflicts in
admission/taskq_bridge.py: both PRs rewritetaskq_pending_ids_for_async. - All three PRs merge cleanly onto
origin/main(f02788522a) on their own.
What I verified
- I read the full source diff at the head SHA, plus
taskq_cancel_queuedandTaskStore.post/_writer_executor/close. The writes stay on the writer thread:taskq_cancel_queued_eachruns as a singlestore.postjob on the single-worker executor. The inline path is used only with no running loop or withpump_off_loop=False, and that matches_post_store_write's rule. The connection ischeck_same_thread=Falseand guarded by the store's lock, which is the pattern main already uses. Nothing loop-bound is touched from the writer thread. - One row's failure does not skip the rest.
taskq_cancel_queued_eachcatches each id separately._report_stopped_rowsreports every row that succeeded and then re-raises the first error. - Ordering against other stops. The post happens before the pop and before the first await, so later refills and claims queue behind the cancels. A parent-end teardown or stage cancel during the batch posts its own cancel behind the batch, finds no entry, and does not report. Only the synchronous on-loop single
cancel()can interleave, and that is the blocking finding. - Mutations (each run on
test/test_queue_depth_reconcile.py, restored afterwards,git status --shortclean):- Removed the
_refill_apply_stopping_parentsskip:test_a_refill_queued_before_stop_all_windows_none_of_the_parents_rowsfails. - Removed the per-id catch in
taskq_cancel_queued_each:test_a_row_that_could_not_be_unqueued_fails_the_stop_and_reaps_nothing[writer-thread, on-loop]fails. - Replaced
asyncio.shield(applying)with a plainawait:test_a_stop_all_cancelled_mid_batch_still_reports_every_row_it_cancelledfails. - Made the drain take a single snapshot (stopped adding the tasks that joined):
test_shutdown_during_stop_all_drains_the_reports_its_applier_spawnsfails. - Disabled the
row_settledearly return intaskq_settle: all 8test_a_queued_stop_skips_the_settle_its_landed_cancel_already_wrote[...]cases fail. - Moved the pop before the post:
test_a_stop_all_whose_post_raises_leaves_the_window_whole[writer-thread, on-loop]fails. - Removed the batch's
_repost_unlanded_cancel:test_stop_all_reposts_a_row_cancel_that_did_not_land[writer-thread, on-loop]fails. - Removed the DEBUG downgrade:
test_a_refused_settle_is_a_warning_only_when_the_row_holds_another_state, plus bothrepostscases, fail.
- Removed the
- Not verified: the PR's gate numbers, mypy, and the frontend shards. I took those from CI and the PR body only.
|
Reviewed, not approving yet: buluoray has changes requested on the current head (9d957c0), and auto-merge is on, so my approval would ship it past that request. This batches bulk-stop store cancels on the writer thread (~800 lines in the subagent stop path). I will review once buluoray's points are resolved. |
9d957c0 to
7916330
Compare
|
GPT 6.1 F1 on 2aef22a ( The mechanism is as described. The fix is what the finding proposes:
Each row is still cancelled once and reported once, and its Test: |
|
Design CONCERNS on 2aef22a. Watch item 1 is accepted for the reader it missed, and the placeholder design is declined. Watch item 2 is answered. New head 1. "Invisible in-flight state guarded only by join points." The concern was right. The GPT lane found a fourth reader that had no join: the parent-end snapshot. It is fixed in
The Why not a placeholder
So the side map plus one join per reader is the smaller change. The table above is the list of readers that the Clears-when asks for. 2. "Backwards-compatibility section cut off." That was the review's description truncation, not the description itself. In full, the section reads: "Compatible: no API, config or persisted shape changes. |
|
@buluoray, one more push since #16394 (comment): The GPT lane found that a parent end during Stop all's batch let the batch's stop reports inject into the conversation that had just ended. The batched rows are in neither |
|
Design CONCERNS on 657193e (advisory). Item 1 is kept, and the list of readers is given below. Item 2 is answered. No code change this round. 1. "The new in-flight state has no single owner." I am keeping the shape and listing the readers, which is the second option in Clears-when. The table in #16394 (comment) covers every reader that acts on an unstarted row's id. The rest of the
Why I am not keeping the entries in the window marked "stopping": every 2. "The take-away check is capped." The cap is the review's 8000-byte read of the description, not a gap in the description. The full |
657193e to
f4e92f0
Compare
|
@buluoray, a rebase-only push since #16394 (comment): the head is now Your blocking item (a single Could you take another look when you have a moment? |
|
Design CONCERNS on f4e92f0, Watch span=6fe33ebfa498 ("each new race got its own side table"): rebutted, shape kept. Rebase-only head; the code is unchanged from 657193e. The full reasoning and the reader list are in #16394 (comment) and the reader table in #16394 (comment). In short: every |
|
Design CONCERNS on f4e92f0, Watch span=61cecebc315b ("the compatibility check could only partly run"): fixed in the PR description. The |
|
Design CONCERNS on f4e92f0, Suggestion span=a353a30629ec ("keep each popped row in a state existing readers already see"): declined. A |
Stop all cancelled each queued row with a synchronous store call on the event loop. Whenever the task store was contended, that held the loop for the busy timeout once per row; one measured Stop all froze the gateway for 10 s. Each Stop all pass now queues ONE writer-thread job for every row's cancel (taskq_post_cancel_queued) and drops the window entries before it first awaits. A tracked, shielded task applies the answers, so a request cancelled mid-batch still reports every row it cancelled. The parent is held in _stopping_parents across both passes, and _refill_apply windows none of its rows meanwhile, so a refill fetch queued before the stop can neither re-window a row being cancelled nor window a store-only row the pending read would then skip. An id the pump registered as a run after the pass read its ids is left out of the job, so the batch never cancels a live run's row; the running sweep reaps it, as #16390 made the synchronous pass do. Releasing the fence schedules one pump pass: a pass that ran under it windowed nothing of the parent's, and a row spawned after the passes read their ids would otherwise wait on disk beside a free slot. A cancel that landed is the row's terminal write. Its queued-stop report now skips the settle's finish and keeps the propagation (row_settled). Stop all, the stage Cancel, the parent-end teardown and a single cancel all pass it. A refused settle whose row already holds the requested state logs at DEBUG instead of WARNING. cancel_all's report drain re-reads _report_tasks after each wait inside its one budget: a Stop all's applier spawns each row's report after the drain's first snapshot, and a drain that never re-read returned with those reports still pending as the loop closed. Specs: subagent.md (cancel_for_parent, Cancel, cancel_all) and taskq.md (on-loop classes, refused settle, cancel callers).
A single cancel of a queued row, made while a Stop all batch was on the writer thread, reported that row twice. The batch had already popped the row's window entry and the row has no _agents record, so the single cancel took it for a store-only row: it cancelled it on the loop, landed first and reported it. The batch's own cancel then came back empty, so it re-posted the cancel with a WARNING and reported the row again with a fresh SubagentInfo, whose claim also succeeded: two subagent_done, two completions to the parent, and the member counted twice. Each row whose entry the batch popped is now filed in _batched_stops until the batch reports it, and cancel_impl joins that answer instead of cancelling the row itself. A batch that never answers (the store closed under it) resolves every join to False, so a joined cancel cannot hang. A parent-end teardown or a stage Cancel whose cancel was queued before the batch lands it first the same way, so the queued-stop claim is also taken per row: _report_queued_stop reports nothing for a row that holds its finalized queued-stop record, and the batch re-posts nothing for it but still counts it, so Stop all's queued count does not depend on which stop reported the row first. A row whose batched cancel raised gets its window entry back, at the tail where a refill would put it, as on main, where the failed unqueue never popped it. cancel_all's drain tests membership against a set. A claim could also register between the moment Stop all posted its batch and the moment the answer came back. The pump's post-claim re-read had been queued ahead of the cancel job, so it answered "still current", and no queued-stop record existed yet. The pump registered the run, the cancel then ended the run's admitted row under it, and Stop all counted the one agent twice: (1, 1). Main, with its synchronous cancel, gets (0, 1). Every id of the batch, store-only ones included, is now filed in _batched_stops. A claimer that finds its row there waits for the batch's answer and re-reads behind the cancel, so it refuses the start. The batch also no longer counts or reports a store-only row that registered as a run after the post. The running sweep reaps that run and counts it. A parent-end teardown that ran while a batch was on the writer thread let the batch's reports inject into the ended conversation: its snapshot reads only _queue and _agents, and once the cancels land its store sweep cannot name the rows either. Each batched id is now filed with its parent in _batched_stop_parents, and the snapshot puts that parent's batched ids in the delivery gate without returning them for cancel. Specs: subagent.md (cancel_for_parent: the join, the per-row claim and count, the store closing under a batch, the restored entry, the claim that joins the batch, the teardown gate; snapshot_teardown_children) and taskq.md (the claimer's re-read).
f4e92f0 to
90ee1fd
Compare
buluoray
left a comment
There was a problem hiding this comment.
Verdict: APPROVE
Reviewed head: 90ee1fdb342a2ed8ab83b6a3176663200fe4a259 (two commits on base 595ad420c3, which already contains #16390 ad9fc4a19e). Re-review after my REQUEST CHANGES at 9d957c0210. CI at this head: 83 success, 12 skipped, 0 failure. Auto-merge is ON, and no one else has reviewed yet (my CHANGES_REQUESTED at 9d957c0210 is the only review), so this approval merges it.
What I ran, all in the PR worktree (kiro_crew.__file__ resolved to the worktree's src/):
- My original repro as a scratch test file, deleted afterwards. It uses the PR's fixtures:
_manager(pump_off_loop=True, max_concurrent=1), a parked slot holder, 3 queued rows for_PARENT,_contend_row_cancels,cancel_for_parentuntilentered,mgr.cancel(rows[-1].id), release, and an assertion that thesubagent_doneids are unique. test_queue_depth_reconcile.py,test_taskq_admission_integration.pyandtest_overload_integration_glue.pyat head: 296 passed.- Mutations of every new guard, each followed by a restore.
git status --shortis clean at the end.
Blocking
None. The previous blocking finding is closed.
The previous finding: a single cancel() of a queued row, made while Stop all's batched cancel job is on the writer thread, reported that row twice. It is now closed by two independent layers:
- Join.
cancellation.py:1495:cancel_implfinds the row in_batched_stopsand awaits the batch's per-row answer instead of cancelling and reporting the row itself. The futures are filed at:1352, before the post's first await, and every one of them is resolved in_apply'sfinally(:1356-1367). - Per-row claim.
_queued_stop_reported(:464) is checked at:407in_report_queued_stop_impland at:1434in_report_stopped_rows. A row that already holds its finalized queued-stop record is not reported again and not re-posted, but it is still counted.
Mutation results:
| Mutation | Result |
|---|---|
Remove both layers (join, :407, :1434) |
My repro reddens with the exact original defect: subagent_done ids [a, b, c, a], len 4 vs 3. |
| Remove the join only | No duplicate, because layer 2 dedupes. The pinning test test_queue_depth_reconcile.py:1000::test_a_single_cancel_during_stop_all_joins_the_batch_and_reports_once still reddens, on assert True not in on_loop (the single cancel ran its store write on the loop). test_a_single_cancel_joined_to_a_batch_that_never_answers_returns also reddens. |
Remove the per-row claim only (:407, :1434) |
test_a_teardown_cancel_queued_before_stop_all_reports_its_row_once[teardown] and [batch] redden. |
#16390's guards survived the rebase. I checked with git range-diff a45b3edcaa..9d957c0210 595ad420c3..d851d34456 and by reading the head:
- The guard "never install a synthetic record over a registered run" is still in
_report_queued_stop_impl(:431). It is refactored into the helper_registered_run, with the same predicate (record is not None and not record.queued). - #16390's skip of ids registered during the store read is ported to the batched path as a pre-post filter at
:1329. - There is also a post-answer check,
entry is None and self._registered_run(agent_id), in_report_stopped_rows.
Non-blocking follow-ups
Status of the four earlier follow-ups:
- Window entry lost when a batched cancel raised: addressed.
:1428puts the entry back at the tail. Neutering it reddenstest_a_row_that_could_not_be_unqueued_fails_the_stop_and_reaps_nothing[writer-thread]and[on-loop]. - Quadratic
not in pending_reports: addressed.:1688usesseen = set(pending_reports). I did not mutation-test this one; it is a performance change and no test pins it. - Store closing mid-batch: addressed.
_apply'sfinallyresolves every join to False. Neutering it reddenstest_a_single_cancel_joined_to_a_batch_that_never_answers_returns. - Unbounded wait (N × busy timeout): not addressed, accepted on purpose. The author's disposition: a joined cancel waits exactly as long as the Stop all it joined, and that wait is off the loop. A separate timeout would bring back the guess that causes the double report. I agree it adds no new hang class, but the wait is still unbounded if the writer thread stalls. If it ever matters, a request-level deadline on Stop all is the place to bound it.
New mechanisms this push added, each mutation-checked:
- Claim waits on the batch (
pump.py:577). Neutering it reddenstest_taskq_admission_integration.py::test_a_claim_resumed_between_stop_alls_post_and_answer_refuses_the_start[claim_joins]. - Post-answer registered-run skip. Neutering it reddens
[start_unjoined]of the same test. - Teardown gate for batched ids (
cancellation.py:576). Neutering it reddenstest_a_parent_end_during_stop_all_gates_the_batchs_reports.
Fail-open vs fail-closed:
- Join that raises: it re-raises the batch's exception. The row stays waiting and its entry is restored, so this fails closed.
- Batch that never answers: joins resolve to False and nothing is reported, so this fails closed.
- Claim wait: it re-reads behind the cancel and refuses the start, so this fails closed.
- Residual exposure: the design keeps several side tables, each tied to a specific reader. If a future reader of an unstarted row's id misses
_batched_stops, the worst outcome is a double report, not starting a row the user stopped. That is the better way for it to fail. The design bot's suggested ordering table insubagent.md(fence → what it orders against → pinning test) would help keep that true.
Bot dispositions
- GPT 6.1: no findings. Agree.
- Opus 5.5: no findings. Agree.
- Design: PASS. Watch 1 (a stuck writer thread leaves joins waiting forever): the same as the unbounded-wait follow-up above. Accepted, not gating. Watch 2 (a combined fence ordering table): a good non-blocking docs follow-up.
- First-principles: CONCERNS ("use teardown's await-then-unqueue shape and delete the six mechanisms"). Advisory and not gating. Awaiting each row's cancel before unqueueing would leave the window entries visible to the pump and stagger timers while the cancels are on the writer thread. The pre-PR spec and tests pin "queue removal happens before the first suspending await, so a scheduled drain cannot start work after the stop request". The PR keeps that invariant and moves the store write off the loop, which is why the extra coordination exists. Teardown can use the other shape because it does not cancel admitted rows (
allow_admitted=False). The simplification is worth exploring separately, but it is a design preference and not a defect here. - Human override on GPT at
9d957c0210(30 s straggler drain): stale for this head. GPT is clean at90ee1fdb34.
Benefit: Stop all on a parent with many queued subagents no longer freezes the gateway on per-row SQLite cancels or logs a warning for every row. A row stopped twice, or one whose claim races the stop, is still reported exactly once.
Problem / Motivation
Goal: Stop all on a parent with many queued subagents no longer freezes the gateway, and a row whose stop already wrote its final state gets no second terminal write.
Stop all ran each queued row's store cancel as a synchronous SQLite call on the event loop, one row at a time. When the task store was contended, every one of those calls held the loop for the store's busy timeout. One measured Stop all froze the whole gateway (chat, heartbeat, every other session) for 10 s. Each stopped row's queued-stop report then wrote a second
finishto the store. That write was always refused, because the cancel had already made the rowcancelled, and it still cost a writer job and logged a "did not commit" WARNING for every row.Why it matters
Stop all is the button people press when the queue has grown large, and that is also when the store is most likely to be contended. A loop freeze there stalls every conversation on the gateway, and a long enough freeze trips the watchdog. The per-row WARNINGs made a normal stop look like a failure in the logs.
Not a goal
cancelof one queued row off the loop. It stays in the documented on-loop class (taskq/store.py, taskq.md). It gains the skipped settle, and it joins a Stop all batch that already holds its row (see_batched_stopsbelow).Backwards compatibility
Compatible: no API, config or persisted shape changes.
_report_queued_stop,_claim_finalizeandtaskq_settlegain a keyword-onlyrow_settled=False, so existing callers behave as before.Removes nothing: the skipped
finishcould only be refused (the row is alreadycancelled). The propagation it came with still runs, and the downgraded log line still fires at WARNING for every refusal that lost a state.What changed (motivation → approach → change)
Symptom: the loop freezes for the busy timeout once per row → root cause:
_stop_queuedcalled_unqueue, which calls the synchronoustaskq_cancel_queued, on the loop for every row → fix: one writer-thread job per pass.cancellation.py_stop_queued(now async). Each pass queues one writer-thread job for every row's cancel (taskq_post_cancel_queued→taskq_cancel_queued_each, with per-row answers and per-row errors). It then drops the window entries (_take_window_entry). Both steps happen before the pass first suspends, so a stagger timer finds nothing to start, and a refill or claim queued after the job lands behind the cancels. The job goes first so that a post that raises leaves the window untouched. A tracked task applies the answers and the request awaits it throughasyncio.shield, so a request cancelled mid-batch still reports every row the job cancelled._report_stopped_rowskeeps the old failure rules: a row whose cancel raised is still waiting and fails the pass before the running sweep; a row whose report raised still counts.cancel_for_parentholds the parent in_stopping_parentsacross both passes, and_refill_applywindows none of that parent's rows while it is held. Without this, a refill fetch queued on the writer thread BEFORE the stop runs ahead of the cancels and reads the parent's rows as waiting. Rows the stop was cancelling came back as window entries for cancelled rows. Store-only rows got windowed, and the stop's pending read, which skips windowed rows, left themqueuedafter Stop all reported done. A review probe showed that interleaving stopping 2 of 4 rows; the base stopped all 4. This fence follows the existing_boundary_cancellation_pendingfilter on the same line._unqueue_implis split into_take_window_entryand_repost_unlanded_cancel. A single cancel behaves as before.taskq_settle(info, row_settled=True)skips the redundantfinishand keeps the propagation to a waiting parent. Every queued-stop path whose own cancel landed passes it: Stop all, the stage Cancel (_apply_boundary_cancelled_rows, only for its confirmed cancels), the parent-end teardown, and a singlecancel.cancel_implnow runs the store phase itself and hands the result to_unqueue(store_cancelled=True), which is the same behaviour, so it knows whether the cancel landed. A cancel that did not land keeps the settle and is re-posted.taskq_report_refused_settlelogs at DEBUG when the row already holds the state the write asked for, and keeps the WARNING otherwise. The case that still reaches it is a queued stop whose cancel did not land: the re-posted cancel runs ahead of the settle'sfinish, so thefinishis refused overcancelled, and_repost_unlanded_cancelhas already logged a WARNING for it.cancel_all's report drain re-reads_report_tasksafter each wait, inside its one_REPORT_DRAIN_TIMEOUTbudget. A Stop all's applier spawns each row's queued-stop report after the drain's first snapshot, so a drain that read the set only once returned while those reports were still pending.pump_off_loop=False) the batch also runs inline, which is_post_store_write's ordering rule._stop_queuedleaves out of its job any id that is a registered run by the time the pass posts (_registered_run, an_agentsrecord that is notqueued). The pending read is an await, and the pump can claim and register a row meanwhile. The batch therefore never cancels a live run's row, and the running sweep reaps it. fix(subagent): never start a row stop all cancelled after its claim #16390's_report_queued_stop_implguard (never lay a queued-stop record over a registered run) is kept for a run that registers between the post and the applier._batched_stops, andclaim_and_startwaits for that answer (shielded) and re-reads behind the cancel, so it refuses the start, as on main._report_stopped_rowsalso neither counts nor reports a store-only row that is a registered run by the time the answer comes back, so a start that did not join is reaped and counted once by the running sweep._stopping_parentsschedules one pump pass (call_soon(_drain_queue), after the running sweep has taken its ids). A pass that ran under the fence windowed nothing of the parent's, so a row the parent spawned after the passes read their ids could wait on disk beside a free slot with no pass due to bring it in._batched_stopsuntil the batch reports it, andcancel_impljoins that answer (shielded) instead of cancelling the row on the loop and reporting it a second time. The join is bounded by the batch:_apply'sfinallyresolves every join to False when the job never answers (the store closed under it). A writer thread that never finishes a job at all is not bounded separately. In that state every store write on the gateway is stuck, and the joined cancel waits exactly as long as the Stop all request itself, which is accepted._queued_stop_reported). A parent-end teardown or a stage Cancel whose cancel was queued before the batch can land first, so_report_queued_stopreports nothing for a row that already holds its finalized queued-stop record, and the batch re-posts nothing for it. The batch still counts that row, because it took it from the window, so Stop all's queued count does not depend on which stop reported the row first.2aef22a562): a reset while the batch is on the writer thread took a snapshot that reads only_queueand_agents, and a store sweep that runs after the cancels land, so neither named the batch's rows, and the batch's reports injected into the ended conversation. Each batched id is now filed with its parent in_batched_stop_parents, andsnapshot_teardown_childrenputs that parent's batched ids in the delivery gate (_teardown_cancelled_ids). It does not return them for cancellation, because the batch owns their cancel.docs/system-specs/modules/subagent.md(cancel_for_parent,snapshot_teardown_children, the Cancel bullet,cancel_all) anddocs/system-specs/modules/taskq.md(the on-loop classes, the refused-settle paragraph, the cancel callers table).Implements mainline-plan U8 (ghost-queue PR 2). Depends on #16191 (merged), and is based on main.
Tests
All in
test/test_queue_depth_reconcile.py. Each one was checked with a mutation that removes the behaviour it pins, and failed under it:test_shutdown_during_stop_all_drains_the_reports_its_applier_spawns: shutdown starts while the cancel job is held on the writer thread, and the job is released only once the report drain is waiting on the applier alone. Whencancel_allreturns, no report task may be pending and every row has itssubagent_done. It fails on a drain that takes a single snapshot.test_stop_all_over_store_only_rows_never_blocks_the_loop: every off-loop cancel waits until the loop releases it, so the stop can only finish if the loop kept turning. A cancel taken on the loop is recorded and does not wait, so there is no wall-clock race. The strict on-loop guard is armed andloop_thread_callsstays flat. The stop ends with Nsubagent_done,{queued: 0}, and 0 rows waiting.test_stop_all_queues_its_cancels_and_drops_its_entries_before_it_suspends(both pump modes): after one loop turn the job is posted and the window holds none of the parent's rows. It is caught by an await placed before the post, and by pop → await → post.test_a_refill_queued_before_stop_all_windows_none_of_the_parents_rows: the review's interleaving, with a refill fetch held on the writer thread when Stop all starts. All 4 rows endcancelled, nothing is left in the window, the last frame is{queued: 0}, and_stopping_parentsends empty. It fails without the_refill_applyfence.test_a_stop_all_whose_post_raises_leaves_the_window_whole(both modes): it fails if the entries are dropped before the post.test_stop_all_reposts_a_row_cancel_that_did_not_land(both modes): the first store cancel raisesTaskStoreUnavailable. The row endscancelledafter two attempts and its settle is not skipped. It fails without the re-post, and fails whenrow_settledis forced to True.test_a_stop_all_cancelled_mid_batch_still_reports_every_row_it_cancelled: it fails without the shield.test_a_queued_stop_skips_the_settle_its_landed_cancel_already_wrote(Stop all, stage Cancel, parent end and single cancel, each in both modes): nofinish, no "did not commit" WARNING, the propagation runs, and the rows endcancelled.test_a_refused_settle_is_a_warning_only_when_the_row_holds_another_state.test_a_row_that_could_not_be_unqueued_fails_the_stop_and_reaps_nothingnow injects its failure at the row's store cancel, and asserts that the row's window entry is put back (follow-up commit).test_a_single_cancel_during_stop_all_joins_the_batch_and_reports_once(follow-up commit, for the human review's blocking item): a singlecancelof a row whose entry Stop all's batch already popped waits on the batch's answer (_batched_stops). It no longer cancels and reports the row a second time. The test fails on the first commit alone.test_a_single_cancel_joined_to_a_batch_that_never_answers_returns: if the store closes under the batch, the joined cancel returns False and does not hang.test_a_teardown_cancel_queued_before_stop_all_reports_its_row_once(both orders): onesubagent_done, and Stop all returns(0, 2)whichever stop reported the contested row. Without the count in the skip branch, the[teardown]order returns(0, 1).test_a_parent_end_during_stop_all_gates_the_batchs_reports(follow-up commit): Stop all's job is held on the writer thread, and the parent-end snapshot and cancel run before it is released. The snapshot returns(), Stop all returns(0, 3), every row gets itssubagent_doneand endscancelled, and_on_doneis never called for any of them. Without the snapshot's gate, all three report into the ended parent.test_a_row_spawned_after_stop_alls_read_starts_once_the_stop_ends: Stop all is held after its pending read, the parent spawns a row (store-only, FIFO), and another parent's run is stopped so its pump pass runs entirely under the fence. Once the stop ends, the row starts. Without the pass that the fence's release schedules, it never starts.test_taskq_admission_integration.py::test_a_row_registered_during_stop_alls_read_is_reaped_not_replaced(from fix(subagent): never start a row stop all cancelled after its claim #16390) now runs on the batched path and also asserts that the pass's cancel job never names the registered run's row. Without the_registered_runfilter, the stop returns(1, 1).test_taskq_admission_integration.py::test_a_claim_resumed_between_stop_alls_post_and_answer_refuses_the_start(follow-up commit): barriers hold the claimer's re-read answer until the batch is posted and the batch's answer until the claimer has looked up its join or registered.[claim_joins]gets(0, 1)and the row never starts;[start_unjoined](the join disabled) gets(1, 0). On the previous head both return(1, 1)._report_queued_stop(intest_taskq_admission_integration.pyandtest_overload_integration_glue.py) now accept the new keyword.Gate for the teardown fix (
657193ee30): black (touched files),check_black_formatting.py,check_subprocess_encoding.py, isort, flake8, mypy,docs-lint.sh, and the three directly related files (298 passed). The firstlocal-gate.pyshard of the related set passed 38,447 tests before the run hit the local time cap.Gate, on the rebase onto 5bb27bd: black (touched files),
check_black_formatting.py,check_subprocess_encoding.py, isort, flake8, mypy on the changed module,docs-lint.sh,check_harness_parity.py, andscripts/local-gate.py. The local gate ran before the last edit (the pass the fence's release schedules, and its test). Its backend related set ran 52,453 passed and 1 failed:test_onboarding_import_refactor_contract.py::TestFacadeSurface::test_a_pre_split_public_name_is_still_bound_here[Lesson], a class-identity check onkiro_crew.learn.Lessonthat this diff does not touch. That file passes on its own (387 passed), so the failure is an ordering effect of another test in the same worker. After the last edits, the directly related files passed again:test_queue_depth_reconcile.py,test_taskq_admission_integration.py,test_overload_integration_glue.py,test_subagent.py,test_session.py,test_taskq_store.py,test_subagent_scale.pyandtest_handlers_messaging_coverage.py(1607 tests). This diff touches nothing underwebsite/.Manual verification
N/A. The change is in loop and writer-thread ordering, and the tests drive the real manager and the real SQLite store in both pump modes, with the interleavings forced by events rather than sleeps.
Related Issues
Part of #16480
Part of the ghost-queue work after #16191 (#15131). Mainline-plan U8.
Pattern harvest
Rule candidate: review-prompt
Pattern: "work moved from a synchronous on-loop call to a queued writer job must check jobs queued BEFORE it, not only after: a read already queued computes its exclusions when it runs"
Checklist