Skip to content

perf(subagent): batch bulk-stop store cancels on the writer thread - #16394

Merged
bolichen97 merged 2 commits into
mainfrom
perf/subagent-bulk-stop-writer-thread
Oct 5, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
perf/subagent-bulk-stop-writer-thread

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop all on a parent with many queued subagents no longer freezes the gateway, and a row whose stop already wrote its final state gets no second terminal write.

Stop all ran each queued row's store cancel as a synchronous SQLite call on the event loop, one row at a time. When the task store was contended, every one of those calls held the loop for the store's busy timeout. One measured Stop all froze the whole gateway (chat, heartbeat, every other session) for 10 s. Each stopped row's queued-stop report then wrote a second finish to the store. That write was always refused, because the cancel had already made the row cancelled, and it still cost a writer job and logged a "did not commit" WARNING for every row.

Why it matters

Stop all is the button people press when the queue has grown large, and that is also when the store is most likely to be contended. A loop freeze there stalls every conversation on the gateway, and a long enough freeze trips the watchdog. The per-row WARNINGs made a normal stop look like a failure in the logs.

Not a goal

  • Moving a single cancel of one queued row off the loop. It stays in the documented on-loop class (taskq/store.py, taskq.md). It gains the skipped settle, and it joins a Stop all batch that already holds its row (see _batched_stops below).
  • The Stop-all-versus-pump-claim race, where an admitted row starts after a stop. That was mainline-plan U9 (fix(subagent): never start a row stop all cancelled after its claim #16390, now merged); this PR keeps its guards on the batched path (see below).
  • Changing what Stop all stops, the depth frames it publishes, or the sandbox.

Backwards compatibility

Compatible: no API, config or persisted shape changes. _report_queued_stop, _claim_finalize and taskq_settle gain a keyword-only row_settled=False, so existing callers behave as before.

Removes nothing: the skipped finish could only be refused (the row is already cancelled). The propagation it came with still runs, and the downgraded log line still fires at WARNING for every refusal that lost a state.

What changed (motivation → approach → change)

Symptom: the loop freezes for the busy timeout once per row → root cause: _stop_queued called _unqueue, which calls the synchronous taskq_cancel_queued, on the loop for every row → fix: one writer-thread job per pass.

  • cancellation.py _stop_queued (now async). Each pass queues one writer-thread job for every row's cancel (taskq_post_cancel_queued → taskq_cancel_queued_each, with per-row answers and per-row errors). It then drops the window entries (_take_window_entry). Both steps happen before the pass first suspends, so a stagger timer finds nothing to start, and a refill or claim queued after the job lands behind the cancels. The job goes first so that a post that raises leaves the window untouched. A tracked task applies the answers and the request awaits it through asyncio.shield, so a request cancelled mid-batch still reports every row the job cancelled. _report_stopped_rows keeps the old failure rules: a row whose cancel raised is still waiting and fails the pass before the running sweep; a row whose report raised still counts.
  • cancel_for_parent holds the parent in _stopping_parents across both passes, and _refill_apply windows none of that parent's rows while it is held. Without this, a refill fetch queued on the writer thread BEFORE the stop runs ahead of the cancels and reads the parent's rows as waiting. Rows the stop was cancelling came back as window entries for cancelled rows. Store-only rows got windowed, and the stop's pending read, which skips windowed rows, left them queued after Stop all reported done. A review probe showed that interleaving stopping 2 of 4 rows; the base stopped all 4. This fence follows the existing _boundary_cancellation_pending filter on the same line.
  • _unqueue_impl is split into _take_window_entry and _repost_unlanded_cancel. A single cancel behaves as before.
  • Skipped settle: taskq_settle(info, row_settled=True) skips the redundant finish and keeps the propagation to a waiting parent. Every queued-stop path whose own cancel landed passes it: Stop all, the stage Cancel (_apply_boundary_cancelled_rows, only for its confirmed cancels), the parent-end teardown, and a single cancel. cancel_impl now runs the store phase itself and hands the result to _unqueue(store_cancelled=True), which is the same behaviour, so it knows whether the cancel landed. A cancel that did not land keeps the settle and is re-posted.
  • taskq_report_refused_settle logs at DEBUG when the row already holds the state the write asked for, and keeps the WARNING otherwise. The case that still reaches it is a queued stop whose cancel did not land: the re-posted cancel runs ahead of the settle's finish, so the finish is refused over cancelled, and _repost_unlanded_cancel has already logged a WARNING for it.
  • cancel_all's report drain re-reads _report_tasks after each wait, inside its one _REPORT_DRAIN_TIMEOUT budget. A Stop all's applier spawns each row's queued-stop report after the drain's first snapshot, so a drain that read the set only once returned while those reports were still pending.
  • With the pump inline (pump_off_loop=False) the batch also runs inline, which is _post_store_write's ordering rule.
  • Ported from fix(subagent): never start a row stop all cancelled after its claim #16390: _stop_queued leaves out of its job any id that is a registered run by the time the pass posts (_registered_run, an _agents record that is not queued). The pending read is an await, and the pump can claim and register a row meanwhile. The batch therefore never cancels a live run's row, and the running sweep reaps it. fix(subagent): never start a row stop all cancelled after its claim #16390's _report_queued_stop_impl guard (never lay a queued-stop record over a registered run) is kept for a run that registers between the post and the applier.
  • Follow-up commit, the post-claim window: the pump's post-claim re-read can be queued ahead of the batch's cancel job, answer "still current", and resume the claimer before the batch's answer installs any record. Every id of the job, store-only ones included, is now filed in _batched_stops, and claim_and_start waits for that answer (shielded) and re-reads behind the cancel, so it refuses the start, as on main. _report_stopped_rows also neither counts nor reports a store-only row that is a registered run by the time the answer comes back, so a start that did not join is reaped and counted once by the running sweep.
  • Releasing _stopping_parents schedules one pump pass (call_soon(_drain_queue), after the running sweep has taken its ids). A pass that ran under the fence windowed nothing of the parent's, so a row the parent spawned after the passes read their ids could wait on disk beside a free slot with no pass due to bring it in.
  • Follow-up commit, the human review's blocking item: each row whose entry the batch popped is filed in _batched_stops until the batch reports it, and cancel_impl joins that answer (shielded) instead of cancelling the row on the loop and reporting it a second time. The join is bounded by the batch: _apply's finally resolves every join to False when the job never answers (the store closed under it). A writer thread that never finishes a job at all is not bounded separately. In that state every store write on the gateway is stuck, and the joined cancel waits exactly as long as the Stop all request itself, which is accepted.
  • Follow-up commit: the queued-stop claim is per row (_queued_stop_reported). A parent-end teardown or a stage Cancel whose cancel was queued before the batch can land first, so _report_queued_stop reports nothing for a row that already holds its finalized queued-stop record, and the batch re-posts nothing for it. The batch still counts that row, because it took it from the window, so Stop all's queued count does not depend on which stop reported the row first.
  • Follow-up commit, a parent end during a batch (GPT review on 2aef22a562): a reset while the batch is on the writer thread took a snapshot that reads only _queue and _agents, and a store sweep that runs after the cancels land, so neither named the batch's rows, and the batch's reports injected into the ended conversation. Each batched id is now filed with its parent in _batched_stop_parents, and snapshot_teardown_children puts that parent's batched ids in the delivery gate (_teardown_cancelled_ids). It does not return them for cancellation, because the batch owns their cancel.
  • Specs updated in the same commit: docs/system-specs/modules/subagent.md (cancel_for_parent, snapshot_teardown_children, the Cancel bullet, cancel_all) and docs/system-specs/modules/taskq.md (the on-loop classes, the refused-settle paragraph, the cancel callers table).

Implements mainline-plan U8 (ghost-queue PR 2). Depends on #16191 (merged), and is based on main.

Tests

All in test/test_queue_depth_reconcile.py. Each one was checked with a mutation that removes the behaviour it pins, and failed under it:

  • test_shutdown_during_stop_all_drains_the_reports_its_applier_spawns: shutdown starts while the cancel job is held on the writer thread, and the job is released only once the report drain is waiting on the applier alone. When cancel_all returns, no report task may be pending and every row has its subagent_done. It fails on a drain that takes a single snapshot.
  • test_stop_all_over_store_only_rows_never_blocks_the_loop: every off-loop cancel waits until the loop releases it, so the stop can only finish if the loop kept turning. A cancel taken on the loop is recorded and does not wait, so there is no wall-clock race. The strict on-loop guard is armed and loop_thread_calls stays flat. The stop ends with N subagent_done, {queued: 0}, and 0 rows waiting.
  • test_stop_all_queues_its_cancels_and_drops_its_entries_before_it_suspends (both pump modes): after one loop turn the job is posted and the window holds none of the parent's rows. It is caught by an await placed before the post, and by pop → await → post.
  • test_a_refill_queued_before_stop_all_windows_none_of_the_parents_rows: the review's interleaving, with a refill fetch held on the writer thread when Stop all starts. All 4 rows end cancelled, nothing is left in the window, the last frame is {queued: 0}, and _stopping_parents ends empty. It fails without the _refill_apply fence.
  • test_a_stop_all_whose_post_raises_leaves_the_window_whole (both modes): it fails if the entries are dropped before the post.
  • test_stop_all_reposts_a_row_cancel_that_did_not_land (both modes): the first store cancel raises TaskStoreUnavailable. The row ends cancelled after two attempts and its settle is not skipped. It fails without the re-post, and fails when row_settled is forced to True.
  • test_a_stop_all_cancelled_mid_batch_still_reports_every_row_it_cancelled: it fails without the shield.
  • test_a_queued_stop_skips_the_settle_its_landed_cancel_already_wrote (Stop all, stage Cancel, parent end and single cancel, each in both modes): no finish, no "did not commit" WARNING, the propagation runs, and the rows end cancelled.
  • test_a_refused_settle_is_a_warning_only_when_the_row_holds_another_state.
  • test_a_row_that_could_not_be_unqueued_fails_the_stop_and_reaps_nothing now injects its failure at the row's store cancel, and asserts that the row's window entry is put back (follow-up commit).
  • test_a_single_cancel_during_stop_all_joins_the_batch_and_reports_once (follow-up commit, for the human review's blocking item): a single cancel of a row whose entry Stop all's batch already popped waits on the batch's answer (_batched_stops). It no longer cancels and reports the row a second time. The test fails on the first commit alone.
  • test_a_single_cancel_joined_to_a_batch_that_never_answers_returns: if the store closes under the batch, the joined cancel returns False and does not hang.
  • test_a_teardown_cancel_queued_before_stop_all_reports_its_row_once (both orders): one subagent_done, and Stop all returns (0, 2) whichever stop reported the contested row. Without the count in the skip branch, the [teardown] order returns (0, 1).
  • test_a_parent_end_during_stop_all_gates_the_batchs_reports (follow-up commit): Stop all's job is held on the writer thread, and the parent-end snapshot and cancel run before it is released. The snapshot returns (), Stop all returns (0, 3), every row gets its subagent_done and ends cancelled, and _on_done is never called for any of them. Without the snapshot's gate, all three report into the ended parent.
  • test_a_row_spawned_after_stop_alls_read_starts_once_the_stop_ends: Stop all is held after its pending read, the parent spawns a row (store-only, FIFO), and another parent's run is stopped so its pump pass runs entirely under the fence. Once the stop ends, the row starts. Without the pass that the fence's release schedules, it never starts.
  • test_taskq_admission_integration.py::test_a_row_registered_during_stop_alls_read_is_reaped_not_replaced (from fix(subagent): never start a row stop all cancelled after its claim #16390) now runs on the batched path and also asserts that the pass's cancel job never names the registered run's row. Without the _registered_run filter, the stop returns (1, 1).
  • test_taskq_admission_integration.py::test_a_claim_resumed_between_stop_alls_post_and_answer_refuses_the_start (follow-up commit): barriers hold the claimer's re-read answer until the batch is posted and the batch's answer until the claimer has looked up its join or registered. [claim_joins] gets (0, 1) and the row never starts; [start_unjoined] (the join disabled) gets (1, 0). On the previous head both return (1, 1).
  • Three test fakes of _report_queued_stop (in test_taskq_admission_integration.py and test_overload_integration_glue.py) now accept the new keyword.

Gate for the teardown fix (657193ee30): black (touched files), check_black_formatting.py, check_subprocess_encoding.py, isort, flake8, mypy, docs-lint.sh, and the three directly related files (298 passed). The first local-gate.py shard of the related set passed 38,447 tests before the run hit the local time cap.

Gate, on the rebase onto 5bb27bd: black (touched files), check_black_formatting.py, check_subprocess_encoding.py, isort, flake8, mypy on the changed module, docs-lint.sh, check_harness_parity.py, and scripts/local-gate.py. The local gate ran before the last edit (the pass the fence's release schedules, and its test). Its backend related set ran 52,453 passed and 1 failed: test_onboarding_import_refactor_contract.py::TestFacadeSurface::test_a_pre_split_public_name_is_still_bound_here[Lesson], a class-identity check on kiro_crew.learn.Lesson that this diff does not touch. That file passes on its own (387 passed), so the failure is an ordering effect of another test in the same worker. After the last edits, the directly related files passed again: test_queue_depth_reconcile.py, test_taskq_admission_integration.py, test_overload_integration_glue.py, test_subagent.py, test_session.py, test_taskq_store.py, test_subagent_scale.py and test_handlers_messaging_coverage.py (1607 tests). This diff touches nothing under website/.

Manual verification

N/A. The change is in loop and writer-thread ordering, and the tests drive the real manager and the real SQLite store in both pump modes, with the interleavings forced by events rather than sleeps.

Related Issues

Part of #16480

Part of the ghost-queue work after #16191 (#15131). Mainline-plan U8.

Pattern harvest

Rule candidate: review-prompt
Pattern: "work moved from a synchronous on-loop call to a queued writer job must check jobs queued BEFORE it, not only after: a read already queued computes its exclusions when it runs"

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 3, 2026 06:42
@bolichen97
bolichen97 requested a review from CrysisDeu October 3, 2026 06:42
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 90ee1fdb342a2ed8ab83b6a3176663200fe4a259 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 90ee1fd

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 90ee1fdb342a2ed8ab83b6a3176663200fe4a259: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 90ee1fdb342a2ed8ab83b6a3176663200fe4a259 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Dropping window entries before the batched cancel lands brings in six compensating mechanisms. The PR never weighs them against the await-then-unqueue shape that teardown already uses.

Not justified as shipped

  • Items 4–9 — oversized: they exist only because the window entries are dropped "before the pass first suspends", while the cancels are still on the writer thread. That gap needs a refill fence, the _batched_stops joins in cancel_impl and claim_and_start, _batched_stop_parents, a per-row reported check, a drain re-read loop and an extra pump pass. Teardown already uses a different shape at cancellation.py:783: it awaits taskq_cancel_queued_async and then calls _unqueue(stored=…, store_cancelled=True). That keeps the loop free and needs none of the six.
    Subtraction: write Stop all's queued pass in the teardown's await-then-unqueue shape, and delete _stopping_parents, _batched_stops/_batched_stop_of, _batched_stop_parents, the call_soon(_drain_queue) release and the drain re-read loop.
    Clears when: a test or a reproduction shows the teardown shape lets a stagger timer start a stopped row that Stop all must not start, or the six mechanisms are removed.

What this change ships

Inventory (10 items) — 4 justified

Intent: Stop all on a parent with many queued subagents should no longer freeze the gateway or log a warning for every row — a FIX (perf).

  1. Stop all's queued cancels run as one writer-thread job instead of N on-loop SQLite calls — justified
  2. A queued stop whose cancel landed skips the second, always-refused finish — justified
  3. A refused settle that already holds the requested state logs at DEBUG instead of WARNING — justified
  4. Refill windows none of a parent's rows while its Stop all runs — oversized
  5. An extra pump pass is scheduled when Stop all releases the parent — oversized
  6. A single cancel or claim of a batched row waits for the batch's answer — oversized
  7. Parent-end teardown gates the reports of rows the batch is still cancelling — oversized
  8. A row that is already reported gets no second queued-stop report — oversized
  9. Shutdown's report drain re-reads tasks the applier spawned — oversized
  10. A single queued cancel now gets the skipped settle — justified

[FIRST-PRINCIPLES-REVIEWED] 90ee1fd

Description read

[DESCRIPTION-READ] 03ae1fcef1a9083c19d9b941cbaa94abc9d2974d54e5d54003c1267bb0541e83

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 90ee1fdb342a2ed8ab83b6a3176663200fe4a259 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that a stuck writer thread can't leave claim_and_start and cancel_impl waiting forever on a _batched_stops future. Only the store-closed path is bounded.

Watch

  • Three follow-up commits each added a fence for a race that review found: _stopping_parents, _batched_stops joins, and the per-row _queued_stop_reported claim. A future change can break their combined behaviour without any single test noticing.
    Clears when: subagent.md has one ordering table that lists each fence, what it orders against, and the test that pins it.

[DESIGN-REVIEWED] 90ee1fd

Description read

[DESCRIPTION-READ] 4a90f540210f69e62594406cf690c192b9fcf6ae985bc0d164bcefac5978f749

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 90ee1fdb342a2ed8ab83b6a3176663200fe4a259 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 90ee1fd

Verdict parsed from the review's SHA-scoped output markers for commit 90ee1fdb342a2ed8ab83b6a3176663200fe4a259.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 90ee1fdb342a2ed8ab83b6a3176663200fe4a259: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 3, 2026
@bolichen97
bolichen97 force-pushed the perf/subagent-bulk-stop-writer-thread branch from 5bc4c0c to d9f7e59 Compare October 3, 2026 16:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

GPT 6.1 F1 (cancellation.py:1137, "Preserve bulk-stop reports across shutdown") — fixed in part, rebutted in part. Head 9d957c0 (rebased again over #15869; the code these address is unchanged).

The finding makes two claims, and I handled them differently.

1. "If application finishes during the drain, its newly spawned reporters also escape the drain's snapshot." Fixed. This was a real defect. cancel_all took one snapshot of _report_tasks, and while the job was pending that snapshot held only the applier. When the applier finished, asyncio.wait returned, and the queued-stop reports it had just spawned were neither waited for nor cancelled before the loop closed. The drain now re-reads _report_tasks after each wait and waits on any task that joined, all inside the one _REPORT_DRAIN_TIMEOUT budget. A task still pending at the deadline counts as a straggler like the rest (cancellation.py, cancel_all_impl).
Pinned by test_queue_depth_reconcile.py::test_shutdown_during_stop_all_drains_the_reports_its_applier_spawns. That test starts shutdown while the cancel job is held on the writer thread, releases the job only once the report drain is waiting on the applier alone, and then asserts that no report task is pending and that every row's subagent_done fired when cancel_all returned. It fails on the previous head: one report task is still pending, assert [<Task pending ...>] == []. Spec: subagent.md § cancel_all.

2. "The applier is cancelled at the 30 s drain deadline, so rows cancelled in the store lose their reports." Not widened by this PR, so no code change. To reach that state, shutdown has to begin during a Stop all whose writer job outlives the 30 s drain. Two things make that a non-regression:

  • The process is SIGKILLed before that deadline. gateway_shutdown_budget.TOTAL_SHUTDOWN_BUDGET_SECS is 20 s (GRACEFUL_SHUTDOWN_SECS 10 + SIGNAL_MARGIN_SECS 10), which is shared by systemd and launchd as the SIGTERM-to-SIGKILL deadline. A job that runs past 30 s is therefore killed mid-run, and the applier is never cancelled by cancel_all.
  • The base loses the same reports in the same scenario. On main, Stop all ran each row's synchronous taskq_cancel_queued on the loop. Under the same contention (rows x busy timeout) that blocks shutdown itself. Once that passes the 20 s budget, the process is killed partway through the row loop: the rows already cancelled hold cancelled, and their reports, which were spawned but never ran, are lost. The PR moves the same work onto the writer thread, and the drain now waits for it and for every report it spawns within the same budget. A queued-stop report that misses the drain has no orphan-recovery path on main either: there is no folder or tombstone for a row that never started, so clear_tombstone is a no-op for it.

Making a cancelled row's report survive a SIGKILL would need a persisted pending-report record. That is new durable state for a pre-existing shutdown residual, which is out of scope for U8.

@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

First-principles CONCERNS, item 5 (the DEBUG downgrade of a same-state refused settle): kept, with the path it serves now named and pinned. Head 9d957c0 (rebased again over #15869; the code these address is unchanged).

The review's condition for clearing was: "someone names a path that still settles a row whose cancel already landed". Here is one: a queued stop whose own cancel did not land.
_report_stopped_rows (and _unqueue_impl) calls _repost_unlanded_cancel, which posts taskq_cancel_queued to the writer thread and logs a WARNING ("no store row was cancelled — re-posting the cancel"). Then _report_queued_stop(row, row_settled=False) posts the settle's finish(cancelled) behind it. The writer is FIFO, so once the store answers again the re-posted cancel writes cancelled first. The finish is then refused over the exact state it asked for. Without the branch, that refusal is a second WARNING ("did not commit; the row is cancelled") for a row that lost nothing.

The review was right that the comment named a case this PR had removed (a single cancel, which now passes row_settled). I fixed that:

  • The taskq_report_refused_settle comment, the docstring of test_a_refused_settle_is_a_warning_only_when_the_row_holds_another_state and the sentence in taskq.md now name the re-post path.
  • test_queue_depth_reconcile.py::test_stop_all_reposts_a_row_cancel_that_did_not_land (both pump modes) now also asserts that the run logs no "did not commit" WARNING. With the branch replaced by a plain _glue_logger.warning, it fails in both modes, which I checked by mutation.

A refusal over any other state still logs at WARNING, so the audibility rule in taskq.md holds for every refusal that lost an outcome.

@bolichen97
bolichen97 force-pushed the perf/subagent-bulk-stop-writer-thread branch from d9f7e59 to 9d957c0 Compare October 3, 2026 17:05
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

GPT 6.1 F1 on 9d957c0 (cancellation.py:1131, "Shutdown permanently drops committed bulk-stop completion reports"): rebutted. Not reachable as described, and not widened relative to main.

The finding's mechanism is: "remaining cancels exceed the 30-second drain budget -> cancel_all_impl cancels this ownerless applier". A gateway shutdown never reaches that branch:

  1. The gateway cuts shutdown at 10 s, before the 30 s straggler cancel can run. slack/gateway.py:13119 runs await asyncio.wait_for(self._shutdown(), timeout=GRACEFUL_SHUTDOWN_SECS), which is 10 s (gateway_shutdown_budget.py). It then logs "Graceful shutdown timed out — force exiting" and goes on to os._exit. cancel_all() is one of _shutdown()'s cleanup tasks (gateway.py:10621), so its _REPORT_DRAIN_TIMEOUT = 30 drain is itself cancelled at ≤ 10 s. The straggler-cancel block the finding cites (cancellation.py ~1405-1450) is never reached from the gateway, and the supervisor's SIGKILL comes at 20 s (TOTAL_SHUTDOWN_BUDGET_SECS) in any case.
  2. What the 10 s cut-off loses is the same on main. Whatever is still pending at the cut-off dies with the loop. That includes every terminal report of any kind, not only this applier. On main, Stop all ran the same per-row taskq_cancel_queued synchronously on the loop, under the same contention (rows x the 2 s busy timeout). That froze the loop, the SIGTERM handler with it, and the 20 s SIGKILL kept counting. So a Stop all long enough to outlast shutdown on main was killed partway through its row loop: the rows it had cancelled stay cancelled, and the reports it spawned never ran. Nothing on main replays those either. A never-started row has no folder or tombstone, so clear_tombstone re-admits nothing for it. This PR stops freezing the loop, and with the drain fix on this head (cancel_all re-reads _report_tasks, pinned by test_shutdown_during_stop_all_drains_the_reports_its_applier_spawns) every report the applier spawns before the cut-off is drained in the same budget as every other report.
  3. The proposed fix is new durable state for a residual that predates this PR. "Persist recoverable pending-report state with each committed cancellation and replay on startup" would add a persisted outbox and a boot-time replay for queued-stop reports. That is a new taskq schema surface, out of scope for U8 (move bulk Stop all's store cancels off the loop). The residual is real, but it belongs to the pre-existing class "a terminal report not delivered before the shutdown cut-off". It is not a property of this change.

A narrower follow-up remains possible without new durable state: report each row the moment its cancel commits, through a call_soon_threadsafe callback from taskq_cancel_queued_each, rather than once the whole job answers. That would shrink the window to one row in flight. I am leaving it out of this PR because it changes the per-pass failure semantics this PR pins, and nothing here makes the residual worse than main.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 9d957c0: the 30 s straggler cancel is unreachable from the gateway, which cuts shutdown at 10 s and force-exits. Reports still pending at that cut-off are lost on main too (and there Stop all froze the loop). This PR drains every report its applier spawns within the same budget. A durable outbox for queued-stop reports is new scope and goes to a follow-up.

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Human judgment recorded

@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 9d957c0210f3c25bdebef30c8d9f322409bf053e.

the 30 s straggler cancel is unreachable from the gateway, which cuts shutdown at 10 s and force-exits. Reports still pending at that cut-off are lost on main too (and there Stop all froze the loop). This PR drains every report its applier spawns within the same budget. A durable outbox for queued-stop reports is new scope and goes to a follow-up.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 3, 2026 20:44

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES

Reviewed head 9d957c0210f3c25bdebef30c8d9f322409bf053e. CI on that SHA: 83 success, 12 skipped, 0 failure. Auto-merge is off and no human review has been posted. I did not run the full suite. I ran only test/test_queue_depth_reconcile.py, under the mutations listed below, plus one probe.

Blocking

A single cancel() of a windowed row while Stop all's batched job is on the writer thread produces two subagent_done reports for that row (cancellation.py:1121-1133, _stop_queued, together with _report_stopped_rows at :1160-1170).

_stop_queued now pops the window entries and then suspends on the writer-thread job. While it is suspended, cancel_impl (:1196-1209) can run for one of those rows. The row has no _agents record yet, so cancel_impl calls the synchronous taskq_cancel_queued on the loop. That call lands, because the batch has not reached the row yet. _unqueue finds no window entry and returns stored, and _report_queued_stop(..., row_settled=True) reports the row. When the batch then reaches the row, its cancel returns None. _report_stopped_rows sees entry is not None and stored is None, so it calls _repost_unlanded_cancel (a spurious WARNING plus an extra writer job) and then _report_queued_stop(entry) a second time. _report_queued_stop_impl creates a fresh SubagentInfo on every call, and _claim_finalize_impl keys the claim on that object's _finalized. So the second claim also succeeds: you get a second subagent_done and a second completion injected into the parent, and batch accounting counts that member twice.

This is fail-open: it emits a second terminal response. It is new in this PR. On main, _stop_queued contains no await, so no other stop can run between a row's unqueue and its report. It is reachable in the production default (pump_off_loop = True, admission/__init__.py:73). The inline mode does not hit it, because there the batch runs synchronously.

Repro: I added a test to the PR's own fixtures and confirmed it fails, then deleted it.

mgr = await _manager(monkeypatch, pump_off_loop=True, max_concurrent=1)
with patch.object(SubagentManager, "_run", new=_park):
    mgr.spawn("holds the slot", parent_session_key="dash:elsewhere")
    rows = [mgr.spawn(f"t{i}", parent_session_key=_PARENT) for i in range(3)]
    await _settle(mgr)
events = _record(mgr)
entered, release, _ = _contend_row_cancels(monkeypatch)
stop = asyncio.ensure_future(mgr.cancel_for_parent(_PARENT))
await _until(entered.is_set, "batch on writer thread")
await mgr.cancel(rows[-1].id)          # user presses Stop on one queued card
release.set(); await stop; await _settle(mgr)
# subagent_done ids: [c521…, 8174…, 902c…, c521…]  <- rows[-1] reported twice

Suggested fix: before reporting a row whose batched cancel did not land, check whether another stop has already reported it. For example, skip the row in _report_stopped_rows (and skip _repost_unlanded_cancel for it) when _agents.get(id) is a queued record that is already _finalized. Alternatively, make _report_queued_stop_impl refuse when an existing queued record for that id is finalized. Add the scenario above as a pinning test. #16390's new guard in _report_queued_stop_impl does not cover this case: it only skips records that are not queued.

Non-blocking follow-ups

  1. A row whose batched cancel raised loses its window entry. _stop_queued pops every entry before it knows any outcome (:1123), and _report_stopped_rows never restores the entry of a row whose cancel raised. On main, _unqueue raised before the pop, so that row stayed in the window. The behavior is fail-closed: the store row is still queued, and the next refill windows it again once _stopping_parents is released. test_a_row_that_could_not_be_unqueued_fails_the_stop_and_reaps_nothing asserts the store state but not the window state. Either restore the entry or say in the docstring that the entry is dropped on purpose.
  2. No bound on the request's wait. await asyncio.shield(applying) waits on a single job that runs N cancels serially, each of which can take up to the store's busy timeout. The loop stays free, but the Stop all request and the writer thread can both be held for N × busy timeout, and every other session's posted settles and claims queue behind that job. The design bot raised the same point. Consider chunking the job, or a per-pass deadline that leaves the remaining rows to the re-post path.
  3. Drain membership test is quadratic. cancel_all's drain checks t not in pending_reports against a list (cancellation.py:~1391). Use a set.
  4. Store close mid-batch. TaskStore.close() shuts the executor down with cancel_futures=True, so a queued batch future is cancelled, _apply never reports, and the window entries are already gone. This only happens at shutdown, and the rows stay queued on disk. Worth one line in the spec's failure list.

Bot dispositions

  • GPT F1 (shutdown drops reports), human override. I checked the override reasoning: GRACEFUL_SHUTDOWN_SECS = 10 (gateway_shutdown_budget.py:6), and gateway.py:13119 wraps _shutdown() in that budget, so the 30 s straggler cancel cannot be reached from the gateway. The override holds.
  • First-principles PASS with a check on the _stopping_parents scope. The fence is a counter that the finally at cancellation.py:781-785 releases on every exit, including failure and cancellation, so a failed pass does not keep its rows out of the window.
  • Design PASS with a check that one N-row job could starve the writer. That concern is real and does not gate this PR (follow-up 2).

Overlap with #16390 and #16395 (merge order)

  • #16394 × #16390: real conflict. git merge-tree reports content conflicts in src/kiro_crew/subagent_manager/cancellation.py (_stop_queued, which #16390 patches in its old synchronous form and #16394 rewrites) and in docs/system-specs/modules/subagent.md. The conflict is semantic as well as textual. #16390's "registered after the ids were read, skip" check has to move into the new _stop_queued before taskq_post_cancel_queued, so the batch never posts a cancel for a registered run. #16390's _report_queued_stop_impl guard must also be kept, because a run can register between the post and the moment the applier runs. Whichever PR lands second has to do that port and re-run both PRs' tests. #16390's taskq_pending_ids_for_async(include_admitted=True) composes with the batch (allow_admitted defaults to True), so that part needs no change.
  • #16394 × #16395: clean merge (git merge-tree exits clean). #16395 adds an include_window keyword to taskq_pending_ids_for_async, and #16394 calls that function positionally, so the call still works. #16395 adds no new caller of _stop_queued or _report_queued_stop. Its parent-end edits sit next to #16394's row_settled change at the teardown report (cancellation.py:683) and merge cleanly. Neither order creates a dependency.
  • For reference, #16390 × #16395 conflicts in admission/taskq_bridge.py: both PRs rewrite taskq_pending_ids_for_async.
  • All three PRs merge cleanly onto origin/main (f02788522a) on their own.

What I verified

  • I read the full source diff at the head SHA, plus taskq_cancel_queued and TaskStore.post/_writer_executor/close. The writes stay on the writer thread: taskq_cancel_queued_each runs as a single store.post job on the single-worker executor. The inline path is used only with no running loop or with pump_off_loop=False, and that matches _post_store_write's rule. The connection is check_same_thread=False and guarded by the store's lock, which is the pattern main already uses. Nothing loop-bound is touched from the writer thread.
  • One row's failure does not skip the rest. taskq_cancel_queued_each catches each id separately. _report_stopped_rows reports every row that succeeded and then re-raises the first error.
  • Ordering against other stops. The post happens before the pop and before the first await, so later refills and claims queue behind the cancels. A parent-end teardown or stage cancel during the batch posts its own cancel behind the batch, finds no entry, and does not report. Only the synchronous on-loop single cancel() can interleave, and that is the blocking finding.
  • Mutations (each run on test/test_queue_depth_reconcile.py, restored afterwards, git status --short clean):
    • Removed the _refill_apply _stopping_parents skip: test_a_refill_queued_before_stop_all_windows_none_of_the_parents_rows fails.
    • Removed the per-id catch in taskq_cancel_queued_each: test_a_row_that_could_not_be_unqueued_fails_the_stop_and_reaps_nothing[writer-thread, on-loop] fails.
    • Replaced asyncio.shield(applying) with a plain await: test_a_stop_all_cancelled_mid_batch_still_reports_every_row_it_cancelled fails.
    • Made the drain take a single snapshot (stopped adding the tasks that joined): test_shutdown_during_stop_all_drains_the_reports_its_applier_spawns fails.
    • Disabled the row_settled early return in taskq_settle: all 8 test_a_queued_stop_skips_the_settle_its_landed_cancel_already_wrote[...] cases fail.
    • Moved the pop before the post: test_a_stop_all_whose_post_raises_leaves_the_window_whole[writer-thread, on-loop] fails.
    • Removed the batch's _repost_unlanded_cancel: test_stop_all_reposts_a_row_cancel_that_did_not_land[writer-thread, on-loop] fails.
    • Removed the DEBUG downgrade: test_a_refused_settle_is_a_warning_only_when_the_row_holds_another_state, plus both reposts cases, fail.
  • Not verified: the PR's gate numbers, mypy, and the frontend shards. I took those from CI and the PR body only.

@iamwhatever

Copy link
Copy Markdown
Contributor

Reviewed, not approving yet: buluoray has changes requested on the current head (9d957c0), and auto-merge is on, so my approval would ship it past that request. This batches bulk-stop store cancels on the writer thread (~800 lines in the subagent stop path). I will review once buluoray's points are resolved.

@bolichen97
bolichen97 force-pushed the perf/subagent-bulk-stop-writer-thread branch from 9d957c0 to 7916330 Compare October 4, 2026 06:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 4, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

GPT 6.1 F1 on 2aef22a (cancellation.py:1296, "In-flight batch stops escape parent-end delivery suppression"): accepted as a real defect and fixed in 657193ee30.

The mechanism is as described. _stop_queued takes each row's window entry before its first await. Until the applier reports a row, that row is in neither _queue nor _agents. snapshot_teardown_children reads only those two, so it did not put the rows in _teardown_cancelled_ids. The teardown's store sweep then runs behind the batch's job on the writer thread, finds the rows already cancelled, and does not name them either. When the batch answered, _report_queued_stop sent each row's stop into the conversation that had just been reset.

The fix is what the finding proposes:

  • _stop_queued takes the parent and files each id beside its _batched_stops join in _batched_stop_parents. The entry is removed in the same finally that clears the join.
  • snapshot_teardown_children puts that parent's batched ids in the delivery gate. It does not return them for cancellation, because the batch owns their cancel. The undelivered and approval-parked runs are handled the same way: they are gated, not cancelled.

Each row is still cancelled once and reported once, and its subagent_done still goes out. Only the injection into the ended parent is dropped.

Test: test_queue_depth_reconcile.py::test_a_parent_end_during_stop_all_gates_the_batchs_reports. Stop all's job is held on the writer thread, and the snapshot and cancel_for_teardown run before the job is released. The snapshot returns (), Stop all returns (0, 3), every row ends cancelled with one subagent_done, and _on_done is never called for any of them. With the snapshot's new gate line removed, the test fails with "a retired row reported home". Specs: subagent.md (snapshot_teardown_children and cancel_for_parent).

@bolichen97

Copy link
Copy Markdown
Contributor Author

Design CONCERNS on 2aef22a. Watch item 1 is accepted for the reader it missed, and the placeholder design is declined. Watch item 2 is answered. New head 657193ee30.

1. "Invisible in-flight state guarded only by join points." The concern was right. The GPT lane found a fourth reader that had no join: the parent-end snapshot. It is fixed in 657193ee30 (see the GPT disposition). Here is every loop reader that acts on an unstarted row's id while a batch holds it, and the test that pins each one:

reader behaviour during a pending batch pinned by
cancel_impl (single cancel) joins the batch's answer test_a_single_cancel_during_stop_all_joins_the_batch_and_reports_once, test_a_single_cancel_joined_to_a_batch_that_never_answers_returns
claim_and_start (pump claim) waits for the answer and re-reads behind the cancel test_taskq_admission_integration.py::test_a_claim_resumed_between_stop_alls_post_and_answer_refuses_the_start
snapshot_teardown_children (parent end) gates the batch's report and does not cancel the row test_a_parent_end_during_stop_all_gates_the_batchs_reports (new)
teardown / stage Cancel store cancels queued before the batch per-row queued-stop claim (_queued_stop_reported) test_a_teardown_cancel_queued_before_stop_all_reports_its_row_once (both orders)

The _queue scanners (pump grant loop, refill census, eviction, reserve, depth) should NOT see the row. Their not seeing it is what stops a stagger timer from starting a row the user stopped. The refill is fenced separately by _stopping_parents.

Why not a placeholder _agents record installed when the entry is popped: it does not remove the audit, it inverts it. Every _agents reader would then have to tell the placeholder apart from a real record:

  • The teardown snapshot's live list would select it, so the teardown would cancel() it and reach cancel_impl's live branch, which runs _force_reap on a record that has no task.
  • _registered_run would treat it as a registered run unless it carried queued=True.
  • _report_queued_stop_impl installs its own record under the same key and claims it through _claim_finalize.

So the side map plus one join per reader is the smaller change. The table above is the list of readers that the Clears-when asks for.

2. "Backwards-compatibility section cut off." That was the review's description truncation, not the description itself. In full, the section reads: "Compatible: no API, config or persisted shape changes. _report_queued_stop, _claim_finalize and taskq_settle gain a keyword-only row_settled=False, so existing callers behave as before. Removes nothing: the skipped finish could only be refused (the row is already cancelled)..." _stop_queued (now async, and now taking the parent) and the _unqueue_impl split have no callers outside this patch.

@bolichen97

Copy link
Copy Markdown
Contributor Author

@buluoray, one more push since #16394 (comment): 657193ee30. The branch still has two commits and the base is still 5bb27bda8e. The change is folded into the follow-up commit.

The GPT lane found that a parent end during Stop all's batch let the batch's stop reports inject into the conversation that had just ended. The batched rows are in neither _queue nor _agents, so the teardown snapshot did not gate them. They now are, through _batched_stop_parents. The rows are not returned for cancellation, because the batch still owns their cancel. Test: test_queue_depth_reconcile.py::test_a_parent_end_during_stop_all_gates_the_batchs_reports. Details: #16394 (comment). Nothing else changed apart from one spec sentence. It now says the double report only applies to rows whose window entry the batch popped.

@bolichen97

bolichen97 commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor Author

Design CONCERNS on 657193e (advisory). Item 1 is kept, and the list of readers is given below. Item 2 is answered. No code change this round.

1. "The new in-flight state has no single owner." I am keeping the shape and listing the readers, which is the second option in Clears-when. The table in #16394 (comment) covers every reader that acts on an unstarted row's id. The rest of the _queue / _agents readers do not need the check, for these reasons:

  • _queue scanners (the pump grant loop, _refill_apply's lane census, eviction, resume_reserve, _window_depth). Not seeing a popped row is the point of popping it: a stagger timer or a grant must not start a row the user stopped, and the depth must not count it. The only one that could put the row back, the refill, is fenced by _stopping_parents.
  • _agents scanners that iterate records (the running sweep in cancel_for_parent, the reaper and stall monitor, /api/spawn listings, orphan reconciliation). A batched row has no record. These readers act on runs that have records, and on store rows that are running or orphaned. A queued row that the batch is cancelling is neither, so ignoring it is correct.
  • Readers keyed by a row's id that would act on "no record": cancel_impl joins the batch, claim_and_start joins and re-reads, and snapshot_teardown_children gates the report. A stop that cancels through the store and lands first is covered by the per-row claim in _report_queued_stop. Each of these has its own test.

Why I am not keeping the entries in the window marked "stopping": every _queue scanner above would then need to skip that mark. That is five readers that would fail toward starting a stopped row, instead of three that fail toward a double report. A single is_batched_stop(id) accessor is a reasonable cleanup. It would not change what any reader does, so I am leaving it as a follow-up and not adding another push to a head that is waiting for human re-review.

2. "The take-away check is capped." The cap is the review's 8000-byte read of the description, not a gap in the description. The full ## Backwards compatibility section lists the log-level change. It reads: "Removes nothing: the skipped finish could only be refused (the row is already cancelled). The propagation it came with still runs, and the downgraded log line still fires at WARNING for every refusal that lost a state." The operator-visible change is the DEBUG level for a refused settle where current == state. It is dispositioned at #16394 (comment). It is pinned by test_a_refused_settle_is_a_warning_only_when_the_row_holds_another_state.

@bolichen97
bolichen97 force-pushed the perf/subagent-bulk-stop-writer-thread branch from 657193e to f4e92f0 Compare October 4, 2026 22:43
@bolichen97

Copy link
Copy Markdown
Contributor Author

@buluoray, a rebase-only push since #16394 (comment): the head is now f4e92f0d16, on origin/main at fd705fb7c7. The rebase was clean and the code is unchanged from 657193ee30.

Your blocking item (a single cancel during Stop all's batch reporting a row twice) and the four follow-ups are handled as described in #16394 (comment) and the later updates. The related tests pass on the new head (test_queue_depth_reconcile.py, test_taskq_admission_integration.py, test_overload_integration_glue.py, test_subagent.py, test_taskq_store.py, test_subagent_scale.py: 574 passed), as do black, isort, flake8, mypy on the changed modules, and docs-lint.sh.

Could you take another look when you have a moment?

@bolichen97
bolichen97 requested a review from buluoray October 4, 2026 22:44
@bolichen97

Copy link
Copy Markdown
Contributor Author

Design CONCERNS on f4e92f0, Watch span=6fe33ebfa498 ("each new race got its own side table"): rebutted, shape kept. Rebase-only head; the code is unchanged from 657193e.

The full reasoning and the reader list are in #16394 (comment) and the reader table in #16394 (comment). In short: every _queue scanner must not see a popped row (a grant or stagger timer must not start a stopped row), _agents scanners act only on rows that have records, and the three readers keyed by a row id that would act on "no record" (cancel_impl, claim_and_start, snapshot_teardown_children) each consult the batch and each have their own test. Keeping a "stopping" mark in the window would make five readers fail toward starting a stopped row. A single is_batched_stop(id) accessor is a cleanup that changes no reader's behaviour, so it is left out of this PR.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Design CONCERNS on f4e92f0, Watch span=61cecebc315b ("the compatibility check could only partly run"): fixed in the PR description.

The ## Backwards compatibility section now comes right after ## Not a goal, at about byte 1600 of the description, well inside the 8000-byte read. Its text is unchanged: no API, config or persisted shape changes; _report_queued_stop, _claim_finalize and taskq_settle gain a keyword-only row_settled=False; the only operator-visible change is DEBUG for a refused settle where current == state.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Design CONCERNS on f4e92f0, Suggestion span=a353a30629ec ("keep each popped row in a state existing readers already see"): declined.

A stopping window entry would be seen by every _queue scanner (the pump grant loop, the refill lane census, eviction, resume_reserve, _window_depth), and each would then need to skip it; a missed skip starts a row the user stopped. A terminal-pending _agents record would be seen by the reaper, stall monitor, /api/spawn listings and orphan reconciliation, which act on runs. The current design fails the other way: a reader that misses the batch can only double-report, and the three readers that could are covered and tested. See #16394 (comment).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 4, 2026
Stop all cancelled each queued row with a synchronous store call on the
event loop. Whenever the task store was contended, that held the loop
for the busy timeout once per row; one measured Stop all froze the
gateway for 10 s.

Each Stop all pass now queues ONE writer-thread job for every row's
cancel (taskq_post_cancel_queued) and drops the window entries before
it first awaits. A tracked, shielded task applies the answers, so a
request cancelled mid-batch still reports every row it cancelled. The
parent is held in _stopping_parents across both passes, and
_refill_apply windows none of its rows meanwhile, so a refill fetch
queued before the stop can neither re-window a row being cancelled nor
window a store-only row the pending read would then skip. An id the
pump registered as a run after the pass read its ids is left out of the
job, so the batch never cancels a live run's row; the running sweep
reaps it, as #16390 made the synchronous pass do. Releasing the fence
schedules one pump pass: a pass that ran under it windowed nothing of
the parent's, and a row spawned after the passes read their ids would
otherwise wait on disk beside a free slot.

A cancel that landed is the row's terminal write. Its queued-stop
report now skips the settle's finish and keeps the propagation
(row_settled). Stop all, the stage Cancel, the parent-end teardown and
a single cancel all pass it. A refused settle whose row already holds
the requested state logs at DEBUG instead of WARNING.

cancel_all's report drain re-reads _report_tasks after each wait inside
its one budget: a Stop all's applier spawns each row's report after the
drain's first snapshot, and a drain that never re-read returned with
those reports still pending as the loop closed.

Specs: subagent.md (cancel_for_parent, Cancel, cancel_all) and taskq.md
(on-loop classes, refused settle, cancel callers).
A single cancel of a queued row, made while a Stop all batch was on the
writer thread, reported that row twice. The batch had already popped the
row's window entry and the row has no _agents record, so the single
cancel took it for a store-only row: it cancelled it on the loop, landed
first and reported it. The batch's own cancel then came back empty, so
it re-posted the cancel with a WARNING and reported the row again with a
fresh SubagentInfo, whose claim also succeeded: two subagent_done, two
completions to the parent, and the member counted twice.

Each row whose entry the batch popped is now filed in _batched_stops
until the batch reports it, and cancel_impl joins that answer instead of
cancelling the row itself. A batch that never answers (the store closed
under it) resolves every join to False, so a joined cancel cannot hang.

A parent-end teardown or a stage Cancel whose cancel was queued before
the batch lands it first the same way, so the queued-stop claim is also
taken per row: _report_queued_stop reports nothing for a row that holds
its finalized queued-stop record, and the batch re-posts nothing for it
but still counts it, so Stop all's queued count does not depend on which
stop reported the row first.

A row whose batched cancel raised gets its window entry back, at the
tail where a refill would put it, as on main, where the failed unqueue
never popped it. cancel_all's drain tests membership against a set.

A claim could also register between the moment Stop all posted its batch
and the moment the answer came back. The pump's post-claim re-read had
been queued ahead of the cancel job, so it answered "still current", and
no queued-stop record existed yet. The pump registered the run, the
cancel then ended the run's admitted row under it, and Stop all counted
the one agent twice: (1, 1). Main, with its synchronous cancel, gets
(0, 1). Every id of the batch, store-only ones included, is now filed in
_batched_stops. A claimer that finds its row there waits for the batch's
answer and re-reads behind the cancel, so it refuses the start. The
batch also no longer counts or reports a store-only row that registered
as a run after the post. The running sweep reaps that run and counts it.

A parent-end teardown that ran while a batch was on the writer thread
let the batch's reports inject into the ended conversation: its snapshot
reads only _queue and _agents, and once the cancels land its store sweep
cannot name the rows either. Each batched id is now filed with its
parent in _batched_stop_parents, and the snapshot puts that parent's
batched ids in the delivery gate without returning them for cancel.

Specs: subagent.md (cancel_for_parent: the join, the per-row claim and
count, the store closing under a batch, the restored entry, the claim
that joins the batch, the teardown gate; snapshot_teardown_children)
and taskq.md (the claimer's re-read).
@bolichen97
bolichen97 force-pushed the perf/subagent-bulk-stop-writer-thread branch from f4e92f0 to 90ee1fd Compare October 5, 2026 01:55
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 5, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE

Reviewed head: 90ee1fdb342a2ed8ab83b6a3176663200fe4a259 (two commits on base 595ad420c3, which already contains #16390 ad9fc4a19e). Re-review after my REQUEST CHANGES at 9d957c0210. CI at this head: 83 success, 12 skipped, 0 failure. Auto-merge is ON, and no one else has reviewed yet (my CHANGES_REQUESTED at 9d957c0210 is the only review), so this approval merges it.

What I ran, all in the PR worktree (kiro_crew.__file__ resolved to the worktree's src/):

  • My original repro as a scratch test file, deleted afterwards. It uses the PR's fixtures: _manager(pump_off_loop=True, max_concurrent=1), a parked slot holder, 3 queued rows for _PARENT, _contend_row_cancels, cancel_for_parent until entered, mgr.cancel(rows[-1].id), release, and an assertion that the subagent_done ids are unique.
  • test_queue_depth_reconcile.py, test_taskq_admission_integration.py and test_overload_integration_glue.py at head: 296 passed.
  • Mutations of every new guard, each followed by a restore. git status --short is clean at the end.

Blocking

None. The previous blocking finding is closed.

The previous finding: a single cancel() of a queued row, made while Stop all's batched cancel job is on the writer thread, reported that row twice. It is now closed by two independent layers:

  1. Join. cancellation.py:1495: cancel_impl finds the row in _batched_stops and awaits the batch's per-row answer instead of cancelling and reporting the row itself. The futures are filed at :1352, before the post's first await, and every one of them is resolved in _apply's finally (:1356-1367).
  2. Per-row claim. _queued_stop_reported (:464) is checked at :407 in _report_queued_stop_impl and at :1434 in _report_stopped_rows. A row that already holds its finalized queued-stop record is not reported again and not re-posted, but it is still counted.

Mutation results:

Mutation Result
Remove both layers (join, :407, :1434) My repro reddens with the exact original defect: subagent_done ids [a, b, c, a], len 4 vs 3.
Remove the join only No duplicate, because layer 2 dedupes. The pinning test test_queue_depth_reconcile.py:1000::test_a_single_cancel_during_stop_all_joins_the_batch_and_reports_once still reddens, on assert True not in on_loop (the single cancel ran its store write on the loop). test_a_single_cancel_joined_to_a_batch_that_never_answers_returns also reddens.
Remove the per-row claim only (:407, :1434) test_a_teardown_cancel_queued_before_stop_all_reports_its_row_once[teardown] and [batch] redden.

#16390's guards survived the rebase. I checked with git range-diff a45b3edcaa..9d957c0210 595ad420c3..d851d34456 and by reading the head:

  • The guard "never install a synthetic record over a registered run" is still in _report_queued_stop_impl (:431). It is refactored into the helper _registered_run, with the same predicate (record is not None and not record.queued).
  • #16390's skip of ids registered during the store read is ported to the batched path as a pre-post filter at :1329.
  • There is also a post-answer check, entry is None and self._registered_run(agent_id), in _report_stopped_rows.

Non-blocking follow-ups

Status of the four earlier follow-ups:

  • Window entry lost when a batched cancel raised: addressed. :1428 puts the entry back at the tail. Neutering it reddens test_a_row_that_could_not_be_unqueued_fails_the_stop_and_reaps_nothing[writer-thread] and [on-loop].
  • Quadratic not in pending_reports: addressed. :1688 uses seen = set(pending_reports). I did not mutation-test this one; it is a performance change and no test pins it.
  • Store closing mid-batch: addressed. _apply's finally resolves every join to False. Neutering it reddens test_a_single_cancel_joined_to_a_batch_that_never_answers_returns.
  • Unbounded wait (N × busy timeout): not addressed, accepted on purpose. The author's disposition: a joined cancel waits exactly as long as the Stop all it joined, and that wait is off the loop. A separate timeout would bring back the guess that causes the double report. I agree it adds no new hang class, but the wait is still unbounded if the writer thread stalls. If it ever matters, a request-level deadline on Stop all is the place to bound it.

New mechanisms this push added, each mutation-checked:

  • Claim waits on the batch (pump.py:577). Neutering it reddens test_taskq_admission_integration.py::test_a_claim_resumed_between_stop_alls_post_and_answer_refuses_the_start[claim_joins].
  • Post-answer registered-run skip. Neutering it reddens [start_unjoined] of the same test.
  • Teardown gate for batched ids (cancellation.py:576). Neutering it reddens test_a_parent_end_during_stop_all_gates_the_batchs_reports.

Fail-open vs fail-closed:

  • Join that raises: it re-raises the batch's exception. The row stays waiting and its entry is restored, so this fails closed.
  • Batch that never answers: joins resolve to False and nothing is reported, so this fails closed.
  • Claim wait: it re-reads behind the cancel and refuses the start, so this fails closed.
  • Residual exposure: the design keeps several side tables, each tied to a specific reader. If a future reader of an unstarted row's id misses _batched_stops, the worst outcome is a double report, not starting a row the user stopped. That is the better way for it to fail. The design bot's suggested ordering table in subagent.md (fence → what it orders against → pinning test) would help keep that true.

Bot dispositions

  • GPT 6.1: no findings. Agree.
  • Opus 5.5: no findings. Agree.
  • Design: PASS. Watch 1 (a stuck writer thread leaves joins waiting forever): the same as the unbounded-wait follow-up above. Accepted, not gating. Watch 2 (a combined fence ordering table): a good non-blocking docs follow-up.
  • First-principles: CONCERNS ("use teardown's await-then-unqueue shape and delete the six mechanisms"). Advisory and not gating. Awaiting each row's cancel before unqueueing would leave the window entries visible to the pump and stagger timers while the cancels are on the writer thread. The pre-PR spec and tests pin "queue removal happens before the first suspending await, so a scheduled drain cannot start work after the stop request". The PR keeps that invariant and moves the store write off the loop, which is why the extra coordination exists. Teardown can use the other shape because it does not cancel admitted rows (allow_admitted=False). The simplification is worth exploring separately, but it is a design preference and not a defect here.
  • Human override on GPT at 9d957c0210 (30 s straggler drain): stale for this head. GPT is clean at 90ee1fdb34.

Benefit: Stop all on a parent with many queued subagents no longer freezes the gateway on per-row SQLite cancels or logs a warning for every row. A row stopped twice, or one whose claim races the stop, is still reported exactly once.

@bolichen97
bolichen97 merged commit a41aeb0 into main Oct 5, 2026
98 checks passed
@bolichen97
bolichen97 deleted the perf/subagent-bulk-stop-writer-thread branch October 5, 2026 03:39
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants