Repository navigation
fix(subagent): publish queued depth 0 after the last row is popped - #14878
Conversation
e0a10e1 to
b07956e
Compare
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify fairness.py's store reads (lines 253/284/383/400) intentionally keep [DESIGN-REVIEWED] 409aba3 Description read[DESCRIPTION-READ] cc875115f5225ab47b696be5e854d2d8710020e42b4a304dce378b4532603c41 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (7 items) — 6 justifiedIntent: FIX — clear the phantom "1 waiting to start" the progress bar keeps after a subagent wave finishes (issue #14877; test fails on base).
Stop/cancel and pending-work reads deliberately still count a popped-unclaimed row; not a change. [FIRST-PRINCIPLES-REVIEWED] 409aba3 Description read[DESCRIPTION-READ] fcde27fbe19242605e131262f728be6e48f7a7ee57eea980b79ce223fd7dc70d That sha256 names the PR description this verdict read. Recompute it with |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsAdvisory only: no blocking defect — every leak/double-dispatch candidate died (the emit's store errors are all converted to FINDING — src/kiro_crew/subagent_manager/admission/taskq_bridge.py:1282 — the refill now excludes [OPUS-REVIEWED] 409aba3 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
b07956e to
41c5383
Compare
|
Round 2, head 41c5383 (rebased onto main at 9932337). The two Windows shard reds on b07956e were: (1) the pid-reader ratchet, inherited from a base that predates #14883's repin; the rebase clears it. (2) |
41c5383 to
91d7c25
Compare
91d7c25 to
b059a22
Compare
b059a22 to
eccd969
Compare
The pump emits the parent's queued depth right after popping a row from the window. With a durable store attached that row is still QUEUED until its claim lands and is in none of the count's exclusion sets, so every depth read between pop and claim counts it once more than the window does: a 4-cap, 5-spawn wave publishes 4, 3, 2, 1 and never 0. The dashboard chip clears its waiting count only on 0, so it keeps showing one agent waiting under a finished turn. Mark a popped row as dispatching until spawn has answered for it, and exclude dispatching rows from the store-backed pending count.
eccd969 to
409aba3
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
Reviewed. The popped row is marked in _dispatching_ids before the depth emit, and the mark is cleared on every exit path (not-a-claim branch, non-registering claim, gate deferred emit, per-dispatch finally, pass-level finally), so a durable row cannot be hidden from refill. taskq_excluded_ids() is unchanged, so cancel and the counters still see the popped row. Moving picked/granting above the try also fixes a latent UnboundLocalError. Tests name a mutation each.
Non-blocking: test_last_pop_publishes_zero_with_a_store_attached polls with a 15s deadline plus fixed settle sleeps and may be slow-shard sensitive.
Four main fixes for what users on 0.8 hit with parallel subagents: - #14878: a wave's "N waiting to start" count ends at "1 waiting" under a finished turn, because a popped but unclaimed row was counted once more than the window holds. A popped row is now marked dispatching and left out of the count, so the last pop publishes 0. - #16191: the count is re-published after every terminal, Stop all and stage Cancel, coalesced per parent; a failed store read publishes nothing instead of a false 0; a resident resume entry and a deferred spawn_async row are counted correctly; the pump's idle wake no longer spins (carries Matt Cohen's #15376 commit). - #15022: a subagent inside the wait tool is not reaped as stalled before its declared duration. - #16012: a spawn the memory gate deferred exists only as a task-store row, so spawn_status answered 404, spawn_list left it out and spawn_sub_agents reported it as an error, which made the model dispatch the work again. These ids now answer as queued, and the sub-agent synthesis waits while a sibling is still queued. Backport to release/0.8.0. #14878 and #15022 apply cleanly. #16012 is applied before #16191, its order on main, because #16191 was built on it. #16012's conflicts are resolved against 0.8 code: - api_spawn_list has no persisted half and no per-row audit on 0.8, so the queued half is listed beside the live half only. - The persistence fallback of GET /api/spawn/{id} keeps 0.8's shape (a run folder reads done); the "no recorded ending is not done" change needs main's ending classifier and is left out. - The Slack gateway and chat runner keep 0.8's orchestrator mode, so the synthesis arm stays chat-mode only. The tests that read main-only APIs are adapted to those shapes, and test/conftest.py gains main's close_subagent_managers fixture, which the new tests request. The subagent spec drops the sentences that describe the main-only fallback and audit. Backports the fixes for #14877, #15131, #15011 and #15045. Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com> Co-authored-by: Will Laws <will.laws@me.com> Co-authored-by: Matt Cohen <mattcohe@amazon.com> (cherry picked from commit 0f785ef) (cherry picked from commit 4bb1d11) (cherry picked from commit 6d99710) (cherry picked from commit c759e04)
Four main fixes for what users on 0.8 hit with parallel subagents: - #14878: a wave's "N waiting to start" count ends at "1 waiting" under a finished turn, because a popped but unclaimed row was counted once more than the window holds. A popped row is now marked dispatching and left out of the count, so the last pop publishes 0. - #16191: the count is re-published after every terminal, Stop all and stage Cancel, coalesced per parent; a failed store read publishes nothing instead of a false 0; a resident resume entry and a deferred spawn_async row are counted correctly; the pump's idle wake no longer spins (carries Matt Cohen's #15376 commit). - #15022: a subagent inside the wait tool is not reaped as stalled before its declared duration. - #16012: a spawn the memory gate deferred exists only as a task-store row, so spawn_status answered 404, spawn_list left it out and spawn_sub_agents reported it as an error, which made the model dispatch the work again. These ids now answer as queued, and the sub-agent synthesis waits while a sibling is still queued. Backport to release/0.8.0. #14878 and #15022 apply cleanly. #16012 is applied before #16191, its order on main, because #16191 was built on it. #16012's conflicts are resolved against 0.8 code: - api_spawn_list has no persisted half and no per-row audit on 0.8, so the queued half is listed beside the live half only. - The persistence fallback of GET /api/spawn/{id} keeps 0.8's shape (a run folder reads done); the "no recorded ending is not done" change needs main's ending classifier and is left out. - The Slack gateway and chat runner keep 0.8's orchestrator mode, so the synthesis arm stays chat-mode only. The tests that read main-only APIs are adapted to those shapes, and test/conftest.py gains main's close_subagent_managers fixture, which the new tests request. The subagent spec drops the sentences that describe the main-only fallback and audit. The rebase onto #16567 keeps main's memory_pressure entry in QUEUED_KIND_TEXT, so a later read of a row the pressure hold keeps still names the pressure instead of "waiting to start". The main handle's trusted wait identity (#15022) gains a behavioural pin: an identity that is not provenance-verified must not name kirocrew-core on the dispatch snapshot. Backports the fixes for #14877, #15131, #15011 and #15045. Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com> Co-authored-by: Will Laws <will.laws@me.com> Co-authored-by: Matt Cohen <mattcohe@amazon.com> (cherry picked from commit 0f785ef) (cherry picked from commit 4bb1d11) (cherry picked from commit 6d99710) (cherry picked from commit c759e04)
Four main fixes for what users on 0.8 hit with parallel subagents: - #14878: a wave's "N waiting to start" count ends at "1 waiting" under a finished turn, because a popped but unclaimed row was counted once more than the window holds. A popped row is now marked dispatching and left out of the count, so the last pop publishes 0. - #16191: the count is re-published after every terminal, Stop all and stage Cancel, coalesced per parent; a failed store read publishes nothing instead of a false 0; a resident resume entry and a deferred spawn_async row are counted correctly; the pump's idle wake no longer spins (carries Matt Cohen's #15376 commit). - #15022: a subagent inside the wait tool is not reaped as stalled before its declared duration. - #16012: a spawn the memory gate deferred exists only as a task-store row, so spawn_status answered 404, spawn_list left it out and spawn_sub_agents reported it as an error, which made the model dispatch the work again. These ids now answer as queued, and the sub-agent synthesis waits while a sibling is still queued. Backport to release/0.8.0. #14878 and #15022 apply cleanly. #16012 is applied before #16191, its order on main, because #16191 was built on it. #16012's conflicts are resolved against 0.8 code: - api_spawn_list has no persisted half and no per-row audit on 0.8, so the queued half is listed beside the live half only. - The persistence fallback of GET /api/spawn/{id} keeps 0.8's shape (a run folder reads done); the "no recorded ending is not done" change needs main's ending classifier and is left out. - The Slack gateway and chat runner keep 0.8's orchestrator mode, so the synthesis arm stays chat-mode only. The tests that read main-only APIs are adapted to those shapes, and test/conftest.py gains main's close_subagent_managers fixture, which the new tests request. The subagent spec drops the sentences that describe the main-only fallback and audit. The rebase onto #16567 keeps main's memory_pressure entry in QUEUED_KIND_TEXT, so a later read of a row the pressure hold keeps still names the pressure instead of "waiting to start". The main handle's trusted wait identity (#15022) gains a behavioural pin: an identity that is not provenance-verified must not name kirocrew-core on the dispatch snapshot. Backports the fixes for #14877, #15131, #15011 and #15045. Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com> Co-authored-by: Will Laws <will.laws@me.com> Co-authored-by: Matt Cohen <mattcohe@amazon.com> (cherry picked from commit 0f785ef) (cherry picked from commit 4bb1d11) (cherry picked from commit 6d99710) (cherry picked from commit c759e04)
Four main fixes for what users on 0.8 hit with parallel subagents: - #14878: a wave's "N waiting to start" count ends at "1 waiting" under a finished turn, because a popped but unclaimed row was counted once more than the window holds. A popped row is now marked dispatching and left out of the count, so the last pop publishes 0. - #16191: the count is re-published after every terminal, Stop all and stage Cancel, coalesced per parent; a failed store read publishes nothing instead of a false 0; a resident resume entry and a deferred spawn_async row are counted correctly; the pump's idle wake no longer spins (carries Matt Cohen's #15376 commit). - #15022: a subagent inside the wait tool is not reaped as stalled before its declared duration. - #16012: a spawn the memory gate deferred exists only as a task-store row, so spawn_status answered 404, spawn_list left it out and spawn_sub_agents reported it as an error, which made the model dispatch the work again. These ids now answer as queued, and the sub-agent synthesis waits while a sibling is still queued. Backport to release/0.8.0. #14878 and #15022 apply cleanly. #16012 is applied before #16191, its order on main, because #16191 was built on it. #16012's conflicts are resolved against 0.8 code: - api_spawn_list has no persisted half and no per-row audit on 0.8, so the queued half is listed beside the live half only. - The persistence fallback of GET /api/spawn/{id} keeps 0.8's shape (a run folder reads done); the "no recorded ending is not done" change needs main's ending classifier and is left out. - The Slack gateway and chat runner keep 0.8's orchestrator mode, so the synthesis arm stays chat-mode only. The tests that read main-only APIs are adapted to those shapes, and test/conftest.py gains main's close_subagent_managers fixture, which the new tests request. The subagent spec drops the sentences that describe the main-only fallback and audit. The rebase onto #16567 keeps main's memory_pressure entry in QUEUED_KIND_TEXT, so a later read of a row the pressure hold keeps still names the pressure instead of "waiting to start". The main handle's trusted wait identity (#15022) gains a behavioural pin: an identity that is not provenance-verified must not name kirocrew-core on the dispatch snapshot. Backports the fixes for #14877, #15131, #15011 and #15045. Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com> Co-authored-by: Will Laws <will.laws@me.com> Co-authored-by: Matt Cohen <mattcohe@amazon.com> (cherry picked from commit 0f785ef) (cherry picked from commit 4bb1d11) (cherry picked from commit 6d99710) (cherry picked from commit c759e04)
Problem / Motivation
Goal: The subagent progress bar clears its "waiting to start" count when the last queued subagent actually starts, so a finished wave no longer shows a phantom waiting agent.
After a turn that spawned a wave of subagents finishes, the subagent progress bar keeps showing "1 waiting to start - queued behind the concurrency limit" with 0 running and every card Complete. The spinner and the Stop all button stay until the page is reloaded. The Subagents panel shows the same phantom "1 waiting to start" row above the completed cards.
Why it matters
The bar says work is still pending when nothing is. A user watching a
spawn_runwave cannot tell a finished turn from one with a stuck subagent, and the only way to clear it is a page reload.Not a goal
queued: 0; the backend never sent it).What changed (motivation → approach → change)
Symptom to root cause:
sseSubagentQueuedremoves the waiting count only when it receives asubagent_queuedevent withqueued: 0._emit_queue_depth: rows in the in-memory window plusstore.count_pending(...)for the parent, minustaskq_excluded_ids()(window rows, registered runs, rows still inspawn_asyncadmission).spawnclaims it. Between pop and claim that row is stillQUEUEDin the durable store and is in none of the exclusion sets, so the store count includes it. Every emit in that window reads one higher than the window: a 5-spawn / cap-4 wave publishes 4, 3, 2, 1 and never 0.The change gives the pump its own exclusion set.
SubagentManager._dispatching_idsholds the id of a row for exactly one dispatch attempt: set when the pump pops it, cleared the momentspawnanswers, whatever the answer is (started, re-queued, parked, refused, claim unavailable, or raised). A separatetaskq_dispatch_excluded_ids()(the ownership set plus_dispatching_ids) is what the pump's own reads use: the window refill, and the depth the chip shows (taskq_overflow(..., for_dispatch=True)). Cancellation and every pending-work read (has_pending_work_for,queued_count_for, the reset-deferral guards) keep the unchangedtaskq_excluded_ids(), so a parent's Stop still reaches a row in exactly this popped-unclaimed state, and a cron teardown still sees it as the parent's unfinished work. The emit right after the pop, and any refill or overflow read in the meantime, leave the popped row out, so the final emit reads 0. When the attempt does not take the row, the mark is dropped before that path's own depth emit, so a row that is still QUEUED is counted again immediately; a mark cannot outlive its pass (afinallyon both pump paths and on the pass itself), so no durable row is ever hidden from refill.Files:
src/kiro_crew/subagent.py(the set),src/kiro_crew/subagent_manager/admission/pump.py(mark on pop, clear on every attempt exit),src/kiro_crew/subagent_manager/admission/gate.py(clear before the failed-claim emit),src/kiro_crew/subagent_manager/admission/taskq_bridge.py(the dispatch-only exclusion set; refill uses it,taskq_overflow*only underfor_dispatch=True),src/kiro_crew/subagent_manager/run.py(only the depth emit asks for the dispatch reading).Backwards compatibility
Compatible: no schema, event shape, or public method changes; the
subagent_queuedevent carries the same fields with a corrected count.Tests
test/test_subagent_scale.py::TestQueuedDepthReachesZero::test_last_pop_publishes_zero_with_a_store_attached: a realSubagentManagerwith the durable store, cap 4, 5 spawns for one parent,_runpatched to release its slot and drain. Asserts the recordedsubagent_queueddepths end at 0 and stay 0. Fails onorigin/mainwithdepth sequence never reached 0and passes with the fix.test_dispatching_rows_are_excluded_from_dispatch_reads_only: a marked id is intaskq_dispatch_excluded_ids()and not intaskq_excluded_ids().test_a_popped_but_unclaimed_row_still_counts_as_pending_work: with the pump parked between pop and claim,has_pending_work_for_asyncis True andqueued_count_foris 1 while the chip's depth is 0. Makingtaskq_overflowdefault to the dispatch set turns it red.test_a_raise_before_the_pick_is_logged_not_rethrown: a store error in the awaits ahead of the pick is logged by the pass's handler and the pass returns; the mark cleanup walks an empty pick. Moving thepickedbinding back inside thetryturns it red withUnboundLocalError.test_stop_all_reaches_a_popped_but_unclaimed_row: the pump is parked between pop and claim (its off-loop policy read held) whencancel_for_parentruns; the store sweep stops the row and it never starts. Pointing the sweep at the dispatch set turns it red.test_an_unavailable_claim_publishes_the_row_as_waiting: the store'sclaimraisesTaskStoreUnavailablefor the popped row; the depth published by that path is 1 and no mark remains. Removing the gate-side discard turns it red.test_a_dispatch_that_raises_leaves_no_mark:spawnraises while dispatching the popped row; no mark remains and the row is not intaskq_excluded_ids(). Removing thefinallyon the inline pump turns it red.test/test_queue_depth_after_deferred_start.py(the memory-deferral re-queue path) passes.Manual verification
A standalone probe of the same shape (cap 4, 5 spawns, scratch
KIROCREW_HOME) on both pump paths (off-loop and on-loop) publishes... 1, 0with the fix and... 1without. mypy clean on the two admission modules; comment-history gate passes.Related Issues
Closes #14877
Pattern harvest
Rule candidate: review-prompt
Pattern: a count derived from a durable store must exclude every in-flight state the in-memory owner already accounts for; adding a new in-flight phase (popped-but-unclaimed) without adding it to the exclusion set silently double-counts.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)