Repository navigation
fix(subagent): backport queue-depth fixes to 0.8 - #16388
bolichen97 wants to merge 2 commits into
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Full backend CI never runs on release-branch PRs — confirm a full-suite run (not just the targeted files) before the next 0.8 tag ships this. [DESIGN-REVIEWED] 79d8c3d |
UX Review (Fable 5) — ✅ PASSUX-level review of UX-Verdict: PASS Verify [UX-REVIEWED] 79d8c3d |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the six hand-resolved rebase conflicts ( Every item traces to a linked issue (#14877, #15131, #15011, #15045) and a merged main PR, tests are stated red on base, and specs updated in the same commit. The two rewritten tests ( What this change shipsInventory (10 items) — 10 justifiedIntent: FIX — backport four merged subagent queue-depth/liveness/visibility fixes from main onto release/0.8.0.
[FIRST-PRINCIPLES-REVIEWED] 79d8c3d |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsOne advisory finding; nothing blocks. FINDING — src/kiro_crew/dashboard/handlers/messaging.py:1431 — [OPUS-REVIEWED] 79d8c3d Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
f41f47e to
0977c21
Compare
|
|
|
0977c21 to
4636ff9
Compare
|
Pushed
|
4636ff9 to
2d90fd4
Compare
|
Four main fixes for what users on 0.8 hit with parallel subagents: - #14878: a wave's "N waiting to start" count ends at "1 waiting" under a finished turn, because a popped but unclaimed row was counted once more than the window holds. A popped row is now marked dispatching and left out of the count, so the last pop publishes 0. - #16191: the count is re-published after every terminal, Stop all and stage Cancel, coalesced per parent; a failed store read publishes nothing instead of a false 0; a resident resume entry and a deferred spawn_async row are counted correctly; the pump's idle wake no longer spins (carries Matt Cohen's #15376 commit). - #15022: a subagent inside the wait tool is not reaped as stalled before its declared duration. - #16012: a spawn the memory gate deferred exists only as a task-store row, so spawn_status answered 404, spawn_list left it out and spawn_sub_agents reported it as an error, which made the model dispatch the work again. These ids now answer as queued, and the sub-agent synthesis waits while a sibling is still queued. Backport to release/0.8.0. #14878 and #15022 apply cleanly. #16012 is applied before #16191, its order on main, because #16191 was built on it. #16012's conflicts are resolved against 0.8 code: - api_spawn_list has no persisted half and no per-row audit on 0.8, so the queued half is listed beside the live half only. - The persistence fallback of GET /api/spawn/{id} keeps 0.8's shape (a run folder reads done); the "no recorded ending is not done" change needs main's ending classifier and is left out. - The Slack gateway and chat runner keep 0.8's orchestrator mode, so the synthesis arm stays chat-mode only. The tests that read main-only APIs are adapted to those shapes, and test/conftest.py gains main's close_subagent_managers fixture, which the new tests request. The subagent spec drops the sentences that describe the main-only fallback and audit. The rebase onto #16567 keeps main's memory_pressure entry in QUEUED_KIND_TEXT, so a later read of a row the pressure hold keeps still names the pressure instead of "waiting to start". The main handle's trusted wait identity (#15022) gains a behavioural pin: an identity that is not provenance-verified must not name kirocrew-core on the dispatch snapshot. Backports the fixes for #14877, #15131, #15011 and #15045. Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com> Co-authored-by: Will Laws <will.laws@me.com> Co-authored-by: Matt Cohen <mattcohe@amazon.com> (cherry picked from commit 0f785ef) (cherry picked from commit 4bb1d11) (cherry picked from commit 6d99710) (cherry picked from commit c759e04)
kirocrew spawn list printed each row's agent-authored id and task straight to the terminal, so an escape sequence or newline in a spawn's task reached the operator's terminal live and could forge another row. The queued rows this backport adds and the running, approval-waiting and done rows below them now share one formatter that passes both fields through safe_terminal_line. Main has the same gap; the forward-port is tracked in #16698.
2d90fd4 to
79d8c3d
Compare
Problem / Motivation
Goal: (release/0.8.0 backport of #14878, #16191, #15022 and #16012) on 0.8, the "N waiting to start" count for parallel subagents is exact, a subagent inside the wait tool is not reaped as stalled, and a spawn the memory gate deferred answers as queued instead of 404, so the model stops dispatching the same work twice.
On release/0.8.0, users who run parallel subagents hit four bugs that main has fixed:
waitis flagged as stalled and reaped before its declared duration (subagents: a sub-agent inside kirocrew-core wait is marked "stalled" - the reaper declines non-shell tools before the oracle's wait contract #15011).spawn_statusanswers 404,spawn_listleaves it out, andspawn_sub_agentsreports it as an error, so the model dispatches the work again (subagents: a gate-deferred run is invisible to spawn_status/spawn_list, reported as an error by spawn_sub_agents, and skipped by the synthesis guard #15045).Why it matters
0.8 is the line users install. Until the next 0.8 tag carries these fixes, users with parallel subagents see a count that never clears, lose long waits to a false stall reap, and get duplicated subagent work, each copy costing a full run.
Not a goal
classify_persisted_endingoutcome model, which 0.8 does not have. Flipping 0.8'sdone: truefallback without that model could leavekirocrew spawn runpolling forever on a dead folder.GET /api/spawnand its per-row audit. 0.8 has neither, so the queued half sits only beside the live half.What changed (motivation → approach → change)
This PR implements mainline-plan unit R3 / G4 (
fix(subagent): backport queue-depth fixes to 0.8). It depends only on the four main PRs, all merged: #14878, #16191, #15022 and #16012. It is not stacked on another open PR.The first commit is one squashed commit of four
cherry-pick -xpicks, with a(cherry picked from commit …)line for each:0f785ef26a(#14878),4bb1d11cf1(#16191),6d99710085(#15022) andc759e04543(#16012). Authors are credited with Co-authored-by: Raymond Chen (#14878), Will Laws (#15022) and Matt Cohen (#16012, and his #15376 commit inside #16191). #16012 was applied before #16191, which is its order on main, because #16191 was built on it. The final tree does not depend on the order.spawn_asyncrow are counted correctly.TaskStore.next_eligible_atnow wakes only for rows that time holds, so the idle wake no longer spins.GET /api/spawn/{id},GET /api/spawn?queued=1,spawn_statusandspawn_sub_agents. An unreadable queue answers 503, never a definitive 404. The sub-agent synthesis waits while a sibling is still queued.The second commit strips terminal controls from every
kirocrew spawn listrow. The first commit also carries the memory-pressure kind text the #16567 rebase dropped and a behavioural pin for the trusted wait identity; see "Review findings carried from main". The two review fixes used to share the second commit; they were split by subject, and the pin moved into the backport commit because it tests #15022 code, which keeps the PR at two commits.Rebased onto #16567. release/0.8.0 gained the macOS memory-pressure hold (#16567) after this PR opened; #16315's 2 GiB floor was already under it. The rebase conflicted in six files, each resolved as the union main already has: the
subagent_queuedspec paragraph and_emit_queue_depthdocstring keep this PR's coalesced-burst text plus #16567's "memory-pressure carries no figures" clause (main's wording);subagent.pykeeps both sets of manager state (_pressure_*/_held_approval_modesand_queue_depth_*/_admitting_waiting/_dispatching_ids) and both timers in the shutdown probe;cancellation.pycancels both the depth retries and the pressure re-check at shutdown, in main's order;pump.pykeeps bothentry_is_resident_resumeandentry_is_childstubs;mcp_tools/spawn.pyimports bothMEMORY_PRESSURE_PHRASEandqueued_wait_text.#14878, #15022 and #16191 applied with only context-line differences. #16012 was resolved against 0.8 code:
dashboard/handlers/messaging.pyapi_spawn_list: the queued half is listed beside the live half only, because 0.8 has no persisted half and no_audit_*helpers. An internal caller sees only its own queued rows, as in the live half. An app caller also sees only its own app's queued rows, filtered in the store read before its cap. The live half does not apply that app bound on 0.8, and the comment and spec now say so.api_spawn_status: the persistence fallback keeps 0.8's shape (a run folder readsdone: true).slack/gateway.py: the synthesis arm sits under 0.8'sif not _is_orchestrator, because orchestrator mode runs its own stage synthesis.Spec updates in this PR:
docs/system-specs/modules/subagent.md(queued visibility; the three main-only claims are removed),taskq.md,acp-client.md,cli.md,learn-cron-dashboard.md, and the user docsrc/kiro_crew/docs/subagents.md.Backwards compatibility
Compatible. Nothing 0.8 accepts today is rejected, renamed or removed.
GET /api/spawnadds aqueuedkey only with?queued=1and only when it is non-empty, so the default payload is unchanged. A deferred id that used to answer 404 now answers 200 withqueued: true, or 503 when the queue cannot be read.Tests
From the picked commits, adapted to 0.8 where noted. Each was confirmed red on a clean
origin/release/0.8.0tree with the branch's test files copied in (89 failed plus 1 import error):test/test_queue_depth_reconcile.py(fix(subagent): keep the queued count exact and cheap #16191 F4): after each exit, the published depth equals the store count, which equals the card reducer.test/test_subagent_scale.py::TestQueuedDepthReachesZero::test_last_pop_publishes_zero_with_a_store_attached(fix(subagent): publish queued depth 0 after the last row is popped #14878).test/test_spawn_queued_scope.py::test_every_queued_child_of_a_wave_answers_queued_to_its_parentandtest/test_member_memory_api.py::test_a_queued_run_is_owned_by_its_rows_session_and_reported_queued(fix(subagent): show gate-deferred spawns as queued and hold synthesis #16012): a deferred id answers queued at the HTTP route, not 404.test/test_subagent_queued_visibility.py::test_a_deferred_spawn_is_readable_until_it_starts(fix(subagent): show gate-deferred spawns as queued and hold synthesis #16012, manager read) and::test_an_unreadable_queue_answers_503_not_404.test/test_spawn_queued_reason.py::TestSpawnSubAgentsNamesTheDeferral::test_a_not_found_answer_is_an_error_never_a_queued_member(fix(subagent): show gate-deferred spawns as queued and hold synthesis #16012).test/test_subagent_stall_attribution.py::test_a_subagent_inside_a_declared_wait_is_not_flagged(fix(subagents): honour the wait tool's declared duration in the stall reaper #15022).test/test_taskq_store.py::test_next_eligible_at_is_a_wake_only_for_rows_time_holds(fix(subagent): keep the queued count exact and cheap #16191).Adapted to 0.8: the list-route tests no longer patch main's
PanelRecords. The inline-collect test uses 0.8's plain-id deliveries instead ofSubagentDelivery. The member-memory queued-listing test no longer patches_audit_*.test/conftest.pygains main'sclose_subagent_managersfixture verbatim, because the new tests request it.test_a_folder_with_no_recorded_ending_is_not_doneis dropped (see Not a goal).Local gate at the first head: black gate, isort, flake8,
mypy src/kiro_crew(1862 files), the subprocess-encoding gate,scripts/docs-lint.shand the harness-parity gate againstorigin/release/0.8.0all pass. 165 targeted test files (every subagent, taskq, spawn, queued, liveness and stall file, plus every test file this diff touches): 6761 passed, 28 skipped. Frontend vitest was not run locally (npm registry E401).waitToolTitle.tsand its test are byte-identical to main's post-#15022 blobs, which main's CI tested.After the rebase onto #16567, at
0977c21a0e: black (changed files), isort, flake8 andmypy src/kiro_crew(1864 files) pass. Targeted tests: every subagent, taskq, queue-depth, spawn-queued, spawn-floor, spawn-memory, admission-gate, queue-boundary, liveness, Slack gateway and chat-runner file (111 files, 4902 passed, 4 skipped), plus the remaining files either commit touches (test_cli,test_eager_spawn,test_resource_status,test_platform_compat,test_messaging_commands,test_kiro_prerequisite,test_acp_stale_recovery,test_mcp_core_more_coverage,test_dashboard_chat_handlers_coverage: 1952 passed, 30 skipped). The backend CI suite runs only on main-targeted PRs, so these local runs are this branch's test evidence.After the rebase onto
04b1c8300d(#16694), at4636ff956f: black (changed files) and the black gate, isort, flake8,mypy src/kiro_crew(1864 files), the subprocess-encoding gate,scripts/docs-lint.shand the harness-parity gate againstorigin/release/0.8.0pass. Targeted tests (test_cli*,test_spawn_*,test_subagent_queued*,test_queue_depth*,test_admission_gate,test_acp_liveness,test_messaging_commands,test_subagent_stall*,test_member_memory_api,test_taskq_store,test_subagent_scale): 3414 passed, 4 skipped.Review findings carried from main
The pre-push review found three test gaps, each inherited from main (the same lines and test files are on main, unpinned there too):
session_handle.py: the main-agent wiring of the trusted wait identity (mcp_identity_trusted) had no behavioural test (fix(subagents): honour the wait tool's declared duration in the stall reaper #15022). Pinned here in the backport commit, after GPT raised it on review:test/test_acp_liveness.py::test_main_dispatch_names_the_server_only_from_a_trusted_identityfails when the guard is removed.mcp_tools/spawn.py_held_by_a_deferral: theresuming is not Trueclause is not pinned (fix(subagent): show gate-deferred spawns as queued and hold synthesis #16012). Left for main.admission/gate.py: the_dispatching_ids.discardon a re-queue is not pinned (fix(subagent): publish queued depth 0 after the last row is popped #14878). Left for main.The AI review of the first head also found that the queued
kirocrew spawn listrow printed the agent-authored id and task raw. That row is new on 0.8 with this backport, so it is fixed here (safe_terminal_line, pinned bytest/test_cli.py::TestSpawnCliAuth::test_spawn_list_queued_row_prints_no_terminal_controls). The post-push review then found that the running, approval-waiting and done rows a few lines below still printed the id and task raw, so the second commit now formats every row through one helper (test/test_cli.py::TestSpawnCliAuth::test_spawn_list_agent_row_prints_no_terminal_controls, one case per icon). Main has the same rows and the same gap. The forward-port of the spawn-list sanitization and the trusted-wait identity pin to main is tracked in #16698.The post-push review also found that the rebase onto #16567 left
QUEUED_REASON_MEMORY_PRESSUREout of this PR'sQUEUED_KIND_TEXT. A row the pressure hold keeps has nodeferredevent, so every later read (spawn_status,spawn_list,spawn_sub_agents,kirocrew spawn list) fell back to "waiting to start". The entry is restored as main has it (MEMORY_PRESSURE_DETAIL). It is pinned bytest/test_spawn_queued_reason.py::TestSpawnStatusAndListShowAQueuedRun::test_a_row_the_pressure_hold_keeps_says_why_on_a_later_readand::test_every_reason_the_gate_labels_has_words, which fail without the entry.GPT's other first-head finding, the function-local
from kiro_crew import taskq as _taskqimports intaskq_bridge.py, is not changed: it is that module's existing convention on both 0.8 and main (29 sites on the 0.8 base, 36 on main), the added sites are byte-identical to main's, and hoisting them is a module-wide refactor that would make every later main-to-0.8 pick of the file conflict. GPT did not raise it on the second head. The second head's one GPT finding (gate.py:383, downgraded on adjudication) is rebutted as inherited from main in its disposition comment.Manual verification
N/A. Unit and route-level coverage drives the real
SubagentManager, the task store and the aiohttp handlers. The backport changes no user-visible UI.Screenshots / video
Why no screenshot: the only frontend change is a doc comment in
website/src/utils/waitToolTitle.tsand its test;WAIT_TITLE_REand every rendered surface are unchanged.Related Issues
Part of #16480 (release/0.8.0 backport line)
Refs #14877
Refs #15131
Refs #15011
Refs #15045
Backports the fixes for #14877, #15131, #15011 and #15045. These were already closed by the main PRs, and closing keywords do nothing on a non-default branch. Backport of #14878, #16191, #15022 and #16012.
Pattern harvest
Not generalizable: a backport of four main fixes. Their pattern harvests are recorded on the main PRs.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement