Skip to content

fix(subagent): backport queue-depth fixes to 0.8 - #16388

Closed
bolichen97 wants to merge 2 commits into
release/0.8.0from
backport/0.8.0-subagent-queue-depth
Closed

bolichen97 wants to merge 2 commits into
release/0.8.0from
backport/0.8.0-subagent-queue-depth

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: (release/0.8.0 backport of #14878, #16191, #15022 and #16012) on 0.8, the "N waiting to start" count for parallel subagents is exact, a subagent inside the wait tool is not reaped as stalled, and a spawn the memory gate deferred answers as queued instead of 404, so the model stops dispatching the same work twice.

On release/0.8.0, users who run parallel subagents hit four bugs that main has fixed:

Why it matters

0.8 is the line users install. Until the next 0.8 tag carries these fixes, users with parallel subagents see a count that never clears, lose long waits to a false stall reap, and get duplicated subagent work, each copy costing a full run.

Not a goal

  • fix(subagent): show gate-deferred spawns as queued and hold synthesis #16012's "a run folder with no recorded ending is not done" change. It depends on main's classify_persisted_ending outcome model, which 0.8 does not have. Flipping 0.8's done: true fallback without that model could leave kirocrew spawn run polling forever on a dead folder.
  • Main's persisted half of GET /api/spawn and its per-row audit. 0.8 has neither, so the queued half sits only beside the live half.
  • Removing 0.8's orchestrator mode from the Slack gateway and the chat runner.
  • New tests for gaps that main also has, beyond the review fixes listed in "Review findings carried from main" below). Those land on main first and are then picked.

What changed (motivation → approach → change)

This PR implements mainline-plan unit R3 / G4 (fix(subagent): backport queue-depth fixes to 0.8). It depends only on the four main PRs, all merged: #14878, #16191, #15022 and #16012. It is not stacked on another open PR.

The first commit is one squashed commit of four cherry-pick -x picks, with a (cherry picked from commit …) line for each: 0f785ef26a (#14878), 4bb1d11cf1 (#16191), 6d99710085 (#15022) and c759e04543 (#16012). Authors are credited with Co-authored-by: Raymond Chen (#14878), Will Laws (#15022) and Matt Cohen (#16012, and his #15376 commit inside #16191). #16012 was applied before #16191, which is its order on main, because #16191 was built on it. The final tree does not depend on the order.

The second commit strips terminal controls from every kirocrew spawn list row. The first commit also carries the memory-pressure kind text the #16567 rebase dropped and a behavioural pin for the trusted wait identity; see "Review findings carried from main". The two review fixes used to share the second commit; they were split by subject, and the pin moved into the backport commit because it tests #15022 code, which keeps the PR at two commits.

Rebased onto #16567. release/0.8.0 gained the macOS memory-pressure hold (#16567) after this PR opened; #16315's 2 GiB floor was already under it. The rebase conflicted in six files, each resolved as the union main already has: the subagent_queued spec paragraph and _emit_queue_depth docstring keep this PR's coalesced-burst text plus #16567's "memory-pressure carries no figures" clause (main's wording); subagent.py keeps both sets of manager state (_pressure_* / _held_approval_modes and _queue_depth_* / _admitting_waiting / _dispatching_ids) and both timers in the shutdown probe; cancellation.py cancels both the depth retries and the pressure re-check at shutdown, in main's order; pump.py keeps both entry_is_resident_resume and entry_is_child stubs; mcp_tools/spawn.py imports both MEMORY_PRESSURE_PHRASE and queued_wait_text.

#14878, #15022 and #16191 applied with only context-line differences. #16012 was resolved against 0.8 code:

  • dashboard/handlers/messaging.py api_spawn_list: the queued half is listed beside the live half only, because 0.8 has no persisted half and no _audit_* helpers. An internal caller sees only its own queued rows, as in the live half. An app caller also sees only its own app's queued rows, filtered in the store read before its cap. The live half does not apply that app bound on 0.8, and the comment and spec now say so.
  • api_spawn_status: the persistence fallback keeps 0.8's shape (a run folder reads done: true).
  • slack/gateway.py: the synthesis arm sits under 0.8's if not _is_orchestrator, because orchestrator mode runs its own stage synthesis.

Spec updates in this PR: docs/system-specs/modules/subagent.md (queued visibility; the three main-only claims are removed), taskq.md, acp-client.md, cli.md, learn-cron-dashboard.md, and the user doc src/kiro_crew/docs/subagents.md.

Backwards compatibility

Compatible. Nothing 0.8 accepts today is rejected, renamed or removed. GET /api/spawn adds a queued key only with ?queued=1 and only when it is non-empty, so the default payload is unchanged. A deferred id that used to answer 404 now answers 200 with queued: true, or 503 when the queue cannot be read.

Tests

From the picked commits, adapted to 0.8 where noted. Each was confirmed red on a clean origin/release/0.8.0 tree with the branch's test files copied in (89 failed plus 1 import error):

Adapted to 0.8: the list-route tests no longer patch main's PanelRecords. The inline-collect test uses 0.8's plain-id deliveries instead of SubagentDelivery. The member-memory queued-listing test no longer patches _audit_*. test/conftest.py gains main's close_subagent_managers fixture verbatim, because the new tests request it. test_a_folder_with_no_recorded_ending_is_not_done is dropped (see Not a goal).

Local gate at the first head: black gate, isort, flake8, mypy src/kiro_crew (1862 files), the subprocess-encoding gate, scripts/docs-lint.sh and the harness-parity gate against origin/release/0.8.0 all pass. 165 targeted test files (every subagent, taskq, spawn, queued, liveness and stall file, plus every test file this diff touches): 6761 passed, 28 skipped. Frontend vitest was not run locally (npm registry E401). waitToolTitle.ts and its test are byte-identical to main's post-#15022 blobs, which main's CI tested.

After the rebase onto #16567, at 0977c21a0e: black (changed files), isort, flake8 and mypy src/kiro_crew (1864 files) pass. Targeted tests: every subagent, taskq, queue-depth, spawn-queued, spawn-floor, spawn-memory, admission-gate, queue-boundary, liveness, Slack gateway and chat-runner file (111 files, 4902 passed, 4 skipped), plus the remaining files either commit touches (test_cli, test_eager_spawn, test_resource_status, test_platform_compat, test_messaging_commands, test_kiro_prerequisite, test_acp_stale_recovery, test_mcp_core_more_coverage, test_dashboard_chat_handlers_coverage: 1952 passed, 30 skipped). The backend CI suite runs only on main-targeted PRs, so these local runs are this branch's test evidence.

After the rebase onto 04b1c8300d (#16694), at 4636ff956f: black (changed files) and the black gate, isort, flake8, mypy src/kiro_crew (1864 files), the subprocess-encoding gate, scripts/docs-lint.sh and the harness-parity gate against origin/release/0.8.0 pass. Targeted tests (test_cli*, test_spawn_*, test_subagent_queued*, test_queue_depth*, test_admission_gate, test_acp_liveness, test_messaging_commands, test_subagent_stall*, test_member_memory_api, test_taskq_store, test_subagent_scale): 3414 passed, 4 skipped.

Review findings carried from main

The pre-push review found three test gaps, each inherited from main (the same lines and test files are on main, unpinned there too):

The AI review of the first head also found that the queued kirocrew spawn list row printed the agent-authored id and task raw. That row is new on 0.8 with this backport, so it is fixed here (safe_terminal_line, pinned by test/test_cli.py::TestSpawnCliAuth::test_spawn_list_queued_row_prints_no_terminal_controls). The post-push review then found that the running, approval-waiting and done rows a few lines below still printed the id and task raw, so the second commit now formats every row through one helper (test/test_cli.py::TestSpawnCliAuth::test_spawn_list_agent_row_prints_no_terminal_controls, one case per icon). Main has the same rows and the same gap. The forward-port of the spawn-list sanitization and the trusted-wait identity pin to main is tracked in #16698.

The post-push review also found that the rebase onto #16567 left QUEUED_REASON_MEMORY_PRESSURE out of this PR's QUEUED_KIND_TEXT. A row the pressure hold keeps has no deferred event, so every later read (spawn_status, spawn_list, spawn_sub_agents, kirocrew spawn list) fell back to "waiting to start". The entry is restored as main has it (MEMORY_PRESSURE_DETAIL). It is pinned by test/test_spawn_queued_reason.py::TestSpawnStatusAndListShowAQueuedRun::test_a_row_the_pressure_hold_keeps_says_why_on_a_later_read and ::test_every_reason_the_gate_labels_has_words, which fail without the entry.

GPT's other first-head finding, the function-local from kiro_crew import taskq as _taskq imports in taskq_bridge.py, is not changed: it is that module's existing convention on both 0.8 and main (29 sites on the 0.8 base, 36 on main), the added sites are byte-identical to main's, and hoisting them is a module-wide refactor that would make every later main-to-0.8 pick of the file conflict. GPT did not raise it on the second head. The second head's one GPT finding (gate.py:383, downgraded on adjudication) is rebutted as inherited from main in its disposition comment.

Manual verification

N/A. Unit and route-level coverage drives the real SubagentManager, the task store and the aiohttp handlers. The backport changes no user-visible UI.

Screenshots / video

Why no screenshot: the only frontend change is a doc comment in website/src/utils/waitToolTitle.ts and its test; WAIT_TITLE_RE and every rendered surface are unchanged.

Related Issues

Part of #16480 (release/0.8.0 backport line)

Refs #14877
Refs #15131
Refs #15011
Refs #15045

Backports the fixes for #14877, #15131, #15011 and #15045. These were already closed by the main PRs, and closing keywords do nothing on a non-default branch. Backport of #14878, #16191, #15022 and #16012.

Pattern harvest

Not generalizable: a backport of four main fixes. Their pattern harvests are recorded on the main PRs.

Checklist

  • At most two commits (two here: the backport and its review fixes), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@bolichen97
bolichen97 requested a review from a team as a code owner October 3, 2026 06:16
@bolichen97
bolichen97 requested review from cruisercohen and removed request for a team October 3, 2026 06:16
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 79d8c3df0bd49a3dcc2cfe4213151f50dae33f46 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Full backend CI never runs on release-branch PRs — confirm a full-suite run (not just the targeted files) before the next 0.8 tag ships this.

[DESIGN-REVIEWED] 79d8c3d

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ✅ PASS

UX-level review of 79d8c3df0bd49a3dcc2cfe4213151f50dae33f46 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: PASS

Verify kirocrew spawn list output once in a real terminal — the 🕒 queued rows and ⏳ running rows ship untested against real-width tasks.

[UX-REVIEWED] 79d8c3d

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 79d8c3df0bd49a3dcc2cfe4213151f50dae33f46 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the six hand-resolved rebase conflicts (subagent.py, cancellation.py, pump.py, mcp_tools/spawn.py, two specs) against main's union — the only place this backport can diverge from the decided shape.

Every item traces to a linked issue (#14877, #15131, #15011, #15045) and a merged main PR, tests are stated red on base, and specs updated in the same commit. The two rewritten tests (test_running_agents_defer_synthesis, test_a_member_deferred_at_accept_is_reported_queued_with_its_reason) strengthen their pins rather than reverse them — the running-half hold is re-pinned in the new parametrized TestFinishQueueCycle cases. No new env var or config key; the new HTTP surface (?queued=1, queued_truncated, 503/409) has counted consumers (CLI spawn list, mcp_tools/spawn.py, chat-runner fire gate). The declared "Not a goal" scope-outs are respected in the diff.

What this change ships

Inventory (10 items) — 10 justified

Intent: FIX — backport four merged subagent queue-depth/liveness/visibility fixes from main onto release/0.8.0.

  1. "N waiting to start" count is exact; last pop publishes 0 — justified
  2. Count re-published after Stop all / cancel / terminal; failed store read publishes nothing, not a false 0; idle wake stops spinning — justified
  3. A subagent inside a declared wait is no longer reaped as stalled before its duration — justified
  4. A gate-deferred spawn answers 200 queued instead of 404; unreadable queue answers 503 — justified
  5. GET /api/spawn?queued=1 and kirocrew spawn list now show queued rows (🕒, partial-list note) — justified
  6. A control aimed at a queued spawn answers 409 "queued — not started", not 404 — justified
  7. Post-fan-out synthesis waits for queued siblings, with a bounded re-check when the store is unreadable — justified
  8. spawn list rows strip terminal control sequences from agent-authored text — rides along (second commit, declared); harm is the external-content boundary, so justified
  9. Memory-pressure kind text restored after the fix(subagent): backport the macOS memory-pressure hold to 0.8 #16567 rebase dropped it — rides along (declared review fix carried from main)
  10. wait tool's 1800s ceiling named once (WAIT_TOOL_MAX_SECS) across schema, clamp and liveness — rides along inside the fix(subagents): honour the wait tool's declared duration in the stall reaper #15022 pick, no behavior change

[FIRST-PRINCIPLES-REVIEWED] 79d8c3d

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 79d8c3df0bd49a3dcc2cfe4213151f50dae33f46 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 79d8c3d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 79d8c3df0bd49a3dcc2cfe4213151f50dae33f46: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 79d8c3df0bd49a3dcc2cfe4213151f50dae33f46 — this comment is updated in place on each push.

Review details

One advisory finding; nothing blocks.

FINDING — src/kiro_crew/dashboard/handlers/messaging.py:1431 — return _queue_unreadable() fires before the persisted-record fallback, so a GET /api/spawn/<id> for a finished run that is no longer in _agents (post-restart orphan, result on disk) answers 503 taskq_unavailable whenever store.get raises TaskStoreUnavailable (a locked read under journal_mode=delete, a closed store, or a row this build cannot model) — the base branch read read_state/result.txt and returned the real result, and the same ordering at :580 makes _spawn_scope_refusal refuse delete/steer/retry on the same run → Fix: in both sites, do the read_state/registry fallback first and only answer _queue_unreadable() when neither the registry nor the persisted record names the run.

[OPUS-REVIEWED] 79d8c3d

Verdict parsed from the review's SHA-scoped output markers for commit 79d8c3df0bd49a3dcc2cfe4213151f50dae33f46.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 79d8c3df0bd49a3dcc2cfe4213151f50dae33f46: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision labels Oct 3, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 3, 2026
@bolichen97
bolichen97 force-pushed the backport/0.8.0-subagent-queue-depth branch from f41f47e to 0977c21 Compare October 4, 2026 06:22
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=abaf2747bd41 gate.py:383 terminal record registered before the durable fail commits: rebutted (inherited from main, backport stays faithful)

Outcome: rebutted, no change in this backport.
_refuse_row (gate.py:370-383) is byte-identical to main's (origin/main:src/kiro_crew/subagent_manager/admission/gate.py:386-393); it arrives here with the #16012 pick. A release backport does not diverge from main on a design question main has not decided.
The adjudication's own trace shows the window closes itself: a lost taskq_fail leaves the row QUEUED, the refill re-pulls it (a done record is not a live run), and the gate re-refuses it on the next drain, so refused work never executes. Converting the synchronous spawn_impl choke point to await the write is a main-first design change, not a backport fix.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=8aefccc152d8 cli_commands.py:363 queued spawn list row prints agent-authored text raw: fixed in 0977c21

Outcome: fixed. The queued row's id and task now pass through safe_terminal_line before printing.
Test: test/test_cli.py::TestSpawnCliAuth::test_spawn_list_queued_row_prints_no_terminal_controls feeds an OSC title sequence in the id and a CSI clear plus a newline in the task. It fails without the fix (ESC reaches stdout) and passes with it. Opus 5 re-reviewed 0977c21 with no findings. Main still has the same row; forward-port tracked in #16698.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Outcome: accepted. The description's "Review findings carried from main" paragraph is corrected: it now says the trusted-wait-identity pin and the queued-row sanitization land here first, as review fixes for code this backport ships, and that main carries both gaps until #16698 lands.
The forward-port to main is tracked in #16698 (open, labelled deferred-finding, owned, Due: 2026-10-11). It names both changes and their tests, so a later main-to-0.8 pick of these files can be checked against it.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 4, 2026
@bolichen97
bolichen97 force-pushed the backport/0.8.0-subagent-queue-depth branch from 0977c21 to 4636ff9 Compare October 4, 2026 08:18
@bolichen97

Copy link
Copy Markdown
Contributor Author

Pushed 4636ff956f (lease pinned to 0977c21a0e), rebased onto release/0.8.0 at 04b1c8300d (#16694).

@bolichen97

Copy link
Copy Markdown
Contributor Author

Outcome: accepted. The race is real, but this backport did not introduce it, and the fix is a separate change that has to land after this one.
Reproduced: a probe with explicit barriers, no sleeps, lets Stop all's list_pending read the row while it is queued, commits the pump's claim (admitted), then lets Stop all cancel the row before the pump resumes. This is the read_before_claim case of main's test_stop_all_between_claim_and_start_keeps_the_row_stopped. It fails the same way on release/0.8.0 at f4a3695108 and on this head. Both times Stop all returns (0, 1), the store row is cancelled, and the run task exists anyway.
This diff leaves both halves of the race as they are on 0.8. claim_and_start re-reads only when point.boundary_owner is set (pump.py:400, and release/0.8.0 pump.py:369 is the same), and Stop all's store pass reads taskq_excluded_ids exactly as before. The new _dispatching_ids set is deliberately left out of that read (see the taskq_dispatch_excluded_ids docstring).
Main's fix is #16390 (ad9fc4a19e, merged after this backport was cut). It does not apply to release/0.8.0 without this PR: its tests extend test/test_queue_depth_reconcile.py and use settle_depth_emits and _dispatching_ids, and this PR is what adds all three. Folding it in here would add a third logical change to a PR already at the two-commit limit. #16882 (open, deferred-finding, owned, Due: 2026-10-12) tracks the stacked backport, and it is done when all three of #16390's test cases pass on 0.8.

bolichen97 and others added 2 commits October 5, 2026 01:59
Four main fixes for what users on 0.8 hit with parallel subagents:

- #14878: a wave's "N waiting to start" count ends at "1 waiting" under
  a finished turn, because a popped but unclaimed row was counted once
  more than the window holds. A popped row is now marked dispatching
  and left out of the count, so the last pop publishes 0.
- #16191: the count is re-published after every terminal, Stop all and
  stage Cancel, coalesced per parent; a failed store read publishes
  nothing instead of a false 0; a resident resume entry and a deferred
  spawn_async row are counted correctly; the pump's idle wake no longer
  spins (carries Matt Cohen's #15376 commit).
- #15022: a subagent inside the wait tool is not reaped as stalled
  before its declared duration.
- #16012: a spawn the memory gate deferred exists only as a task-store
  row, so spawn_status answered 404, spawn_list left it out and
  spawn_sub_agents reported it as an error, which made the model
  dispatch the work again. These ids now answer as queued, and the
  sub-agent synthesis waits while a sibling is still queued.

Backport to release/0.8.0. #14878 and #15022 apply cleanly. #16012 is
applied before #16191, its order on main, because #16191 was built on
it. #16012's conflicts are resolved against 0.8 code:
- api_spawn_list has no persisted half and no per-row audit on 0.8, so
  the queued half is listed beside the live half only.
- The persistence fallback of GET /api/spawn/{id} keeps 0.8's shape (a
  run folder reads done); the "no recorded ending is not done" change
  needs main's ending classifier and is left out.
- The Slack gateway and chat runner keep 0.8's orchestrator mode, so
  the synthesis arm stays chat-mode only.
The tests that read main-only APIs are adapted to those shapes, and
test/conftest.py gains main's close_subagent_managers fixture, which
the new tests request. The subagent spec drops the sentences that
describe the main-only fallback and audit.

The rebase onto #16567 keeps main's memory_pressure entry in
QUEUED_KIND_TEXT, so a later read of a row the pressure hold keeps
still names the pressure instead of "waiting to start". The main
handle's trusted wait identity (#15022) gains a behavioural pin: an
identity that is not provenance-verified must not name kirocrew-core
on the dispatch snapshot.

Backports the fixes for #14877, #15131, #15011 and #15045.

Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com>
Co-authored-by: Will Laws <will.laws@me.com>
Co-authored-by: Matt Cohen <mattcohe@amazon.com>
(cherry picked from commit 0f785ef)
(cherry picked from commit 4bb1d11)
(cherry picked from commit 6d99710)
(cherry picked from commit c759e04)
kirocrew spawn list printed each row's agent-authored id and task
straight to the terminal, so an escape sequence or newline in a spawn's
task reached the operator's terminal live and could forge another row.
The queued rows this backport adds and the running, approval-waiting
and done rows below them now share one formatter that passes both
fields through safe_terminal_line.

Main has the same gap; the forward-port is tracked in #16698.
@bolichen97
bolichen97 force-pushed the backport/0.8.0-subagent-queue-depth branch from 2d90fd4 to 79d8c3d Compare October 5, 2026 01:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 5, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Closing: the owner decided the remaining subagent fixes ship in the next release from main rather than as further 0.8.x backports. The backports already on release/0.8.0 (#16315, #16567, #16302) stay.

@bolichen97 bolichen97 closed this Oct 5, 2026
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant