Repository navigation
refactor(subagent): let memory, not a count cap, bound concurrency - #17017
Conversation
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Nothing limits the running count after admission any more: build-heavy children admitted at settled size can grow past free memory, and no control lowers the count. Watch
[DESIGN-REVIEWED] 5f0dd25 Description read[DESCRIPTION-READ] bd0f3596577609c2bc7437650f08246243fb35e2f201abdd22f6b1dbc5ebda3c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS The degrade reason names only "Backend starts", so when Subagent runs are also lowered, nothing on the card says why. Watch
Suggestions
[UX-REVIEWED] 5f0dd25 Description read[DESCRIPTION-READ] d286c9f69db46ff0101eb275605b5ec4cf65e9987dfbb065177153b4aaebf562 That sha256 names everything this verdict read: the PR description, plus the 7 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ human override accepted@bolichen97 overrode this lane for |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings block merge; one advisory finding on the runner lane's memory bound. FINDING — src/kiro_crew/adaptive/policy.py:203 — the runner lane's ceiling is the manager's [OPUS-REVIEWED] 5f0dd25 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
d835d50 to
b954628
Compare
|
The screenshots now attached to the description show the same card's capacity column labels its second lane "Backend starts" ("Up to 2 at once, lowered from your 8"), directly above the Degrade reason row. The pause and probe copy names that same unit, so a reader matches the reason to the lane it lowered. "Agent starts" would be wrong: the spawn gate holds MCP backend initializations at its floor, not agent starts, and the "Subagent runs" lane next to it is not paused (the execution cap never pauses in this PR). |
The pause holds the spawn gate (MCP backend initializations) at its floor. Agent runs are bounded by the separate "Subagent runs" lane, which this PR makes un-pausable. Naming agents would tell the user their subagents are stopped when they are not. The current copy matches the "Backend starts" lane label on the same card (see the attached screenshots). The missing-screenshot evidence gap is closed: both states are now attached to the description. |
|
Agreed: |
|
New |
The probe is removed rather than moved to a worker thread. Both sites (mcp_broker.py and gateway_runtime/admission.py) now call |
#16480 item 2 is this symptom, reported by the owner: "More than two subagents throttle each other. Hidden count caps and an adaptive controller that reacts to event-loop lag limit concurrency well below what free memory allows, and one chat's wave can starve another chat's starts." The PR is "Part of #16480" (unit U4 of that redesign). #14013 covers the memory-priced auto cap. On the title: the change ships new defaults, so the description names them as behaviour changes in "What changed" and "Backwards compatibility". The conventional-commit type does not change what the review reads. |
#16480 item 2 names "an adaptive controller that reacts to event-loop lag" and limits "well below what free memory allows" as the defect. Starting the exec track at 4 is one of the count bounds that produced it. Memory is bounded per start by the floor (#16355, merged), so the exec cap does not have to earn its way up to protect memory. |
A 250 ms lag blip (a GC pause, a large read) halved the cap for every chat at once, which is exactly the "reacts to event-loop lag" defect in #16480. Lag and memory still cut, pause and probe the spawn gate (unchanged thresholds), and work evidence (timeouts, slow starts, failing inits, fd/process exhaustion) still cuts the exec cap. The deleted tests pinned the old exec-track rule; the new ones pin the split (test_subagent_memory_bounds_concurrency.py (ii) and (v), test_adaptive_policy.py). |
The new shape (exec track starts at its ceiling, cut only on work evidence; lag and memory shape the spawn gate only) is the owner-directed fix for #16480 item 2, built on the per-start memory floor that #16355 merged. |
b954628 to
87f88b7
Compare
|
Fixed in 87f88b7. |
The finding is accurate about the shape. The lane's ceiling is the manager's
What is missing is the per-start price for a runner-lane start, which the subagent spawn floor gives to subagent starts. Putting a memory term back on the exec track would bring back the count-under-free-memory throttle that #16480 symptom b removes, so it is not the fix. Pricing runner-lane admissions with |
The description's |
The pause and probe strings lead with "Backend starts" so that, in the global Degrade reason row, the reader can tell which lane the reason explains (the fix for the round-two finding on 87f88b7). A lane-free variant under the lane would need a second catalog key per reason and a selection in the placement code. The repeat is redundant, not misleading, and the lane's own line still carries the number. This round's single push was reserved for the runner-lane defect. |
"Backend starts: testing one before raising the limit" names what a probe does: it admits one backend start, and a clean start raises the limit while a failed one keeps the floor. "Recovering" would promise the outcome before the probe has run. |
The execution track cuts on file-descriptor and process exhaustion as well as on timeouts and slow or failing starts ( |
The description's Repeated from 4fd9dc3 on the rebased head d28bb78. The rebase onto main changed no behavior of this PR: it resolved conflicts with #16852 (the memory-sized figure now prices a typical run and reserves the heaviest agent's p90 once), which this PR keeps for the TaskRunner auto value and the disabled-floor fallback, and updated the matching docstrings, |
The pause and probe strings lead with "Backend starts" so that, in the global Degrade reason row, the reader can tell which lane the reason explains (the fix for the round-two finding on 87f88b7). A lane-free variant under the lane would need a second catalog key per reason and a selection in the placement code. The repeat is redundant, not misleading, and the lane's own line still carries the number. This round's single push was reserved for the runner-lane defect. |
"Backend starts: testing one before raising the limit" names what a probe does: it admits one backend start, and a clean start raises the limit while a failed one keeps the floor. "Recovering" would promise the outcome before the probe has run. |
The execution track cuts on file-descriptor and process exhaustion as well as on timeouts and slow or failing starts ( Repeated from 4fd9dc3 on the rebased head d28bb78. The rebase onto main changed no behavior of this PR: it resolved conflicts with #16852 (the memory-sized figure now prices a typical run and reserves the heaviest agent's p90 once), which this PR keeps for the TaskRunner auto value and the disabled-floor fallback, and updated the matching docstrings, |
That wording says agent starts are slowed, which is the misstatement described in the "Backend starts" disposition on this head. The Subagent runs lane is not throttled by memory or load in this PR (the execution track ignores lag and memory, RFC Q11), so a reader told "new agents are slowed" would expect the wrong lane to drop. The wrap comes from the narrow value column of the Degrade reason row, which this PR does not change. The string is 14 words and states the floor ("held to the minimum") and both release conditions. On 87f88b7 the shorter "Paused" was replaced because it contradicted the lane's "Up to 1 at once". Losing either condition would be a regression in accuracy to fix a layout issue. |
|
The execution track cuts on file-descriptor and process exhaustion ( |
The description's Repeated from d28bb78 on head 5f0dd25. This push rebased onto main (picking up #17145) and changed only one test fixture and assertion in |
"Backend starts: testing one before raising the limit" names what a probe does: it admits one backend start, and a clean start raises the limit while a failed one keeps the floor. "Recovering" would promise the outcome before the probe has run. |
The execution track cuts on file-descriptor and process exhaustion as well as on timeouts and slow or failing starts ( |
Repeated from d28bb78 on head 5f0dd25. This push rebased onto main (picking up #17145) and changed only one test fixture and assertion in |
|
Reviewed for merge triage, not approving from my side: ~2.6k added / ~1.3k removed lines across 68 files replacing the subagent count cap with a memory bound. That changes how many subagents may run at once on every host, a capacity/product decision for a maintainer. Auto-merge is already on, so a single approval merges it; please make sure a maintainer reviews it directly. CI is green. |
Problem / Motivation
Goal: Free host memory, not a count cap, decides how many subagents run at once, so one chat's subagents stop throttling another's while memory is still free.
With more than two chats using subagents, starts queued behind each other although the host had plenty of free memory. Three count bounds sat under the per-start memory floor (
agent.spawn_min_memory_gb), and each one bound first:max_subagents=0) was sized from host memory, and priced every slot at the heaviest agent's p90 peak (Auto subagent cap priced every slot at the heaviest agent's p90 peak #14013);Why it matters
Anyone who fans out work across several chats or a wide subagent tree gets serialized starts and long queues on a host with tens of GiB free. Waits show as
concurrency_limitand look like a capacity problem, but no resource is actually short. A loop-lag blip (a GC pause, a large file read) could halve the cap for every chat at once.Not a goal
max_subagentsof 1 or 2 (still floored at 3, per the owner's decision).What changed (motivation → approach → change)
Symptom: starts queued under a count while memory was free. Root cause: three count bounds each bound before the memory floor did. The change turns each of them into a high ceiling or a work-evidence rule, and leaves memory to the per-start floor.
C2: count ceiling.
compute_max_subagentsreturnsagent.subagent_auto_max(32), never below 3. It falls back to 3 only when host memory cannot be read, because the floor then admits everything.agent.spawn_min_memory_gb <= 0) admits everything too, so there the auto cap stays sized from host memory (compute_memory_sized_parallel_cap, main's figure), and the floor switch plus that figure's inputs (subagent_mem_buffer_pct,subagent_cost_gb,session.pool_size) stay live sizing paths on the manager.compute_memory_sized_parallel_cap, which only the TaskRunner's auto value uses: no per-start floor prices a TaskRunner step.max_subagents=Nis unchanged.C3: adaptive controller split into two tracks.
policy._exec_step) reads work signals only: timeouts, slow starts, failing inits, and fd or process exhaustion. It needs two of them to cut, it never pauses, and it starts at its ceiling. A raised ceiling is followed at once if the cap sat at the old one.signals.HOST_ONLY_SIGNALS) shape only the spawn-gate track, which keeps the old rules: either alone can cut it, and severe pressure pauses or probes it.agent.adaptive_initialis inert. It is deliberately not flaggeddeprecated: every save writes it, so a deprecation notice would fire on every launch.C4: spawn gate ceiling.
derive_spawn_gate_ceilingreturns max(mcp_gateway.spawn_concurrency_max, subagent ceiling).subagent_count_ceiling: the explicit pin floored at 3, elsesubagent_auto_max), never from the subagent manager and never through a host-memory probe. On boot the broker starts before_init_subagentsbuilds the manager. A manager-derived figure therefore launched the daemon at 8 while the controller targeted 32, and the daemon clamped every raise back. The boot path before readiness takes no memory reading; the gate needs only the upper bound the cap can reach.spawn_concurrency_initialand climbs only on clean backend inits.C1/F7: every consumer has a defined figure.
{{MAX_SUBAGENTS}}prompt token and the spawn tool text show the ceiling. The tool text adds that each start also waits for host memory./api/systemsubagent_capreportsresolve_max_subagents, including an explicit pin.B5 and copy.
adaptive_cap_zerowait reason is gone (it is unreachable now, and cron never used it), together with its UI strings.exec_signals, RFC Q11): Subagent runs are lowered only on evidence from the work itself (timeouts, slow or failing starts, running out of file handles or processes), Backend starts also by low memory or load. French uses U+202F before ";" and ":". Every translated catalog is updated anden-XAregenerated.subagent_mem_buffer_pct,subagent_cost_gbandspawn_min_memory_gb) andconfig-baseline.jsonare updated.TaskRunner steps re-check memory at a parked grant. The runner lane's limit is the execution cap, which no longer pauses on memory, so a step that waited behind a full lane would start on whatever memory read at the grant: its pressure check predated the wait.
RunnerAdmission.admitnow re-runs that check when the step had to park, and a refusal gives the slot back and defers again (taskq.md).Accepted decision: settled, not peak, start pricing. The old auto cap priced every slot at the heaviest agent's p90 whole-run peak, so growth after admission was bounded by the count. The floor prices a start at its settled size, so a wave of build-heavy children can grow past the floor after admission and nothing lowers the running count. This is accepted, not measured: peak pricing is what held a second chat's subagents behind the first's while memory was free. Growth is still answered by every later start re-reading live memory, by the spawn gate cutting and pausing backend starts at the critical line, and on macOS by the kernel memory-pressure hold. An operator with build-heavy fan-outs pins
agent.max_subagentsor raisesagent.spawn_min_memory_gb. Recorded insubagent.md(Memory guard) and in the RFC's Q11 residual.Specs updated in the same commit:
rfc-overload-resilience.md(v7): §5.2's execution-cap column and decision Q11 record the start-at-ceiling cap cut only on work evidence, with lag and memory left to the spawn gate (owner direction, the fix for subagent: admission and queueing redesign (memory floor, bounded wait, accurate queued state) #16480 symptom b);adaptive-concurrency.md,subagent.md(the Windows residual Windows: subagent spawn floor reads physical memory, not commit headroom #16983, the slot held during an approval wait under an explicit N, settled rather than peak pricing, and the live-config table now listing everySIZING_CONFIG_PATHSkey),dynamic-subagent-sizing.md;config.md,taskq.md,taskrunner.md,mcp.md,resource-protection.md,subagents.md,configuration.md.Implements
rfc-overload-resilience.mddecision Q11 (unit U4 of the #16480 subagent admission redesign). It depends on U3 (fix(subagent): queue every capacity verdict and read memory off the loop), already on main via #16355, and is not stacked on another branch. U5 (b) builds on it.Backwards compatibility
Compatible: every key main accepts is still accepted and round-trips.
max_subagents=0still means auto; an explicit N still caps at N (at least 3). Withagent.spawn_min_memory_gbat 0 or below the auto cap is main's memory-sized figure, unchanged.The diff takes away the
adaptive_cap_zerowait reason, the memory sizing of the auto subagent cap, thedeprecatedflag onagent.adaptive_initial, and the controller's use ofadaptive_initial:Reader: src/kiro_crew/subagent_wait_reasons.py:QUEUED_REASON_ADAPTIVE_CAP_ZERO -- subagent -- constant and its text removed; a 0 cap is an ordinary
concurrency_limitwait (test_spawn_admission_verdict_census.py::test_every_capacity_condition_queues_and_only_policy_refuses)Reader: src/kiro_crew/subagent_manager/admission/gate.py (both sites) -- subagent -- now report
concurrency_limit(same test)Reader: src/kiro_crew/subagent_manager/admission/taskq_bridge.py -- subagent -- same (same test)
Reader: website/src/pages/chat/subagentQueuedReason.ts -- chat -- case and catalog key removed; any other reason falls to the default queued line (ActivityViewer.test.tsx, subagentActivity.test.ts)
Reader: src/kiro_crew/subagent.py:compute_max_subagents -- chat/subagent -- with the floor disabled the auto cap keeps main's memory sizing (test_a_disabled_floor_sizes_the_cap_from_host_memory, test_disabling_the_spawn_floor_recomputes_the_cap, test_pool_size_change_recomputes_the_cap)
Reader: src/kiro_crew/taskrunner.py:auto max_parallel_steps -- app/task runner -- keeps memory sizing through compute_memory_sized_parallel_cap (test_the_taskrunner_auto_value_is_never_zero)
Reader: src/kiro_crew/dashboard/handlers_system.py:subagent_cap -- crew page -- reads resolve_max_subagents (test_resource_health_endpoint.py)
Reader: src/kiro_crew/config/service_sections.py -- release/config -- help text only
Reader: src/kiro_crew/adaptive/controller.py:watched keys -- chat/subagent -- no longer watches agent.adaptive_initial; the ceiling keys still hot-reload (test_adaptive_controller.py)
Reader: src/kiro_crew/adaptive/policy.py:params_from_config -- chat/subagent -- no longer reads adaptive_initial; exec_initial defaults to None, which means start at the ceiling (test_adaptive_policy.py)
Reader: src/kiro_crew/doctor_checks/workload.py -- release (doctor) -- drops the
initial=fieldReader: src/kiro_crew/config/loader.py:adaptive_initial -- config -- still loaded, clamped and saved unchanged
Tests
New
test/test_subagent_memory_bounds_concurrency.py, one test per acceptance criterion:max_subagents=0, 20 spawns on a host simulated at 40 GiB with the real controller attached: all 20 start, paced only by the stagger.subagent_auto_maxon any host, and 3 when memory cannot be read.max_subagents=3still caps (3 run, 2 queue). A yielded slot goes to the waiting parent's child, not to a queued root.test_the_daemon_is_launched_with_the_subagent_ceiling_at_bootdrives_init_mcp_gatewaywith no subagent manager, as on boot. It asserts that the GatewaySpec carries 32 and that the controller'sgate_ceilingequals it. It fails on the previous revision (assert 8 == 32).test_the_boot_path_sizes_the_gate_without_probing_host_memory:_init_mcp_gatewayand the controller wiring size the gate with_available_memory_gbnever called. It fails on the previous revision (called 2 times).test_a_disabled_floor_sizes_the_cap_from_host_memory(0 and -1) andtest_disabling_the_spawn_floor_recomputes_the_cap: with the floor off, 8 GB at 0.5 GB per agent gives 12, not 32, and a live switch re-resolves the cap. Both fail on the previous revision.test_the_inert_adaptive_initial_does_not_warn_on_every_launch: a default save validates with no deprecation warning. It fails on the previous revision.test_a_step_granted_after_a_lane_wait_rechecks_memory_pressure(test_taskq_runner_adapter.py): a step parked behind a full lane whose grant lands while memory is critical defers instead of claiming. It fails on the previous revision.test_the_count_ceiling_keeps_an_explicit_pin_floored_at_three(50 gives 50; 2 and 1 give 3) andtest_the_count_ceiling_is_the_auto_ceiling_when_unpinnedpinsubagent_count_ceilingdirectly, with any host memory read failing the test.Updated:
TasksCapacityCard.test.tsx, which renders the real catalog, so it fails on the old pause/probe copy.Local gate:
mypy src/kiro_crew, check_black_formatting, check_subprocess_encoding, check_harness_parity and docs-lint all pass.scripts/local-gate.pyrelated shards pass, except 5 tests intest_stt_transcribe_vocabulary.py. Those fail identically on main becausebotocoreis missing from the local venv.test_taskq_runner_adapter.py(the new test fails without the fix), i18n:check and the i18n style tests against origin/main,TasksCapacityCard.test.tsx, andscripts/local-gate.pyon this round's delta.Manual verification
N/A: unit and wiring tests cover each criterion, including the boot-order daemon launch. Per the harness safety rules, no gateway was started.
Screenshots / video
The Tasks Capacity card with the new degrade copy, captured from the real built SPA with the
website/scripts/capture-tasks-capacity.mjsharness (stubbed/api/tasks/summary, the spawn gate at its floor of 1 as a paused gate is).Spawn gate paused (
adaptive_pause), both lanes lowered, so the reason lands in the Degrade reason row: "Backend starts held to the minimum until this computer frees up memory or load eases".Spawn gate probing (
adaptive_probe): "Backend starts: testing one before raising the limit".Only the Backend starts lane lowered, Subagent runs at the full limit: the same line sits under the lane it explains.
The Degrade reason tip, opened: the two-track split.
Related Issues
Part of #16480. Addresses #14013 (the auto cap no longer prices slots from memory). That issue is left for the triager to close because it carries no
triagedlabel. Follow-up: #16983 (Windows commit-charge residual).Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement