Skip to content

refactor(subagent): let memory, not a count cap, bound concurrency - #17017

Merged
chenmingwei23 merged 1 commit into
mainfrom
refactor/subagent-memory-bounds-concurrency
Oct 6, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
refactor/subagent-memory-bounds-concurrency

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Free host memory, not a count cap, decides how many subagents run at once, so one chat's subagents stop throttling another's while memory is still free.

With more than two chats using subagents, starts queued behind each other although the host had plenty of free memory. Three count bounds sat under the per-start memory floor (agent.spawn_min_memory_gb), and each one bound first:

  • the auto cap (max_subagents=0) was sized from host memory, and priced every slot at the heaviest agent's p90 peak (Auto subagent cap priced every slot at the heaviest agent's p90 peak #14013);
  • the adaptive execution cap started at 4 and was halved by 250 ms of gateway event-loop lag or by low memory alone;
  • the MCP gateway spawn gate topped out at 8 backend initializations, so a wide fan-out queued behind eight windows.

Why it matters

Anyone who fans out work across several chats or a wide subagent tree gets serialized starts and long queues on a host with tens of GiB free. Waits show as concurrency_limit and look like a capacity problem, but no resource is actually short. A loop-lag blip (a GC pause, a large file read) could halve the cap for every chat at once.

Not a goal

What changed (motivation → approach → change)

Symptom: starts queued under a count while memory was free. Root cause: three count bounds each bound before the memory floor did. The change turns each of them into a high ceiling or a work-evidence rule, and leaves memory to the per-start floor.

C2: count ceiling.

  • compute_max_subagents returns agent.subagent_auto_max (32), never below 3. It falls back to 3 only when host memory cannot be read, because the floor then admits everything.
  • A disabled floor (agent.spawn_min_memory_gb <= 0) admits everything too, so there the auto cap stays sized from host memory (compute_memory_sized_parallel_cap, main's figure), and the floor switch plus that figure's inputs (subagent_mem_buffer_pct, subagent_cost_gb, session.pool_size) stay live sizing paths on the manager.
  • The memory arithmetic moved to compute_memory_sized_parallel_cap, which only the TaskRunner's auto value uses: no per-start floor prices a TaskRunner step.
  • An explicit max_subagents=N is unchanged.

C3: adaptive controller split into two tracks.

  • The execution track (policy._exec_step) reads work signals only: timeouts, slow starts, failing inits, and fd or process exhaustion. It needs two of them to cut, it never pauses, and it starts at its ceiling. A raised ceiling is followed at once if the cap sat at the old one.
  • Loop lag and memory (signals.HOST_ONLY_SIGNALS) shape only the spawn-gate track, which keeps the old rules: either alone can cut it, and severe pressure pauses or probes it.
  • agent.adaptive_initial is inert. It is deliberately not flagged deprecated: every save writes it, so a deprecation notice would fire on every launch.

C4: spawn gate ceiling.

  • derive_spawn_gate_ceiling returns max(mcp_gateway.spawn_concurrency_max, subagent ceiling).
  • The broker's daemon command line and the controller's gate track both read the subagent ceiling from config alone (subagent_count_ceiling: the explicit pin floored at 3, else subagent_auto_max), never from the subagent manager and never through a host-memory probe. On boot the broker starts before _init_subagents builds the manager. A manager-derived figure therefore launched the daemon at 8 while the controller targeted 32, and the daemon clamped every raise back. The boot path before readiness takes no memory reading; the gate needs only the upper bound the cap can reach.
  • The gate still starts at spawn_concurrency_initial and climbs only on clean backend inits.

C1/F7: every consumer has a defined figure.

  • The {{MAX_SUBAGENTS}} prompt token and the spawn tool text show the ceiling. The tool text adds that each start also waits for host memory.
  • /api/system subagent_cap reports resolve_max_subagents, including an explicit pin.

B5 and copy.

  • The adaptive_cap_zero wait reason is gone (it is unreachable now, and cron never used it), together with its UI strings.
  • The Tasks Capacity card's pause and probe degrade copy leads with the lane it lowered, "Backend starts", the only thing that pauses now, and says "this computer" as the rest of the catalog does: "Backend starts held to the minimum until this computer frees up memory or load eases", "Backend starts: testing one before raising the limit". A paused gate still admits its floor, so the copy says "held to the minimum", which agrees with the lane's own "Up to 1 at once". The Degrade reason tip names the two-track split, as the code reads it (exec_signals, RFC Q11): Subagent runs are lowered only on evidence from the work itself (timeouts, slow or failing starts, running out of file handles or processes), Backend starts also by low memory or load. French uses U+202F before ";" and ":". Every translated catalog is updated and en-XA regenerated.
  • Help texts (including the disabled-floor case for subagent_mem_buffer_pct, subagent_cost_gb and spawn_min_memory_gb) and config-baseline.json are updated.

TaskRunner steps re-check memory at a parked grant. The runner lane's limit is the execution cap, which no longer pauses on memory, so a step that waited behind a full lane would start on whatever memory read at the grant: its pressure check predated the wait. RunnerAdmission.admit now re-runs that check when the step had to park, and a refusal gives the slot back and defers again (taskq.md).

Accepted decision: settled, not peak, start pricing. The old auto cap priced every slot at the heaviest agent's p90 whole-run peak, so growth after admission was bounded by the count. The floor prices a start at its settled size, so a wave of build-heavy children can grow past the floor after admission and nothing lowers the running count. This is accepted, not measured: peak pricing is what held a second chat's subagents behind the first's while memory was free. Growth is still answered by every later start re-reading live memory, by the spawn gate cutting and pausing backend starts at the critical line, and on macOS by the kernel memory-pressure hold. An operator with build-heavy fan-outs pins agent.max_subagents or raises agent.spawn_min_memory_gb. Recorded in subagent.md (Memory guard) and in the RFC's Q11 residual.

Specs updated in the same commit:

Implements rfc-overload-resilience.md decision Q11 (unit U4 of the #16480 subagent admission redesign). It depends on U3 (fix(subagent): queue every capacity verdict and read memory off the loop), already on main via #16355, and is not stacked on another branch. U5 (b) builds on it.

Backwards compatibility

Compatible: every key main accepts is still accepted and round-trips. max_subagents=0 still means auto; an explicit N still caps at N (at least 3). With agent.spawn_min_memory_gb at 0 or below the auto cap is main's memory-sized figure, unchanged.

The diff takes away the adaptive_cap_zero wait reason, the memory sizing of the auto subagent cap, the deprecated flag on agent.adaptive_initial, and the controller's use of adaptive_initial:
Reader: src/kiro_crew/subagent_wait_reasons.py:QUEUED_REASON_ADAPTIVE_CAP_ZERO -- subagent -- constant and its text removed; a 0 cap is an ordinary concurrency_limit wait (test_spawn_admission_verdict_census.py::test_every_capacity_condition_queues_and_only_policy_refuses)
Reader: src/kiro_crew/subagent_manager/admission/gate.py (both sites) -- subagent -- now report concurrency_limit (same test)
Reader: src/kiro_crew/subagent_manager/admission/taskq_bridge.py -- subagent -- same (same test)
Reader: website/src/pages/chat/subagentQueuedReason.ts -- chat -- case and catalog key removed; any other reason falls to the default queued line (ActivityViewer.test.tsx, subagentActivity.test.ts)
Reader: src/kiro_crew/subagent.py:compute_max_subagents -- chat/subagent -- with the floor disabled the auto cap keeps main's memory sizing (test_a_disabled_floor_sizes_the_cap_from_host_memory, test_disabling_the_spawn_floor_recomputes_the_cap, test_pool_size_change_recomputes_the_cap)
Reader: src/kiro_crew/taskrunner.py:auto max_parallel_steps -- app/task runner -- keeps memory sizing through compute_memory_sized_parallel_cap (test_the_taskrunner_auto_value_is_never_zero)
Reader: src/kiro_crew/dashboard/handlers_system.py:subagent_cap -- crew page -- reads resolve_max_subagents (test_resource_health_endpoint.py)
Reader: src/kiro_crew/config/service_sections.py -- release/config -- help text only
Reader: src/kiro_crew/adaptive/controller.py:watched keys -- chat/subagent -- no longer watches agent.adaptive_initial; the ceiling keys still hot-reload (test_adaptive_controller.py)
Reader: src/kiro_crew/adaptive/policy.py:params_from_config -- chat/subagent -- no longer reads adaptive_initial; exec_initial defaults to None, which means start at the ceiling (test_adaptive_policy.py)
Reader: src/kiro_crew/doctor_checks/workload.py -- release (doctor) -- drops the initial= field
Reader: src/kiro_crew/config/loader.py:adaptive_initial -- config -- still loaded, clamped and saved unchanged

Tests

New test/test_subagent_memory_bounds_concurrency.py, one test per acceptance criterion:

  • (i) max_subagents=0, 20 spawns on a host simulated at 40 GiB with the real controller attached: all 20 start, paced only by the stagger.
  • (C2) The auto ceiling is subagent_auto_max on any host, and 3 when memory cannot be read.
  • (ii) 3 s of loop lag leaves the exec cap unchanged, at the policy level and through the real controller. Memory at the critical line leaves it unchanged too, while the gate still backs off.
  • (iii) An explicit max_subagents=3 still caps (3 run, 2 queue). A yielded slot goes to the waiting parent's child, not to a queued root.
  • (iv/C4) The derived ceiling reaches 32; a SpawnGate built with it runs 20 inits at once; the gate track climbs past 8 on clean evidence; HostBudget admits 20 x 5 backends at 40 GiB.
  • (iv/C4, boot wiring) test_the_daemon_is_launched_with_the_subagent_ceiling_at_boot drives _init_mcp_gateway with no subagent manager, as on boot. It asserts that the GatewaySpec carries 32 and that the controller's gate_ceiling equals it. It fails on the previous revision (assert 8 == 32).
  • (v) Timeouts on two servers and fd plus process exhaustion still cut the exec cap; a provider 429 never does.
  • (vi) The prompt figure and the spawn tool text name the ceiling and the memory wait.
  • (vii) The TaskRunner auto value is never 0.
  • test_the_boot_path_sizes_the_gate_without_probing_host_memory: _init_mcp_gateway and the controller wiring size the gate with _available_memory_gb never called. It fails on the previous revision (called 2 times).
  • test_a_disabled_floor_sizes_the_cap_from_host_memory (0 and -1) and test_disabling_the_spawn_floor_recomputes_the_cap: with the floor off, 8 GB at 0.5 GB per agent gives 12, not 32, and a live switch re-resolves the cap. Both fail on the previous revision.
  • test_the_inert_adaptive_initial_does_not_warn_on_every_launch: a default save validates with no deprecation warning. It fails on the previous revision.
  • test_a_step_granted_after_a_lane_wait_rechecks_memory_pressure (test_taskq_runner_adapter.py): a step parked behind a full lane whose grant lands while memory is critical defers instead of claiming. It fails on the previous revision.
  • test_the_count_ceiling_keeps_an_explicit_pin_floored_at_three (50 gives 50; 2 and 1 give 3) and test_the_count_ceiling_is_the_auto_ceiling_when_unpinned pin subagent_count_ceiling directly, with any host memory read failing the test.

Updated:

  • the adaptive policy and controller, sizing, census, queued-reason, taskrunner and startup-memory tests;
  • TasksCapacityCard.test.tsx, which renders the real catalog, so it fails on the old pause/probe copy.

Local gate:

  • black (touched files), isort, flake8, mypy src/kiro_crew, check_black_formatting, check_subprocess_encoding, check_harness_parity and docs-lint all pass.
  • Website: i18n:check, lint:i18n, tsc and the i18n catalog tests pass.
  • scripts/local-gate.py related shards pass, except 5 tests in test_stt_transcribe_vocabulary.py. Those fail identically on main because botocore is missing from the local venv.
  • This round (the runner-lane re-check, the pause/probe copy and the Degrade reason tip): the gates above plus check_comment_history, test_config_baseline, test_taskq_runner_adapter.py (the new test fails without the fix), i18n:check and the i18n style tests against origin/main, TasksCapacityCard.test.tsx, and scripts/local-gate.py on this round's delta.

Manual verification

N/A: unit and wiring tests cover each criterion, including the boot-order daemon launch. Per the harness safety rules, no gateway was started.

Screenshots / video

The Tasks Capacity card with the new degrade copy, captured from the real built SPA with the website/scripts/capture-tasks-capacity.mjs harness (stubbed /api/tasks/summary, the spawn gate at its floor of 1 as a paused gate is).

Spawn gate paused (adaptive_pause), both lanes lowered, so the reason lands in the Degrade reason row: "Backend starts held to the minimum until this computer frees up memory or load eases".

Tasks Capacity card, spawn gate paused, reason in the Degrade reason row

Spawn gate probing (adaptive_probe): "Backend starts: testing one before raising the limit".

Tasks Capacity card, spawn gate probing

Only the Backend starts lane lowered, Subagent runs at the full limit: the same line sits under the lane it explains.

Tasks Capacity card, only Backend starts lowered, the line under that lane

The Degrade reason tip, opened: the two-track split.

Tasks Capacity card, Degrade reason tip open

Related Issues

Part of #16480. Addresses #14013 (the auto cap no longer prices slots from memory). That issue is left for the triager to close because it carries no triaged label. Follow-up: #16983 (Windows commit-charge residual).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@bolichen97
bolichen97 requested a review from a team as a code owner October 5, 2026 11:53
@bolichen97
bolichen97 requested a review from CrysisDeu October 5, 2026 11:54
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 5f0dd2568f78536a16a02dcdaa269bd2f8167e87 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Nothing limits the running count after admission any more: build-heavy children admitted at settled size can grow past free memory, and no control lowers the count.

Watch

  • Settled-size pricing has no backstop on running work. The description says: "a wave of build-heavy children can grow past the floor after admission and nothing lowers the running count." The chain:

    • compute_max_subagents now returns 32, and the execution track ignores memory (HOST_ONLY_SIGNALS reach only the gate).
    • The floor reads memory once, at each start, so many children can pass it at their small settled size.
    • Those children then grow together. The spawn gate only holds back new backend starts, so on a small Linux or Windows host (no macOS pressure hold) the result is swap or the OOM killer, not a queue.
    • The only remedy is a manual max_subagents pin, which the operator sets after the first OOM.

    The description says this is "accepted, not measured".
    Clears when: a measured fan-out run (e.g. 20 build-heavy children on a 16 GiB Linux host) shows the gate and later floors keep the host out of OOM, or the execution track regains a critical-memory cut (cut only, never pause).

  • The compatibility check is capped. ## Backwards compatibility is the last section before the [description TRUNCATED notice, so its reader list could not be checked in full. In this tree, the removed adaptive_cap_zero reason has two readers: subagentQueuedReason.ts (falls back to the default text) and test_queue_depth_reconcile.py (treats it as an opaque string). Neither breaks.
    Clears when: the description fits within 8000 bytes, or the Backwards compatibility section moves above the cut.

[DESIGN-REVIEWED] 5f0dd25

Description read

[DESCRIPTION-READ] bd0f3596577609c2bc7437650f08246243fb35e2f201abdd22f6b1dbc5ebda3c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — 🟡 CONCERNS

UX-level review of 5f0dd2568f78536a16a02dcdaa269bd2f8167e87 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: CONCERNS

The degrade reason names only "Backend starts", so when Subagent runs are also lowered, nothing on the card says why.

Watch

  • Subagent cut has no stated reason. In shot-01, Subagent runs show "lowered from your 14" while the Degrade reason row reads only "Backend starts held to the minimum…". The blind reader said it "names only backend starts, though, and doesn't say why subagent runs were lowered." The new pause copy is scoped to one lane, so the other lane's cut is left unexplained. This is moderate: it shows up every time both lanes are degraded, which is the usual case under pressure. Fix: when the execution lane is below its ceiling, give it its own reason line under "Subagent runs".
  • "Concurrency lowered under load" now contradicts the new tip. This degrade_adaptive_decrease string is unchanged. The rewritten degrade_reason_tip says Subagent runs are lowered "only on evidence from the work itself", but this string still blames "load". A user who reads both gets two opposite causes. Fix: change it to name the evidence, e.g. "Subagent runs lowered after timeouts or failing starts".
  • The probe copy doesn't say the limit is still held. In shot-02, "Backend starts: testing one before raising the limit" sits next to an unchanged "Up to 1 at once, lowered from your 32". The reader "can't tell whether the limit is about to go up or is still held down." This is low severity: the probe state is short-lived. Fix: "Backend starts held at 1 while one test start runs".

Suggestions

  • degrade_reason_tip: cut the parenthetical list ("file handles or processes") down to "such as timeouts or failing starts". The reader found the tip "wordy and technical", and in shot-04 it covers the numbers it explains.

[UX-REVIEWED] 5f0dd25

Description read

[DESCRIPTION-READ] d286c9f69db46ff0101eb275605b5ec4cf65e9987dfbb065177153b4aaebf562

That sha256 names everything this verdict read: the PR description, plus the 7 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ human override accepted

@bolichen97 overrode this lane for 5f0dd2568f78536a16a02dcdaa269bd2f8167e87 via /ai-review override (this commit only).

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 5f0dd2568f78536a16a02dcdaa269bd2f8167e87 — this comment is updated in place on each push.

Review details

No findings block merge; one advisory finding on the runner lane's memory bound.

FINDING — src/kiro_crew/adaptive/policy.py:203 — the runner lane's ceiling is the manager's max_concurrent (runner_admission_for → RunnerLane(_cap)), and that now starts at initial = self.exec_ceiling (32 by default under auto) and never reacts to memory. Workflow ctx.agent() calls and TaskRunner steps run on this lane through the session pool, not through the subagent spawn floor. So several concurrent workflow runs (4 calls each, _wf_concurrency = 4) or TaskRunner runs (each with a cap sized from the same free memory) start together where the lane used to hold them at 4 and cut them under memory pressure. The only remaining guard is cached_admission_check, which refuses only at CRITICAL posture and serves one cached verdict per TTL window, so a burst of starts can overcommit host memory. → Fix: give runner-lane admissions the per-start spawn_min_memory_gb pricing that subagent starts get, or keep a memory term on the exec track for the lane's share. Either needs machinery outside the changed lines.

[OPUS-REVIEWED] 5f0dd25

Verdict parsed from the review's SHA-scoped output markers for commit 5f0dd2568f78536a16a02dcdaa269bd2f8167e87.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5f0dd2568f78536a16a02dcdaa269bd2f8167e87: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 5f0dd2568f78536a16a02dcdaa269bd2f8167e87 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5f0dd25

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5f0dd2568f78536a16a02dcdaa269bd2f8167e87: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the refactor/subagent-memory-bounds-concurrency branch from d835d50 to b954628 Compare October 5, 2026 12:51
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 5, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Fixed in b954628: a disabled floor keeps the memory-sized auto cap. span=2481be1ed4a8

compute_max_subagents now returns compute_memory_sized_parallel_cap(cfg) (main's figure) when agent.spawn_min_memory_gb <= 0, and the floor switch plus that figure's inputs (subagent_mem_buffer_pct, subagent_cost_gb, session.pool_size) are back in LIVE_CONFIG_PATHS / SIZING_CONFIG_PATHS, so a live switch re-resolves the cap. Tests: test_a_disabled_floor_sizes_the_cap_from_host_memory (0 and -1: 8 GB at 0.5 GB gives 12, not 32), test_disabling_the_spawn_floor_recomputes_the_cap, test_pool_size_change_recomputes_the_cap; all fail on d835d50. Specs, help text and the description's compatibility section updated.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Kept "backend start": it is the card's own lane label. span=2f3fc5792d92

The screenshots now attached to the description show the same card's capacity column labels its second lane "Backend starts" ("Up to 2 at once, lowered from your 8"), directly above the Degrade reason row. The pause and probe copy names that same unit, so a reader matches the reason to the lane it lowered. "Agent starts" would be wrong: the spawn gate holds MCP backend initializations at its floor, not agent starts, and the "Subagent runs" lane next to it is not paused (the execution cap never pauses in this PR).

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Suggestion not taken: "New agent starts paused" would misstate what pauses. span=2cd7a3b48433

The pause holds the spawn gate (MCP backend initializations) at its floor. Agent runs are bounded by the separate "Subagent runs" lane, which this PR makes un-pausable. Naming agents would tell the user their subagents are stopped when they are not. The current copy matches the "Backend starts" lane label on the same card (see the attached screenshots). The missing-screenshot evidence gap is closed: both states are now attached to the description.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Fixed in b954628: the workflow bounded_limit ride-along is deleted. span=2a86dbbdf153

dsl.bounded_limit and dsl.DEFAULT_AGENT_CONCURRENCY are removed, and runner.py is back to main's asyncio.Semaphore(concurrency) if (concurrency and concurrency > 0) else None. Only the stale docstrings that named resolve_max_subagents() as the runner's prod limit are corrected (the dashboard passes its fixed workflow cap). The test and the workflows.md paragraph for it are removed, and the description no longer claims it.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Fixed in b954628, with the deletion above: no non-positive caller to guard. span=cc2beb403fa9

Agreed: workflows/service.py:840 gets the dashboard's fixed _wf_concurrency = 4, and service.py:793 already falls back to 4. The bounding code is deleted rather than kept as a second copy.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Fixed in b954628, with the deletion above: the None caller keeps main's behaviour. span=9286754bb979

apps/builtins/workflows/server.py:112 passes no concurrency, so None (no limit) as on main. With bounded_limit gone, nothing about that path changes in this PR.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Fixed in b954628: the boot path sizes the gate from config alone. span=2fa8fbccf3f0

New subagent.subagent_count_ceiling(cfg): the explicit pin floored at 3, else max(3, subagent_auto_max), an O(1) config read with no host-memory probe. _init_mcp_gateway (mcp_broker.py) and _start_adaptive_controller (gateway_runtime/admission.py) both use it, so the daemon's clamp and the controller's gate ceiling still agree. test_the_boot_path_sizes_the_gate_without_probing_host_memory patches _available_memory_gb and asserts it is never called across both wiring paths; it fails on d835d50 (called 2 times).

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Fixed in b954628: no memory probe on the loop at either wiring site. span=8933962e632c

The probe is removed rather than moved to a worker thread. Both sites (mcp_broker.py and gateway_runtime/admission.py) now call subagent_count_ceiling(cfg), a config-only upper bound with no /proc, cgroup or Mach read. The gate needs only the bound the cap can reach. test_the_boot_path_sizes_the_gate_without_probing_host_memory asserts _available_memory_gb is never called; it fails on d835d50.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Rebutted: the symptom is tracked in an open issue. span=da41e1ab3138

#16480 item 2 is this symptom, reported by the owner: "More than two subagents throttle each other. Hidden count caps and an adaptive controller that reacts to event-loop lag limit concurrency well below what free memory allows, and one chat's wave can starve another chat's starts." The PR is "Part of #16480" (unit U4 of that redesign). #14013 covers the memory-priced auto cap. On the title: the change ships new defaults, so the description names them as behaviour changes in "What changed" and "Backwards compatibility". The conventional-commit type does not change what the review reads.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Rebutted: start-at-ceiling is the owner-directed redesign, not a ride-along. span=c7740d030cf1

#16480 item 2 names "an adaptive controller that reacts to event-loop lag" and limits "well below what free memory allows" as the defect. Starting the exec track at 4 is one of the count bounds that produced it. Memory is bounded per start by the floor (#16355, merged), so the exec cap does not have to earn its way up to protect memory. rfc-overload-resilience.md is partial and predates the per-start floor. The owner accepted this direction under #16480; the RFC text is a maintainer decision (override requested).

@bolichen97

Copy link
Copy Markdown
Contributor Author

A 250 ms lag blip (a GC pause, a large read) halved the cap for every chat at once, which is exactly the "reacts to event-loop lag" defect in #16480. Lag and memory still cut, pause and probe the spawn gate (unchanged thresholds), and work evidence (timeouts, slow starts, failing inits, fd/process exhaustion) still cuts the exec cap. The deleted tests pinned the old exec-track rule; the new ones pin the split (test_subagent_memory_bounds_concurrency.py (ii) and (v), test_adaptive_policy.py).

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Escalated: the RFC-vs-owner-decision question goes to a maintainer. span=96c4151e980d

The new shape (exec track starts at its ceiling, cut only on work evidence; lag and memory shape the spawn gate only) is the owner-directed fix for #16480 item 2, built on the per-start memory floor that #16355 merged. rfc-overload-resilience.md (partial) predates that floor. A maintainer decides between amending the RFC on main first and a /ai-review override first-principles on this head. This PR does not edit the RFC itself, because the lane only accepts an RFC merged on base.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 5, 2026
@bolichen97
bolichen97 force-pushed the refactor/subagent-memory-bounds-concurrency branch from b954628 to 87f88b7 Compare October 5, 2026 13:51
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Fixed in 87f88b7: the RFC now records the shape. span=96c4151e980d (items c7740d030cf1, 7ab3363c2b67)

docs/request-for-change/rfc-overload-resilience.md (v7, status partial) is amended in the same commit. §5.2's ExecutionCap column now reads user_max / 1 / user_max for initial / floor / ceiling, a cut on two or more distinct work signals, no lag or memory hysteresis, and no pause or probe. The fresh-start sentence says the cap begins at user_max and is cut only on work evidence. New decision Q11 (2026-10-05, accepted) records the owner-directed fix for #16480 symptom b. The subagent exec track starts at its ceiling and is cut only on work evidence: attributable timeouts, slow or failing starts, and fd or process exhaustion, with provider 429s scoped to their own lane share. Loop lag alone and memory do not cut the subagent cap: the floor bounds memory per start (Q9, #16355), and lag and memory still cut or pause the spawn gate. Q2's fixed reversal carries an "Amended by Q11" note. The grounds, the residual and the reversal flags are written in the RFC's own Q-entry format. The "mainline-plan unit U4" citation in the description now points at Q11.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Recorded as an accepted decision, with its reason: settled, not peak, start pricing. span=8bd04f1a0b40

Fixed in 87f88b7. subagent.md Memory guard now has an Accepted decision: starts are priced settled, not at peak paragraph, and the RFC's Q11 carries the same residual. Before, read_learned_cost took the max per-agent p90 of the whole-run peak, so the count bounded growth after admission. Now the floor prices each start at its settled size (capped at _SETTLED_PROJECTION_CEILING_GB). A wave of build-heavy children can grow past the floor after admission, and nothing lowers the running count. This is accepted, not measured. Peak pricing is what held a second chat's subagents behind the first chat's while memory was free (#16480 symptom b), and §5.2 already records that a burst-priced prediction pins a 96 GB host at its fresh-start cap. Growth is still answered three ways: every later start re-reads live free memory, the spawn gate cuts and then pauses backend starts at the critical line, and on macOS the kernel pressure hold vetoes starts at WARN. An operator with build-heavy fan-outs pins agent.max_subagents or raises agent.spawn_min_memory_gb. The PR description records the move under "Accepted decision".

@bolichen97

bolichen97 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Accepted residual, not fixed in this unit: runner-lane starts get no per-start memory price. src/kiro_crew/adaptive/policy.py:203 (advisory: the runner lane's ceiling starts at the exec ceiling and never reacts to memory) span=cf9f0f2ca649

The finding is accurate about the shape. The lane's ceiling is the manager's max_concurrent, which now starts at exec_ceiling, and nothing on the execution track cuts it on memory (RFC Q11). Three bounds still apply to a runner-lane burst:

  • Per run: A TaskRunner run stays capped by compute_memory_sized_parallel_cap, which is sized from free memory (taskrunner.md; subagent.md says why no per-start floor prices a step). A workflow run is held to the fixed 4 agent calls (workflows.md).
  • At the grant: RunnerAdmission.admit re-checks memory pressure when a parked step is granted. While the host is critical, the step hands its slot back and defers (fixed in 881f1df and 4fd9dc3, pinned in test_taskq_runner_adapter.py).
  • At backend start: The spawn gate still lowers and pauses on loop lag and memory, and it holds backend initializations at its floor below the critical line. That covers the per-session MCP servers of every lane, including this one.

What is missing is the per-start price for a runner-lane start, which the subagent spawn floor gives to subagent starts. Putting a memory term back on the exec track would bring back the count-under-free-memory throttle that #16480 symptom b removes, so it is not the fix. Pricing runner-lane admissions with spawn_min_memory_gb needs a projection per lane kind and a settle-tracking row for steps and workflow calls. As the finding says, that machinery lies outside the changed lines, and it is a separate unit under #16480, not part of U4's concurrency bound. Until it lands, an operator who runs many concurrent TaskRunner or workflow runs on a small host can pin agent.max_subagents, which bounds this lane's ceiling too.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Not shortened; repeated from 881f1df. Watch item: the Backwards compatibility section is cut off span=5d0aa60219d1

The description's ## Backwards compatibility section has its full reader list (12 Reader: lines, from subagent_wait_reasons.py to config/loader.py, each naming its test). The cut-off is the review's 8000-byte input window, not the description. The lane's own checks (adaptive_cap_zero, adaptive_pause_text, QUEUED_REASON_ADAPTIVE_CAP_ZERO, SUFFICIENT_ALONE) found no reader that breaks. On the skipped item: agent.adaptive_initial is read by no app or cron code; it stays a valid config key, so a saved config still loads.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Kept: the row form names its lane because the row is not under one. Watch item: the lane name repeats when the sentence sits under its lane (shot 3) span=6b1e95fbdb25

The pause and probe strings lead with "Backend starts" so that, in the global Degrade reason row, the reader can tell which lane the reason explains (the fix for the round-two finding on 87f88b7). A lane-free variant under the lane would need a second catalog key per reason and a selection in the placement code. The repeat is redundant, not misleading, and the lane's own line still carries the number. This round's single push was reserved for the runner-lane defect.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Kept; repeated from 881f1df. Watch item: probe copy does not say the lane is recovering span=3bc197d1774a

"Backend starts: testing one before raising the limit" names what a probe does: it admits one backend start, and a clean start raises the limit while a failed one keeps the floor. "Recovering" would promise the outcome before the probe has run.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Not taken; repeated from 881f1df. Suggestion: drop "file handles or processes" from degrade_reason_tip span=87306b547f4f

The execution track cuts on file-descriptor and process exhaustion as well as on timeouts and slow or failing starts (adaptive/signals.py, exec_signals = signals - HOST_ONLY_SIGNALS; RFC Q11). "Subagent runs drop only when runs time out or fail to start" would tell an operator whose runs were lowered by fd exhaustion to look for a timeout that does not exist. The tip is on-demand detail behind a glyph and the only place this list appears.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Not shortened; repeated from 881f1df. Watch item: the Backwards compatibility section is cut off span=d84ef51f90a0

The description's ## Backwards compatibility section has its full reader list (12 Reader: lines, from subagent_wait_reasons.py to config/loader.py, each naming its test). The cut-off is the review's 8000-byte input window, not the description. The lane's own checks (adaptive_cap_zero, adaptive_pause_text, QUEUED_REASON_ADAPTIVE_CAP_ZERO, SUFFICIENT_ALONE) found no reader that breaks. On the skipped item: agent.adaptive_initial is read by no app or cron code; it stays a valid config key, so a saved config still loads.

Repeated from 4fd9dc3 on the rebased head d28bb78. The rebase onto main changed no behavior of this PR: it resolved conflicts with #16852 (the memory-sized figure now prices a typical run and reserves the heaviest agent's p90 once), which this PR keeps for the TaskRunner auto value and the disabled-floor fallback, and updated the matching docstrings, dynamic-subagent-sizing.md and one sentence of subagent.md.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Kept: the row form names its lane because the row is not under one. Watch item: the lane name repeats when the sentence sits under its lane (shot 3) span=6b1e95fbdb25

The pause and probe strings lead with "Backend starts" so that, in the global Degrade reason row, the reader can tell which lane the reason explains (the fix for the round-two finding on 87f88b7). A lane-free variant under the lane would need a second catalog key per reason and a selection in the placement code. The repeat is redundant, not misleading, and the lane's own line still carries the number. This round's single push was reserved for the runner-lane defect.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Kept; repeated from 881f1df. Watch item: probe copy does not say the lane is recovering span=3bc197d1774a

"Backend starts: testing one before raising the limit" names what a probe does: it admits one backend start, and a clean start raises the limit while a failed one keeps the floor. "Recovering" would promise the outcome before the probe has run.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Not taken; repeated from 881f1df. Suggestion: drop "file handles or processes" from degrade_reason_tip span=87306b547f4f

The execution track cuts on file-descriptor and process exhaustion as well as on timeouts and slow or failing starts (adaptive/signals.py, exec_signals = signals - HOST_ONLY_SIGNALS; RFC Q11). "Subagent runs drop only when runs time out or fail to start" would tell an operator whose runs were lowered by fd exhaustion to look for a timeout that does not exist. The tip is on-demand detail behind a glyph and the only place this list appears.

Repeated from 4fd9dc3 on the rebased head d28bb78. The rebase onto main changed no behavior of this PR: it resolved conflicts with #16852 (the memory-sized figure now prices a typical run and reserves the heaviest agent's p90 once), which this PR keeps for the TaskRunner auto value and the disabled-floor fallback, and updated the matching docstrings, dynamic-subagent-sizing.md and one sentence of subagent.md.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Suggestion not taken: "Starting new agents is slowed until memory or load eases". Watch item: the pause line wraps over six lines (shot-01) span=79bb550032b8

That wording says agent starts are slowed, which is the misstatement described in the "Backend starts" disposition on this head. The Subagent runs lane is not throttled by memory or load in this PR (the execution track ignores lag and memory, RFC Q11), so a reader told "new agents are slowed" would expect the wrong lane to drop. The wrap comes from the narrow value column of the Degrade reason row, which this PR does not change. The string is 14 words and states the floor ("held to the minimum") and both release conditions. On 87f88b7 the shorter "Paused" was replaced because it contradicted the lane's "Up to 1 at once". Losing either condition would be a regression in accuracy to fix a layout issue.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Not changed here; repeated from 4fd9dc3. Watch item: the reason can read as explaining the wrong lane span=a7f7975bf7bb

degrade_reason is a single global token, and the card's placement rule (causeAtLane = degradeText != null && loweredCount === 1) predates this PR. A per-lane cause needs a payload and card change. That is outside this unit, and the full reply is in the repeated disposition on this head.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Not taken; repeated from 881f1df. Watch item and suggestion: the tooltip's "file handles", "processes" and "lane" are jargon span=e9d3a5f585b3

The execution track cuts on file-descriptor and process exhaustion (adaptive/signals.py). "System limits" would not tell an operator which limit to raise, and this on-demand tip is the only place the list appears. "Lane" is the card's own name for its two rows. The tip text did not change on this head: the rebase onto main touched no catalog string, and en-XA regenerated with no change.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Not shortened; repeated from 881f1df. Watch item: the Backwards compatibility section is cut off span=b1be10021c1b

The description's ## Backwards compatibility section has its full reader list (12 Reader: lines, from subagent_wait_reasons.py to config/loader.py, each naming its test). The cut-off is the review's 8000-byte input window, not the description. The lane's own checks (adaptive_cap_zero, adaptive_pause_text, QUEUED_REASON_ADAPTIVE_CAP_ZERO, SUFFICIENT_ALONE) found no reader that breaks. On the skipped item: agent.adaptive_initial is read by no app or cron code; it stays a valid config key, so a saved config still loads.

Repeated from d28bb78 on head 5f0dd25. This push rebased onto main (picking up #17145) and changed only one test fixture and assertion in test/test_subagent_cost.py, two docstrings in subagent_cost.py and one sentence of dynamic-subagent-sizing.md. No behavior and no catalog string changed.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Kept; repeated from 881f1df. Watch item: probe copy does not say the lane is recovering span=5ba7e0fa7f9f

"Backend starts: testing one before raising the limit" names what a probe does: it admits one backend start, and a clean start raises the limit while a failed one keeps the floor. "Recovering" would promise the outcome before the probe has run.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Not taken; repeated from 881f1df. Suggestion: drop "file handles or processes" from degrade_reason_tip span=e9ca49f425ac

The execution track cuts on file-descriptor and process exhaustion as well as on timeouts and slow or failing starts (adaptive/signals.py, exec_signals = signals - HOST_ONLY_SIGNALS; RFC Q11). "Subagent runs drop only when runs time out or fail to start" would tell an operator whose runs were lowered by fd exhaustion to look for a timeout that does not exist. The tip is on-demand detail behind a glyph and the only place this list appears.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Not changed here: "Concurrency lowered under load" is the one token for a cut on either track, and this PR does not touch it. Watch item: degrade_adaptive_decrease contradicts the new tip span=5f0830eadf4b

_degrade_reason (slack/gateway_runtime/admission.py) maps the policy's last decrease action to adaptive_decrease, and that action is emitted for a spawn-gate cut (loop lag, memory) as well as an execution-track cut (work evidence). "Under load" is the common cause of both, so it does not contradict the tip, which says Backend starts are lowered when the computer is overloaded. "Subagent runs lowered after timeouts or failing starts" would be false for every gate-only cut. Naming the track needs a per-track reason token in the payload, the same payload and card change as the Subagent-runs cut disposition on this head, outside #16480 U4. The string is unchanged on main and in this PR.

Repeated from d28bb78 on head 5f0dd25. This push rebased onto main (picking up #17145) and changed only one test fixture and assertion in test/test_subagent_cost.py, two docstrings in subagent_cost.py and one sentence of dynamic-subagent-sizing.md. No behavior and no catalog string changed.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 6, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 6, 2026 04:41
@iamwhatever

Copy link
Copy Markdown
Contributor

Reviewed for merge triage, not approving from my side: ~2.6k added / ~1.3k removed lines across 68 files replacing the subagent count cap with a memory bound. That changes how many subagents may run at once on every host, a capacity/product decision for a maintainer. Auto-merge is already on, so a single approval merges it; please make sure a maintainer reviews it directly. CI is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants