Repository navigation
fix(subagent): cap concurrent startups, pressure-aware start deadline - #13762
Closed
bolichen97 wants to merge 2 commits into
Closed
bolichen97 wants to merge 2 commits into
bolichen97 wants to merge 2 commits into
Conversation
bolichen97
force-pushed
the
fix/subagent-startup-pressure
branch
from
September 25, 2026 09:16
9f8fde8 to
224056f
Compare
A fan-out was bounded by two quantities: subagent_spawn_stagger_secs bounds the RATE of starts and _max_concurrent bounds the RUNNING population. Nothing bounded how many admitted agents were IN STARTUP at once, the dedicated-process path charged its SessionStartGate queue time to the startup watchdog, and the watchdog reaped at a fixed 120s blind to the crowd. On a 623-item fan-out, waves of 24-45 lost ~2% and a wave of 120 lost ~50%, every loss a healthy start recorded as "Failed to start within 120s". Gate-exit reset on the dedicated path: _run_inner passes on_gate_acquired=_gate_exit_reset(info) into get_or_create; it rides the provider factory (a NAMED parameter in _acp) into AcpProvider and from there into the process's own create_session, so time parked at the gate is not charged as startup time on either path. One definition serves both; the shared path's inline closure becomes a call to it. Guard A: agent.subagent_max_concurrent_startups (0 = derive max(2 x session_start_concurrency, ceil(cap / 4)), never above the cap). _should_stagger_queue gains a third clause and the drain pump holds its pick under the same test; spawns wait in the EXISTING queue. The queue wakes on a runtime PID or first stream (_note_startup_progress) and on the slot-release drain of every terminal, including the watchdog's reap of a wedged start. Guard B: the watchdog compares against _startup_deadline_for(info): base + base/8 per OTHER agent in startup, capped at 3x base. Alone in startup the deadline is the base, so the single-agent contract is unchanged. The reaper records the deadline that fired before its awaits and the error names it.
…hdog dynamic-subagent-sizing.md: add agent.subagent_max_concurrent_startups to the config table, correct the sentence that called the cap a startup count limit (it bounds the running population), and state that gate queue time is never charged as startup time on either start path. subagent.md: live-config row, the admission invariant, the dedicated-path gate-exit reset and the watchdog invariant. configuration.md: key row.
bolichen97
force-pushed
the
fix/subagent-startup-pressure
branch
from
September 25, 2026 09:23
224056f to
2ed219d
Compare
Contributor
Opus 5 Review (fork) — ✅ no blocking findingsReviewed |
bolichen97
enabled auto-merge (squash)
September 25, 2026 15:48
5 tasks done
Contributor
Author
|
Superseded by #13890, which carries the identical diff at the same head Nothing was wrong with the code here. This PR could not reach readiness because it was raised from a fork:
The one genuine finding on this PR, |
auto-merge was automatically disabled
September 25, 2026 17:33
Pull request was closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem / Motivation
A wide sub-agent fan-out loses a large share of its members to a false
Failed to start within 120s (no runtime launched, no turn produced)while the members are healthy and merely slow to start.Measured on a 623-item fan-out (2026-09-25), success rate by wave size:
Every loss in the 120-wave was reported as the startup-watchdog error above, and the losses clustered at task-list positions beyond the live concurrency cap.
Root cause, read from the code:
_should_stagger_queue_impl(subagent_manager/admission/pump.py) queues a spawn on exactly two conditions: no free slot, or a spawn started withinsubagent_spawn_stagger_secs. The stagger bounds the RATE of admissions._max_concurrentbounds the RUNNING population.model/reasoning_effortspawn is a dedicated process (~3-5s, ~400 MB); itssession/newthen queues behindSessionStartGate(agent.session_start_concurrency, default 2), so the gate serves 2 starts persession/newround-trip. The decisive mechanism is the mismatch between the ADMISSION rate and the gate's THROUGHPUT, not the stagger value. If one round-trip takes ~10-20s under provider throttling the gate serves ~0.1-0.2 starts/s; at the stock stagger of 0.25s (config/sections.py) admission delivers 4/s, at a 2.0s stagger 0.5/s. The queue at the gate grows either way, and the tail waits minutes._create_shared_session(shared-runtime path) passeson_gate_acquiredso_exec_startedis reset at gate exit and gate queue time is excluded; the dedicated path (get_or_create, which everymodel/reasoning_effortspawn takes -- exactly the measured waves) had no such callback. TheSessionStartGatebounds outstandingsession/newRPCs, not admission: agents were still admitted, still counted as running, still burning the deadline while parked at the gate._is_startup_stalled_impl) reaped at a fixed_startup_deadline(120s), blind to the crowd. A healthy start past 120s -> reaped -> recorded as a failure (terminal.py) -> the orchestrator re-dispatches -> more startup pressure -> more false reaps. That loop is the ~50%.The measured host did NOT run the stock stagger: its
config.jsonsetssubagent_spawn_stagger_secs: 2.0(andmax_subagents: 64), so admissions arrived every 2.0s and a cap of ~37 filled over ~74s -- and it still lost ~50% of a 120-wave. That is the stronger evidence: slowing admission down by 8x does not fix it, because a smaller stagger only makes the pile-up arrive faster; the admission/throughput mismatch and the charged gate wait are what create it. A stock install at 0.25s is worse.Why it matters
Anyone running a fan-out wider than the live cap on a host where starts are slow (model-pinned spawns, a busy session-start gate, provider throttling) loses roughly half the wave to a watchdog that was meant to catch wedged handshakes, and the retries the losses provoke keep the crowd that caused them alive. The failure text blames the run, so operators chase the provider instead of admission.
What changed (motivation → approach → change)
Three changes, each verified against the code before adding.
0 -- stop charging gate wait as startup time on the dedicated path. Mirror of the shared path's existing behaviour, one definition for both:
RunEventCoordinator._gate_exit_reset_impl(info)returns theon_gate_acquiredcallback (resets_exec_started/last_activityat gate EXIT, records_start_queue_wait_ms);_create_shared_sessionnow calls it instead of its inline closure, and_run_innerpasseson_gate_acquired=self._manager._gate_exit_reset(info)into bothget_or_createcall sites. It ridesextra_factory_kwargsto the provider factory (config/loader.py_acp, where it is a NAMED parameter -- the**_kwargscatch-all would otherwise swallow it silently, the same reasonpermission_modeis named), intoAcpProvider(on_gate_acquired=...), and from_start_kiro_runtime_implinto the dedicated process's ownruntime.create_session(...). Not passed toload_session: asession/loadresume takes no gate permit. Every other factory caller passes nothing and getsNone, whichcreate_sessionalready treats as absent. The reset fires only once the permit is HELD, so a start wedged before the gate keeps its original clock and is still caught, and one wedged after the permit is caught at the base deadline from gate exit -- the startup deadline measures time spent STARTING, not time spent WAITING FOR A PERMIT, on both paths.A -- bound the in-startup population (structural). New key
agent.subagent_max_concurrent_startups(config/sections.py,config/loader.py,config-baseline.jsonregenerated, inLIVE_CONFIG_PATHSandapply_limits)._should_stagger_queue_implgains a third clause:_startup_population() >= _startup_cap()queues the spawn exactly as a full cap does, and_drain_queue_sync_implholds its pick under the same test (after the resume grants -- a resume is not a start). No second queueing mechanism: the spawn waits in the existing queue / durable window._in_startup(info)is ONE predicate for both guards and mirrors the watchdog's own shape:_exec_startedset,turns == 0,_pid is None,_first_stream_started is None, not done / reaping. A queued or approval-parked agent is NOT in startup, exactly as it is invisible to the watchdog (that keying is unchanged and load-bearing).0= derivemax(2 × session_start_concurrency, ceil(cap / 4)), clamped to[1, cap]. Why: twice the session-start gate's width keeps one full next round already admitted and waiting at the gate, so the gate never idles between admissions (the stagger refills a freed startup slot within 0.25s); a quarter of the running cap keeps the startup burst -- the phase that spawns a ~400 MB process and initialises its MCP servers -- to a quarter of the steady-state footprint the cap was sized for. At the default gate width of 2: cap 8 -> 4, cap 40 -> 10, cap 64 -> 16; a cap of 3 or below is unaffected (bound = cap).session_start_concurrencyisrestart=True, so it is captured at construction only; the derived bound tracks the gate that actually exists._force_reapsetsdone(which removes it from the population) and then takes the slot-release path interminal.py, which already calls_drain_queue(). The pump also wakes on the two non-terminal exits from startup -- a runtime PID and the first provider stream -- via_note_startup_progressfrom_run_inner(three call sites, at most twice per start; the first-stream site fires only on turn 0). A held drain arms no timer: every edge that frees a startup slot already pumps, so a timer would only poll for those same edges.Subagent queued (...)log line now carriesin_startup=n/capso a hold is diagnosable.B -- make the startup deadline pressure-aware.
_is_startup_stalled_implcompares against_startup_deadline_for(info)=base + others × base × _STARTUP_DEADLINE_PEER_FRACTION(one eighth of the base per OTHER agent in startup, 15s at the 120s default), capped atbase × _STARTUP_DEADLINE_CEILING_FACTOR(3×, 360s). With exactly one agent in startup the deadline is the base, so the single-agent contract is byte-identical and a lone wedged agent is reaped when it always was. At cap 40 (derived bound 10) a start earns up to120 + 9 × 15 = 255s; a truly wedged agent still cannot outlive 360s however large the crowd. The reaper writes the deadline it decided on toinfo._startup_deadline_firedBEFORE its awaits, and thestartup_timeouterror names that value (the population keeps moving through the teardown, so a re-read at record time would name a different number).Both constants scale with the base, so
SubagentManager(startup_timeout=10)in the suite behaves proportionally.Not changed: the
_exec_startedkeying of the watchdog (documented as load-bearing) and the shared path's gate-exit reset semantics (only its definition moved)._start_sem(_MAX_CONCURRENT_COLD_STARTS = 4, the SessionManager's cold-start semaphore) is acquired BEFOREspawn()+create_sessionon the dedicated path, so gate exit is downstream of it and the reset covers that wait too.This is a defect fix in an existing mechanism (the watchdog reaping starts it was never meant to reap), not a product-shape change; the new key defaults to the derived bound and the derived bound leaves any cap ≤ 3 untouched.
Backwards compatibility
Compatible: no key is renamed, removed or narrowed. The new key is optional and defaults to
0; a config without it loads as before.AcpProvider.__init__and the_acpfactory gain one optional keyword defaulting toNone; every existing caller is unchanged. The only observable difference is that a fan-out wider than the derived startup bound fills over a few more stagger intervals instead of piling into startup at once, and a startup reap under pressure fires later (never later than 3× the base).Tests
New
test/test_subagent_startup_pressure.py(41 tests):TestInStartupPredicate-- the predicate's shape and every way out of it;_startup_populationcounts only registered agents in startup and honoursexclude.TestStartupCap-- the derivation table (test_zero_derives_from_the_running_cap, 7 cases incl. cap 0 and cap 3), configured value taken and clamped,apply_limitsadopts the key live and it is inLIVE_CONFIG_PATHS, loader default / negative / non-numeric clamp to 0.TestGateHoldsAtTheStartupCap--_should_stagger_queuereturns(True, True)while startup is full (held, slot still free), opens when one agent records a PID, and a reaped wedged agent no longer counts.test_queue_holds_at_the_startup_cap_and_resumes_on_progress-- end to end throughspawn()with a_runthat enters startup and parks: cap 8, bound 2, four spawns -> two start, two wait in the existing queue with six free slots; each PID admits the next.test_queue_drains_after_the_watchdog_reaps_a_wedged_start-- bound 1, a wedged start and a waiter:_force_reap(reason="startup_timeout")drains the queue and the waiter starts; the error names the base 120s.TestPressureAwareDeadline-- alone = base; +base/8 per peer; peers out of startup add nothing; capped at 3× (and exactly at the knee); scales with a non-default base.TestWatchdogUnderPressure-- a lone wedged agent is reaped at the base deadline; a crowd earns a slow start more time but not past its own deadline; the ceiling holds against 100 peers; pressure is measured at decision time, not registration.test_force_reap_error_names_the_deadline_that_firedandtest_reaper_sweep_records_the_fired_deadline_before_reaping(one real_reaper_loopsweep with_REAPER_INTERVAL=0): the wedged agent is reaped understartup_timeoutwith_startup_deadline_fired == 135.0(one peer) while its within-deadline peer is left alone.TestGateExitResetIsOneDefinition-- the reset moves_exec_started/last_activityand records the wait; the watchdog then measures from gate exit (200s queued + 100s starting is not stalled; wedged 120.5s after the permit IS reaped at the base deadline); the shared path'son_gate_acquiredis_gate_exit_reset's.TestDedicatedPathGateExitReset--test_run_inner_hands_get_or_create_the_gate_exit_reset(a model-pinned_run_innerpasses a callableon_gate_acquiredintoget_or_createthat resets this run's clock);test_provider_factory_names_the_kwarg_and_forwards_it(named in_acp, forwarded toAcpProvider,Nonewhen absent);test_acp_provider_forwards_the_reset_to_its_own_create_session(real_start_kiro_runtimeagainst a fake runtime:create_sessionreceives the callback,load_sessionis not used);test_a_provider_built_without_the_callback_passes_none.Unchanged and green:
test/test_subagent_startup_watchdog.py(all),test/test_subagent_coverage.py::TestStartupStall(managers withstartup_timeout=10) -- the "pressure of one = base" rule is what keeps them so, confirmed by running them.Ran locally: the whole
test_subagent*,test_taskq*,test_spawn*,test_overload*,test_session*,test_acp_provider*set (6050 passed) plus the config suites (test_config_baseline,test_config_loader,test_configuration_doc_defaults,test_config_schema,test_config_api,test_config_live,test_subagent_config_hot_reload,test_adaptive_startup_memory,test_admission_gate,test_fairness_lanes: 991 passed;test_headless_memory_recognition,test_session_start_gate,test_session_sharinggreen);black --check,isort --check-only,flake8,mypy src/kiro_crew/(no issues),scripts/docs_lint.py(passed),BRAND_BASE_REF=origin/main scripts/check_brand_name.py(clean),COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py(clean).Manual verification
N/A -- unit coverage sufficient: the admission hold, the wake edges, the reap-drain path and the deadline arithmetic are all exercised through the real manager, gate, pump and reaper loop; the measured numbers above are the field evidence this reproduces the mechanism of.
Related Issues
no linked issue: this was found by running the fan-out, not from a filed report, and the evidence is the measurement in Problem / Motivation above rather than a tracker item.
Pattern harvest
Rule candidate: review-prompt
Pattern: "a fixed timeout guarding a step whose duration scales with concurrent contenders (queue depth at a shared gate) -- the timeout needs the contention term, or admission needs to bound the contenders".
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)src/kiro_crew/docs/dynamic-subagent-sizing.md(config row; the sentence calling the cap "a startup count limit" corrected to a running-population bound; a section on the three bounds and the pressure-aware watchdog),docs/system-specs/modules/subagent.md(live-config row, admission invariant, dedicated-path gate-exit reset, watchdog invariant),src/kiro_crew/docs/configuration.md(key row)