Skip to content

fix(subagent): cap concurrent startups, pressure-aware start deadline - #13762

Closed
bolichen97 wants to merge 2 commits into
kirodotdev:mainfrom
bolichen97:fix/subagent-startup-pressure
Closed

bolichen97 wants to merge 2 commits into
kirodotdev:mainfrom
bolichen97:fix/subagent-startup-pressure

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

A wide sub-agent fan-out loses a large share of its members to a false Failed to start within 120s (no runtime launched, no turn produced) while the members are healthy and merely slow to start.

Measured on a 623-item fan-out (2026-09-25), success rate by wave size:

wave size success
24-45 ~98%
50-60 84-98%
120 ~50%

Every loss in the 120-wave was reported as the startup-watchdog error above, and the losses clustered at task-list positions beyond the live concurrency cap.

Root cause, read from the code:

  1. _should_stagger_queue_impl (subagent_manager/admission/pump.py) queues a spawn on exactly two conditions: no free slot, or a spawn started within subagent_spawn_stagger_secs. The stagger bounds the RATE of admissions.
  2. _max_concurrent bounds the RUNNING population.
  3. Nothing bounded how many admitted agents were simultaneously IN STARTUP. One start is admitted per stagger interval regardless of how long each takes. A model / reasoning_effort spawn is a dedicated process (~3-5s, ~400 MB); its session/new then queues behind SessionStartGate (agent.session_start_concurrency, default 2), so the gate serves 2 starts per session/new round-trip. The decisive mechanism is the mismatch between the ADMISSION rate and the gate's THROUGHPUT, not the stagger value. If one round-trip takes ~10-20s under provider throttling the gate serves ~0.1-0.2 starts/s; at the stock stagger of 0.25s (config/sections.py) admission delivers 4/s, at a 2.0s stagger 0.5/s. The queue at the gate grows either way, and the tail waits minutes.
  4. On the dedicated-process path that wait was CHARGED to the startup watchdog. _create_shared_session (shared-runtime path) passes on_gate_acquired so _exec_started is reset at gate exit and gate queue time is excluded; the dedicated path (get_or_create, which every model / reasoning_effort spawn takes -- exactly the measured waves) had no such callback. The SessionStartGate bounds outstanding session/new RPCs, not admission: agents were still admitted, still counted as running, still burning the deadline while parked at the gate.
  5. The watchdog (_is_startup_stalled_impl) reaped at a fixed _startup_deadline (120s), blind to the crowd. A healthy start past 120s -> reaped -> recorded as a failure (terminal.py) -> the orchestrator re-dispatches -> more startup pressure -> more false reaps. That loop is the ~50%.

The measured host did NOT run the stock stagger: its config.json sets subagent_spawn_stagger_secs: 2.0 (and max_subagents: 64), so admissions arrived every 2.0s and a cap of ~37 filled over ~74s -- and it still lost ~50% of a 120-wave. That is the stronger evidence: slowing admission down by 8x does not fix it, because a smaller stagger only makes the pile-up arrive faster; the admission/throughput mismatch and the charged gate wait are what create it. A stock install at 0.25s is worse.

Why it matters

Anyone running a fan-out wider than the live cap on a host where starts are slow (model-pinned spawns, a busy session-start gate, provider throttling) loses roughly half the wave to a watchdog that was meant to catch wedged handshakes, and the retries the losses provoke keep the crowd that caused them alive. The failure text blames the run, so operators chase the provider instead of admission.

What changed (motivation → approach → change)

Three changes, each verified against the code before adding.

0 -- stop charging gate wait as startup time on the dedicated path. Mirror of the shared path's existing behaviour, one definition for both: RunEventCoordinator._gate_exit_reset_impl(info) returns the on_gate_acquired callback (resets _exec_started / last_activity at gate EXIT, records _start_queue_wait_ms); _create_shared_session now calls it instead of its inline closure, and _run_inner passes on_gate_acquired=self._manager._gate_exit_reset(info) into both get_or_create call sites. It rides extra_factory_kwargs to the provider factory (config/loader.py _acp, where it is a NAMED parameter -- the **_kwargs catch-all would otherwise swallow it silently, the same reason permission_mode is named), into AcpProvider(on_gate_acquired=...), and from _start_kiro_runtime_impl into the dedicated process's own runtime.create_session(...). Not passed to load_session: a session/load resume takes no gate permit. Every other factory caller passes nothing and gets None, which create_session already treats as absent. The reset fires only once the permit is HELD, so a start wedged before the gate keeps its original clock and is still caught, and one wedged after the permit is caught at the base deadline from gate exit -- the startup deadline measures time spent STARTING, not time spent WAITING FOR A PERMIT, on both paths.

A -- bound the in-startup population (structural). New key agent.subagent_max_concurrent_startups (config/sections.py, config/loader.py, config-baseline.json regenerated, in LIVE_CONFIG_PATHS and apply_limits). _should_stagger_queue_impl gains a third clause: _startup_population() >= _startup_cap() queues the spawn exactly as a full cap does, and _drain_queue_sync_impl holds its pick under the same test (after the resume grants -- a resume is not a start). No second queueing mechanism: the spawn waits in the existing queue / durable window.

  • _in_startup(info) is ONE predicate for both guards and mirrors the watchdog's own shape: _exec_started set, turns == 0, _pid is None, _first_stream_started is None, not done / reaping. A queued or approval-parked agent is NOT in startup, exactly as it is invisible to the watchdog (that keying is unchanged and load-bearing).
  • Default 0 = derive max(2 × session_start_concurrency, ceil(cap / 4)), clamped to [1, cap]. Why: twice the session-start gate's width keeps one full next round already admitted and waiting at the gate, so the gate never idles between admissions (the stagger refills a freed startup slot within 0.25s); a quarter of the running cap keeps the startup burst -- the phase that spawns a ~400 MB process and initialises its MCP servers -- to a quarter of the steady-state footprint the cap was sized for. At the default gate width of 2: cap 8 -> 4, cap 40 -> 10, cap 64 -> 16; a cap of 3 or below is unaffected (bound = cap). session_start_concurrency is restart=True, so it is captured at construction only; the derived bound tracks the gate that actually exists.
  • No deadlock. A wedged agent holds a startup slot only until the watchdog reaps it: _force_reap sets done (which removes it from the population) and then takes the slot-release path in terminal.py, which already calls _drain_queue(). The pump also wakes on the two non-terminal exits from startup -- a runtime PID and the first provider stream -- via _note_startup_progress from _run_inner (three call sites, at most twice per start; the first-stream site fires only on turn 0). A held drain arms no timer: every edge that frees a startup slot already pumps, so a timer would only poll for those same edges.
  • The Subagent queued (...) log line now carries in_startup=n/cap so a hold is diagnosable.

B -- make the startup deadline pressure-aware. _is_startup_stalled_impl compares against _startup_deadline_for(info) = base + others × base × _STARTUP_DEADLINE_PEER_FRACTION (one eighth of the base per OTHER agent in startup, 15s at the 120s default), capped at base × _STARTUP_DEADLINE_CEILING_FACTOR (3×, 360s). With exactly one agent in startup the deadline is the base, so the single-agent contract is byte-identical and a lone wedged agent is reaped when it always was. At cap 40 (derived bound 10) a start earns up to 120 + 9 × 15 = 255s; a truly wedged agent still cannot outlive 360s however large the crowd. The reaper writes the deadline it decided on to info._startup_deadline_fired BEFORE its awaits, and the startup_timeout error names that value (the population keeps moving through the teardown, so a re-read at record time would name a different number).

Both constants scale with the base, so SubagentManager(startup_timeout=10) in the suite behaves proportionally.

Not changed: the _exec_started keying of the watchdog (documented as load-bearing) and the shared path's gate-exit reset semantics (only its definition moved). _start_sem (_MAX_CONCURRENT_COLD_STARTS = 4, the SessionManager's cold-start semaphore) is acquired BEFORE spawn() + create_session on the dedicated path, so gate exit is downstream of it and the reset covers that wait too.

This is a defect fix in an existing mechanism (the watchdog reaping starts it was never meant to reap), not a product-shape change; the new key defaults to the derived bound and the derived bound leaves any cap ≤ 3 untouched.

Backwards compatibility

Compatible: no key is renamed, removed or narrowed. The new key is optional and defaults to 0; a config without it loads as before. AcpProvider.__init__ and the _acp factory gain one optional keyword defaulting to None; every existing caller is unchanged. The only observable difference is that a fan-out wider than the derived startup bound fills over a few more stagger intervals instead of piling into startup at once, and a startup reap under pressure fires later (never later than 3× the base).

Tests

New test/test_subagent_startup_pressure.py (41 tests):

  • TestInStartupPredicate -- the predicate's shape and every way out of it; _startup_population counts only registered agents in startup and honours exclude.
  • TestStartupCap -- the derivation table (test_zero_derives_from_the_running_cap, 7 cases incl. cap 0 and cap 3), configured value taken and clamped, apply_limits adopts the key live and it is in LIVE_CONFIG_PATHS, loader default / negative / non-numeric clamp to 0.
  • TestGateHoldsAtTheStartupCap -- _should_stagger_queue returns (True, True) while startup is full (held, slot still free), opens when one agent records a PID, and a reaped wedged agent no longer counts.
  • test_queue_holds_at_the_startup_cap_and_resumes_on_progress -- end to end through spawn() with a _run that enters startup and parks: cap 8, bound 2, four spawns -> two start, two wait in the existing queue with six free slots; each PID admits the next.
  • test_queue_drains_after_the_watchdog_reaps_a_wedged_start -- bound 1, a wedged start and a waiter: _force_reap(reason="startup_timeout") drains the queue and the waiter starts; the error names the base 120s.
  • TestPressureAwareDeadline -- alone = base; +base/8 per peer; peers out of startup add nothing; capped at 3× (and exactly at the knee); scales with a non-default base.
  • TestWatchdogUnderPressure -- a lone wedged agent is reaped at the base deadline; a crowd earns a slow start more time but not past its own deadline; the ceiling holds against 100 peers; pressure is measured at decision time, not registration.
  • test_force_reap_error_names_the_deadline_that_fired and test_reaper_sweep_records_the_fired_deadline_before_reaping (one real _reaper_loop sweep with _REAPER_INTERVAL=0): the wedged agent is reaped under startup_timeout with _startup_deadline_fired == 135.0 (one peer) while its within-deadline peer is left alone.
  • TestGateExitResetIsOneDefinition -- the reset moves _exec_started / last_activity and records the wait; the watchdog then measures from gate exit (200s queued + 100s starting is not stalled; wedged 120.5s after the permit IS reaped at the base deadline); the shared path's on_gate_acquired is _gate_exit_reset's.
  • TestDedicatedPathGateExitReset -- test_run_inner_hands_get_or_create_the_gate_exit_reset (a model-pinned _run_inner passes a callable on_gate_acquired into get_or_create that resets this run's clock); test_provider_factory_names_the_kwarg_and_forwards_it (named in _acp, forwarded to AcpProvider, None when absent); test_acp_provider_forwards_the_reset_to_its_own_create_session (real _start_kiro_runtime against a fake runtime: create_session receives the callback, load_session is not used); test_a_provider_built_without_the_callback_passes_none.

Unchanged and green: test/test_subagent_startup_watchdog.py (all), test/test_subagent_coverage.py::TestStartupStall (managers with startup_timeout=10) -- the "pressure of one = base" rule is what keeps them so, confirmed by running them.

Ran locally: the whole test_subagent*, test_taskq*, test_spawn*, test_overload*, test_session*, test_acp_provider* set (6050 passed) plus the config suites (test_config_baseline, test_config_loader, test_configuration_doc_defaults, test_config_schema, test_config_api, test_config_live, test_subagent_config_hot_reload, test_adaptive_startup_memory, test_admission_gate, test_fairness_lanes: 991 passed; test_headless_memory_recognition, test_session_start_gate, test_session_sharing green); black --check, isort --check-only, flake8, mypy src/kiro_crew/ (no issues), scripts/docs_lint.py (passed), BRAND_BASE_REF=origin/main scripts/check_brand_name.py (clean), COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py (clean).

Manual verification

N/A -- unit coverage sufficient: the admission hold, the wake edges, the reap-drain path and the deadline arithmetic are all exercised through the real manager, gate, pump and reaper loop; the measured numbers above are the field evidence this reproduces the mechanism of.

Related Issues

no linked issue: this was found by running the fan-out, not from a filed report, and the evidence is the measurement in Problem / Motivation above rather than a tracker item.

Pattern harvest

Rule candidate: review-prompt
Pattern: "a fixed timeout guarding a step whose duration scales with concurrent contenders (queue depth at a shared gate) -- the timeout needs the contention term, or admission needs to bound the contenders".

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) -- src/kiro_crew/docs/dynamic-subagent-sizing.md (config row; the sentence calling the cap "a startup count limit" corrected to a running-population bound; a section on the three bounds and the pressure-aware watchdog), docs/system-specs/modules/subagent.md (live-config row, admission invariant, dedicated-path gate-exit reset, watchdog invariant), src/kiro_crew/docs/configuration.md (key row)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and pepmach September 25, 2026 08:52
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-startup-pressure branch from 9f8fde8 to 224056f Compare September 25, 2026 09:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
A fan-out was bounded by two quantities: subagent_spawn_stagger_secs bounds
the RATE of starts and _max_concurrent bounds the RUNNING population.
Nothing bounded how many admitted agents were IN STARTUP at once, the
dedicated-process path charged its SessionStartGate queue time to the startup
watchdog, and the watchdog reaped at a fixed 120s blind to the crowd. On a
623-item fan-out, waves of 24-45 lost ~2% and a wave of 120 lost ~50%, every
loss a healthy start recorded as "Failed to start within 120s".

Gate-exit reset on the dedicated path: _run_inner passes
on_gate_acquired=_gate_exit_reset(info) into get_or_create; it rides the
provider factory (a NAMED parameter in _acp) into AcpProvider and from there
into the process's own create_session, so time parked at the gate is not
charged as startup time on either path. One definition serves both; the
shared path's inline closure becomes a call to it.

Guard A: agent.subagent_max_concurrent_startups (0 = derive
max(2 x session_start_concurrency, ceil(cap / 4)), never above the cap).
_should_stagger_queue gains a third clause and the drain pump holds its pick
under the same test; spawns wait in the EXISTING queue. The queue wakes on a
runtime PID or first stream (_note_startup_progress) and on the slot-release
drain of every terminal, including the watchdog's reap of a wedged start.

Guard B: the watchdog compares against _startup_deadline_for(info): base +
base/8 per OTHER agent in startup, capped at 3x base. Alone in startup the
deadline is the base, so the single-agent contract is unchanged. The reaper
records the deadline that fired before its awaits and the error names it.
…hdog

dynamic-subagent-sizing.md: add agent.subagent_max_concurrent_startups to the
config table, correct the sentence that called the cap a startup count limit
(it bounds the running population), and state that gate queue time is never
charged as startup time on either start path. subagent.md: live-config row,
the admission invariant, the dedicated-path gate-exit reset and the watchdog
invariant. configuration.md: key row.
@bolichen97
bolichen97 force-pushed the fix/subagent-startup-pressure branch from 224056f to 2ed219d Compare September 25, 2026 09:23
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 224056fcea834ee5447df85c4ae9d498f1b6a4c0 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 224056f

@bolichen97

Copy link
Copy Markdown
Contributor Author

Superseded by #13890, which carries the identical diff at the same head 2ed219d8fd from a branch in this repository.

Nothing was wrong with the code here. This PR could not reach readiness because it was raised from a fork:

  • Backend Lint & Type Check (3.12) was killed with exit 143 in its black step (oom_kill 0, ~6.5 min in), which is a killed process rather than a formatting violation — a violation exits 1 with a diff. Reproduced on other contributors' fork PRs (test(acp): hold the prune clock in the backlog-drain cap test #13738, test(knowledge): wait on the committed audit instead of a fixed sleep #13737), so it was not this change.
  • GPT 5.6 Review, Opus 5 Review and Internal Content Scan reported "could not run"; Security Scope Review reported [scope-floor:unsettled]. None had a re-runnable workflow run in this repository.
  • The two failures that WERE re-runnable (Resolve desktop matrix, Publish readiness signal, both API rate limit exceeded for installation) were re-run and cleared here.

The one genuine finding on this PR, semgrep.kirocrew.mock-pid-allocatable, was fixed at this head before it was re-raised. Same path as #13604 → #13721.

@bolichen97 bolichen97 closed this Sep 25, 2026
auto-merge was automatically disabled September 25, 2026 17:33

Pull request was closed

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant