Skip to content

fix(subagent): cap concurrent startups, reset start clock at gate exit - #13890

Merged
iamwhatever merged 2 commits into
mainfrom
fix/subagent-startup-pressure
Sep 25, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
fix/subagent-startup-pressure

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Supersedes #13762. It was opened at the same head 2ed219d8fd with the identical diff; this PR has since moved to b6acb084da, where the pressure-aware deadline was withdrawn in response to the First Principles review (see Withdrawn below). That PR was raised from a fork, where it could not reach readiness: Backend Lint & Type Check (3.12) was killed with exit 143 in its black step (oom_kill 0) on fork PRs generally — reproduced on #13738 and #13737, not only on mine — and the four review lanes (GPT 5.6, Opus 5, Internal Content Scan, Security Scope) reported "could not run" / [scope-floor:unsettled] with no re-runnable workflow available. Zero code findings were outstanding when it was re-raised here; the one real finding (semgrep.kirocrew.mock-pid-allocatable) was fixed on this head.

Problem / Motivation

A wide sub-agent fan-out loses a large share of its members to a false Failed to start within 120s (no runtime launched, no turn produced) while the members are healthy and merely slow to start.

Measured on a 623-item fan-out (2026-09-25), success rate by wave size:

wave size success
24-45 ~98%
50-60 84-98%
120 ~50%

Every loss in the 120-wave was reported as the startup-watchdog error above, and the losses clustered at task-list positions beyond the live concurrency cap.

Root cause, read from the code:

  1. _should_stagger_queue_impl (subagent_manager/admission/pump.py) queues a spawn on exactly two conditions: no free slot, or a spawn started within subagent_spawn_stagger_secs. The stagger bounds the RATE of admissions.
  2. _max_concurrent bounds the RUNNING population.
  3. Nothing bounded how many admitted agents were simultaneously IN STARTUP. One start is admitted per stagger interval regardless of how long each takes. A model / reasoning_effort spawn is a dedicated process (~3-5s, ~400 MB); its session/new then queues behind SessionStartGate (agent.session_start_concurrency, default 2), so the gate serves 2 starts per session/new round-trip. The decisive mechanism is the mismatch between the ADMISSION rate and the gate's THROUGHPUT, not the stagger value. If one round-trip takes ~10-20s under provider throttling the gate serves ~0.1-0.2 starts/s; at the stock stagger of 0.25s (config/sections.py) admission delivers 4/s, at a 2.0s stagger 0.5/s. The queue at the gate grows either way, and the tail waits minutes.
  4. On the dedicated-process path that wait was CHARGED to the startup watchdog. _create_shared_session (shared-runtime path) passes on_gate_acquired so _exec_started is reset at gate exit and gate queue time is excluded; the dedicated path (get_or_create, which every model / reasoning_effort spawn takes -- exactly the measured waves) had no such callback. The SessionStartGate bounds outstanding session/new RPCs, not admission: agents were still admitted, still counted as running, still burning the deadline while parked at the gate.
  5. The watchdog (_is_startup_stalled_impl) reaped at _startup_deadline (120s) measured from admission, so on the dedicated path the whole gate wait counted. A healthy start past 120s -> reaped -> recorded as a failure (terminal.py) -> the orchestrator re-dispatches -> more startup pressure -> more false reaps. That loop is the ~50%.

The measured host did NOT run the stock stagger: its config.json sets subagent_spawn_stagger_secs: 2.0 (and max_subagents: 64), so admissions arrived every 2.0s and a cap of ~37 filled over ~74s -- and it still lost ~50% of a 120-wave. That is the stronger evidence: slowing admission down by 8x does not fix it, because a smaller stagger only makes the pile-up arrive faster; the admission/throughput mismatch and the charged gate wait are what create it. A stock install at 0.25s is worse.

Why it matters

Anyone running a fan-out wider than the live cap on a host where starts are slow (model-pinned spawns, a busy session-start gate, provider throttling) loses roughly half the wave to a watchdog that was meant to catch wedged handshakes, and the retries the losses provoke keep the crowd that caused them alive. The failure text blames the run, so operators chase the provider instead of admission.

What changed (motivation → approach → change)

Two changes, each verified against the code before adding. A third (a pressure-aware watchdog deadline) was in an earlier revision of this PR and has been removed; see "Withdrawn" below.

0 -- the startup clock never counts time queued for a gate permit, on either path. runtime.create_session fires two callbacks around its SessionStartGate wait: on_gate_queued immediately before the wait for a permit begins, and on_gate_acquired at gate EXIT with the queue wait. RunEventCoordinator._gate_wait_mark_impl(info) stamps _gate_wait_started on the first, and while it is set _is_startup_stalled reads the start clock as frozen at that moment; _gate_exit_reset_impl(info) clears it and restarts the clock (_exec_started / last_activity, records _start_queue_wait_ms) on the second. _create_shared_session hands the pair to the parent runtime's create_session directly; _run_inner passes both into get_or_create, and they ride extra_factory_kwargs to the provider factory (config/loader.py _acp, where both are NAMED parameters -- the **_kwargs catch-all would otherwise swallow them silently), into AcpProvider(on_gate_acquired=, on_gate_queued=), and from _start_kiro_runtime_impl into the dedicated process's own runtime.create_session(...). Not passed to load_session: a session/load resume takes no gate permit. Every other factory caller passes nothing and gets None, which create_session treats as absent.

Invariant enforced (GPT span=599ab81e58eb, "the startup cap does not bound pre-permit wait" -- correct, and this is the fix): the startup deadline measures time spent STARTING -- before the gate (a process spawn on the dedicated path) and with a permit held (session/new) -- and never time queued for a permit, however long the queue. A start wedged before the gate is on a running clock and is reaped at the base deadline; a start queued at the gate is not reaped for waiting; a start wedged after its permit is reaped at the base deadline from gate exit. The earlier form of the argument ("the last waiter's pre-permit wait is one round, and a round is bounded by the deadline") does not hold and is gone from every docstring and doc: a holder's clock resets at acquisition, so a session/new taking 119s is not reaped, the permit stays held, and a waiter's un-reset clock keeps accumulating -- a healthy waiter could be reaped exactly as it was about to be served. Freezing the waiter's clock removes the charge instead of trying to bound the round. The wait itself is finite without a deadline of its own: every permit holder is on a running clock from acquisition and is reaped at the base deadline if its session/new has not returned (_pid is set only after create_session returns), the request has its own budget (agent.session_start_timeout_secs), and the gate keeps a headroom of permits no StartCollector may hold (collector_hold_ceiling).

A -- bound the in-startup population (structural). SubagentManager._startup_cap() bounds how many admitted agents may be in startup at once; there is no config key for it (see below). _should_stagger_queue_impl gains a third clause: _startup_population() >= _startup_cap() queues the spawn exactly as a full cap does, and _drain_queue_sync_impl holds its pick under the same test (after the resume grants -- a resume is not a start). No second queueing mechanism: the spawn waits in the existing queue / durable window.

  • _in_startup(info) mirrors the watchdog's own shape: _exec_started set, turns == 0, _pid is None, _first_stream_started is None, not done / reaping. _startup_population() also counts _startup_reservations: a durable-store ClaimPoint reserved before its claim is awaited and registered only on the re-entry that skips the gate. A queued spawn is not in it (not registered until admitted), and neither is an agent PARKED at the spawn-approval prompt.
  • The bound is 2 × session_start_concurrency, clamped to [1, cap] -- tied to the session-start GATE, not to the running cap. The gate is the one resource every start in startup contends for: session/new runs under G permits, so at most G starts make progress at any moment, and every other admitted start is a spawned process (dedicated path) or a claimed slot holding nothing but a place in the gate's queue. With change 0 the queue's length reaps nothing, so what the bound decides is how much of the running cap may sit in startup contending for G permits at once. 2G is the smallest value that never idles the gate -- one round holding permits and one round already admitted to take them the moment they free -- and admitting more buys no starts, since the gate serves G per round however many are queued: it only lengthens the queue and grows the population of admitted-but-idle starts (each a live process on the dedicated path). A cap-derived term (max(2 × G, ceil(cap / 4)), say) would do exactly that: at cap 64 it admits 16 against a 2-permit gate, seven rounds queued for two permits. At the default gate width of 2 the bound is 4 at cap 8, 40 and 64 alike; cap below 4; 1 at a cap of 0 (an adaptive squeeze -- the running cap, not this bound, pauses admission there). session_start_concurrency is restart=True, so it is captured at construction only; the bound tracks the gate that actually exists. It also coincides with the SessionManager's dedicated cold-start semaphore (_MAX_CONCURRENT_COLD_STARTS = 4).
  • There is deliberately no config key. 2G is both the floor and the ceiling of the useful range: fewer than 2G idles the gate between rounds (no next round already admitted when the current one releases), and more than 2G buys no starts -- the gate serves G per round however many are queued behind it -- only a longer queue of idle admitted starts. A knob could therefore only make the system worse, and the operator's real lever already exists: agent.session_start_concurrency sizes the gate, and the bound tracks it. config/sections.py, config/loader.py's agent block, config-baseline.json and configuration.md are unchanged from main; LIVE_CONFIG_PATHS / apply_limits gain nothing (a cap change re-clamps the derived value). _startup_cap()'s docstring states this reason so the absence is explained, not silent.
  • The bulk-grant flood is bounded by metering the RELEASE, not by counting parked agents (GPT span=599ab81e58eb closed; Design span=3d20de435a65 answered). Admission is the only place the bound is checked, and _spawn_with_approval used to go from approval straight to _run; a bulk grant (tool_approval:bulk_trust / bulk_yolo in chat_handlers.py) resolves EVERY pending prompt in one pass, so a whole wave admitted to park entered startup together. The previous head counted parked agents while parked; that closed the flood but self-inflicted the converse -- at the default bound of 4, four ignored prompts queued every other spawn on the host, auto-approved ones from unrelated parents included, with no load needed to trigger it. Now a parked agent consumes nothing (it is starting nothing), and its release re-enters through the pump: _spawn_with_approval awaits _admit_released_start, which appends a resident entry to the EXISTING queue (_resume_id + _startup_release + the waiter, so every unstarted-spawn scan -- queued-stop, refill census, eviction, continuation lookup -- already leaves it alone) and waits on info._start_release. The pump's first phase, _release_admitted_start, meters ONE released start per pass under the same stagger and _startup_population() >= _startup_cap() test a fresh spawn faces, stamps _last_spawn_ts, and re-arms at the stagger boundary. It answers "released", "held" or "" (none waiting) and the pass CONTINUES on every answer (Opus span=dff7a69117d0, correct: a phase that ended the pass on its hold withheld the resume grants and pick_window_index for as long as the bound stayed full): a resume waits on a lane slot, never on the startup bound or the stagger, so the grants run whether a start was released or is being held, and the fresh-spawn pick applies the same two checks itself. It runs before the capacity check (the run already holds its slot; at a full cap any_slot is False and a released start would otherwise wait on itself) and before the resume grants (it is older than anything admitted after it); a start whose prompt resolves while gateway admission is closed is refused at release (spawn rejected: the gateway closed admission before this approved spawn could start) -- the same admission_closed guard every registration in the admission package sits behind, which is also what makes the released-start queue append the sixth registration site in test_update_check_install_aware.py's admission/census contract; pick_window_index and the resume-first loop skip it, and it does not arm the child reserve. While it waits it is registered, holds its running slot and shows in its parent's queue depth; it joins _startup_population only when released. A stop or reap while waiting wakes it with False and drops its entry (_force_reap names the state: "waiting to be admitted into startup after spawn approval"); a slow self-re-arming re-pump (_RELEASE_REPUMP_SECS, 1s) backs the edge-driven wake so a swallowed pump failure cannot strand it. The watchdog stays blind to parked and released-waiting agents (a human prompt has no deadline). Every other path into _run -- yolo, approval_mode="auto", parent trust, hooks.auto_approve_subagent_spawn, POST /api/spawn, the MCP tools, spawn_continue -- creates its run task inside spawn() immediately after the gate, so none had the hole; the cancel-recovery respawn re-runs an already-admitted run.
  • No deadlock. A wedged agent holds a startup slot only until the watchdog reaps it: _force_reap sets done (which removes it from the population) and then takes the slot-release path in terminal.py, which already calls _drain_queue(). The pump also wakes on the two non-terminal exits from startup -- a runtime PID and the first provider stream -- via _note_startup_progress from _run_inner (three call sites, at most twice per start; the first-stream site fires only on turn 0). A held drain arms no timer: every edge that frees a startup slot already pumps, so a timer would only poll for those same edges.
  • The Subagent queued (...) log line now carries in_startup=n/cap so a hold is diagnosable.

Withdrawn -- a pressure-aware startup deadline. An earlier revision of this PR also made _is_startup_stalled_impl compare against base + base/8 × (other agents in startup), capped at 3× the base. It is removed, for two reasons, both raised in review and both correct:

  • Not justified by the measurement. The ~50% figure was taken before any of the changes here existed. This PR's own text names the gate-exit reset as the strongest single mechanism behind that loss, and guard A bounds the crowd at admission. Nothing shows a healthy start still missing the BASE deadline once queue time stops being charged and the in-startup population is bounded. I could not write a test showing that without contriving the input (a test that declares a 130s post-permit start "healthy" proves only its own premise), so the mechanism has no evidence behind it and is not kept on the strength of being already written.
  • Not monotonic. others was sampled at SWEEP time but compared against now - _exec_started, which spans the whole crowded period, so the effective deadline SHRANK as the crowd drained: an agent inside a 225s window at one sweep could be reaped at the next under 120s. Removing the term dissolves the defect rather than patching it.

The watchdog therefore compares against the fixed _startup_deadline, exactly as on main; terminal.py is unchanged from main. The reaper's warning now also names the in-startup population, as diagnostics only. If a future measurement WITH changes 0 and A in force shows healthy starts reaped at the base deadline, that measurement is the place to start a pressure term from -- and any such term must be latched per agent so a granted window can only grow.

Not changed: the _exec_started keying of the watchdog (documented as load-bearing) and the shared path's gate-exit reset semantics (only its definition moved). _start_sem (_MAX_CONCURRENT_COLD_STARTS = 4, the SessionManager's cold-start semaphore) is acquired BEFORE spawn() + create_session on the dedicated path, so gate exit is downstream of it and the reset covers that wait too.

This is a defect fix in an existing mechanism (the watchdog reaping starts it was never meant to reap), not a product-shape change; the new key defaults to the derived bound and the derived bound leaves any cap ≤ 3 untouched.

Backwards compatibility

Compatible: no key is added, renamed, removed or narrowed -- the config schema, config-baseline.json and configuration.md are unchanged from main. AcpProvider.__init__ and the _acp factory gain one optional keyword defaulting to None; every existing caller is unchanged. The only observable difference is that a fan-out wider than the startup bound (4 at the default gate width) is admitted into startup one gate round at a time instead of piling in at once; admission throughput is unchanged, since the gate already served only G starts per round. The startup deadline itself is unchanged.

Tests

New test/test_subagent_startup_pressure.py (42 tests); plus one in test_session_start_gate.py, a shape assertion for on_gate_queued in test_headless_memory_recognition.py, and test_update_check_install_aware.py's admission/census contract updated to six subagent registration sites (the sixth: the released-start queue append in _admit_released_start_impl, behind the same admission_closed guard):

  • TestInStartupPredicate -- the predicate's shape; a queued or approval-parked agent is NOT in it (and the watchdog is blind to it); every way out of it (including a mid-run tool prompt); _startup_population counts only registered agents and honours exclude; test_a_reservation_counts_until_its_info_is_registered (a ClaimPoint reservation holds the gate until release_reservation).
  • TestStartupCap -- the derivation table (test_derives_two_gate_rounds_clamped_to_the_cap, 8 cases incl. cap 0, cap 3, and cap 64 at gate widths 2 and 4), test_derived_bound_does_not_grow_with_the_cap (caps 8..128 all derive 4 at gate width 2), test_there_is_no_config_key_for_the_bound (no schema field, not in LIVE_CONFIG_PATHS, no manager setting; apply_limits only re-clamps).
  • TestGateHoldsAtTheStartupCap (gate width 1, bound 2) -- _should_stagger_queue returns (True, True) while startup is full (held, slot still free), opens when one agent records a PID, and a reaped wedged agent no longer counts.
  • test_ignored_prompts_do_not_block_an_unrelated_auto_approved_spawn -- through the REAL _spawn_with_approval (no yolo, no parent trust, no hook auto-approve; an on_spawn_approval that parks each prompt on a future): cap 8, gate width 1 (bound 2), FOUR prompted spawns nobody answers, then an approval_mode="auto" spawn from another parent starts at once, and the bound still meters the auto path itself. Fails on the head that counted parked agents.
  • test_bulk_approval_cannot_release_more_than_the_startup_cap -- same harness: four prompted spawns all park (population 0, four slots held); every pending prompt is resolved in one pass with trust flipped on, exactly as the bulk grant does -> two enter _run, two wait as _startup_release entries in the existing queue, each released on PID progress. Fails on the head with no metering (all four entered _run together).
  • test_a_released_start_that_is_stopped_while_waiting_never_runs -- a released start reaped while waiting for the bound ends without a run and its entry is dropped.
  • test_queue_holds_at_the_startup_cap_and_resumes_on_progress -- end to end through spawn() with a _run that enters startup and parks: cap 8, gate width 1 (bound 2), four spawns -> two start, two wait in the existing queue with six free slots; each PID admits the next.
  • test_queue_drains_after_the_watchdog_reaps_a_wedged_start -- gate width 1 (bound 2), two wedged starts and a waiter: _force_reap(reason="startup_timeout") on one wedged start drains the queue and the waiter starts; the error names the 120s deadline.
  • TestWatchdogIgnoresTheCrowd -- a lone wedged agent is reaped at the base deadline; four peers in startup buy a wedged agent no extra time; and the monotonicity invariant the withdrawn term violated: an agent inside its window at one sweep with peers present is still inside it at the next after every peer has left.
  • test_reaper_sweep_reaps_at_the_base_deadline_under_a_crowd (one real _reaper_loop sweep with _REAPER_INTERVAL=0, two agents in startup): the one past the base deadline is reaped under startup_timeout at +125s -- the point where the withdrawn term would have granted 135s -- while its within-deadline peer is left alone.
  • TestGateExitResetIsOneDefinition -- the reset moves _exec_started / last_activity and records the wait; the watchdog measures from gate exit (200s queued + 100s starting is not stalled; wedged 120.5s after the permit IS reaped); test_the_clock_does_not_run_while_queued_for_a_permit (30s into its start, a run waits 400s for a permit and is not reaped; at acquisition the clock restarts from zero); test_a_start_wedged_before_the_gate_keeps_its_running_clock; the shared path's on_gate_queued / on_gate_acquired are _gate_wait_mark's / _gate_exit_reset's.
  • test_session_start_gate.py::test_gate_entry_callback_fires_before_the_wait_and_exit_after -- against the real AcpRuntime.create_session and a filled gate: on_gate_queued fires while queued, on_gate_acquired only after a permit frees, then session/new goes out.
  • test_a_held_release_does_not_starve_a_queued_resume -- gate width 1 (bound 2), two starts fill the bound, a third released from its prompt is held; a running worker yields its lane slot (yield_slot, row running -> waiting_children) and requests a resume: the resume is granted on the same pass that holds the release. Fails on the head whose release phase ended the pass on a hold.
  • TestDedicatedPathGateExitReset -- test_run_inner_hands_get_or_create_the_gate_exit_reset (a model-pinned _run_inner passes callable on_gate_acquired AND on_gate_queued into get_or_create; the mark freezes and the reset restarts this run's clock); test_provider_factory_names_the_kwarg_and_forwards_it (both named in _acp, forwarded to AcpProvider, None when absent); test_acp_provider_forwards_the_reset_to_its_own_create_session (real _start_kiro_runtime against a fake runtime: create_session receives both callbacks, load_session is not used); test_a_provider_built_without_the_callback_passes_none.

Removed with the withdrawn mechanism: TestPressureAwareDeadline (5), TestWatchdogUnderPressure (4), test_force_reap_error_names_the_deadline_that_fired, test_reaper_sweep_records_the_fired_deadline_before_reaping.

Unchanged and green: test/test_subagent_startup_watchdog.py (all), test/test_subagent_coverage.py::TestStartupStall (managers with startup_timeout=10) -- the watchdog's deadline arithmetic is untouched from main.

Ran locally: the whole test_subagent*, test_taskq*, test_spawn*, test_overload*, test_session*, test_acp_provider* set (6050 passed) plus the config suites (test_config_baseline, test_config_loader, test_configuration_doc_defaults, test_config_schema, test_config_api, test_config_live, test_subagent_config_hot_reload, test_adaptive_startup_memory, test_admission_gate, test_fairness_lanes: 991 passed; test_headless_memory_recognition, test_session_start_gate, test_session_sharing green); black --check, isort --check-only, flake8, mypy src/kiro_crew/ (no issues), scripts/docs_lint.py (passed), BRAND_BASE_REF=origin/main scripts/check_brand_name.py (clean), COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py (clean).

Manual verification

N/A -- unit coverage sufficient: the admission hold, the wake edges, the reap-drain path and the deadline arithmetic are all exercised through the real manager, gate, pump and reaper loop; the measured numbers above are the field evidence this reproduces the mechanism of.

Related Issues

no linked issue: this was found by running the fan-out, not from a filed report, and the evidence is the measurement in Problem / Motivation above rather than a tracker item.

Pattern harvest

Rule candidate: review-prompt
Pattern: "a fixed timeout guarding a step whose duration scales with concurrent contenders (queue depth at a shared gate) -- start the clock after the contended wait and bound the contenders at admission; do not widen the timeout by a term sampled at check time, which is not monotonic and has no measurement behind it once the first two are in place".

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) -- src/kiro_crew/docs/dynamic-subagent-sizing.md (the sentence calling the cap "a startup count limit" corrected to a running-population bound; a section on the three bounds, when gate queue time stops being charged, and why the watchdog deadline stays fixed), docs/system-specs/modules/subagent.md (admission invariant, dedicated-path gate-exit reset, fixed-deadline invariant)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and cixuuz September 25, 2026 17:33
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 25, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 25, 2026 17:35
@bolichen97
bolichen97 disabled auto-merge September 25, 2026 17:35
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 3c9ccd6116375675e1ca8f5aebc9bc99385fb92e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Re-run the wide (120+) fan-out on this head — the ~50% loss recovery is mechanism-tested, not yet field re-measured.

[DESIGN-REVIEWED] 3c9ccd6

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 3c9ccd6116375675e1ca8f5aebc9bc99385fb92e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Re-run the 120-item wave on this head: the fix's evidence is the author's own 2026-09-25 measurement, and no test reproduces the measured loss end-to-end.

What this change ships

Inventory (10 items) — 10 justified

Intent: stop the startup watchdog from reaping healthy subagents in wide fan-outs by not charging gate-queue time and bounding the in-startup crowd — a FIX. Provenance is strong: base's own session_start_concurrency field doc already pins "queue time is not counted against the start budget or the startup watchdog" (config/sections.py:1485), which the dedicated path violated.

  1. Model-pinned spawns no longer reaped for gate-queue time (clock resets at gate exit) — justified
  2. Clock freezes while queued for a permit, both paths (on_gate_queued) — justified
  3. At most 2×gate-width agents in startup; excess waits in the existing queue, no config key — justified
  4. Bulk-approved starts metered one-per-pass through the pump instead of starting all at once — justified
  5. An approved start released while gateway admission is closed is now refused — justified (the pinned every-registration-site guard; 5→6 count updated)
  6. New error text for a stop while awaiting post-approval admission — justified
  7. Queue/reaper/reconfigure logs now report in-startup population — rides along
  8. ClaimPoint reservations counted in the startup population — justified
  9. Startup deadline pinned fixed, not pressure-aware (docs + tests) — justified
  10. AcpProvider/factory gain the two callback parameters (plumbing, 2 counted consumers of on_gate_queued: shared path and dedicated path) — justified

Checked: the bound duplicates nothing — spawn_min_memory_gb (default 4.0) is a per-spawn memory floor, not a count bound; grepped max_concurrent_startups|startup_cap, no prior mechanism. No pin deleted or reversed; the shared-path callback was moved, not changed. No lens-9 item: the fix restores behaviour a spec on base already pins.

[FIRST-PRINCIPLES-REVIEWED] 3c9ccd6

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 3c9ccd6116375675e1ca8f5aebc9bc99385fb92e and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/subagent_manager/run.py:1824 -- _note_startup_progress(info) runs before the first event arrives, so a hung stream evades startup accounting and the watchdog -> Fix: mark progress after receiving the first event.
FINDING -- src/kiro_crew/subagent_manager/admission/pump.py:794 -- _admit_released_start(info) can reject after _log_spawned(info) records an unstarted run as spawned -> Fix: log only after admission succeeds.
[GPT-REVIEWED] 3c9ccd6

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3c9ccd6116375675e1ca8f5aebc9bc99385fb92e: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 3c9ccd6116375675e1ca8f5aebc9bc99385fb92e — this comment is updated in place on each push.

Review details

Approved spawns that the user stops while parked are reported as a gateway admission failure.

FINDING — src/kiro_crew/subagent_manager/admission/pump.py:902 — _admit_released_start_impl collapses four distinct refusals into one return granted and not (info.done or info.user_stopped or info.reaped or info._reap_started), so a Stop or reap landing while an approved spawn waits on the in-startup bound is announced with "spawn rejected: the gateway closed admission before this approved spawn could start" and logged as reason="admission_closed", losing the neutral user-stop record → Fix: return the refusal reason (admission closed vs. already terminal) and widen the caller's if not info.done: at pump.py:795 to the same four flags so a terminal run takes the existing stop/reap path.

[OPUS-REVIEWED] 3c9ccd6

Verdict parsed from the review's SHA-scoped output markers for commit 3c9ccd6116375675e1ca8f5aebc9bc99385fb92e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3c9ccd6116375675e1ca8f5aebc9bc99385fb92e: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-startup-pressure branch from 2ed219d to b6acb08 Compare September 25, 2026 18:14
@bolichen97 bolichen97 changed the title fix(subagent): cap concurrent startups, pressure-aware start deadline fix(subagent): cap concurrent startups, reset start clock at gate exit Sep 25, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Sep 25, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-startup-pressure branch from b6acb08 to 302191a Compare September 25, 2026 18:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-startup-pressure branch from 302191a to 80d3395 Compare September 25, 2026 19:07
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-startup-pressure branch 2 times, most recently from 9c41f66 to 40aaf77 Compare September 25, 2026 20:39
@bolichen97
bolichen97 force-pushed the fix/subagent-startup-pressure branch from 40aaf77 to e0902ee Compare September 25, 2026 21:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
A fan-out was bounded by two quantities: subagent_spawn_stagger_secs bounds
the RATE of starts and _max_concurrent bounds the RUNNING population.
Nothing bounded how many admitted agents were IN STARTUP at once, and the
dedicated-process path charged its SessionStartGate queue time to the startup
watchdog. On a 623-item fan-out, waves of 24-45 lost ~2% and a wave of 120
lost ~50%, every loss a healthy start recorded as "Failed to start within
120s".

Gate clock on both paths: runtime.create_session fires on_gate_queued
immediately before the wait for a SessionStartGate permit and on_gate_acquired
at gate exit. _run_inner passes _gate_wait_mark(info) / _gate_exit_reset(info)
into get_or_create (they ride the provider factory -- NAMED parameters in _acp
-- into AcpProvider and from there into the process's own create_session) and
_create_shared_session hands the same pair to the parent runtime. The mark
stamps _gate_wait_started and the startup watchdog reads the clock as frozen
at that moment; the reset clears it and restarts the clock. Invariant: the
startup deadline measures time spent starting -- before the gate and with a
permit held -- and never time queued for a permit, however long the queue. The
wait is finite because every holder is on a running clock from acquisition and
is reaped at the base deadline if session/new has not returned, the request has
its own budget, and the gate keeps a collector-free headroom of permits. One
definition per callback serves both paths.

In-startup bound: _startup_cap() = 2 x session_start_concurrency, clamped to
[1, cap] (so 1 rather than 0 at a cap of 0 -- the running cap pauses admission
there). The bound is tied to the session-start gate and NOT to the running
cap because the gate is the one resource every start in startup contends for:
G permits serve G starts per round, so 2G (one round holding, one admitted to
take over) is the smallest value that never idles the gate and admitting more
buys no starts -- only a longer queue of idle admitted starts, whose queue time
the frozen clock does not charge. A cap-sized bound (ceil(cap / 4) at cap 64
is 16 against 2 permits) queues seven rounds for two permits. There is no config key for the bound: 2G is both floor and ceiling of the
useful range (below idles the gate, above adds only pre-permit wait), so an
override could only make it worse, and agent.session_start_concurrency is
already the operator's lever -- the bound tracks it. Earlier revisions of this
branch used max(2 x G, ceil(cap / 4)) (16 at cap 64 against a 2-permit gate,
seven rounds of pre-permit wait) and shipped the bound as
agent.subagent_max_concurrent_startups with 0 = derive; both are dropped.
The population the bound is checked against is the agents actually STARTING
(_exec_started set, no PID / stream / turn) plus a ClaimPoint reservation not
yet re-entered and registered. An agent parked at the spawn-approval prompt is
NOT counted: it starts nothing, and counting it would let N unanswered prompts
stall every spawn on the host, auto-approved ones from unrelated parents
included. Its RELEASE is what is bounded, because a bulk trust/yolo grant
resolves every pending prompt in one pass: _spawn_with_approval no longer goes
straight to _run but awaits _admit_released_start, which appends a resident
entry (_resume_id + _startup_release, so every unstarted-spawn scan already
leaves it alone) to the EXISTING queue; the pump's first phase
(_release_admitted_start, ahead of the capacity check since the run already
holds its slot) meters one released start per pass under the same stagger and
in-startup checks a fresh spawn faces and stamps the stagger clock; it answers
"released" / "held" / "" and the pass continues on every answer, so a held
release never withholds a resume grant. A start released while gateway
admission is closed is refused, the same guard every registration in the
admission package sits behind. A stop
while waiting wakes the waiter with False and drops the entry. The watchdog
keeps reaping only the executing population (a human prompt has no deadline).
_should_stagger_queue gains a third clause and the drain pump holds its pick
under the same test; spawns wait in the EXISTING queue. The queue wakes on a
runtime PID or first stream (_note_startup_progress) and on the slot-release
drain of every terminal, including the watchdog's reap of a wedged start.

The watchdog deadline itself stays the fixed _startup_deadline. A
pressure-aware deadline (base + base/8 per other agent in startup, 3x
ceiling) was in an earlier revision of this branch and is withdrawn: the
measurement above predates both changes here, so nothing shows a healthy
start still missing the base deadline once queue time stops being charged
and the crowd is bounded; and the term was not monotonic -- sampled at sweep
time against now - _exec_started, which spans the whole crowded period, it
shrank as the crowd drained and could reap at one sweep an agent the previous
sweep had left inside its window. The reaper's warning keeps the in-startup
population as diagnostics only.
… clock

dynamic-subagent-sizing.md: describe the in-startup bound (2 x the session-start
gate's width, clamped to [1, cap] so 1 at a cap of 0), say why it is gate-tied
rather than cap-tied (the pre-permit wait the gate-exit reset cannot exclude)
and why it has no config key (2G is floor and ceiling of the useful range;
session_start_concurrency is the lever), correct the sentence that called the cap a startup count
limit (it bounds the running population), and state the gate-clock invariant
(the startup deadline never counts time queued for a permit: the clock freezes
at gate entry and restarts at acquisition) and why the wait is finite. Record
that the deadline stays fixed and why a pressure-aware one was withdrawn.
subagent.md: the admission invariant (including why an approval-parked agent
is NOT in the bounded population and how its release is metered through the
pump, and why an unregistered reservation is), the dedicated-path
gate-exit reset and the fixed-deadline invariant.
@bolichen97
bolichen97 force-pushed the fix/subagent-startup-pressure branch from e0902ee to 3c9ccd6 Compare September 25, 2026 22:23
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@iamwhatever
iamwhatever merged commit ba5d430 into main Sep 25, 2026
100 of 102 checks passed
@iamwhatever
iamwhatever deleted the fix/subagent-startup-pressure branch September 25, 2026 23:10
@buluoray

Copy link
Copy Markdown
Contributor

Post-merge record: this landed while I was composing the write-up, so the verification below finished after it merged and the approve it was headed for would have gated nothing. Posting it as a record instead.

Verdict had it still been open: approve. No blocking findings. The one real gap is a missing test, which is now a follow-up on main rather than a pre-merge ask.

Verified against source at this head, not the description

  • The callback chain is intact end to end, including the trap the description names. _acp declares on_gate_acquired / on_gate_queued as NAMED parameters (config/loader.py:5675-5676) rather than letting **_kwargs swallow them, forwards both to AcpProvider (:5816-5817), which stores them (providers/acp.py:434,437) and hands them to its own create_session (:1286-1287). runtime.create_session fires on_gate_queued immediately before the permit wait and on_gate_acquired(permit.queue_wait_ms) after it (acp/runtime.py:6163-6173), each inside its own try/except so a raising callback cannot break a start. load_session is not given them, which is correct — a resume takes no permit.
  • The freeze/reset pair is one definition for both paths, and the reasoning in _gate_wait_mark_impl's docstring is the correct one: a holder's clock resets at acquisition, so without freezing the waiter you can reap a healthy start exactly as it is about to be served. Withdrawing the pressure-aware deadline in favour of this is the right trade — the withdrawn term was non-monotonic (sampled at sweep time, compared against a span), and dissolving it beats patching it.
  • _startup_reservations rides an existing invariant rather than inventing a second one. It is incremented at precisely the reserve-then-commit site that increments _running_count (admission/gate.py:1082) and decremented at precisely the two sites that give that reservation back — release_reservation (admission/pump.py:415-417) and the _dispatch_now registration hand-off (gate.py:1180-1182). A leak in the new counter would therefore require a leak in _running_count, which would already starve admission today. This is what makes a plain int counter acceptable for a value whose over-count would wedge admission.
  • 2 × session_start_concurrency is argued correctly and the no-config-key decision is right. The gate serves G per round however many queue behind it, so anything above 2G buys no starts and only grows a population of admitted-but-idle starts — each a live process on the dedicated path. A cap-derived term really would admit 16 against a 2-permit gate at cap 64. session_start_concurrency is the operator's existing lever and the bound tracks it.
  • Tests: 223 passed across test_subagent_startup_pressure.py, test_subagent_startup_watchdog.py, test_session_start_gate.py, test_headless_memory_recognition.py, test_update_check_install_aware.py on the merge ref. Mutation-verified two of the three load-bearing guards: forcing startup_full = False in _should_stagger_queue_impl reddens 6 tests, and disabling the _gate_wait_started stamp reddens 3 (including test_the_clock_does_not_run_while_queued_for_a_permit). Both guards are real.

🟡 [Test coverage] Follow-up on main: the one guard standing between this change and a silent wedge has no test

Deleting all three _note_startup_progress(info) call sites (subagent_manager/run.py:1540, :1824, :3285) leaves the suite fully green — 77 passed, zero failures.

Why that matters more here than a typical coverage gap: those calls are the only edge that announces a non-terminal exit from startup. _note_startup_progress's own docstring says so — "the slot-release drain fires only on a terminal, and the pump does not poll" — and I confirmed it: the only other _drain_queue reachable from the monitoring module is taskq_on_wake (monitoring.py:238), which is a coordinator wake for a parked row, not a periodic sweep. So there is no timer backstop for the main queue hold, unlike the released-start path, which you deliberately gave _RELEASE_REPUMP_SECS.

Consequence chain if one of those edges is ever lost to a refactor (moving where _pid is assigned would do it): a fan-out wider than the bound — 4 at the stock gate width — sits queued with free slots and nothing to move it until an unrelated terminal or a new arrival happens. That is a permanent silent hang, which is strictly worse than the false reap this PR fixes, and no test would go red.

test_queue_holds_at_the_startup_cap_and_resumes_on_progress and test_bulk_approval_cannot_release_more_than_the_startup_cap both describe themselves as admitting the next start "on PID progress", but the mutation shows neither actually depends on the wake edge firing — they reach the pump another way. The hold half is well covered; the wake half is not covered at all.

Suggestion: one test that sets _pid through the real _run_inner PID-record path with a spawn queued behind a full startup bound, asserting the queued spawn starts without any explicit _drain_queue() call — and confirm it reddens when the :1540 call is removed. The turns == 0 first-stream site at :1824 deserves the same, since that is the only exit for a provider that creates its child lazily.

Not blocking, and not a reason to revert: the calls are present and correct today, and the PR's "no deadlock" argument is sound as written. I am flagging it because that argument is currently an argument rather than a pinned invariant, and it is the half that fails toward a hang.

Also noted, no action asked

A spawn_continue run takes no gate permit (session/load) yet still matches _in_startup and so occupies one of the bound's slots while contending for nothing. It does spawn a process on the dedicated path, so charging it is defensible; mentioning it only so it is a known consequence rather than a surprise if resume-heavy workloads ever look slower to admit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants