Repository navigation
fix(subagent): size the auto cap from a typical run, not the heaviest agent's peak - #14015
Pearcekieser wants to merge 1 commit into
Conversation
|
Intent: When |
4718d81 to
8175327
Compare
Design Review (Fable 5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The pooled p90 can hide the heavy tail entirely, removing the last static bound on aggregate memory over-commit — the OOM the old divisor priced in. WatchIn the PR's own kill case ( [DESIGN-REVIEWED] 024d3dc Description read[DESCRIPTION-READ] 83caf8b946bb0caa9d217e0963fbabe30efea1b6bc5c10d72f77ce350096a7bd That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the live gates alone suffice when a heavy agent that runs less often than every 40 runs stays permanently outside the recent window pricing the cap. Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: FIX — under auto, size the subagent cap from a typical run's cost instead of the heaviest agent's lifetime p90 (Fixes #14013; premise verified on base: the spawn gate already prices every start at
[FIRST-PRINCIPLES-REVIEWED] 024d3dc Description read[DESCRIPTION-READ] 66d1a488870c7037dd1ccaae6137aab16d9523ce5c78e520c24587c47dbd6dee That sha256 names the PR description this verdict read. Recompute it with |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5 Review (fork) — ✅ no blocking findingsReviewed Review details
FINDING — src/kiro_crew/docs/dynamic-subagent-sizing.md:148 — the new [OPUS-REVIEWED] 024d3dc |
8175327 to
94ad1de
Compare
|
Pushed
The earlier |
94ad1de to
052a755
Compare
|
Pushed |
052a755 to
7a35f11
Compare
|
Pushed |
7a35f11 to
2d31979
Compare
|
Pushed
Locally, 475 targeted tests pass, black, flake8 and isort are clean, and mypy is clean on the changed module. The PR body is updated to match. |
3fc016c to
2fe036d
Compare
|
Pushed |
|
@iamwhatever (author of #14800) and maintainers, a direction question before I push again. #14800 changes the ground this PR stands on. Every revision here priced the auto cap's slots at a typical run, because the spawn reserve priced each start at that agent's own learned p90. Most of the GPT 5.6 rounds were about keeping those two prices consistent. #14800 removed the learned per-agent start price. Every warming start is now reserved at I see three ways to go:
I have a local revision ready for the pre-#14800 base and will not push it. Which direction do you want? For the record, the reds on
|
2fe036d to
af94f94
Compare
|
Pushed The auto subagent cap now starts from On the open findings:
Locally 2,829 tests pass across the subagent, sizing, cost, adaptive, admission, monitoring, task-runner and resource suites. black ( |
af94f94 to
0500432
Compare
|
Pushed |
0500432 to
9931936
Compare
|
Pushed
Locally, 648 tests pass across the hot-reload, sizing, cost, resource-endpoint, subagent-coverage and task-runner suites. black ( |
9931936 to
44700e0
Compare
|
Pushed
All Linux backend shards, the other seven Windows shards, lint and the remaining E2E jobs passed on |
|
Pushed |
|
Pushed |
|
Pushed
Locally, the sizing, cost, hot-reload, resource-endpoint, subagent-coverage, task-runner and session-export suites pass on the new base. |
|
Pushed
Locally, the sizing, cost, hot-reload, resource-endpoint, subagent-coverage and task-runner suites pass (649 tests). black (
|
With max_subagents = 0 the auto cap divided free memory by the heaviest agent's own lifetime p90, so one agent whose window was 12% release builds priced every slot on a 171.8 GB host and held the cap at 4 to 8. kirodotdev#14800 already prices every warming start at agent.subagent_cost_gb because a lifetime subtree peak is not a start cost. The subagent cap now prices one slot at max(subagent_cost_gb, p90 of the last 100 cost samples across every agent). With no history that is subagent_cost_gb, and the cap never exceeds what that price alone gives. The reaper re-sizes an auto cap from the latest samples on every sweep, so heavier recent runs lower it and it rises back as they age out, with no restart or reload; an explicit max_subagents pin is never touched and a live reload landing during the read wins. The task runner's cap and the warm-pool reserve keep the worst-case price (read_learned_cost), because those starts do not pass the spawn gate. Fixes kirodotdev#14013
|
Pushed Blocking finding, a torn Finding on Finding on the re-size log line showing only the effective cap: fixed. The line now logs the ceiling and the effective cap, each with its previous value. Locally, 705 tests pass across the sizing, cost, hot-reload, resource-endpoint, subagent-coverage, task-runner and session-export suites. black ( |
|
Thank you, @Pearcekieser, for this work and for the careful thinking behind sizing from a typical run. |
Problem / Motivation
Goal: When
max_subagentsis left at auto, the subagent cap reflects what a typical run on the host costs, so one agent's occasional release builds no longer hold the whole host to a handful of slots.With
agent.max_subagents = 0(auto, the default), a 171.8 GB host with little else running started with a subagent cap of 8, and recomputing gave 4. One agent's occasional release builds set the price of every slot.Why it matters
Auto is the default, so most installs get this cap. A host that fits dozens of workers queues work behind a ceiling of 4 to 8. The ceiling also flips between 4 and 32 across restarts, depending on whether one agent's last 50 runs happen to include a sixth build.
Not a goal
max_subagentsis set explicitly.What changed (motivation → approach → change)
Symptom: with
max_subagents = 0, the auto cap divided free memory by the heaviest agent's own lifetime p90. One agent whose window was 12% release builds (21 to 25 GB) priced every slot on a 171.8 GB host, so the cap was 4 to 8. Since #14800, the spawn gate prices every warming start atagent.subagent_cost_gb, on the argument that a lifetime subtree peak is not a start cost. The cap still used that peak.Approach: the subagent cap starts from the same price the gate reserves, and follows recent runs so it scales down on its own when the host's work gets heavier.
read_recent_costinsrc/kiro_crew/subagent_cost.pyreturns the p90 of the last 40 cost samples in the log across every agent (_RECENT_WINDOW, kept below the 50 per-agent samples compaction retains so an agent that stopped running cannot hold the window). It streams the log and holds only the window, so an old heavy run ages out once 40 newer runs have landed.compute_max_subagents(cfg, recent=True)insrc/kiro_crew/subagent.pyprices one slot atmax(agent.subagent_cost_gb, read_recent_cost("mem_gb")). With no history that issubagent_cost_gb, and the cap is never above whatsubagent_cost_gbalone would give.resolve_max_subagents(the subagent cap) uses it, and the dashboard'ssubagent_capreadout callsresolve_max_subagents, so it shows an explicit pin when one is set. The worst-case read for the warm pool runs only whensession.pool_size > 0.compute_max_subagents(cfg)is unchanged, so the task runner's parallel-step cap keeps the worst case (read_learned_cost, the max of per-agent p90s). The memory held back for the warm pool (session.pool_size) also keeps the worst case, because warm-pool runtimes and task-runner steps start without the spawn gate.SubagentManager._refresh_auto_capre-sizes an auto cap from the latest samples, and the reaper calls it on every sweep (src/kiro_crew/subagent_manager/monitoring.py). Heavier recent runs therefore lower the cap on the next sweep, and it rises back as they age out, with no restart or reload. An explicitmax_subagentspin is never touched. The re-size reads the config the live watcher has adopted, so a tornconfig.jsoncannot discard a pin, and a live reload that lands during the read wins.spawn_min_memory_gb, the posture gate and the adaptive controller.src/kiro_crew/docs/dynamic-subagent-sizing.mdanddocs/system-specs/modules/subagent.mddescribe the new divisor.Backwards compatibility
Compatible: no config key, API or file format changes, and an explicit
max_subagentspin is unaffected. Under auto, the subagent cap rises where one agent's peaks priced every slot, and it now moves between gateway restarts as recent runs change. The task runner's cap and the warm-pool reserve are unchanged.Tests
test/test_subagent_sizing.pyTestAutoCapFollowsRecentRuns: with no history the cap comes fromsubagent_cost_gb; a heavier recent window lowers it; heavy runs that aged out of the window raise it back; the cap never exceeds thesubagent_cost_gb-only value; the reported host gives 32 for the subagent cap and 5 for the task runner's worst case; the warm-pool reserve keeps the worst case.test/test_subagent_cost.py:read_recent_costreturns None without history, pools across agents, reads only the newest window, and skips unusable values.test/test_subagent_config_hot_reload.pyTestTheAutoCapFollowsRecentRuns: the reaper re-sizes the cap each sweep, down and back up; an explicit pin is never re-sized; a reload that lands during the read wins.test/test_resource_health_endpoint.py: the dashboard readout goes throughresolve_max_subagents.test_without_a_pool_the_worst_case_is_not_readchecks the worst-case read is skipped when there is no warm pool.Manual verification
N/A — unit coverage sufficient: the sizing tests drive
resolve_max_subagentsend to end from a real cost log file, and the reaper test drives the re-size through_reaper_loop.Related Issues
Fixes #14013
Pattern harvest
Rule candidate: review-prompt
Pattern: a count of concurrent slots priced at the worst-case member's lifetime peak instead of a recent, bounded window, when the per-start gate already prices a start at its start cost.
Checklist