Repository navigation
fix(subagent): price a sub-agent start at its start cost - #14800
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The burst under-reserve #13294 guarded against returns on heavy-settling hosts, and the compensating post-start guard is deferred to a change nobody has filed. WatchThe deleted code carries its own incident record: "Pricing it at the 0.5 GB fallback while the store already knew a ~6 GB p90 let four starts each clear a raw free-memory check and then grow into the same headroom together" (the fan-out that exhausted a 64 GB host). The PR's rebuttal — a settled kiro-cli runtime is ~0.5 GB — is measured, but its own numbers say ~1.5 GB with an MCP roster, so a fan-out on such a host is under-reserved ~3× inside the first-sample window, with the remedy a knob operators learn about after the OOM. The description defers the mitigation ("Guarding heavy work a sub-agent runs after it starts. That is a separate change") with no issue linked. SuggestionsThe sweep already reads each worker's RSS at settle; recording that as a second sample kind and pricing warming starts at its p90 fixes the root defect (wrong quantity sampled — lifetime subtree peak) while keeping per-host adaptivity, instead of abandoning learning for the reserve. [DESIGN-REVIEWED] ef4e6f9 Description read[DESCRIPTION-READ] 5742496d0cbb8c57a9ee4d9395a1ee3494219f10dd50a02ee4053b3265c483ad That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Reinstates the flat 0.5 GB start price a deleted pin blamed for exhausting a 64 GB host, while the promised post-start-growth guard has no tracked issue. Not justified as shipped
What this change shipsInventory (6 items) — 5 justifiedIntent: stop pricing a sub-agent start at a run's lifetime whole-tree peak so spawns admit when start memory is free — a FIX.
[FIRST-PRINCIPLES-REVIEWED] ef4e6f9 Description read[DESCRIPTION-READ] 6716ac00242e45e13e80c02fc6b662717f01969d20fc7c69698a3fbe78dcdd33 That sha256 names the PR description this verdict read. Recompute it with |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/config/sections.py:1579 -- with False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNothing blocks; one added spec sentence contradicts the new reserve arithmetic. FINDING — src/kiro_crew/docs/dynamic-subagent-sizing.md:262 — the added paragraph says "a settled worker owes nothing" and then "Parents waiting without a slot retain their reservation", but [OPUS-REVIEWED] ef4e6f9 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
4c16720 to
88aeffb
Compare
|
|
|
|
|
|
|
|
|
88aeffb to
59e1ee8
Compare
|
|
|
|
|
|
59e1ee8 to
9c7d2ed
Compare
|
Review from a reproduction of the reported bar, on 1. It fixes the reported estimate (verified)A user report showed
2. Linux measurement for First Principles' "Clears when"That review asks for the settled-runtime figure on a Linux host. I measured settled kiro-cli 2.26.0 ACP runtimes on a 32-core, 123 GB Linux host by summing
So 0.5 GB holds for a bare runtime on Linux, which backs the macOS figure. A runtime with a heavy MCP roster settles at about 3x that. The 4 GB floor absorbs the difference for a small burst, and -A settled kiro-cli runtime with its MCP servers measured 0.3–0.6 GB resident on a macOS host.
+A settled kiro-cli runtime measured 0.3–0.6 GB resident on a macOS host and 0.47 GB on Linux
+with no MCP servers; with a roster of ~17 MCP-server processes it settled at ~1.5 GB on Linux.
+Raise `agent.subagent_cost_gb` toward that figure when agents carry a heavy MCP roster.For comparison, the same host's own 3. Remaining effect, outside this PR's scope: the same samples still pin the auto cap
After this PR the start bar is fixed, but a fan-out on that host is still slot-queued at the floor of 3, and the cause is the same whole-tree peak. "Not a goal" in the description already excludes this, so it isn't a blocker here. Could it get a linked follow-up issue? The issue's own direction covers it: record the agent's own processes, not the whole tree. 4. Suggested regression test: pin the reported numberThe new tests use a synthetic heavy settled worker. Nothing in this PR ties the fix to the cost log that produced the reported bar. This test drives class TestALearnedWholeTreePeakNeverPricesAStart:
"""#13489: a cost log whose p90 is a whole-tree peak must not raise the start bar."""
def test_a_132_gb_learned_p90_leaves_the_bar_at_floor_plus_start_cost(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
import json
import time
from kiro_crew.subagent import SubagentManager
monkeypatch.setenv("KIROCREW_HOME", str(tmp_path))
log = tmp_path / "subagents" / "cost_samples.jsonl"
log.parent.mkdir(parents=True)
now = time.time()
log.write_text(
"".join(
json.dumps({"agent": "kirocrew", "mem_gb": v, "cpu_cores": 1.0, "ts": now - i})
+ "\n"
for i, v in enumerate([1.2] * 5 + [132.3] * 5)
)
)
sessions = MagicMock()
sessions.get_agent_selection.return_value = ("template", "")
mgr = SubagentManager(
sessions=sessions, ctx_builder=MagicMock(), on_done=MagicMock(), max_concurrent=3
)
# Where the 0.8.0 reaper published that log's p90 for the gate to read.
mgr._learned_costs_gb = {"kirocrew": 132.3}
asked: list[float] = []
def _check(min_gb: float) -> tuple[bool, float]:
asked.append(min_gb)
return (85.9 >= min_gb, 85.9)
with (
patch("kiro_crew.subagent.check_memory_available", side_effect=_check),
patch("kiro_crew.subagent.KiroCrewConfig") as mock_cfg,
patch("kiro_crew.subagent.cached_admission_check", return_value=_admitted()),
patch("kiro_crew.subagent.sel"),
):
mock_cfg.load.return_value.agent.spawn_min_memory_gb = 4.0
mock_cfg.load.return_value.agent.subagent_cost_gb = 0.5
info = mgr.spawn(task="t", parent_session_key="s")
assert asked == [pytest.approx(4.5)]
assert info is not None and info.queued is FalseWithout the 5. The only red check is inherited
BackportThis head, #14722 and #14661 all cherry-pick onto |
* fix(subagent): backport start-cost pricing to release/0.8.0 Price every warming start at agent.subagent_cost_gb instead of the learned p90 or a live peak, and remove the learned-cost refresh only the gate read. On 0.8.0 a cost log whose p90 was a whole-tree peak (build and test children included) set the start bar at floor + p90: a reported host with 85.9 GB free was told a sub-agent needed 136.3 GB. Backport of #14800, applied without conflict resolution. (cherry picked from commit 2399227) Co-authored-by: Joe Guo <zejiangg@amazon.com> * fix(subagent): backport page-cache headroom and queued-count fixes Count a cgroup's inactive page cache as memory headroom, so a build-heavy host does not read ~0 GB free and defer every start as memory-critical, and re-emit the queued count once a claimed row starts, so the chip stops showing "1 waiting" with a stale low-memory reason. Backport of #14722 and #14661, applied without conflict resolution. (cherry picked from commit 62fa20d) (cherry picked from commit 17981aa) Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com> Co-authored-by: Joe Guo <zejiang.guo@gmail.com> --------- Co-authored-by: Joe Guo <zejiangg@amazon.com> Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com> Co-authored-by: Joe Guo <zejiang.guo@gmail.com>
2399227 to
c8fa832
Compare
|
@bolichen97 thanks, this is very useful. All handled in
Rebased: main fixed |
Price every warming start at agent.subagent_cost_gb instead of the learned p90 or a live peak, and remove the learned-cost refresh only the gate read.
c8fa832 to
ef4e6f9
Compare
With max_subagents = 0 the auto cap divided free memory by the heaviest agent's own lifetime p90, so one agent whose window was 12% release builds priced every slot on a 171.8 GB host and held the cap at 4 to 8. kirodotdev#14800 already prices every warming start at agent.subagent_cost_gb because a lifetime subtree peak is not a start cost. The subagent cap now prices one slot at max(subagent_cost_gb, p90 of the last 100 cost samples across every agent). With no history that is subagent_cost_gb, and the cap never exceeds what that price alone gives. The reaper re-sizes an auto cap from the latest samples on every sweep, so heavier recent runs lower it and it rises back as they age out, with no restart or reload; an explicit max_subagents pin is never touched and a live reload landing during the read wins. The task runner's cap and the warm-pool reserve keep the worst-case price (read_learned_cost), because those starts do not pass the spawn gate. Fixes kirodotdev#14013
With max_subagents = 0 the auto cap divided free memory by the heaviest agent's own lifetime p90, so one agent whose window was 12% release builds priced every slot on a 171.8 GB host and held the cap at 4 to 8. kirodotdev#14800 already prices every warming start at agent.subagent_cost_gb because a lifetime subtree peak is not a start cost. The subagent cap now prices one slot at max(subagent_cost_gb, p90 of the last 100 cost samples across every agent). With no history that is subagent_cost_gb, and the cap never exceeds what that price alone gives. The reaper re-sizes an auto cap from the latest samples on every sweep, so heavier recent runs lower it and it rises back as they age out, with no restart or reload; an explicit max_subagents pin is never touched and a live reload landing during the read wins. The task runner's cap and the warm-pool reserve keep the worst-case price (read_learned_cost), because those starts do not pass the spawn gate. Fixes kirodotdev#14013
Problem / Motivation
Goal: A sub-agent starts when there is enough memory to start it.
Since #13294, users see
waiting to start — needs 4.5GB of free memory, 4.2GB free nowmore often, and the bar can climb past 10 GB. The spawn guard prices every warming start at the learned p90 of past runs, or at the biggest live worker's peak. Those numbers measure a run's whole process tree over its whole life, test suites and builds included. A fresh runtime needs a fraction of that.Why it matters
On a laptop, ordinary spawns queue for memory the start will never use. Deferred runs record no new samples, so a high p90 can hold the bar above what the host ever clears.
Not a goal
compute_max_subagentsstill reads them).What changed (motivation → approach → change)
Symptom: spawns wait for 4.5–10+ GB free. Cause: the start price is a lifetime peak, not a start cost. Change: price the start at its start cost.
src/kiro_crew/subagent.py_startup_memory_reserve_gb,src/kiro_crew/subagent_manager/admission/gate.py). Every warming start is priced atagent.subagent_cost_gb(0.5 GB by default) less what it already holds. A settled worker owes nothing; its memory is already in the free reading. One spawn needsspawn_min_memory_gb + subagent_cost_gbfree, 4.5 GB by default. The deferral message prints the need with one decimal and names the start price (need 4.5 GB (0.5 GB per warming start)), not a learned one.src/kiro_crew/subagent_manager/monitoring.py,subagent.py,subagent_cost.py). The learned-cost refresh only the gate read (_refresh_learned_cost,_learned_costs_gb,_startup_cost_gb,_effective_next_start_gb) and the now-unusedlearned_cost_forandcost_log_identityhelpers.src/kiro_crew/config/sections.py,config-baseline.json).agent.subagent_cost_gbis now described as the free memory each start must find, and still the cap-sizing fallback until a learned value accumulates.mem_gbsample waspeak_rss_gb: the highest/procsubtree RSS seen over the whole run (_proc_subtree_sampleinsubagent_manager/monitoring.py, recorded by_record_cost_implat run end). A subtree peak includes every child the run spawns, so a run that launched a test suite or build recorded that child's memory as its price.docs/system-specs/modules/subagent.md,adaptive-concurrency.md,dynamic-subagent-sizing.mdandsrc/kiro_crew/docs/subagents.mddescribe the new price and theagent.subagent_cost_gbknob. A settled kiro-cli runtime measured 0.3–0.6 GB on macOS and 0.47 GB on Linux with no MCP servers; with ~17 MCP-server processes it settled at ~1.5 GB on Linux, which is when to raise the knob.Backwards compatibility
Compatible: nothing is rejected or renamed. Spawns admit at a lower bar than today. The deferral SEL record drops
learned_cost_gb.Tests
test/test_admission_gate.py: the pending start is priced at the configured cost, even with a heavy settled worker live; the deferral reports what it saw and needs; a cost log whose learned p90 is 132.3 GB (the reportedneeds 136.3GBbar) still asks for 4.5 GB.test/test_adaptive_startup_memory.py: a peak never prices a start; a settled worker owes nothing; learned-price and refresh tests removed with the code.test/test_subagent_reap_race.py,test/test_subagent_startup_pressure.py: the respawn reserve check uses the new signature.Manual verification
N/A — unit coverage exercises the gate arithmetic; no UI changed.
Related Issues
Follow-up to #13294.
no linked issue: this follow-up to #13294 closes no tracked issue.
Pattern harvest
Not generalizable: a one-off pricing choice. The start guard read a lifetime measure (peak RSS of a whole process tree) as a start cost.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)