Skip to content

fix(subagent): price a sub-agent start at its start cost - #14800

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/subagent-loose-start-heavy-gate
Sep 30, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/subagent-loose-start-heavy-gate

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A sub-agent starts when there is enough memory to start it.

Since #13294, users see waiting to start — needs 4.5GB of free memory, 4.2GB free now more often, and the bar can climb past 10 GB. The spawn guard prices every warming start at the learned p90 of past runs, or at the biggest live worker's peak. Those numbers measure a run's whole process tree over its whole life, test suites and builds included. A fresh runtime needs a fraction of that.

Why it matters

On a laptop, ordinary spawns queue for memory the start will never use. Deferred runs record no new samples, so a high p90 can hold the bar above what the host ever clears.

Not a goal

What changed (motivation → approach → change)

Symptom: spawns wait for 4.5–10+ GB free. Cause: the start price is a lifetime peak, not a start cost. Change: price the start at its start cost.

  • Start price (src/kiro_crew/subagent.py _startup_memory_reserve_gb, src/kiro_crew/subagent_manager/admission/gate.py). Every warming start is priced at agent.subagent_cost_gb (0.5 GB by default) less what it already holds. A settled worker owes nothing; its memory is already in the free reading. One spawn needs spawn_min_memory_gb + subagent_cost_gb free, 4.5 GB by default. The deferral message prints the need with one decimal and names the start price (need 4.5 GB (0.5 GB per warming start)), not a learned one.
  • Removed (src/kiro_crew/subagent_manager/monitoring.py, subagent.py, subagent_cost.py). The learned-cost refresh only the gate read (_refresh_learned_cost, _learned_costs_gb, _startup_cost_gb, _effective_next_start_gb) and the now-unused learned_cost_for and cost_log_identity helpers.
  • Help text (src/kiro_crew/config/sections.py, config-baseline.json). agent.subagent_cost_gb is now described as the free memory each start must find, and still the cap-sizing fallback until a learned value accumulates.
  • Why the old ~6 GB was not a start cost. The learned mem_gb sample was peak_rss_gb: the highest /proc subtree RSS seen over the whole run (_proc_subtree_sample in subagent_manager/monitoring.py, recorded by _record_cost_impl at run end). A subtree peak includes every child the run spawns, so a run that launched a test suite or build recorded that child's memory as its price.
  • Docs. docs/system-specs/modules/subagent.md, adaptive-concurrency.md, dynamic-subagent-sizing.md and src/kiro_crew/docs/subagents.md describe the new price and the agent.subagent_cost_gb knob. A settled kiro-cli runtime measured 0.3–0.6 GB on macOS and 0.47 GB on Linux with no MCP servers; with ~17 MCP-server processes it settled at ~1.5 GB on Linux, which is when to raise the knob.

Backwards compatibility

Compatible: nothing is rejected or renamed. Spawns admit at a lower bar than today. The deferral SEL record drops learned_cost_gb.

Tests

  • test/test_admission_gate.py: the pending start is priced at the configured cost, even with a heavy settled worker live; the deferral reports what it saw and needs; a cost log whose learned p90 is 132.3 GB (the reported needs 136.3GB bar) still asks for 4.5 GB.
  • test/test_adaptive_startup_memory.py: a peak never prices a start; a settled worker owes nothing; learned-price and refresh tests removed with the code.
  • test/test_subagent_reap_race.py, test/test_subagent_startup_pressure.py: the respawn reserve check uses the new signature.

Manual verification

N/A — unit coverage exercises the gate arithmetic; no UI changed.

Related Issues

Follow-up to #13294.

no linked issue: this follow-up to #13294 closes no tracked issue.

Pattern harvest

Not generalizable: a one-off pricing choice. The start guard read a lifetime measure (peak RSS of a whole process tree) as a start cost.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested a review from a team as a code owner September 28, 2026 18:48
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 28, 2026
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of ef4e6f913bcaafb3f16a16af4d42c09f72d576a6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The burst under-reserve #13294 guarded against returns on heavy-settling hosts, and the compensating post-start guard is deferred to a change nobody has filed.

Watch

The deleted code carries its own incident record: "Pricing it at the 0.5 GB fallback while the store already knew a ~6 GB p90 let four starts each clear a raw free-memory check and then grow into the same headroom together" (the fan-out that exhausted a 64 GB host). The PR's rebuttal — a settled kiro-cli runtime is ~0.5 GB — is measured, but its own numbers say ~1.5 GB with an MCP roster, so a fan-out on such a host is under-reserved ~3× inside the first-sample window, with the remedy a knob operators learn about after the OOM. The description defers the mitigation ("Guarding heavy work a sub-agent runs after it starts. That is a separate change") with no issue linked.
Clears when: a filed follow-up for the post-start growth guard is linked, or the warming price is derived from settle-time RSS rather than a flat constant.

Suggestions

The sweep already reads each worker's RSS at settle; recording that as a second sample kind and pricing warming starts at its p90 fixes the root defect (wrong quantity sampled — lifetime subtree peak) while keeping per-host adaptivity, instead of abandoning learning for the reserve.

[DESIGN-REVIEWED] ef4e6f9

Description read

[DESCRIPTION-READ] 5742496d0cbb8c57a9ee4d9395a1ee3494219f10dd50a02ee4053b3265c483ad

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of ef4e6f913bcaafb3f16a16af4d42c09f72d576a6 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Reinstates the flat 0.5 GB start price a deleted pin blamed for exhausting a 64 GB host, while the promised post-start-growth guard has no tracked issue.

Not justified as shipped

  • Item 1 — premise risk: the deleted pins recorded a decision — "priced at the 0.5 GB fallback a burst of ~6 GB dedicated runtimes each cleared the raw free-memory check and then grew into the same headroom together" ("the ~5 GB/session fan-out that exhausted a 64 GB host") — and this PR restores exactly that pricing. Its evidence that the old price measured the wrong thing is verified in-repo (peak_rss_gb is the lifetime whole-subtree high-water, monitoring.py:806, written by _record_cost_impl:828), but the burst-growth window that reserve bounded is now bounded by nothing after settle, and the deferred guard ("Guarding heavy work a sub-agent runs after it starts. That is a separate change") is, per the description itself, "no linked issue".
    Clears when: the deferred post-start-growth guard is a linked issue, or evidence the remaining 4 GB floor alone prevents the recorded fan-out incident.

What this change ships

Inventory (6 items) — 5 justified

Intent: stop pricing a sub-agent start at a run's lifetime whole-tree peak so spawns admit when start memory is free — a FIX.

  1. A spawn's memory bar is floor + configured 0.5 GB per warming start; learned p90s and live/settled peaks never raise it, and a settled worker owes nothing — premise risk (re-opens the OOM incident the deleted pins recorded; entry above)
  2. Deferral/refusal text prints the need to one decimal with the flat per-start price, dropping the price-source and reset remedies — justified
  3. The deferral SEL record drops learned_cost_gb — justified
  4. The learned-cost refresh machinery is deleted (_refresh_learned_cost, _learned_costs_gb, _startup_cost_gb, _effective_next_start_gb, learned_cost_for, cost_log_identity, read_learned_costs_checked, the 30-day horizon, dedicated-only reads; grepped src/ at HEAD: 0 remaining consumers; _cost_bucket and read_learned_cost keep theirs at monitoring.py:827, subagent.py:1965) — justified
  5. agent.subagent_cost_gb help text redefined as the per-start free-memory price — justified
  6. Subagent docs rewritten around the new price with measured settle figures — justified

[FIRST-PRINCIPLES-REVIEWED] ef4e6f9

Description read

[DESCRIPTION-READ] 6716ac00242e45e13e80c02fc6b662717f01969d20fc7c69698a3fbe78dcdd33

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of ef4e6f913bcaafb3f16a16af4d42c09f72d576a6 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/config/sections.py:1579 -- with spawn_min_memory_gb=0, “each sub-agent start must find” the configured cost is false because the memory guard is disabled -> Fix: qualify this as applying only while the memory guard is enabled.
[GPT-REVIEWED] ef4e6f9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ef4e6f913bcaafb3f16a16af4d42c09f72d576a6: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed ef4e6f913bcaafb3f16a16af4d42c09f72d576a6 — this comment is updated in place on each push.

Review details

Nothing blocks; one added spec sentence contradicts the new reserve arithmetic.

FINDING — src/kiro_crew/docs/dynamic-subagent-sizing.md:262 — the added paragraph says "a settled worker owes nothing" and then "Parents waiting without a slot retain their reservation", but _startup_memory_reserve_gb (src/kiro_crew/subagent.py:2001) sums gaps only for info in dedicated if info._rss_samples < _RSS_SAMPLES_TO_SETTLE, so a yielded parent past two sweeps — the normal case for a parent waiting on children — contributes 0.0 like any settled row, and an operator sizing headroom from this text (repeated in docs/system-specs/modules/subagent.md:184 and docs/system-specs/modules/adaptive-concurrency.md:120) over-estimates what the gate reserves → Fix: in all three sentences say a yielded parent is priced by the same warming/settled test as any other dedicated row (it is not dropped the way a queued/terminal row is), instead of "retain their reservation".

[OPUS-REVIEWED] ef4e6f9

Verdict parsed from the review's SHA-scoped output markers for commit ef4e6f913bcaafb3f16a16af4d42c09f72d576a6.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ef4e6f913bcaafb3f16a16af4d42c09f72d576a6: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 28, 2026
@iamwhatever
iamwhatever force-pushed the fix/subagent-loose-start-heavy-gate branch from 4c16720 to 88aeffb Compare September 28, 2026 21:11
@iamwhatever

iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=f4492d261210 Non-critical release can still exhaust cgroup memory

rebutted -- tradeoff recorded, needs a maintainer override to clear.
The tight-band case is real: a heavy command can start with 2-4 GB free.
No threshold closes it; holding at tight still admits a 6 GB command at 4.5 GB free.
The only fix is a per-command cost reserved ahead of time, i.e. the learned-p90 price this PR removes.
That price is what queued ordinary spawns behind 4.5-10+ GB on dev machines after #13294.
This head still holds heavy commands at CRITICAL; v0.7.1 held none.
Override reason: tight band is advisory by design; a per-command reserve is the p90 machinery #13294 over-blocked dev machines with.

@iamwhatever

iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=8ba58aa868ef Requirement message rounds 4.5 GB down to 4 GB

fixed in 88aeffb: both requirement messages in admission/gate.py format with one decimal (need 4.5 GB).

@iamwhatever

iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=d8c2ed1da6e2 Hold ignores agent.admission_gate: false

fixed in 88aeffb: hold_heavy_command returns at once when _gate_enabled(cfg) is false, the same switch admission_check honours.
The docs no longer name a config key that does not exist.

@iamwhatever

iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=01601ed122ca Hold runs above every deny branch

fixed in 88aeffb: the hold moved into the grant helper in subagent_manager/run.py.
It runs after every deny/reject branch and after human approval, so refused requests never wait and approval cards show at once.

@iamwhatever

iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=d8c2ed1da6e2 Heavy match is not anchored to the program

fixed in 88aeffb: is_heavy_command judges the program each segment runs, wrappers and launchers stripped.
grep -rn pytest test/ and git commit -m "make the build green" are not heavy; test_heavy_command_hold.py pins both.

@iamwhatever

iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=6bee470c85a2 Held commands release together

rebutted -- releasing one held command per poll would not bound the growth.
A test suite or build grows over minutes; a re-probe 10 s after one release cannot see that growth.
So staggering by one poll only delays the same simultaneous growth by seconds.
Bounding it needs a per-command cost known ahead, the reservation this PR removes (see the GPT tradeoff disposition).
The hold is back-pressure at CRITICAL, not a reservation, by design.

@iamwhatever

iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=228be4c87fef Hold vs ACP watchdog windows

rebutted -- the wait does not count as runtime idle.
acp/session_handle.py:1737-1744 sets _parked_since while the consumer holds a yielded event; parked time is consumer time, excluded from the dispatch idle clocks.
The hold runs inside that park, in the consumer's grant helper.
The 300 s HEAVY_COMMAND_HOLD_SECS also fits inside the 600 s default agent.tool_approval_timeout_secs approval window.

@iamwhatever

iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=52d56d517200 Drop the internal build tool name

fixed in 88aeffb: removed from the heavy-command list, its test and the shipped doc; internal-content-scan is clean.

@iamwhatever

iamwhatever commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=59c0190f295b Start price not justified by a measurement

rebutted -- measured.
Live kiro-cli ACP runtimes on a 48 GB macOS host, each with its full child tree (MCP servers included), via libproc RSS:
age 131 s: 0.56 GB; 525 s: 0.62 GB; 570 s: 0.29 GB; 650 s: 0.50 GB.
A settled runtime sits near subagent_cost_gb (0.5 GB), far below the ~6 GB p90 the old price charged.
The multi-GB peaks come from work run inside a session (test suites, builds), which is what the hold gates.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 28, 2026
@iamwhatever
iamwhatever force-pushed the fix/subagent-loose-start-heavy-gate branch from 88aeffb to 59e1ee8 Compare September 28, 2026 22:04
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=a61c5a0e98fc Quoted separators split a command

fixed in 59e1ee8: _COMMAND_SPLIT_RE is replaced by _split_commands, which splits on ;, &&, ||, | and newlines only outside quotes.
git commit -m 'fix; pytest later' is one segment and not heavy; test_heavy_command_hold.py pins it.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=d8c2ed1da6e2 Splitter has no quote state

fixed in 59e1ee8: _split_commands tracks single/double quotes and backslash escapes; an unbalanced quote keeps the rest as one segment and never raises.
git commit -m "fix: pytest hangs; make it bounded" is not heavy; echo "a;b" && pytest -q still is. Both pinned in test_heavy_command_hold.py.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=81c68aa04eef Heavy grants above CRITICAL can grow together

accepted residual, recorded in docs/system-specs/modules/subagent.md in 59e1ee8, as the Clears-when allows.
The hold checks posture at grant time only; several heavy commands granted above CRITICAL, or released together after the bound, can still grow together.
Bounding that needs a per-command cost reservation, which is the learned-p90 price this PR removes because it over-blocked dev machines.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=7186cd2bd83c Runtime and MCP growth above 0.5 GB

fixed in 59e1ee8: subagent.md names raising agent.subagent_cost_gb as the operator knob for heavier runtimes.
It cites the measured figure: a settled kiro-cli runtime with its MCP servers sits at about 0.3-0.6 GB resident on a macOS host.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=bf513283a54c Record fired holds in SEL

rebutted as out of scope: each fired hold already logs held a heavy command Ns for memory with the run id (subagent_manager/run.py).
That log answers whether the convoy occurs; a new SEL record type is a separate observability change, not needed for this PR's goal.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=2d12f35fa9e9 Fresh-runtime start cost not measured

rebutted -- measured on a real host, as the Clears-when asks.
Live kiro-cli ACP runtimes on a 48 GB macOS host, full child tree incl. MCP servers, via libproc RSS:
age 131 s: 0.56 GB; 525 s: 0.62 GB; 570 s: 0.29 GB; 650 s: 0.50 GB.
A fresh dedicated runtime settles near subagent_cost_gb (0.5 GB); the multi-GB peaks are work run inside a session, which the hold gates.

@iamwhatever
iamwhatever force-pushed the fix/subagent-loose-start-heavy-gate branch from 59e1ee8 to 9c7d2ed Compare September 28, 2026 22:37
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in files near this PR (acp/session_handle.py, acp/runtime.py, agent.py and others) since this head's CI started (1518434 -> cbe6d6a); CI re-running. 445 related tests pass locally on the new base.

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 29, 2026
@bolichen97

bolichen97 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Review from a reproduction of the reported bar, on 239922771458eaddb67dad2ac3ad9b9a789085a3. Not an approval; findings only.

1. It fixes the reported estimate (verified)

A user report showed waiting to start — needs 136.3GB of free memory, 85.9GB free now with 0 running, 1 waiting, 4 done. I reproduced that number exactly: a KIROCREW_HOME whose subagents/cost_samples.jsonl holds 5 dedicated kirocrew samples at 1.2 GB and 5 at 132.3 GB (whole-tree peaks), the real _refresh_learned_cost_impl, then spawn() with check_memory_available answering 85.9 GB free:

Tree learned map min_gb the guard asked for outcome
release/0.8.0 (64e850b) {'kirocrew': 132.3} 136.3 (4.0 floor + 132.3 p90) deferred low_memory
this PR (2399227) removed 4.5 admitted

2. Linux measurement for First Principles' "Clears when"

That review asks for the settled-runtime figure on a Linux host. I measured settled kiro-cli 2.26.0 ACP runtimes on a 32-core, 123 GB Linux host by summing /proc/<pid>/status VmRSS over each runtime's whole subtree:

Agent roster processes in subtree subtree RSS
no MCP servers 3 0.47–0.48 GB (2 runtimes)
~17 MCP-server processes (node and native) 20 1.45–1.57 GB (4 runtimes)

So 0.5 GB holds for a bare runtime on Linux, which backs the macOS figure. A runtime with a heavy MCP roster settles at about 3x that. The 4 GB floor absorbs the difference for a small burst, and agent.subagent_cost_gb is the right knob, so I'm not treating this as blocking. It would help if the docs gave the Linux figure next to the macOS one, so an operator knows when to raise the knob. Suggested wording for docs/system-specs/modules/subagent.md / src/kiro_crew/docs/dynamic-subagent-sizing.md:

-A settled kiro-cli runtime with its MCP servers measured 0.3–0.6 GB resident on a macOS host.
+A settled kiro-cli runtime measured 0.3–0.6 GB resident on a macOS host and 0.47 GB on Linux
+with no MCP servers; with a roster of ~17 MCP-server processes it settled at ~1.5 GB on Linux.
+Raise `agent.subagent_cost_gb` toward that figure when agents carry a heavy MCP roster.

For comparison, the same host's own cost_samples.jsonl (dedicated, last 50) gives the default bucket a median of 1.73 GB, a p90 of about 3.4 GB and a max of 10.23 GB. Those numbers are whole-tree peaks, which is what the PR's premise says.

3. Remaining effect, outside this PR's scope: the same samples still pin the auto cap

_host_mem_term still reads read_learned_cost("mem_gb"), which is the maximum per-bucket p90. With the cost log above and 93 GB available, on this PR's head:

compute_max_subagents -> 3   (mem_term = 0)    with the 132.3 GB samples
compute_max_subagents -> 32  (mem_term = 148)  after the log is removed

After this PR the start bar is fixed, but a fan-out on that host is still slot-queued at the floor of 3, and the cause is the same whole-tree peak. "Not a goal" in the description already excludes this, so it isn't a blocker here. Could it get a linked follow-up issue? The issue's own direction covers it: record the agent's own processes, not the whole tree.

4. Suggested regression test: pin the reported number

The new tests use a synthetic heavy settled worker. Nothing in this PR ties the fix to the cost log that produced the reported bar. This test drives spawn() with that log on disk and the p90 published where 0.8.0 read it. I checked it both ways: on this head it passes, and on release/0.8.0 it fails with assert [136.3] == [4.5 ± 4.5e-06].

class TestALearnedWholeTreePeakNeverPricesAStart:
    """#13489: a cost log whose p90 is a whole-tree peak must not raise the start bar."""

    def test_a_132_gb_learned_p90_leaves_the_bar_at_floor_plus_start_cost(
        self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
    ) -> None:
        import json
        import time

        from kiro_crew.subagent import SubagentManager

        monkeypatch.setenv("KIROCREW_HOME", str(tmp_path))
        log = tmp_path / "subagents" / "cost_samples.jsonl"
        log.parent.mkdir(parents=True)
        now = time.time()
        log.write_text(
            "".join(
                json.dumps({"agent": "kirocrew", "mem_gb": v, "cpu_cores": 1.0, "ts": now - i})
                + "\n"
                for i, v in enumerate([1.2] * 5 + [132.3] * 5)
            )
        )
        sessions = MagicMock()
        sessions.get_agent_selection.return_value = ("template", "")
        mgr = SubagentManager(
            sessions=sessions, ctx_builder=MagicMock(), on_done=MagicMock(), max_concurrent=3
        )
        # Where the 0.8.0 reaper published that log's p90 for the gate to read.
        mgr._learned_costs_gb = {"kirocrew": 132.3}
        asked: list[float] = []

        def _check(min_gb: float) -> tuple[bool, float]:
            asked.append(min_gb)
            return (85.9 >= min_gb, 85.9)

        with (
            patch("kiro_crew.subagent.check_memory_available", side_effect=_check),
            patch("kiro_crew.subagent.KiroCrewConfig") as mock_cfg,
            patch("kiro_crew.subagent.cached_admission_check", return_value=_admitted()),
            patch("kiro_crew.subagent.sel"),
        ):
            mock_cfg.load.return_value.agent.spawn_min_memory_gb = 4.0
            mock_cfg.load.return_value.agent.subagent_cost_gb = 0.5
            info = mgr.spawn(task="t", parent_session_key="s")

        assert asked == [pytest.approx(4.5)]
        assert info is not None and info.queued is False

Without the _learned_costs_gb line, the test also passes on release/0.8.0. The gate reads only what the reaper published, not the log itself, so that line is what lets the test catch the old behavior.

5. The only red check is inherited

Backend Tests (Windows) (1) fails at test/test_apps_registry_coverage.py:886 and :3235 (symlink loop / link depth). Main fixed both in #15081 and #15065, which are not in this head's base. git merge-tree origin/main against this head merges cleanly, so a rebase should clear the check and the one PR Readiness item that depends on it.

Backport

This head, #14722 and #14661 all cherry-pick onto release/0.8.0 without conflicts. The related suites pass there: 5,962 passed and 34 skipped across the subagent, config, cost, admission, sizing and docs tests. The backport is #15247, against release/0.8.0. It tracks this PR and is marked not to merge before this lands on main, and I'll re-cut it from the merged commit if review changes this.

bolichen97 added a commit that referenced this pull request Sep 30, 2026
* fix(subagent): backport start-cost pricing to release/0.8.0

Price every warming start at agent.subagent_cost_gb instead of the learned
p90 or a live peak, and remove the learned-cost refresh only the gate read.
On 0.8.0 a cost log whose p90 was a whole-tree peak (build and test children
included) set the start bar at floor + p90: a reported host with 85.9 GB free
was told a sub-agent needed 136.3 GB.

Backport of #14800, applied without conflict resolution.

(cherry picked from commit 2399227)
Co-authored-by: Joe Guo <zejiangg@amazon.com>

* fix(subagent): backport page-cache headroom and queued-count fixes

Count a cgroup's inactive page cache as memory headroom, so a build-heavy
host does not read ~0 GB free and defer every start as memory-critical, and
re-emit the queued count once a claimed row starts, so the chip stops
showing "1 waiting" with a stale low-memory reason.

Backport of #14722 and #14661, applied without conflict resolution.

(cherry picked from commit 62fa20d)
(cherry picked from commit 17981aa)
Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com>
Co-authored-by: Joe Guo <zejiang.guo@gmail.com>

---------

Co-authored-by: Joe Guo <zejiangg@amazon.com>
Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com>
Co-authored-by: Joe Guo <zejiang.guo@gmail.com>
@iamwhatever
iamwhatever force-pushed the fix/subagent-loose-start-heavy-gate branch from 2399227 to c8fa832 Compare September 30, 2026 05:28
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 30, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

@bolichen97 thanks, this is very useful. All handled in c8fa832669, rebased onto main 272cef4480:

Item Result
1. Reported 136.3 GB bar Thanks for the reproduction.
2. Linux figures in docs fixed: docs/system-specs/modules/subagent.md now gives the macOS, bare-Linux (0.47 GB) and heavy-MCP (~1.5 GB) figures and says when to raise agent.subagent_cost_gb.
3. Auto cap still pinned by whole-tree peaks accepted-and-deferred: #15298 (out of this PR's goal, listed under "Not a goal").
4. Regression test pinning the reported number fixed: added as TestALearnedWholeTreePeakNeverPricesAStart in test/test_admission_gate.py, with the _learned_costs_gb line kept so it fails on 0.8.0.
5. Windows (1) red Rebased onto main, which carries #15081 and #15065. CI re-running.

Rebased: main fixed Backend Tests (Windows) (1); CI re-running.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 30, 2026
Price every warming start at agent.subagent_cost_gb instead of the learned p90 or a live peak, and remove the learned-cost refresh only the gate read.
@iamwhatever
iamwhatever force-pushed the fix/subagent-loose-start-heavy-gate branch from c8fa832 to ef4e6f9 Compare September 30, 2026 05:41
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved 8 commits in files near this PR since c8fa832 (272cef4 -> latest main). This push also drops an issue number from a new test docstring, which the Comment History Gate flagged. CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 30, 2026
@chenmingwei23
chenmingwei23 merged commit fe05527 into main Sep 30, 2026
97 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/subagent-loose-start-heavy-gate branch September 30, 2026 07:13
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 30, 2026
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 30, 2026
With max_subagents = 0 the auto cap divided free memory by the heaviest
agent's own lifetime p90, so one agent whose window was 12% release builds
priced every slot on a 171.8 GB host and held the cap at 4 to 8. kirodotdev#14800
already prices every warming start at agent.subagent_cost_gb because a
lifetime subtree peak is not a start cost.

The subagent cap now prices one slot at max(subagent_cost_gb, p90 of the
last 100 cost samples across every agent). With no history that is
subagent_cost_gb, and the cap never exceeds what that price alone gives.
The reaper re-sizes an auto cap from the latest samples on every sweep,
so heavier recent runs lower it and it rises back as they age out, with
no restart or reload; an explicit max_subagents pin is never touched and
a live reload landing during the read wins. The task runner's cap and the
warm-pool reserve keep the worst-case price (read_learned_cost), because
those starts do not pass the spawn gate.

Fixes kirodotdev#14013
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 30, 2026
With max_subagents = 0 the auto cap divided free memory by the heaviest
agent's own lifetime p90, so one agent whose window was 12% release builds
priced every slot on a 171.8 GB host and held the cap at 4 to 8. kirodotdev#14800
already prices every warming start at agent.subagent_cost_gb because a
lifetime subtree peak is not a start cost.

The subagent cap now prices one slot at max(subagent_cost_gb, p90 of the
last 100 cost samples across every agent). With no history that is
subagent_cost_gb, and the cap never exceeds what that price alone gives.
The reaper re-sizes an auto cap from the latest samples on every sweep,
so heavier recent runs lower it and it rises back as they age out, with
no restart or reload; an explicit max_subagents pin is never touched and
a live reload landing during the read wins. The task runner's cap and the
warm-pool reserve keep the worst-case price (read_learned_cost), because
those starts do not pass the spawn gate.

Fixes kirodotdev#14013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants