Skip to content

fix(subagent): size the auto cap from a typical run, not the heaviest agent's peak - #14015

Closed
Pearcekieser wants to merge 1 commit into
kirodotdev:mainfrom
Pearcekieser:fix/subagent-auto-ceiling
Closed

Pearcekieser wants to merge 1 commit into
kirodotdev:mainfrom
Pearcekieser:fix/subagent-auto-ceiling

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When max_subagents is left at auto, the subagent cap reflects what a typical run on the host costs, so one agent's occasional release builds no longer hold the whole host to a handful of slots.

With agent.max_subagents = 0 (auto, the default), a 171.8 GB host with little else running started with a subagent cap of 8, and recomputing gave 4. One agent's occasional release builds set the price of every slot.

Why it matters

Auto is the default, so most installs get this cap. A host that fits dozens of workers queues work behind a ceiling of 4 to 8. The ceiling also flips between 4 and 32 across restarts, depending on whether one agent's last 50 runs happen to include a sixth build.

Not a goal

  • Changing how the spawn gate's memory reserve prices a single start, or its 30-day sample horizon.
  • Changing the task runner's parallel-step cap, which keeps the worst-case per-agent price.
  • Changing behaviour when max_subagents is set explicitly.

What changed (motivation → approach → change)

Symptom: with max_subagents = 0, the auto cap divided free memory by the heaviest agent's own lifetime p90. One agent whose window was 12% release builds (21 to 25 GB) priced every slot on a 171.8 GB host, so the cap was 4 to 8. Since #14800, the spawn gate prices every warming start at agent.subagent_cost_gb, on the argument that a lifetime subtree peak is not a start cost. The cap still used that peak.

Approach: the subagent cap starts from the same price the gate reserves, and follows recent runs so it scales down on its own when the host's work gets heavier.

  • read_recent_cost in src/kiro_crew/subagent_cost.py returns the p90 of the last 40 cost samples in the log across every agent (_RECENT_WINDOW, kept below the 50 per-agent samples compaction retains so an agent that stopped running cannot hold the window). It streams the log and holds only the window, so an old heavy run ages out once 40 newer runs have landed.
  • compute_max_subagents(cfg, recent=True) in src/kiro_crew/subagent.py prices one slot at max(agent.subagent_cost_gb, read_recent_cost("mem_gb")). With no history that is subagent_cost_gb, and the cap is never above what subagent_cost_gb alone would give. resolve_max_subagents (the subagent cap) uses it, and the dashboard's subagent_cap readout calls resolve_max_subagents, so it shows an explicit pin when one is set. The worst-case read for the warm pool runs only when session.pool_size > 0.
  • The default compute_max_subagents(cfg) is unchanged, so the task runner's parallel-step cap keeps the worst case (read_learned_cost, the max of per-agent p90s). The memory held back for the warm pool (session.pool_size) also keeps the worst case, because warm-pool runtimes and task-runner steps start without the spawn gate.
  • SubagentManager._refresh_auto_cap re-sizes an auto cap from the latest samples, and the reaper calls it on every sweep (src/kiro_crew/subagent_manager/monitoring.py). Heavier recent runs therefore lower the cap on the next sweep, and it rises back as they age out, with no restart or reload. An explicit max_subagents pin is never touched. The re-size reads the config the live watcher has adopted, so a torn config.json cannot discard a pin, and a live reload that lands during the read wins.
  • Growth after a start stays with the live gates: spawn_min_memory_gb, the posture gate and the adaptive controller.
  • src/kiro_crew/docs/dynamic-subagent-sizing.md and docs/system-specs/modules/subagent.md describe the new divisor.

Backwards compatibility

Compatible: no config key, API or file format changes, and an explicit max_subagents pin is unaffected. Under auto, the subagent cap rises where one agent's peaks priced every slot, and it now moves between gateway restarts as recent runs change. The task runner's cap and the warm-pool reserve are unchanged.

Tests

  • test/test_subagent_sizing.py TestAutoCapFollowsRecentRuns: with no history the cap comes from subagent_cost_gb; a heavier recent window lowers it; heavy runs that aged out of the window raise it back; the cap never exceeds the subagent_cost_gb-only value; the reported host gives 32 for the subagent cap and 5 for the task runner's worst case; the warm-pool reserve keeps the worst case.
  • test/test_subagent_cost.py: read_recent_cost returns None without history, pools across agents, reads only the newest window, and skips unusable values.
  • test/test_subagent_config_hot_reload.py TestTheAutoCapFollowsRecentRuns: the reaper re-sizes the cap each sweep, down and back up; an explicit pin is never re-sized; a reload that lands during the read wins.
  • test/test_resource_health_endpoint.py: the dashboard readout goes through resolve_max_subagents. test_without_a_pool_the_worst_case_is_not_read checks the worst-case read is skipped when there is no warm pool.

Manual verification

N/A — unit coverage sufficient: the sizing tests drive resolve_max_subagents end to end from a real cost log file, and the reaper test drives the re-size through _reaper_loop.

Related Issues

Fixes #14013

Pattern harvest

Rule candidate: review-prompt
Pattern: a count of concurrent slots priced at the worst-case member's lifetime peak instead of a recent, bounded window, when the per-start gate already prices a start at its start cost.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and buluoray September 25, 2026 23:12
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Intent: When agent.max_subagents is 0 (auto), size the cap from the memory a typical run on this host uses, so one agent with occasional heavy builds no longer holds the whole host to a handful of subagents.
Not a goal: changing the per-spawn memory reserve, re-sizing the cap on a timer, or changing explicit max_subagents pins or the adaptive controller.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/subagent-auto-ceiling branch from 4718d81 to 8175327 Compare September 25, 2026 23:19
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of 024d3dc17b2393c0cc524575cf23ff72e99df5a8 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The pooled p90 can hide the heavy tail entirely, removing the last static bound on aggregate memory over-commit — the OOM the old divisor priced in.

Watch

In the PR's own kill case (test_one_build_heavy_agent_no_longer_sizes_every_slot: 6×24 GB among 150 samples), heavy runs are ~4% of the pooled 40-sample window, so the p90 is a light run and the cap hits 32. The old divisor kept the all-heavy case inside host memory (≈6×24 GB < 171.8 GB); the new one does not (32 slots admitted at ~1 GB start price can include several builds that balloon to 24 GB after admission). The named guards — spawn_min_memory_gb, posture, adaptive controller — are admission/throttle controls and cannot reclaim growth already admitted, and the recent-window re-size is lagging (samples land at run end, cap moves at the next sweep). The spec hunk this PR edits still says "memory over-commit is an unrecoverable OOM," so the design knowingly swaps that static guarantee for admission-time gating; a human should confirm that trade is intended, since the failure is a host OOM an operator can only pre-empt by pinning max_subagents.
Clears when: the PR (description or subagent.md) states why simultaneous admission of several heavy runs before they balloon is bounded by the live gates — e.g. the posture gate's reaction time versus a build's ramp — or adds a term that bounds the all-heavy aggregate.

[DESIGN-REVIEWED] 024d3dc

Description read

[DESCRIPTION-READ] 83caf8b946bb0caa9d217e0963fbabe30efea1b6bc5c10d72f77ce350096a7bd

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 024d3dc17b2393c0cc524575cf23ff72e99df5a8 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the live gates alone suffice when a heavy agent that runs less often than every 40 runs stays permanently outside the recent window pricing the cap.

Not justified as shipped

  • Item 7 — undeclared: the non-positive subagent_cost_gb guard (start_cost = ... if agent.subagent_cost_gb > 0 else 0.5) appears in no description sentence; it is required by the new recent branch, whose max(start_cost, …) divisor would otherwise hit zero.

What this change ships

Inventory (8 items) — 7 justified

Intent: FIX — under auto, size the subagent cap from a typical run's cost instead of the heaviest agent's lifetime p90 (Fixes #14013; premise verified on base: the spawn gate already prices every start at subagent_cost_gb, admission/gate.py:738, subagent.py:2041).

  1. Auto cap priced at max(start price, recent cross-agent p90), not the heaviest agent's lifetime p90 — justified
  2. Auto cap now re-sizes itself every reaper sweep, no restart needed — justified
  3. Task runner cap and warm-pool reserve keep the worst-case price — justified
  4. Dashboard subagent_cap readout now shows the real cap, including an explicit pin — justified
  5. New helper read_recent_cost (1 consumer: _host_mem_term; grepped read_recent_cost, src) — justified
  6. recent= parameter on compute_max_subagents (both branches have real consumers: resolve_max_subagents, taskrunner.py:774) — justified
  7. Non-positive configured start cost now falls back to 0.5 in both branches — undeclared: no description sentence mentions it; harm-free rider the new divisor needs
  8. Both sizing docs updated in the same commit — justified (AGENTS.md same-commit rule)

[FIRST-PRINCIPLES-REVIEWED] 024d3dc

Description read

[DESCRIPTION-READ] 66d1a488870c7037dd1ccaae6137aab16d9523ce5c78e520c24587c47dbd6dee

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 024d3dc17b2393c0cc524575cf23ff72e99df5a8 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 024d3dc

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 024d3dc17b2393c0cc524575cf23ff72e99df5a8 via the fork AI-review pipeline; updated in place on each push.

Review details

subagent_cost_gb is now a permanent floor on the auto cap's slot price, but all three shipped copies of its help text still call it a fallback "until a learned value accumulates".

FINDING — src/kiro_crew/docs/dynamic-subagent-sizing.md:148 — the new mem_cost = max(start_cost, read_recent_cost("mem_gb") or 0.0) in _host_mem_term(recent=True) makes subagent_cost_gb a permanent floor under every slot price, so raising it unconditionally and permanently lowers the subagent cap, yet the table row still reads also the cap-sizing fallback (GB/agent) until a learned cost exists — contradicting the paragraph this same PR added 50 lines above it — and the identical claim ships to operators in config/sections.py:1579 ("the per-agent fallback used to auto-size the cap until a learned value accumulates") and byte-identically in config-baseline.json:1209, so an operator who raises it to be conservative on a heavy host silently shrinks the cap instead of only the admission reserve → Fix: in all three copies, state that it is both the per-start price the admission gate reserves and the floor under the auto cap's slot price (requires editing config/sections.py and config-baseline.json, which this PR does not touch).

[OPUS-REVIEWED] 024d3dc

@Pearcekieser
Pearcekieser force-pushed the fix/subagent-auto-ceiling branch from 8175327 to 94ad1de Compare September 26, 2026 23:51
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 94ad1dec10 (still one commit, rebased on current main). It addresses both review findings on 817532719f:

  • GPT 5.6 (blocking, task runner OOM): fixed. compute_max_subagents now takes pooled: bool = False. Only resolve_max_subagents (the subagent cap) and the dashboard subagent_cap readout pass pooled=True. The task runner's _clamp_parallel_steps call is unchanged, so parallel step groups keep main's worst-case price (read_learned_cost, restored), because those steps have no per-start reserve. test_one_build_heavy_agent_does_not_size_every_slot now asserts both: the subagent cap is 32 and the default-priced (task runner) cap stays at 5 on the same cost log.
  • Opus 5 (stale samples in the pool): fixed. read_pooled_cost passes max_age_secs=_SAMPLE_MAX_AGE_SECS, the same 30-day horizon the spawn guard's reserve reader uses, and test_expired_samples_do_not_price_a_start asserts the pooled figure ignores expired rows while the worst-case reader keeps main's behaviour.

The earlier Backend Tests (3.12, 8) failure (test_sandbox_protected_name_holds leaf-only count) was inherited from main at the old base; it passes on the new base. Locally, 475 sizing, cost, taskrunner, adaptive and resource-endpoint tests pass, and mypy is clean on the changed modules.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/subagent-auto-ceiling branch from 94ad1de to 052a755 Compare September 27, 2026 00:24
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 052a755f79: the only change from 94ad1dec10 is removing test/test_resource_health_endpoint.py from .github/black-baseline.txt. The file became black-clean in the previous revision, and the Lint lane asks for it to leave the baseline. The shard 2 failure (test_ai_review_workflows.py::TestReviewLanesPublishOnlyTheReview::test_the_trim_and_count_execute_as_written) is in a file this PR does not touch, and it passes locally on the same base. Coverage Gate failed because it depends on the test shards.

@Pearcekieser
Pearcekieser force-pushed the fix/subagent-auto-ceiling branch from 052a755 to 7a35f11 Compare September 27, 2026 00:55
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 7a35f112be, a timestamp-only amend of 052a755f79 with an identical tree and no code change. It is there to re-trigger the lanes that failed between 00:25 and 00:33Z on the CI app installation rate limit: Fork PR Description, readiness, desktop matrix, the GPT 5.6 and Opus 5 reviews, Security Scope Review and Internal Content Scan. The backend test matrix, lint and E2E all passed on 052a755f79.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/subagent-auto-ceiling branch from 7a35f11 to 2d31979 Compare September 27, 2026 01:19
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 2d31979054, which addresses the GPT 5.6 review of 7a35f112be:

  • Blocking F1 (a 30-day horizon on the pooled reader drops a long-idle host to the 0.5 GB fallback): fixed by removing the horizon, so read_pooled_cost reads the retained history the way main's read_learned_cost always did. I had added the horizon in the previous round for the Opus 5 finding about stale samples. Of the two risks, falling back to 0.5 GB and sizing 32 slots on a host whose workers cost 6 GB is the worse one. The Opus scenario needs a retired roster of light agents to outnumber the current heavy one, and even then the spawn gate still prices each start at its own agent's p90. test_expired_samples_do_not_price_a_start now asserts that both cap readers keep old samples.
  • F2 (the dashboard stubs accepted any kwargs): the compute_max_subagents stubs in test/test_resource_health_endpoint.py now take a keyword-only pooled argument and raise unless it is True, so dropping pooled=True from the readout fails both tests.

Locally, 475 targeted tests pass, black, flake8 and isort are clean, and mypy is clean on the changed module. The PR body is updated to match.

@Pearcekieser
Pearcekieser force-pushed the fix/subagent-auto-ceiling branch from 3fc016c to 2fe036d Compare September 30, 2026 07:00
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 2fe036d8e8, a timestamp-only amend of 3fc016cf7c with an identical tree and no code change. Seven lanes on 3fc016cf7c failed between 06:30 and 06:34Z on the CI app's installation rate limit (HTTP 403 in the job logs): Fork PR Description, Publish readiness signal, Resolve desktop matrix, GPT 5.6 Review, Opus 5 Review, Internal Content Scan and Security Scope Review. This push re-runs them. The backend test matrix, lint and E2E completed on 3fc016cf7c with no failures.

@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 30, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

@iamwhatever (author of #14800) and maintainers, a direction question before I push again. #14800 changes the ground this PR stands on.

Every revision here priced the auto cap's slots at a typical run, because the spawn reserve priced each start at that agent's own learned p90. Most of the GPT 5.6 rounds were about keeping those two prices consistent. #14800 removed the learned per-agent start price. Every warming start is now reserved at agent.subagent_cost_gb, and its "Not a goal" section leaves cap sizing alone. On main, therefore, compute_max_subagents still divides by the heaviest agent's lifetime p90, and nothing else prices a heavy agent. The symptom in #14013 is unchanged on main: the auto cap is 4 to 8 on a 171 GB host because one agent's release builds were 12% of its runs. The rebase also conflicts in five files.

I see three ways to go:

  1. Rebuild this PR small on top of fix(subagent): price a sub-agent start at its start cost #14800. The auto cap divides by the pooled p90 of recent runs, the task runner and warm pool keep the worst case, and the live gates (spawn_min_memory_gb, the posture gate and the adaptive controller) guard growth after start. This follows fix(subagent): price a sub-agent start at its start cost #14800's own reasoning that a lifetime subtree peak is not a start cost, but the fenced GPT lane will very likely block it without an override.
  2. Keep the worst-case divisor and close this PR. Auto subagent cap priced every slot at the heaviest agent's p90 peak #14013 stays open for whoever takes on guarding post-start growth, which fix(subagent): price a sub-agent start at its start cost #14800 lists as separate work.
  3. Something you prefer, for example sizing the auto cap from subagent_cost_gb and leaving the live gates to back off.

I have a local revision ready for the pre-#14800 base and will not push it. Which direction do you want?

For the record, the reds on 2fe036d8e8 are not from this diff:

  • Backend Tests (3.12, 5): test_taskq_startup.py::test_an_unheld_manager_pumps_a_recovered_row_at_once passes on this branch and on a detached origin/main locally.
  • Frontend Tests (1): GitPanel.test.tsx is in a file this PR does not touch.
  • The coverage lanes follow from those two shards.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed af94f94211, a fresh single commit on current main (not a rebase) that answers the direction question above. Pearce chose option 3.

The auto subagent cap now starts from agent.subagent_cost_gb, the price #14800 reserves for every start. It prices one slot at max(subagent_cost_gb, p90 of the last 100 cost samples across every agent). With no history the price is subagent_cost_gb, and the cap never exceeds what that price alone gives. The reaper re-sizes an auto cap from the latest samples on every sweep, so heavier recent runs lower it within a minute and it rises back as they age out of the window. No restart or reload is needed. The task runner's cap and the warm-pool reserve keep the worst case (read_learned_cost), because those starts skip the spawn gate. Growth after a start stays with spawn_min_memory_gb, the posture gate and the adaptive controller.

On the open findings:

  • GPT 5.6 F1 on 2fe036d8e8 (the warm-pool subtraction was priced at the pooled figure): addressed. The warm-pool reserve uses the worst case, and test_the_warm_pool_reserve_keeps_the_worst_case covers it.
  • The earlier rounds about consistency between the cap and the per-agent spawn reserve no longer apply. fix(subagent): price a sub-agent start at its start cost #14800 removed that reserve's learned per-agent price, and this commit contains none of the machinery those rounds were about (the reserve-priced agent set, the price republication, the boot adoption, the log-identity bracket).
  • The Frontend Tests (1) (GitPanel.test.tsx) and Backend Tests (3.12, 5) (test_taskq_startup.py) failures on 2fe036d8e8 are in files this PR does not touch, and the backend one passes on a detached origin/main.

Locally 2,829 tests pass across the subagent, sizing, cost, adaptive, admission, monitoring, task-runner and resource suites. black (--target-version py310) and flake8 are clean on the changed files, and mypy is clean on subagent_cost.py and monitoring.py.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 30, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/subagent-auto-ceiling branch from af94f94 to 0500432 Compare September 30, 2026 23:15
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 0500432f13. The only change from af94f94211 is one comment in src/kiro_crew/subagent_cost.py: its wording ("no longer counts") tripped the comment-history gate, and it now reads "only the newest this-many samples count". The Fast Gate failed on that marker and skipped the heavy jobs, and this push re-runs them. COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py passes locally.

@Pearcekieser
Pearcekieser force-pushed the fix/subagent-auto-ceiling branch from 0500432 to 9931936 Compare October 1, 2026 00:16
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 9931936669. It addresses the GPT 5.6 finding on 0500432f13 and both Opus 5 findings on the same head.

  • GPT 5.6 F1 (fenced; also Opus's third finding): _refresh_auto_cap took its generation snapshot after the config load. A reload that pinned the cap during that load could therefore be overwritten. The snapshot now comes before the first await. test_a_reload_landing_during_the_config_load_wins fails on 0500432f13 and passes here.
  • Opus 5 blocking: a non-positive agent.subagent_cost_gb made the divisor zero when there was no history. The start price now falls back to 0.5 GB unless the configured value is positive. test_a_non_positive_start_cost_does_not_divide_by_zero covers this.
  • Opus 5 window finding: with _RECENT_WINDOW = 100 and compaction keeping 50 samples per bucket, a heavy agent that had stopped running could hold the window indefinitely. _RECENT_WINDOW is now _DEFAULT_WINDOW * 4 // 5, which is 40. test_aged_out_heavy_runs_raise_it_back now builds its log through compact_cost_log, and test_the_recent_window_is_shorter_than_a_compacted_bucket pins the relationship. The docs and PR body say 40.

Locally, 648 tests pass across the hot-reload, sizing, cost, resource-endpoint, subagent-coverage and task-runner suites. black (--target-version py310), flake8 and the comment-history gate are clean.

@Pearcekieser
Pearcekieser force-pushed the fix/subagent-auto-ceiling branch from 9931936 to 44700e0 Compare October 1, 2026 01:15
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 44700e0950, a timestamp-only amend of 9931936669 with an identical tree and no code change. It re-runs the lanes that failed on 9931936669. None of them touches this diff:

  • GPT 5.6 Review, Opus 5 Review and Internal Content Scan reported "could not run" within the same second (00:40:55Z), and Security Scope Review failed the same way at 00:49Z. This is the CI app rate-limit pattern seen on earlier heads.
  • E2E Gateway Boot (windows-latest): the throwaway KIROCREW_HOME could not be removed because Windows still had the MCP probe temp dirs (run/mcp-tmp/probe-*) locked.
  • Backend Tests (Windows) (2): two test_native_skill_projection.py lock-file sweep tests failed. This PR does not touch that file.

All Linux backend shards, the other seven Windows shards, lint and the remaining E2E jobs passed on 9931936669.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 1, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 22a81f977f, which is the same commit as 44700e0950 rebased onto current main (ecdc59abf4). There are no code changes and no conflicts. The rebase re-runs the two Windows shards that were red on 44700e0950. Shard 2's test_native_skill_projection.py lock-file tests were failing on main itself until ecdc59abf4 turned that shard green. Shard 8's test_members_dm_thread.py failure is in a module this PR does not touch. Every AI review lane passed on 44700e0950. Locally, 648 targeted tests pass on the new base.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 4f52245f45, a timestamp-only amend of 22a81f977f with an identical tree and no code change. It re-runs Backend Tests (Windows) (3), the only failing check on 22a81f977f. The failing test was test_session_export.py::test_a_sent_export_streams_its_staged_file_and_removes_it, an assertion that a staged export file was removed, in a module this PR does not touch. The same shard passed on main's last three runs. Every other check on 22a81f977f passed, including the GPT 5.6, Opus 5, Security Scope, UX and Internal Content Scan lanes (Design and First Principles are advisory and neutral) and both Windows shards that were red before the resync.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 12f02c6134, which is the same commit rebased onto current main (7038f5fc9b). It has no code changes and no conflicts. It re-runs the reds on 4f52245f45, none of which touches this diff:

  • Integration (in-process gateway) failed its second-boot module-globals check on security.paths._path_resolve_thread_waits. It passed on the identical tree 22a81f977f and on main.
  • Backend Tests (Windows) (3) failed test_session_export.py::test_a_sent_export_streams_its_staged_file_and_removes_it, a Windows timing check that a staged export file is unlinked right after an aiohttp send.
  • Backend Tests (3.12, 3) failed test_agent_panel_crew_log.py::test_eviction_ranks_on_fold_order_not_on_a_stamp.
  • Coverage Gate failed because it depends on those shards.

Locally, the sizing, cost, hot-reload, resource-endpoint, subagent-coverage, task-runner and session-export suites pass on the new base.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 046282d2c9. On 12f02c6134 the Opus 5 lane reported "review incomplete", so this replies to the Opus 5 findings from 4f52245f45 instead. None of those findings was blocking.

  • handlers_system.py readout ignored the pin: fixed. The dashboard's subagent_cap now calls resolve_max_subagents(cfg), so it shows an explicit max_subagents pin when one is set and the recent-run cap otherwise.
  • Unconditional worst-case log scan: fixed. read_learned_cost runs only when session.pool_size > 0, because the warm-pool reserve is the only thing it prices on the recent path. test_without_a_pool_the_worst_case_is_not_read covers this.
  • A heavy agent's own fan-out is no longer priced at its own p90 by the cap: this follows from the design Pearce chose (option 3 in the direction comment). The slot price is max(subagent_cost_gb, recent p90), and a heavier recent mix lowers the cap on the next reaper sweep. A heavy agent's runs that have not yet entered the window are left to the live gates (spawn_min_memory_gb, the posture gate, the adaptive controller). Pricing a start by the spawning agent's own cost at the spawn gate is a change to fix(subagent): price a sub-agent start at its start cost #14800's start price, which is outside this PR's goal.

Locally, the sizing, cost, hot-reload, resource-endpoint, subagent-coverage and task-runner suites pass (649 tests). black (--target-version py310), flake8 and the comment-history gate are clean.

Backend Tests (3.12, 3) on 12f02c6134 failed test_session_export.py::test_a_sent_export_streams_its_staged_file_and_removes_it. That is the same assertion that failed twice on the Windows shard, that a staged export file is gone right after an aiohttp send completes. The test is in a module this PR does not touch. It passes locally, both alone and in one process together with this PR's new test files (234 tests), so it looks like a timing race in that test rather than an interaction with this change. Maintainers may want to look at it separately.

With max_subagents = 0 the auto cap divided free memory by the heaviest
agent's own lifetime p90, so one agent whose window was 12% release builds
priced every slot on a 171.8 GB host and held the cap at 4 to 8. kirodotdev#14800
already prices every warming start at agent.subagent_cost_gb because a
lifetime subtree peak is not a start cost.

The subagent cap now prices one slot at max(subagent_cost_gb, p90 of the
last 100 cost samples across every agent). With no history that is
subagent_cost_gb, and the cap never exceeds what that price alone gives.
The reaper re-sizes an auto cap from the latest samples on every sweep,
so heavier recent runs lower it and it rises back as they age out, with
no restart or reload; an explicit max_subagents pin is never touched and
a live reload landing during the read wins. The task runner's cap and the
warm-pool reserve keep the worst-case price (read_learned_cost), because
those starts do not pass the spawn gate.

Fixes kirodotdev#14013
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 024d3dc17b, which addresses the Opus 5 findings on 046282d2c9.

Blocking finding, a torn config.json discarding a pin: fixed. _refresh_auto_cap now reads live.snapshot(), the config the live watcher has adopted. A torn or unreadable file loads as defaults, but the watcher keeps the last valid config, so an operator's max_subagents pin stays in force. It falls back to a disk read only when no watcher is running. test_a_torn_config_file_cannot_discard_a_pin covers the case where the file loads as defaults while the watcher holds the pin.

Finding on read_recent_cost having no min_samples floor: fixed. It returns None until the window holds _DEFAULT_MIN_SAMPLES (3) samples, the same floor the per-agent readers apply, so one heavy first run on a fresh host does not price every slot. test_recent_cost_needs_min_samples covers it.

Finding on the re-size log line showing only the effective cap: fixed. The line now logs the ceiling and the effective cap, each with its previous value.

Locally, 705 tests pass across the sizing, cost, hot-reload, resource-endpoint, subagent-coverage, task-runner and session-export suites. black (--target-version py310), flake8, mypy on subagent_cost.py and the comment-history gate are clean.

@bolichen97

Copy link
Copy Markdown
Contributor

Thank you, @Pearcekieser, for this work and for the careful thinking behind sizing from a typical run.
We're closing it because the 2 GiB floor redesign (G1/WP2a) replaces the moving count caps with a single rule: a subagent starts only while at least 2 GiB of memory stays available, and is otherwise queued visibly with a bounded wait.
That removes auto-sizing of the count cap, and the cost of a dedicated start will come from the settled RSS measured in #15472.
The replacement will take over the fix for #14013. You know this area well, so we would really welcome your review on it when it opens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Auto subagent cap priced every slot at the heaviest agent's p90 peak

2 participants