Repository navigation
fix(subagent): backport the 2 GiB spawn floor to 0.8 - #16315
Conversation
First Principles Review (Fable 5) — ✅ human override accepted@bolichen97 overrode this lane for |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The macOS pressure-veto release dependency lives only in PR prose; nothing on the branch stops a 0.8 tag shipping the 2 GiB floor alone. Watch
Suggestions
[DESIGN-REVIEWED] 6654bbf |
GPT 5.6 Review — ✅ human override acceptedHuman judgment by @bolichen97 overrides the GPT 5.6 finding for [GPT-OVERRIDE] 6654bbf This comment is updated in place on each push. The model was not re-run because an authorized human decision supersedes it. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
…runs (#15890) The spawn floor defaulted to 4.0 GB with one flat 0.5 GB per start on top, so a 16 GB laptop rarely had the 4.5 GB a first start needed and subagents sat in the queue. The flat price was also wrong both ways: a start that shares its parent's runtime was charged as a process, and a dedicated runtime with the default agent's MCP servers settles near 1 GB. The floor now means what must remain AFTER a start, default 2.0 GB from one constant behind every consumer; a stored 4.0 adopts to 2.0 once. Each start is priced by its run's own sharing decision and carries that price until it settles: dedicated at the agent's learned settled PSS (capped) or a measured 1.0 GB, never below subagent_cost_gb; shared at that less the 0.35 GB process it skips. A shared start that turns dedicated is re-priced and re-checked before its process launches. Pending claims are charged their checked price. The learned settled size is a new quiet PSS capture per dedicated process, written as the cost record's settled_gb beside the unchanged mem_gb peak. RFC overload-resilience Q9 (#15887) records the decision and supersedes Q8. Backport to release/0.8.0: the RFC on this branch has no Q8 or Q9, so its limits table takes the new spawn-floor and posture rows without the Q9 pointer. run.py keeps 0.8's claim path (no claim-time re-projection) and gains only the dedicated top-up and the shared-flag reset. The whole-tree p90 admission test that #15247 left out comes along with its new price. That test and the run.py top-up test close their manager at teardown, so neither leaves the durable task store open. Fixes #15244 Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com> (cherry picked from commit 0288a88)
… usage (#15929) A finite cgroup memory limit whose usage file cannot be read gives zero headroom, so the spawn gate defers the run. The deferral then read "low memory: 0.0 GB available", which looks like real exhaustion. The cgroup probe now records a per-thread cause when it hits that case, and the gate reads it right after its own memory check. The warning, the refusal text, the deferral reason and the audit metadata now say the headroom is unknown and ask to restore read access to memory.current / memory.usage_in_bytes. Admission is unchanged: the run is still deferred. Backport to release/0.8.0: the new test file closes every manager it builds at teardown, so none leaves the durable task store open. Closes #12285 Co-authored-by: Joe Guo <zejiangg@amazon.com> (cherry picked from commit f38d63f)
616bf31 to
6654bbf
Compare
|
|
|
|
|
|
|
|
|
|
/ai-review override gpt 6654bbf: faithful backport. The synchronous fallback in _host_available_gb_off_loop is merged main code (#15890), byte-identical and pinned by test_a_pool_that_cannot_answer_gets_the_synchronous_reading. Any change to it lands on main first and is then backported, not changed only on the release branch. |
Human judgment recorded@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
Human judgment recorded@bolichen97 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
Backport of #15876 to release/0.8.0. #16315 already gave this branch the 2 GiB spawn floor; the critique of that change was that a 2 GiB floor on a 16 GB Mac admits into compressor pressure unless the kernel's own pressure verdict can veto it. With this change the next 0.8 tag carries both the floor and the veto. * fix(subagent): veto macOS starts under kernel memory pressure On macOS the memory floor now has a second input: the kernel's kern.memorystatus_vm_pressure_level. At WARN or worse a root start waits while a dedicated runtime of this gateway is running, even when the Mach reclaimable figure clears the floor. The level is the kernel's lagging verdict (it has read NORMAL with 4.9 GB compressed and 4.8/6.0 GB of swap in use), so it backs the figure up; it does not replace it. - platform_compat.memory_pressure_level() reads the sysctl through sysctlbyname on the cached libc handle, now built once under a lock. None off macOS and on any failure, and None fails open. - The gate reads the level fresh (resource_status .read_memory_pressure_level). The hold is off with the floor (0), never applies to a nested child or a claim re-entry, and only counts dedicated-priced runtimes of ours, so a shared wave never holds itself. A start priced shared is held too: it still launches the agent's MCP servers. - It is a capacity-style wait, not a store deferral: the row stays in the window as memory_pressure (restricted rows too), the pump pick passes over held roots, every slot release and a 15 s timer re-check it. Past 1800 s a row is ended, never started ("never started: waiting for memory"), and an episode that outlives that bound ends new starts at once until it eases. The same rule is re-checked at approval release (per entry, so a held root never blocks a child) and at the shared-to-dedicated top-up. - A critical posture wins over the hold, and agent validation now runs before the memory gates, so an unknown agent is refused at once; the event-loop callers run its directory scan off the loop. - An unreadable Mach figure is reported on macOS as on Linux (memory_check_unavailable). The level is shown in resource_status, the [RESOURCES] line and the diagnostics bundle; the eager-spawn admission takes no pre-warm while the hold applies. * fix(subagent): keep a waiting row's approval and answer refusals Three defects on the paths a held or deferred start now takes routinely. Each predates the pressure hold. - A window refill rebuilds an entry from the store, which never carries approval_mode, so a held or deferred App Kit spawn (approval_mode "auto") drained into a spawn prompt nobody can answer. The accepting process now keeps that row's mode in _held_approval_modes and the refill restores it; a restart still replays the row without it. - finish_parked_defer announced a refusal on TaskStoreUnavailable while the row stayed QUEUED and due, so one id could get a refusal and a later completion. It now answers still-queued and re-pumps. - The channel spawn reply said "Spawned" for a refused terminal record. It now says the subagent was not started, and why. Backport to release/0.8.0. The hold, its tests and its UI land as on main; only the code around them differs, and none of #16191, #16012 or #15900 comes along: - The 30-minute bound stays the named constant _PRESSURE_HOLD_MAX_WAIT_SECS, because the agent.subagent_queue_max_wait_secs key does not exist on 0.8. - subagent_wait_reasons has no QUEUED_KIND_TEXT or queued_wait_text on 0.8, so the new kind adds its constants only. - The manager gains only the pressure-hold and approval-mode fields; the queue-depth emit and retry state, _admitting_waiting and _dispatching_ids stay main-only, and shutdown cancels only the pressure re-check timer. - The [RESOURCES] comment is in context.py (0.8 has no context_assembly/turn.py). The spec keeps 0.8's subagent_queued paragraph and producer paths and adds the pressure rows to them. - conftest gains only the kernel-pressure pin. 0.8's conftest has no close_subagent_managers fixture, so test_admission_gate carries the module-local autouse copy that test_subagent_sizing and seven other 0.8 test files already use, and every manager it builds is closed at teardown, not at GC. (cherry picked from commit 44ea05c)
Problem / Motivation
Goal: (release/0.8.0 backport of #15890 and #15929) on 0.8, a 16 GB laptop can start subagents again instead of leaving them queued, and a deferral caused by unreadable cgroup usage says the headroom is unknown.
On release/0.8.0 the spawn floor defaults to 4.0 GB, and every start adds a flat 0.5 GB. A first start therefore needs 4.5 GB available. A 16 GB laptop with a browser and an IDE open rarely has that much, so subagents sit in the queue (#15244). The flat price is also wrong in both directions: a start that shares its parent's runtime is charged as if it were a process, and a dedicated runtime with the default agent's MCP servers settles near 1 GB, not 0.5 GB.
Separately, when a finite cgroup memory limit has a usage file that cannot be read, headroom computes as zero. The resulting deferral reads "low memory: 0.0 GB available", which looks like real exhaustion (#12285).
Why it matters
0.8 is the line users install. Until the next 0.8 tag carries this, those users get none of the main fix, and subagents on a typical 16 GB machine keep waiting for memory they will never get. The misleading cgroup message sends container operators after a memory problem they do not have.
Not a goal
reproject_claimed_session,replace_stale_shared_session,invalidate_stale_kas_session). 0.8 keeps its plainget_or_createclaim path.What changed (motivation → approach → change)
This PR implements mainline-plan unit R1 / G2 (
fix(subagent): backport the 2 GiB spawn floor to 0.8). It has two commits, each acherry-pick -xof a merged main PR, with original authorship and Co-authored-by trailers kept.The product decision (the 2.0 GB floor) is recorded on main in rfc-overload-resilience.md Q9 (accepted, #15887). Q10 there (accepted) adds the macOS veto as part of the same floor rule.
1.
0288a888d4(#15890): admit on a 2 GiB floor and price each start by how it runs.constants.DEFAULT_SPAWN_MIN_MEMORY_GB. All four former 4.0 sites read it:config/sections.py,config/loader.py,subagent.py(two) andsubagent_manager/admission/gate.py.config/superseded_defaults.pyrow withauto_adopt=True.subagent_cost_gb;Six files conflicted on 0.8. They were resolved as follows:
docs/architecture/resource-protection.md: the 0.8 row text is kept, and only fix(subagent): admit on a 2 GiB floor and price each start by how it runs #15890's edit is applied. The 0.8 gate already defers durable spawns, so the edit is accurate here.docs/request-for-change/rfc-overload-resilience.md: the single stalecheck_memory_available / cached_admission_check | REFUSErow is replaced by fix(subagent): admit on a 2 GiB floor and price each start by how it runs #15890's two rows, the spawn floor and the posture gate with its cron deferral. 0.8'scron.pydefers interval and one-shot firings, so the cron row is accurate. The "(Q9)" pointer is dropped, because this branch's RFC has no Q9.docs/system-specs/modules/subagent.md: takes fix(subagent): admit on a 2 GiB floor and price each start by how it runs #15890's whole "Memory guard: what must remain after the start" block. It replaces the older start-price paragraph.src/kiro_crew/config/sections.py: keeps the two new constants imports. Main'sservice_sectionsimport is dropped, because that module does not exist on 0.8.src/kiro_crew/subagent_manager/run.py: 0.8's claim path is kept. Two of fix(subagent): admit on a 2 GiB floor and price each start by how it runs #15890's seams are added:_ensure_dedicated_start_pricedtop-up, in both the shared-runtime fallback arm and the dedicated arm;_session_sharing/_shared_providerreset in the dedicated arm.test/test_admission_gate.py: brings inTestALearnedWholeTreePeakNeverPricesAStartat fix(subagent): admit on a 2 GiB floor and price each start by how it runs #15890's version. The earlier backport fix(subagent): backport start-cost pricing to release/0.8.0 #15247 left this class out. The file now matches main's post-fix(subagent): admit on a 2 GiB floor and price each start by how it runs #15890 blob.2.
f38d63f988(#15929): say when a spawn deferral comes from unreadable cgroup usage. This applied cleanly. The cgroup probe records a per-thread cause, and the gate reads it right after its memory check. The warning, refusal text, deferral reason and audit metadata now say the headroom is unknown. Admission is unchanged: the run is still deferred.Spec updates in this PR:
docs/system-specs/modules/subagent.md,docs/architecture/resource-protection.md,docs/request-for-change/rfc-overload-resilience.md(limits table only).Release gating: the next 0.8 tag must also carry the macOS pressure veto
Do not cut a 0.8 tag that ships this floor to 16 GB Macs without the macOS pressure veto (the #15876 backport, mainline-plan R2). On macOS, available memory includes reclaimable pages, so a 2.0 GB floor alone can admit a start while the kernel already reports memory pressure. Main's RFC records the veto as part of the 2.0 GB floor rule (Q10). This PR can merge on its own, but the tag that releases it must also include R2.
Backwards compatibility
Compatible. Nothing that 0.8 accepts today is rejected, renamed or removed.
agent.spawn_min_memory_gbkeeps its key and type. Only the default changes, and an explicitly stored 4.0 adopts to 2.0 once through the existing superseded-defaults mechanism. Any other stored value is kept as is. The cost record gains asettled_gbfield next to the unchangedmem_gbpeak; readers ofmem_gbare not affected.Tests
All of these come from the cherry-picked commits and run on the branch:
test/test_spawn_floor_defaults.py(F2/F3) drives the realSubagentManager.spawnpath at the shipped defaults:On plain release/0.8.0 this file fails at collection (
ImportError: DEFAULT_SPAWN_MIN_MEMORY_GB).test/test_config_superseded_defaults.py: a stored 4.0 adopts to 2.0 once.test/test_admission_gate.py:TestSpawnAdmissionGate::test_the_pending_start_is_priced_at_its_dedicated_projection;TestALearnedWholeTreePeakNeverPricesAStart: a learned whole-tree p90 never prices a start.test/test_session_sharing.py::TestSessionSharingFallback::test_a_dedicated_start_is_repriced_before_its_process_launches[fallback|dedicated-arm]pins the run.py top-up in both arms. A review mutation confirmed each variant fails when its top-up is removed.test/test_spawn_memory_cause.py: an unreadable cgroup usage file is reported as unknown headroom.test/test_subagent_sizing.py,test/test_subagent_cost.py: start pricing and the settled-PSS capture.Local results, after rebasing onto
3d390545fe:test_config*,test_docs*,test_resource*: 2183 passed, 34 skipped.mypy src/kiro_crewpasses. Also passing: subprocess-encoding, docs-lint, the harness-parity gate (base release/0.8.0) and the comment-history gate (base release/0.8.0).check_black_formatting.pyflags one file,test/test_symlinked_home_member_sources.py. This PR does not touch it, and it fails the same way on plain release/0.8.0 (it came in with fix(members): read member sources on a home reached through a symlink (release/0.8.0) #16265).Manual verification
N/A: unit coverage is sufficient. The spawn-floor scenarios run the real
SubagentManager.spawnadmission path with only the memory probe faked. The behaviour is identical to main, where these PRs have already been merged and exercised.Related Issues
Fixes #15244 (release/0.8.0). Fixes #12285 (release/0.8.0). Main: #15890, #15929.
Depends on: nothing to merge. Release dependency: the macOS pressure veto backport (#15876, mainline-plan R2) must ride in the same 0.8 tag.
Pattern harvest
Not generalizable: this is a backport. The pattern harvest for the defect (one constant behind every default of a threshold, pinned by
test_every_default_of_the_floor_is_the_one_constant) was done in #15890.Checklist