Repository navigation
fix(subagent): backport the macOS memory-pressure hold to 0.8 - #16567
Conversation
Backport of #15876 to release/0.8.0. #16315 already gave this branch the 2 GiB spawn floor; the critique of that change was that a 2 GiB floor on a 16 GB Mac admits into compressor pressure unless the kernel's own pressure verdict can veto it. With this change the next 0.8 tag carries both the floor and the veto. * fix(subagent): veto macOS starts under kernel memory pressure On macOS the memory floor now has a second input: the kernel's kern.memorystatus_vm_pressure_level. At WARN or worse a root start waits while a dedicated runtime of this gateway is running, even when the Mach reclaimable figure clears the floor. The level is the kernel's lagging verdict (it has read NORMAL with 4.9 GB compressed and 4.8/6.0 GB of swap in use), so it backs the figure up; it does not replace it. - platform_compat.memory_pressure_level() reads the sysctl through sysctlbyname on the cached libc handle, now built once under a lock. None off macOS and on any failure, and None fails open. - The gate reads the level fresh (resource_status .read_memory_pressure_level). The hold is off with the floor (0), never applies to a nested child or a claim re-entry, and only counts dedicated-priced runtimes of ours, so a shared wave never holds itself. A start priced shared is held too: it still launches the agent's MCP servers. - It is a capacity-style wait, not a store deferral: the row stays in the window as memory_pressure (restricted rows too), the pump pick passes over held roots, every slot release and a 15 s timer re-check it. Past 1800 s a row is ended, never started ("never started: waiting for memory"), and an episode that outlives that bound ends new starts at once until it eases. The same rule is re-checked at approval release (per entry, so a held root never blocks a child) and at the shared-to-dedicated top-up. - A critical posture wins over the hold, and agent validation now runs before the memory gates, so an unknown agent is refused at once; the event-loop callers run its directory scan off the loop. - An unreadable Mach figure is reported on macOS as on Linux (memory_check_unavailable). The level is shown in resource_status, the [RESOURCES] line and the diagnostics bundle; the eager-spawn admission takes no pre-warm while the hold applies. * fix(subagent): keep a waiting row's approval and answer refusals Three defects on the paths a held or deferred start now takes routinely. Each predates the pressure hold. - A window refill rebuilds an entry from the store, which never carries approval_mode, so a held or deferred App Kit spawn (approval_mode "auto") drained into a spawn prompt nobody can answer. The accepting process now keeps that row's mode in _held_approval_modes and the refill restores it; a restart still replays the row without it. - finish_parked_defer announced a refusal on TaskStoreUnavailable while the row stayed QUEUED and due, so one id could get a refusal and a later completion. It now answers still-queued and re-pumps. - The channel spawn reply said "Spawned" for a refused terminal record. It now says the subagent was not started, and why. Backport to release/0.8.0. The hold, its tests and its UI land as on main; only the code around them differs, and none of #16191, #16012 or #15900 comes along: - The 30-minute bound stays the named constant _PRESSURE_HOLD_MAX_WAIT_SECS, because the agent.subagent_queue_max_wait_secs key does not exist on 0.8. - subagent_wait_reasons has no QUEUED_KIND_TEXT or queued_wait_text on 0.8, so the new kind adds its constants only. - The manager gains only the pressure-hold and approval-mode fields; the queue-depth emit and retry state, _admitting_waiting and _dispatching_ids stay main-only, and shutdown cancels only the pressure re-check timer. - The [RESOURCES] comment is in context.py (0.8 has no context_assembly/turn.py). The spec keeps 0.8's subagent_queued paragraph and producer paths and adds the pressure rows to them. - conftest gains only the kernel-pressure pin. 0.8's conftest has no close_subagent_managers fixture, so test_admission_gate carries the module-local autouse copy that test_subagent_sizing and seven other 0.8 test files already use, and every manager it builds is closed at teardown, not at GC. (cherry picked from commit 44ea05c)
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the auto-triggered [DESIGN-REVIEWED] 0ec1373 |
UX Review (Fable 5) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS Terminal error opens with "waiting for memory" after the hold has permanently given up — the reader couldn't tell whether starts would still happen. Watch
[UX-REVIEWED] 0ec1373 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Cited decision Q10 of rfc-overload-resilience is main-only; the 0.8 base RFC's decisions end at Q7, so the macOS hold rests on a generic "pressure defers" clause. Not justified as shipped
What this change shipsInventory (10 items) — 5 justifiedIntent: FIX — backport merged #15876 so 0.8's 2 GiB spawn floor stops admitting kiro-cli starts into macOS compressor pressure; tests fail 75+8 on base.
[FIRST-PRINCIPLES-REVIEWED] 0ec1373 |
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/subagent_manager/admission/gate.py:709 -- Per-row side tables bypass the bounded dispatch window (origin: validation) Adjudication (Opus 5) — is blocking on each finding proportionate?HARM: HIGH — process-local growth proportional to the durable backlog, recoverable only by restart (the maps are rebuilt empty, Conditions confirmed in this run: both maps are retained per row before the window decision — REMEDY COST: real and non-trivial — a named pending-start cap must sit before [ADJUDICATION] 0ec1373 total=1 uphold=1 downgrade=0 False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNo findings that block. FINDING — src/kiro_crew/resource_status.py:96 — [OPUS-REVIEWED] 0ec1373 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Four main fixes for what users on 0.8 hit with parallel subagents: - #14878: a wave's "N waiting to start" count ends at "1 waiting" under a finished turn, because a popped but unclaimed row was counted once more than the window holds. A popped row is now marked dispatching and left out of the count, so the last pop publishes 0. - #16191: the count is re-published after every terminal, Stop all and stage Cancel, coalesced per parent; a failed store read publishes nothing instead of a false 0; a resident resume entry and a deferred spawn_async row are counted correctly; the pump's idle wake no longer spins (carries Matt Cohen's #15376 commit). - #15022: a subagent inside the wait tool is not reaped as stalled before its declared duration. - #16012: a spawn the memory gate deferred exists only as a task-store row, so spawn_status answered 404, spawn_list left it out and spawn_sub_agents reported it as an error, which made the model dispatch the work again. These ids now answer as queued, and the sub-agent synthesis waits while a sibling is still queued. Backport to release/0.8.0. #14878 and #15022 apply cleanly. #16012 is applied before #16191, its order on main, because #16191 was built on it. #16012's conflicts are resolved against 0.8 code: - api_spawn_list has no persisted half and no per-row audit on 0.8, so the queued half is listed beside the live half only. - The persistence fallback of GET /api/spawn/{id} keeps 0.8's shape (a run folder reads done); the "no recorded ending is not done" change needs main's ending classifier and is left out. - The Slack gateway and chat runner keep 0.8's orchestrator mode, so the synthesis arm stays chat-mode only. The tests that read main-only APIs are adapted to those shapes, and test/conftest.py gains main's close_subagent_managers fixture, which the new tests request. The subagent spec drops the sentences that describe the main-only fallback and audit. The rebase onto #16567 keeps main's memory_pressure entry in QUEUED_KIND_TEXT, so a later read of a row the pressure hold keeps still names the pressure instead of "waiting to start". The main handle's trusted wait identity (#15022) gains a behavioural pin: an identity that is not provenance-verified must not name kirocrew-core on the dispatch snapshot. Backports the fixes for #14877, #15131, #15011 and #15045. Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com> Co-authored-by: Will Laws <will.laws@me.com> Co-authored-by: Matt Cohen <mattcohe@amazon.com> (cherry picked from commit 0f785ef) (cherry picked from commit 4bb1d11) (cherry picked from commit 6d99710) (cherry picked from commit c759e04)
Four main fixes for what users on 0.8 hit with parallel subagents: - #14878: a wave's "N waiting to start" count ends at "1 waiting" under a finished turn, because a popped but unclaimed row was counted once more than the window holds. A popped row is now marked dispatching and left out of the count, so the last pop publishes 0. - #16191: the count is re-published after every terminal, Stop all and stage Cancel, coalesced per parent; a failed store read publishes nothing instead of a false 0; a resident resume entry and a deferred spawn_async row are counted correctly; the pump's idle wake no longer spins (carries Matt Cohen's #15376 commit). - #15022: a subagent inside the wait tool is not reaped as stalled before its declared duration. - #16012: a spawn the memory gate deferred exists only as a task-store row, so spawn_status answered 404, spawn_list left it out and spawn_sub_agents reported it as an error, which made the model dispatch the work again. These ids now answer as queued, and the sub-agent synthesis waits while a sibling is still queued. Backport to release/0.8.0. #14878 and #15022 apply cleanly. #16012 is applied before #16191, its order on main, because #16191 was built on it. #16012's conflicts are resolved against 0.8 code: - api_spawn_list has no persisted half and no per-row audit on 0.8, so the queued half is listed beside the live half only. - The persistence fallback of GET /api/spawn/{id} keeps 0.8's shape (a run folder reads done); the "no recorded ending is not done" change needs main's ending classifier and is left out. - The Slack gateway and chat runner keep 0.8's orchestrator mode, so the synthesis arm stays chat-mode only. The tests that read main-only APIs are adapted to those shapes, and test/conftest.py gains main's close_subagent_managers fixture, which the new tests request. The subagent spec drops the sentences that describe the main-only fallback and audit. The rebase onto #16567 keeps main's memory_pressure entry in QUEUED_KIND_TEXT, so a later read of a row the pressure hold keeps still names the pressure instead of "waiting to start". The main handle's trusted wait identity (#15022) gains a behavioural pin: an identity that is not provenance-verified must not name kirocrew-core on the dispatch snapshot. Backports the fixes for #14877, #15131, #15011 and #15045. Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com> Co-authored-by: Will Laws <will.laws@me.com> Co-authored-by: Matt Cohen <mattcohe@amazon.com> (cherry picked from commit 0f785ef) (cherry picked from commit 4bb1d11) (cherry picked from commit 6d99710) (cherry picked from commit c759e04)
Four main fixes for what users on 0.8 hit with parallel subagents: - #14878: a wave's "N waiting to start" count ends at "1 waiting" under a finished turn, because a popped but unclaimed row was counted once more than the window holds. A popped row is now marked dispatching and left out of the count, so the last pop publishes 0. - #16191: the count is re-published after every terminal, Stop all and stage Cancel, coalesced per parent; a failed store read publishes nothing instead of a false 0; a resident resume entry and a deferred spawn_async row are counted correctly; the pump's idle wake no longer spins (carries Matt Cohen's #15376 commit). - #15022: a subagent inside the wait tool is not reaped as stalled before its declared duration. - #16012: a spawn the memory gate deferred exists only as a task-store row, so spawn_status answered 404, spawn_list left it out and spawn_sub_agents reported it as an error, which made the model dispatch the work again. These ids now answer as queued, and the sub-agent synthesis waits while a sibling is still queued. Backport to release/0.8.0. #14878 and #15022 apply cleanly. #16012 is applied before #16191, its order on main, because #16191 was built on it. #16012's conflicts are resolved against 0.8 code: - api_spawn_list has no persisted half and no per-row audit on 0.8, so the queued half is listed beside the live half only. - The persistence fallback of GET /api/spawn/{id} keeps 0.8's shape (a run folder reads done); the "no recorded ending is not done" change needs main's ending classifier and is left out. - The Slack gateway and chat runner keep 0.8's orchestrator mode, so the synthesis arm stays chat-mode only. The tests that read main-only APIs are adapted to those shapes, and test/conftest.py gains main's close_subagent_managers fixture, which the new tests request. The subagent spec drops the sentences that describe the main-only fallback and audit. The rebase onto #16567 keeps main's memory_pressure entry in QUEUED_KIND_TEXT, so a later read of a row the pressure hold keeps still names the pressure instead of "waiting to start". The main handle's trusted wait identity (#15022) gains a behavioural pin: an identity that is not provenance-verified must not name kirocrew-core on the dispatch snapshot. Backports the fixes for #14877, #15131, #15011 and #15045. Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com> Co-authored-by: Will Laws <will.laws@me.com> Co-authored-by: Matt Cohen <mattcohe@amazon.com> (cherry picked from commit 0f785ef) (cherry picked from commit 4bb1d11) (cherry picked from commit 6d99710) (cherry picked from commit c759e04)
Problem / Motivation
Goal: (release/0.8.0 backport of #15876) on 0.8, a root subagent start on macOS waits while the kernel reports memory pressure and one of this gateway's own dedicated runtimes is running, so the next 0.8 tag ships the 2 GiB floor together with its macOS backstop.
#16315 (R1) already gave release/0.8.0 the 2 GiB spawn floor. On macOS that floor reads only the Mach reclaimable page count, which does not show the compressor. A Mac can report gigabytes "reclaimable" while the kernel already rates its memory pressure as WARN, and the gate then admits another kiro-cli runtime into that state. #16315's own release-gating note says the tag that releases the floor must also carry this veto.
Why it matters
0.8 is the line users install, and a 16 GB Mac is the common laptop case. Without the veto, the lower floor admits starts into compressor pressure and swap thrash, and the whole machine slows, not just Kiro Crew. With this PR merged, the next 0.8 tag carries both the floor and the veto, as main's RFC (rfc-overload-resilience Q10) records them: one floor rule.
Not a goal
_admitting_waiting,_dispatching_ids), fix(update): derive the auto-update effect once and act on it #15900,QUEUED_KIND_TEXT/queued_wait_text,context_assembly/turn.py, or conftest's sharedclose_subagent_managersfixture.agent.subagent_queue_max_wait_secsto 0.8. The 30-minute bound stays the named constant.AdmissionDecision.admitted, posture tiers from the level, AIMD integration).What changed (motivation → approach → change)
This PR implements mainline-plan unit R2 / G3-080 (
g3-backport). It is one commit, acherry-pick -xof #15876's squash44ea05c0ed(which already holds both of #15876's commits) onto release/0.8.0. The commit keeps #15876's message verbatim, with a lead paragraph on the 0.8 tag and a "Backport to release/0.8.0" list of the adaptations.What the hold does is unchanged from main; the full statement is
docs/system-specs/modules/subagent.md(macOS: the kernel memory-pressure hold):platform_compat.memory_pressure_level()readskern.memorystatus_vm_pressure_level(None off macOS or on failure, which fails open).memory_pressurewhile a dedicated runtime of ours runs. Nested children and claim re-entries are never held, and the hold is off with the floor (0).resource_status,[RESOURCES]and diagnostics. The eager-spawn path takes no pre-warm while the hold applies.approval_mode, a defer-store outage answers still-queued instead of refused, and the channel spawn reply reports refusals.Most files took #15876's patch byte for byte. Twelve conflicted, and each was resolved against 0.8's code:
subagent_wait_reasons.py: adds the memory-pressure constants only. 0.8 has noQUEUED_KIND_TEXT, so that entry is left out.subagent.py: the manager gains only the_pressure_*fields and_held_approval_modes._pressure_recheck_handlejoins the intentional-cancel timer set. The_PRESSURE_HOLD_MAX_WAIT_SECScomment says release/0.8.0 has noagent.subagent_queue_max_wait_secs.subagent_manager/cancellation.py: shutdown cancels only the pressure re-check timer, because 0.8 has no depth retries.subagent_manager/admission/pump.py: the TYPE_CHECKING block gets only theentry_is_childstub.subagent_manager/run.py,mcp_tools/spawn.py: only the imports 0.8 needs.context.py: the[RESOURCES]comment lands here, because 0.8 has nocontext_assembly/turn.py.test/conftest.py: only the_no_live_kernel_memory_pressurepin.test/test_admission_gate.py: main's one-line autouse requests conftest'sclose_subagent_managers, which 0.8 lacks. The module therefore carries its own_close_subagent_managersautouse, the same body 8 test files on 0.8 already use (test_subagent_sizing.pyand others). Every manager is closed at teardown.subagent.mdkeeps 0.8'ssubagent_queuedparagraph and adds the memory-pressure no-figures note.injected-messages.mdkeeps 0.8's producer paths and adds the[RESOURCES]row.learn-cron-dashboard.mdadds the pre-warm hold sentence to 0.8'sws.pytext.Spec updates in this PR:
docs/system-specs/modules/subagent.md,docs/system-specs/common/injected-messages.md,docs/system-specs/common/platform-compat.md,docs/system-specs/modules/learn-cron-dashboard.md,docs/system-specs/modules/taskq.md,docs/architecture/context-management.md. User docs:src/kiro_crew/docs/subagents.md,configuration.md,dynamic-subagent-sizing.md.Backwards compatibility
Compatible. Nothing 0.8 accepts today is rejected or renamed, same as #15876 on main:
spawn_min_memory_gb = 0, every new branch is inert.memory_pressurereason.agent_not_foundbefore any memory or capacity wait instead of after it.spawnreply changes from "Spawned ..." to "not started: ".approval_mode="auto"on refill instead of raising a prompt.Tests
All tests come from #15876 and run on this branch against 0.8 source. Run against base source with base conftest, the cherry-picked test files give 75 failed and 8 errors.
test/test_admission_gate.py(96 tests): the hold queues in the window with no store defer, the five pass-through cases, posture-critical wins, the per-row and episode bounds, slot-release and timer re-checks, approval release per entry, the real readers end to end,_owns_dedicated_runtime[*], and the defer-outage path. It also checks that the user docs state the bounds the code uses. 58 of these fail on 0.8 source.test/test_platform_compat.py::TestMemoryPressureLevel: each level through a real ctypes pointer, plus failure shapes. On base it fails at collection.test/test_resource_status.py: the level read, summary lines,[RESOURCES]and the held flag (13 fail on base).test/test_spawn_floor_defaults.py: the top-up waits on the hold for a root, never for a nested child (2 fail on base).test/test_eager_spawn.py,test/test_messaging_commands.py: no pre-warm under the hold, and the refusal reply.SubagentRunCard.test.tsx,ActivityViewer.test.tsx, and the i18n gate, strict-rule, plural and all-languages tests.Local results:
facd28b5a0: 5735 passed, 32 skipped. They cover every touched test file, everytest_subagent*,test_spawn*,test_taskq*,mcp_core,chat_runner, fairness, resource and config-baseline, plus the test files the new base commits touched.test_admission_gate.pyserial with a descriptor probe: fds 12→12 and threads 1→1 at session end (without the module autouse: 12→54 and 6 livetaskq-writerthreads).tscand eslint on the touched files,i18n:checkagainstorigin/release/0.8.0(PASS) andlint:i18n. The three rebases since then brought in backend-only commits.6997221a88(fix(memory): leave over-budget guides out instead of refusing the turn (release/0.8.0) #16556 also editscontext.py), the overlapping set gave 1300 passed and 28 skipped: admission gate, resource status, context management, essential context, eager spawn, messaging, spawn floor and platform compat.mypy src/kiro_crew(no issues in 1864 files),docs-lint.sh, and the harness-parity gate against release/0.8.0.local-gate.py --base origin/release/0.8.0planned effectively the full suite (2833 backend files), so it was not run. CI runs the suite.Manual verification
N/A: unit coverage is sufficient for a backport. The behaviour is identical to main, where #15876 merged with its macOS-only test reading the real sysctl on the macOS CI runners. As on main, no real Mac was available here. The XNU source verification of the sysctl values is in #15876's description.
Screenshots / video
The frontend patch (
SubagentRunCard,subagentQueuedReason.ts,backendPhrases.json, all 12 locales) applied byte for byte from #15876, so the rendered surfaces are the ones captured there from the real built SPA at #15876's head.Held, run card (dark):
Every member ended at the bound, run card (dark):
Mixed wave, run card (dark):
Light theme and the Subagents panel
Related Issues
Part of #16480. Main: #15876. Refs #15244 #15890 #16315.
Depends on: nothing to merge. #16315 (R1, the 2 GiB floor) is already on release/0.8.0. Release dependency: the next 0.8 tag should carry this together with #16315.
If R3 (#16388, the queue-depth state) lands after this, its conflict is to add its depth-retry handles next to
_pressure_recheck_handle, in the intentional-cancel set and at shutdown.Pattern harvest
Not generalizable: this is a backport. The pattern harvest for the defect (an OS memory signal must back up a measured figure, never replace it) was done in #15876.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement