Repository navigation
fix(subagent): veto macOS starts under kernel memory pressure - #15876
Conversation
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS A spent episode refuses new root starts outright: on a Mac that sits at WARN, every new root spawn ends "never started" while a long dedicated run continues. Watch
[DESIGN-REVIEWED] 97e3119 Description read[DESCRIPTION-READ] feee671bede179d121b78f52ae3157166b29234df7c5be617b972f27f2fee52b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The macOS hold is aimed at a real gap, but its by-default 1800 s "never started" ending has no recorded decision, and the Not justified as shipped
What this change shipsInventory (9 items) — 7 justifiedIntent: stop new subagent starts pushing a Mac already under kernel memory pressure further into swap — FIX.
[FIRST-PRINCIPLES-REVIEWED] 97e3119 Description read[DESCRIPTION-READ] 084381e362406f73382c01ab4696f0ba0fe539492ae0df8a4428d8bf420e7acc That sha256 names the PR description this verdict read. Recompute it with |
UX Review (Opus 5.5) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS The queued copy is a three-clause run-on, and its bound, "starts give up once the pressure has lasted 30 minutes", confused the blind reader. Watch
Suggestions
[UX-REVIEWED] 97e3119 Description read[DESCRIPTION-READ] 4614ddc5ff77d232301a865b172fdb7887b9da345919e786252bcbdac0133fb5 That sha256 names everything this verdict read: the PR description, plus the 9 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
3f47aaf to
6785435
Compare
|
|
|
|
|
6785435 to
7d47a0a
Compare
|
|
|
|
7d47a0a to
77934d1
Compare
|
|
|
|
|
|
|
|
On macOS the memory floor now has a second input: the kernel's
kern.memorystatus_vm_pressure_level. At WARN or worse a root start waits
while a dedicated runtime of this gateway is running, even when the Mach
reclaimable figure clears the floor. The level is the kernel's lagging
verdict (it has read NORMAL with 4.9 GB compressed and 4.8/6.0 GB of
swap in use), so it backs the figure up; it does not replace it.
- platform_compat.memory_pressure_level() reads the sysctl through
sysctlbyname on the cached libc handle, now built once under a lock.
None off macOS and on any failure, and None fails open.
- The gate reads the level fresh (resource_status
.read_memory_pressure_level). The hold is off with the floor (0),
never applies to a nested child or a claim re-entry, and only counts
dedicated-priced runtimes of ours, so a shared wave never holds
itself. A start priced shared is held too: it still launches the
agent's MCP servers.
- It is a capacity-style wait, not a store deferral: the row stays in
the window as memory_pressure (restricted rows too), the pump pick
passes over held roots, every slot release and a 15 s timer re-check
it. Past 1800 s a row is ended, never started ("never started: waiting
for memory"), and an episode that outlives that bound ends new starts
at once until it eases. The same rule is re-checked at approval
release (per entry, so a held root never blocks a child) and at the
shared-to-dedicated top-up.
- A critical posture wins over the hold, and agent validation now runs
before the memory gates, so an unknown agent is refused at once; the
event-loop callers run its directory scan off the loop.
- An unreadable Mach figure is reported on macOS as on Linux
(memory_check_unavailable). The level is shown in resource_status,
the [RESOURCES] line and the diagnostics bundle; the eager-spawn
admission takes no pre-warm while the hold applies.
Three defects on the paths a held or deferred start now takes routinely. Each predates the pressure hold. - A window refill rebuilds an entry from the store, which never carries approval_mode, so a held or deferred App Kit spawn (approval_mode "auto") drained into a spawn prompt nobody can answer. The accepting process now keeps that row's mode in _held_approval_modes and the refill restores it; a restart still replays the row without it. - finish_parked_defer announced a refusal on TaskStoreUnavailable while the row stayed QUEUED and due, so one id could get a refusal and a later completion. It now answers still-queued and re-pumps. - The channel spawn reply said "Spawned" for a refused terminal record. It now says the subagent was not started, and why.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Problem / Motivation
Goal: On macOS, a root subagent start waits while the kernel reports memory pressure and one of this gateway's own dedicated runtimes is running. New starts then stop pushing a Mac that is already compressing and swapping further into swap.
On macOS the memory floor (
agent.spawn_min_memory_gb) reads only the Mach reclaimable page count (free + inactive + speculative + purgeable). That count does not show the compressor. A Mac can report gigabytes "reclaimable" while the kernel already rates its memory pressure as WARN, and the gate then admits another kiro-cli runtime into that state.This matters now that #15890 lowered the floor to 2 GiB. The bar a dedicated start must clear is about 3 GiB, and the only WARN reading on record (2.59-2.77 GB) sits right at it.
Why it matters
A 16 GB Mac is the common laptop case. Without a backstop, a lower floor admits starts into compressor pressure and swap thrash. The whole machine slows, not just Kiro Crew. The kernel's pressure verdict is the one signal macOS gives for "short of memory whatever the counters say". This PR makes the floor honour it for the starts we control, as the floor's second input rather than a second rule.
Not a goal
AdmissionDecision.admitted. Cron, the runner lane and workflowctx.agentkeep their figure-based gate (follow-up).SIGNAL_MEMORYon WARN is a follow-up.What changed (motivation → approach → change)
Motivation → approach. The level is the kernel's lagging verdict, not a measure of free memory (#15244 read NORMAL with 4.8 of 6.0 GB swap in use), so it backs the figure up and never replaces it. It is read fresh on every check. The hold is part of the floor rule: off with the floor (0), and it applies only while a dedicated runtime of ours is running or warming; otherwise the pressure is not ours to relieve. The rule's full statement is
docs/system-specs/modules/subagent.md(macOS: the kernel memory-pressure hold).Commit 1:
fix(subagent): veto macOS starts under kernel memory pressureplatform_compat.memory_pressure_level()readskern.memorystatus_vm_pressure_levelviasysctlbyname(1/2/4 = NORMAL/WARN/CRITICAL;Noneoff macOS or on failure, which fails open).admission/gate.py) keeps a ROOT start, shared-priced included (a shared start still launches its own MCP servers; the owner's scope decision, and the maintainer's ruling that only dedicated runtimes count as ours: comment). Nested children are never held.memory_pressure; slot releases and a 15 s timer re-check it._PRESSURE_HOLD_MAX_WAIT_SECS), and an episode that has lasted that long ends the starts it would hold at once (owner decision; failing open after the bound was weighed and rejected, because the start would then launch into the very pressure it waited on). Either way the start ends "never started: waiting for memory", with a WARNING and anever_started_memory_pressureSEL row whoseexpired_bynames the bound.agent.subagent_queue_max_wait_secsis a separate PR; whichever lands second re-points the constant.AgentCheck): the reordered re-validation runs from the pump on the event loop, and the repo's blocking ruleno-blocking-call-on-event-loopwas upheld against the on-loop version on this PR.resource_status, the[RESOURCES]line and diagnostics; prewarm admission skips while the hold applies.memory_pressurequeued kind, a card header and countedErrorNoticefor starts that never started, in all 12 catalogs.Commit 2:
fix(subagent): keep a waiting row's approval and answer refusals(three pre-existing defects on paths the hold now takes routinely)approval_mode, so a held App Kit spawn drained into an unanswerable prompt; the refill now restores it (in-process).finish_parked_deferannounced a refusal onTaskStoreUnavailablefor a row still QUEUED; it now answers queued.spawnreply said "Spawned" for a refused record; it now says it was not started, and why.Backwards compatibility
Compatible: nothing that works today is rejected or renamed.
spawn_min_memory_gb = 0, every new branch is inert.memory_pressurereason is additive; they fall back to the default queued text.agent_not_foundbefore any memory or capacity wait instead of after it.spawn. A refused spawn's reply changes from "Spawned ..." to "not started: ".approval_mode="auto"instead of raising a prompt.Tests
test/test_admission_gate.py: the hold queues a start in the window (WARN and CRITICAL) with no store defer; five cases let a start through (no runtime of ours, level unreadable, NORMAL, floor 0, nested child); posture-critical wins; an unknown agent is refused first; a restricted spawn waits in memory; a held member keeps its asking turn; a shared wave never holds itself while a shared-priced start is held; WARNING once per level and again after an episode ends; the per-row bound ends a row never started (gate, pump and approval release), and a spent episode ends new starts at once; a held wave starts on the slot release with the pick never re-entering the gate meanwhile; a dedicated wave is released one runtime at a time with clocks kept; the recheck timer notices pressure easing and is cancelled at shutdown; relabelling at the pick and at the end of the hold; the label over a full cap; approval release held, and a held root never blocks a child behind it; the unreadable macOS figure; the real readers end to end; the coroutine pump keeps the row without a defer; a direct table for_owns_dedicated_runtime; queued-stop cleanup; the user docs state the bounds the code uses. Commit 2 adds the App Kit refill and the defer-outage tests.test/test_spawn_floor_defaults.py: the top-up waits on the hold for a root and never for a nested child.test/test_eager_spawn.py: the hold blocks a new pre-warm.test/test_platform_compat.py::TestMemoryPressureLevel: each level through a real ctypes function pointer, failure shapes, a concurrent first build, and a macOS-only read of the real sysctl.test/test_resource_status.py,test/test_messaging_commands.py,website/src/test/ActivityViewer.test.tsx.Frontend Tests (1)andFrontend Tests (2)fail onsrc/test/SessionActionsMenu.duplicate.test.tsxandsrc/test/hoverVariantPolicy.test.ts, which fix(website): gate hover so iOS taps land on the first try #14807's hover gate broke. Both reproduce on a clean main worktree, both fixes are in flight (fix: repair two frontend tests broken by the hover-gate cross-merge #16210, with fix(website): repair hover policy and SessionActionsMenu tests after #14807 #16223 overlapping it), and this PR touches neither file.TestPeakRssIsThisProcesss::test_a_child_of_a_bloated_parent_reports_its_own_small_peak,test_platform_compat_refactor_facade::test_every_owner_is_loaded_with_the_facade,test_telegram_transport_dispatch_composition_contract::test_a_fresh_import_loads_every_owner.Review rebuttals
assert events == []is vacuous in the restricted-spawn test." Verified it can fail under the inline pump conftest pins; the test is rewritten anyway, since restricted spawns now queue.resource_status._cached_decision." The cached verdict has no reader in the suite (healthy_host_memoryand the spawn-pin ratchet cover every spawning test), and the hold reads the level fresh, so the conftest level pin covers it.caplog.at_levelalready isolate the counts.list_agentsis cached on the directory's stat signature.Follow-ups (held)
AdmissionDecision.admitted(cron, runner lane, workflowctx.agent).resource_statusPosture line).SIGNAL_MEMORYon WARN).low_memory/posture_criticalSTORE-deferral path (the pressure hold avoids them by not deferring; approval is now covered for all three kinds by commit 2).spawn_min_memory_gb > 0.Manual verification
Not run on a real Mac. No macOS host was available. What was verified instead:
bsd/kern/kern_memorystatus_notify.c'ssysctl_memorystatus_vm_pressure_levelanswersconvert_internal_pressure_level_to_dispatch_level(memorystatus_vm_pressure_level), which maps Normal / Warning+Urgent / Critical toNOTE_MEMORYSTATUS_PRESSURE_NORMAL/_WARN/_CRITICAL(0x1 / 0x2 / 0x4 inbsd/sys/event_private.h), the same values as libdispatch's publicDISPATCH_MEMORYPRESSURE_*.PRIV_VM_PRESSUREonly under!XNU_TARGET_OS_OSX, so an ordinary macOS process can read it. The new macOS-only test reads the real sysctl on the macOS CI runners.Still worth doing on a Mac:
sysctl kern.memorystatus_vm_pressure_levelshould print 1 on an idle machine, andmemory_pressure -S -l warn(Apple's simulator, needs sudo) should move it to 2 and show the chip sentence on a queued spawn.Screenshots / video
Captured from the real built SPA (
npm run build) with every/api/**call and the websocket stubbed, usingwebsite/scripts/capture-subagent-visibility.mjs. Thesubagent_queuedframes carry the gate's figure-freereason: "memory_pressure"; the timed-out shots pushsubagent_doneframes withoutcome: "failed"and the gate's own never-started error. No gateway was running. All 12 catalogs carry these strings. Every image below was captured at this PR's current head.Held, run card (dark and light):
Every member ended at the bound, run card (dark and light): the header says what happened, and a counted error notice directly under the card (the shared ErrorNotice, outside the card button) gives the cause and names the control that retries.
A mixed wave (two ran, one was ended at the bound) says how many ran, and the counted notice binds the cause to the one that did not:
Every member ended at the bound, Subagents panel (dark)
Related Issues
Refs #15244 #15270 #15890
No linked issue: this is the macOS backstop for the 2 GiB floor (#15890); #15244 stays open because there the kernel read NORMAL, which this change does not touch.
Pattern harvest
Rule candidate: review-prompt
Pattern: an OS memory signal (a kernel pressure level, a PSI reading) used as a measure of free memory, or a NORMAL reading taken as proof of headroom. Such signals lag, so they back up a measured figure; they never replace it.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement