Repository navigation
fix(subagent): name unreadable cgroup usage in spawn deferrals - #15929
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the subagent spec's probe paragraph (subagent.md:145) doesn't need a one-line mention of the new diagnostic cause, per the same-commit doc rule. [DESIGN-REVIEWED] 29947c1 Description read[DESCRIPTION-READ] 893c2642444a42cbba83adb9f0ee453a61d0c555f5ff5d9132e140ffef139c50 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The probe's cause reaches only one of its two consumers: a session import still shows an unreadable cgroup usage file as 0 bytes free. Not justified as shipped
What this change shipsInventory (5 items) — 4 justifiedIntent: a spawn deferred by an unreadable cgroup usage file should say so instead of reading as real memory exhaustion — a FIX (closes #12285).
[FIRST-PRINCIPLES-REVIEWED] 29947c1 Description read[DESCRIPTION-READ] 645b67b28fc956bb2ca5f6d7a0f4df890e3a7ad2a13eff55db44c115c25b28a7 That sha256 names the PR description this verdict read. Recompute it with |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/subagent_manager/admission/gate.py:825 -- Low host memory plus unreadable cgroup usage makes False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNothing blocks; one advisory — the deferral's published figure still asserts a measured FINDING — src/kiro_crew/subagent_manager/admission/gate.py:883 — the [OPUS-REVIEWED] 29947c1 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
e4facb7 to
b60419e
Compare
|
|
|
b60419e to
dae5410
Compare
|
|
dae5410 to
a6f84cd
Compare
… usage A finite cgroup memory limit whose usage file cannot be read gives zero headroom, so the spawn gate defers the run. The deferral then read "low memory: 0.0 GB available", which looks like real exhaustion. The cgroup probe now records a per-thread cause when it hits that case, and the gate reads it right after its own memory check. The warning, the refusal text, the deferral reason and the audit metadata now say the headroom is unknown and ask to restore read access to memory.current / memory.usage_in_bytes. Admission is unchanged: the run is still deferred. Closes #12285
a6f84cd to
29947c1
Compare
… usage (#15929) A finite cgroup memory limit whose usage file cannot be read gives zero headroom, so the spawn gate defers the run. The deferral then read "low memory: 0.0 GB available", which looks like real exhaustion. The cgroup probe now records a per-thread cause when it hits that case, and the gate reads it right after its own memory check. The warning, the refusal text, the deferral reason and the audit metadata now say the headroom is unknown and ask to restore read access to memory.current / memory.usage_in_bytes. Admission is unchanged: the run is still deferred. Backport to release/0.8.0: the new test file closes every manager it builds at teardown, so none leaves the durable task store open. Closes #12285 Co-authored-by: Joe Guo <zejiangg@amazon.com> (cherry picked from commit f38d63f)
* fix(subagent): admit on a 2 GiB floor and price each start by how it runs (#15890) The spawn floor defaulted to 4.0 GB with one flat 0.5 GB per start on top, so a 16 GB laptop rarely had the 4.5 GB a first start needed and subagents sat in the queue. The flat price was also wrong both ways: a start that shares its parent's runtime was charged as a process, and a dedicated runtime with the default agent's MCP servers settles near 1 GB. The floor now means what must remain AFTER a start, default 2.0 GB from one constant behind every consumer; a stored 4.0 adopts to 2.0 once. Each start is priced by its run's own sharing decision and carries that price until it settles: dedicated at the agent's learned settled PSS (capped) or a measured 1.0 GB, never below subagent_cost_gb; shared at that less the 0.35 GB process it skips. A shared start that turns dedicated is re-priced and re-checked before its process launches. Pending claims are charged their checked price. The learned settled size is a new quiet PSS capture per dedicated process, written as the cost record's settled_gb beside the unchanged mem_gb peak. RFC overload-resilience Q9 (#15887) records the decision and supersedes Q8. Backport to release/0.8.0: the RFC on this branch has no Q8 or Q9, so its limits table takes the new spawn-floor and posture rows without the Q9 pointer. run.py keeps 0.8's claim path (no claim-time re-projection) and gains only the dedicated top-up and the shared-flag reset. The whole-tree p90 admission test that #15247 left out comes along with its new price. That test and the run.py top-up test close their manager at teardown, so neither leaves the durable task store open. Fixes #15244 Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com> (cherry picked from commit 0288a88) * fix(subagent): say when a spawn deferral comes from unreadable cgroup usage (#15929) A finite cgroup memory limit whose usage file cannot be read gives zero headroom, so the spawn gate defers the run. The deferral then read "low memory: 0.0 GB available", which looks like real exhaustion. The cgroup probe now records a per-thread cause when it hits that case, and the gate reads it right after its own memory check. The warning, the refusal text, the deferral reason and the audit metadata now say the headroom is unknown and ask to restore read access to memory.current / memory.usage_in_bytes. Admission is unchanged: the run is still deferred. Backport to release/0.8.0: the new test file closes every manager it builds at teardown, so none leaves the durable task store open. Closes #12285 Co-authored-by: Joe Guo <zejiangg@amazon.com> (cherry picked from commit f38d63f) --------- Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com> Co-authored-by: Joe Guo <12429733+iamwhatever@users.noreply.github.com> Co-authored-by: Joe Guo <zejiangg@amazon.com>
Problem / Motivation
Goal: When a spawn is deferred because a container's cgroup memory usage cannot be read, the deferral says so instead of reading like real memory exhaustion.
A finite cgroup memory limit with an unreadable
memory.current(v1:memory.usage_in_bytes) gives zero headroom on purpose (#12203), so the spawn is deferred. The text then saidlow memory: 0.0 GB available, need 4.5 GB. An operator reads that as "the container is full", while the real cause is a permission or mount problem with the usage file.Why it matters
Operators chase the wrong fix (freeing memory, raising limits, or turning off
spawn_min_memory_gb) when the real fix is to make the usage file readable. Runs stay parked with no hint why.Not a goal
spawn_min_memory_gb=0.What changed (motivation → approach → change)
Symptom: an unreadable cgroup usage file produced the same "low memory" text as a full container.
Cause: the cgroup probe folded "usage unknown" into a plain
0.0headroom, so nothing downstream could tell the two apart.Change:
src/kiro_crew/subagent.py: the cgroup probe records a cause,cgroup_usage_unreadable, when a finite limit (at the process's cgroup or any visible ancestor) has unreadable usage. The cause is kept per thread, so a probe on another thread cannot change what this thread's gate reads. The headroom value it returns is unchanged.src/kiro_crew/subagent_manager/admission/gate.py: the spawn gate clears that cause, runs its one memory check, and reads the cause right after. The warning log, the refusal text, the deferral reason and the audit metadata then say the headroom is unknown and ask to restore read access to the usage file. With no cause, every text is exactly as before.Backwards compatibility
Compatible: admission verdicts, return values and the existing low-memory text are unchanged; the audit metadata only gains an optional
causekey when the new case fires.Tests
test/test_spawn_memory_cause.pyfabricates the cgroup files throughsubagent's own seams, so it runs on every OS:memory.maxwith unreadablememory.current→(False, 0.0)and the cause is set;memory.max→ host figure, no cause;/proc/meminfoplus a finite limit with unknown usage → still not admitted, cause set;low memory:text and no cause;spawn_min_memory_gb=0still skips the guard.test/test_subagent_spawn_host_pin.pylists the new file in its exclusions: it feeds the real memory check fabricated kernel files, so the shared host-memory pin would replace the check under test.Each of the three production edits was mutated away and the file went red each time.
Manual verification
N/A — unit coverage sufficient: the change is message text keyed off one probe branch, and the tests drive that branch and the gate end to end.
Related Issues
Closes #12285
Pattern harvest
Not generalizable: a one-off where a probe collapsed "unknown" into a numeric sentinel; the probe now carries the reason beside the number.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)