Skip to content

fix(subagent): name unreadable cgroup usage in spawn deferrals - #15929

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/12285-cgroup-unreadable-deferral
Oct 2, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/12285-cgroup-unreadable-deferral

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When a spawn is deferred because a container's cgroup memory usage cannot be read, the deferral says so instead of reading like real memory exhaustion.

A finite cgroup memory limit with an unreadable memory.current (v1: memory.usage_in_bytes) gives zero headroom on purpose (#12203), so the spawn is deferred. The text then said low memory: 0.0 GB available, need 4.5 GB. An operator reads that as "the container is full", while the real cause is a permission or mount problem with the usage file.

Why it matters

Operators chase the wrong fix (freeing memory, raising limits, or turning off spawn_min_memory_gb) when the real fix is to make the usage file readable. Runs stay parked with no hint why.

Not a goal

  • Changing admission. An unknown finite cgroup budget still defers (durable) or refuses (legacy) exactly as today.
  • Any change to the host reading, the agents-slice reading, the startup reservation, or spawn_min_memory_gb=0.
  • New config, new polling, or a new resource subsystem.
  • macOS/Windows: the cgroup path is Linux-only and never runs there.

What changed (motivation → approach → change)

Symptom: an unreadable cgroup usage file produced the same "low memory" text as a full container.

Cause: the cgroup probe folded "usage unknown" into a plain 0.0 headroom, so nothing downstream could tell the two apart.

Change:

  • src/kiro_crew/subagent.py: the cgroup probe records a cause, cgroup_usage_unreadable, when a finite limit (at the process's cgroup or any visible ancestor) has unreadable usage. The cause is kept per thread, so a probe on another thread cannot change what this thread's gate reads. The headroom value it returns is unchanged.
  • src/kiro_crew/subagent_manager/admission/gate.py: the spawn gate clears that cause, runs its one memory check, and reads the cause right after. The warning log, the refusal text, the deferral reason and the audit metadata then say the headroom is unknown and ask to restore read access to the usage file. With no cause, every text is exactly as before.

Backwards compatibility

Compatible: admission verdicts, return values and the existing low-memory text are unchanged; the audit metadata only gains an optional cause key when the new case fires.

Tests

test/test_spawn_memory_cause.py fabricates the cgroup files through subagent's own seams, so it runs on every OS:

  • finite memory.max with unreadable memory.current → (False, 0.0) and the cause is set;
  • readable usage near the limit → low reading, no cause;
  • absent or unlimited memory.max → host figure, no cause;
  • unreadable /proc/meminfo plus a finite limit with unknown usage → still not admitted, cause set;
  • a finite ancestor with unknown usage → cause set;
  • a real spawn with unreadable usage stays queued, its deferral says "headroom unknown … unreadable", and the audit metadata carries the cause;
  • a real spawn with measured low memory keeps the old low memory: text and no cause;
  • a stale cause from an earlier reading does not leak into a later check;
  • a probe on another thread does not set this thread's cause;
  • spawn_min_memory_gb=0 still skips the guard.

test/test_subagent_spawn_host_pin.py lists the new file in its exclusions: it feeds the real memory check fabricated kernel files, so the shared host-memory pin would replace the check under test.

Each of the three production edits was mutated away and the file went red each time.

Manual verification

N/A — unit coverage sufficient: the change is message text keyed off one probe branch, and the tests drive that branch and the gate end to end.

Related Issues

Closes #12285

Pattern harvest

Not generalizable: a one-off where a probe collapsed "unknown" into a numeric sentinel; the probe now carries the reason beside the number.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested a review from a team as a code owner October 1, 2026 16:27
@iamwhatever
iamwhatever requested a review from CrysisDeu October 1, 2026 16:27
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 29947c14bbbb55c2b357bc6059f89a2ceef7c0fe — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the subagent spec's probe paragraph (subagent.md:145) doesn't need a one-line mention of the new diagnostic cause, per the same-commit doc rule.

[DESIGN-REVIEWED] 29947c1

Description read

[DESCRIPTION-READ] 893c2642444a42cbba83adb9f0ee453a61d0c555f5ff5d9132e140ffef139c50

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 29947c14bbbb55c2b357bc6059f89a2ceef7c0fe — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The probe's cause reaches only one of its two consumers: a session import still shows an unreadable cgroup usage file as 0 bytes free.

Not justified as shipped

  • Item 4 — symptom-level: the description says "the probe now carries the reason beside the number" and "Not generalizable: a one-off", but _container_cgroup_available_gb has 2 production consumers (grep _container_cgroup_available_gb): the spawn gate (fixed) and session_transfer.py:1600, where the same unreadable-usage 0.0 becomes available = 0, so a session import waits out SESSION_IMPORT_MEMORY_WAIT_SECS and fails as a memory timeout with the identical "full container" misread — 1 counted unfixed sibling the description never names. The deferral is acceptable; the silence is the finding.
    Clears when: the session-import wait also reads the recorded cause, or the sibling at session_transfer.py:1600 is named as deliberately left.

What this change ships

Inventory (5 items) — 4 justified

Intent: a spawn deferred by an unreadable cgroup usage file should say so instead of reading as real memory exhaustion — a FIX (closes #12285).

  1. The spawn deferral for unreadable cgroup usage now says the headroom is unknown and names the file to fix — justified
  2. The warning log and the legacy refusal text say the same; measured-low text unchanged — justified
  3. The spawn audit record gains an optional cause key when that case fires — justified
  4. The memory probe records why it reported zero headroom (thread-local, pop_memory_check_cause) — symptom-level: 1 of its 2 consumers left showing unknown as 0
  5. New test file pins both texts; the host-memory pin ratchet excludes it with a stated reason — justified

[FIRST-PRINCIPLES-REVIEWED] 29947c1

Description read

[DESCRIPTION-READ] 645b67b28fc956bb2ca5f6d7a0f4df890e3a7ad2a13eff55db44c115c25b28a7

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 29947c14bbbb55c2b357bc6059f89a2ceef7c0fe and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/subagent_manager/admission/gate.py:825 -- Low host memory plus unreadable cgroup usage makes "unknown_usage" prescribe restoring cgroup access even though host memory independently blocks spawning -> Fix: suppress the cause when the host reading is already below min_gb.
[GPT-REVIEWED] 29947c1

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 29947c14bbbb55c2b357bc6059f89a2ceef7c0fe: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 29947c14bbbb55c2b357bc6059f89a2ceef7c0fe — this comment is updated in place on each push.

Review details

Nothing blocks; one advisory — the deferral's published figure still asserts a measured 0.0 the new wording says was never measured.

FINDING — src/kiro_crew/subagent_manager/admission/gate.py:883 — the unknown_usage branch rewrites only the reason text while the sibling wait={… "available_gb": round(float(avail_gb), 2) …} in the same _deferred(...) call still publishes 0.0, so _emit_queue_depth → subagent_queued → queuedWaitText's low_memory figures branch renders "needs 4 GB of free memory, 0 GB free now; free up memory to continue" on the chip (SubagentRunCard / ActivityViewer / SubagentProgressBar), i.e. the dashboard keeps giving the operator the misdiagnosis the other three surfaces were changed to drop, for a deferral freeing memory can never clear → Fix: omit available_gb from that wait dict when unknown_usage, so the chip falls to the existing low_memory_no_figures string instead of asserting a measured zero.

[OPUS-REVIEWED] 29947c1

Verdict parsed from the review's SHA-scoped output markers for commit 29947c14bbbb55c2b357bc6059f89a2ceef7c0fe.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 29947c14bbbb55c2b357bc6059f89a2ceef7c0fe: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the fix/12285-cgroup-unreadable-deferral branch from e4facb7 to b60419e Compare October 1, 2026 17:01
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: memory-check cause is now thread-local span=56eaf48ad59f

Fixed in b60419e: _memory_check_cause is a threading.local(), so a probe on another thread (sizing sampler, session transfer) cannot write the cause this thread's gate reads.
The gate clears, checks and reads on one thread with no await between, so the read always belongs to its own check.
The return arity of check_memory_available stays unchanged on purpose: dozens of tests mock it as a fixed two-tuple.
Pinned by test_a_probe_on_another_thread_does_not_set_this_threads_cause.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: wait-record cause key deleted span=217aa958d0b5

Fixed in b60419e: the cause key is gone from the wait= dict, so subagent_queued carries exactly what it carried before.
The cause stays only where it has readers: the deferral reason text, the refusal text, the warning log and the audit metadata.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: process-global cause races with concurrent memory probes span=599ab81e58eb

Fixed in b60419e: the cause slot is now threading.local(); a probe on another thread writes only its own thread's slot.
The gate clears, runs its synchronous check and reads the cause on one thread with no await between, so no other probe can interleave.
The cause is diagnostic text only; admission still keys off the unchanged (ok, gb) result, so even a wrong cause could never admit work.
Pinned by test_a_probe_on_another_thread_does_not_set_this_threads_cause.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • accepted-and-deferred: dashboard queued chip for unknown headroom span=30454b4c12d6

Deferred to #15956 (deferred-finding, owner @iamwhatever, Due 2026-10-31): carry the cause on the wait= record and render a "headroom unknown" chip string.
This PR's goal is the existing deferral/status TEXT: the deferral reason reaches the spawn tool result, CLI, Slack and /api/spawn as queued_reason_detail (mcp_tools/spawn.py, cli_commands.py, messaging/commands.py, dashboard/handlers/messaging.py), and that text now says "memory headroom unknown".
The previous round's First Principles review asked to delete the wait-record cause key because no consumer read it; adding the chip here would mean a frontend surface, i18n keys and screenshots, which is new scope.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: side channel kept because the (ok, gb) signature cannot change cheaply span=1ae150efd56a

Named callers: check_memory_available returns (ok, gb) to gate.py and to subagent.py's session-transfer sizing (_ok, host_gb = check_memory_available(min_gb=0.0)), and 74 test references across 16 files mock it as a fixed two-tuple, including the shared healthy_host_memory fixture in test/conftest.py.
A third return value would rewrite every one of those mocks for a diagnostic string; the side channel keeps them all unchanged.
The thread-local and the pre-clear exist only to make that one read correct: the pre-clear stops a cause left by a mocked or earlier check leaking in, and the thread-local stops another thread's probe overwriting it (last round's GPT blocker).

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@iamwhatever
iamwhatever force-pushed the fix/12285-cgroup-unreadable-deferral branch from dae5410 to a6f84cd Compare October 2, 2026 06:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 2, 2026
… usage

A finite cgroup memory limit whose usage file cannot be read gives zero
headroom, so the spawn gate defers the run. The deferral then read
"low memory: 0.0 GB available", which looks like real exhaustion.

The cgroup probe now records a per-thread cause when it hits that case,
and the gate reads it right after its own memory check. The warning, the
refusal text, the deferral reason and the audit metadata now say the
headroom is unknown and ask to restore read access to memory.current /
memory.usage_in_bytes. Admission is unchanged: the run is still deferred.

Closes #12285
@iamwhatever
iamwhatever force-pushed the fix/12285-cgroup-unreadable-deferral branch from a6f84cd to 29947c1 Compare October 2, 2026 07:06
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 2, 2026
@chenmingwei23
chenmingwei23 merged commit f38d63f into main Oct 2, 2026
97 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/12285-cgroup-unreadable-deferral branch October 2, 2026 15:57
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 2, 2026
bolichen97 pushed a commit that referenced this pull request Oct 3, 2026
… usage (#15929)

A finite cgroup memory limit whose usage file cannot be read gives zero
headroom, so the spawn gate defers the run. The deferral then read
"low memory: 0.0 GB available", which looks like real exhaustion.

The cgroup probe now records a per-thread cause when it hits that case,
and the gate reads it right after its own memory check. The warning, the
refusal text, the deferral reason and the audit metadata now say the
headroom is unknown and ask to restore read access to memory.current /
memory.usage_in_bytes. Admission is unchanged: the run is still deferred.

Backport to release/0.8.0: the new test file closes every manager it
builds at teardown, so none leaves the durable task store open.

Closes #12285

Co-authored-by: Joe Guo <zejiangg@amazon.com>
(cherry picked from commit f38d63f)
bolichen97 added a commit that referenced this pull request Oct 3, 2026
* fix(subagent): admit on a 2 GiB floor and price each start by how it runs (#15890)

The spawn floor defaulted to 4.0 GB with one flat 0.5 GB per start on top,
so a 16 GB laptop rarely had the 4.5 GB a first start needed and subagents
sat in the queue. The flat price was also wrong both ways: a start that
shares its parent's runtime was charged as a process, and a dedicated
runtime with the default agent's MCP servers settles near 1 GB.

The floor now means what must remain AFTER a start, default 2.0 GB from one
constant behind every consumer; a stored 4.0 adopts to 2.0 once. Each start
is priced by its run's own sharing decision and carries that price until it
settles: dedicated at the agent's learned settled PSS (capped) or a measured
1.0 GB, never below subagent_cost_gb; shared at that less the 0.35 GB process
it skips. A shared start that turns dedicated is re-priced and re-checked
before its process launches. Pending claims are charged their checked price.
The learned settled size is a new quiet PSS capture per dedicated process,
written as the cost record's settled_gb beside the unchanged mem_gb peak.

RFC overload-resilience Q9 (#15887) records the decision and supersedes Q8.

Backport to release/0.8.0: the RFC on this branch has no Q8 or Q9, so its
limits table takes the new spawn-floor and posture rows without the Q9
pointer. run.py keeps 0.8's claim path (no claim-time re-projection) and
gains only the dedicated top-up and the shared-flag reset. The whole-tree
p90 admission test that #15247 left out comes along with its new price.
That test and the run.py top-up test close their manager at teardown,
so neither leaves the durable task store open.

Fixes #15244

Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com>
(cherry picked from commit 0288a88)

* fix(subagent): say when a spawn deferral comes from unreadable cgroup usage (#15929)

A finite cgroup memory limit whose usage file cannot be read gives zero
headroom, so the spawn gate defers the run. The deferral then read
"low memory: 0.0 GB available", which looks like real exhaustion.

The cgroup probe now records a per-thread cause when it hits that case,
and the gate reads it right after its own memory check. The warning, the
refusal text, the deferral reason and the audit metadata now say the
headroom is unknown and ask to restore read access to memory.current /
memory.usage_in_bytes. Admission is unchanged: the run is still deferred.

Backport to release/0.8.0: the new test file closes every manager it
builds at teardown, so none leaves the durable task store open.

Closes #12285

Co-authored-by: Joe Guo <zejiangg@amazon.com>
(cherry picked from commit f38d63f)

---------

Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com>
Co-authored-by: Joe Guo <12429733+iamwhatever@users.noreply.github.com>
Co-authored-by: Joe Guo <zejiangg@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Explain unreadable cgroup usage in spawn memory deferrals

2 participants