Repository navigation
fix(resources): admit and hand-shake against the agents slice ceiling - #11807
Conversation
5f2d84c to
ee4ebf3
Compare
Design Review (Fable 5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The non-transient overload verdict leaks: type-based handlers still classify Watch
[DESIGN-REVIEWED] 6fa85f6 |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING -- src/kiro_crew/acp/runtime.py:2177 -- |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Description twice says the throttled init budget is "120 s"; the diff ships 90 s (watchdog-bounded) — reconcile the numbers, especially against #11483's unconditional 120 s. Not justified as shipped
Inventory (7 items) — 6 justifiedIntent: FIX — stop admitting spawns into, and killing healthy-but-slow cold starts under, the memory throttle the sandbox itself imposes (reproduction in description; accepted
Verified on base: [FIRST-PRINCIPLES-REVIEWED] 6fa85f6 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
ee4ebf3 to
5cd5bf8
Compare
|
Review response for First Principles item 4 (zero consumers of First Principles item 6 (duplicate cgroup reader) -- accepted. One reader remains: GPT finding (local import in Design review (description drift) -- accepted. The description named a helper that does not exist ( GPT BLOCKING F1 ("overload still kills the live runtime") -- rebutted; requesting the fence be lifted. The reaped process is a cold-start runtime that never completed Local gates on |
5cd5bf8 to
1dd20cf
Compare
1dd20cf to
e7b493d
Compare
1492109 to
1588e2a
Compare
1588e2a to
4ea6fdf
Compare
|
Review response for GPT finding (late throttle detection still reaps at the plain deadline) -- accepted, fixed. On a fresh gateway Fix: Also on this head, not from the diff: |
4ea6fdf to
f68dc74
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Approved after verified readiness: passed, PR Readiness success on this head, mergeable, no open change requests. [operator: bolichen97 batch review 2026-09-23]
Problem / Motivation
On a bare Linux host (no container cgroup), the sandbox places every agent process in
kirocrew-agents.slicewithMemoryHighat 75% of RAM, but the spawn-admission probe (resource_status,_probe_resources) never reads that slice. Its "cgroup-clamped" branch reads/sys/fs/cgroup/memory.max(the container root), which does not exist on a bare host, so it falls back to host-wideMemAvailable. With plenty of host memory free the probe reportedampleand admitted new spawns while the kernel was already throttling the entire agent subtree.Observed on a 32 GB host: the slice's
memory.events highcounter climbed from 0 to 3.7M over 40 minutes (9.6 min of accumulated stall) with zero admission refusals logged. Under that throttle, kiro-cli could not answerinitializeinside the fixed 30 s handshake budget; the runtime killed it (process_state=runningon every one of nine consecutive kills, i.e. a healthy but slow process) and respawned into the same throttled slice, deepening the throttle. That spiral killed a running subagent mid-prompt.Why it matters
Anyone running the gateway on a bare Linux host with the systemd slice active. The gate that exists to prevent overload is blind to the one ceiling the product itself imposes, so it admits work into an already-throttled slice, and the handshake reaper then converts "slow" into "dead" and retries, making the overload worse instead of backing off. The user sees agents and chat sessions dying with
AcpProcessDied ... killedand no explanation.What changed (motivation → approach → change)
Symptom:
resource_statussaysamplewhile the agents slice is throttling; healthy runtimes are killed at 30.0 s and respawned.Root cause: (1)
_cgroup_available_gbconsults only the container root cgroup; (2) the init-handshake budget is fixed and any timeout is treated as a dead process.Two changes, kept separate:
Slice-aware admission (
sandbox.py,subagent.py). New_agents_slice_available_gb()insubagentreadsmin(memory.high, memory.max) - memory.currentfrom the existingsandbox._agents_slice_cgroup_dir()._cgroup_available_gbnow takes the minimum of the process-ancestry figure (main's_container_cgroup_available_gbwalker) and the slice figure, so posture flips totight/criticalwhen the slice is the binding limit and spawns are refused there (resource_status,admission_check,compute_max_subagentsall sit behind that probe). The slice is a sibling of the gateway's cgroup, not an ancestor, and the walker never readsmemory.high, so the slice term is not subsumed by it.subagentkeeps its own_read_int_file(main's sizing tests isolate the kernel by patching that module'sopen);sandbox.read_cgroup_intserves the sandbox-side probes, and a test pins both readers to the same sentinel contract.Throttle-aware handshake (
acp/runtime.py,acp/session_handle.py). New typedAcpRuntimeOverloaded(AcpRequestTimeout)withtransient = False. The handshake helper checkssandbox.agents_slice_throttling()(memory.current >= memory.high, or thememory.events highcounter climbing between probes): when the slice is throttling, the init budget is extended to 120 s; if the process is still alive at the deadline, the failure is raised asAcpRuntimeOverloaded. The uninitialized cold-start process is still reaped by the existing anti-leak guard (it holds no session and no prompt), but because the verdict is non-transient the retry layers that read.transientstructurally (llm_helpers.acp_error_is_transient, the taskqacp_providerclassifier) do NOT respawn a fresh cold start into the same throttle -- the error surfaces to the caller naming the remedy (free agent memory). When the slice is not throttling, behaviour is unchanged (30 s, plainAcpRequestTimeout, retried as before).Related: #11483 widens the handshake timeout to 120 s unconditionally. This PR raises it only while the slice is throttling and adds the typed refusal, so it is complementary; if #11483 lands first this rebases down to the throttle-aware refusal on top of its budget.
Backwards compatibility
Compatible: on hosts with no agents slice (macOS, containers, systemd unavailable) both readers return
Noneand every code path falls through to today's behaviour. No public signature, config key or error code is removed or renamed;AcpRuntimeOverloadedis additive.Tests
test/test_agents_slice_admission.py(27 new tests; 760 passed with the existing runtime/sandbox/subagent-sizing neighbours at-n 2):TestAgentsSliceHeadroom: lower ofmemory.high/memory.maxbinds; usage above the soft ceiling clamps to 0 not negative;maxsentinel onhighfalls back to the hard ceiling; no ceiling at all does not constrain.available_gbwhen it is the binding limit; container-root figure still wins when it is lower; missing slice leaves the probe unchanged; posture reachescriticalfrom the slice alone.AcpRuntimeOverloaded;AcpRuntimeOverloaded.transient is FalsewhileAcpRequestTimeout.transient is True, andacp_error_is_transientreturns the matching verdict for each (the no-respawn pin); non-throttling slice keeps the 30 s path; dead process at deadline still reports the death, not overload; error exported from the runtime module.subagent._read_int_file,sandbox.read_cgroup_int) agree on every sentinel:max, garbage and an absent file all read asNone.memory.events highadvance is held for 60 s (_SLICE_THROTTLE_EDGE_AT) instead of being consumed by the first probe.Revert-mutation probes were run for both headline claims (dropping the slice term from the probe; replacing the overload conversion with a plain re-raise) and each fails exactly the named tests.
Manual verification
Reproduced the original failure signature from the journal on the affected host (40 min continuous
memory.events highclimb, ninefailed init handshakekills all withprocess_state=running, zero admission refusals). With the patch, the live slice readsheadroom_gbcorrectly against the 23 GBMemoryHigh. A full re-trigger of the throttle spiral was not re-run on the shared host; the handshake path is covered by the unit tests above.Related Issues
Related: #11483, #10166, #10521.
Pattern harvest
Rule candidate: review-prompt
Pattern: "admission/health probe reads a ceiling the product does not set while ignoring the ceiling the product itself imposes" -- any resource gate should read the same cgroup/limit that the sandbox writes.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)