Repository navigation
fix(sandbox): OOM-bias session hosts that no cgroup scope bounds - #22683
unstablebrainiac wants to merge 1 commit into
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change only raises Notes
[DESIGN-REVIEWED] 483d4b6 Description read[DESCRIPTION-READ] 19c4fed54cd425eba4226d98462635f5200b2f10553ba21c0fe6c1eaa0b1ccfe That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: this is a point fix for #22681 that adds one profile, picked at the only long-lived session-host launch site. Notes
What this change shipsInventory (4 items) — 3 justifiedIntent: when memory runs out on a Linux host with no cgroup scope, the kernel should kill an agent process, not the gateway. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 483d4b6 Description read[DESCRIPTION-READ] e47377801ef6127f0b8692b763bf37b90bb991cf5138422ce4332fed244bc6bc That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
bolichen97
left a comment
There was a problem hiding this comment.
Looks good to ship on the code. Two maintainer checks remain before approval.
Head reviewed: 51e7c36. Bots read: Design PASS, First Principles PASS, UX skipped, Opus no blocking, GPT no blocking, readiness checking.
Findings: 0 must-fix, 0 should-fix.
What I verified on this head:
cgroup_scope_argvreturns the same argv only on its two degrade paths (no cgroup v2 delegation, or no trustedsystemd-run). Every wrapped path prependssystemd-run. Soargv == unscoped_argvis a correct "no scope" test.request.scope_argvruns after the choice and only adds--unitto an argv that is already wrapped. It cannot turn an unscoped spawn into a scoped one, so the profile choice cannot be stale.- The new profile reuses the session host NOFILE spec and preexec, and only adds the bias entry in
_PROFILE_OOM_BIAS. Off Linux the shim skips the bias, and on Windows the shim prefix is empty. - The bias only raises
oom_score_adj. It widens no fence and lowers no limit. - The golden test goes red if the launch choice is inverted or ignored. The real-child test checks the grandchild, which is the path the harness shell commands take, and its waits are bounded.
Remaining maintainer checks:
- PR Readiness is still pending on this head. Backend test shards are still running.
- The diff changes
sandbox.pyandsandbox_rlimit.py. A change under the sandbox modules needs a maintainer review, and the Sensitive Change Review check is red on this head. Please fill the sensitive change template in the description so that check can pass.
Not repeated: the Design Review note that a killed session host can take every session on that runtime, not one session. The doc row and the sandbox_rlimit.py comment say "one session lost". Consider "one agent runtime lost" in a later push, but this does not block.
|
Sensitive Change Review cannot pass on this PR, with or without the template. On main, The filled template is in the description now, at the end, and passes the checker's own "One session lost": agreed. A session host can serve several sessions, a parent and the subagents created on its runtime, so killing it ends all of them. The |
51e7c36 to
c83441a
Compare
c83441a to
865c993
Compare
|
Hold for maintainer review at 865c993. This changes sandbox code and which process the kernel kills under memory pressure. An unscoped session host and everything it runs now start at — crew-pr-reviewer (r117) |
865c993 to
f453df0
Compare
f453df0 to
1fe83c4
Compare
1212174 to
783b647
Compare
|
@bolichen97 Dropped the runtime skip in The test passes from a shell at |
|
Thanks, 783b647 fixes my point: the runtime Could you update this branch onto main? Backend Tests (3.12, 6) fails only on |
783b647 to
ea352fe
Compare
|
@bolichen97 Rebased onto |
|
@unstablebrainiac thanks for the update onto main. The patch is unchanged, and my earlier point stays fixed. One thing blocks on ea352fe. Sensitive Change Review reports "Sensitive path changed without a reasoning template for this head." The rebase gave the PR a new head sha, so the lane needs the Opus 5.5, Internal Content Scan and Security Scope Review are also red. They failed when the CI app hit its API rate limit at 07:56 UTC, not on your change. I'll approve once the lanes are green. |
Sensitive change review
|
…ateway On a host without cgroup v2 delegation the agent tree has no memory.max, so the kernel's host-wide OOM killer is the only memory ceiling. Session hosts were spawned with oom_score_adj 0, tying the gateway, so the kernel picked by size and killed the gateway, ending every session. The ACP launch tail now picks unbounded_session_host when cgroup_scope_argv did not wrap the spawn. It keeps session_host's NOFILE headroom and adds the OOM bias, which every process in the agent tree inherits, including shell commands the harness runs itself.
ea352fe to
483d4b6
Compare
Code change approver recommendation: ApproveWhy: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. Adds an An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
|
I'm picking this up to address the requested changes and the red CI. Per the maintainer review, a sandbox change from a fork cannot pass Sensitive Change Review, so I'll carry @unstablebrainiac's commit on a branch of this repository with authorship kept, rebase it onto current main, run the real Linux sandbox check on an unscoped host, and link the new PR here. Thanks @unstablebrainiac for the fix and the careful iterations. |
|
Carried on a repository branch as #39189: the same commit, authored by @unstablebrainiac, rebased onto current main. That PR also has a real-host check on Amazon Linux 2 (cgroup v1, no user bus): a live session's host, harness command, grandchild, MCP servers and a subagent all read oom_score_adj 1000 while the gateway stays at 0. Review and CI continue there. |
|
Landed as #39189 (merge commit f47c07b), carrying @unstablebrainiac's commit with authorship kept; #22681 is closed. Closing this one as superseded. Thanks for the fix. |
Pull request was closed
Problem / Motivation
Goal: When a Linux host without cgroup v2 runs out of memory, the kernel kills a process in one agent session instead of the gateway.
On such a host nothing bounds an agent's memory, so the kernel OOM killer is the only ceiling left. Every session host runs at
oom_score_adj0, the same as the gateway, and so does everything it runs. The kernel then ranks by size, and the gateway is usually the largest process (#22681).Why it matters
Losing the gateway ends every chat and subagent on the host at once, and the restarted gateway starts into the same memory shortage. Losing one agent process ends at most the sessions on that agent's runtime.
Not a goal
kiro-clicheck, which also usessession_hostbut runs no agent work.What changed (motivation → approach → change)
When no cgroup scope bounds a session host, it now starts at
oom_score_adj1000, and so does everything it runs. Out of memory, the kernel then kills a process in one agent's tree instead of the gateway.acp/launch.py, foracp/client.pyandacp/runtime.pysession_host: NOFILE headroom, no OOM bias, as todayunbounded_session_host: the same NOFILE headroom plus--oom-biasagent_sdk/pod_child_probe.pysession_host, unchangedThe launch tail compares the argv before and after
cgroup_scope_argv. An unchanged argv means no scope, so nomemory.maxbounds the tree, and the launch picks the biased profile. The spawn shim then writesoom_score_adj=1000on the session host, and every process in its tree inherits it. That includes the shell commands the harness runs itself, which thetoolprofile never reaches. A scoped session host stays unbiased, because its scope already bounds it.sandbox_rlimit.pydefines the profile and maps it to the bias,sandbox.pyforwards the name, anddocs/architecture/resource-protection.mdlists it. A shim-less install keeps the session host's descriptor headroom and drops only the bias.Backwards compatibility
Compatible: a session host the cgroup scope wraps spawns exactly as before. One it cannot wrap gets today's limits plus the OOM bias, which only changes which process the kernel kills once memory is gone.
Removes nothing: the change adds one profile and picks it in one new case. No path, key, record or public function is taken away.
Tests
test/test_acp_launch_goldens.py: the launch pickssession_hostwhen the scope wraps the spawn andunbounded_session_hostwhen it does not.test/test_spawn_exec_shim.py:RLIMIT_NOFILE:hardand adds--oom-bias;oom_score_adj1000, whilesession_hostpasses the parent's own value through. The test skips when the runner is itself at 1000, because both profiles would then read alike. Its waits are bounded, and a probe still running at the bound is killed.test/test_sandbox_refactor_facade.py:sandboxforwards the new name.Disabling each piece in turn (the bias-map entry, the shim's NOFILE rule for the new profile, the shim-less fallback, and the launch's choice inverted, always biased, or ignored) makes at least one of these tests fail. Each new or changed test passed 20 runs at
-n0, 10 runs at-n 4 --dist loadgroupbeside its sibling files, and one shuffled run (seed 15928), all from a shell atoom_score_adj0.Manual verification
On Linux 5.10 (Amazon Linux 2, cgroup v1, systemd 219), from a shell at
oom_score_adj0:/proc/<pid>/cgroupreadsmemory:/system.slice/kirocrew.service, the gateway's own cgroup, socgroup_scope_argvdid not wrap them._compute_cgroup_scope_probe()gives the reason:(False, 'no XDG_RUNTIME_DIR (no systemd user session)'). On this host the launch therefore picksunbounded_session_hostevery time.unbounded_session_hostand 0 undersession_host.--timeout=20, making the probe's grandchild hang fails the real-child test in 7 s withsubprocess.TimeoutExpired, and making the probe itself hang fails it in 12 s naming the probe. Neither leaves a process behind.A local patch that biases every
session_hostspawn has run on this host since 2026-10-19. No spawn here is scoped, so for agent sessions it does what this PR does./proc/<pid>/oom_scorereads 1336 to 1340 for the largest agent processes and 693 for the gateway, so the kernel's first choice is now an agent process.Related Issues
Closes #22681
Related: #20489 (the loop-stall restart that ended the incident in #22681 before the kernel chose a victim)
Pattern harvest
Not generalizable: the session host is the only long-lived trusted spawn left unbiased on purpose, and
acp/launch.pyis its only launch site, so there is no wider class for a rule to catch.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Sensitive change review
cgroup_scope_argvcannot wrap a session host, the ACP launch tail spawns it with the newunbounded_session_hostprofile instead ofsession_host. That profile issession_host'sRLIMIT_NOFILE:hardplus the spawn shim's--oom-bias. A scoped session host keepssession_host.oom_score_adj, 0 on the reporting host. After, they run at 1000. No mount, path mask, seccomp rule, environment scrub or descriptor limit changes, and nothing is newly allowed or blocked.oom_score_adj, which any unprivileged process may do.unbounded_session_hostand 0 undersession_host. Every live session host there readsmemory:/system.slice/kirocrew.service, so none is scoped, and_compute_cgroup_scope_probe()returns(False, 'no XDG_RUNTIME_DIR (no systemd user session)'). Not run: the OS sandbox backend, because the shell these checks ran in refusesunshare(CLONE_NEWUSER)with EPERM.git,gh,pytestand MCP tools all worked there.session_hostexactly as before, pinned bytest_only_a_session_host_without_a_cgroup_scope_is_an_oom_target. No config key, path or public name is removed, and a shim-less install keepssession_host_preexec, pinned bytest_unbounded_session_host_fallback_keeps_the_session_host_preexec.test/test_spawn_exec_shim.py,test/test_acp_launch_goldens.pyandtest/test_sandbox_refactor_facade.py, 660 tests, passed on this head. 5,262 relatedacp_*,spawn_exec*andsandbox*tests passed on the same production change.session_hostfor every session host again, and no config or data needs cleaning up. Session hosts already running keep 1000 until they restart.