Repository navigation
fix(sandbox): OOM-bias session hosts that no cgroup scope bounds - #39189
Conversation
Sensitive Change ReviewSensitive files changed:
|
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the bias is applied only when the cgroup wrap left the argv unchanged, which is the exact case where no Notes
[DESIGN-REVIEWED] fc17b88 Description read[DESCRIPTION-READ] 7b851130e1fe7d45851625ad52583c020ac774aefb667d8896534e732f8a9f8f That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: a fix for the reported gateway OOM kill (#22681), working at the cause on its one agent launch site. The only rider is test-only and costs nothing. What this change shipsInventory (6 items) — 5 justifiedIntent: when a Linux host without a cgroup scope runs out of memory, the kernel kills one agent's process instead of the gateway. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] fc17b88 Description read[DESCRIPTION-READ] 4c3538c95bed0d8f078de24ceb8ae63aac5d3a275e8ffa9306592b1343105dbc That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
…ateway On a host without cgroup v2 delegation the agent tree has no memory.max, so the kernel's host-wide OOM killer is the only memory ceiling. Session hosts were spawned with oom_score_adj 0, tying the gateway, so the kernel picked by size and killed the gateway, ending every session. The ACP launch tail now picks unbounded_session_host when cgroup_scope_argv did not wrap the spawn. It keeps session_host's NOFILE headroom and adds the OOM bias, which every process in the agent tree inherits, including shell commands the harness runs itself.
0eb6b2f to
fc17b88
Compare
Sensitive change review
|
|
span=10c58b4db3bd
|
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approving. On head fc17b88 the review checked these points:
- The new profile is picked only when
cgroup_scope_argvreturns the argv unchanged. Every wrapped argv starts withsystemd-runand carriesMemoryMax, so a scoped host can never be marked "unbounded" by mistake. - Every reader of the profile handles it.
_rlimit_specand_preexec_for_profilekeep thesession_hostNOFILE headroom and its shim-less fallback, and_PROFILE_OOM_BIASadds--oom-bias. - Each production hunk has a test that fails when that hunk is reverted.
- The real-child test classes now share one
xdist_group.
Security: the sandbox files changed here add no seal, mask, refusal or allowance. The only effect is a higher oom_score_adj for session trees on Linux hosts where no cgroup scope bounds them, which is what #22681 asks for. Scoped hosts and Windows behave as before. On macOS the bias step returns early, so nothing changes there either.
Done well: the Linux test proves that a grandchild inherits the bias, which is the real threat. It also keeps a session_host control in the same run, so a reading of 1000 is not just the runner's own baseline.
Optional:
- Close #22683 as superseded once this lands.
- The sibling sweep in the description could also list
src/kiro_crew/coder/client.py, which is not affected because it runs a short Coder CLI command and is not a session host.
…ateway (#39189) On a host without cgroup v2 delegation the agent tree has no memory.max, so the kernel's host-wide OOM killer is the only memory ceiling. Session hosts were spawned with oom_score_adj 0, tying the gateway, so the kernel picked by size and killed the gateway, ending every session. The ACP launch tail now picks unbounded_session_host when cgroup_scope_argv did not wrap the spawn. It keeps session_host's NOFILE headroom and adds the OOM bias, which every process in the agent tree inherits, including shell commands the harness runs itself. Co-authored-by: Sajal Narang <sajalnarang@gmail.com> (cherry picked from commit f47c07b)
…ateway (#39189) On a host without cgroup v2 delegation the agent tree has no memory.max, so the kernel's host-wide OOM killer is the only memory ceiling. Session hosts were spawned with oom_score_adj 0, tying the gateway, so the kernel picked by size and killed the gateway, ending every session. The ACP launch tail now picks unbounded_session_host when cgroup_scope_argv did not wrap the spawn. It keeps session_host's NOFILE headroom and adds the OOM bias, which every process in the agent tree inherits, including shell commands the harness runs itself. Co-authored-by: Sajal Narang <sajalnarang@gmail.com> (cherry picked from commit f47c07b)
…ateway (#39189) On a host without cgroup v2 delegation the agent tree has no memory.max, so the kernel's host-wide OOM killer is the only memory ceiling. Session hosts were spawned with oom_score_adj 0, tying the gateway, so the kernel picked by size and killed the gateway, ending every session. The ACP launch tail now picks unbounded_session_host when cgroup_scope_argv did not wrap the spawn. It keeps session_host's NOFILE headroom and adds the OOM bias, which every process in the agent tree inherits, including shell commands the harness runs itself. Co-authored-by: Sajal Narang <sajalnarang@gmail.com> (cherry picked from commit f47c07b)
…ateway (#39189) On a host without cgroup v2 delegation the agent tree has no memory.max, so the kernel's host-wide OOM killer is the only memory ceiling. Session hosts were spawned with oom_score_adj 0, tying the gateway, so the kernel picked by size and killed the gateway, ending every session. The ACP launch tail now picks unbounded_session_host when cgroup_scope_argv did not wrap the spawn. It keeps session_host's NOFILE headroom and adds the OOM bias, which every process in the agent tree inherits, including shell commands the harness runs itself. Co-authored-by: Sajal Narang <sajalnarang@gmail.com> (cherry picked from commit f47c07b)
… main (#39387) Backports to release/0.13.0 the fixes from #39342, squashed into one commit and brought up to date with release/0.13.0 after the Kiro Memory preview backport (#39360). Picks, in order; each [pick sha] is the commit on the #39342 branch, whose body has main's cherry-picked-from line: - fix(monitor): wake the session once when its watched PR merges or closes (#37776) [pick 84da419] - fix(security): match deny rules against Windows program suffixes and cmd/PowerShell in-word escapes (#33569) [pick e4941b7] - fix(mimocode): refuse a chat that would answer from Xiaomi's hosted model unsigned (#37629) [pick 1bfdce3] - fix(acp): end a quiet agent-started cycle on the shared runtime's stale gate (#37932) [pick fa870a2] - fix(conductor): re-read goal-conductor after a compaction (#37917) [pick ccdd67c] - fix(security): refuse a command nested past 32 substitutions before the floors walk it (#37875) [pick 2f9bd67] - fix(mates): carry a moved Bot's shell toolsSettings entry as execute_bash instead of holding it (#37973) [pick 16022f4] - fix: hold an app to its own sessions, jobs and runs (#37804) [pick 3dff0c0] - fix(acp): keep a KAS chat's stored record when its resume fails after the load (#37955) [pick 6ceb7b5] - fix(gateway): ignore a repeat SIGTERM during graceful shutdown (#38031) [pick b48287d] - fix(security): keep a quote in a $( ) from hiding a later ssh (#37558) [pick b9fed61] - fix(agent): rebuild takes a server's disabledTools from the kiro global (#38147) [pick c50bbcc] - fix(mcp_gateway): keep the backend when a stub leaves an open listen (#38121) [pick 3e9c7f9] - fix: refresh plan and model list after a kiro-cli sign-in to another account (#38053) [pick 5ad3d36] - fix(slack): keep a picture sent in a linked thread for the dashboard turn (#38076) [pick 7f8903e] - fix: keep a resolved tool policy when the gateway is too slow to answer (#36669) [pick 9be5691] - fix: open owner-only files and prune overlay folders without following links (#36994) [pick 6a1acc9] - fix: refuse the local token mint to app backends (#38113) [pick d166dab] - fix(subagent): hold a spawn's first prompt for its MCP servers (#38127) [pick e3e0298] - fix(security): refuse a headless git --output write with no approver (#36326) [pick 1abee40] - fix(restore): require an explicit mode, never default to replace (#38094) [pick ca86c06] - fix: make the deny gate's shell walk linear in command length (#36437) [pick 22c100b] - fix(chat): let a cancelled background start finish its session teardown (#36664) [pick b1dd34f] - fix(acp): refuse a start when two differing specs declare the agent's name (#38411) [pick 80ccfff] - fix: probe a silent turn while its co-tenant keeps the process busy (#38662) [pick 6354490] - fix(kas): let a reopened chat switch to models its load select omits (#38376) [pick 1640822] - fix(cloud): install a patched node 22 on run on aws instances (#38732) [pick 059cb90] - fix: write member record files without following links (#37303) [pick aac9642] - fix: fail a kas start fast once the host has turned its mcp off (#38677) [pick e147238] - fix(decisions): fall back to HTTP/1.1 when h2 is missing (#38759) [pick fc16482] - fix(mcp_gateway): free a spawn permit when its backend exits before initialize (#38793) [pick a084ba2] - fix(kas): start an app chat whose tool policy hides every kirocrew-cron tool (#38761) [pick ccea5a9] - fix(acp): give the tool-stall watchdog a margin over native waits (#34117) [pick b5d9e89] - fix(mcp): bind the working-folder answer to its launch; observe launch changes behind name grants (#38983) [pick a1069be] - fix(mcp): hide the gateway secret files from the npm install and third-party probe spawns (#39000) [pick 14d8571] - fix(sandbox): fold env name case on Windows in the agent env scrub (#33997) [pick 9bdce76] - fix(apps): let app turns follow the Auto approval mode (#38830) [pick e8336df] - fix: end a dashboard turn stopped while it is still being prepared (#36478) [pick 37fff93] - fix(acp): cleanup_session deletes the record on the session's own host (#39046) [pick 31b5ec4] - fix(acp): scan only the new part of a streamed tool result (#38852) [pick e2870db] - fix(sandbox): make an unscoped session host the OOM victim, not the gateway (#39189) [pick 2be6dde] - fix(security): read aws s3 downloads piped into tail, with 2>&1, or with an AWS_PROFILE prefix (#38271) [pick bf65d97] - fix: redact four display and log paths and use a fixed 502 message for session switch refusals (#38982) [pick 55d9ec7] - fix(sandbox): pin a credential-masking app's folder after the masks on the namespace backend (#34339) [pick afd2b14] - fix(acp): wait for systemd scope admission as its own startup stage (#37008) [pick 3e29c86] - fix(acp): wait for systemd scope admission on the AcpClient spawn path (#38303) [pick 85f0a25] - fix(acp): stamp the spawn instance on the AcpClient child (#38810) [pick f36eb5a] - fix: end the turn when the agent never confirms a stop (#38512) [pick 2665c9d] - fix(kas): renew a refused crew sign-in instead of failing every start (#38978) [pick 2f48cac] - fix(security): keep self-host detection when the ssh verb is a glob (#34056) [pick ab53f10] - fix(security): judge each substitution body for ssh-to-self (#37616) [pick d5b7ac3] - test: check the ssh floor's nesting bound below the gate's depth cap (#37964) [pick 6c9ae1d] - fix(mate): apply held Bot limits on shared claude sessions and refuse unnarrowed KAS children (#39212) [pick 721bbbe] - test(integration): wait out and reset the hidden-chat refresh between boots (#37933) [pick e77586d] - fix(browser): fall back to Kiro's Chromium under Automatic when Chrome's remote debugging is off (#39265) [pick 31912eb] Left out of #39342's set (not needed for 0.13.0): the provider_labels refactor, #34045, #36668, #39120 and #37627. Merge with release/0.13.0: kept both imports in runtime_session_load, runtime_spawn_launch and session_provider; session_allocation lists the memory_policy owner; the keep-record resume path passes kas_memory_mode as main does; dropped the ACP_BACKENDS_SESSION_EVICTION import that moved to session_provider_new_conversation.
Problem / Motivation
Goal: When a Linux host without cgroup v2 runs out of memory, the kernel kills a process in one agent session instead of the gateway.
On such a host nothing bounds an agent's memory, so the kernel OOM killer is the only ceiling left. Every session host runs at
oom_score_adj0, the same as the gateway, and so does everything it runs. The kernel then ranks by size, and the gateway is usually the largest process (#22681).This PR carries @unstablebrainiac's commit from #22683 (authorship kept) on a branch of this repository, because a fork PR that changes the sandbox modules cannot pass Sensitive Change Review. The maintainer accepted the design on #22683; the code is unchanged apart from the rebase onto current
main.Why it matters
Losing the gateway ends every chat and subagent on the host at once, and the restarted gateway starts into the same memory shortage. Losing one agent process ends at most the sessions on that agent's runtime.
Not a goal
kiro-clicheck, which also usessession_hostbut runs no agent work.What changed (motivation → approach → change)
When no cgroup scope bounds a session host, it now starts at
oom_score_adj1000, and so does everything it runs. Out of memory, the kernel then kills a process in one agent's tree instead of the gateway.acp/launch.pysession_host, as todayunbounded_session_host: same NOFILE headroom plus--oom-biasagent_sdk/pod_child_probe.pysession_host, unchangedThe launch tail compares the argv before and after
cgroup_scope_argv. An unchanged argv means no scope and nomemory.max, so it picks the biased profile. The spawn shim writesoom_score_adj=1000on the session host and its whole tree inherits it, including shell commands the harness runs itself.sandbox_rlimit.pydefines the profile,sandbox.pyforwards the name, anddocs/architecture/resource-protection.mdlists it. A shim-less install keeps the NOFILE headroom and drops only the bias.Backwards compatibility
Compatible: a session host the cgroup scope wraps spawns exactly as before. One it cannot wrap gets today's limits plus the OOM bias, which only changes which process the kernel kills once memory is gone.
Removes nothing: the change adds one profile and picks it in one new case. No path, key, record or public function is taken away.
session_host, unbiasedoom_score_adj(0)cgroup_scope_argvnever wraps;session_hostunbounded_session_host; the shim's bias is a no-op off Linux, NOFILE same, so behaviour unchangedTests
test/test_acp_launch_goldens.py: the launch pickssession_hostwhen the scope wraps the spawn andunbounded_session_hostwhen it does not.test/test_spawn_exec_shim.py: the new profile's shim argv keepsRLIMIT_NOFILE:hardand adds--oom-bias; a shim-less install falls back to the session host's preexec, not the tool's 1024 cap; a real child under the new profile, and a command that child runs, both readoom_score_adj1000 whilesession_hostpasses the parent's value through (Linux only; no runtime skip).test/test_sandbox_refactor_facade.py:sandboxforwards the new name.TestRealChildandTestSyncRealChildnow share onexdist_group, so every real interpreter child in the module runs on one worker in turn instead of cold-starting side by side.Proof: not provable -- other: reverting all production hunks removes the new constant, so the tests error at collection; per-hunk mutations were run by hand instead. Forcing the launch to always pick
session_hostfailstest_only_a_session_host_without_a_cgroup_scope_is_an_oom_target; setting the bias-map entry to False failstest_unbounded_session_host_keeps_headroom_and_is_an_oom_targetandtest_unbounded_session_host_biases_what_its_harness_runs((0, 0) == (1000, 1000)).Determinism: the four new tests passed 10 repeated runs at
-n0and one run in reversed order, from a shell atoom_score_adj0. The three touched files plustest_flake_pattern_ratchet.pypass on this head (835 passed,-n 2).Manual verification
See Real-machine verification: one real session, its harness shell command, a grandchild, its MCP servers and a subagent were all read from
/proc.Spawn paths near this change:
unbounded_session_hoston this hostspawn_run) session hostcreate_subprocess_limitedtool spawnstool, already biased before this PRReal-machine verification
Machine: own dev host, Amazon Linux 2, kernel 5.10, aarch64, 16 vCPU / 62 GiB, cgroup v1 (
/sys/fs/cgroupis tmpfs), no swap, no systemd user bus.Before (main 42d0d66, the live gateway on this host): a shell run by a live session reads its chain from
/proc/<pid>/oom_score_adj: shell 0,kiro-cli0, session host 0, gateway 0, and a grandchildsleep0.After (head 0eb6b2f, same production code as fc17b88): a throwaway gateway from this branch with its own
KIROCREW_HOMEon 127.0.0.1, onekirocrew chat --headlesssession runningsh probe.sh:A second session listed every process under its session host:
kiro-cli, the Kiro Crew Python MCP servers,builder-mcp,arccand the other MCP servers all read 1000, the gateway 0. A subagent it started viaspawn_runran the same probe: its session host, command and grandchild read 1000, the gateway 0.Not run: the namespace sandbox backend, because the shell these checks ran in refuses nested
unshare(CLONE_NEWUSER)(EPERM); the test gateway usedagent.sandbox_allow_unsandboxed_exec=truein its throwaway data home only. The bias is set by the spawn shim before the sandbox wrapper execs, andoom_score_adjsurvivesexecandunshare.Cleanup: own computer; the test gateway was stopped by PID and left no process at 1000.
Related Issues
Closes #22681
Supersedes #22683 (same commit, carried on a repository branch).
Related: #20489 (the loop-stall restart that ended the incident in #22681 before the kernel chose a victim)
Sibling sweep
Swept: git grep -n "RLIMIT_PROFILE_SESSION_HOST" -- src/
Sibling: src/kiro_crew/agent_sdk/pod_child_probe.py -- not affected -- a 20-second login probe that runs no agent work
Pattern harvest
Not generalizable: the session host is the only long-lived trusted spawn left unbiased on purpose, and
acp/launch.pyis its only launch site, so there is no wider class for a rule to catch.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Sensitive change review
cgroup_scope_argvcannot wrap a session host, the ACP launch tail spawns it with the newunbounded_session_hostprofile instead ofsession_host. That profile issession_host'sRLIMIT_NOFILE:hardplus the spawn shim's--oom-bias. A scoped session host keepssession_host.oom_score_adj, 0 on this host. After, they run at 1000. No mount, path mask, seccomp rule, environment scrub or descriptor limit changes, and nothing is newly allowed or blocked.oom_score_adj, which any unprivileged process may do.unshare(CLONE_NEWUSER)with EPERM.spawn_runsubagent all worked at 1000.session_hostexactly as before, pinned bytest_only_a_session_host_without_a_cgroup_scope_is_an_oom_target. A shim-less install keepssession_host_preexec, pinned bytest_unbounded_session_host_fallback_keeps_the_session_host_preexec. No config key, path or public name is removed.test/test_spawn_exec_shim.py,test/test_acp_launch_goldens.py,test/test_sandbox_refactor_facade.pyandtest/test_flake_pattern_ratchet.py, 835 tests.session_hostfor every session host again, and no config or data needs cleaning up. Session hosts already running keep 1000 until they restart.