Repository navigation
fix(sandbox): backport the post-logout scope probe fix to release/0.8.0 - #15624
Conversation
The cgroup-scope probe was computed once per process from the presence of XDG_RUNTIME_DIR and cached for life. On a host without lingering, logind stops the per-user manager and removes /run/user/<uid> at the last logout while the variable stays set, so a gateway started inside an SSH session kept prepending `systemd-run --user --scope` and every agent respawn, cron script and app backend died with "Failed to connect to bus" until the gateway was restarted. The probe now requires a bus socket that accepts a non-blocking connect() (which also covers a stale socket file and a seccomp policy refusing the connect, #11473), and its cache is keyed on a stat-only fingerprint of the runtime dir, bus sockets and user slice, re-checked per spawn, with a 60s TTL behind it. When the manager is gone, spawns take the existing no-scope path a gateway started after logout takes, with a SECURITY warning naming `loginctl enable-linger`; the ceiling comes back on its own when the manager returns. The cpu-delegation cache is refreshed with it. Refs: #11473 (cherry picked from commit 02837e8)
a720a08 to
eb3e959
Compare
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify #15622 has merged on main unchanged before this backport lands, so release/0.8.0 never diverges from the mainline fix. [DESIGN-REVIEWED] eb3e959 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The root cause has three session-scoped caches; this fixes two — Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: FIX — a gateway started over SSH keeps spawning after the last logout instead of failing every spawn until restart (backport of #15622; linked ticket, red-on-main test).
[FIRST-PRINCIPLES-REVIEWED] eb3e959 |
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/sandbox.py:14415 -- Probe refresh walks PATH on the event loop BLOCKING -- src/kiro_crew/sandbox.py:14312 -- Escaped D-Bus paths disable cgroup ceilings FINDING -- src/kiro_crew/acp/runtime.py:1057 -- [BLOCK-MERGE] eb3e959 Adjudication (Opus 5) — is blocking on each finding proportionate?Opened F1 — Harm rung: HIGH (gateway loop stall, no self-recovery while the mount hangs). Conditions confirmed: Linux only ( F2 — Harm rung: HIGH (fork-bomb/memory ceiling silently absent for new spawns), but logged. Conditions confirmed: raw append with no unquoting ( [ADJUDICATION] eb3e959 total=0 uphold=0 downgrade=0 [ADJUDICATION-FENCED] eb3e959 fenced=2 flagged=1 🏷️ Fenced finding(s) machine-flagged as likely edge caseThe security fence keeps these findings blocking regardless of adjudication; the only clearance path is a human override. The annotate-only pass judged the condition combination each one requires extreme, and pre-drafted the override rationale below. A repository writer must independently verify a rationale before posting it — it is machine-authored, and a wrong override on a security-class finding ships exactly the class the fence exists to stop.
False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNo findings. FINDING — src/kiro_crew/sandbox.py:13204 — the recovery branch resets [OPUS-REVIEWED] eb3e959 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Problem / Motivation
Backport of #15622 (merged on main as 321fd99) and its review follow-up #15636 to
release/0.8.0: two commits, eachcherry-pick -x. The one doc conflict, indocs/system-specs/modules/security.md, was resolved by keeping the release-branch paragraph and appending the same sentence. Draft until #15636 merges on main. The follow-up resolvessystemd-runthroughplatform_compat.trusted_system_bininstead ofshutil.which, so a probe refresh never walksPATHon the event loop, and percent-decodesunix:path=bus addresses.Goal: A gateway that was started inside an SSH session keeps spawning agents, cron scripts and app backends after the user's last session logs out, instead of failing every spawn with "Failed to connect to bus" until it is restarted.
On a Linux host without lingering, logind stops the per-user systemd manager and removes
/run/user/<uid>(anduser-<uid>.slice) when the last login session ends.XDG_RUNTIME_DIRstays set in the gateway's environment._probe_cgroup_scope()decided "a user session exists" from the presence of that variable and cached the answer for the life of the process, so a gateway that probed while the user was logged in kept prependingsystemd-run --user --scopeto every spawn.systemd-runcan no longer reach any bus and exits 1 withFailed to connect to bus: No such file or directorybefore it execs the command it wraps. Every agent respawn, script cron and app backend then fails, with no self-heal, until someone restarts the gateway; auto-paused crons also stay paused after that restart. A gateway restarted after the logout probes correctly and degrades, so two gateways disagreed about the same host.Reported on a Cloud Desktop (internal ticket: https://t.corp.amazon.com/187b5c6e-b5cb-4d1b-9c36-ccf8c598e4f4), where the only workaround,
loginctl enable-linger, needs sudo.Why it matters
Every Linux user who starts or auto-updates the gateway over SSH on a host without lingering loses all sandboxed spawns at logout. The gateway looks healthy while every spawn fails, and the error only shows up as a per-cron
last_erroror as agents that cannot respawn.Not a goal
What changed (motivation → approach → change)
Symptom → root cause: the probe's session half (user manager, bus, delegated slice) belongs to the user's login, but it was cached as if it were process-stable, and it trusted an environment variable instead of the bus.
Approach: make the cached answer follow the session, cheaply enough to check at every spawn on the event loop, and use the fallback the sandbox already has.
src/kiro_crew/sandbox.py:_compute_cgroup_scope_probenow requires a reachable bus (_user_bus_reachable). It tries a non-blockingAF_UNIXconnect()on the socketssystemd-run --userdials: theunix:path=entries ofDBUS_SESSION_BUS_ADDRESS, then$XDG_RUNTIME_DIR/systemd/privateand$XDG_RUNTIME_DIR/bus. A missing socket, a stale socket file with no listener, or a seccomp/LSMEPERM(cgroup scope probe reports available when D-Bus socket is unreachable #11473) all count as unavailable.EAGAINfrom a full backlog counts as reachable, so the connect can never stall the loop or drop the ceiling under load. The "unavailable" reason names the remedy (loginctl enable-linger $USER)._probe_cgroup_scopekeys its cache on_cgroup_scope_session_fingerprint(): stat-only(st_dev, st_ino)of the runtime dir, each bus socket and the user slice'scgroup.controllers, plus the locator values. It recomputes when the fingerprint changes, and in any case every_CGROUP_SCOPE_PROBE_TTL_SECONDS(60s). The TTL catches a manager that died and left its socket behind. A recompute runs under a lock and also drops the_cpu_controller_delegatedcache, which was cached for life too._log_cgroup_scope_transition. A loss is a SECURITY warning stating that new spawns run without the fork-bomb/memory ceilings and naming the reason and remedy, and it is emitted even if the one-time startup warning already fired. A restore is logged at info. The ceiling therefore never drops silently, and it comes back on its own when the user logs in again.Guarantees that stay the same:
systemd-runscope, properties and slice).cgroup_scope_argvpassthrough, the same path a gateway started without a manager takes.cgroup_scope_bus_envreads the same gate, so no bus locator is forwarded to a spawn that is not wrapped. If the two calls insandboxed_spawn_argvstraddle a flip, theenv -ushim stays inside the wrapped argv, so the child still never sees the locators.create_subprocess_limited/run_limited/popen_limitedvia the spawn shim) is untouched, soRLIMIT_NOFILEand the other rlimits still apply after exec on the fallback path.Docs:
docs/architecture/resource-protection.md§Availability and fallback anddocs/system-specs/modules/security.mddescribe the session-following gate.Backwards compatibility
Compatible: when a user manager is reachable the probe returns the same
(True, "ok")and the argv is unchanged. Hosts without one already degraded. The only behaviour change is on hosts where the old probe said "available" butsystemd-runcould not run, where every spawn previously failed.Tests
test/test_cgroup_scope_probe_liveness.py(new, Linux-only) stands up a fake user manager: a real listeningAF_UNIXsocket in a runtime dir undertmp_pathplus a fakecgroup.controllers. The host's real user manager is never contacted, because the conftest session floor already strips the real locators.test_logout_after_probe_stops_wrapping_spawns_in_systemd_runis the reported sequence: probe while live, then remove the runtime dir with the variables still set. The spawn must be unwrapped, no locators are forwarded, and a SECURITY warning namesenable-linger. Red on main: argv was still/usr/bin/systemd-run ….test_ceiling_returns_when_the_manager_comes_back: re-login bounds spawns again without a restart.test_transition_warns_once_not_per_spawn,test_a_second_outage_is_reported_too: one warning per flip, including a later outage after the startup warning.test_bus_gone_but_slice_still_delegated_is_unavailable,test_stale_bus_socket_with_no_listener_is_unavailable,test_bus_connect_refused_by_policy_is_unavailable(cgroup scope probe reports available when D-Bus socket is unreachable #11473): the bus gate on its own.test_a_listener_with_a_full_backlog_is_reachable_and_never_blocks:EAGAINcounts as reachable.test_unchanged_session_reuses_the_cached_probe,test_cached_probe_is_rechecked_after_its_ttl: one compute per session state, plus the TTL.test_cpu_delegation_is_reread_with_the_session.On unfixed
release/0.8.010 of the original 11 tests fail; with both commits all 13 pass (includingtest_a_percent_escaped_bus_address_is_decodedandtest_a_probe_refresh_never_walks_path). Probe-consuming suites on the release branch pass (4973), flake8 and mypy clean. Each invariant line was mutation-checked and reds at least one test: fingerprint comparison removed, bus connect removed, cpu-cache reset removed, transition log removed,EAGAINtreated as unreachable, TTL removed, remedy dropped from the reason. Existing suites that use the probe (test_sandbox_argv,test_per_runtime_process_resources,test_sandbox_agent_slice_identity,test_session_scope_reapand every other test referencingcgroup_scope) pass. flake8, mypy (src/kiro_crew, 0 issues), black, isort and docs-lint are clean.Manual verification
On a Linux host with systemd 252 (the real user manager was left running and never stopped):
env XDG_RUNTIME_DIR=/tmp/<gone> DBUS_SESSION_BUS_ADDRESS=unix:path=/tmp/<gone>/bus systemd-run --user --scope -q -- trueprintedFailed to connect to bus: No such file or directoryand exited 1. This is the reported symptom, with the env a logged-out gateway carries.(True, 'ok')with the live session. Pointing the locators at the removed dir madecgroup_scope_argvreturn the unwrapped argv, which ran and printed its output, and logged the SECURITY transition warning namingloginctl enable-linger. Restoring the locators wrapped the next spawn in/usr/bin/systemd-runagain and logged "available again".Related Issues
Backport of #15622 (Fixes #11473 on main)
Internal report: https://t.corp.amazon.com/187b5c6e-b5cb-4d1b-9c36-ccf8c598e4f4. Related but not duplicated: #7363 and #14955 (pod-side handling of hosts with no per-user manager), #11352.
Pattern harvest
Rule candidate: review-prompt
Pattern: a process-lifetime cache of host state that belongs to a login session or other external lifecycle (the user manager,
/run/user/<uid>, a bus) must be keyed on a cheap fingerprint of that state or a TTL; "the environment is process-stable" does not hold for the systemd user session.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement