Repository navigation
fix(sandbox): stop wrapping spawns in systemd-run after logout - #15622
Conversation
The cgroup-scope probe was computed once per process from the presence of XDG_RUNTIME_DIR and cached for life. On a host without lingering, logind stops the per-user manager and removes /run/user/<uid> at the last logout while the variable stays set, so a gateway started inside an SSH session kept prepending `systemd-run --user --scope` and every agent respawn, cron script and app backend died with "Failed to connect to bus" until the gateway was restarted. The probe now requires a bus socket that accepts a non-blocking connect() (which also covers a stale socket file and a seccomp policy refusing the connect, #11473), and its cache is keyed on a stat-only fingerprint of the runtime dir, bus sockets and user slice, re-checked per spawn, with a 60s TTL behind it. When the manager is gone, spawns take the existing no-scope path a gateway started after logout takes, with a SECURITY warning naming `loginctl enable-linger`; the ceiling comes back on its own when the manager returns. The cpu-delegation cache is refreshed with it. Refs: #11473
02837e8 to
ec4d5e3
Compare
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/sandbox.py:15500 -- Re-login leaves aggregate cgroup state stale Adjudication (Opus 5) — is blocking on each finding proportionate?Harm rung: MEDIUM-to-HIGH residual, but the condition combination is the PR's own motivating scenario, not an extreme one. [ADJUDICATION] ec4d5e3 total=0 uphold=0 downgrade=0 [ADJUDICATION-FENCED] ec4d5e3 fenced=1 flagged=0 False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Confirm the deliberate fail-open: a same-uid process killing the user manager now yields unbounded-but-logged spawns where it previously yielded failed spawns. [DESIGN-REVIEWED] ec4d5e3 Description read[DESCRIPTION-READ] 854f7c8ea900177e979650a281a5eb40b3c1070acd521d293583ca2ae8e47872 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the one unverified residual: a live session dbus with a dead user manager still probes "reachable" — spawns there fail exactly as on base, no worse. What this change shipsInventory (7 items) — 7 justifiedIntent: FIX — a gateway started over SSH keeps wrapping spawns in
[FIRST-PRINCIPLES-REVIEWED] ec4d5e3 Description read[DESCRIPTION-READ] edff69bf0d9643e5743fcc9e7ce2a0c91c74990a4d345dcc53d1bd4e30738f75 That sha256 names the PR description this verdict read. Recompute it with |
|
This merged at ec4d5e3 while the GPT 5.6 review's two blocking findings were still open: F1, the TTL refresh resolving |
Opus 5 Review — 🔴 changes requested (blocking)Reviewed BLOCKING — the new fail-open bus probe lets the sandboxed agent delete its own fork-bomb / memory ceiling. BLOCKING — src/kiro_crew/sandbox.py:15543
FINDING — src/kiro_crew/sandbox.py:15500 — the recompute block resets only [BLOCK-MERGE] ec4d5e3 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Two defects the post-logout probe fix (#15622) introduced, from its review: - The probe now refreshes on a TTL, and every refresh resolved systemd-run with shutil.which on the event loop. A PATH entry on a stalled mount would hang the gateway's loop. The probe now uses platform_compat.trusted_system_bin, the same fixed-directory lookup cgroup_scope_argv already pins its wrapper with. - DBUS_SESSION_BUS_ADDRESS unix:path= values may percent-escape any byte. Probing the escaped spelling literally misses a live bus that lives outside XDG_RUNTIME_DIR and drops the cgroup ceiling. The value is now decoded before it is dialed. (cherry picked from commit e5a68df) (cherry picked from commit 623f2b4)
…#15636) * fix(sandbox): keep the scope probe refresh off PATH, decode bus paths Two defects the post-logout probe fix (#15622) introduced, from its review: - The probe now refreshes on a TTL, and every refresh resolved systemd-run with shutil.which on the event loop. A PATH entry on a stalled mount would hang the gateway's loop. The probe now uses platform_compat.trusted_system_bin, the same fixed-directory lookup cgroup_scope_argv already pins its wrapper with. - DBUS_SESSION_BUS_ADDRESS unix:path= values may percent-escape any byte. Probing the escaped spelling literally misses a live bus that lives outside XDG_RUNTIME_DIR and drops the cgroup ceiling. The value is now decoded before it is dialed. * fix(sandbox): resolve the probe's systemd-run without any PATH walk Second-round review of this PR: trusted_system_bin's miss diagnostic still calls shutil.which, so a host with no trusted systemd-run walked PATH on the event loop at the first probe refresh. The probe now uses platform_compat.trusted_system_bin_quiet, the same fixed-directory lookup without the diagnostic, and reports the miss in its reason. A D-Bus unix:path= value that decodes to an empty path or one holding a NUL cannot name a socket and made os.stat raise ValueError on the spawn path; such values are now skipped.
…ease/0.8.0 (#15638) * fix(sandbox): keep the scope probe refresh off PATH, decode bus paths Two defects the post-logout probe fix (#15622) introduced, from its review: - The probe now refreshes on a TTL, and every refresh resolved systemd-run with shutil.which on the event loop. A PATH entry on a stalled mount would hang the gateway's loop. The probe now uses platform_compat.trusted_system_bin, the same fixed-directory lookup cgroup_scope_argv already pins its wrapper with. - DBUS_SESSION_BUS_ADDRESS unix:path= values may percent-escape any byte. Probing the escaped spelling literally misses a live bus that lives outside XDG_RUNTIME_DIR and drops the cgroup ceiling. The value is now decoded before it is dialed. (cherry picked from commit e5a68df) (cherry picked from commit 623f2b4) * fix(sandbox): resolve the probe's systemd-run without any PATH walk Second-round review of this PR: trusted_system_bin's miss diagnostic still calls shutil.which, so a host with no trusted systemd-run walked PATH on the event loop at the first probe refresh. The probe now uses platform_compat.trusted_system_bin_quiet, the same fixed-directory lookup without the diagnostic, and reports the miss in its reason. A D-Bus unix:path= value that decodes to an empty path or one holding a NUL cannot name a socket and made os.stat raise ValueError on the spawn path; such values are now skipped. (cherry picked from commit 5843231)
… release/0.8.0 (#15619) * fix(windows): backport the directory-junction hardening to release/0.8.0 (#15666) Backport of #14327, #14962, #15310 to release/0.8.0, cherry-picked in main's merge order. * fix(windows): anchor six directory-link fences to pinned descriptors (#14327) (cherry picked from commit 6b0a6d5) * fix(config): refuse a link or junction at the workspace dir name (#14962) (cherry picked from commit 4bb50ef) * fix(windows): make four is_symlink guards see a junction (#12379, #10492) (#15310) (cherry picked from commit 6d54165) * style(test): black-format the failed-start reap test and drop its history wording on release/0.8.0 (#15665) test/test_acp_runtime_failed_start_reap.py arrived with #15625 not black-formatted (fails scripts/check_black_formatting.py on the release tip), and the _gone docstring said "no longer runs", which scripts/check_comment_history.py flags as history narration. Black 26.3.1 (--target-version py310) reflow, and the docstring now reads "has stopped running", the wording the #15664 forward-port uses. Test file only. * fix(adaptive): keep a same-window gate step from eating idle recovery (#15595) Idle recovery (#15541) took its exec step only when nothing else had changed on the sample. A spawn-gate increase earned on the same sample set that shared flag, so the idle exec step due on it was skipped, and the gate step also restarted the one-increase-per-window clock: the exec cap then waited a further window for a step it had already been due. Judge the idle step on the exec track alone. Both tracks keep sharing one increase-per-window clock across samples, as the earn rules do. An increase's reason now names each track it moved: the exec note only when the exec cap moved, and every gate step as "spawn gate target X -> Y" (the policy's target; the daemon confirms or clamps it). A gate-only step used to read "clean window earned +1", and a gate step beside an exec step was not mentioned at all. (cherry picked from commit f23a39f) * fix(acp): treat only JSON whitespace as an empty mcp.json (#15673) #15627 read an empty or whitespace-only MCP settings file as absent, using bytes.strip() with no argument. That also strips \x0b and \x0c, which JSON does not allow, so a file holding only those bytes read as "no restrictions" instead of failing closed like any other malformed document. The emptiness check now strips only space, tab, CR and LF (RFC 8259); anything else goes to the parser and fails closed as before #15627. Refs #15616 (cherry picked from commit 157fd51) * fix(sandbox): backport the scope probe PATH and bus-path fixes to release/0.8.0 (#15638) * fix(sandbox): keep the scope probe refresh off PATH, decode bus paths Two defects the post-logout probe fix (#15622) introduced, from its review: - The probe now refreshes on a TTL, and every refresh resolved systemd-run with shutil.which on the event loop. A PATH entry on a stalled mount would hang the gateway's loop. The probe now uses platform_compat.trusted_system_bin, the same fixed-directory lookup cgroup_scope_argv already pins its wrapper with. - DBUS_SESSION_BUS_ADDRESS unix:path= values may percent-escape any byte. Probing the escaped spelling literally misses a live bus that lives outside XDG_RUNTIME_DIR and drops the cgroup ceiling. The value is now decoded before it is dialed. (cherry picked from commit e5a68df) (cherry picked from commit 623f2b4) * fix(sandbox): resolve the probe's systemd-run without any PATH walk Second-round review of this PR: trusted_system_bin's miss diagnostic still calls shutil.which, so a host with no trusted systemd-run walked PATH on the event loop at the first probe refresh. The probe now uses platform_compat.trusted_system_bin_quiet, the same fixed-directory lookup without the diagnostic, and reports the miss in its reason. A D-Bus unix:path= value that decodes to an empty path or one holding a NUL cannot name a socket and made os.stat raise ValueError on the spawn path; such values are now skipped. (cherry picked from commit 5843231) * fix(chat): restore the interrupted turn when Resume lands on a natively loaded session (#15643) (#15644) * fix(chat): restore the interrupted turn when Resume lands on a natively loaded session (#15643) kiro-cli appends a prompt to <sid>.jsonl only once the model has answered it. When the process serving a turn dies mid-answer (a gateway restart, a crash, a recycled runtime) that request is never written, so session/load brings the conversation back without it. A natively resumed session gets no Kiro Crew replay, so the Resume press on that turn sent a bare "finish the user's most recent request" with the request missing: the model answered the turn before it, or said there was nothing to continue. Reproduced against kiro-cli 2.26.1: after a completed turn and a second turn killed mid-stream, session/load restores only the first. The path is any native resume: a dashboard chat with agent.tool_search off, a dashboard chat linked to a channel, and the automatic connection-lost recovery on a respawned runtime. When the Resume message or the connection-lost recovery runs on a session that this turn natively resumed, _run_chat now prepends the interrupted turn from the slot's own transcript (the opener the turn-in-flight marker restores, plus the partial answer), bracketed [INTERRUPTED TURN] so the context scanner attributes it. A cold start that replays already carries the turn and is unchanged. A session whose only turn was interrupted was already covered: SessionMap.get prunes a sid whose log holds no turn. Backport of #15643 to release/0.8.0. Conflict: context.py still carries compress_thread_history here (removed on main by #14952); the new helper is placed after it. * fix(chat): open the interrupted-turn restore at the right row, and mint it after the scrub (#15643) A cron, app or synthesis delivery opens its turn as an inject row. The interrupted-turn walk only stopped at user, nudge and sub-agent rows, so for such a turn it reached the already-answered request before it and restored that as the user's most recent request. Openers now include the inject kinds dashboard.state.TURN_OPENING_INJECT_KINDS names (the dispatching kinds minus recovery and user_replay, which continue a turn). The session-busy recovery (_BUSY_RECOVER_MSG) also resets the session and session/loads the same native conversation, so it gets the restore too. The frame tells the model the request inside it is the one to carry on with, so its markers join _STRUCTURAL_MARKER_RES and the runner prepends it after the egress scrub, like the task checklist; the builder scrubs the restored payload itself. The turn-in-flight marker now copies a sub-agent completion opener too (_LOCAL_TURN_OPENER_ROLES / _LOCAL_TURN_PROMPT_ROLES): without it a crash before the flush loses that row, and the restore would walk past it to the previous, already-answered request. * fix(mcp): repair Windows command casing at every resolver, not only the rewriter On Windows `shutil.which` spells the extension it appends exactly as PATHEXT spells it, so a bare `demo-mcp` resolves to `demo-mcp.EXE` while the file on disk is `demo-mcp.exe`. A launcher that dispatches on its own argv[0] basename case-sensitively (a tool manager's multiplexer shim) refuses to run under that spelling and the MCP server never starts. The rewriter already restored the on-disk basename, but the agent-config resolver (`agent._resolve_command`) and the dashboard probe (`mcp_discovery`) consumed raw `which` output, and the resolver's result is PERSISTED as the spec's absolute `command`. An absolute command is accepted verbatim on every later pass, so the uppercase spelling written once looked operator-authored to the rewriter from then on and its repair never ran. Move the helper to `kiro_crew.env.resolved_command_casing`, beside the `mcp_search_path` the three resolvers already share, and apply it on both return paths of `_resolve_command` and in the probe. Repairing at the resolver also keeps the provenance record's `emitted` value repaired, so `command_is_ours` still recognises the entry and re-derivation stays on. Fixes #11627 (cherry picked from commit aa7c0bd, #15071; applied as its pre-#14919 revision 457210d plus the round-2 test and docstring delta, since release/0.8.0 predates the agent.py split)
Problem / Motivation
Goal: A gateway that was started inside an SSH session keeps spawning agents, cron scripts and app backends after the user's last session logs out, instead of failing every spawn with "Failed to connect to bus" until it is restarted.
On a Linux host without lingering, logind stops the per-user systemd manager and removes
/run/user/<uid>(anduser-<uid>.slice) when the last login session ends.XDG_RUNTIME_DIRstays set in the gateway's environment._probe_cgroup_scope()decided "a user session exists" from the presence of that variable and cached the answer for the life of the process, so a gateway that probed while the user was logged in kept prependingsystemd-run --user --scopeto every spawn.systemd-runcan no longer reach any bus and exits 1 withFailed to connect to bus: No such file or directorybefore it execs the command it wraps. Every agent respawn, script cron and app backend then fails, with no self-heal, until someone restarts the gateway; auto-paused crons also stay paused after that restart. A gateway restarted after the logout probes correctly and degrades, so two gateways disagreed about the same host.Reported on a Cloud Desktop (internal ticket: https://t.corp.amazon.com/187b5c6e-b5cb-4d1b-9c36-ccf8c598e4f4), where the only workaround,
loginctl enable-linger, needs sudo.Why it matters
Every Linux user who starts or auto-updates the gateway over SSH on a host without lingering loses all sandboxed spawns at logout. The gateway looks healthy while every spawn fails, and the error only shows up as a per-cron
last_erroror as agents that cannot respawn.Not a goal
What changed (motivation → approach → change)
Symptom → root cause: the probe's session half (user manager, bus, delegated slice) belongs to the user's login, but it was cached as if it were process-stable, and it trusted an environment variable instead of the bus.
Approach: make the cached answer follow the session, cheaply enough to check at every spawn on the event loop, and use the fallback the sandbox already has.
src/kiro_crew/sandbox.py:_compute_cgroup_scope_probenow requires a reachable bus (_user_bus_reachable). It tries a non-blockingAF_UNIXconnect()on the socketssystemd-run --userdials: theunix:path=entries ofDBUS_SESSION_BUS_ADDRESS, then$XDG_RUNTIME_DIR/systemd/privateand$XDG_RUNTIME_DIR/bus. A missing socket, a stale socket file with no listener, or a seccomp/LSMEPERM(cgroup scope probe reports available when D-Bus socket is unreachable #11473) all count as unavailable.EAGAINfrom a full backlog counts as reachable, so the connect can never stall the loop or drop the ceiling under load. The "unavailable" reason names the remedy (loginctl enable-linger $USER)._probe_cgroup_scopekeys its cache on_cgroup_scope_session_fingerprint(): stat-only(st_dev, st_ino)of the runtime dir, each bus socket and the user slice'scgroup.controllers, plus the locator values. It recomputes when the fingerprint changes, and in any case every_CGROUP_SCOPE_PROBE_TTL_SECONDS(60s). The TTL catches a manager that died and left its socket behind. A recompute runs under a lock and also drops the_cpu_controller_delegatedcache, which was cached for life too._log_cgroup_scope_transition. A loss is a SECURITY warning stating that new spawns run without the fork-bomb/memory ceilings and naming the reason and remedy, and it is emitted even if the one-time startup warning already fired. A restore is logged at info. The ceiling therefore never drops silently, and it comes back on its own when the user logs in again.Guarantees that stay the same:
systemd-runscope, properties and slice).cgroup_scope_argvpassthrough, the same path a gateway started without a manager takes.cgroup_scope_bus_envreads the same gate, so no bus locator is forwarded to a spawn that is not wrapped. If the two calls insandboxed_spawn_argvstraddle a flip, theenv -ushim stays inside the wrapped argv, so the child still never sees the locators.create_subprocess_limited/run_limited/popen_limitedvia the spawn shim) is untouched, soRLIMIT_NOFILEand the other rlimits still apply after exec on the fallback path.Docs:
docs/architecture/resource-protection.md§Availability and fallback anddocs/system-specs/modules/security.mddescribe the session-following gate.Backwards compatibility
Compatible: when a user manager is reachable the probe returns the same
(True, "ok")and the argv is unchanged. Hosts without one already degraded. The only behaviour change is on hosts where the old probe said "available" butsystemd-runcould not run, where every spawn previously failed.Tests
test/test_cgroup_scope_probe_liveness.py(new, Linux-only) stands up a fake user manager: a real listeningAF_UNIXsocket in a runtime dir undertmp_pathplus a fakecgroup.controllers. The host's real user manager is never contacted, because the conftest session floor already strips the real locators.test_logout_after_probe_stops_wrapping_spawns_in_systemd_runis the reported sequence: probe while live, then remove the runtime dir with the variables still set. The spawn must be unwrapped, no locators are forwarded, and a SECURITY warning namesenable-linger. Red on main: argv was still/usr/bin/systemd-run ….test_ceiling_returns_when_the_manager_comes_back: re-login bounds spawns again without a restart.test_transition_warns_once_not_per_spawn,test_a_second_outage_is_reported_too: one warning per flip, including a later outage after the startup warning.test_bus_gone_but_slice_still_delegated_is_unavailable,test_stale_bus_socket_with_no_listener_is_unavailable,test_bus_connect_refused_by_policy_is_unavailable(cgroup scope probe reports available when D-Bus socket is unreachable #11473): the bus gate on its own.test_a_listener_with_a_full_backlog_is_reachable_and_never_blocks:EAGAINcounts as reachable.test_unchanged_session_reuses_the_cached_probe,test_cached_probe_is_rechecked_after_its_ttl: one compute per session state, plus the TTL.test_cpu_delegation_is_reread_with_the_session.All 9 of the original tests failed on main; all 11 pass. Each invariant line was mutation-checked and reds at least one test: fingerprint comparison removed, bus connect removed, cpu-cache reset removed, transition log removed,
EAGAINtreated as unreachable, TTL removed, remedy dropped from the reason. Existing suites that use the probe (test_sandbox_argv,test_per_runtime_process_resources,test_sandbox_agent_slice_identity,test_session_scope_reapand every other test referencingcgroup_scope) pass. flake8, mypy (src/kiro_crew, 0 issues), black, isort and docs-lint are clean.Manual verification
On a Linux host with systemd 252 (the real user manager was left running and never stopped):
env XDG_RUNTIME_DIR=/tmp/<gone> DBUS_SESSION_BUS_ADDRESS=unix:path=/tmp/<gone>/bus systemd-run --user --scope -q -- trueprintedFailed to connect to bus: No such file or directoryand exited 1. This is the reported symptom, with the env a logged-out gateway carries.(True, 'ok')with the live session. Pointing the locators at the removed dir madecgroup_scope_argvreturn the unwrapped argv, which ran and printed its output, and logged the SECURITY transition warning namingloginctl enable-linger. Restoring the locators wrapped the next spawn in/usr/bin/systemd-runagain and logged "available again".Related Issues
Fixes #11473
Internal report: https://t.corp.amazon.com/187b5c6e-b5cb-4d1b-9c36-ccf8c598e4f4. Related but not duplicated: #7363 and #14955 (pod-side handling of hosts with no per-user manager), #11352.
Pattern harvest
Rule candidate: review-prompt
Pattern: a process-lifetime cache of host state that belongs to a login session or other external lifecycle (the user manager,
/run/user/<uid>, a bus) must be keyed on a cheap fingerprint of that state or a TTL; "the environment is process-stable" does not hold for the systemd user session.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement