Skip to content

fix(sandbox): backport the post-logout scope probe fix to release/0.8.0 - #15624

Merged
bolichen97 merged 1 commit into
release/0.8.0from
backport/0.8.0-scope-probe-bus-loss
Oct 1, 2026
Merged

bolichen97 merged 1 commit into
release/0.8.0from
backport/0.8.0-scope-probe-bus-loss

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Backport of #15622 (merged on main as 321fd99) and its review follow-up #15636 to release/0.8.0: two commits, each cherry-pick -x. The one doc conflict, in docs/system-specs/modules/security.md, was resolved by keeping the release-branch paragraph and appending the same sentence. Draft until #15636 merges on main. The follow-up resolves systemd-run through platform_compat.trusted_system_bin instead of shutil.which, so a probe refresh never walks PATH on the event loop, and percent-decodes unix:path= bus addresses.

Goal: A gateway that was started inside an SSH session keeps spawning agents, cron scripts and app backends after the user's last session logs out, instead of failing every spawn with "Failed to connect to bus" until it is restarted.

On a Linux host without lingering, logind stops the per-user systemd manager and removes /run/user/<uid> (and user-<uid>.slice) when the last login session ends. XDG_RUNTIME_DIR stays set in the gateway's environment. _probe_cgroup_scope() decided "a user session exists" from the presence of that variable and cached the answer for the life of the process, so a gateway that probed while the user was logged in kept prepending systemd-run --user --scope to every spawn. systemd-run can no longer reach any bus and exits 1 with Failed to connect to bus: No such file or directory before it execs the command it wraps. Every agent respawn, script cron and app backend then fails, with no self-heal, until someone restarts the gateway; auto-paused crons also stay paused after that restart. A gateway restarted after the logout probes correctly and degrades, so two gateways disagreed about the same host.

Reported on a Cloud Desktop (internal ticket: https://t.corp.amazon.com/187b5c6e-b5cb-4d1b-9c36-ccf8c598e4f4), where the only workaround, loginctl enable-linger, needs sudo.

Why it matters

Every Linux user who starts or auto-updates the gateway over SSH on a host without lingering loses all sandboxed spawns at logout. The gateway looks healthy while every spawn fails, and the error only shows up as a per-cron last_error or as agents that cannot respawn.

Not a goal

What changed (motivation → approach → change)

Symptom → root cause: the probe's session half (user manager, bus, delegated slice) belongs to the user's login, but it was cached as if it were process-stable, and it trusted an environment variable instead of the bus.

Approach: make the cached answer follow the session, cheaply enough to check at every spawn on the event loop, and use the fallback the sandbox already has.

src/kiro_crew/sandbox.py:

  • _compute_cgroup_scope_probe now requires a reachable bus (_user_bus_reachable). It tries a non-blocking AF_UNIX connect() on the sockets systemd-run --user dials: the unix:path= entries of DBUS_SESSION_BUS_ADDRESS, then $XDG_RUNTIME_DIR/systemd/private and $XDG_RUNTIME_DIR/bus. A missing socket, a stale socket file with no listener, or a seccomp/LSM EPERM (cgroup scope probe reports available when D-Bus socket is unreachable #11473) all count as unavailable. EAGAIN from a full backlog counts as reachable, so the connect can never stall the loop or drop the ceiling under load. The "unavailable" reason names the remedy (loginctl enable-linger $USER).
  • _probe_cgroup_scope keys its cache on _cgroup_scope_session_fingerprint(): stat-only (st_dev, st_ino) of the runtime dir, each bus socket and the user slice's cgroup.controllers, plus the locator values. It recomputes when the fingerprint changes, and in any case every _CGROUP_SCOPE_PROBE_TTL_SECONDS (60s). The TTL catches a manager that died and left its socket behind. A recompute runs under a lock and also drops the _cpu_controller_delegated cache, which was cached for life too.
  • An availability flip is logged by _log_cgroup_scope_transition. A loss is a SECURITY warning stating that new spawns run without the fork-bomb/memory ceilings and naming the reason and remedy, and it is emitted even if the one-time startup warning already fired. A restore is logged at info. The ceiling therefore never drops silently, and it comes back on its own when the user logs in again.

Guarantees that stay the same:

  • When the manager is present, the argv is byte-identical (same systemd-run scope, properties and slice).
  • Without a manager, spawns take the existing cgroup_scope_argv passthrough, the same path a gateway started without a manager takes.
  • cgroup_scope_bus_env reads the same gate, so no bus locator is forwarded to a spawn that is not wrapped. If the two calls in sandboxed_spawn_argv straddle a flip, the env -u shim stays inside the wrapped argv, so the child still never sees the locators.
  • The post-exec rlimit delivery (create_subprocess_limited / run_limited / popen_limited via the spawn shim) is untouched, so RLIMIT_NOFILE and the other rlimits still apply after exec on the fallback path.

Docs: docs/architecture/resource-protection.md §Availability and fallback and docs/system-specs/modules/security.md describe the session-following gate.

Backwards compatibility

Compatible: when a user manager is reachable the probe returns the same (True, "ok") and the argv is unchanged. Hosts without one already degraded. The only behaviour change is on hosts where the old probe said "available" but systemd-run could not run, where every spawn previously failed.

Tests

test/test_cgroup_scope_probe_liveness.py (new, Linux-only) stands up a fake user manager: a real listening AF_UNIX socket in a runtime dir under tmp_path plus a fake cgroup.controllers. The host's real user manager is never contacted, because the conftest session floor already strips the real locators.

  • test_logout_after_probe_stops_wrapping_spawns_in_systemd_run is the reported sequence: probe while live, then remove the runtime dir with the variables still set. The spawn must be unwrapped, no locators are forwarded, and a SECURITY warning names enable-linger. Red on main: argv was still /usr/bin/systemd-run ….
  • test_ceiling_returns_when_the_manager_comes_back: re-login bounds spawns again without a restart.
  • test_transition_warns_once_not_per_spawn, test_a_second_outage_is_reported_too: one warning per flip, including a later outage after the startup warning.
  • test_bus_gone_but_slice_still_delegated_is_unavailable, test_stale_bus_socket_with_no_listener_is_unavailable, test_bus_connect_refused_by_policy_is_unavailable (cgroup scope probe reports available when D-Bus socket is unreachable #11473): the bus gate on its own.
  • test_a_listener_with_a_full_backlog_is_reachable_and_never_blocks: EAGAIN counts as reachable.
  • test_unchanged_session_reuses_the_cached_probe, test_cached_probe_is_rechecked_after_its_ttl: one compute per session state, plus the TTL.
  • test_cpu_delegation_is_reread_with_the_session.

On unfixed release/0.8.0 10 of the original 11 tests fail; with both commits all 13 pass (including test_a_percent_escaped_bus_address_is_decoded and test_a_probe_refresh_never_walks_path). Probe-consuming suites on the release branch pass (4973), flake8 and mypy clean. Each invariant line was mutation-checked and reds at least one test: fingerprint comparison removed, bus connect removed, cpu-cache reset removed, transition log removed, EAGAIN treated as unreachable, TTL removed, remedy dropped from the reason. Existing suites that use the probe (test_sandbox_argv, test_per_runtime_process_resources, test_sandbox_agent_slice_identity, test_session_scope_reap and every other test referencing cgroup_scope) pass. flake8, mypy (src/kiro_crew, 0 issues), black, isort and docs-lint are clean.

Manual verification

On a Linux host with systemd 252 (the real user manager was left running and never stopped):

  • env XDG_RUNTIME_DIR=/tmp/<gone> DBUS_SESSION_BUS_ADDRESS=unix:path=/tmp/<gone>/bus systemd-run --user --scope -q -- true printed Failed to connect to bus: No such file or directory and exited 1. This is the reported symptom, with the env a logged-out gateway carries.
  • In one process with the fixed code: the probe was (True, 'ok') with the live session. Pointing the locators at the removed dir made cgroup_scope_argv return the unwrapped argv, which ran and printed its output, and logged the SECURITY transition warning naming loginctl enable-linger. Restoring the locators wrapped the next spawn in /usr/bin/systemd-run again and logged "available again".

Related Issues

Backport of #15622 (Fixes #11473 on main)

Internal report: https://t.corp.amazon.com/187b5c6e-b5cb-4d1b-9c36-ccf8c598e4f4. Related but not duplicated: #7363 and #14955 (pod-side handling of hosts with no per-user manager), #11352.

Pattern harvest

Rule candidate: review-prompt
Pattern: a process-lifetime cache of host state that belongs to a login session or other external lifecycle (the user manager, /run/user/<uid>, a bus) must be keyed on a cheap fingerprint of that state or a TTL; "the environment is process-stable" does not hold for the systemd user session.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@bolichen97 bolichen97 changed the title fix(sandbox): stop wrapping spawns in systemd-run after logout fix(sandbox): backport the post-logout scope probe fix to release/0.8.0 Oct 1, 2026
The cgroup-scope probe was computed once per process from the presence
of XDG_RUNTIME_DIR and cached for life. On a host without lingering,
logind stops the per-user manager and removes /run/user/<uid> at the
last logout while the variable stays set, so a gateway started inside
an SSH session kept prepending `systemd-run --user --scope` and every
agent respawn, cron script and app backend died with "Failed to
connect to bus" until the gateway was restarted.

The probe now requires a bus socket that accepts a non-blocking
connect() (which also covers a stale socket file and a seccomp policy
refusing the connect, #11473), and its cache is keyed on a stat-only
fingerprint of the runtime dir, bus sockets and user slice, re-checked
per spawn, with a 60s TTL behind it. When the manager is gone, spawns
take the existing no-scope path a gateway started after logout takes,
with a SECURITY warning naming `loginctl enable-linger`; the ceiling
comes back on its own when the manager returns. The cpu-delegation
cache is refreshed with it.

Refs: #11473
(cherry picked from commit 02837e8)
@bolichen97
bolichen97 force-pushed the backport/0.8.0-scope-probe-bus-loss branch from a720a08 to eb3e959 Compare October 1, 2026 00:34
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of eb3e9592de5dc6e6e244cb631f27fadf85ca3a21 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify #15622 has merged on main unchanged before this backport lands, so release/0.8.0 never diverges from the mainline fix.

[DESIGN-REVIEWED] eb3e959

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of eb3e9592de5dc6e6e244cb631f27fadf85ca3a21 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The root cause has three session-scoped caches; this fixes two — _SLICE_MEMHIGH_APPLIED still outlives the login, so the aggregate ceiling silently stays off after re-login.

Not justified as shipped

  • Item 4 — symptom-level: "it bounds spawns again once the manager returns" holds for per-scope ceilings only. The root cause named ("session-scoped state cached as process-stable") has three instances in sandbox.py (grepped _CGROUP_SCOPE_PROBE|_CPU_DELEGATED|_SLICE_MEMHIGH_APPLIED): the probe and _CPU_DELEGATED are fixed, but _ensure_agent_slice_memory_high still early-returns on desired == _SLICE_MEMHIGH_APPLIED (src/kiro_crew/sandbox.py:13409) although its own docstring says the --runtime drop-in "vanishes with the login session" (:13397) — so after the re-login this PR now handles, the slice-level MemoryHigh is never re-applied and no transition warning fires.
    Clears when: the recompute that already drops _CPU_DELEGATED (sandbox.py:13188) also drops _SLICE_MEMHIGH_APPLIED (in fix(sandbox): stop wrapping spawns in systemd-run after logout #15622 first, then this backport), or the description states the aggregate ceiling is left unfixed after re-login.

What this change ships

Inventory (8 items) — 7 justified

Intent: FIX — a gateway started over SSH keeps spawning after the last logout instead of failing every spawn until restart (backport of #15622; linked ticket, red-on-main test).

  1. Post-logout spawns run unwrapped instead of failing with "Failed to connect to bus" — justified
  2. Probe now requires a reachable bus socket; a stale socket file or policy-refused connect counts as unavailable — justified
  3. Cached probe re-validated per spawn against a stat fingerprint plus a 60s TTL — justified
  4. Ceiling returns on re-login without a restart — symptom-level: the aggregate MemoryHigh sibling cache is left unfixed (see above)
  5. Mid-run availability flips are logged; a loss is a SECURITY warning naming the remedy — justified
  6. Unavailable reason now names loginctl enable-linger $USER — justified
  7. CPU-delegation answer is re-read with the session — justified
  8. Docs updated in the same commit — justified

[FIRST-PRINCIPLES-REVIEWED] eb3e959

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging eb3e9592de5dc6e6e244cb631f27fadf85ca3a21. 2 of 2 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/sandbox.py:14415 -- Probe refresh walks PATH on the event loop
result = _compute_cgroup_scope_probe() → shutil.which("systemd-run")
TTL expiry -> dashboard spawn path -> synchronous PATH traversal -> gateway loop freezes on a stalled mount.
Anchor: no-blocking-call-on-event-loop
Fix: Use platform_compat.trusted_system_bin("systemd-run").

BLOCKING -- src/kiro_crew/sandbox.py:14312 -- Escaped D-Bus paths disable cgroup ceilings
paths.append(value)
Percent-escaped unix:path= with no runtime-dir socket -> live bus is missed -> agent processes launch without memory/process ceilings.
Anchor: residual/security
Fix: Decode with os.fsdecode(urllib.parse.unquote_to_bytes(value)).

FINDING -- src/kiro_crew/acp/runtime.py:1057 -- _runtime_rpc_exception and the other unrelated backports materially exceed the stated sandbox-only scope -> Fix: remove the six unrelated backport commits from this PR.

[BLOCK-MERGE] eb3e959
[GPT-REVIEWED] eb3e959

Adjudication (Opus 5) — is blocking on each finding proportionate?

Opened sandbox.py:14241-14470, platform_compat.py:2946-2976, and the on-loop callers at dashboard/handlers/sessions.py:815,1152 versus the offloaded ones at acp/runtime.py:2753, acp/client.py:9521.

F1 — Harm rung: HIGH (gateway loop stall, no self-recovery while the mount hangs). Conditions confirmed: Linux only (sandbox.py:14454); a cache miss now recurs every 60s or on any fingerprint change (sandbox.py:14401-14418, a line this PR adds — before it the compute ran once per process at the old if _CGROUP_SCOPE_PROBE is None); the recompute runs on the loop at dashboard/handlers/sessions.py:815 and in the periodic usage-scrape refresh at :1152, which deliberately offload wrap_argv to an executor but not cgroup_scope_argv; shutil.which stats one entry per PATH directory (sandbox.py:14456). A PATH leading with home-backed directories on a hung network mount is ordinary on exactly the managed hosts this code's own remedy text names, and platform_compat.py:2949-2951 documents agent-writable leading PATH entries as legitimate for a gateway — so the condition set is not extreme. Real fix: one-line swap to the existing trusted_system_bin helper, which probes fixed system dirs.

F2 — Harm rung: HIGH (fork-bomb/memory ceiling silently absent for new spawns), but logged. Conditions confirmed: raw append with no unquoting (sandbox.py:14305-14312); the probe returns early unless XDG_RUNTIME_DIR is set (sandbox.py:14461), so the escaped entry can only decide the answer when the var IS set and both derived candidates — $XDG_RUNTIME_DIR/systemd/systemd/private… precisely systemd/private and bus (:14315-14316) — are unreachable, since the first reachable path short-circuits True (:14366-14374). A live user manager creates its private socket inside that same runtime dir, so reaching F2 needs an already-inconsistent environment (runtime dir without the manager's sockets) and a bus path containing a character outside dbus's unreserved set, which systemd's own /run/user/<uid>/bus never does. Recovery: the flip is logged as a SECURITY warning naming each unreachable path with its errno (:14376, :14421-14422, :14443-14449), RLIMIT_NOFILE and the OS sandbox are unaffected, and the TTL recompute restores the ceiling on its own once a reachable socket appears (:14404). Real fix: one-line unquote_to_bytes + os.fsdecode.

[ADJUDICATION] eb3e959 total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] eb3e959

[ADJUDICATION-FENCED] eb3e959 fenced=2 flagged=1
UPHOLD-FENCED F1 src/kiro_crew/sandbox.py:14415 -- The PR turns a once-per-process PATH walk into a recurring on-loop one at sessions.py:815/1152, and a home-backed PATH entry on a hung network mount is ordinary on the managed hosts this code targets, so no rarity argument is available.
FLAG F2 src/kiro_crew/sandbox.py:14312 -- The escaped entry can only decide the answer when XDG_RUNTIME_DIR is set (14461) yet both manager-derived sockets are unreachable (14315-14316, 14366-14374) while a live bus sits at a path needing percent-escaping, a combination systemd's own socket naming cannot produce; and the resulting ceiling loss is logged with each failing path (14376, 14443-14449) and self-restores on the next TTL recompute (14404).
[GPT-ADJUDICATED-FENCED] eb3e959

🏷️ Fenced finding(s) machine-flagged as likely edge case

The security fence keeps these findings blocking regardless of adjudication; the only clearance path is a human override. The annotate-only pass judged the condition combination each one requires extreme, and pre-drafted the override rationale below. A repository writer must independently verify a rationale before posting it — it is machine-authored, and a wrong override on a security-class finding ships exactly the class the fence exists to stop.

  • F2 src/kiro_crew/sandbox.py:14312 — The escaped entry can only decide the answer when XDG_RUNTIME_DIR is set (14461) yet both manager-derived sockets are unreachable (14315-14316, 14366-14374) while a live bus sits at a path needing percent-escaping, a combination systemd's own socket naming cannot produce; and the resulting ceiling loss is logged with each failing path (14376, 14443-14449) and self-restores on the next TTL recompute (14404).

    /ai-review override gpt eb3e9592de5dc6e6e244cb631f27fadf85ca3a21: The escaped entry can only decide the answer when XDG_RUNTIME_DIR is set (14461) yet both manager-derived sockets are unreachable (14315-14316, 14366-14374) while a live bus sits at a path needing percent-escaping, a combination systemd's own socket naming cannot produce; and the resulting ceiling loss is logged with each failing path (14376, 14443-14449) and self-restores on the next TTL recompute (14404).
    

False positive or not applicable? A repository writer can comment:
/ai-review override gpt eb3e9592de5dc6e6e244cb631f27fadf85ca3a21: <one-sentence reason>

@bolichen97
bolichen97 marked this pull request as ready for review October 1, 2026 00:56
@bolichen97
bolichen97 requested a review from a team as a code owner October 1, 2026 00:56
@bolichen97
bolichen97 requested review from cruisercohen and removed request for a team October 1, 2026 00:56
@bolichen97
bolichen97 merged commit b8a35c5 into release/0.8.0 Oct 1, 2026
20 of 21 checks passed
@bolichen97
bolichen97 deleted the backport/0.8.0-scope-probe-bus-loss branch October 1, 2026 00:56
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed eb3e9592de5dc6e6e244cb631f27fadf85ca3a21 — this comment is updated in place on each push.

Review details

No findings.

FINDING — src/kiro_crew/sandbox.py:13204 — the recovery branch resets _CPU_DELEGATED (in _probe_cgroup_scope) but not the login-scoped slice memos, so after a logout/re-login the --runtime MemoryHigh drop-in that logind destroyed with /run/user/<uid> is never re-applied — _ensure_agent_slice_memory_high early-returns on if desired == _SLICE_MEMHIGH_APPLIED (host-derived, process-constant), and a reconcile worker that lands during the outage latches _SLICE_MEMHIGH_DISABLED = True for the process's life — leaving the aggregate soft ceiling off while this branch logs "are bounded by the fork-bomb / memory-DoS ceilings once more" (the hard slice cap self-heals via _slice_oom_report; MemoryHigh has no equivalent) → Fix: in the available branch clear _SLICE_MEMHIGH_APPLIED = None and _SLICE_MEMHIGH_DISABLED = False alongside the existing _CPU_DELEGATED reset, and narrow the message to the per-scope ceilings it actually re-establishes.

[OPUS-REVIEWED] eb3e959

Verdict parsed from the review's SHA-scoped output markers for commit eb3e9592de5dc6e6e244cb631f27fadf85ca3a21.

False positive or not applicable? A repository writer can comment:
/ai-review override fable eb3e9592de5dc6e6e244cb631f27fadf85ca3a21: <one-sentence reason>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant