Skip to content

fix(sandbox): run the mount-source pin scan in a spawned child - #33195

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/33171-pin-scan-child
Nov 8, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/33171-pin-scan-child

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: On a busy Linux host, the sandbox mount-source sweep's pin scan no longer competes with the gateway's own threads for the Python interpreter lock, and it still never reclaims a directory it could not prove unused.

The pin scan reads /proc for every task on the host (a stat, two link probes and a mountinfo read each). It ran inside the gateway, so every one of those syscalls had to take the GIL back. On hosts with thousands of processes the crew-log writers, the context-build pool and the event loop queued behind it, and the scan itself was slowed so much by the other threads that it often ended "incomplete", leaving the stand-in directories unreclaimed (#33171).

Why it matters

Operators on busy shared hosts saw event-loop lag, slow dashboard pages and messages that never confirmed delivery while a sweep pass ran, plus a sweep that logged pin scan incomplete and descendant coverage unproven every pass for days.

Not a goal

What changed (motivation -> approach -> change)

Symptom: gateway threads parked in take_gil during sweep passes, and a pin scan that took minutes and came back incomplete. Cause: the scan's tens of thousands of /proc syscalls ran in the gateway's interpreter. Its answer is small (the pinned names plus two flags), so it moves to its own process.

  • New src/kiro_crew/sandbox_pin_scan_child.py: on Linux, a scan of the host's /proc runs as python -P -m kiro_crew.sandbox_pin_scan_child keyed|legacy, a fresh interpreter started through subprocess (no fork of the threaded gateway) at a lower CPU priority. It writes one JSON line with the pinned names, complete and covered, then exits. This follows the consolidation write child's launch pattern.
  • Fail closed: a child that cannot start, runs past 300 s, crashes, exits non-zero or writes anything but that one well-formed line answers as an incomplete scan with coverage unproven. The pass then reclaims no directory, the same as an incomplete scan today, and logs a warning naming the reason.
  • In src/kiro_crew/sandbox_mount_sweep.py, _mount_pinned_source_names and _bound_source_basenames send the host /proc scan on Linux to the child. Any other proc root, a custom matcher and every other platform keep the in-process scan, so the unit tests over synthetic proc trees and macOS/Windows behave as before.
  • The scan still reads every task, and each mount view once (Stale-sandbox pin scan re-parses thousands of identical mountinfo tables per tick, holding the GIL and stalling the event loop #12719). The issue's uid pre-filter (skip other users' tasks) was left out because skipping a task cannot be proven safe for non-sticky shared temp roots.
  • docs/system-specs/modules/security.md: the sandbox_mount_sweep paragraph now states the child process and its fail-closed rule.

Measured on a 64-core host with about 1,400 processes and 7,200 threads (about 3,100 pinned names). A small stand-in for the gateway ran the scan on a worker thread, as the maintenance executor does, beside an asyncio heartbeat (every 50 ms) and 16 probe threads that sleep 2 ms and record how late they wake. "Before" is the parent commit, "after" this branch:

Run Before (scan in the gateway) After (scan in the child)
Pass time, nothing else running 1.7 s 1.6 to 1.8 s
Pass time, 4 threads doing 1 ms of Python every 5 ms 9.0 to 9.9 s 1.6 to 1.7 s
Pass time, 1 thread that never pauses 199 s, scan incomplete, coverage unproven 2.4 s, complete and covered
Probe wake-up lag (max) with the 4 threads 2.3 to 11.9 ms 2.0 to 11.7 ms (no-scan floor 1.8 to 2.0 ms)
Heartbeat lag (max) with the 4 threads 1.7 to 2.8 ms 0.8 to 4.1 ms (no-scan floor 0.7 to 1.8 ms)

On this host the lag of the other threads stays within noise in both versions. The clear effect is on the scan: in the gateway it slows by 5x next to modest Python work, and next to one busy thread it takes over three minutes and ends incomplete, which is the "pin scan incomplete" warning the issue reports. In the child it takes the same time whatever the gateway is doing. Native stack samples (eu-stack) of the stand-in showed no thread in take_gil in either version with nothing else running.

Backwards compatibility

Compatible: the sweep's inputs, decisions and log lines are unchanged; only where the scan runs moves. A failed child answers like an incomplete scan, which callers already handle by keeping every directory.

Removes nothing: no path, record, key or function is taken away; the scan reads the same tasks as before.

Tests

New test/test_sandbox_pin_scan_child.py:

  • test_real_child_scans_the_host_proc: a real spawned child reads the real /proc, and on Linux both _mount_pinned_source_names and _bound_source_basenames answer through it (keyed, then legacy).
  • test_failed_child_answers_none[...], one case per failure: cannot start, timeout (the child would reply well-formed after its sleep, so only the timeout yields None), crash (SIGKILL), non-zero exit with a valid reply, not JSON, wrong shape, empty output. Each checks the warning names the reason.
  • test_failed_child_keeps_a_dead_pid_directory and test_failed_child_leaves_the_legacy_scan_unproven: a failed child through the real sweep keeps a dead-pid directory and leaves the legacy scan unproven.
  • test_child_scan_runs_the_matching_in_process_scan[keyed|legacy] and test_child_entry_writes_one_reply_line: the child side, run in-process, scans /proc with the right name shape and writes exactly one reply line that the parent parses back.
  • test_well_formed_reply_is_returned, test_child_entry_refuses_an_unknown_kind.

The spawn audit and the internal-Python isolation list register the new spawn site. The existing sweep suites (test_sandbox_mount_source_sweep.py, test_sandbox_mount_sweep_views.py, the facade and monkeypatch-undo tests) pass unchanged.

Determinism: the new file passed 20 runs at -n0, 10 runs at -n 4 --dist loadgroup beside two sibling files, and a shuffled order (seed 20811).

Proof: not provable -- other: the fix adds a new child module, so reverting the production hunks fails test collection

Manual verification

Ran the bench above against the parent commit and this branch on the same host, alternating the two so load drift hit both alike, and compared the child's pinned set and flags with the in-process scan on the same host.

Related Issues

Closes #33171

Related: #12719, #32177, #31730

Sibling sweep

Swept: git grep -nE 'listdir("/proc")|listdir(proc_root)|Path("/proc").iterdir' -- src/
Sibling: src/kiro_crew/dashboard/handlers_system.py:511 -- not affected -- one cmdline read per process, per page request, not a per-thread mountinfo walk on every sweep tick
Sibling: src/kiro_crew/platform_compat_proctable.py:1219 -- not affected -- a per-call descriptor lookup for one file, not part of the periodic sweep
Sibling: src/kiro_crew/platform_compat_liveness.py:66 -- not affected -- a single directory listing with no per-task reads

Pattern harvest

Not generalizable: the cost came from this one scan's size (every thread's mountinfo on the host, on a timer); the other /proc readers read one small file per process on demand.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/sandbox_mount_sweep.py

  • src/kiro_crew/sandbox_pin_scan_child.py

  • @chenmingwei23 posted a complete reasoning template for 7fb162d.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 7fb162d39487c0efd60d96f7b293f1502ea7037d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 7fb162d39487c0efd60d96f7b293f1502ea7037d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the scan's answer is small, the child returns exactly the fields the gate reads (pinned, complete, covered), and every way the child can fail counts as an unproven scan, so that pass reclaims no directory.

Notes

[DESIGN-REVIEWED] 7fb162d

Description read

[DESCRIPTION-READ] 51b2cdcd4cc2bf953aa248479e3e9a2203691130dc81ed88a1aa0e85deb042f4

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 7fb162d39487c0efd60d96f7b293f1502ea7037d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the scan leaves the gateway interpreter, which is the cause the issue names, and a failed child is treated as an incomplete scan.

Notes

  • Item 6 — zero consumers: the argv= and timeout= parameters on scan_in_child are passed only by tests (the two sweep callers pass just kind), per the docstring: "tests use it to make a child that hangs, crashes or replies badly". Subtraction: drop both parameters, use the SCAN_TIMEOUT_SECS constant, and have the tests monkeypatch platform_compat.isolated_python_argv / SCAN_TIMEOUT_SECS instead.

What this change ships

Inventory (7 items) — 6 justified

Intent: on busy Linux hosts, stop the mount-source sweep's pin scan from slowing the gateway and from ending "incomplete" every pass (#33171). FIX.

  1. On Linux, the host /proc pin scan (keyed and legacy) runs in a spawned child process instead of in the gateway — justified
  2. A child that cannot start, runs past 300 s, crashes or replies malformed counts as an incomplete scan with coverage unproven, so no directory is reclaimed — justified
  3. A new warning log line names why the child failed — justified
  4. The child runs at nice +10 — rides along, undeclared
  5. Synthetic proc roots, custom matchers and non-Linux platforms keep the in-process scan — justified
  6. scan_in_child takes argv / timeout parameters that only tests use — zero consumers
  7. The security spec states the child process and its fail-closed rule; the spawn audit lists the new spawn site — justified

[FIRST-PRINCIPLES-REVIEWED] 7fb162d

Description read

[DESCRIPTION-READ] ce18df00571ab342f8b5ed107f93823d4c54d15243ce0c308fead327433b8e87

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 7fb162d39487c0efd60d96f7b293f1502ea7037d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7fb162d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7fb162d39487c0efd60d96f7b293f1502ea7037d false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 7fb162d39487c0efd60d96f7b293f1502ea7037d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7fb162d

Verdict parsed from the review's SHA-scoped output markers for commit 7fb162d39487c0efd60d96f7b293f1502ea7037d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7fb162d39487c0efd60d96f7b293f1502ea7037d false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 8, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/33171-pin-scan-child branch from 60d596d to d5788a4 Compare November 8, 2026 14:04
@chenmingwei23

chenmingwei23 commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=b63ed0c39aa2 Bare undo fails the required determinism gate

fixed in d5788a4: test_another_users_task_is_not_read scopes its two patches with with monkeypatch.context() as mp: and no longer calls monkeypatch.undo(), so the autouse floor patches stay in place for the rest of the test.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: d5788a4
  • What rule changed: none of the sandbox's allow/deny rules. The sweep that reclaims the namespace launcher's leftover bind-mount stand-ins now runs its /proc pin scan in a spawned child process on Linux, and the scan no longer reads the mountinfo of another real user's task.
  • Before -> after (what was allowed/blocked, now): before, the gateway read every task's mountinfo in its own interpreter; after, a fresh python -P -m kiro_crew.sandbox_pin_scan_child reads it and returns the pinned names and coverage verdict as one JSON line. Which directories may be reclaimed is decided by the same gate (entry not pinned and (complete or covered)).
  • Worst case if wrong (breaks commands / leaks secret / locks user out): the child keeps failing, so every pass counts as an unproven scan and stand-in directories accumulate on the runtime tmpfs as they do today on a host whose scan never completes; nothing is removed that would not have been. A wrong skip of another user's task could at most drop a pin that user's process held, and such a process cannot bind a source this uid staged.
  • Evidence checked (which sandbox result proves it): test_sandbox_pin_scan_child.py runs the real child against the real /proc, and drives each failure (no start, timeout, crash, non-zero exit, malformed reply) through the real sweep, which keeps a dead-pid directory; test_another_users_task_is_not_read shows the foreign task's mountinfo is never opened while coverage holds.
  • No regression:
    • Tools still work (which tools/commands were run, result): the child scan and the in-process scan returned the same pinned count and flags on a 1,400-process host; the bench in the PR body ran both versions.
    • Backward compatible (old config / old data / old callers still work, how checked): no config or stored data changes; non-Linux platforms and synthetic proc roots keep the in-process scan, covered by the unchanged sweep suites.
    • Existing tests (which suites passed on this head): test_sandbox_mount_source_sweep.py, test_sandbox_mount_sweep_views.py, test_sandbox_refactor_facade.py, test_home_pin_survives_monkeypatch_undo.py, test_spawn_audit.py, test_internal_python_isolation.py and the new test_sandbox_pin_scan_child.py, all green locally on this tree.
  • How to undo: revert this commit; the scan runs in the gateway again.

@chenmingwei23
chenmingwei23 force-pushed the fix/33171-pin-scan-child branch from d5788a4 to 39d911f Compare November 8, 2026 14:16
@chenmingwei23

chenmingwei23 commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=f28364d8feb2 security.md sweep paragraph describes the scan wrongly

fixed in 39d911f: the sandbox_mount_sweep paragraph of docs/system-specs/modules/security.md now says another real uid's tasks are skipped without reading their mountinfo and lower only complete, that the Linux /proc scan runs in the spawned sandbox_pin_scan_child, and that a child that cannot start, times out, crashes, exits non-zero or replies malformed reclaims no directory.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 39d911f
  • What rule changed: none of the sandbox's allow/deny rules. The sweep that reclaims the namespace launcher's leftover bind-mount stand-ins now runs its /proc pin scan in a spawned child process on Linux, and the scan no longer reads the mountinfo of another real user's task.
  • Before -> after (what was allowed/blocked, now): before, the gateway read every task's mountinfo in its own interpreter; after, a fresh python -P -m kiro_crew.sandbox_pin_scan_child reads it and returns the pinned names and coverage verdict as one JSON line. Which directories may be reclaimed is decided by the same gate (entry not pinned and (complete or covered)).
  • Worst case if wrong (breaks commands / leaks secret / locks user out): the child keeps failing, so every pass counts as an unproven scan and stand-in directories accumulate on the runtime tmpfs as they do today on a host whose scan never completes; nothing is removed that would not have been. A wrong skip of another user's task could at most drop a pin that user's process held, and such a process cannot bind a source this uid staged.
  • Evidence checked (which sandbox result proves it): test_sandbox_pin_scan_child.py runs the real child against the real /proc, and drives each failure (no start, timeout, crash, non-zero exit, malformed reply) through the real sweep, which keeps a dead-pid directory; test_another_users_task_is_not_read shows the foreign task's mountinfo is never opened while coverage holds.
  • No regression:
    • Tools still work (which tools/commands were run, result): the child scan and the in-process scan returned the same pinned count and flags on a 1,400-process host; the bench in the PR body ran both versions.
    • Backward compatible (old config / old data / old callers still work, how checked): no config or stored data changes; non-Linux platforms and synthetic proc roots keep the in-process scan, covered by the unchanged sweep suites.
    • Existing tests (which suites passed on this head): test_sandbox_mount_source_sweep.py, test_sandbox_mount_sweep_views.py, test_sandbox_refactor_facade.py, test_home_pin_survives_monkeypatch_undo.py, test_spawn_audit.py, test_internal_python_isolation.py and the new test_sandbox_pin_scan_child.py, all green locally on this tree.
  • How to undo: revert this commit; the scan runs in the gateway again.

@chenmingwei23
chenmingwei23 force-pushed the fix/33171-pin-scan-child branch from 39d911f to baa9e9c Compare November 8, 2026 14:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 8, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author
  • span=f14748bcbc1a Preserve foreign-UID pins during a root sweep

fixed in baa9e9c: the other-user skip in _scan_mount_pins now applies only when the sweeper is not root (own_uid != 0). A root sweep reads every task and keeps another user's pins, so the unfiltered keyed sweep cannot remove a still-bound source; test_a_root_sweep_reads_every_users_task pins it. A non-root sweeper cannot remove another user's entry in the sticky /dev/shm or tmp roots, or in that user's /run/user dir.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: baa9e9c
  • What rule changed: none of the sandbox's allow/deny rules. The sweep that reclaims the namespace launcher's leftover bind-mount stand-ins now runs its /proc pin scan in a spawned child process on Linux, and, when the sweep does not run as root, the scan no longer reads the mountinfo of another real user's task (a root sweep still reads every task).
  • Before -> after (what was allowed/blocked, now): before, the gateway read every task's mountinfo in its own interpreter; after, a fresh python -P -m kiro_crew.sandbox_pin_scan_child reads it and returns the pinned names and coverage verdict as one JSON line. Which directories may be reclaimed is decided by the same gate (entry not pinned and (complete or covered)).
  • Worst case if wrong (breaks commands / leaks secret / locks user out): the child keeps failing, so every pass counts as an unproven scan and stand-in directories accumulate on the runtime tmpfs as they do today on a host whose scan never completes; nothing is removed that would not have been. A wrong skip of another user's task could at most drop a pin that user's process held, and such a process cannot bind a source this uid staged.
  • Evidence checked (which sandbox result proves it): test_sandbox_pin_scan_child.py runs the real child against the real /proc, and drives each failure (no start, timeout, crash, non-zero exit, malformed reply) through the real sweep, which keeps a dead-pid directory; test_another_users_task_is_not_read shows a non-root scan never opens the foreign task's mountinfo while coverage holds, and test_a_root_sweep_reads_every_users_task shows a root scan still reads it and keeps its pin.
  • No regression:
    • Tools still work (which tools/commands were run, result): the child scan and the in-process scan returned the same pinned count and flags on a 1,400-process host; the bench in the PR body ran both versions.
    • Backward compatible (old config / old data / old callers still work, how checked): no config or stored data changes; non-Linux platforms and synthetic proc roots keep the in-process scan, covered by the unchanged sweep suites.
    • Existing tests (which suites passed on this head): test_sandbox_mount_source_sweep.py, test_sandbox_mount_sweep_views.py, test_sandbox_refactor_facade.py, test_home_pin_survives_monkeypatch_undo.py, test_spawn_audit.py, test_internal_python_isolation.py and the new test_sandbox_pin_scan_child.py, all green locally on this tree.
  • How to undo: revert this commit; the scan runs in the gateway again.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 8, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/33171-pin-scan-child branch from baa9e9c to f80cc51 Compare November 8, 2026 15:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 8, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: f80cc51
  • What rule changed: none of the sandbox's allow/deny rules. The sweep that reclaims the namespace launcher's leftover bind-mount stand-ins now runs its /proc pin scan in a spawned child process on Linux, and, when the sweep does not run as root, the scan no longer reads the mountinfo of another real user's task (a root sweep still reads every task).
  • Before -> after (what was allowed/blocked, now): before, the gateway read every task's mountinfo in its own interpreter; after, a fresh python -P -m kiro_crew.sandbox_pin_scan_child reads it and returns the pinned names and coverage verdict as one JSON line. Which directories may be reclaimed is decided by the same gate (entry not pinned and (complete or covered)).
  • Worst case if wrong (breaks commands / leaks secret / locks user out): the child keeps failing, so every pass counts as an unproven scan and stand-in directories accumulate on the runtime tmpfs as they do today on a host whose scan never completes; nothing is removed that would not have been. A wrong skip of another user's task could at most drop a pin that user's process held, and such a process cannot bind a source this uid staged.
  • Evidence checked (which sandbox result proves it): test_sandbox_pin_scan_child.py runs the real child against the real /proc, and drives each failure (no start, timeout, crash, non-zero exit, malformed reply) through the real sweep, which keeps a dead-pid directory; test_another_users_task_is_not_read shows a non-root scan never opens the foreign task's mountinfo while coverage holds, and test_a_root_sweep_reads_every_users_task shows a root scan still reads it and keeps its pin.
  • No regression:
    • Tools still work (which tools/commands were run, result): the child scan and the in-process scan returned the same pinned count and flags on a 1,400-process host; the bench in the PR body ran both versions.
    • Backward compatible (old config / old data / old callers still work, how checked): no config or stored data changes; non-Linux platforms and synthetic proc roots keep the in-process scan, covered by the unchanged sweep suites.
    • Existing tests (which suites passed on this head): test_sandbox_mount_source_sweep.py, test_sandbox_mount_sweep_views.py, test_sandbox_refactor_facade.py, test_home_pin_survives_monkeypatch_undo.py, test_spawn_audit.py, test_internal_python_isolation.py and the new test_sandbox_pin_scan_child.py, all green locally on this tree.
  • How to undo: revert this commit; the scan runs in the gateway again.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 8, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 8, 2026
On Linux the sweep's pin scan of /proc now runs in a fresh interpreter
(python -P -m kiro_crew.sandbox_pin_scan_child keyed|legacy) that writes
the pinned names and the coverage verdict as one JSON line. In the
gateway every /proc syscall of the scan had to take the GIL back, so on
a busy host the scan slowed itself and every other gateway thread.

A child that cannot start, runs past 300 s, crashes, exits non-zero or
replies malformed counts as an incomplete scan with coverage unproven:
the pass reclaims no directory, as an incomplete in-process scan does.

The scan also stops reading another real user's tasks; they cannot bind
a source this uid staged, and they now lower only the host-wide flag.

Other platforms and any other proc root keep the in-process scan.

Closes #33171
@chenmingwei23
chenmingwei23 force-pushed the fix/33171-pin-scan-child branch from f80cc51 to 7fb162d Compare November 8, 2026 16:16
@chenmingwei23

Copy link
Copy Markdown
Contributor Author
  • span=f14748bcbc1a Preserve foreign-owned sources in non-sticky temp roots

fixed in 7fb162d: the other-uid skip is removed from _scan_mount_pins, so the scan reads every task again (as on main) and another user's live source stays pinned on any root, sticky or not. The spec sentence, docstring and the two foreign-uid tests went with it.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 7fb162d
  • What rule changed: none of the sandbox's allow/deny rules. The sweep that reclaims the namespace launcher's leftover bind-mount stand-ins now runs its /proc pin scan in a spawned child process on Linux,. The scan still reads every task, as before.
  • Before -> after (what was allowed/blocked, now): before, the gateway read every task's mountinfo in its own interpreter; after, a fresh python -P -m kiro_crew.sandbox_pin_scan_child reads it and returns the pinned names and coverage verdict as one JSON line. Which directories may be reclaimed is decided by the same gate (entry not pinned and (complete or covered)).
  • Worst case if wrong (breaks commands / leaks secret / locks user out): the child keeps failing, so every pass counts as an unproven scan and stand-in directories accumulate on the runtime tmpfs as they do today on a host whose scan never completes; nothing is removed that would not have been.
  • Evidence checked (which sandbox result proves it): test_sandbox_pin_scan_child.py runs the real child against the real /proc, and drives each failure (no start, timeout, crash, non-zero exit, malformed reply) through the real sweep, which keeps a dead-pid directory.
  • No regression:
    • Tools still work (which tools/commands were run, result): the child scan and the in-process scan returned the same pinned count and flags on a 1,400-process host; the bench in the PR body ran both versions.
    • Backward compatible (old config / old data / old callers still work, how checked): no config or stored data changes; non-Linux platforms and synthetic proc roots keep the in-process scan, covered by the unchanged sweep suites.
    • Existing tests (which suites passed on this head): test_sandbox_mount_source_sweep.py, test_sandbox_mount_sweep_views.py, test_sandbox_refactor_facade.py, test_home_pin_survives_monkeypatch_undo.py, test_spawn_audit.py, test_internal_python_isolation.py and the new test_sandbox_pin_scan_child.py, all green locally on this tree.
  • How to undo: revert this commit; the scan runs in the gateway again.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 8, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 8, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 8, 2026 17:29

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. On this head the review read the whole diff and checked each claim against the code. The child runs the same in-process scan (_mount_pinned_source_names_in_process) and shares the gateway's uid and mount namespace. The spawn runs only on the maintenance executor, never on the event loop, and no file on the boot path changed. The security.md paragraph matches the code. The new test file passes, and a mutation run showed that the child-fails path and the routing to the child are both pinned by tests.

Security: the PR adds a subprocess inside sandbox code that decides when bind-mount sources are reclaimed. Every child failure fails closed: it cannot start, it times out, it is killed, it exits non-zero, or its output is not JSON or has the wrong shape. Each of these returns None, which _scan_via_child turns into an incomplete scan with coverage.covered = False. The directory gate then keeps every candidate, and the legacy pass reclaims nothing and leaves its marker unset. A reply can only lower coverage, never raise it. The argv is fixed, runs on this interpreter with -P, takes no agent input, and is registered in the spawn audit. No seal, mask or refusal is added.

Done well: there is one test per child failure mode, and each failure is driven through the real sweep, which keeps a dead-pid directory and leaves the legacy scan unproven.

Optional:

  • When the child replies successfully, nothing pins how the parent reads the reply. Three fail-open mutations at sandbox_mount_sweep.py:389-392 all stayed green: dropping the covered propagation, returning an empty pinned set, and forcing complete=True. A follow-up test could stub scan_in_child with fixed replies such as ({"kirocrew_sb_X_a"}, True, True), (set(), False, False) and (set(), False, True), and assert what the sweep keeps and what it reclaims.
  • The benchmark proves that the scan now completes (199 s and incomplete before, 2.4 s and complete after). It does not show the gateway-lag relief described in "Why it matters". The body itself calls the lag within noise.
  • The argv= and timeout= parameters of scan_in_child, and the nice +10, are used only by tests or go beyond the goal. Both follow the consolidation_write_child.py precedent. Neither needs a change, but a short disposition on each lane note would close it.

@KiroCrewCaptain
KiroCrewCaptain merged commit 2da9ec8 into main Nov 8, 2026
112 of 116 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/33171-pin-scan-child branch November 8, 2026 17:30
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 8, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

The code looks good to me on 7fb162d. One step before I can approve: could you update this branch onto main, with a rebase rather than a merge commit (PR Hygiene allows only one or two commits)?

macOS Tests / Backend Tests (macOS) (8) fails on test_the_watchdog_budget_falls_back_to_the_managed_service_grace in test_dashboard_server_composition_contract_owners.py, twice on this head, and this change does not touch that code. The test was fixed on main by #33322, which merged at 19:05 UTC today, and this branch is 112 commits behind main without it. Once it is updated and CI is green, I will approve.

buluoray added a commit that referenced this pull request Nov 8, 2026
On Linux the sweep's pin scan of /proc now runs in a fresh interpreter
(python -P -m kiro_crew.sandbox_pin_scan_child keyed|legacy) that writes
the pinned names and the coverage verdict as one JSON line. In the
gateway every /proc syscall of the scan had to take the GIL back, so on
a busy host the scan slowed itself and every other gateway thread.

A child that cannot start, runs past 300 s, crashes, exits non-zero or
replies malformed counts as an incomplete scan with coverage unproven:
the pass reclaims no directory, as an incomplete in-process scan does.

The scan also stops reading another real user's tasks; they cannot bind
a source this uid staged, and they now lower only the host-wide flag.

Other platforms and any other proc root keep the in-process scan.

Closes #33171

Co-authored-by: Raymond Chen <chenmingwei23@users.noreply.github.com>
Original author: Raymond Chen
(cherry picked from commit 2da9ec8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sandbox mount-source pin scan holds the gateway GIL on busy hosts: run it in a child process

3 participants