Repository navigation
fix(sandbox): run the mount-source pin scan in a spawned child - #33195
Conversation
Sensitive Change ReviewSensitive files changed:
|
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the scan's answer is small, the child returns exactly the fields the gate reads ( Notes
[DESIGN-REVIEWED] 7fb162d Description read[DESCRIPTION-READ] 51b2cdcd4cc2bf953aa248479e3e9a2203691130dc81ed88a1aa0e85deb042f4 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the scan leaves the gateway interpreter, which is the cause the issue names, and a failed child is treated as an incomplete scan. Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: on busy Linux hosts, stop the mount-source sweep's pin scan from slowing the gateway and from ending "incomplete" every pass (#33171). FIX.
[FIRST-PRINCIPLES-REVIEWED] 7fb162d Description read[DESCRIPTION-READ] ce18df00571ab342f8b5ed107f93823d4c54d15243ce0c308fead327433b8e87 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
60d596d to
d5788a4
Compare
|
Sensitive change review
|
d5788a4 to
39d911f
Compare
|
Sensitive change review
|
39d911f to
baa9e9c
Compare
|
Sensitive change review
|
baa9e9c to
f80cc51
Compare
Sensitive change review
|
On Linux the sweep's pin scan of /proc now runs in a fresh interpreter (python -P -m kiro_crew.sandbox_pin_scan_child keyed|legacy) that writes the pinned names and the coverage verdict as one JSON line. In the gateway every /proc syscall of the scan had to take the GIL back, so on a busy host the scan slowed itself and every other gateway thread. A child that cannot start, runs past 300 s, crashes, exits non-zero or replies malformed counts as an incomplete scan with coverage unproven: the pass reclaims no directory, as an incomplete in-process scan does. The scan also stops reading another real user's tasks; they cannot bind a source this uid staged, and they now lower only the host-wide flag. Other platforms and any other proc root keep the in-process scan. Closes #33171
f80cc51 to
7fb162d
Compare
|
Sensitive change review
|
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approving. On this head the review read the whole diff and checked each claim against the code. The child runs the same in-process scan (_mount_pinned_source_names_in_process) and shares the gateway's uid and mount namespace. The spawn runs only on the maintenance executor, never on the event loop, and no file on the boot path changed. The security.md paragraph matches the code. The new test file passes, and a mutation run showed that the child-fails path and the routing to the child are both pinned by tests.
Security: the PR adds a subprocess inside sandbox code that decides when bind-mount sources are reclaimed. Every child failure fails closed: it cannot start, it times out, it is killed, it exits non-zero, or its output is not JSON or has the wrong shape. Each of these returns None, which _scan_via_child turns into an incomplete scan with coverage.covered = False. The directory gate then keeps every candidate, and the legacy pass reclaims nothing and leaves its marker unset. A reply can only lower coverage, never raise it. The argv is fixed, runs on this interpreter with -P, takes no agent input, and is registered in the spawn audit. No seal, mask or refusal is added.
Done well: there is one test per child failure mode, and each failure is driven through the real sweep, which keeps a dead-pid directory and leaves the legacy scan unproven.
Optional:
- When the child replies successfully, nothing pins how the parent reads the reply. Three fail-open mutations at
sandbox_mount_sweep.py:389-392all stayed green: dropping thecoveredpropagation, returning an empty pinned set, and forcingcomplete=True. A follow-up test could stubscan_in_childwith fixed replies such as({"kirocrew_sb_X_a"}, True, True),(set(), False, False)and(set(), False, True), and assert what the sweep keeps and what it reclaims. - The benchmark proves that the scan now completes (199 s and incomplete before, 2.4 s and complete after). It does not show the gateway-lag relief described in "Why it matters". The body itself calls the lag within noise.
- The
argv=andtimeout=parameters ofscan_in_child, and the nice +10, are used only by tests or go beyond the goal. Both follow theconsolidation_write_child.pyprecedent. Neither needs a change, but a short disposition on each lane note would close it.
|
The code looks good to me on 7fb162d. One step before I can approve: could you update this branch onto main, with a rebase rather than a merge commit (PR Hygiene allows only one or two commits)? macOS Tests / Backend Tests (macOS) (8) fails on |
On Linux the sweep's pin scan of /proc now runs in a fresh interpreter (python -P -m kiro_crew.sandbox_pin_scan_child keyed|legacy) that writes the pinned names and the coverage verdict as one JSON line. In the gateway every /proc syscall of the scan had to take the GIL back, so on a busy host the scan slowed itself and every other gateway thread. A child that cannot start, runs past 300 s, crashes, exits non-zero or replies malformed counts as an incomplete scan with coverage unproven: the pass reclaims no directory, as an incomplete in-process scan does. The scan also stops reading another real user's tasks; they cannot bind a source this uid staged, and they now lower only the host-wide flag. Other platforms and any other proc root keep the in-process scan. Closes #33171 Co-authored-by: Raymond Chen <chenmingwei23@users.noreply.github.com> Original author: Raymond Chen (cherry picked from commit 2da9ec8)
Problem / Motivation
Goal: On a busy Linux host, the sandbox mount-source sweep's pin scan no longer competes with the gateway's own threads for the Python interpreter lock, and it still never reclaims a directory it could not prove unused.
The pin scan reads
/procfor every task on the host (a stat, two link probes and amountinforead each). It ran inside the gateway, so every one of those syscalls had to take the GIL back. On hosts with thousands of processes the crew-log writers, the context-build pool and the event loop queued behind it, and the scan itself was slowed so much by the other threads that it often ended "incomplete", leaving the stand-in directories unreclaimed (#33171).Why it matters
Operators on busy shared hosts saw event-loop lag, slow dashboard pages and messages that never confirmed delivery while a sweep pass ran, plus a sweep that logged
pin scan incomplete and descendant coverage unprovenevery pass for days.Not a goal
/procreader out of the gateway.What changed (motivation -> approach -> change)
Symptom: gateway threads parked in
take_gilduring sweep passes, and a pin scan that took minutes and came back incomplete. Cause: the scan's tens of thousands of/procsyscalls ran in the gateway's interpreter. Its answer is small (the pinned names plus two flags), so it moves to its own process.src/kiro_crew/sandbox_pin_scan_child.py: on Linux, a scan of the host's/procruns aspython -P -m kiro_crew.sandbox_pin_scan_child keyed|legacy, a fresh interpreter started throughsubprocess(no fork of the threaded gateway) at a lower CPU priority. It writes one JSON line with the pinned names,completeandcovered, then exits. This follows the consolidation write child's launch pattern.src/kiro_crew/sandbox_mount_sweep.py,_mount_pinned_source_namesand_bound_source_basenamessend the host/procscan on Linux to the child. Any other proc root, a custom matcher and every other platform keep the in-process scan, so the unit tests over synthetic proc trees and macOS/Windows behave as before.docs/system-specs/modules/security.md: thesandbox_mount_sweepparagraph now states the child process and its fail-closed rule.Measured on a 64-core host with about 1,400 processes and 7,200 threads (about 3,100 pinned names). A small stand-in for the gateway ran the scan on a worker thread, as the maintenance executor does, beside an asyncio heartbeat (every 50 ms) and 16 probe threads that sleep 2 ms and record how late they wake. "Before" is the parent commit, "after" this branch:
On this host the lag of the other threads stays within noise in both versions. The clear effect is on the scan: in the gateway it slows by 5x next to modest Python work, and next to one busy thread it takes over three minutes and ends incomplete, which is the "pin scan incomplete" warning the issue reports. In the child it takes the same time whatever the gateway is doing. Native stack samples (eu-stack) of the stand-in showed no thread in
take_gilin either version with nothing else running.Backwards compatibility
Compatible: the sweep's inputs, decisions and log lines are unchanged; only where the scan runs moves. A failed child answers like an incomplete scan, which callers already handle by keeping every directory.
Removes nothing: no path, record, key or function is taken away; the scan reads the same tasks as before.
Tests
New
test/test_sandbox_pin_scan_child.py:test_real_child_scans_the_host_proc: a real spawned child reads the real/proc, and on Linux both_mount_pinned_source_namesand_bound_source_basenamesanswer through it (keyed, then legacy).test_failed_child_answers_none[...], one case per failure: cannot start, timeout (the child would reply well-formed after its sleep, so only the timeout yields None), crash (SIGKILL), non-zero exit with a valid reply, not JSON, wrong shape, empty output. Each checks the warning names the reason.test_failed_child_keeps_a_dead_pid_directoryandtest_failed_child_leaves_the_legacy_scan_unproven: a failed child through the real sweep keeps a dead-pid directory and leaves the legacy scan unproven.test_child_scan_runs_the_matching_in_process_scan[keyed|legacy]andtest_child_entry_writes_one_reply_line: the child side, run in-process, scans /proc with the right name shape and writes exactly one reply line that the parent parses back.test_well_formed_reply_is_returned,test_child_entry_refuses_an_unknown_kind.The spawn audit and the internal-Python isolation list register the new spawn site. The existing sweep suites (
test_sandbox_mount_source_sweep.py,test_sandbox_mount_sweep_views.py, the facade and monkeypatch-undo tests) pass unchanged.Determinism: the new file passed 20 runs at
-n0, 10 runs at-n 4 --dist loadgroupbeside two sibling files, and a shuffled order (seed 20811).Proof: not provable -- other: the fix adds a new child module, so reverting the production hunks fails test collection
Manual verification
Ran the bench above against the parent commit and this branch on the same host, alternating the two so load drift hit both alike, and compared the child's pinned set and flags with the in-process scan on the same host.
Related Issues
Closes #33171
Related: #12719, #32177, #31730
Sibling sweep
Swept: git grep -nE 'listdir("/proc")|listdir(proc_root)|Path("/proc").iterdir' -- src/
Sibling: src/kiro_crew/dashboard/handlers_system.py:511 -- not affected -- one cmdline read per process, per page request, not a per-thread mountinfo walk on every sweep tick
Sibling: src/kiro_crew/platform_compat_proctable.py:1219 -- not affected -- a per-call descriptor lookup for one file, not part of the periodic sweep
Sibling: src/kiro_crew/platform_compat_liveness.py:66 -- not affected -- a single directory listing with no per-task reads
Pattern harvest
Not generalizable: the cost came from this one scan's size (every thread's mountinfo on the host, on a timer); the other /proc readers read one small file per process on demand.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)