Repository navigation
perf(sandbox): parse each distinct mount table once per pin scan - #12720
Conversation
|
Intent: Stop the stale-sandbox pin scan from splitting and matching every line of thousands of byte-identical mountinfo tables in Python on every cleanup tick, which held the GIL on a maintenance thread and stalled the event loop (32 of 275 GIL samples). Each distinct table is parsed once per scan; repeats cost the read alone. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify on the profiled host that loop_lag stalls actually clear — scan wall time barely drops (4.2s→3.7s); the fix rests on the GIL-held share. [DESIGN-REVIEWED] 0e10983 |
First Principles Review (Fable 5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The undeclared cache-bound machinery exists only because raw table bytes are retained; a digest set needs no bounds, no fallback branch, no extra test. Not justified as shipped
What this change shipsInventory (4 items) — 3 justifiedIntent: stop the cleanup tick's mount-table parsing from holding the GIL and stalling the gateway loop — a FIX (py-spy profile; new test fails on base).
[FIRST-PRINCIPLES-REVIEWED] 0e10983 |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING -- src/kiro_crew/sandbox.py:8360 -- Changing |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsI've confirmed the parsing: For CANDIDATE 1's harm to occur, a So outcome (c) is not reachable — a live sandbox source cannot be dropped from No Step 2 finding survives the same bar: the dedup/cache logic issues the read for every task (preserving per-caller OSError coverage accounting), bounds both entries and bytes before adding, and any over-split of a foreign line can only create a harmless conservative false-positive pin. No findings. [OPUS-REVIEWED] 0e10983 |
e9db037 to
b778cb5
Compare
|
|
Pushed aceba35: two CI test fixes plus a rebase.
|
b778cb5 to
aceba35
Compare
aceba35 to
275a822
Compare
275a822 to
7ccff21
Compare
|
Rebased: main moved in import-overlap files (adaptive/controller.py, config/sections.py, eventlog/service.py, subagent.py and tests) since this head (-> 1256a48); CI re-running. No code change. |
|
64e5690 to
7257b12
Compare
|
Re-pushed the identical tree as 7257b12: the fork review lanes for 64e5690 reported "could not run" (GitHub App rate limit) and Windows shard 3 failed on |
|
The stale-sandbox mount-source sweep proves absence of a pin by reading the mountinfo of every task that could hold a sandbox namespace: every thread-group leader plus, for this uid's, root's and the overflow uid's groups, every sibling thread. Every thread of a group shares its leader's mount table unless it unshared one, so on a busy host the scan is thousands of byte-identical reads: 12.5k tasks and 1.3M mountinfo lines for 18 distinct tables, measured on one host, and every one of those lines was split and matched in Python. py-spy --gil put 32 of 275 GIL-holding samples on that per-line work in the maintenance thread, which starves the event loop through GIL contention and shows up in the adaptive concurrency controller as loop_lag pressure. _collect reads each table as bytes, dismisses one that names no keyed source with a single substring check, and splits and matches a distinct table once, remembering its bytes for the scan. Repeats cost the read alone, which releases the GIL. Nothing about WHAT is reclaimed changes: the read is still issued for every task, so the OSError the coverage accounting keys on is raised as it was, and identical bytes contribute identical pins.
7257b12 to
0e10983
Compare
|
|
bolichen97
left a comment
There was a problem hiding this comment.
Approved after verified readiness: passed, PR Readiness success on this head, mergeable, no open change requests. [operator: bolichen97 batch review 2026-09-23]
…odotdev#12720) The stale-sandbox mount-source sweep proves absence of a pin by reading the mountinfo of every task that could hold a sandbox namespace: every thread-group leader plus, for this uid's, root's and the overflow uid's groups, every sibling thread. Every thread of a group shares its leader's mount table unless it unshared one, so on a busy host the scan is thousands of byte-identical reads: 12.5k tasks and 1.3M mountinfo lines for 18 distinct tables, measured on one host, and every one of those lines was split and matched in Python. py-spy --gil put 32 of 275 GIL-holding samples on that per-line work in the maintenance thread, which starves the event loop through GIL contention and shows up in the adaptive concurrency controller as loop_lag pressure. _collect reads each table as bytes, dismisses one that names no keyed source with a single substring check, and splits and matches a distinct table once, remembering its bytes for the scan. Repeats cost the read alone, which releases the GIL. Nothing about WHAT is reclaimed changes: the read is still issued for every task, so the OSError the coverage accounting keys on is raised as it was, and identical bytes contribute identical pins.
Problem / Motivation
The gateway's event loop stalls for 0.4-1.1 s at a time on an idle 128-core host, and the adaptive concurrency controller reads the stalls as
loop_lagpressure and pins the subagent cap at 1. py-spy--gil --threads(120 s, 275 GIL-holding samples) puts 37 of them on maintenance threadmc-maint_2in the stale-sandbox mount-source sweep, 32 of those inside_mount_pinned_source_names._collect, 27 on its per-line loop.Why it matters
A worker thread that holds the GIL for long stretches of pure-Python work starves the asyncio loop just as effectively as work on the loop itself: every request and the concurrency controller stall behind it. This scan runs on every cleanup tick (60-300 s) on any host that holds a directory reclaim candidate, which on the profiled host is permanent.
What changed (motivation → approach → change)
The pin scan must read the mountinfo of every task that could hold a sandbox mount namespace, sibling threads included, because a thread can
setnsinto a namespace its leader is not in. But every thread that has not done so reports its leader's table verbatim, so the scan is thousands of byte-identical reads: measured 12.5k tasks, 1.32 M mountinfo lines, 18 distinct tables on one host, with every line split and matched in Python on every tick._collectinsrc/kiro_crew/sandbox.pyreads each table as bytes, dismisses one that names no keyed source with a single substring check, and splits and matches a distinct table once per scan, remembering its bytes. The remembered set is bounded: at most_MOUNT_TABLE_CACHE_MAX_ENTRIES(256) distinct tables or_MOUNT_TABLE_CACHE_MAX_BYTES(64 MiB) of table bytes per scan; a table arriving past either bound is parsed and matched without being remembered (cache_full), so the bound degrades deduplication, never coverage. Repeats cost the read alone, which releases the GIL. The read is still issued for every task, so theOSErrorthe coverage accounting keys on is raised exactly as it was; identical bytes contribute identical pins, so WHAT the sweep reclaims is unchanged. The sweep's coverage rules, passes, and the directory gate are untouched.Backwards compatibility
Compatible: the function's signature, return value, coverage accounting and every error path are unchanged; only the amount of Python-level work per duplicate table drops.
Tests
TestMountPinnedSourceNames::test_identical_mount_tables_are_parsed_once_per_scan: a leader with 20 sibling threads and a second process all reporting the same two-line table; the matcher (which sees every line of a parsed table) is called exactly twice, the pins and coverage are unchanged. Fails on main (22 groups' worth of lines are matched), passes with the fix.test/test_sandbox_mount_source_sweep.pyandtest/test_home_pin_survives_monkeypatch_undo.pypass unchanged, including the real-/procscan test.test/test_jsonl_util.py: the("sandbox.py", "/proc/<pid>/mountinfo")entry leaves the handle-iteration excused list because the reader is no longer afor line in fhloop; the audit's staleness check requires the removal.Manual verification
Read-only script against the live
/procof the profiled host: the current parse costs ~4.2 s per scan (of which ~3.5 s is the kernel generating mountinfo with the GIL released and ~0.7 s is the Python per-line loop); the dedup path costs ~3.7 s with the Python share reduced to 18 table parses. The kernel read cost is out of scope here.Related Issues
Related: #12710 / #12711 (the slots broadcast, the larger offender in the same profile).
Pattern harvest
Rule candidate: review-prompt
Pattern: per-item pure-Python parse inside a
/proc-wide walk on a worker thread, where most items are byte-identical (dedupe by content or namespace identity before parsing).Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Fixes #12719