Skip to content

perf(sandbox): parse each distinct mount table once per pin scan - #12720

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:perf/sandbox-sweep-gil
Sep 23, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:perf/sandbox-sweep-gil

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

The gateway's event loop stalls for 0.4-1.1 s at a time on an idle 128-core host, and the adaptive concurrency controller reads the stalls as loop_lag pressure and pins the subagent cap at 1. py-spy --gil --threads (120 s, 275 GIL-holding samples) puts 37 of them on maintenance thread mc-maint_2 in the stale-sandbox mount-source sweep, 32 of those inside _mount_pinned_source_names._collect, 27 on its per-line loop.

Why it matters

A worker thread that holds the GIL for long stretches of pure-Python work starves the asyncio loop just as effectively as work on the loop itself: every request and the concurrency controller stall behind it. This scan runs on every cleanup tick (60-300 s) on any host that holds a directory reclaim candidate, which on the profiled host is permanent.

What changed (motivation → approach → change)

The pin scan must read the mountinfo of every task that could hold a sandbox mount namespace, sibling threads included, because a thread can setns into a namespace its leader is not in. But every thread that has not done so reports its leader's table verbatim, so the scan is thousands of byte-identical reads: measured 12.5k tasks, 1.32 M mountinfo lines, 18 distinct tables on one host, with every line split and matched in Python on every tick.

_collect in src/kiro_crew/sandbox.py reads each table as bytes, dismisses one that names no keyed source with a single substring check, and splits and matches a distinct table once per scan, remembering its bytes. The remembered set is bounded: at most _MOUNT_TABLE_CACHE_MAX_ENTRIES (256) distinct tables or _MOUNT_TABLE_CACHE_MAX_BYTES (64 MiB) of table bytes per scan; a table arriving past either bound is parsed and matched without being remembered (cache_full), so the bound degrades deduplication, never coverage. Repeats cost the read alone, which releases the GIL. The read is still issued for every task, so the OSError the coverage accounting keys on is raised exactly as it was; identical bytes contribute identical pins, so WHAT the sweep reclaims is unchanged. The sweep's coverage rules, passes, and the directory gate are untouched.

Backwards compatibility

Compatible: the function's signature, return value, coverage accounting and every error path are unchanged; only the amount of Python-level work per duplicate table drops.

Tests

  • TestMountPinnedSourceNames::test_identical_mount_tables_are_parsed_once_per_scan: a leader with 20 sibling threads and a second process all reporting the same two-line table; the matcher (which sees every line of a parsed table) is called exactly twice, the pins and coverage are unchanged. Fails on main (22 groups' worth of lines are matched), passes with the fix.
  • The existing 89 tests in test/test_sandbox_mount_source_sweep.py and test/test_home_pin_survives_monkeypatch_undo.py pass unchanged, including the real-/proc scan test.

test/test_jsonl_util.py: the ("sandbox.py", "/proc/<pid>/mountinfo") entry leaves the handle-iteration excused list because the reader is no longer a for line in fh loop; the audit's staleness check requires the removal.

Manual verification

Read-only script against the live /proc of the profiled host: the current parse costs ~4.2 s per scan (of which ~3.5 s is the kernel generating mountinfo with the GIL released and ~0.7 s is the Python per-line loop); the dedup path costs ~3.7 s with the Python share reduced to 18 table parses. The kernel read cost is out of scope here.

Related Issues

Related: #12710 / #12711 (the slots broadcast, the larger offender in the same profile).

Pattern harvest

Rule candidate: review-prompt
Pattern: per-item pure-Python parse inside a /proc-wide walk on a worker thread, where most items are byte-identical (dedupe by content or namespace identity before parsing).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Fixes #12719

@Pearcekieser
Pearcekieser requested review from a team and cixuuz September 22, 2026 05:37
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Intent: Stop the stale-sandbox pin scan from splitting and matching every line of thousands of byte-identical mountinfo tables in Python on every cleanup tick, which held the GIL on a maintenance thread and stalled the event loop (32 of 275 GIL samples). Each distinct table is parsed once per scan; repeats cost the read alone.
Not a goal: Changing what the sweep reclaims, its coverage rules, its pass structure or cadence; reducing the kernel-side cost of reading mountinfo for every task; touching the slots broadcast (#12711).

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 0e10983a649036203e7e9bb61de0074278da6b5a via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify on the profiled host that loop_lag stalls actually clear — scan wall time barely drops (4.2s→3.7s); the fix rests on the GIL-held share.

[DESIGN-REVIEWED] 0e10983

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 0e10983a649036203e7e9bb61de0074278da6b5a via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The undeclared cache-bound machinery exists only because raw table bytes are retained; a digest set needs no bounds, no fallback branch, no extra test.

Not justified as shipped

  • Item 3 — undeclared: the description says only "remembering its bytes"; the diff also ships _MOUNT_TABLE_CACHE_MAX_ENTRIES, _MOUNT_TABLE_CACHE_MAX_BYTES, a cache_full degradation branch and a parametrized test — surface guarding only the memory the bytes-set itself creates (18 distinct tables measured; each constant has exactly one consumer, _collect).
    Subtraction: replace parsed_tables: set[bytes] with a set of hashlib.sha256(data).digest() (~32 B per distinct table, worst case ~400 KB at 12.5k tasks); delete both constants, cache_full, parsed_table_bytes, and test_tables_past_cache_limit_remain_parsed.
    Clears when: that deletion is made, or a host is named where retained distinct-table bytes measurably need a cap that digests cannot satisfy.

What this change ships

Inventory (4 items) — 3 justified

Intent: stop the cleanup tick's mount-table parsing from holding the GIL and stalling the gateway loop — a FIX (py-spy profile; new test fails on base).

  1. Each distinct mount table is split and matched once per pin scan; duplicate sibling-thread reads cost the read alone — justified
  2. A table naming no keyed source is dismissed on one whole-table check before any line work — justified
  3. In-scan cache bounds: two size constants plus a parse-without-retaining fallback once full — undeclared; smaller alternative (digest set) never considered
  4. mountinfo is now read whole-file instead of line-streamed, so sandbox.py leaves the handle-iteration excused list — justified (the audit's staleness test mandates the removal)

[FIRST-PRINCIPLES-REVIEWED] 0e10983

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 0e10983a649036203e7e9bb61de0074278da6b5a via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING -- src/kiro_crew/sandbox.py:8360 -- Changing len(parsed_tables) >= _MOUNT_TABLE_CACHE_MAX_ENTRIES to > passes the new tests, silently allowing 257 cached tables -> Fix: repeat the first table in the zero-limit test and assert it is reparsed. (origin: validation)
[GPT-REVIEWED] 0e10983

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 0e10983a649036203e7e9bb61de0074278da6b5a via the fork AI-review pipeline; updated in place on each push.

Review details

I've confirmed the parsing: source = fields[3] is the mountinfo "root" field, positioned after three kernel-generated numeric fields (mount ID, parent ID, major:minor), and the sandbox source names are kirocrew_sb_* — a fixed, control-char-free naming scheme.

For CANDIDATE 1's harm to occur, a kirocrew_sb_* line would have to be split by .splitlines() such that its source no longer sits at fields[3]. But .splitlines() only adds split points; it never merges lines. The extra boundaries it recognizes over universal-newline iteration (\v, \f, \x1c–\x1e; the \x85/\u2028/\u2029 cases require valid UTF-8 that errors="replace" would mangle to U+FFFD, not a boundary) would have to appear at or before field index 3 of the kirocrew line to displace the source. Those first four fields are three numeric kernel fields plus the controlled kirocrew_sb_* name — none can carry such a byte. A control char in the mount-point field (index 4+) splits the line only after the source, so the first fragment still holds the source at fields[3] and it still pins. Splitting an unrelated non-kirocrew line only misparses that line, which never matched anyway.

So outcome (c) is not reachable — a live sandbox source cannot be dropped from pinned by this change. The candidate itself could exhibit no such mount and rated itself low. It dies under falsification.

No Step 2 finding survives the same bar: the dedup/cache logic issues the read for every task (preserving per-caller OSError coverage accounting), bounds both entries and bytes before adding, and any over-split of a foreign line can only create a harmless conservative false-positive pin.

No findings.

[OPUS-REVIEWED] 0e10983

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • mount-table cache retains unbounded external data — fixed in b778cb5

The per-scan dedup set is now bounded: at most _MOUNT_TABLE_CACHE_MAX_ENTRIES (256) distinct tables and _MOUNT_TABLE_CACHE_MAX_BYTES (64 MiB) retained.
Reaching either cap stops new insertions for that scan; cached tables keep deduplicating and every uncached table is parsed directly.
Every table is still read and parsed past the cap, so pin discovery, OSError coverage accounting and reclaim semantics are unchanged.
Test test_tables_past_cache_entry_limit_remain_parsed pins that a table past the cap is still matched and the set does not grow.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed aceba35: two CI test fixes plus a rebase.

  • test_jsonl_util.TestNoUnboundedHandleIteration: sandbox.py no longer iterates the mountinfo handle line-by-line (it reads the table as bytes), so it is removed from the _KERNEL_PSEUDO_FILE_READERS excused list -- the audit`s guard-the-guard test required that.
  • test_tables_past_cache_entry_limit_remain_parsed: the matcher-call assertion depended on /proc task iteration order (Windows shard 5 saw pid 500 before pid 1); it now asserts both distinct tables were parsed and that a past-cap table is parsed rather than dropped, independent of order.
  • Rebased: main moved in test/test_pending_boundary_consumer_semantics.py (fix(test): credit busy-helper callers in the slot-reader tripwire #12718 fixes the shard-4 slot-reader tripwire) (0a80050 -> 64fbda5); CI re-running.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/executors.py, test/test_security.py, test/test_security_path_resolve_bounded.py, scripts/measure_path_gate_ttl.py (import/same-dir overlap only) since this head went green (64fbda5 -> 37315e6); CI re-running. No code change.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention labels Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in import-overlap files (adaptive/controller.py, config/sections.py, eventlog/service.py, subagent.py and tests) since this head (-> 1256a48); CI re-running. No code change.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • past-cap test leaves the byte cap unpinned — fixed in 64e5690

test_tables_past_cache_limit_remain_parsed is parameterized over _MOUNT_TABLE_CACHE_MAX_ENTRIES and _MOUNT_TABLE_CACHE_MAX_BYTES, each forced to zero, and asserts exactly one matcher call per table read (three), independent of /proc iteration order.
Removing or loosening either cap now fails a test; the existing dedup test still requires a single parse when both caps are in force.
Also rebased onto main 80bd0a8 (import-only overlap); no production change in this push.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Re-pushed the identical tree as 7257b12: the fork review lanes for 64e5690 reported "could not run" (GitHub App rate limit) and Windows shard 3 failed on test_session_capabilities::test_actual_runtime_cwd_must_match_saved_project (source_unreadable), which this PR does not touch and which is green on main at the same base 80bd0a8. CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Whole-file mount-table read enables memory exhaustion — rebutted

The read is of /proc/<pid>/mountinfo, a kernel-synthesized pseudo-file whose size is bounded by fs.mount-max (default 100k mounts, low MB); an agent cannot write to it, and the repo already records this position for exactly this class of read at test/test_jsonl_util.py:682-683, where ("sandbox.py", "/proc/<pid>/mountinfo") sat on the excused list until this PR changed the reader shape from for line in fh to a whole read that the scanner does not detect.
Peak allocation is one bounded mountinfo plus the dedup cache, itself capped at 64 MiB / 256 entries and enforced in this diff. The adjudicator FLAGged this finding as disproportionate on the same grounds (fenced=1 flagged=1).

The stale-sandbox mount-source sweep proves absence of a pin by reading
the mountinfo of every task that could hold a sandbox namespace: every
thread-group leader plus, for this uid's, root's and the overflow uid's
groups, every sibling thread. Every thread of a group shares its leader's
mount table unless it unshared one, so on a busy host the scan is
thousands of byte-identical reads: 12.5k tasks and 1.3M mountinfo lines
for 18 distinct tables, measured on one host, and every one of those
lines was split and matched in Python. py-spy --gil put 32 of 275
GIL-holding samples on that per-line work in the maintenance thread,
which starves the event loop through GIL contention and shows up in the
adaptive concurrency controller as loop_lag pressure.

_collect reads each table as bytes, dismisses one that names no keyed
source with a single substring check, and splits and matches a distinct
table once, remembering its bytes for the scan. Repeats cost the read
alone, which releases the GIL. Nothing about WHAT is reclaimed changes:
the read is still issued for every task, so the OSError the coverage
accounting keys on is raised as it was, and identical bytes contribute
identical pins.
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Re-pushed the identical tree as 0e10983: the only CI red on 7257b12 was the Windows installer smoke test ("Silent uninstall left a KiroCrew registration behind"), an installer/registry teardown check this sandbox.py change cannot influence and green on main at the same base 80bd0a8. CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision labels Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Item 3 — undeclared cache bounds — rebutted

Declared: the PR body now names _MOUNT_TABLE_CACHE_MAX_ENTRIES (256), _MOUNT_TABLE_CACHE_MAX_BYTES (64 MiB) and the cache_full parse-past-cap branch, and what they bound (deduplication, never coverage).
The bounds are not a rider: the GPT lane required a bound on the remembered set in this PR's round 2 (head 5cf9267c-era finding "unbounded dedup set"), and a digest set would put a SHA-256 of every table for every task on the exact hot path this PR is trimming to a substring check, trading a memory bound the profile never needed for CPU under the GIL it did.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • >= vs > at the entry cap is not pinned by the zero-limit test — rebutted

The difference is one extra remembered table (257 rather than 256) on a soft memory bound whose byte cap still holds; it changes neither coverage nor what the sweep reclaims. The zero-limit cases already pin the behaviour that matters: past the cap every table is parsed and matched. Adding a mutation-killing repeat for an off-by-one on a soft bound is more test than the property warrants.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after verified readiness: passed, PR Readiness success on this head, mergeable, no open change requests. [operator: bolichen97 batch review 2026-09-23]

@bolichen97
bolichen97 merged commit a7da6e6 into kirodotdev:main Sep 23, 2026
116 of 122 checks passed
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…odotdev#12720)

The stale-sandbox mount-source sweep proves absence of a pin by reading
the mountinfo of every task that could hold a sandbox namespace: every
thread-group leader plus, for this uid's, root's and the overflow uid's
groups, every sibling thread. Every thread of a group shares its leader's
mount table unless it unshared one, so on a busy host the scan is
thousands of byte-identical reads: 12.5k tasks and 1.3M mountinfo lines
for 18 distinct tables, measured on one host, and every one of those
lines was split and matched in Python. py-spy --gil put 32 of 275
GIL-holding samples on that per-line work in the maintenance thread,
which starves the event loop through GIL contention and shows up in the
adaptive concurrency controller as loop_lag pressure.

_collect reads each table as bytes, dismisses one that names no keyed
source with a single substring check, and splits and matches a distinct
table once, remembering its bytes for the scan. Repeats cost the read
alone, which releases the GIL. Nothing about WHAT is reclaimed changes:
the read is still issued for every task, so the OSError the coverage
accounting keys on is raised as it was, and identical bytes contribute
identical pins.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stale-sandbox pin scan re-parses thousands of identical mountinfo tables per tick, holding the GIL and stalling the event loop

2 participants