Repository navigation
fix(security): resolve sensitive-path anchors in a stdlib child so GIL contention cannot stall the gate - #12739
Pearcekieser wants to merge 1 commit into
Conversation
|
👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review. When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically. |
…L contention cannot stall the gate The realpath work behind is_sensitive_path -- the agent-supplied candidate, the root anchors re-resolved on every gate call and the ~70-call anchor rebuild -- ran on an mc-pathres thread of the gateway interpreter. realpath releases and re-acquires the GIL once per path component, so beside a few CPU-bound sibling threads a rebuild that costs 2 ms idle measured 1.2 s with one hog and 20-29 s with eight, past its 8 s budget on a healthy disk, and every gate refused ordinary project paths. The pool worker now sends the paths as one JSON line to a long-lived `python -I -S -c <source>` child with its own GIL and reads one line back. _run_resolution_bounded keeps every piece of its bookkeeping; what changes is what the worker does with its budget. The root anchors travel as one list request and the rebuild as one list request. A child wedged on a dead mount is killed at the deadline so the pinned worker is freed; the stall classifier samples the child's /proc/<pid>/syscall while a request is in flight; a transport fault is refused fail-closed without charging a cooldown; a host that cannot start the child falls back in-process for a cool-off. Measured on a 128-core host, 70-anchor rebuild under 0/1/2/4/8 GIL hogs: in-process 2 ms / 1.2 s / 7.6 s / 15 s / 20 s median; through the child 2 / 11 / 27 / 11 / 8 ms median (342 ms worst).
f4271d8 to
566a2dc
Compare
|
👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review. When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically. |
1 similar comment
|
👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review. When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically. |
|
@bolichen97 @chenmingwei23 -- since you have both been working this stack today: this is the smaller alternative to #10256 (same fix, ~1/5 the diff; cross-referenced there). It has been waiting for fork-workflow approval since 07:07 UTC and cannot get a CI or review round without it. Could one of you approve the pending workflows? #10256 and #11223 are both at |
|
👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review. When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically. |
2 similar comments
|
👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review. When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically. |
|
👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review. When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically. |
|
Closing in favour of #12879, the wiring PR for the subprocess pool that landed in #12709. #12709 built the child interpreter and The tests this PR contributed that still apply (GIL-hog reproduction, child/in-process parity, one request per rebuild, fail-closed on transport fault, empty-spelling guard) were carried over. Thanks to @bolichen97 and @chenmingwei23 for the rebases and review rounds here; they shaped the wiring PR directly. |
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread (#12879) Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs #10255, #12709, #10256, #12739
Problem / Motivation
The sensitive-path gate (
is_sensitive_path/sensitive_path_refusal, run on the event loop fromhooks.on_tool_call) refuses ordinary project files as unverifiable on healthy local disks whenever the gateway interpreter is busy (#10255). Everyrealpathbehind the gate -- the agent-supplied candidate, the root anchors re-resolved on every call, and the ~70-call anchor rebuild every 0.1 s -- runs on anmc-pathresthread inside the gateway process, under the gateway GIL.os.path.realpathreleases and re-acquires the GIL once per path component, so beside a few CPU-bound sibling threads each re-acquisition waits a switch interval and the rebuild blows through its 8 s budget. Measured on this 128-core host with a local disk, 70-anchor rebuild, pure-Python hogs, 5 trials (median / worst):Candidate resolution, worst of 5: in-process 31 / 112 / 779 / 1,120 ms at 1/2/4/8 hogs; through the child 10 / 36 / 66 / 31 ms. A 3-trial re-run on the rebased head reproduced the shape: in-process rebuild 1.5 s / 9.3 s / 17.7 s / 4.0 s median (worst 21.4 s at four hogs), through the child 11 / 27 / 16 / 36 ms median.
Why it matters
Each expiry is a fail-closed refusal of the tool call; the fourth expiry in 10 s under one prefix opens a 30 s-doubling cooldown, and since #12647 the rebuild's prefix is the user's home directory -- every worktree and checkout. A maintainer counted nine refusals in ~40 minutes on an idle 64-core host, including a refused worker-session start. #12460 (wider pool) and #12664 (adaptive anchor TTL, merged) reduce how often the cost is paid; #12664's description names "making a single cold rebuild cheap (that needs the resolution to leave this interpreter)" as separate work. This is that work.
What changed (motivation → approach → change)
src/kiro_crew/security/pathres_helper.py(stdlib only, 72 lines) is a JSON-lines loop over stdin: a path answers[realpath, resolve], a list of paths answers onerealpatheach, a malformed line exits.src/kiro_crew/security/pathres_client.pyowns one long-livedsys.executable -I -S -c <source captured at import>child with aPATH-only environment, one request lock, kill-at-deadline (abort_if_inflight), an in-process fallback that latches for 60 s only when the child cannot start or dies before its first answer, andblocked_in_filesystem, which samples the child's/proc/<pid>/syscallattributed to the requesting worker.src/kiro_crew/security/paths.pychanges only what the pool worker does with its budget._resolved_spellings(candidate) calls the child;_realpaths_or_noneis the batched anchor seam and_realpath_or_noneits single-path form;_resolve_root_anchorssends$HOMEand every override root as one deduplicated list;_home_dir_targets_uncachedruns its anchoring logic once with no answers to collect the paths it needs, resolves them in one request, then builds the set._run_resolution_boundedkeeps its budget, grace, load arm, cooldown, wedged tracking and leave-one-free guard intact; it gains a re-raise for aPathResolutionStalledthe worker itself raised (a transport fault must refuse, not fall back to the lexical forms), and the charged-stall arm kills the wedged child so the pinned worker unwinds._worker_blocked_in_filesystemprefers the child's sample while a request is in flight and otherwise samples the thread as it did.test_spawn_audit.pyallowlists the spawn: argv fixed, env minimal, agent-supplied paths travel as stdin data, and the resolver cannot run inside the agent's sandbox view because it exists to see through it.docs/system-specs/modules/security.mdgains one paragraph with the measurements.Based on
origin/mainafter #12664 (theresolution_differedmarker is computed over the batched answers); this supersedes #10256 (2,033 lines across 21 files), whose wording and shell-command halves landed as #11221 and are pending as #11223; bolichen97's four rebases there kept it current, and the helper program, the client's spawn/kill/latch core, the root-anchor dedupe and thirteen tests are salvaged from it.Pre-declared residuals: (1) one pipe round-trip per resolution still runs on a pool thread under the GIL, so a saturated interpreter can delay the write/readline by a few switch intervals -- tens of milliseconds, measured above, not seconds; (2) a child killed at the deadline faults at most the one other request queued behind the lock, which refuses fail-closed with no cooldown; (3) Windows has no
/proceither way, so the grace wait remains the discriminator there, and the list form exists because a child spawn and each round-trip cost a scheduler tick on Windows CI; (4) independent of and additive to #12664 (frequency vs cost); (5)KIROCREW_PATH_RESOLVE_WORKERSstill sizes the thread pool that fronts the child; (6) out of scope: caching the candidate side, the cooldown key, the load-arm probe allowance, #11223.Backwards compatibility
Compatible: every public gate keeps its signature and verdicts; the only behavioural change is that a resolver transport fault is refused as unverifiable (no cooldown) instead of matching on lexical forms, which is the fail-closed direction the spec already mandates for a stall.
_realpath_or_noneremains and is unchanged in contract; tests that stubbed it for the anchors stub_realpaths_or_noneinstead.Tests
test/test_pathres_client.py(19 tests): the reproduction (test_resolution_completes_inside_the_budget_while_a_sibling_thread_hogs_the_gil-- proven to fail with in-process resolution substituted: refused as unverifiable under one hog), helper parity with in-process answers, ordered single/list replies, exit on each malformed-request shape, surrogate-escaped bytes across the pipe, stdlib-only import pin, boot-captured-I -S -csource and minimal env, root key and rebuild as one deduplicated request each, killing a wedged child frees the pool worker and respawns,abort_if_inflightandblocked_in_filesystemanswer only the owning worker, cannot-start latches once and re-probes after the cool-off, dies-before-first-answer latches, transport faults refuse fail-closed on both halves without a cooldown, and a fault inside the grace wait refuses instead of going lexical.test_security_path_resolve_bounded.pyandtest_pathres_loop_starvation.py: the anchor stubs move to the batch seam; every existing stall, cooldown, grace and one-pool-job assertion is unchanged.Manual verification
Measurement table above from
~/.kiro/crew/workspace/investigations/pr_c_measure.py(stdlib hogs, gateway venv, Python 3.12.13). Gates run locally: scoped backend tests, black, isort, flake8, mypy, comment-history, docs-lint, spawn audit and the remaining scripted gates.Related Issues
Fixes #10255. Supersedes #10256. Rebases onto #12664 when it lands.
Pattern harvest
Rule candidate: review-prompt
Pattern: pure-Python syscall loops (realpath, os.walk, stat batches) placed on a worker thread for liveness still pay one GIL re-acquisition per syscall; under interpreter contention the budget measures the GIL, not the disk.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)