Repository navigation
fix: resolve sensitive-path symlinks out of process so GIL contention cannot expire the budget - #10256
fix: resolve sensitive-path symlinks out of process so GIL contention cannot expire the budget#10256Pearcekieser wants to merge 1 commit into
Conversation
|
Intent: Stop the sensitive-path gate from refusing ordinary files when the gateway is busy, by taking symlink resolution off the interpreter whose GIL the resolver was waiting on, and make a resolution that could not complete say so instead of claiming a match. |
e5223a2 to
a0c4dec
Compare
Design Review (Fable 5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Duplicates the just-landed Watch
[DESIGN-REVIEWED] 0761537 |
First Principles Review (Fable 5, fork) — 🔴 BLOCKPremise-level review of First-Principles-Verdict: BLOCK The base already ships this mechanism: Not justified as shipped
What this change shipsInventory (10 items) — 6 justifiedIntent: stop refusing ordinary project files as "sensitive" when the gateway's own CPU load, not the disk, expires the resolve budget — a FIX (repro test fails on base).
BlockersDuplicate out-of-process resolver beside the one the base already carries. [FIRST-PRINCIPLES-REVIEWED] 0761537 |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsBased on my analysis: the candidate list contained no candidates (the discovery pass found nothing), so Step 1 has nothing to falsify. I worked through the production code in Step 2 to see whether any grounded defect was missed — the helper-process delegation in No findings. [OPUS-REVIEWED] 0761537 |
Security Scope Review (fork) — 🟡 CONCERNSWhich legitimate operations does the reviewer's own judgement; no confirmed regression Scope-Verdict: CONCERNS TIGHTENING: the sensitive-path resolver's failure handling went fail-closed on both halves of the check -- a candidate or anchor whose resolution does not complete is now refused instead of degraded to its lexical spelling. The one threat this refuses is a workspace symlink into a credential store (and, on the spawn path, a symlinked credential home left out of the OS sandbox deny mask) passing while resolution is unavailable, quoting the ADJUDICATED: 12 candidates, 0 confirmed newly-refused PLATFORM COVERAGE: posix (7 rows) and windows (5 rows); no UNADJUDICATED:
[SCOPE-REVIEWED] d43f587Security scope review -- adjudicated candidates
No candidate flipped from allowed to refused. |
a0c4dec to
fe80f15
Compare
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: no
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
fe80f15 to
75186d5
Compare
|
self-added: yes
|
|
self-added: yes
|
|
Taking the review-surface point: this change is now split so each piece can be judged on its own, and none of them needs the helper process.
On the contention half: since #10276 raised the anchor-rebuild budget to 8 s plus up to 4 s of grace, one contending thread no longer expires the rebuild on my host (62-anchor rebuild: 2 ms idle, ~2.0 s under a pure-Python GIL hog, ~4.5 s under a long C-regex hold). I will measure sustained multi-thread contention against current main before proposing anything further there; if the window holds, this PR closes in favour of the two above. |
|
A maintainer attempted a conflict-only rebase of this PR onto main What blocks a mechanical rebase:
Once rebased onto current main with the above reconciled, readiness should re-evaluate on the new head. |
|
Rebased onto current What changed in the resolution — read this before re-reviewing. Merged PR #11221 has since landed the wording and classification half of this PR in a refined form, so that half is now taken from
What remains is this PR's own contribution: the out-of-process resolver ( Three resolutions combined both sides rather than picking one, since
|
|
CI triage on The real one, and it was mine to fix
Fixed with an order-preserving The same bug in the hotter place
Local: 2162 targeted tests pass; black / isort / flake8 / mypy clean. The other nine failures, each attributed
The still-running macOS attempt on the old head was cancelled before this push rather than left to burn a runner on a superseded commit. The #11223 overlap noted in my previous comment is unchanged: whichever of the two merges first forces the other to rebase, and this PR still carries the broader |
|
Rebased onto The flakes from the previous head, resolved
One conflict this rebase had to resolve
Both are kept. The caller-supplied path is checked first and returns straight through the shared cache helper: if roots is not None:
return _cached_home_dir_targets(home_dirs, roots, inline=inline)That is correct rather than merely conflict-free: the guard Local: black / flake8 / mypy clean on all 20 changed files; 2139 targeted tests pass across the resolver, gate, hooks, deny-guidance, executor and pod-floor suites. The #11223 overlap is unchanged — that PR is now at |
|
CI is finished on That one is issue #12099, not this PR. The fork Opus lane never reaches a model: its runner has no outbound network (Ubuntu's mirror, Everything that WAS this PR's has been fixed, and every other red was attributed rather than re-run until it went away:
The readiness label stays |
|
/ai-review override fable c161ae6: the lane never reached a model — its fork runner has no egress (the Ubuntu mirror, dl.google.com and packages.microsoft.com all resolve to 54.185.253.63 and refuse), so bubblewrap/socat fail and the Claude Code install exits 7 in the discovery step; tracked as #12099, fork-only, and it failed identically on four fresh heads, so a re-run is blind retry. Authorized by bolichen97 in the driving session. GPT 5.6, Internal Content Scan and Security Scope are green on this head. |
Human judgment recorded@bolichen97 marked the fable AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
Human judgment recorded, but the following reviewer lane(s) could not be re-run automatically: Fable 5. Re-run the lane's latest workflow run for |
|
Rebased onto Dropped from this diff, because it now belongs elsewhere:
Final Local gates on this head: 1534 targeted tests pass (resolver, gate, hooks, deny-guidance, llm_helpers, ACP client, spawn audit, pod suites); black / isort / flake8 / mypy clean; comment-history, docs-lint, brand, feature-map, changelog, harness-parity, memory-store-seam, focus-cue, builtin-skill-scope, loop-bound-locks, testpaths-coverage and vendor-manifest gates pass. One sandbox-only failure ( Known upstream-red on current |
|
Same-diff rebase onto On |
|
Smaller alternative now open: #12739 is the same fix at roughly a fifth of this diff (10 files, +783/-66; ~430 source lines) -- a stdlib helper child, a one-lock client with kill-at-deadline and in-process fallback, and the anchor/candidate seams in I am keeping this PR open and green as the reference implementation until maintainers indicate which shape they prefer; the tests here that #12739 does not carry are listed in its body under "not carried over". Happy to close either in favour of the other. |
|
|
|
… cannot expire the budget The sensitive-path gate refused ordinary project files as "access to sensitive path" whenever the gateway was busy. The resolver was not waiting on the disk: os.path.realpath re-acquires the GIL once per path component, the anchor rebuild does ~400 of those, and each waits a full switch interval when another thread is CPU-bound. Measured: 2 ms idle, 2000-4200 ms under one busy sibling thread, against a 2 s budget. One host logged 57 expiries in a day, each after an event-loop-lag warning. Resolution now runs in a helper process (own GIL): the pool worker pays one pipe round-trip. A helper wedged on a dead mount is killed at the deadline so the worker is reclaimed, and its /proc syscall is the stall sample. A stall is refused as "could not be verified within the resolver budget" instead of as a match, with its own deny-guidance class; shell command text no longer enters the path tier; the stall log names what it observed instead of guessing "stalled mount?".
|
Closing in favour of #12879, which implements the same change. Evidence. #12879's own description states it supersedes this PR's helper and client: "the child-pid syscall sampling, batched anchor request and fail-closed transport posture come from #10256 (@bolichen97) and #12739, whose own helper/client this supersedes." Why this one and not #12879. Same author, same problem; #12879 is the correctly-scoped implementation that keeps the work from here. This is not a judgement on the work -- the two changes were written independently and this one is not the weaker of the pair, it is the one that does not need to land twice. If you disagree, or if #12879 turns out to miss something this branch handles, say so and I will reopen this. From a full relationship audit of every open PR against |
Problem / Motivation
Ordinary project files get refused as
Blocked: access to sensitive path: <file>whenever the gateway is busy. Once it starts, every tool call under that path prefix is refused for 30 s, then 60 s, then 120 s. It happened 57 times in one day on one host, each preceded by anevent-loop heartbeat: lagwarning and none by a slow disk. Agents read the message literally: they hunt for a credential in a source file, decide the session is "locked down", or try another spelling.Why it matters
The gate's fail-closed refusal is right, but its trigger is the gateway's own load, not the disk. So the busier the gateway, the more often its agents are told a normal file is a secret. Two review sessions lost an afternoon each to it, and a resolver worker wedged on a dead mount is never reclaimed.
What changed (motivation → approach → change)
The resolver was waiting on the GIL, not the disk.
os.path.realpathis pure Python issuing onelstatand onereadlinkper path component, and every syscall releases and re-acquires the GIL. The ~60-anchor rebuild every 100 ms is ~400 re-acquisitions, each waiting a full 5 ms switch interval when any other thread is CPU-bound: measured 2 ms idle, 2000–4200 ms beside one busy thread. 400 × 5 ms is the 2 s budget. The fix is to spend the budget in an interpreter with its own GIL.Resolution now runs in a helper process (
security/pathres_helper.py, owned bysecurity/pathres_client.py); themc-pathrespool worker pays one pipe round-trip per request. Round-trips are the cost that scales (CPU-saturated Windows CI: 236 s per gate-heavy test file at one request per anchor per matcher, 37 s onmain), so: the per-call root anchors travel as one request (_realpaths_or_none), the anchor rebuild resolves its ~60 anchors in one request, andsensitive_path_refusal/is_sensitive_write_pathresolve the candidate and the roots once for both matchers (candidates=/roots=seams). Each batch is deduped before it is sent. Caller-supplied roots go straight to_cached_home_dir_targetsand still key the cache.Three properties follow. (1) A helper wedged on a dead mount is killed at the deadline (
ResolverHelper.abort()), so the blocked worker reads EOF and is reclaimed; the adapter preflight does the same at its 60 s deadline (abort_if_inflight, attributed to the expired worker only). (2) The single-threaded helper's/proc/<pid>/syscallis the stall sample_worker_blocked_in_filesystemprefers, attributed to the in-flight worker only; a budget spent queued or spawning is the load arm on every host. (3) In-process resolution is used only when the interpreter cannot run in the child (Popenraising, or a child dying before its first answer), never after a kill the gateway performed, and re-probes after 60 s.Fail-closed is preserved: a transport fault on a candidate is refused as unverifiable; one on an anchor raises
PathResolutionStalledso the sandbox deny mask is never built from a lexical root (the preflight maps it to its existing retryableAcpError). The helper's code is fixed at boot: the source is read once at import and run as-I -S -c <source>by the gateway's own interpreter, never the file by path, with aPATH-only environment (plusSystemRoot/TEMP/TMPon Windows).Pre-declared residuals (recorded in
docs/system-specs/modules/security.md): one helper per gateway, serialised by a request lock, so a candidate wedged on a dead mount holds other path checks for that request's deadline (2 s read gate, 60 s preflight) before the abort frees it -- a check expiring in that window is refused on the load arm, uncharged; and the wedged-mount syscall sample is inferred from analogue kernel waits, not measured on a real wedged mount.Not in this PR: the stall wording,
is_unverifiable_path_refusaland thepath_unverifieddeny class landed in #11221 and are taken frommain; the shell-text exemption of the path tier is #11223.flowchart LR subgraph Before A1[on_tool_call]:::ctx --> B1[mc-pathres thread]:::ctx --> C1[realpath in-process<br/>~400 GIL handoffs]:::removed --> D1[timeout → 'access to sensitive path']:::removed end subgraph After A2[on_tool_call]:::ctx --> B2[mc-pathres thread]:::ctx --> C2[pathres helper process<br/>own GIL, 1 round-trip]:::added --> D2[verdict: sensitive / clear /<br/>'could not be verified']:::changed end classDef added fill:#DCFCE7,stroke:#16A34A,color:#14532D,stroke-width:2px classDef changed fill:#FEF3C7,stroke:#D97706,color:#78350F,stroke-width:2px classDef removed fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-dasharray:4 3 classDef ctx fill:#E0F2FE,stroke:#0284C7,color:#0C4A6E linkStyle 1,2 stroke:#DC2626,stroke-dasharray:4 3 linkStyle 4,5 stroke:#16A34A,stroke-width:2px🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged
Backwards compatibility
Compatible: every public gate keeps its signature and verdicts (
is_sensitive_path,sensitive_path_refusal,is_sensitive_write_path,is_sensitive_resolved_path); the newcandidates=/roots=parameters are keyword-only withNonedefaults, so every existing caller resolves exactly as before. Helper answers are pinned identical to in-process resolution (test_helper_answers_are_identical_to_in_process_resolution), a host that cannot spawn the helper keeps the pre-helper in-process behaviour, and a fault degrades to a refusal, never to a pass.pathres_clientand_realpaths_or_noneare added to thekiro_crew.securityexports; nothing is removed or renamed.Tests
test_resolution_completes_inside_the_budget_while_a_sibling_thread_hogs_the_gil— the reproduction: cold-cache resolution under one GIL-hogging thread is not refused, finishes inside the budget, charges no prefix. With the helper bypassed the same workload expires the budget and fails.test_killing_a_wedged_helper_frees_the_pool_worker,test_abort_kills_without_waiting_on_the_event_loop,test_an_aborted_request_does_not_resubmit_its_wedged_path,test_abort_if_inflight_kills_only_for_the_owning_worker— the deadline kill and its attribution.test_the_helper_is_chosen_under_the_request_lock,test_a_dead_handle_found_at_the_lock_is_replaced_before_the_write,test_a_kill_landing_after_the_handle_was_chosen_is_a_fault_not_an_answer— a handle another request's deadline killed is never used.test_a_transport_fault_on_a_candidate_fails_closed_not_lexical,test_a_transport_fault_on_an_anchor_refuses_the_sandbox_mask,test_a_transport_fault_during_the_grace_wait_refuses_instead_of_going_lexical— a fault is a refusal, never a lexical pass, and charges no cooldown.test_a_queued_request_never_samples_another_requests_wedged_helper,test_a_worker_still_spawning_or_queued_is_the_load_arm_even_without_proc,test_an_abandoned_queued_worker_returns_at_promotion_without_sending,test_exhausted_load_arm_charges_the_prefix_but_does_not_kill_a_healthy_helper— sample attribution and the worker phases.test_the_root_key_resolves_every_anchor_in_one_helper_request,test_root_anchors_name_each_spelling_once_in_the_request,test_anchor_batches_carry_one_spelling,test_realpaths_answers_in_order_and_matches_single_requests,test_pathres_loop_starvation.py::test_duplicate_override_leaves_resolve_once_per_fresh_build— batching and the pre-send dedupe (negative controls fail without it).test_helper_answers_are_identical_to_in_process_resolution,test_a_path_with_undecodable_bytes_survives_the_pipe,test_the_helper_program_*(order, malformed request, surrogate-escaped bytes, stdlib-only imports),test_the_helper_receives_a_minimal_environment,test_the_helper_runs_boot_captured_source_not_the_file_on_disk— the transport contract.test_a_helper_that_cannot_start_falls_back_in_process_once,test_a_helper_that_dies_before_its_first_answer_latches_to_in_process,test_an_abort_never_latches_to_in_process,test_an_in_process_latch_re_probes_after_the_cool_off— the fallback latch.test_a_starved_worker_is_logged_as_contention_not_as_a_mount,test_a_real_stall_is_logged_as_a_mount_that_is_not_answering,test_a_stall_reads_as_unverifiable_and_a_healthy_path_as_clear— the log line and the verdict.test_acp_client_more_coverage.py::test_sandbox_preflight_expiry_aborts_the_resolver_helper;test_spawn_audit.pybenign-spawn entry.Manual verification
Measured on a 64-core Linux host (local XFS): the 62-anchor rebuild in-process is 2 ms idle and 2006 ms / 4193 ms beside one CPU-bound thread; through the helper it stays inside the budget. Locally on this head: 1534 targeted tests pass across the resolver, gate, hooks, deny-guidance, ACP client and pod suites; black / isort / flake8 / mypy clean; the fast-gate scripts (comment-history, docs-lint, brand, feature-map, changelog, harness-parity, memory-store-seam, focus-cue, builtin-skill-scope, loop-bound-locks, testpaths-coverage, vendor-manifest) pass. One sandbox-only failure (
test_pod_windows_seed,Permission denied: 'workplace') reproduces on untouchedmain.Related Issues
Fixes #10255
Pattern harvest
Rule candidate: review-prompt
Pattern: a wall-clock budget spent inside the interpreter (GIL handoffs per syscall) is not a filesystem stall; a bounded check that can only distinguish the two by where the time went must sample the waiting thread, not the clock.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)