Skip to content

fix: resolve sensitive-path symlinks out of process so GIL contention cannot expire the budget - #10256

Closed
Pearcekieser wants to merge 1 commit into
kirodotdev:mainfrom
Pearcekieser:fix/sensitive-path-resolver-gil-starvation
Closed

Pearcekieser wants to merge 1 commit into
kirodotdev:mainfrom
Pearcekieser:fix/sensitive-path-resolver-gil-starvation

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Ordinary project files get refused as Blocked: access to sensitive path: <file> whenever the gateway is busy. Once it starts, every tool call under that path prefix is refused for 30 s, then 60 s, then 120 s. It happened 57 times in one day on one host, each preceded by an event-loop heartbeat: lag warning and none by a slow disk. Agents read the message literally: they hunt for a credential in a source file, decide the session is "locked down", or try another spelling.

Why it matters

The gate's fail-closed refusal is right, but its trigger is the gateway's own load, not the disk. So the busier the gateway, the more often its agents are told a normal file is a secret. Two review sessions lost an afternoon each to it, and a resolver worker wedged on a dead mount is never reclaimed.

What changed (motivation → approach → change)

The resolver was waiting on the GIL, not the disk. os.path.realpath is pure Python issuing one lstat and one readlink per path component, and every syscall releases and re-acquires the GIL. The ~60-anchor rebuild every 100 ms is ~400 re-acquisitions, each waiting a full 5 ms switch interval when any other thread is CPU-bound: measured 2 ms idle, 2000–4200 ms beside one busy thread. 400 × 5 ms is the 2 s budget. The fix is to spend the budget in an interpreter with its own GIL.

Resolution now runs in a helper process (security/pathres_helper.py, owned by security/pathres_client.py); the mc-pathres pool worker pays one pipe round-trip per request. Round-trips are the cost that scales (CPU-saturated Windows CI: 236 s per gate-heavy test file at one request per anchor per matcher, 37 s on main), so: the per-call root anchors travel as one request (_realpaths_or_none), the anchor rebuild resolves its ~60 anchors in one request, and sensitive_path_refusal / is_sensitive_write_path resolve the candidate and the roots once for both matchers (candidates= / roots= seams). Each batch is deduped before it is sent. Caller-supplied roots go straight to _cached_home_dir_targets and still key the cache.

Three properties follow. (1) A helper wedged on a dead mount is killed at the deadline (ResolverHelper.abort()), so the blocked worker reads EOF and is reclaimed; the adapter preflight does the same at its 60 s deadline (abort_if_inflight, attributed to the expired worker only). (2) The single-threaded helper's /proc/<pid>/syscall is the stall sample _worker_blocked_in_filesystem prefers, attributed to the in-flight worker only; a budget spent queued or spawning is the load arm on every host. (3) In-process resolution is used only when the interpreter cannot run in the child (Popen raising, or a child dying before its first answer), never after a kill the gateway performed, and re-probes after 60 s.

Fail-closed is preserved: a transport fault on a candidate is refused as unverifiable; one on an anchor raises PathResolutionStalled so the sandbox deny mask is never built from a lexical root (the preflight maps it to its existing retryable AcpError). The helper's code is fixed at boot: the source is read once at import and run as -I -S -c <source> by the gateway's own interpreter, never the file by path, with a PATH-only environment (plus SystemRoot/TEMP/TMP on Windows).

Pre-declared residuals (recorded in docs/system-specs/modules/security.md): one helper per gateway, serialised by a request lock, so a candidate wedged on a dead mount holds other path checks for that request's deadline (2 s read gate, 60 s preflight) before the abort frees it -- a check expiring in that window is refused on the load arm, uncharged; and the wedged-mount syscall sample is inferred from analogue kernel waits, not measured on a real wedged mount.

Not in this PR: the stall wording, is_unverifiable_path_refusal and the path_unverified deny class landed in #11221 and are taken from main; the shell-text exemption of the path tier is #11223.

flowchart LR
  subgraph Before
    A1[on_tool_call]:::ctx --> B1[mc-pathres thread]:::ctx --> C1[realpath in-process<br/>~400 GIL handoffs]:::removed --> D1[timeout → 'access to sensitive path']:::removed
  end
  subgraph After
    A2[on_tool_call]:::ctx --> B2[mc-pathres thread]:::ctx --> C2[pathres helper process<br/>own GIL, 1 round-trip]:::added --> D2[verdict: sensitive / clear /<br/>'could not be verified']:::changed
  end
  classDef added fill:#DCFCE7,stroke:#16A34A,color:#14532D,stroke-width:2px
  classDef changed fill:#FEF3C7,stroke:#D97706,color:#78350F,stroke-width:2px
  classDef removed fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-dasharray:4 3
  classDef ctx fill:#E0F2FE,stroke:#0284C7,color:#0C4A6E
  linkStyle 1,2 stroke:#DC2626,stroke-dasharray:4 3
  linkStyle 4,5 stroke:#16A34A,stroke-width:2px
Loading

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

Backwards compatibility

Compatible: every public gate keeps its signature and verdicts (is_sensitive_path, sensitive_path_refusal, is_sensitive_write_path, is_sensitive_resolved_path); the new candidates= / roots= parameters are keyword-only with None defaults, so every existing caller resolves exactly as before. Helper answers are pinned identical to in-process resolution (test_helper_answers_are_identical_to_in_process_resolution), a host that cannot spawn the helper keeps the pre-helper in-process behaviour, and a fault degrades to a refusal, never to a pass. pathres_client and _realpaths_or_none are added to the kiro_crew.security exports; nothing is removed or renamed.

Tests

  • test_resolution_completes_inside_the_budget_while_a_sibling_thread_hogs_the_gil — the reproduction: cold-cache resolution under one GIL-hogging thread is not refused, finishes inside the budget, charges no prefix. With the helper bypassed the same workload expires the budget and fails.
  • test_killing_a_wedged_helper_frees_the_pool_worker, test_abort_kills_without_waiting_on_the_event_loop, test_an_aborted_request_does_not_resubmit_its_wedged_path, test_abort_if_inflight_kills_only_for_the_owning_worker — the deadline kill and its attribution.
  • test_the_helper_is_chosen_under_the_request_lock, test_a_dead_handle_found_at_the_lock_is_replaced_before_the_write, test_a_kill_landing_after_the_handle_was_chosen_is_a_fault_not_an_answer — a handle another request's deadline killed is never used.
  • test_a_transport_fault_on_a_candidate_fails_closed_not_lexical, test_a_transport_fault_on_an_anchor_refuses_the_sandbox_mask, test_a_transport_fault_during_the_grace_wait_refuses_instead_of_going_lexical — a fault is a refusal, never a lexical pass, and charges no cooldown.
  • test_a_queued_request_never_samples_another_requests_wedged_helper, test_a_worker_still_spawning_or_queued_is_the_load_arm_even_without_proc, test_an_abandoned_queued_worker_returns_at_promotion_without_sending, test_exhausted_load_arm_charges_the_prefix_but_does_not_kill_a_healthy_helper — sample attribution and the worker phases.
  • test_the_root_key_resolves_every_anchor_in_one_helper_request, test_root_anchors_name_each_spelling_once_in_the_request, test_anchor_batches_carry_one_spelling, test_realpaths_answers_in_order_and_matches_single_requests, test_pathres_loop_starvation.py::test_duplicate_override_leaves_resolve_once_per_fresh_build — batching and the pre-send dedupe (negative controls fail without it).
  • test_helper_answers_are_identical_to_in_process_resolution, test_a_path_with_undecodable_bytes_survives_the_pipe, test_the_helper_program_* (order, malformed request, surrogate-escaped bytes, stdlib-only imports), test_the_helper_receives_a_minimal_environment, test_the_helper_runs_boot_captured_source_not_the_file_on_disk — the transport contract.
  • test_a_helper_that_cannot_start_falls_back_in_process_once, test_a_helper_that_dies_before_its_first_answer_latches_to_in_process, test_an_abort_never_latches_to_in_process, test_an_in_process_latch_re_probes_after_the_cool_off — the fallback latch.
  • test_a_starved_worker_is_logged_as_contention_not_as_a_mount, test_a_real_stall_is_logged_as_a_mount_that_is_not_answering, test_a_stall_reads_as_unverifiable_and_a_healthy_path_as_clear — the log line and the verdict.
  • test_acp_client_more_coverage.py::test_sandbox_preflight_expiry_aborts_the_resolver_helper; test_spawn_audit.py benign-spawn entry.

Manual verification

Measured on a 64-core Linux host (local XFS): the 62-anchor rebuild in-process is 2 ms idle and 2006 ms / 4193 ms beside one CPU-bound thread; through the helper it stays inside the budget. Locally on this head: 1534 targeted tests pass across the resolver, gate, hooks, deny-guidance, ACP client and pod suites; black / isort / flake8 / mypy clean; the fast-gate scripts (comment-history, docs-lint, brand, feature-map, changelog, harness-parity, memory-store-seam, focus-cue, builtin-skill-scope, loop-bound-locks, testpaths-coverage, vendor-manifest) pass. One sandbox-only failure (test_pod_windows_seed, Permission denied: 'workplace') reproduces on untouched main.

Related Issues

Fixes #10255

Pattern harvest

Rule candidate: review-prompt
Pattern: a wall-clock budget spent inside the interpreter (GIL handoffs per syscall) is not a filesystem stall; a bounded check that can only distinguish the two by where the time went must sample the waiting thread, not the clock.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and krishdhasmana September 12, 2026 00:28
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 12, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Intent: Stop the sensitive-path gate from refusing ordinary files when the gateway is busy, by taking symlink resolution off the interpreter whose GIL the resolver was waiting on, and make a resolution that could not complete say so instead of claiming a match.
Not a goal: Changing what the gate refuses (unverifiable is still refused, fail-closed), widening or narrowing the sensitive-path list, or fixing the separate inline-Python and self-kill argv-floor false positives.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 12, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/sensitive-path-resolver-gil-starvation branch from e5223a2 to a0c4dec Compare September 12, 2026 01:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 12, 2026
@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of 0761537473c13c58c8890f32726501d033894001 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Duplicates the just-landed subprocess_pool (#12709): main already ships a separate-interpreter realpath primitive naming this resolver as its first consumer.

Watch

  • Two parallel out-of-process resolvers now coexist. subprocess_pool/_child_realpath.py (HEAD~4) says it is "the pool's FIRST CONSUMER… it belongs next to the sensitive-path resolver it serves and moves there when that caller is wired up" — this PR wires a bespoke security/pathres_client.py instead, so merging leaves an orphaned duplicate with contradictory design rationales (length-prefix framing chosen there because newline framing was judged unsafe; newline-JSON here — safe via ensure_ascii, but two written contracts for one job).
    Clears when: the PR (or an agreed immediate follow-up) names which mechanism survives — either the helper's transport moves onto SubprocessPoolExecutor, or _child_realpath.py is deleted and its docstring promise retired.
  • The transport shape may not hold at the top of the load range it targets. subprocess_pool/executor.py's in-tree measurement table puts "same child via a thread pool" at 1097 ms / 2680 ms for 24 / 48 CPU-bound contenders — past the 2 s budget — and made its Executor caller-thread-lazy for exactly that reason; this PR keeps the mc-pathres pool worker driving the pipe ("the mc-pathres pool worker pays one pipe round-trip per request"), so under sustained heavy contention the load arm exhausts its 3 uncharged probes and the cooldown falsely refusing ordinary paths — the PR's own motivating bug — recurs. The reproduction test pins one hog only.
    Clears when: the reproduction is re-measured at ~24–48 CPU-bound contenders and stays inside the budget, or the transport adopts the caller-thread-wait shape.

[DESIGN-REVIEWED] 0761537

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🔴 BLOCK

Premise-level review of 0761537473c13c58c8890f32726501d033894001 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: BLOCK

The base already ships this mechanism: subprocess_pool's _child_realpath names the sensitive-path resolver as its first consumer, and this PR builds a second one beside it.

Not justified as shipped

  • Item 1 — duplicate of src/kiro_crew/subprocess_pool/: "Resolution now runs in a helper process (security/pathres_helper.py, owned by security/pathres_client.py)" re-solves the job the base's merged mechanism (feat(subprocess-pool): an Executor that runs work in a separate interpreter #12709) was built for, and the readable motivation/approach never engages it.
    Subtraction: delete pathres_client.py/pathres_helper.py; wire the gate to SubprocessPoolExecutor.realpath_spellings and add the batch op per that module's documented adoption recipe.
    Clears when: the gate consumes kiro_crew.subprocess_pool and the two new modules are gone, or a maintainer records that the pool is superseded.
  • Item 2 — duplicate of SubprocessPoolExecutor's request-ceiling reaper: abort()/abort_if_inflight re-implement kill-at-deadline the executor already provides (kill_async, DEFAULT_REQUEST_CEILING_SECS).
    Clears when: the same consumption above.
  • Item 3 — duplicate of the pool's two-child design: the declared residual ("one helper per gateway, serialised by a request lock") is exactly the workers=1 consequence executor.py warns against — "one bad mount refusing paths under every other prefix too".
    Clears when: the same consumption above.
  • Item 7 — rides along: load-arm/stall log rewording and debug→WARNING promotion; harm named (operators diagnosed the wrong layer), nothing more.

What this change ships

Inventory (10 items) — 6 justified

Intent: stop refusing ordinary project files as "sensitive" when the gateway's own CPU load, not the disk, expires the resolve budget — a FIX (repro test fails on base).

  1. Busy gateway no longer refuses ordinary files; resolution runs in a new helper process — duplicate of src/kiro_crew/subprocess_pool/
  2. A check wedged on a dead mount now frees its worker via a deadline kill — duplicate of SubprocessPoolExecutor's ceiling reaper
  3. One wedged check now holds all others for its deadline (single serialized helper) — duplicate of the pool's two-child design
  4. Anchors and roots resolve in one batched request, shared across both matchers — justified
  5. A host that cannot spawn the helper falls back in-process, re-probing after 60 s — justified
  6. Adapter spawn refuses with a retryable error when an anchor cannot resolve — justified
  7. Timeout logs now name interpreter contention vs. a mount, at WARNING — rides along
  8. pathres_client / _realpaths_or_none exported from kiro_crew.security — justified
  9. Empty paths short-circuit before reaching the resolver — justified
  10. Spec, stale hooks doc line, and spawn-audit allowlist updated in the same commit — justified

Blockers

Duplicate out-of-process resolver beside the one the base already carries.
The merged pool's child says on base: "This is the subprocess pool's FIRST CONSUMER … it belongs next to the sensitive-path resolver it serves and moves there when that caller is wired up" (_child_realpath.py). Grepped subprocess_pool|SubprocessPoolExecutor|realpath_spellings across src/: 4 files — the package's own 3 plus an unrelated name in executors.py — so the pool has ZERO consumers and awaits exactly this wiring. The new transport also silently reverses that module's recorded choices: newline framing where it pins "Length prefixes, never a newline", and one serialized child where it pins two.
Subtraction: consume kiro_crew.subprocess_pool (extend _child_realpath with the batch op and phase hooks) and delete pathres_client.py/pathres_helper.py.
Clears when: the gate resolves through subprocess_pool with the two new modules deleted, or a maintainer records the pool as superseded by this design.

[FIRST-PRINCIPLES-REVIEWED] 0761537

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 0761537473c13c58c8890f32726501d033894001 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0761537

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 12, 2026
@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 0761537473c13c58c8890f32726501d033894001 via the fork AI-review pipeline; updated in place on each push.

Review details

Based on my analysis: the candidate list contained no candidates (the discovery pass found nothing), so Step 1 has nothing to falsify. I worked through the production code in Step 2 to see whether any grounded defect was missed — the helper-process delegation in pathres_client.py, the batched anchor resolution and dedupe in _home_dir_targets_uncached/_resolve_root_anchors (every real[...]/resolved[...] key is provably present in wanted after dict.fromkeys), the fail-closed PathResolutionStalled propagation through _resolved_spellings/_realpath(s)_or_none and its handlers in _run_preflight_bounded and the grace/outer ladders, and the owner/abort attribution. Each path fails closed and I could not ground (a)+(b)+(c) for any new defect at the 80+ bar.

No findings.

[OPUS-REVIEWED] 0761537

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review (fork) — 🟡 CONCERNS

Which legitimate operations does d43f58781b65958ed91ecebc5384efb439bc70f7 newly refuse? Model proposes, scripts/deny_diff.py decides — updated in place on each push. A script-confirmed regression blocks PR readiness.

the reviewer's own judgement; no confirmed regression

Scope-Verdict: CONCERNS

TIGHTENING: the sensitive-path resolver's failure handling went fail-closed on both halves of the check -- a candidate or anchor whose resolution does not complete is now refused instead of degraded to its lexical spelling. The one threat this refuses is a workspace symlink into a credential store (and, on the spawn path, a symlinked credential home left out of the OS sandbox deny mask) passing while resolution is unavailable, quoting the _resolved_spellings hunk: "A transport fault (the helper died, or answered garbage) is a RESOLUTION THAT DID NOT COMPLETE, and it raises PathResolutionStalled for this path exactly as a timeout does -- it must not return the empty set, because an empty set reads to the caller as 'resolved, no other spelling' and leaves only the lexical forms, which for a workspace symlink into a credential store is a pass." The same posture is extended to anchors (_realpath_or_none/_realpaths_or_none raise where the base returned None and kept the lexical anchor) and to the adapter preflight (_run_preflight_bounded now catches PathResolutionStalled and refuses to start the adapter: "The adapter is not started without its mask; retry shortly").

ADJUDICATED: 12 candidates, 0 confirmed newly-refused
No harness verdict reported any candidate flipping from allowed at 80bd0a8 to refused at d43f587, and a regression without a script verdict is not a finding. On a healthy host the string-visible matching is unchanged: the set of strings reaching the resolver is the same (the only new short-circuit, the empty title, is a loosening), the per-path OSError/ValueError lexical-fallback semantics are preserved inside the helper's _resolve_one, and a helper that cannot spawn latches to the exact in-process resolution the base ran.

PLATFORM COVERAGE: posix (7 rows) and windows (5 rows); no any rows, because every row carries a path, a separator, or an executable name. The posix rows cover the diff's POSIX assumptions: the /local/home container-of-homes cooldown prefix, home-rooted reads and workspace symlinks through the new helper round-trip, the .aws/.aws-sam anchor component boundary, the ASCII-JSON pipe protocol against a non-ASCII filename, and the documented KIROCREW_PATH_RESOLVE_WORKERS knob. The windows rows cover the diff's explicit Windows assumptions: the SystemRoot/TEMP/TMP child-interpreter environment, %VAR% and backslash spellings, the one-spelling anchor batching motivated by handle-per-component cost, the _is_unc_path lexical shortcut, and the Windows test-lane interpreter path. One platform assumption I could not express as a row: the helper's /proc/<pid>/syscall stall sample is Linux-only and the diff's new phase classifier exists precisely for hosts without it (Windows, macOS) -- that behavior is runtime state, not a command spelling, and is named under UNADJUDICATED. Per-leg verdicts are the deterministic fold's to report, not mine.

UNADJUDICATED:

  1. Adapter-spawn admission (acp/client.py _run_preflight_bounded). Call path: session start → AcpClient._spawn → _run_preflight_bounded(_sandbox_preflight) → sandbox_credential_targets → _resolve_root_anchors → _realpaths_or_none → resolver helper. At base, an anchor whose realpath raised degraded to its lexical spelling and the adapter started; at head, a helper transport fault or an open stall cooldown raises PathResolutionStalled and the adapter is refused (retryable AcpError), and the expiry arm now also abort_if_inflights the helper. No shell row reaches this guard. Candidate legitimate inputs I would have submitted: starting an ordinary codex/claude session on a healthy host; a session start on a host where the gateway cannot fork a child (ENOMEM, a sandboxed install -- covered by the in-process latch, but the latch window is the input to probe); a session start during an active stall cooldown on the home prefix, where the base started the adapter with lexical anchors and the head refuses until the cooldown lapses.
  2. The fault-conditioned resolver arms. The transport-fault refusals on candidates and anchors fire only when the helper process dies or answers garbage -- a runtime condition no classified command string can produce, so the zero-flip adjudication above is not evidence about them. The fail-open cover is the in-process latch (Popen raising, or a child dead before its first answer, resolves in-process for 60 s); the input to probe is any ordinary file read on a host where subprocess spawn is denied.
  3. NARROWING, arguable but not confirmable by the classifier: the load arm lost its grace. At base, a resolution that missed its budget with the worker NOT blocked in a filesystem syscall and the prefix's three uncharged probes spent fell through to the bounded grace wait, and a late completion was ALLOWED with no charge (base paths.py:1842-1853). At head, that exact case is refused immediately and _mark_stalled opens the per-prefix cooldown with no grace at all, so on a CPU-loaded Linux host an ordinary project read is refused one grace-window earlier than base and takes every path under its prefix with it for the 30 s-to-30 min cooldown. The helper is expected to make this arm rare (its purpose is removing the GIL convoy), but when the arm fires the head is strictly stricter than the base. Narrow it: keep the grace wait ahead of the charge in the exhausted-probes contention arm -- refuse-and-charge only after the grace also expires, which still keeps a many-token call from paying the budget per token.

[SCOPE-REVIEWED] d43f587

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 7 classified, 0 newly refused, 7 still allowed.
  • ubuntu-latest (platform posix): 7 classified, 0 newly refused, 7 still allowed.
  • windows-latest (platform windows): 5 classified, 0 newly refused, 5 still allowed.

No candidate flipped from allowed to refused.

@Pearcekieser
Pearcekieser force-pushed the fix/sensitive-path-resolver-gil-starvation branch from a0c4dec to fe80f15 Compare September 12, 2026 01:30
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: anchor transport fault raises PathResolutionStalled; preflight maps it to a retryable AcpError

  • fixed — Anchor transport faults can drop canonical credential masks (_realpath_or_none), fixed in fe80f15

A helper transport fault on an ANCHOR no longer returns None: _realpath_or_none raises PathResolutionStalled, so no caller can degrade a root to its lexical spelling. None is reserved for a realpath that raised (OSError/ValueError), which is the pre-existing documented case.
On the spawn path sandbox_credential_targets propagates that stall and _run_preflight_bounded maps it to the same retryable AcpError its 60 s timeout raises, so the adapter is refused rather than started with a mask missing the canonical credential home. Pinned by test_a_transport_fault_on_an_anchor_refuses_the_sandbox_mask.
This ruling covers every instance of "transport fault → lexical spelling" on either half of the check: candidates were already fail-closed (_resolved_spellings), anchors now are too.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: helper syscall sample attributed to the requesting worker (native tid)

  • fixed — helper sample used without checking whose request is in flight, fixed in fe80f15

ResolverHelper now records the native thread id of the worker whose request is in flight, and blocked_in_filesystem(fs_syscalls, tid) answers only when tid owns that request; a worker queued on _request_lock gets None and falls back to its own thread sample (which for a worker parked on a lock reads as not blocked in the filesystem, i.e. the load arm, charging no prefix).
Pinned by test_a_queued_request_never_samples_another_requests_wedged_helper.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • rebutted — batch the ~60-anchor rebuild into one resolve([...]) call

Disproportional for this PR. The per-path round-trip costs two GIL re-acquisitions, so the rebuild is ~120 vs the ~400 it replaced, and the measured worst case under a saturating hog thread is 0.3 s alone and 1.04 s under a full 12-worker xdist run, against a 2 s budget: the fix's headroom is what the reproduction test pins.
Batching means restructuring _home_dir_targets_uncached into a record-then-resolve two-pass, and _realpath_or_none is the seam the existing stall tests (test_a_stalled_root_anchor_refuses_within_the_budget, test_a_unc_home_still_has_its_anchors_resolved, ...) pin by call sequence; changing it widens this security PR into the anchor builder. Worth its own change if the headroom ever proves insufficient.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • fixed — sensitive_path_verdict generalized for one consumer, fixed in fe80f15

Folded into sensitive_path_refusal, which now catches PathResolutionStalled itself; VERDICT_SENSITIVE/VERDICT_CLEAR/VERDICT_UNVERIFIABLE are gone from paths.py, security/__init__.py and _exports.py. The tier is one function with one consumer shape (hooks, deny_diff, the conductor).

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: no

  • fixed — safe_read_file still names an ordinary file as a sensitive-path match under a stall, fixed in fe80f15

hooks.safe_read_file now asks sensitive_path_refusal(resolved); a stall is refused with the unverifiable wording, a match keeps Blocked: access to sensitive path: <repr> (the repr is retained because the text reaches log records via exc_info). Every access to sensitive path producer in src/ now distinguishes stall from match.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • fixed — two spellings of the unverifiable anchor, fixed in fe80f15

deny_guidance._CLASS_ANCHORS now uses security.UNVERIFIABLE_PATH_ANCHOR; one spelling remains.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • fixed — fold sensitive_path_verdict into sensitive_path_refusal and drop the VERDICT_* exports, fixed in fe80f15

Done as described: one function, stall caught inside it, three constants and the facade/manifest entries removed.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • fixed — replace the literal anchor tuple in deny_guidance._CLASS_ANCHORS with the constant, fixed in fe80f15

(security.UNVERIFIABLE_PATH_ANCHOR,) is the entry now.

@Pearcekieser
Pearcekieser force-pushed the fix/sensitive-path-resolver-gil-starvation branch from fe80f15 to 75186d5 Compare September 12, 2026 02:19
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: helper source captured at import; child runs python -I -c

  • fixed — Agent-writable helper executes outside the sandbox, fixed in 75186d5

The child no longer runs pathres_helper.py by path. pathres_client reads the helper source ONCE at import (gateway boot, before any agent turn) and spawns [sys.executable, "-I", "-c", _HELPER_SOURCE]; the file on disk is never re-read for a spawn. An edit to the file now takes effect at the next gateway restart -- the operator's action -- exactly like every other product source file, so the on-demand respawn is no longer a no-restart code-execution path.
Pinned by test_the_helper_runs_boot_captured_source_not_the_file_on_disk (argv is -I -c <captured source>, no path argument, and the captured source equals the shipped file).
This ruling covers every instance of "a respawned child reads agent-writable content": the only content the child receives at runtime is the JSON path request on stdin, which it treats as data.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • rebutted — pin that a bare-path shell command is still stopped somewhere

A shell command consisting solely of ~/.aws/credentials EXECUTES that path; it is not a read, and no layer of this gate ever treated command text as a path on purpose (base spec: "The gate does not match PATHS in command text", pinned by TestTheBashGateMatchesNoPaths). What stops credential access from the shell is the OS sandbox, whose coverage is already pinned per mode (test_every_mode_carries_the_derived_set; .aws hidden in cc/strict, visible in standard by design for credential_process). A new test asserting "a bare path as the whole command is denied by X" would pin a control that does not exist at the bash tier by design, and the same residual was written up for the maintainer's ack on an earlier head (design span fe92d308bdc1).

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Taking the review-surface point: this change is now split so each piece can be judged on its own, and none of them needs the helper process.

On the contention half: since #10276 raised the anchor-rebuild budget to 8 s plus up to 4 s of grace, one contending thread no longer expires the rebuild on my host (62-anchor rebuild: 2 ms idle, ~2.0 s under a pure-Python GIL hog, ~4.5 s under a long C-regex hold). I will measure sustained multi-thread contention against current main before proposing anything further there; if the window holds, this PR closes in favour of the two above.

@bolichen97

Copy link
Copy Markdown
Contributor

A maintainer attempted a conflict-only rebase of this PR onto main cff7de54 (2026-09-19) but did not push, because the conflict is semantic rather than textual and resolving it needs changes outside the conflict hunks -- which is the author's call. Nothing on this branch was changed.

What blocks a mechanical rebase:

  • The textual conflicts are resolvable, but the automatic part of the merge changes the non-conflicted src/kiro_crew/deny_guidance.py: it reintroduces substring-based refusal classification that main has since hardened. A focused probe: an AWS-credential refusal classifies as aws_credential on main but as path_unverified after the merge.
  • Keeping main's classifier requires editing that non-conflicted file. The out-of-process symlink-resolution premise itself still holds (main still resolves on resolver threads).

Once rebased onto current main with the above reconciled, readiness should re-evaluate on the new head.

@bolichen97

Copy link
Copy Markdown
Contributor

Rebased onto current main (8f52009dcd) by a maintainer at the author's PR being 563 commits behind. Old head ef244d6dd6, new head 8ebd554a5c. Local gates green: black / isort / flake8 / mypy clean on every changed file, and 1147 targeted tests pass (test_security_path_resolve_bounded, test_hooks, test_deny_guidance, test_deny_diff, test_llm_helpers_tool_input_offload, test_security_conductor_verify_fix, test_hooks_coverage, test_acp_client_more_coverage, test_spawn_audit, test_pipeline_conductor_skill_contract).

What changed in the resolution — read this before re-reviewing. Merged PR #11221 has since landed the wording and classification half of this PR in a refined form, so that half is now taken from main and dropped from the diff:

  • UNVERIFIABLE_PATH_ANCHOR (a substring the classifier searched for) is gone; the tree uses main's UNVERIFIABLE_PATH_PREFIX plus is_unverifiable_path_refusal(), a fixed-prefix test. That is strictly stronger: a substring test lets a path spelled to contain the phrase pass itself off as a stall, and main carries a regression test for exactly that forgery.
  • deny_guidance.py is now byte-identical to main: its DENY_CLASS_PATH_UNVERIFIED class, anchor ordering and remediation text were already there, and this PR's duplicate anchor rule was removed so there is one classification path, not two.
  • test_deny_guidance.py and test_hooks_coverage.py are likewise back to main's versions and no longer appear in the diff.

What remains is this PR's own contribution: the out-of-process resolver (security/pathres_client.py, security/pathres_helper.py), the batched anchor resolution (_realpaths_or_none, one round-trip for ~60 anchors), and the shared per-gate-call resolution threaded into both matchers.

Three resolutions combined both sides rather than picking one, since main grew features this PR predates:

  • _home_dir_targets(home_dirs, *, inline=False, roots=None) — main's inline (resolve on the calling thread, for is_sensitive_resolved_path) and this PR's roots (caller already resolved them). roots still keys the TTL cache, so freshness is unchanged.
  • _path_in_home_dirs / _is_keystone_publish_artifact carry pre_resolved (main) and candidates / roots (this PR).
  • main's memoizing resolve_target helper inside _home_dir_targets_uncached was removed as dead code: real, the dict built from the one batched request, subsumes it and still resolves each repeated spelling once per build.

hooks.py, llm_helpers.py, deny_diff.py and verify_fix.py keep this PR's own is_shell exemption verbatim. Note the overlap: open PR #11223 replaces that broad exemption with a narrower one (only the recovered command of a sandboxed, non-MCP shell, because classify_tool_call can report is_shell=True alongside an MCP server, and a use_aws-shaped frame can carry a credential path as a structured argument). #11223 is the author's own later correction of this same code. Whichever merges second needs one more rebase; this resolution deliberately did not make that design change on the author's behalf.

@bolichen97

Copy link
Copy Markdown
Contributor

CI triage on 8ebd554a5c plus two fixes. New head 1f9a503759, rebased onto f2ea1261ef (which carries #12341, the macOS fix below). Exactly one of the ten failures was this PR's.

The real one, and it was mine to fix

test_pathres_loop_starvation.py::test_duplicate_override_leaves_resolve_once_per_fresh_build failed assert 0 == 1 on Linux shard 8, Windows shard 8 and macOS shard 2. It pins that each identical anchor spelling is resolved once per build, and it stubs _realpath_or_none. The batched rebuild goes through _realpaths_or_none, so the stub saw nothing — but that was only the visible half. The list handed to the helper still carried the duplicate (both crew-home prefixes anchor the same leaf under KIROCREW_HOME), and dict(zip(...)) collapsed it only after the helper had resolved it once per occurrence. One request is not the whole saving; the repeat has to be gone before it goes out.

Fixed with an order-preserving dict.fromkeys dedupe ahead of the request (home stays first — a transport fault raises for the first path, and that prefix is the one the rebuild charges), and the test now stubs _realpaths_or_none with all five assertions and its generation-change freshness control unchanged. Negative control: drop the dedupe and it fails assert 2 == 1.

The same bug in the hotter place

_resolve_root_anchors batches the same way and had the same defect, and it runs on every gate call rather than once per 0.1 s TTL. Several env vars routinely name one directory (an adapter override pointed at the crew home, or one set to $HOME). Same dedupe, plus test_root_anchors_name_each_spelling_once_in_the_request beside the existing one-request test: it asserts the sent list holds each spelling once and that collapsing costs neither root its answer. Negative control fails without the dedupe. This one had no failing test to find it — it was found by looking for the same shape after fixing the first.

Local: 2162 targeted tests pass; black / isort / flake8 / mypy clean.

The other nine failures, each attributed

The still-running macOS attempt on the old head was cancelled before this push rather than left to burn a runner on a superseded commit.

The #11223 overlap noted in my previous comment is unchanged: whichever of the two merges first forces the other to rebase, and this PR still carries the broader if not is_shell: form on purpose rather than adopting #11223's narrower one on the author's behalf.

@bolichen97

Copy link
Copy Markdown
Contributor

Rebased onto e8298981bf (21 commits of main), new head c161ae64e4. The previous head reached zero real failures — every check green except Opus 4.8 Review, which is issue #12099 and not this PR's (see below). This rebase is to pick up main and give that lane a fresh attempt, not to fix anything.

The flakes from the previous head, resolved

One conflict this rebase had to resolve

main split _home_dir_targets in two: it now short-circuits the TTL cache before taking a new _home_targets_inline_lock, then delegates to a new _cached_home_dir_targets helper, so that concurrent inline builders coalesce instead of each rebuilding. This PR's change to the same lines added the roots parameter.

Both are kept. The caller-supplied path is checked first and returns straight through the shared cache helper:

if roots is not None:
    return _cached_home_dir_targets(home_dirs, roots, inline=inline)

That is correct rather than merely conflict-free: the guard main added exists because resolving the roots twice (once for the key, once for the build) lets a root symlink repointed in between file one root's targets under another root's key — a fail-open TOCTOU. When the caller hands the roots in, there is no second resolution to race, and nothing for the inline lock to coalesce. Freshness is unchanged: the caller resolved them on this same call, and they still key the cache.

Local: black / flake8 / mypy clean on all 20 changed files; 2139 targeted tests pass across the resolver, gate, hooks, deny-guidance, executor and pod-floor suites.

The #11223 overlap is unchanged — that PR is now at 9095c00150, still carrying the narrower exemption, and whichever merges first will force the other to rebase.

@bolichen97

Copy link
Copy Markdown
Contributor

CI is finished on c161ae64e4: 100 checks, 91 green, and exactly one failure — Opus 4.8 Review. Nothing in this diff is red.

That one is issue #12099, not this PR. The fork Opus lane never reaches a model: its runner has no outbound network (Ubuntu's mirror, dl.google.com and packages.microsoft.com all resolving to 54.185.253.63 and refusing), so bubblewrap/socat fail to install and the Claude Code install exits 7 in the discovery step. The lane then declines to invent a verdict and reports review incomplete, which is the correct behaviour. Root-cause evidence is on #12099. It is fork-only — every same-repo PR sampled today had the lane green — and it is intermittent rather than permanent, so it has been given four fresh heads rather than blind re-runs.

Everything that WAS this PR's has been fixed, and every other red was attributed rather than re-run until it went away:

The readiness label stays action required on the Opus lane alone. Clearing it needs either that lane fixed or a maintainer /ai-review override — neither is reachable from inside a fork PR, so this is where the automated pass stops.

@bolichen97

Copy link
Copy Markdown
Contributor

/ai-review override fable c161ae6: the lane never reached a model — its fork runner has no egress (the Ubuntu mirror, dl.google.com and packages.microsoft.com all resolve to 54.185.253.63 and refuse), so bubblewrap/socat fail and the Claude Code install exits 7 in the discovery step; tracked as #12099, fork-only, and it failed identically on four fresh heads, so a re-run is blind retry. Authorized by bolichen97 in the driving session. GPT 5.6, Internal Content Scan and Security Scope are green on this head.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@bolichen97 marked the fable AI finding as false positive, not applicable, or explicitly accepted for c161ae64e4e65c4f4d12196b7cd230799548c0e3.

the lane never reached a model — its fork runner has no egress (the Ubuntu mirror, dl.google.com and packages.microsoft.com all resolve to 54.185.253.63 and refuse), so bubblewrap/socat fail and the Claude Code install exits 7 in the discovery step; tracked as #12099, fork-only, and it failed identically on four fresh heads, so a re-run is blind retry. Authorized by bolichen97 in the driving session. GPT 5.6, Internal Content Scan and Security Scope are green on this head.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded, but the following reviewer lane(s) could not be re-run automatically: Fable 5. Re-run the lane's latest workflow run for c161ae64e4e65c4f4d12196b7cd230799548c0e3 manually from the Actions tab.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased onto main 0a80050938 (#12707), new head 8e8d9a69f9, one commit. Thanks to @bolichen97 for the three maintainer rebases and the notes above -- the _home_dir_targets split resolution (roots is not None short-circuits straight to _cached_home_dir_targets) and the order-preserving pre-send dedupe in both _resolve_root_anchors and the anchor rebuild are carried forward exactly as written there.

Dropped from this diff, because it now belongs elsewhere:

  • Everything fix(security): word a resolver stall as unverifiable, not a path match #11221 landed: sensitive_path_refusal as reason-or-None, UNVERIFIABLE_PATH_PREFIX / is_unverifiable_path_refusal, the path_unverified deny class and its tests are taken from main as-is (already the case since the 09-20 rebase; the PR body now says so instead of describing them as this PR's).
  • The shell-text exemption of the path tier (if not is_shell: in hooks.on_tool_call, the is_shell= threading through _title_denial / _first_tool_input_denial, and the matching deny_diff.py / verify_fix.py tier-table changes with their tests in test_hooks.py, test_llm_helpers_tool_input_offload.py, test_deny_diff.py, test_security_conductor_verify_fix.py). That is fix(security): do not hand shell command text to the path tier #11223's narrower correction of the same code and is open separately; this PR no longer overlaps it, so neither forces the other to rebase. hooks.py, llm_helpers.py, deny_diff.py and verify_fix.py are byte-identical to main.
  • The one doc conflict (security.md): main's paragraph kept (pool-size knob, home-container stall prefix), this PR's helper-process text appended, the fix(security): word a resolver stall as unverifiable, not a path match #11221 wording sentences and the shell-tier sentence removed.

Final --stat: 11 files -- security/pathres_client.py (+477), security/pathres_helper.py (+99), security/paths.py (+425/-116 net), acp/client.py (preflight abort + PathResolutionStalled mapping), security/__init__.py / _exports.py, the spec doc, test_security_path_resolve_bounded.py, test_pathres_loop_starvation.py, test_acp_client_more_coverage.py, test_spawn_audit.py.

Local gates on this head: 1534 targeted tests pass (resolver, gate, hooks, deny-guidance, llm_helpers, ACP client, spawn audit, pod suites); black / isort / flake8 / mypy clean; comment-history, docs-lint, brand, feature-map, changelog, harness-parity, memory-store-seam, focus-cue, builtin-skill-scope, loop-bound-locks, testpaths-coverage and vendor-manifest gates pass. One sandbox-only failure (test_pod_windows_seed, Permission denied: 'workplace') is environmental.

Known upstream-red on current main, not this diff: test_pending_boundary_consumer_semantics.py::test_running_and_turn_running_slot_readers_are_enumerated (#12713); if it fails here it will need that fix or a rebase past it.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Same-diff rebase onto main 194f6d51da (#12718), new head 0d1f0e3eab; git diff against the respective bases is byte-identical to 8e8d9a69f9.

On 8e8d9a69f9 every lane was green except Linux shard 4 and Windows shard 4 (and Coverage Gate downstream of them), both on test_pending_boundary_consumer_semantics.py::test_running_and_turn_running_slot_readers_are_enumerated -- issue #12713, red on every merge ref of main until #12718, which this head now carries. GPT 5.6, Opus 4.8, Design, UX, PR Scope and Internal Content Scan all passed on the previous head with no findings.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Smaller alternative now open: #12739 is the same fix at roughly a fifth of this diff (10 files, +783/-66; ~430 source lines) -- a stdlib helper child, a one-lock client with kill-at-deadline and in-process fallback, and the anchor/candidate seams in paths.py batched into one request each. It keeps upstream's _run_resolution_bounded bookkeeping and drops the latch/owner/abort-attribution machinery this PR accreted under review. Measurements on current main (in the #12739 body): the 70-anchor rebuild in-process takes 1.2 s / 7.6 s / 15.4 s / 20.5 s median at 1/2/4/8 GIL hogs against the 8 s + 4 s budget; through the child 11 / 27 / 11 / 8 ms.

I am keeping this PR open and green as the reference implementation until maintainers indicate which shape they prefer; the tests here that #12739 does not carry are listed in its body under "not carried over". Happy to close either in favour of the other.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • fixed — shared resolution bypasses the empty-path guard (paths.py:3256)

Legitimate. Hoisting _candidate_forms in front of both matchers moved it ahead of their if not path_str short-circuits, so an empty tool title reached _resolved_forms_bounded("") and, under an active stall cooldown on that prefix, was refused as an unverifiable path — a refusal of an unrelated tool with no path in it (fail-closed, self-healing, as the adjudication says, but still a regression from base). sensitive_path_refusal and is_sensitive_write_path now return None/False for an empty spelling before any resolution. Pinned by test_an_empty_spelling_never_reaches_the_resolver, which arms a stalled resolver and asserts zero resolver calls for "" through all three funnels.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Correct: that clause was a leftover from before the split, and merged alone it would have had the spec describe a gate hooks.py does not enforce on base. The "(on non-shell tools; … is_sensitive_bash_command's to judge)" clause is dropped from this diff; the bullet keeps only the is_sensitive_path → sensitive_path_refusal rename. #11223 carries its own doc line.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • fixed — _run_preflight_bounded made ownerless resolver-helper requests (acp/client.py:4831)

Taken, advisory or not: the read gate wraps every resolution in pathres_client.owner(...) precisely so a worker promoted onto the helper lock after its deadline returns instead of sending a wedged path with nobody left to abort it, and the preflight path reaches the same lock via sandbox_credential_targets without that guard. The preflight worker now runs inside pathres_client.owner(abandoned) and both give-up branches (TimeoutError, PathResolutionStalled) set the event. Pinned by test_sandbox_preflight_binds_an_owner_the_deadline_gives_up.

… cannot expire the budget

The sensitive-path gate refused ordinary project files as "access to
sensitive path" whenever the gateway was busy. The resolver was not
waiting on the disk: os.path.realpath re-acquires the GIL once per path
component, the anchor rebuild does ~400 of those, and each waits a full
switch interval when another thread is CPU-bound. Measured: 2 ms idle,
2000-4200 ms under one busy sibling thread, against a 2 s budget. One
host logged 57 expiries in a day, each after an event-loop-lag warning.

Resolution now runs in a helper process (own GIL): the pool worker pays
one pipe round-trip. A helper wedged on a dead mount is killed at the
deadline so the worker is reclaimed, and its /proc syscall is the stall
sample. A stall is refused as "could not be verified within the resolver
budget" instead of as a match, with its own deny-guidance class; shell
command text no longer enters the path tier; the stall log names what it
observed instead of guessing "stalled mount?".
@bolichen97

Copy link
Copy Markdown
Contributor

Closing in favour of #12879, which implements the same change.

Evidence. #12879's own description states it supersedes this PR's helper and client: "the child-pid syscall sampling, batched anchor request and fail-closed transport posture come from #10256 (@bolichen97) and #12739, whose own helper/client this supersedes."

Why this one and not #12879. Same author, same problem; #12879 is the correctly-scoped implementation that keeps the work from here.

This is not a judgement on the work -- the two changes were written independently and this one is not the weaker of the pair, it is the one that does not need to land twice. If you disagree, or if #12879 turns out to miss something this branch handles, say so and I will reopen this.

From a full relationship audit of every open PR against main (baseline a992d5cda5). Overlap was measured with git diff against each branch's own merge base, and re-measured after excluding files that 20+ open PRs touch, so shared locale and lockfile edits could not inflate it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sensitive-path gate refuses ordinary files under load: resolver budget is spent on GIL handoffs, not the disk

3 participants