Repository navigation
fix(security): make the path-resolver pool size an operator knob - #12460
Conversation
|
Intent: Let an operator size the sensitive-path resolver pool for their host ( |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the deferred observability follow-up (refusal names its arm) is actually filed — the knob is flown blind without it. [DESIGN-REVIEWED] 47b20f7 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsThe single candidate concerns the Falsifying it: No grounded new findings emerged. No findings. [OPUS-REVIEWED] 47b20f7 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The refusal-window claim is argued from the guard's condition, never measured — only the reporting host's operator can confirm the widened pool ends it. What this change shipsInventory (6 items) — 6 justifiedIntent: let the operator of a CPU-starved host raise the path-resolver pool ceiling so two pinned resolutions stop refusing every tool call — an ADDITION (opt-in knob), despite the
[FIRST-PRINCIPLES-REVIEWED] 47b20f7 |
Security Scope Review -- ✅ PASSWhich legitimate operations does Verdict basis: adjudicated, zero confirmed regressions. Security scope review -- adjudicated candidates
No candidate flipped from allowed to refused. Scope-Verdict: PASS TIGHTENING: No operation gate got stricter. The runtime change makes the ADJUDICATED: 15 candidates, 0 confirmed newly-refused PLATFORM COVERAGE: posix and windows; no [SCOPE-REVIEWED] 47b20f7 |
a405d02 to
77fd6df
Compare
77fd6df to
0ca6cf2
Compare
0ca6cf2 to
629f970
Compare
The sensitive-path gate resolves every tool-call path on its own thread pool, sized by _MAX_PATH_RESOLVE_WORKERS = 2. The cap was justified by "healthy resolution is microseconds, so sustained queueing means the filesystem is wedged". That premise is a property of the host, not of the code: the anchor rebuild behind the 0.1s cache runs near-continuously under ordinary tool traffic, and under GIL contention one rebuild that costs 0.9ms on an idle interpreter measured 873ms with one CPU-bound sibling thread. Where both hold, the pool sits at its wedged floor and sensitive_path_refusal refuses every tool call, fail-closed, on a perfectly healthy mount. Read the size from KIROCREW_PATH_RESOLVE_WORKERS once at import. The default stays 2: on the host the premise was written for, a low ceiling is what makes the wedge signal meaningful, and every install should not pay a wider pool for a problem only some hosts have. An unparseable or out-of-range value keeps the default and logs a warning, so a typo can only ever leave the shipped behaviour in place, never widen it. The accepted range is 2..64. The floor is 2 because of the leave-one-free guard in security.paths: a prefix with a stall on record is refused before submission once W - 1 workers are pinned, which with W = 1 reads "wedged >= 0" and holds with nothing pinned; only a resolution that runs can clear a prefix's record, so one transient stall would refuse that prefix until restart. The leave-one-free test derives its scenario from the pool size instead of pinning a literal 2, new tests pin the shipped default, honoured in-range values, and every fail-soft arm with its warning, and a fresh-interpreter test follows the variable to the pool's max_workers so the parser cannot be disconnected from the pool unnoticed. The security spec and the paths.py docstring no longer state the pool as a fixed two workers.
629f970 to
47b20f7
Compare
buluoray
left a comment
There was a problem hiding this comment.
Verified against source at 47b20f760b3ad943895b4a139a5b388a4bb6495b, not against the description.
The knob cannot widen the ceiling. _path_resolve_worker_count (src/kiro_crew/executors.py:319-355) clamps to 2..64, and I ran the parser directly against 0, 1, -3, 65, 9999999999, "abc", "2.5", "0x10", "1e2", "", " " -- every one returns the default 2, the lower bound. Only an in-range integer widens, so a malformed or hostile value can only make the pool smaller, which refuses more rather than less.
The confidentiality decision is independent of W. _run_resolution_bounded (src/kiro_crew/security/paths.py:1556-1635) still runs every realpath off-loop under a per-call budget and raises PathResolutionStalled on any stall regardless of pool size; a larger pool only delays the saturation refusal at :1623-1628. The guard body is byte-unchanged by this PR (its only diff is the docstring at :2623-2631).
No constant became a fail-open runtime read: _MAX_PATH_RESOLVE_WORKERS is still import-time (executors.py:356), bound once at security/paths.py:50, and the env name appears in no other consumer. The keystone HIDDEN/READONLY/VISIBLE dispositions do not apply -- those scope crew-home leaf files (docs/system-specs/modules/security.md:448), and this is a process env var read before any agent runs, strictly less agent-reachable than a config leaf. No settings-registry entry was added.
The floor of 2 is load-bearing, not cosmetic: at W=1 the leave-one-free guard wedged >= W-1 degenerates to wedged >= 0 and would refuse a stalled prefix forever.
Tests: 16/16 pass. test_the_resolver_pool_knob_reaches_the_pool_in_a_fresh_interpreter is the strongest -- it asserts both the module constant and path_resolve_executor()._max_workers, which catches a parser-disconnected-from-the-pool regression the direct-parser tests would miss.
One note, not blocking: test_a_known_stalled_prefix_is_not_reprobed_onto_the_last_free_worker now derives workers from the constant, which is right, but no test sets the env, so CI still runs it at W=2 where W-1 and 1 coincide -- the leave-one-free arm has no executing coverage at W>2.
…odotdev#12460) The sensitive-path gate resolves every tool-call path on its own thread pool, sized by _MAX_PATH_RESOLVE_WORKERS = 2. The cap was justified by "healthy resolution is microseconds, so sustained queueing means the filesystem is wedged". That premise is a property of the host, not of the code: the anchor rebuild behind the 0.1s cache runs near-continuously under ordinary tool traffic, and under GIL contention one rebuild that costs 0.9ms on an idle interpreter measured 873ms with one CPU-bound sibling thread. Where both hold, the pool sits at its wedged floor and sensitive_path_refusal refuses every tool call, fail-closed, on a perfectly healthy mount. Read the size from KIROCREW_PATH_RESOLVE_WORKERS once at import. The default stays 2: on the host the premise was written for, a low ceiling is what makes the wedge signal meaningful, and every install should not pay a wider pool for a problem only some hosts have. An unparseable or out-of-range value keeps the default and logs a warning, so a typo can only ever leave the shipped behaviour in place, never widen it. The accepted range is 2..64. The floor is 2 because of the leave-one-free guard in security.paths: a prefix with a stall on record is refused before submission once W - 1 workers are pinned, which with W = 1 reads "wedged >= 0" and holds with nothing pinned; only a resolution that runs can clear a prefix's record, so one transient stall would refuse that prefix until restart. The leave-one-free test derives its scenario from the pool size instead of pinning a literal 2, new tests pin the shipped default, honoured in-range values, and every fail-soft arm with its warning, and a fresh-interpreter test follows the variable to the pool's max_workers so the parser cannot be disconnected from the pool unnoticed. The security spec and the paths.py docstring no longer state the pool as a fixed two workers.
Problem / Motivation
The sensitive-path gate resolves every string a tool call names (symlinks followed, then compared to the fenced list) on a dedicated thread pool whose size is a constant:
_MAX_PATH_RESOLVE_WORKERS = 2insrc/kiro_crew/executors.py. The comment beside the number says a healthy resolution takes microseconds, so two workers busy at once could only mean a wedged filesystem.That premise does not hold on a loaded host. When the gateway process itself is starved of CPU, a resolution that takes microseconds on an idle box misses its 2.0 s wall-clock budget (
_PATH_RESOLVE_TIMEOUT_SECS) without touching a slow filesystem. The repository already records why: the docstring of_worker_blocked_in_filesysteminsrc/kiro_crew/security/paths.py, from a measurement on a 48-core host, says that a Python thread "spends most of its wall time waiting on the GIL rather than inside a syscall". A healthy filesystem plus a CPU-hot gateway is enough to pin a worker. When that happens,_run_resolution_boundedin the same module does the following, in this order:_path_resolve_wedgedbefore the timeout is classified._worker_blocked_in_filesystem), refuses this one call, and leaves the prefix uncharged. Its own comment states the consequence: "The future stays tracked as wedged (it does hold a worker until it finishes), but the prefix is NOT charged". It logs at DEBUG only._wedged_workers() >= _MAX_PATH_RESOLVE_WORKERSholds, and the saturation arm refuses every path, from every session, without submitting anything. DEBUG only as well._wedged_workers()prunes finished futures.So on a starved host the pool size is the binding constraint: two CPU-starved resolutions refuse every tool call on the box until they finish, and the operator has no knob. The only way to change the number is to edit the source and redeploy.
Measured on the reporting host during the refusal window: the gateway process at 95.3% CPU with a 6.6 GB RSS, and
/proc/loadavgreading 15.29 / 21.61 / 51.83, so the 15-minute average shows load sustained across that window. A single resolution measured 873 ms there, not microseconds.The arm that did not fire is on record too.
_mark_stalledis the only refusal arm that logs above DEBUG (logger.warning("sensitive-path symlink resolution did not complete in ...")), and it is the only way a prefix gets charged and a cooldown opened. Across that host's gateway log from 02:27 to 07:01 it appears zero times, while the same log carries 4,618 WARNING lines from other modules, including the siblingkiro_crew.security.exfil, so the logger is wired. No prefix was ever charged, the cooldown arm did not fire, and the refusals came from the saturation arm above.Why it matters
The gate fails closed on purpose, and this PR keeps that.
_run_resolution_boundedhas seven ways to refuse (an active cooldown on the prefix, a saturated pool, an exhausted per-thread wait allowance, a resolution that never started, one that ran but was starved of CPU, one whose budget was clamped, and a real timeout blocked in the filesystem). All seven raise the samePathResolutionStalledand reach the user as one message. Exactly one of them, the saturated pool, depends on the worker count -- and on a starved host it is the one that ends up refusing everything.Since the bounded resolver landed, four follow-up changes have shaped what happens after a resolution stalls: #9711 (a worker that only lost the CPU charges no prefix), #9482 (the uncharged queue-wait refusal is logged and pinned), #10276 (one slow resolution no longer fences a whole drive), #11221 (a stalled check is worded as unverifiable, not as a match). Each made a single stall cheaper. None changed how many stalls the pool can carry at once, and that number is what decides whether the host stays usable while it is starved. A knob with an unchanged default lets the owner of such a host raise it, while every other install keeps today's behaviour exactly.
What changed (motivation -> approach -> change)
KIROCREW_PATH_RESOLVE_WORKERS0,1, negative, above 64, or not an integer keep the default and log one warning naming the variable.This PR raises the fail-closed floor and claims nothing else: with a prefix that has stall history the saturation guard refuses at
_MAX_PATH_RESOLVE_WORKERS - 1pinned workers, so at 2 a single pinned worker fails every session closed, and a larger value raises that floor (to 7 at 8) without making any resolution faster. The worker count is not a parallelism setting. A resolution that misses its budget is not cancelled: its thread stays busy until the call returns, and_wedged_workers()counts it as pinned until then. So the count is how many pinned workers the gate can carry and still probe -- stall headroom, not speed -- and that is a property of the host, which is why it becomes a knob instead of a constant.src/kiro_crew/executors.pyreads the variable through_path_resolve_worker_count()and hands the result to the existing pool;src/kiro_crew/security/paths.pykeeps reading the same constant for its saturation guard, so the guard scales with the pool.What a wider pool buys is written in that guard's condition. Without a charged prefix, two workers refuse everything once two are pinned; eight workers, once eight are. With a prefix that has a stall on record, two refuse once one is pinned; eight once seven are. On the reporting host no prefix was ever charged, so the first clause is the one that fired: two starved resolutions, and every call refused.
What this PR does not do. It does not change why the host is starved, and it does not change what is submitted to the pool: every string the gate checks is still resolved, so on a starved host resolutions still miss their budget -- it now takes more of them at once before the pool refuses everything. It does not make any resolution faster or lower its latency; under concurrent load a wider pool makes each resolution slower (measured below). It does not touch the cooldown arm or the dead-mount case: a prefix charged with a real stall stays refused for
_PATH_RESOLVE_COOLDOWN_SECS = 30.0, doubling per repeat, and a worker blocked in a dead mount stays pinned, whatever the pool size. It does not claim an end-to-end refusal count went down: the reduction is argued from the guard's condition, not from a before/after count on the host.The default stays at 2. On the host the premise was written for, a low ceiling is what makes sustained pinning a meaningful wedge signal; raising it for everyone would make ordinary contention look like a dead mount there. A wider pool has a measured cost.
path_resolve_executor()insrc/kiro_crew/executors.pyis aThreadPoolExecutor, and the hot-path work isos.path.realpathin_resolved_spellings(src/kiro_crew/security/paths.py), stdlib Python that walks path components in bytecode, so concurrent workers contend for the one interpreter lock. Measured on a 64-core host at loadavg 16.9, Python 3.12.13, 24000 identical resolutions split across workers:Threads slow the work down and processes speed it up, which places the cost in interpreter contention rather than in the filesystem (moving resolution out of process is #10256, left open). The benchmark ran every worker continuously and saturated, a harsher contention profile than the gate's occasional single resolutions, so the direction is the claim here and the exact ratio is not a statement about production. That cost is one every install would pay for a need only some hosts have, so the host that has the need opts in.
The floor is 2 because of the same guard: with one worker the with-history test reads
wedged >= 0, true with nothing pinned, and only a resolution that runs can clear a prefix's record, so one transient stall would refuse that prefix until restart. The local review lane caught this;1fails soft like any other bad value. Bad values fail soft to the default rather than failing the import, and the default is the lower ceiling, so a typo can only leave today's behaviour in place. The warning is what tells the operator the value was ignored.The security spec (
docs/system-specs/modules/security.md) and theis_sensitive_resolved_pathdocstring insecurity/paths.pystated the pool as a fixed two workers; both now say two by default and name the knob.Backwards compatibility
Compatible: with the variable unset, the pool is created with the same two workers and the same guard thresholds as main. Nothing that works today changes.
Tests
test/test_security_path_resolve_bounded.py:test_the_resolver_pool_ships_with_two_workerspins the default at a literal 2, so a later "just raise it" cannot ride in silently.test_an_in_range_resolver_pool_knob_is_honoured:2,3,8,8and64each produce that pool size.test_the_resolver_pool_knob_reaches_the_pool_in_a_fresh_interpreter: importskiro_crew.executorsin a child interpreter with the variable set to8and asserts both the module constant and the pool's_max_workersare 8, so disconnecting the parser from the pool cannot stay green.test_a_bad_resolver_pool_knob_fails_soft_to_the_default_and_says_so:0,1,-3,65,abcand2.5each give the default and exactly one warning that names the variable and the reason.test_an_empty_resolver_pool_knob_is_the_default_without_a_warning: empty and whitespace-only values are the default, silently.test_a_known_stalled_prefix_is_not_reprobed_onto_the_last_free_workernow derives its scenario from the pool size (W-1 stalls, then one more) instead of pinning a literal 2, so it keeps exercising the with-history arm when the pool is resized.Mutation-verified: disabling the range guard reddens the out-of-range cases (
0 == 2,-3 == 2,65 == 2); silencing the not-an-integer warning reddens theabcand2.5cases on the warning count; replacing_MAX_PATH_RESOLVE_WORKERS = _path_resolve_worker_count()with a literal2reddens the fresh-interpreter test (['2', '2'] == ['8', '8']). The whole file passes with-n0: 71 passed.Manual verification
Every value from the hand check on the reporting host (
0,-3,65,abc, empty -> 2;8-> 8) is now a parametrized case. The mechanism above was established by reading the arms of_run_resolution_boundedin order and by the warning count in the gateway log; the knob was not left at 8 on that host long enough to compare refusal counts before and after, so no such number is claimed.Related Issues
no linked issue: the constant was found unconfigurable while investigating refusals on one host. Why that host's gateway process is pegged is separate work; this PR carries only the knob.
Related, left open: #10256 moves the resolution out of process. This PR is independent of it -- the pool and the guard that reads its size exist either way, and the two do not touch the same hunks.
Pattern harvest
Rule candidate: review-prompt
Pattern: seven refusal arms, one exception, one user-facing message, and six of the seven log at DEBUG only. Diagnosing the reporting host took reading the arms in order and counting a warning that never appeared; an operator who raises the pool size still cannot tell from the logs whether it helped. The refusal should say which arm fired. Known and deferred: that is an observability change with its own scope, not this PR.
The pool size was one constant in a family that has never had a knob:
Two notes for whoever touches the rest. The comment above
_PATH_RESOLVE_TIMEOUT_SECSalready rules out raising it globally (it would relax the latency guarantee on the candidate path, which does not need it), so that is not the next knob. The cooldown cap is: stall #6 on one prefix is 960 seconds and #7 hits the 30-minute ceiling, while the comment's own motivating case is "ssh refused for 30 seconds during a VPN switch". That deserves its own change, not this one.Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)