Skip to content

fix(file-search): bound the walk by wall-clock, slow-store escape hatch (#11419) - #17475

Closed
chenmingwei23 wants to merge 2 commits into
kirodotdev:mainfrom
chenmingwei23:fix/11419-file-search-walk-budget
Closed

chenmingwei23 wants to merge 2 commits into
kirodotdev:mainfrom
chenmingwei23:fix/11419-file-search-walk-budget

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

What broke

The @-mention file search is bounded by a client-side 15s deadline
(FILE_SEARCH_TIMEOUT_MS in website/src/api/client/files.ts, added in #7068).
That bound was calibrated against local disk only: a ceiling-reaching walk
measured 4.20s worst, giving ~3.6x headroom under 15s. The comment conceded no
network mount was available to sample.

The server walk (_walk_file_search in file_api/search.py) is bounded only by
count ceilings — _WALK_MAX_SCAN_SCOPED (50k entries) and
_WALK_MAX_DIRS_VISITED (20k dirs). On an NFS/SSHFS-class mount each
os.stat/scandir is a network round trip, so the same bounded number of
calls
can take many times longer and blow straight past the client's 15s. The
client never retries a deadline it set, and the user's Retry/Refresh re-enters
the same bound — so a healthy-but-slow store fails every attempt. (#11419, the
deferral recorded on #7068.)

Measurement

Worker host has no network mount, so this was measured with a simulated slow
filesystem
in a unit test (test/test_file_search_dirs.py →
TestFileSearchWalkDeadline): a thin os.walk wrapper that sleeps per yielded
directory, standing in for a slow mount's per-syscall latency. With a small tree
that trips neither count ceiling, the walk runs well past the client bound —
the local-disk headroom does not hold on a slow store.

What changed

The escape hatch is a server-side wall-clock budget on the walk itself, so
the walk cannot outrun its own bound regardless of store speed and the client's
15s keeps its margin:

  • _walk_file_search now carries a monotonic deadline. It stops descending at
    the budget and returns a partial result marked truncated — the same
    observable outcome the entry/dir count ceilings already produce, now reached by
    elapsed time too. A per-entry stride (_WALK_DEADLINE_STRIDE, mirroring the
    grep path's _GREP_ROW_DEADLINE_STRIDE) keeps a single enormous directory from
    overrunning the budget between per-directory checks, without paying a clock read
    per entry.
  • Default budget 10s, leaving room under the client's 15s for admission and
    round trips.
  • Slow-store escape hatch: an operator on a store slow enough that 10s
    truncates too eagerly can raise it with the
    KIROCREW_FILE_SEARCH_WALK_BUDGET_MS env var (milliseconds). Read per request
    so a change needs no restart; a missing/empty/non-numeric/non-positive value
    falls back to the default. This follows the repo's existing KIROCREW_*
    env-override convention; a knob (not a hard-coded bump) because the right value
    is store-specific and only the rare slow-mount operator needs to know it.
  • /api/file-search responses now include truncated (optional on the client
    type; a missing field reads as false, so this is backward compatible — the
    walk already truncated silently on the count ceilings).

Smallest option that matches existing patterns: it reuses the walk's own
early-stop + truncated mechanism and the KIROCREW_* override convention,
rather than adding a per-surface knob or a new client contract. The alternative
(a client deadline knob) was rejected: the BROWSE_FILES_TIMEOUT_MS comment
already argues a client-side number is the wrong thing to ask a user to pick, and
a client knob would not stop the walk from pinning a worker on a slow mount.

What was tested

test/test_file_search_dirs.py::TestFileSearchWalkDeadline (4 new tests, all
green):

  • a simulated slow store truncates the walk at the budget and still returns the
    partial matches it collected (the measurement);
  • a fast store / production-class budget is not truncated (control);
  • raising KIROCREW_FILE_SEARCH_WALK_BUDGET_MS lets the same slow store finish
    (the escape hatch);
  • the env-override parsing: positive ms honoured, missing/empty/non-numeric/
    non-positive fall back to 10s.

Full regression: test_file_search.py, test_file_search_dirs.py,
test_dashboard_files_onloop_fs.py, test_dashboard_files_composition_contract.py
— 173 passed. flake8, isort --check, mypy clean on the changed modules. The
composition-contract test was updated to register the new facade helper
(_file_search_walk_budget_secs).

The frontend change is a comment update plus one optional field on a result type
(additive, no caller forced to consume it); the website's node_modules is not
installed on this host, so tsc was not run locally — CI builds the SPA.

Env knob reaches the process (reviewer check)

Confirmed in response to the #17414 finding (its FE_EXTRA_ROOTS was rejected
because minimal_env strips it and nothing reads it). That failure mode is a
subprocess env-scrub boundary: apps/registry_pipeline/subprocess_env.minimal_env
builds a scrubbed env dict for spawned subprocesses (registry installs, app
backends, git clones). This knob is not on that kind of path.

api_file_search is an in-process aiohttp handler, and _walk_file_search runs
on the bounded probe pool (_run_path_probe -> threads in the same process,
explicitly not a subprocess). So the walk reads the gateway's own live
os.environ; minimal_env is nowhere on this path and cannot strip the knob.

Pinned end-to-end by test_the_env_knob_reaches_the_thread_that_runs_the_walk:
it sets KIROCREW_FILE_SEARCH_WALK_BUDGET_MS in os.environ, drives the real
endpoint, and asserts the value the running search used equals what was set --
so a future change that moved the walk behind an env-scrubbing boundary would
fail here instead of silently ignoring the operator. The knob is therefore kept.

Fixes #11419

The @-mention file-search walk was bounded only by entry/dir count
ceilings and a client-side 15s deadline calibrated on local disk. On a
network mount each stat is a round trip, so the same bounded number of
calls can outrun the client bound and its Retry re-enters it, failing
every attempt.

Measured with a simulated slow filesystem (unit test): the local-disk
headroom does not hold on a slow store. Fix: give _walk_file_search its
own monotonic budget (default 10s, raise with
KIROCREW_FILE_SEARCH_WALK_BUDGET_MS) that stops the walk early and marks
the result truncated -- the same signal the count ceilings already raise.
So the walk cannot outrun its own bound regardless of store speed.

Fixes kirodotdev#11419
@chenmingwei23
chenmingwei23 requested a review from a team as a code owner October 6, 2026 18:19
Reviewer check (cf. kirodotdev#17414, whose FE_EXTRA_ROOTS was rejected because
minimal_env strips it): confirm KIROCREW_FILE_SEARCH_WALK_BUDGET_MS is
read in-process, not through a subprocess env scrub.

The file search is an in-process aiohttp handler and the walk runs on
the bounded probe pool (threads, same process), so it reads the gateway's
own live os.environ; minimal_env (which only scrubs spawned-subprocess
env) is nowhere on this path. New end-to-end test
test_the_env_knob_reaches_the_thread_that_runs_the_walk pins it, and the
budget docstring records why the knob is safe here. The knob is kept.
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 6, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Superseded by #17946, which keeps only the server-side wall-clock bound on the file-search fallback walk and drops the configurable slow-store escape hatch this PR also carried. The replacement reproduces the issue on main first (a slow store drives /api/file-search past the client deadline with no server bound) and adds the fixed 10s budget plus the truncated signal. Closing this in favour of #17946.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Measure file-search deadline headroom on network mounts; add a slow-store escape hatch if it does not hold

1 participant