Repository navigation
fix(file-search): bound the walk by wall-clock, slow-store escape hatch (#11419) - #17475
Closed
chenmingwei23 wants to merge 2 commits into
Closed
chenmingwei23 wants to merge 2 commits into
chenmingwei23 wants to merge 2 commits into
Conversation
The @-mention file-search walk was bounded only by entry/dir count ceilings and a client-side 15s deadline calibrated on local disk. On a network mount each stat is a round trip, so the same bounded number of calls can outrun the client bound and its Retry re-enters it, failing every attempt. Measured with a simulated slow filesystem (unit test): the local-disk headroom does not hold on a slow store. Fix: give _walk_file_search its own monotonic budget (default 10s, raise with KIROCREW_FILE_SEARCH_WALK_BUDGET_MS) that stops the walk early and marks the result truncated -- the same signal the count ceilings already raise. So the walk cannot outrun its own bound regardless of store speed. Fixes kirodotdev#11419
Reviewer check (cf. kirodotdev#17414, whose FE_EXTRA_ROOTS was rejected because minimal_env strips it): confirm KIROCREW_FILE_SEARCH_WALK_BUDGET_MS is read in-process, not through a subprocess env scrub. The file search is an in-process aiohttp handler and the walk runs on the bounded probe pool (threads, same process), so it reads the gateway's own live os.environ; minimal_env (which only scrubs spawned-subprocess env) is nowhere on this path. New end-to-end test test_the_env_knob_reaches_the_thread_that_runs_the_walk pins it, and the budget docstring records why the knob is safe here. The knob is kept.
5 tasks done
Contributor
Author
|
Superseded by #17946, which keeps only the server-side wall-clock bound on the file-search fallback walk and drops the configurable slow-store escape hatch this PR also carried. The replacement reproduces the issue on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
The
@-mention file search is bounded by a client-side 15s deadline(
FILE_SEARCH_TIMEOUT_MSinwebsite/src/api/client/files.ts, added in #7068).That bound was calibrated against local disk only: a ceiling-reaching walk
measured 4.20s worst, giving ~3.6x headroom under 15s. The comment conceded no
network mount was available to sample.
The server walk (
_walk_file_searchinfile_api/search.py) is bounded only bycount ceilings —
_WALK_MAX_SCAN_SCOPED(50k entries) and_WALK_MAX_DIRS_VISITED(20k dirs). On an NFS/SSHFS-class mount eachos.stat/scandiris a network round trip, so the same bounded number ofcalls can take many times longer and blow straight past the client's 15s. The
client never retries a deadline it set, and the user's Retry/Refresh re-enters
the same bound — so a healthy-but-slow store fails every attempt. (#11419, the
deferral recorded on #7068.)
Measurement
Worker host has no network mount, so this was measured with a simulated slow
filesystem in a unit test (
test/test_file_search_dirs.py→TestFileSearchWalkDeadline): a thinos.walkwrapper that sleeps per yieldeddirectory, standing in for a slow mount's per-syscall latency. With a small tree
that trips neither count ceiling, the walk runs well past the client bound —
the local-disk headroom does not hold on a slow store.
What changed
The escape hatch is a server-side wall-clock budget on the walk itself, so
the walk cannot outrun its own bound regardless of store speed and the client's
15s keeps its margin:
_walk_file_searchnow carries a monotonic deadline. It stops descending atthe budget and returns a partial result marked
truncated— the sameobservable outcome the entry/dir count ceilings already produce, now reached by
elapsed time too. A per-entry stride (
_WALK_DEADLINE_STRIDE, mirroring thegrep path's
_GREP_ROW_DEADLINE_STRIDE) keeps a single enormous directory fromoverrunning the budget between per-directory checks, without paying a clock read
per entry.
round trips.
truncates too eagerly can raise it with the
KIROCREW_FILE_SEARCH_WALK_BUDGET_MSenv var (milliseconds). Read per requestso a change needs no restart; a missing/empty/non-numeric/non-positive value
falls back to the default. This follows the repo's existing
KIROCREW_*env-override convention; a knob (not a hard-coded bump) because the right value
is store-specific and only the rare slow-mount operator needs to know it.
/api/file-searchresponses now includetruncated(optional on the clienttype; a missing field reads as
false, so this is backward compatible — thewalk already truncated silently on the count ceilings).
Smallest option that matches existing patterns: it reuses the walk's own
early-stop +
truncatedmechanism and theKIROCREW_*override convention,rather than adding a per-surface knob or a new client contract. The alternative
(a client deadline knob) was rejected: the
BROWSE_FILES_TIMEOUT_MScommentalready argues a client-side number is the wrong thing to ask a user to pick, and
a client knob would not stop the walk from pinning a worker on a slow mount.
What was tested
test/test_file_search_dirs.py::TestFileSearchWalkDeadline(4 new tests, allgreen):
partial matches it collected (the measurement);
KIROCREW_FILE_SEARCH_WALK_BUDGET_MSlets the same slow store finish(the escape hatch);
non-positive fall back to 10s.
Full regression:
test_file_search.py,test_file_search_dirs.py,test_dashboard_files_onloop_fs.py,test_dashboard_files_composition_contract.py— 173 passed.
flake8,isort --check,mypyclean on the changed modules. Thecomposition-contract test was updated to register the new facade helper
(
_file_search_walk_budget_secs).The frontend change is a comment update plus one optional field on a result type
(additive, no caller forced to consume it); the website's
node_modulesis notinstalled on this host, so
tscwas not run locally — CI builds the SPA.Env knob reaches the process (reviewer check)
Confirmed in response to the #17414 finding (its
FE_EXTRA_ROOTSwas rejectedbecause
minimal_envstrips it and nothing reads it). That failure mode is asubprocess env-scrub boundary:
apps/registry_pipeline/subprocess_env.minimal_envbuilds a scrubbed env dict for spawned subprocesses (registry installs, app
backends, git clones). This knob is not on that kind of path.
api_file_searchis an in-process aiohttp handler, and_walk_file_searchrunson the bounded probe pool (
_run_path_probe-> threads in the same process,explicitly not a subprocess). So the walk reads the gateway's own live
os.environ;minimal_envis nowhere on this path and cannot strip the knob.Pinned end-to-end by
test_the_env_knob_reaches_the_thread_that_runs_the_walk:it sets
KIROCREW_FILE_SEARCH_WALK_BUDGET_MSinos.environ, drives the realendpoint, and asserts the value the running search used equals what was set --
so a future change that moved the walk behind an env-scrubbing boundary would
fail here instead of silently ignoring the operator. The knob is therefore kept.
Fixes #11419