Repository navigation
fix(dashboard): stop taking probe workers for a mount already stuck - #21782
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check whether any keyed probe-pool call (validate, stat, realpath) can run past 2 s on a slow but healthy NFS workspace. If two do at once, every route under that prefix returns 503 until they finish. [DESIGN-REVIEWED] fec79f6 Description read[DESCRIPTION-READ] 42de02e354e814fbdb8f65b8e7f255d673b150864ce1c1ae6fb439c67931bd95 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS One thing to check before merge: the home-directory key goes one folder deeper than What this change shipsInventory (6 items) — 6 justifiedIntent: stop one dead mount, named over and over by a caller, from using up every dashboard file-probe worker and taking down every file route. This is a FIX (Closes #10518).
[FIRST-PRINCIPLES-REVIEWED] fec79f6 Description read[DESCRIPTION-READ] 06c72e8356d6944b3084081afce5e7b15489207fdff279470b0037e7dee2cc37 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
26d0304 to
867233c
Compare
|
|
|
|
|
|
|
867233c to
fec79f6
Compare
|
Rebased onto main 943a4e5, which carries the red-main fix #21792 ( |
|
|
|
Review-ready at head fec79f6: PR Readiness passed, 106 checks green, mergeable, and the green is fresh against main. GPT and Opus are stamped on this head with no findings; Design and First Principles pass, and each one's ask is answered above; UX skipped (no user-facing surface). The four callers that do not pass their path first are deferred to #21798, and the Windows flake seen on an earlier head is recorded in #21802. Waiting on a maintainer's review. This Crew will not merge, approve, or enable auto-merge. |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the request-path probe pool refuses new calls under a mount prefix once two probes there are stuck, so one dead mount cannot take every worker; tests added. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/security.md.
Problem / Motivation
Goal: A caller who keeps naming one dead mount stops using up the dashboard's file-probe workers once probes there are stuck, while paths on other mounts keep working.
The dashboard file routes run their blocking path work on a small bounded pool (
_run_path_probe). Astaton a dead mount never returns, so each request naming that mount parks one more worker. The pool does not learn which mount is dead. A caller who keeps asking fills every worker, and then every file route answers 503 for every path, healthy ones too.Why it matters
One unresponsive network share or FUSE mount can take the whole file surface down: file read, diff, browse, search and the editor all share the pool. The path comes from the request, so the caller picks which mount to hit.
Not a goal
~namepaths. They keep the plain pool bound.security/paths.py) or its own cooldown.What changed (motivation → approach → change)
Two stuck probes now mark their mount as stuck, so the next request for that mount is turned away before it takes a worker.
_run_path_probe(dashboard/file_api/path_probe.py, with its budgets and in-flight map indashboard/handlers/files.py) keys each call by the stall prefix of its first argument when that is an absolute path. That is the same lexical key (_stall_prefix) the sensitive-path resolver uses for its cooldown, so deciding makes no filesystem call and no user lookup. The worker records its start time in an in-flight map while it runs. Once_PATH_PROBE_STUCK_TO_REFUSE(2) probes under one prefix have each run past_PATH_PROBE_STALL_SECS(2 s), a new call there gets the existing503 path_probe_busyanswer at once. One stuck probe refuses nothing, so a single stray link to a dead share costs one worker, as before. Paths under other prefixes still get workers.Under a home directory the key goes one folder deeper, for this gate only:
/home/<user>/mnt/nas/xkeys to/home/<user>/mnt. So a dead FUSE or sshfs mount inside the home does not refuse/home/<user>/workspace. The resolver's own_stall_prefixkey is unchanged.The Goal is met for path-first routes only. Four probe-pool callers take their request path somewhere other than the first argument and stay unkeyed, bounded by the pool alone:
file_api/desktop.py(_stat_then_launchclosure),file_api/search.py(_probe_fallback_rootsclosure),file_api/pinned_io.py(_classify_request_paths, a list) andhandlers/terminal.py(_resolve_cwd, config first). Relative project paths through_resolve_project_relativeare unkeyed too. Keying them is deferred to #21798.~name, or not a pathThe security spec (
docs/system-specs/modules/security.md) states the rule beside the existing pool description.Backwards compatibility
Compatible: no route, field or error code changes. The only new outcome is the existing
503 path_probe_busy, given before a worker is taken instead of after the 2 s admission wait.Removes nothing: every route, key and helper stays; healthy paths under unstuck prefixes are served exactly as before.
Tests
test/test_dashboard_files_onloop_fs.py::TestWedgedPrefixRefusal:GET /api/file-readand/api/file-diffunder it answer503 path_probe_busy, a path under another prefix still runs on the free worker, and the same file reads200once the stuck thread returns.homeorUsershome,~/mnt/xand~/workspace/fget different keys, and paths in one mount share a key. Two probes stuck under~/mntrefuse that mount, while a real file in~/workspacestill reads200.~/xis keyed fromHOMEalone;~someone/xgets no key, andos.path.expanduseris never called.Determinism: 20 runs at
-n0and 10 at-n 4 --dist loadgroup --max-worker-restart=2besidetest_files_path_probe.pyandtest_file_raw.py, all green; the whole file in a shuffled order (seed 2965), 52 passed. The tests wait onthreading.Eventandasync_wait_until, with no fixed sleep.Manual verification
N/A — unit coverage sufficient: the tests drive the real handlers on a real bounded pool with a probe that parks exactly like a dead-mount
stat.Related Issues
Closes #10518
Pattern harvest
Not generalizable: the shared pattern (a per-prefix stall key) already lives in
security/paths.py; this change reuses that key rather than adding a new one.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)