Skip to content

fix(dashboard): stop taking probe workers for a mount already stuck - #21782

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/path-probe-stall-cooldown-10518
Oct 18, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/path-probe-stall-cooldown-10518

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A caller who keeps naming one dead mount stops using up the dashboard's file-probe workers once probes there are stuck, while paths on other mounts keep working.

The dashboard file routes run their blocking path work on a small bounded pool (_run_path_probe). A stat on a dead mount never returns, so each request naming that mount parks one more worker. The pool does not learn which mount is dead. A caller who keeps asking fills every worker, and then every file route answers 503 for every path, healthy ones too.

Why it matters

One unresponsive network share or FUSE mount can take the whole file surface down: file read, diff, browse, search and the editor all share the pool. The path comes from the request, so the caller picks which mount to hit.

Not a goal

  • Unsticking a thread that is already wedged. The syscall is uninterruptible; nothing in Python can take it back.
  • Keying relative paths or ~name paths. They keep the plain pool bound.
  • A burst that lands before any probe has been stuck for 2 s. Those calls are still bounded only by the pool's size and its 2 s admission wait.
  • A backoff window after the mount recovers. The refusal ends when the stuck thread returns.
  • Changing the sensitive-path resolver (security/paths.py) or its own cooldown.

What changed (motivation → approach → change)

Two stuck probes now mark their mount as stuck, so the next request for that mount is turned away before it takes a worker.

_run_path_probe (dashboard/file_api/path_probe.py, with its budgets and in-flight map in dashboard/handlers/files.py) keys each call by the stall prefix of its first argument when that is an absolute path. That is the same lexical key (_stall_prefix) the sensitive-path resolver uses for its cooldown, so deciding makes no filesystem call and no user lookup. The worker records its start time in an in-flight map while it runs. Once _PATH_PROBE_STUCK_TO_REFUSE (2) probes under one prefix have each run past _PATH_PROBE_STALL_SECS (2 s), a new call there gets the existing 503 path_probe_busy answer at once. One stuck probe refuses nothing, so a single stray link to a dead share costs one worker, as before. Paths under other prefixes still get workers.

Under a home directory the key goes one folder deeper, for this gate only: /home/<user>/mnt/nas/x keys to /home/<user>/mnt. So a dead FUSE or sshfs mount inside the home does not refuse /home/<user>/workspace. The resolver's own _stall_prefix key is unchanged.

The Goal is met for path-first routes only. Four probe-pool callers take their request path somewhere other than the first argument and stay unkeyed, bounded by the pool alone: file_api/desktop.py (_stat_then_launch closure), file_api/search.py (_probe_fallback_roots closure), file_api/pinned_io.py (_classify_request_paths, a list) and handlers/terminal.py (_resolve_cwd, config first). Relative project paths through _resolve_project_relative are unkeyed too. Keying them is deferred to #21798.

Call Under a stuck prefix Can mark a prefix stuck
Probe-pool call (validate, stat) refused, 503 yes
Transfer-pool call (read, walk) refused, 503 no: it may run long on a healthy mount
Relative path, ~name, or not a path not keyed no

The security spec (docs/system-specs/modules/security.md) states the rule beside the existing pool description.

Backwards compatibility

Compatible: no route, field or error code changes. The only new outcome is the existing 503 path_probe_busy, given before a worker is taken instead of after the 2 s admission wait.

Removes nothing: every route, key and helper stays; healthy paths under unstuck prefixes are served exactly as before.

Tests

test/test_dashboard_files_onloop_fs.py::TestWedgedPrefixRefusal:

  • A stuck prefix makes GET /api/file-read and /api/file-diff under it answer 503 path_probe_busy, a path under another prefix still runs on the free worker, and the same file reads 200 once the stuck thread returns.
  • One stuck probe refuses nothing; a second under the same prefix turns refusal on, and it clears when they return.
  • Two healthy probes under one prefix both run: being busy is not being stuck.
  • A long transfer never marks its prefix stuck.
  • A probe that raises leaves no in-flight entry.
  • Only a leading absolute path becomes a key; a relative path does not.
  • Under a home or Users home, ~/mnt/x and ~/workspace/f get different keys, and paths in one mount share a key. Two probes stuck under ~/mnt refuse that mount, while a real file in ~/workspace still reads 200.
  • ~/x is keyed from HOME alone; ~someone/x gets no key, and os.path.expanduser is never called.

Determinism: 20 runs at -n0 and 10 at -n 4 --dist loadgroup --max-worker-restart=2 beside test_files_path_probe.py and test_file_raw.py, all green; the whole file in a shuffled order (seed 2965), 52 passed. The tests wait on threading.Event and async_wait_until, with no fixed sleep.

Manual verification

N/A — unit coverage sufficient: the tests drive the real handlers on a real bounded pool with a probe that parks exactly like a dead-mount stat.

Related Issues

Closes #10518

Pattern harvest

Not generalizable: the shared pattern (a per-prefix stall key) already lives in security/paths.py; this change reuses that key rather than adding a new one.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision fec79f6ba70103b0668633487030c9c8e7b604c8 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of fec79f6ba70103b0668633487030c9c8e7b604c8 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check whether any keyed probe-pool call (validate, stat, realpath) can run past 2 s on a slow but healthy NFS workspace. If two do at once, every route under that prefix returns 503 until they finish.

[DESIGN-REVIEWED] fec79f6

Description read

[DESCRIPTION-READ] 42de02e354e814fbdb8f65b8e7f255d673b150864ce1c1ae6fb439c67931bd95

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of fec79f6ba70103b0668633487030c9c8e7b604c8 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

One thing to check before merge: the home-directory key goes one folder deeper than _stall_prefix, so this gate and the resolver's cooldown now key the same path differently.

What this change ships

Inventory (6 items) — 6 justified

Intent: stop one dead mount, named over and over by a caller, from using up every dashboard file-probe worker and taking down every file route. This is a FIX (Closes #10518).

  1. Once two probes under one mount prefix have each run past 2 s, a new request there gets 503 path_probe_busy right away, before it takes a worker — justified
  2. Transfer calls (read, walk) under a stuck prefix are refused too, but a transfer can never mark a prefix stuck — justified
  3. Under /home/<user> or Users\<user>, the key goes one folder deeper, so a dead ~/mnt does not refuse ~/workspace — justified
  4. ~ and ~/... are keyed from HOME/USERPROFILE only; ~name and relative paths get no key — justified
  5. A worker-written in-flight map plus two module constants (_PATH_PROBE_STALL_SECS, _PATH_PROBE_STUCK_TO_REFUSE) — justified
  6. The security spec now states the rule next to the existing pool description — justified

[FIRST-PRINCIPLES-REVIEWED] fec79f6

Description read

[DESCRIPTION-READ] 06c72e8356d6944b3084081afce5e7b15489207fdff279470b0037e7dee2cc37

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed fec79f6ba70103b0668633487030c9c8e7b604c8 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fec79f6

Verdict parsed from the review's SHA-scoped output markers for commit fec79f6ba70103b0668633487030c9c8e7b604c8.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fec79f6ba70103b0668633487030c9c8e7b604c8: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of fec79f6ba70103b0668633487030c9c8e7b604c8 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fec79f6

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fec79f6ba70103b0668633487030c9c8e7b604c8: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@iamwhatever
iamwhatever force-pushed the fix/path-probe-stall-cooldown-10518 branch from 26d0304 to 867233c Compare October 18, 2026 07:05
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=ac56494e9e3f fixed in 867233c

Under a home directory the probe key now goes one component deeper than _stall_prefix, for this gate only: /home/<user>/mnt/nas/x keys to /home/<user>/mnt, so two probes stuck there no longer refuse /home/<user>/workspace.
test_two_wedged_probes_inside_a_home_leave_the_workspace_served wedges two probes under <home>/mnt and reads a real file in <home>/workspace with 200, on a three-worker pool so the answer comes from the key, not spare capacity.
The resolver's _stall_prefix key is unchanged.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=3f2fcc5bfff0 fixed in 867233c

Taken as suggested: _path_probe_stall_key keys paths under a home or Users home one component deeper, and the resolver's key is not changed.
test_a_path_under_a_home_is_keyed_one_component_below_the_home pins ~/mnt/x and ~/workspace/f to different keys, one mount to one key, and the home and non-home paths to the plain key.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=90829fe77ef0 fixed in 867233c

Taken via the second clear condition: the PR description now names the four probe-pool callers that take their path elsewhere (file_api/desktop.py, file_api/search.py, file_api/pinned_io.py, handlers/terminal.py) as unkeyed and says the fix covers path-first callers only.
Those callers keep the plain pool bound they had before this PR; keying them means changing four call signatures, which is new scope for this fix.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=f9ca2c4284fc fixed in 867233c

The comment above _PATH_PROBE_STALL_SECS in handlers/files.py now says _PATH_PROBE_STUCK_TO_REFUSE probes must each be past the budget, matching _path_probe_prefix_wedged.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=d18d6cf889e6 fixed in 867233c

A dead mount inside the home no longer refuses the whole home: under a home directory the key is one component deeper (/home/alice/mnt/nas -> /home/alice/mnt).
test_two_wedged_probes_inside_a_home_leave_the_workspace_served proves <home>/workspace still reads 200 while two probes are stuck under <home>/mnt.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Taken via the second clear condition: the description now says the Goal is met for path-first routes only, and names the four unkeyed callers plus relative project paths.
Keying them needs an explicit key argument on _run_path_probe and a change at each call site, which is new scope for this fix; #21798 carries that task with the deferred-finding label, an owner and a Due date.
Those callers keep exactly the pool bound they have on main, so this PR makes none of them worse.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • design PASS punchline answered: relative project paths

Checked: relative paths reach _resolve_project_relative and get no key on purpose, because abspath would charge the gateway's working directory instead of the project mount the probe touches.
They keep the plain pool bound they have on main. Keying them from the resolved project directory is part of #21798.

@iamwhatever
iamwhatever force-pushed the fix/path-probe-stall-cooldown-10518 branch from 867233c to fec79f6 Compare October 18, 2026 07:32
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main 943a4e5, which carries the red-main fix #21792 (test_history_bindings.py::test_the_member_set_is_the_frozen_inventory, red on clean main since #21108). Same patch, new head fec79f6; CI re-running. The Windows shard's test_sweep_fails_by_name_when_the_handle_is_never_released red touches no file in this PR and is recorded as flaky in #21802.

@iamwhatever

iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor Author
  • design PASS punchline answered: slow but healthy NFS

Accepted as the trade: refusal needs two keyed probes past 2 s under one key at the same time, and it clears the moment either returns, so a slow-but-healthy mount gets short 503s, not a lockout.
A validate or stat is milliseconds when healthy; 2 s matches the sensitive-path resolver's own stall budget.
Without this gate, the same burst fills the 8-worker pool and every path, on every mount, answers 503 after the 2 s admission wait.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • first-principles PASS punchline answered: two keys for one path

Intended: this gate counts threads still parked in _path_probe_inflight; the resolver's cooldown is a time window in _path_resolve_degraded. They share no state, so a path keyed differently by each cannot make either refuse wrongly.
The gate's key is the resolver's key plus at most one component under a home, so it is never wider and can only refuse less; the resolver's _stall_prefix is unchanged.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 18, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Review-ready at head fec79f6: PR Readiness passed, 106 checks green, mergeable, and the green is fresh against main. GPT and Opus are stamped on this head with no findings; Design and First Principles pass, and each one's ask is answered above; UX skipped (no user-facing surface). The four callers that do not pass their path first are deferred to #21798, and the Windows flake seen on an earlier head is recorded in #21802. Waiting on a maintainer's review. This Crew will not merge, approve, or enable auto-merge.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 18, 2026 09:06

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the request-path probe pool refuses new calls under a mount prefix once two probes there are stuck, so one dead mount cannot take every worker; tests added. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/security.md.

@KiroCrewCaptain
KiroCrewCaptain merged commit 2efc351 into main Oct 18, 2026
107 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/path-probe-stall-cooldown-10518 branch October 18, 2026 09:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bound the executor used by dashboard path probes

2 participants