Repository navigation
fix(knowledge): run request-path filesystem checks off the event loop - #21791
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that Not justified as shipped
What this change shipsInventory (6 items) — 5 justifiedIntent: a hung mount under a caller-chosen knowledge path should no longer freeze the whole gateway. FIX.
[FIRST-PRINCIPLES-REVIEWED] 0ade343 Description read[DESCRIPTION-READ] 0ac2dbd3d50e8347a82b2d65e1266e5da23cb85cb8a45ea8830902bafce6b398 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 0ade343 Description read[DESCRIPTION-READ] a6a397176701e1ce3aae8ec5d6b8f258914ee89c4f6a59176e0e76f4a98a0278 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
7fe9a72 to
483908b
Compare
|
|
483908b to
0ade343
Compare
|
Rebased onto main 943a4e5 (9d088c8 -> 943a4e5) to pick up #21792. Shard 1 (Linux and Windows) failed on |
|
|
Review-ready at 0ade343, waiting on a maintainer review.
No merge, approval or auto-merge from this Crew; the merge is the maintainer's call. |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Every review lane on the head is clean (only runner notices and PR Scope size warnings); held by the triage cron on a path, category, checklist or P0-area rule, not a finding. Approved on the watch holder's request.
Problem / Motivation
Goal: Adding, retrying or skipping a knowledge source on a path whose mount stopped answering no longer freezes the whole gateway.
The knowledge handlers
add_source,retry_fileandskip_filetake a path from the request and callPath.resolve(),is_file()andis_dir()on it inside the coroutine. The folder connector'svalidate_configcallsexists()andis_dir()there too, and_ingest_local_file_taskre-resolves the stored path the same way. These are the five sites counted in #10517 part 2.Why it matters
The gateway runs one event loop. One of these calls on a hung NFS or FUSE mount blocks every endpoint at once: dashboard, tunnel, MCP and crons. A stall past
dashboard.loop_stall_exit_after_secsmakes the loop watchdog kill the process, and every in-flight turn drops. The path is the caller's choice, so the caller picks which mount hangs the gateway.Not a goal
scripts/check_sync_io_in_async.py. That gate needs its own baseline; this PR adds a module-local ratchet forknowledge.pyonly.terminal.py?cwd=): main already runs the request-supplied cwd through_run_path_probe(since feat: run bang commands directly in the session workspace #14008). The remainingos.path.isdirthere checks the configured cwd, not a request value.What changed (motivation → approach → change)
A path from a request was resolved and stat'ed on the event loop, so a hung mount froze the gateway.
The fix reuses the bounded request-path probe pool
files._run_path_probethatfiles.pyalready uses. Each request makes one trip to that pool. A full pool answers the same coded 503 as the file endpoints, via_probe_busy_response.add_sourcevalidate_config, then one resolve feeding both the sensitive check and the file/dir type checkretry_file,skip_file_ingest_local_file_taskpending, so the sweep retries itIn
add_source, one resolve feeds the sensitive check, the type check and the stored path. So no symlink swap can split what was checked from what is stored, and noawaitsits between the check and the use.Two uncoded 403 answers in
add_sourcerepeated the general sensitive guard that runs before them, so they are gone.error-code-baseline.jsonis regenerated:knowledge.pymissing_code60 -> 58. Its_complianttotal moves 2675 -> 2767 because main drifted; the generator prints 2767 on clean main too.Backwards compatibility
Compatible: every request that succeeds today gets the same answer. The only new answer is the coded 503 when the probe pool is full, which today would have blocked the loop instead. For a sensitive
local_filepath, the general guard already answered first on main, so its 403 body is unchanged.Removes nothing: no route, key, field or record is removed or narrowed.
Tests
test/test_knowledge_onloop_fs.py, 12 tests, mirroringtest/test_dashboard_files_onloop_fs.py:async definknowledge.pycalls a filesystem method on the loop. A paired test proves the ratchet can fail.add_source,retry_fileandskip_file, andpending(nevererror) for the ingest task.Determinism: 20 repeated runs of the file, 20/20 green; one shuffled node order (seed 7), 12/12 green.
prove.py --base origin/main: PROVEN (an assertion fails with the fix reverted).Manual verification
N/A: unit coverage sufficient. The tests record the real thread of each call through aiohttp's test server, which is the property a manual hung-mount check would show.
Screenshots / video
N/A: backend only, no user-visible UI change.
Related Issues
Part of #10517
Pattern harvest
Rule candidate: lint
Pattern: "unbounded filesystem syscall (
Path.resolve,is_file,is_dir,exists,os.stat) on a request-supplied path inside anasync def". This is #10517 part 1, a filesystem family forscripts/check_sync_io_in_async.py.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)