Skip to content

fix(knowledge): run request-path filesystem checks off the event loop - #21791

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/10517-offload-request-path-fs
Oct 19, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/10517-offload-request-path-fs

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Adding, retrying or skipping a knowledge source on a path whose mount stopped answering no longer freezes the whole gateway.

The knowledge handlers add_source, retry_file and skip_file take a path from the request and call Path.resolve(), is_file() and is_dir() on it inside the coroutine. The folder connector's validate_config calls exists() and is_dir() there too, and _ingest_local_file_task re-resolves the stored path the same way. These are the five sites counted in #10517 part 2.

Why it matters

The gateway runs one event loop. One of these calls on a hung NFS or FUSE mount blocks every endpoint at once: dashboard, tunnel, MCP and crons. A stall past dashboard.loop_stall_exit_after_secs makes the loop watchdog kill the process, and every in-flight turn drops. The path is the caller's choice, so the caller picks which mount hangs the gateway.

Not a goal

What changed (motivation → approach → change)

A path from a request was resolved and stat'ed on the event loop, so a hung mount froze the gateway.

The fix reuses the bounded request-path probe pool files._run_path_probe that files.py already uses. Each request makes one trip to that pool. A full pool answers the same coded 503 as the file endpoints, via _probe_busy_response.

handler what runs off the loop when the pool is full
add_source connector validate_config, then one resolve feeding both the sensitive check and the file/dir type check coded 503
retry_file, skip_file raw and resolved sensitive check coded 503
_ingest_local_file_task resolved sensitive re-check status stays pending, so the sweep retries it

In add_source, one resolve feeds the sensitive check, the type check and the stored path. So no symlink swap can split what was checked from what is stored, and no await sits between the check and the use.

Two uncoded 403 answers in add_source repeated the general sensitive guard that runs before them, so they are gone. error-code-baseline.json is regenerated: knowledge.py missing_code 60 -> 58. Its _compliant total moves 2675 -> 2767 because main drifted; the generator prints 2767 on clean main too.

Backwards compatibility

Compatible: every request that succeeds today gets the same answer. The only new answer is the coded 503 when the probe pool is full, which today would have blocked the loop instead. For a sensitive local_file path, the general guard already answered first on main, so its 403 body is unchanged.

Removes nothing: no route, key, field or record is removed or narrowed.

Tests

test/test_knowledge_onloop_fs.py, 12 tests, mirroring test/test_dashboard_files_onloop_fs.py:

  • A static AST ratchet: no async def in knowledge.py calls a filesystem method on the loop. A paired test proves the ratchet can fail.
  • Per endpoint, the thread each filesystem call actually ran on is the probe pool, never the loop thread.
  • A full pool gives a coded 503 for add_source, retry_file and skip_file, and pending (never error) for the ingest task.
  • A missing folder still returns the connector's own 400 error.

Determinism: 20 repeated runs of the file, 20/20 green; one shuffled node order (seed 7), 12/12 green. prove.py --base origin/main: PROVEN (an assertion fails with the fix reverted).

Manual verification

N/A: unit coverage sufficient. The tests record the real thread of each call through aiohttp's test server, which is the property a manual hung-mount check would show.

Screenshots / video

N/A: backend only, no user-visible UI change.

Related Issues

Part of #10517

Pattern harvest

Rule candidate: lint
Pattern: "unbounded filesystem syscall (Path.resolve, is_file, is_dir, exists, os.stat) on a request-supplied path inside an async def". This is #10517 part 1, a filesystem family for scripts/check_sync_io_in_async.py.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 0ade343d72f6b8948aa8a52bd1a421f7ad4718b2 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 0ade343d72f6b8948aa8a52bd1a421f7ad4718b2 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that _compliant 2675 → 2767 in error-code-baseline.json comes from base drift, not a miscount; the description only mentions missing_code 60 → 58.

Not justified as shipped

  • Item 5 — undeclared: the regenerated baseline also raises "_compliant": 2767 (+92), which this diff's two deleted 403s can't account for.

What this change ships

Inventory (6 items) — 5 justified

Intent: a hung mount under a caller-chosen knowledge path should no longer freeze the whole gateway. FIX.

  1. Adding a knowledge source no longer runs its path resolve, type check or connector validate_config on the event loop. They use the existing files._run_path_probe pool. — justified
  2. Retry-file and skip-file run their sensitive-path check off the loop. — justified
  3. The ingest-time sensitive re-check runs off the loop. When the pool is full, the file stays pending for the sweep instead of being marked error. — justified
  4. A full probe pool now answers the existing coded 503 (_probe_busy_response) on these three endpoints. — justified
  5. The error-code baseline is regenerated, and _compliant grows by 92 without explanation. — undeclared
  6. A module-local AST ratchet plus a thread-proof test file for knowledge.py. — justified

[FIRST-PRINCIPLES-REVIEWED] 0ade343

Description read

[DESCRIPTION-READ] 0ac2dbd3d50e8347a82b2d65e1266e5da23cb85cb8a45ea8830902bafce6b398

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 0ade343d72f6b8948aa8a52bd1a421f7ad4718b2 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 0ade343

Description read

[DESCRIPTION-READ] a6a397176701e1ce3aae8ec5d6b8f258914ee89c4f6a59176e0e76f4a98a0278

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 0ade343d72f6b8948aa8a52bd1a421f7ad4718b2 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0ade343

Verdict parsed from the review's SHA-scoped output markers for commit 0ade343d72f6b8948aa8a52bd1a421f7ad4718b2.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 0ade343d72f6b8948aa8a52bd1a421f7ad4718b2: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 0ade343d72f6b8948aa8a52bd1a421f7ad4718b2 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0ade343

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0ade343d72f6b8948aa8a52bd1a421f7ad4718b2: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the fix/10517-offload-request-path-fs branch from 7fe9a72 to 483908b Compare October 18, 2026 07:05
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Every connector's validate_config is pure sync with no loop-bound state: rebutted (verified, no change)

Only two definitions exist in src/: the abstract BaseConnector.validate_config (connectors/base.py:18) and LocalFolderConnector.validate_config (connectors/local_folder.py:19).
The one implementation is plain sync code: Path(uri).exists() and is_dir() on the config uri, returning a tuple. No event loop, lock, aiohttp object or app state is touched.
Those two stat calls are the reason it moved to the probe pool: they hit the caller-chosen path, the same hang this PR removes from the loop.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Regenerated baseline _compliant 2675 -> 2767 is base drift: rebutted (verified, no change)

Running python test/test_error_code_contract.py --update on a clean origin/main (9d088c8) prints compliant: 2767 and missing_code: 1046.
On this head it prints compliant: 2767 and missing_code: 1044. So +92 compliant is main's drift, already in the tree; this PR's own delta is knowledge.py missing_code 60 -> 58.
The file is generated output, so the PR keeps the generator's whole snapshot rather than hand-editing one number.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@iamwhatever
iamwhatever force-pushed the fix/10517-offload-request-path-fs branch from 483908b to 0ade343 Compare October 18, 2026 07:32
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main 943a4e5 (9d088c8 -> 943a4e5) to pick up #21792. Shard 1 (Linux and Windows) failed on test/test_history_bindings.py::test_the_member_set_is_the_frozen_inventory, which listed three history members that #21108 added on main (_dir_write_epoch, _dir_write_epochs, warm_search). This PR touches none of those files; the test passes on clean main 943a4e5, and #21792 is its fix. The patch is unchanged (same patch-id); 1350 related tests pass locally on 0ade343. CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@iamwhatever

iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=9de68687f86d _compliant 2675 -> 2767 in error-code-baseline.json: rebutted (base drift); body now declares it

python test/test_error_code_contract.py --update on clean origin/main 9d088c8 printed compliant: 2767, missing_code: 1046; this head prints compliant: 2767, missing_code: 1044.
So +92 compliant is already in main's tree, and this PR's own delta is only knowledge.py missing_code 60 -> 58 (the two removed 403s).
The file is generator output, so the PR keeps the whole snapshot instead of hand-editing one total. The description's What changed now states the drift in one sentence.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Review-ready at 0ade343, waiting on a maintainer review.

No merge, approval or auto-merge from this Crew; the merge is the maintainer's call.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every review lane on the head is clean (only runner notices and PR Scope size warnings); held by the triage cron on a path, category, checklist or P0-area rule, not a finding. Approved on the watch holder's request.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 19, 2026 01:17
@KiroCrewCaptain
KiroCrewCaptain merged commit a5b615b into main Oct 19, 2026
162 of 163 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/10517-offload-request-path-fs branch October 19, 2026 01:48
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants