Repository navigation
fix(memory): check a custom model path off the event loop - #22023
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] afd6194 Description read[DESCRIPTION-READ] a7edb84d4fbeb3b26f9be5ec4cbd2546632c8b4ba2a97e766c13dfa5f638fcfb That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing to check. What this change shipsInventory (3 items) — 3 justifiedIntent: stop a custom embedding-model path on a hung mount from freezing the whole gateway. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] afd6194 Description read[DESCRIPTION-READ] 4f015755a23875f4faa4a753f20c9aae05b3523a3bdee60ea2816ad0d57cff2d That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
|
6874aad to
afd6194
Compare
|
Pushed afd6194: fixes the CI red on |
bolichen97
left a comment
There was a problem hiding this comment.
Good to ship. Reviewed head afd6194.
What I checked:
_probe_model_pathrunsvalidate_custom_model_pathand the size stat in one hop on_run_path_probe. No filesystem call on the request path stays on the event loop.- The busy path uses
_probe_busy_response, the same coded 503 thatfiles.pyandknowledge.pyreturn. The probe runs before any config write, so a 503 changes no state. - Success and 400 answers are unchanged, including
size_bytesand0for an unstatable path. - The worker re-check in
_apply_embedding_modelstill runs, so the sensitive-path gate and the model load stay in the same scope. files.pydoes not import the memory handlers, so the new import adds no cycle.test_path_checks_run_off_the_event_loopgoes red if either call returns to the loop.test_busy_probe_pool_answers_a_coded_503pins the new answer.- The facade
TYPE_CHECKINGmanifest now names the four forwarded names.
Remaining before merge: PR Readiness is still checking on this head. I will approve after it reports passed.
|
Review-ready at afd6194, waiting on a maintainer review.
No merge, approval or auto-merge from this Crew; the merge is the maintainer's call. |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (3 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: moves the custom model path validation and stat onto the bounded path-probe pool so a slow mount cannot block the event loop, with a coded 503 when the pool is busy, plus tests.
Problem / Motivation
Goal: Checking a custom embedding model path on a mount that stopped answering no longer freezes the whole gateway.
POST /api/memory/embedding-modeltakes a model path from the request. Its handler,api_memory_embedding_modelindashboard/handlers/memory_embeddings.py, callsvalidate_custom_model_path(which runsexists(),is_file()andstat()on that path) and thenpath.stat()again, inline in the coroutine.Why it matters
The gateway runs one event loop. One of these calls on a hung NFS or FUSE mount blocks every endpoint at once. A stall past
dashboard.loop_stall_exit_after_secsmakes the loop watchdog kill the process. The path is the caller's choice, so the caller picks which mount hangs the gateway. This is the same defect #21791 fixed inknowledge.py.Not a goal
scripts/check_sync_io_in_async.py. It lands next, in its own PR. That gate'sTestExemplarStaysCleanrequires the memory handlers to stay clean, which is why this fix goes first._apply_embedding_model. It already runs on the embed executor, off the loop.validate_custom_model_pathaccepts or refuses.What changed (motivation → approach → change)
A path from a request was stat'ed on the event loop, so a hung mount froze the gateway.
The handler now makes one trip to the bounded request-path probe pool,
_run_path_probe, the onefiles.pyandknowledge.pyalready use. A new sync helper,_probe_model_path, runs the validation and the size stat together on that pool. A full pool, or a mount already marked stuck, answers the same coded 503 (path_probe_busy) as the file endpoints, via_probe_busy_response.The
memoryfacade lists the three names it now serves fromdashboard.handlers.filesunderTYPE_CHECKING, as its other outside names are.Backwards compatibility
Compatible: every request that succeeds today gets the same answer, including
size_bytesand the 400 codes. The only new answer is the coded 503 when the probe pool is full, which today would have blocked the loop instead.Removes nothing: no route, key, field or record is removed or narrowed.
Tests
In
test/test_handlers_memory_coverage.py,TestEmbeddingModelEndpoint:test_path_checks_run_off_the_event_loop: the validation and the size stat both run on a thread other than the loop's, and the answer is unchanged.test_busy_probe_pool_answers_a_coded_503: a refused probe answers 503 withcode: path_probe_busy.Determinism: the two tests passed 20/20 repeated runs. The whole class passed in one shuffled node order (seed 7).
prove.py --base origin/main: PROVEN (an assertion fails with the fix reverted).Manual verification
N/A: unit coverage sufficient. The test records the real thread of each call, which is what a manual hung-mount check would show.
Screenshots / video
N/A: backend only, no user-visible UI change.
Related Issues
Refs #10517
Pattern harvest
Rule candidate: lint
Pattern: "a filesystem syscall on a request-supplied path inside an
async def". This is #10517 part 1, thefsfamily forscripts/check_sync_io_in_async.py, which opens next.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)