Skip to content

fix(memory): check a custom model path off the event loop - #22023

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/10517-memory-model-path-off-loop
Oct 19, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/10517-memory-model-path-off-loop

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Checking a custom embedding model path on a mount that stopped answering no longer freezes the whole gateway.

POST /api/memory/embedding-model takes a model path from the request. Its handler, api_memory_embedding_model in dashboard/handlers/memory_embeddings.py, calls validate_custom_model_path (which runs exists(), is_file() and stat() on that path) and then path.stat() again, inline in the coroutine.

Why it matters

The gateway runs one event loop. One of these calls on a hung NFS or FUSE mount blocks every endpoint at once. A stall past dashboard.loop_stall_exit_after_secs makes the loop watchdog kill the process. The path is the caller's choice, so the caller picks which mount hangs the gateway. This is the same defect #21791 fixed in knowledge.py.

Not a goal

What changed (motivation → approach → change)

A path from a request was stat'ed on the event loop, so a hung mount froze the gateway.

The handler now makes one trip to the bounded request-path probe pool, _run_path_probe, the one files.py and knowledge.py already use. A new sync helper, _probe_model_path, runs the validation and the size stat together on that pool. A full pool, or a mount already marked stuck, answers the same coded 503 (path_probe_busy) as the file endpoints, via _probe_busy_response.

The memory facade lists the three names it now serves from dashboard.handlers.files under TYPE_CHECKING, as its other outside names are.

Backwards compatibility

Compatible: every request that succeeds today gets the same answer, including size_bytes and the 400 codes. The only new answer is the coded 503 when the probe pool is full, which today would have blocked the loop instead.

Removes nothing: no route, key, field or record is removed or narrowed.

Tests

In test/test_handlers_memory_coverage.py, TestEmbeddingModelEndpoint:

  • test_path_checks_run_off_the_event_loop: the validation and the size stat both run on a thread other than the loop's, and the answer is unchanged.
  • test_busy_probe_pool_answers_a_coded_503: a refused probe answers 503 with code: path_probe_busy.

Determinism: the two tests passed 20/20 repeated runs. The whole class passed in one shuffled node order (seed 7). prove.py --base origin/main: PROVEN (an assertion fails with the fix reverted).

Manual verification

N/A: unit coverage sufficient. The test records the real thread of each call, which is what a manual hung-mount check would show.

Screenshots / video

N/A: backend only, no user-visible UI change.

Related Issues

Refs #10517

Pattern harvest

Rule candidate: lint
Pattern: "a filesystem syscall on a request-supplied path inside an async def". This is #10517 part 1, the fs family for scripts/check_sync_io_in_async.py, which opens next.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision afd6194317bb4d45b50319be878b94840e420bc9 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of afd6194317bb4d45b50319be878b94840e420bc9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] afd6194

Description read

[DESCRIPTION-READ] a7edb84d4fbeb3b26f9be5ec4cbd2546632c8b4ba2a97e766c13dfa5f638fcfb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of afd6194317bb4d45b50319be878b94840e420bc9 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (3 items) — 3 justified

Intent: stop a custom embedding-model path on a hung mount from freezing the whole gateway. This is a FIX.

  1. Checking a custom model path and reading its size now run on the bounded probe pool, not on the event loop. — justified
  2. When the probe pool is full, or the mount is already marked stuck, the endpoint now answers a coded 503 path_probe_busy. — justified
  3. The memory facade now lists _probe_model_path and the three probe names it gets from files, under TYPE_CHECKING. — justified

[FIRST-PRINCIPLES-REVIEWED] afd6194

Description read

[DESCRIPTION-READ] 4f015755a23875f4faa4a753f20c9aae05b3523a3bdee60ea2816ad0d57cff2d

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed afd6194317bb4d45b50319be878b94840e420bc9 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] afd6194

Verdict parsed from the review's SHA-scoped output markers for commit afd6194317bb4d45b50319be878b94840e420bc9.

False positive or not applicable? A repository writer can comment:
/ai-review override fable afd6194317bb4d45b50319be878b94840e420bc9: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of afd6194317bb4d45b50319be878b94840e420bc9 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] afd6194

False positive or not applicable? A repository writer can comment:
/ai-review override gpt afd6194317bb4d45b50319be878b94840e420bc9: <one-sentence reason>

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Every client of POST /api/memory/embedding-model handles path_probe_busy like the file-browse callers: rebutted (verified, no change)

The only client is website/src/pages/overview/EmbeddingModelCard.tsx (vectorValidateEmbedModel / vectorApplyEmbedModel). A non-OK answer throws, and embedModelErrorMessage maps known codes; any other code falls to the server's error prose.
No file in website/src maps path_probe_busy today, so the file-browse callers show the same server prose: "file system probe capacity exhausted; retry shortly". The two surfaces behave alike.
The probe runs before any config write, so a 503 changes nothing, and the Check button stays usable to retry.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • _probe_busy_response(resource=raw, ...) answers a dashboard client like the file endpoints: rebutted (verified, no change)

It is the same function the file endpoints and knowledge.py call (dashboard/file_api/path_probe.py): status 503, body {"error": ..., "code": "path_probe_busy"}, one SEL record. Only resource and operation differ, as at every adopter.
resource=raw is the path the caller sent, the same thing knowledge.py passes as resource=str(uri). operation="memory.embedding_model" is the name this handler's denied and allowed records already use.

@iamwhatever
iamwhatever force-pushed the fix/10517-memory-model-path-off-loop branch from 6874aad to afd6194 Compare October 19, 2026 02:48
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Pushed afd6194: fixes the CI red on Backend Tests (Windows) (2), test/test_facade_mirror_guards.py::test_every_mirrored_facade_manifest_names_what_it_forwards. This PR's new helper _probe_model_path was forwarded by the memory facade but missing from its TYPE_CHECKING manifest; that one line is added. Rebased onto main fcc5eea (650b69f -> fcc5eea), where main had moved in nearby files. 438 related tests pass locally. CI re-running.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to ship. Reviewed head afd6194.

What I checked:

  • _probe_model_path runs validate_custom_model_path and the size stat in one hop on _run_path_probe. No filesystem call on the request path stays on the event loop.
  • The busy path uses _probe_busy_response, the same coded 503 that files.py and knowledge.py return. The probe runs before any config write, so a 503 changes no state.
  • Success and 400 answers are unchanged, including size_bytes and 0 for an unstatable path.
  • The worker re-check in _apply_embedding_model still runs, so the sensitive-path gate and the model load stay in the same scope.
  • files.py does not import the memory handlers, so the new import adds no cycle.
  • test_path_checks_run_off_the_event_loop goes red if either call returns to the loop. test_busy_probe_pool_answers_a_coded_503 pins the new answer.
  • The facade TYPE_CHECKING manifest now names the four forwarded names.

Remaining before merge: PR Readiness is still checking on this head. I will approve after it reports passed.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Review-ready at afd6194, waiting on a maintainer review.

  • PR Readiness passed, 104 checks green, mergeable, one commit; green_age is fresh against main d32ab41.
  • GPT and Opus: no findings. Design and First Principles passed; their earlier questions (how a client sees the new path_probe_busy 503) are answered above.
  • The one CI red on an earlier head (test_facade_mirror_guards, the facade manifest missing _probe_model_path) was this PR's own and is fixed.
  • Refs #10517: this clears the last memory-handler site so the fs family gate (Extend the sync-IO-in-async gate to request-path filesystem calls, and offload knowledge.py's five sites #10517 part 1) can open next with TestExemplarStaysClean unchanged.

No merge, approval or auto-merge from this Crew; the merge is the maintainer's call.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 19, 2026 04:04

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (3 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: moves the custom model path validation and stat onto the bounded path-probe pool so a slow mount cannot block the event loop, with a coded 503 when the pool is busy, plus tests.

@KiroCrewCaptain
KiroCrewCaptain merged commit 1862203 into main Oct 19, 2026
105 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/10517-memory-model-path-off-loop branch October 19, 2026 04:38
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants