Skip to content

fix: Retire embedding cache after failed CPU transfer - #820

Draft
buildingjoshbetter wants to merge 1 commit into
fix/model-process-admission-publish-297-v2from
fix/model-transfer-retention-publish-297-v1
Draft

buildingjoshbetter wants to merge 1 commit into
fix/model-process-admission-publish-297-v2from
fix/model-transfer-retention-publish-297-v1

Conversation

@buildingjoshbetter

Copy link
Copy Markdown
Owner

Change

When embedding inference hits MPS OOM and its transfer to CPU raises, the daemon retained the failed model. A later request returned that cached instance without resolving sticky CPU selection, so it could repeat inference and transfer failure on the same MPS object.

This follow-up invalidates the cache only when it still holds that failed instance, using the existing residency and fast-generation retirement machinery. It rethrows the identical exception, including cancellation, without another retry or construction in the failed request. The next admitted request reconstructs the same requested model on CPU. An unrelated replacement survives; successful transfers stay cached. Models, rerankers, dtype, request limits and retrieval behavior remain unchanged.

Stacked on #819 (fix/model-process-admission-publish-297-v2), a partial follow-up for #297. Exactly three files change.

Validation

  • Independent original-source probes reproduced the loop with synthetic Qwen identity for single and batched requests: one MPS construction, two MPS forwards, the same transfer exception twice, sticky CPU set and owners released. On the corrected source, the first request clears the failed cache and the second constructs the same identity on CPU and succeeds. No native model ran in these probes.
  • All 135 existing assertion nodes are unchanged; the fixture now has 194. The original-source negative control fails 13 assertions with zero errors. Independent partial-transfer, cancellation and alias checks pass eight cases; successful-transfer preservation passes two cases. Weak-reference checks release exception tracebacks before assessing Python object lifetime.
  • Combined author 6efc5830b2e0ce8c6ffa345c59d41e6fe182adc1: 3,067 tests and 2,626 subtests passed, 17 skipped, 3 deselected. Log SHA256: defe64345f65355a04e5fd8de0eee9e29f4faee973621cfeafe4dffb616a0a8b.
  • Exact publication head 7af5eaf9d6b1f527a523c20d327540d62daeff67: 2,257 tests and 670 subtests passed, 17 skipped, 3 deselected. Log SHA256: 142ef530dbd55bb20638a849f8c08432fd82020c2763d69566d871ac3f9dbe98.
  • Both full gates ran sequentially in unprivileged, offline, bounded Linux containers with actual sqlite-vec and synthetic data. Both exited 0 without timeout or OOM. Installed-wheel builds and remote source hashes passed.
  • Independent implementation and publication reviews found no blockers. Production/test files exactly match the author; documentation retains the publication prerequisite plus the exact new suffix. Ruff, Python 3.10 grammar and whitespace checks pass. Privacy and public identity checks cover 204 added lines, three files and one commit with zero findings.

Limits

Cache invalidation does not establish native allocator reclamation, bound CPU-transfer peak memory or prove the original 26.24 GB Mac incident fixed. Tracebacks or active captured work may retain objects until released. The RSS guard remains disabled by default pending calibration; finite budgets, workspace reservations, system headroom, native Mac/MPS latency and thermal acceptance remain open. No main merge or release.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant