Repository navigation
fix: Retire embedding cache after failed CPU transfer - #820
Draft
buildingjoshbetter wants to merge 1 commit into
Draft
buildingjoshbetter wants to merge 1 commit into
buildingjoshbetter wants to merge 1 commit into
Conversation
This was referenced Oct 9, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
When embedding inference hits MPS OOM and its transfer to CPU raises, the daemon retained the failed model. A later request returned that cached instance without resolving sticky CPU selection, so it could repeat inference and transfer failure on the same MPS object.
This follow-up invalidates the cache only when it still holds that failed instance, using the existing residency and fast-generation retirement machinery. It rethrows the identical exception, including cancellation, without another retry or construction in the failed request. The next admitted request reconstructs the same requested model on CPU. An unrelated replacement survives; successful transfers stay cached. Models, rerankers, dtype, request limits and retrieval behavior remain unchanged.
Stacked on #819 (
fix/model-process-admission-publish-297-v2), a partial follow-up for #297. Exactly three files change.Validation
6efc5830b2e0ce8c6ffa345c59d41e6fe182adc1: 3,067 tests and 2,626 subtests passed, 17 skipped, 3 deselected. Log SHA256:defe64345f65355a04e5fd8de0eee9e29f4faee973621cfeafe4dffb616a0a8b.7af5eaf9d6b1f527a523c20d327540d62daeff67: 2,257 tests and 670 subtests passed, 17 skipped, 3 deselected. Log SHA256:142ef530dbd55bb20638a849f8c08432fd82020c2763d69566d871ac3f9dbe98.Limits
Cache invalidation does not establish native allocator reclamation, bound CPU-transfer peak memory or prove the original 26.24 GB Mac incident fixed. Tracebacks or active captured work may retain objects until released. The RSS guard remains disabled by default pending calibration; finite budgets, workspace reservations, system headroom, native Mac/MPS latency and thermal acceptance remain open. No main merge or release.