Repository navigation
Conversation
iaojnh
force-pushed
the
codex/flat-external-vector
branch
from
October 8, 2026 12:01
8e88f03 to
2b9987d
Compare
…#795) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flat can now search externally owned row-major FP32 L2/IP vectors without storing a second copy of their payloads.
VectorSourcesupplies an optional batched iterator for exhaustive scans; candidate searches reuse its existing random-access methods. Callers register IDs withadd_with_source, then bind a source per search request.The scan domain is the intersection of externally enumerated IDs, registered IDs, and the request filter. Callers must keep the source, scan domain, and visibility filter on the same snapshot. Scan batches can retain backing pages through a lease; fetched result vectors borrow source memory.
This includes request isolation, storage-mode validation, reopen coverage, and guards for unsupported formats and operations. Registration skips redundant FP32 payload conversion. Filtered or unregistered rows may omit vector pointers, and reusing query parameters preserves the visibility filter.
Validation: 210 related local tests passed, with 2 platform/size-dependent skips; all changed C++ files pass clang-tidy with warnings treated as errors. The Windows ARM64 C1001 failure is addressed by the upstream test fix from alibaba#795. This branch has also been merged with current main, preserving its memory-lifetime and read-only-close fixes and guarding the new internal-buffer fast path in external-vector mode.
The initial scope excludes quantized/sparse/column-major vectors, cosine, group-by, source-bound refinement, and dump/merge of external Flat indexes.
Related: alibaba#797.