Skip to content

feat(flat): support external vector storage and batched scans - #17

Open
iaojnh wants to merge 17 commits into
mainfrom
codex/flat-external-vector
Open

iaojnh wants to merge 17 commits into
mainfrom
codex/flat-external-vector

Conversation

@iaojnh

@iaojnh iaojnh commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Flat can now search externally owned row-major FP32 L2/IP vectors without storing a second copy of their payloads. VectorSource supplies an optional batched iterator for exhaustive scans; candidate searches reuse its existing random-access methods. Callers register IDs with add_with_source, then bind a source per search request.

The scan domain is the intersection of externally enumerated IDs, registered IDs, and the request filter. Callers must keep the source, scan domain, and visibility filter on the same snapshot. Scan batches can retain backing pages through a lease; fetched result vectors borrow source memory.

This includes request isolation, storage-mode validation, reopen coverage, and guards for unsupported formats and operations. Registration skips redundant FP32 payload conversion. Filtered or unregistered rows may omit vector pointers, and reusing query parameters preserves the visibility filter.

Validation: 210 related local tests passed, with 2 platform/size-dependent skips; all changed C++ files pass clang-tidy with warnings treated as errors. The Windows ARM64 C1001 failure is addressed by the upstream test fix from alibaba#795. This branch has also been merged with current main, preserving its memory-lifetime and read-only-close fixes and guarding the new internal-buffer fast path in external-vector mode.

The initial scope excludes quantized/sparse/column-major vectors, cosine, group-by, source-bound refinement, and dump/merge of external Flat indexes.

Related: alibaba#797.

@iaojnh
iaojnh force-pushed the codex/flat-external-vector branch from 8e88f03 to 2b9987d Compare October 8, 2026 12:01
@iaojnh iaojnh changed the title Codex/flat external vector feat(flat): support external vector storage and batched scans Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants