Skip to content

feat(index): add layered RaBitQ storage and precision queries - #9490

Draft
BubbleCal wants to merge 5 commits into
yang/layered-rq-core-basefrom
yang/layered-rabitq-stack
Draft

BubbleCal wants to merge 5 commits into
yang/layered-rq-core-basefrom
yang/layered-rabitq-stack

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Add opt-in layered IVF_RQ storage for every supported width from 3 through 9 bits, per-level factors, precision selection, per-plane cache loading, and build/merge/remap support. Layered defaults to false; explicitly enabling it with 1 or 2 bits returns an error. Full queries retain native full-width encoding and pruning semantics. Sign/high precision and the explicit candidate cascade remain separate query choices.

Store high and low planes with their actual widths and select the matching high-precision calculator. Cover all seven widths with encoding/decoding, numeric parity and index lifecycle regressions. Preserve main's search-effort field in execution-query protobuf round trips.

Dependencies

Depends on #9470, #9831, and the merged #9829 and #9832. The review-only base yang/layered-rq-core-base combines these prerequisites with Lance main b3c8619886b9. Once the prerequisites land, rebase this PR's own changes onto main and retarget it.

Validation

On Linux x86_64 with Rust 1.98.1, formatting, full-workspace clippy (--all --tests --benches -- -D warnings), 559 BQ unit tests, 22 layered index integration tests, and 25 execution-query wire tests (with substrait enabled) passed on this PR tree.

No new performance measurement is claimed. CI is not awaited.

Tracking

OSS-2343, OSS-2344, OSS-2347, OSS-2348, OSS-2349, OSS-2351, OSS-2354, OSS-2353, OSS-2559

@github-actions github-actions Bot added A-python Python bindings A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-deps Dependency updates A-docs Documentation enhancement New feature or request labels Sep 23, 2026
@github-actions github-actions Bot added the A-encoding Encoding, IO, file reader/writer label Oct 9, 2026
@BubbleCal BubbleCal changed the title feat(index): add layered rabitq precision and candidate reranking feat(index): add layered rabitq lazy full queries Oct 9, 2026
@BubbleCal
BubbleCal force-pushed the yang/layered-rabitq-stack branch from cdd6408 to 0d950bc Compare October 9, 2026 14:16
@BubbleCal BubbleCal changed the title feat(index): add layered rabitq lazy full queries feat(index): add layered RaBitQ storage and precision queries Oct 9, 2026
@BubbleCal
BubbleCal changed the base branch from yang/layered-rabitq-format to yang/layered-rq-encoding-primitives October 9, 2026 14:17
@BubbleCal
BubbleCal force-pushed the yang/layered-rabitq-stack branch from 0d950bc to f14aff3 Compare October 9, 2026 16:48
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-encoding-primitives branch from 90c0415 to 689ea17 Compare October 9, 2026 16:48
@BubbleCal
BubbleCal force-pushed the yang/layered-rabitq-stack branch from f14aff3 to 7a3e52f Compare October 9, 2026 18:55
@github-actions github-actions Bot added A-format On-disk format: protos and format spec docs format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). labels Oct 9, 2026
@BubbleCal
BubbleCal changed the base branch from yang/layered-rq-encoding-primitives to yang/layered-rq-core-base October 9, 2026 18:56

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates A-docs Documentation A-encoding Encoding, IO, file reader/writer A-format On-disk format: protos and format spec docs A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-python Python bindings enhancement New feature or request format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant