Skip to content

feat(index): load resident RaBitQ columns when an index opens - #9842

Draft
BubbleCal wants to merge 4 commits into
yang/layered-rq-plane-rowsfrom
yang/layered-rq-open-residency
Draft

BubbleCal wants to merge 4 commits into
yang/layered-rq-plane-rowsfrom
yang/layered-rq-open-residency

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Load and retain the small resident column store at index open on every origin, so partition opens share its Arc allocation. A Weak registry reuses live stores after cache eviction, and the ordinary cache can retain idle stores. Cover pre-open binding, cache promotion, eviction, reopen and unavailable-residency fallbacks. Auto admission follows the ordinary cache entry limit for both QuickCache and Moka; explicit On can retain a store even with no cache capacity. Lifecycle tests cover eviction with live handles, load failure, reconstruction, session teardown and identical query results.

Dependencies

Stacked directly on #9837.

Integrated Lance main b3c8619886b9 and the updated 3–9-bit layered prerequisites. Standalone benchmark/POC tools remain excluded.

Retain main's sub-index load timing alongside resident-store attachment and query reconstruction.

Validation

This PR tree passed formatting and full-workspace clippy (--all --tests --benches -- -D warnings, Rust 1.98.1, Linux x86_64). The integrated composition passed 670 BQ tests and 22 layered integration tests on x86_64, plus 22 layered and 111 lazy/resident integration tests on macOS arm64. Production and Rust test source equality was verified after the final dependency refresh; the only later source change adds Java test coverage.

No new performance measurement is claimed. CI is not awaited.

Tracking

OSS-2566

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer enhancement New feature or request labels Oct 9, 2026
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-plane-rows branch from c1a8b97 to f899069 Compare October 9, 2026 16:48
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-open-residency branch 2 times, most recently from 0d1febf to f931c45 Compare October 9, 2026 18:55
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-plane-rows branch from f899069 to 4399615 Compare October 9, 2026 18:55
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-plane-rows branch from 53ee247 to 156f013 Compare October 10, 2026 16:03
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-open-residency branch from dd31fc2 to 01a4dd1 Compare October 10, 2026 16:04

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant