Skip to content

feat(index): add origin-aware reads and shared resident columns - #9835

Draft
BubbleCal wants to merge 5 commits into
yang/layered-rq-residency-basefrom
yang/layered-rq-origin-residency
Draft

BubbleCal wants to merge 5 commits into
yang/layered-rq-residency-basefrom
yang/layered-rq-origin-residency

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Add origin-latency hints, origin-aware sparse gather policy and a 256 KiB default coalescing gap for high-latency origins. Share reusable small columns through Arc handles and a Weak registry. Store them in the ordinary index cache; eviction removes the cache entry while live handles retain the allocation. Auto admission uses the backend's ordinary maximum entry size.

Dependencies

Depends on #9834, #9833. The base branch yang/layered-rq-residency-base only merges these prerequisite heads so this PR shows its own feature diff. After the prerequisites land, rebase this PR’s own feature commit onto main (including after squash merges), then retarget it to main. Do not use the review-base branch as the final merge destination.

Integrated Lance main b3c8619886b9 and the updated 3–9-bit layered prerequisites. Standalone benchmark/POC tools remain excluded.

An Arc handle is acquired before remapping or cache lookups can promote other entries, so eviction during index open cannot force a reload of a live store. Both legacy and tagged row-ID remapping remain wired through index open and reconstruction; main's pairwise readers are retained.

Validation

This PR tree passed formatting and full-workspace clippy (--all --tests --benches -- -D warnings, Rust 1.98.1, Linux x86_64). The integrated composition passed 670 BQ tests and 22 layered integration tests on x86_64, plus 22 layered and 111 lazy/resident integration tests on macOS arm64. Production and Rust test source equality was verified after the final dependency refresh; the only later source change adds Java test coverage.

No new performance measurement is claimed. CI is not awaited.

Tracking

OSS-2563

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer enhancement New feature or request labels Oct 9, 2026
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-lazy-full branch from d9ae54b to 455f096 Compare October 9, 2026 16:48
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-origin-residency branch 2 times, most recently from 0e68efc to c2c40eb Compare October 9, 2026 18:55
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-lazy-full branch from 455f096 to 259b911 Compare October 9, 2026 18:55
@github-actions github-actions Bot added the A-encoding Encoding, IO, file reader/writer label Oct 9, 2026
@BubbleCal
BubbleCal changed the base branch from yang/layered-rq-lazy-full to yang/layered-rq-residency-base October 9, 2026 18:56
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-residency-base branch from 00a7387 to f0ce1b8 Compare October 10, 2026 16:03
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-origin-residency branch from 98650a7 to b731b77 Compare October 10, 2026 16:03
@BubbleCal BubbleCal changed the title feat(index): add origin-aware reads and budgeted resident columns feat(index): add origin-aware reads and shared resident columns Oct 10, 2026
@BubbleCal
BubbleCal force-pushed the yang/layered-rq-residency-base branch from f0ce1b8 to 6329b34 Compare October 11, 2026 12:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer A-index Vector index, linalg, tokenizer enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant