You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Python write path classifies sources into "materialized" (wrapped in an in-memory MemTable) vs. "streaming" (one-shot), via _is_materialized / _coerce_reader in python/python/lance/types.py.
Re-readable but not-in-memory sources — LanceDataset, pa.dataset.Dataset, pa.dataset.Scanner — are converted with .to_reader() and fall through to the streaming path. On the Rust side they arrive as a one-shot SendableRecordBatchStream of unknown size, so with conflict_retries > 0 they get drained into a memory/disk spill (spilling_table_provider) on every merge_insert — even though the source could simply be re-scanned. The re-readability is thrown away at the boundary.
Impact is limited to wasted work (an extra buffer/double-store on retrying merges), not correctness, but it's avoidable.
Proposed improvements (coupled)
Re-scannable TableProvider. Add a provider that re-runs the source's scan on each .scan() — no spill, no full materialization — for sources that advertise re-readability (Scanner, LanceDataset, pa.dataset.Dataset). This eliminates both the spill and the double-store for those inputs.
singledispatch source registry. Refactor the source classification/coercion (_is_materialized + _coerce_reader) into a functools.singledispatch registry (as done in LanceDB), returning the source plus a strategy (materialize / re-scannable / one-shot). This collapses the two hand-maintained type lists into one extensible registry and lets first- and third-party types register their own handling — e.g. a re-scannable registration wouldn't spill.
These are really one refactor: (2) is the mechanism, (1) is its first payoff.
Context
Follow-up from #7368 review (thanks @hamersaw for flagging both the type-list maintainability and the auto-inference of spill-on-retry).
Hi @wjones127 , I’d like to work on the Python source registry and re-scannable provider. If nobody is working on this, could you assign it to me? I’ll share a short implementation plan here before opening a draft PR.
Following up on #7895. I looked through the Python write-source conversion and merge retry path, and put together a draft implementation in #9617.
The approach uses a singledispatch registry and a re-scannable TableProvider. Eligible Lance and Arrow datasets can open fresh readers on retry. One-shot inputs and scanners without a repeatability guarantee keep the existing replay path. Third-party sources can provide their own reader factories.
I've tested real conflict retries and verified that eligible sources bypass the replay buffer. The implementation, validation results, and performance tradeoffs are in the PR.
I exposed WriteSource and coerce_source as experimental APIs under lance.types. Does that API surface look reasonable to you?
I'm keeping the PR as a draft for now, so I'm mainly looking for early API feedback rather than a full review.
Activity
easyrev commentedon Sep 29, 2026
Hi @wjones127 , I’d like to work on the Python source registry and re-scannable provider. If nobody is working on this, could you assign it to me? I’ll share a short implementation plan here before opening a draft PR.
easyrev commentedon Oct 11, 2026
Following up on #7895. I looked through the Python write-source conversion and merge retry path, and put together a draft implementation in #9617.
The approach uses a
singledispatchregistry and a re-scannableTableProvider. Eligible Lance and Arrow datasets can open fresh readers on retry. One-shot inputs and scanners without a repeatability guarantee keep the existing replay path. Third-party sources can provide their own reader factories.I've tested real conflict retries and verified that eligible sources bypass the replay buffer. The implementation, validation results, and performance tradeoffs are in the PR.
I exposed
WriteSourceandcoerce_sourceas experimental APIs underlance.types. Does that API surface look reasonable to you?I'm keeping the PR as a draft for now, so I'm mainly looking for early API feedback rather than a full review.