Skip to content

Register re-scannable write sources to avoid spilling re-readable inputs (Scanner/LanceDataset) #7895

Open

Description

@wjones127

Problem

The Python write path classifies sources into "materialized" (wrapped in an in-memory MemTable) vs. "streaming" (one-shot), via _is_materialized / _coerce_reader in python/python/lance/types.py.

Re-readable but not-in-memory sources — LanceDataset, pa.dataset.Dataset, pa.dataset.Scanner — are converted with .to_reader() and fall through to the streaming path. On the Rust side they arrive as a one-shot SendableRecordBatchStream of unknown size, so with conflict_retries > 0 they get drained into a memory/disk spill (spilling_table_provider) on every merge_insert — even though the source could simply be re-scanned. The re-readability is thrown away at the boundary.

Impact is limited to wasted work (an extra buffer/double-store on retrying merges), not correctness, but it's avoidable.

Proposed improvements (coupled)

  1. Re-scannable TableProvider. Add a provider that re-runs the source's scan on each .scan() — no spill, no full materialization — for sources that advertise re-readability (Scanner, LanceDataset, pa.dataset.Dataset). This eliminates both the spill and the double-store for those inputs.

  2. singledispatch source registry. Refactor the source classification/coercion (_is_materialized + _coerce_reader) into a functools.singledispatch registry (as done in LanceDB), returning the source plus a strategy (materialize / re-scannable / one-shot). This collapses the two hand-maintained type lists into one extensible registry and lets first- and third-party types register their own handling — e.g. a re-scannable registration wouldn't spill.

These are really one refactor: (2) is the mechanism, (1) is its first payoff.

Context

Follow-up from #7368 review (thanks @hamersaw for flagging both the type-list maintainability and the auto-inference of spill-on-retry).

Activity

  1. easyrev commented on Sep 29, 2026

    @easyrev

    Hi @wjones127 , I’d like to work on the Python source registry and re-scannable provider. If nobody is working on this, could you assign it to me? I’ll share a short implementation plan here before opening a draft PR.

  2. easyrev commented on Oct 11, 2026

    @easyrev

    Following up on #7895. I looked through the Python write-source conversion and merge retry path, and put together a draft implementation in #9617.

    The approach uses a singledispatch registry and a re-scannable TableProvider. Eligible Lance and Arrow datasets can open fresh readers on retry. One-shot inputs and scanners without a repeatability guarantee keep the existing replay path. Third-party sources can provide their own reader factories.

    I've tested real conflict retries and verified that eligible sources bypass the replay buffer. The implementation, validation results, and performance tradeoffs are in the PR.

    I exposed WriteSource and coerce_source as experimental APIs under lance.types. Does that API surface look reasonable to you?

    I'm keeping the PR as a draft for now, so I'm mainly looking for early API feedback rather than a full review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions