diff --git a/_posts/2026-08-07-analytic-file-formats-sans-io.md b/_posts/2026-08-07-analytic-file-formats-sans-io.md new file mode 100644 index 00000000000..b9aafc89542 --- /dev/null +++ b/_posts/2026-08-07-analytic-file-formats-sans-io.md @@ -0,0 +1,366 @@ +--- +layout: post +title: "Implementing Analytic File Formats Sans I/O" +description: "Why separating Parquet decoding from storage I/O makes analytic file readers more composable, and how arrow-rs puts the design into practice" +date: "2026-08-07 00:00:00" +author: "Andrew Lamb, Xudong Wang, Neil Conway, and Daniël Heres" +categories: [parquet, arrow] +--- + + +Opening a file, constructing a reader, and calling `next()` is a wonderfully +simple interface. It is also where many analytic file libraries accidentally +make a much larger architectural decision: **the decoder, rather than the +application, controls I/O**. + +That choice works well until the decoder is embedded in a system with its own +storage runtime, buffer manager, cache, or scheduler. Such a system may want to +race object-store reads, use `io_uring`, account for every buffered byte, or +reuse pages that are already resident in memory. If the file reader performs +I/O internally, the application is limited to the policies anticipated by the +library author. + +This post presents a different foundation for analytic file readers: **Sans +I/O**. A Sans I/O decoder contains all the knowledge needed to interpret a file, +but never reads the file itself. It reports which byte ranges it needs, and the +application decides how to obtain them. + +> **The core idea:** format knowledge belongs in the decoder; I/O policy belongs +> in the application. + +This post distills our CIDR 2027 paper, *Implementing Analytic File Formats Sans +I/O*, and describes the production implementation in the [Apache Arrow Rust +Parquet reader][arrow-rs-parquet]. + +## A Parquet reader is a small query engine + +[Apache Parquet] is a good example of why the I/O boundary matters. Reading a +Parquet file is not a single sequential operation. Even a simplified decoder +alternates between I/O and CPU work across at least nine states. + +The two diagrams in this post are reproduced from the paper under [CC BY 4.0]. + +
+ Parquet decoding alternates between I/O and CPU stages for the file footer, metadata, optional pruning structures, and data pages +
Figure 1 (paper Figure 2): The Parquet decoding state machine. Pink states perform I/O, yellow states perform CPU work, and dashed states are optional.
+
+ +The footer reveals where the file metadata lives. The metadata identifies row +groups and column chunks. Bloom filters and page indexes may eliminate some of +those chunks, and only then does the decoder know which data pages survive. The +answer to "what bytes are needed next?" therefore depends on information +decoded in earlier states, along with the query's projection, filters, and +limit. + +There are several valid ways to orchestrate these reads: + +- Prefetch the next predicted range while the CPU decodes the current one. +- Coalesce nearby ranges to reduce request count, at the cost of reading and + buffering extra bytes. +- Interleave or race remote reads to reduce tail latency. +- Skip indexes that cannot help the current query or maintenance task. +- Reuse pages already owned by the application's buffer manager. + +The best choice depends on the environment. Local NVMe, remote object storage, +an in-memory cache, `mmap`, RDMA, `O_DIRECT`, and `io_uring` have very different +latency, concurrency, alignment, and memory requirements. There is no single +read policy that is best for all of them. + +## The hidden policy inside a pull API + +Most analytic file libraries expose a pull interface modeled after a database +iterator. For example, the main Parquet Java API is conceptually as simple as: + +```java +ParquetReader reader = ParquetReader.builder( + new GroupReadSupport(), path +).build(); + +Group record; +while ((record = reader.read()) != null) { + process(record); +} +``` + +The caller sees records, not reads. When the decoder needs another page, it +internally calls methods such as `seek` and `readFully` on a +`SeekableInputStream`. This is convenient for both the library implementer and +applications whose own execution model is also an iterator. + +However, a replaceable storage interface is not the same thing as Sans I/O. A +callback such as `read_at(offset, length)` lets the application choose *where* +bytes come from, but the decoder still chooses *which* ranges to read, *when* +to issue each request, and often *where* to buffer the result. + +This has two important consequences: + +1. **I/O optimization is limited to what the reader implements.** Adding a + new policy such as read racing or a workload-specific prefetcher requires a + change inside the format library. +2. **Memory ownership becomes blurred.** Either the reader maintains a sizable + private buffer pool, or the application adapts its own buffer manager to the + reader's I/O abstraction. Both make global memory accounting harder. + +### Async is useful, but it is not Sans I/O + +An asynchronous reader can return control while an I/O operation is pending, +so a thread does not need to sit idle. That changes *how execution waits*. It +does not necessarily change *who owns the I/O policy*. + +In most async readers, a future or stream still computes and issues reads +internally. The caller can await the result, but cannot inspect the decoder's +future byte requirements or substitute a different schedule for those reads. +By contrast, a Sans I/O decoder reports those requirements as data. It can be +driven by synchronous code, async code, a thread pool, a custom runtime, or no +runtime at all. + +## The Sans I/O contract + +We call an analytic file decoder Sans I/O if it performs no I/O, directly or +indirectly, and communicates all byte requirements to its caller. + +
+ Side-by-side comparison of an integrated decoder, where I/O is orchestrated by the library and performed through a storage callback, and a Sans I/O decoder, where I/O is orchestrated and performed by the application +
Figure 2 (paper Figure 1): An integrated reader accepts a storage callback but owns the read schedule. A Sans I/O reader leaves both orchestration and execution of I/O to the application.
+
+ +The contract has a small feedback loop. Each time the application asks the +decoder to advance, the decoder does one of three things: + +1. Produces a decoded batch. +2. Reports that decoding is finished. +3. Returns the precise byte ranges needed to make progress. + +For the third outcome, the application fetches those ranges using any strategy +and pushes the resulting buffers into the decoder. It may supply more data than +was requested, which permits prefetching. The decoder keeps all format +semantics, including pruning and the order in which metadata must be decoded, +while the host owns concurrency, caching, request coalescing, retries, memory, +and cost. + +In simplified Rust, the interaction looks like this: + +```rust +loop { + match decoder.try_decode()? { + DecodeResult::NeedsData(ranges) => { + // Any storage API, cache, runtime, or policy can be used here. + let buffers = io_policy.fetch(&ranges)?; + decoder.push_ranges(ranges, buffers)?; + } + DecodeResult::Data(batch) => consume(batch)?, + DecodeResult::Finished => break, + } +} +``` + +This is a push-style storage API: the application asks the decoder to advance, +then pushes bytes into it when requested. "Push" here describes the boundary +between storage and decoding, not the execution model used by the rest of the +query engine. + +## Putting the design into arrow-rs + +The [`ParquetPushDecoder`] implementation was released in `arrow-rs` `57.1.0`. +It was motivated by two related problems: maintaining parallel synchronous and +asynchronous readers, and repeated requests from users who needed more control +over Parquet I/O. + +A Sans I/O Parquet decoder cannot simply pause in the middle of a normal +function and wait for a read. It must preserve enough state to explain what it +needs and later resume after the caller supplies the bytes. The arrow-rs +implementation therefore models decoding as an explicit state machine. Its +[`try_decode`] method advances until it produces a batch, finishes, or +returns `NeedsData(Vec>)`. [`push_ranges`] adds reference-counted +`Bytes` buffers without prescribing how they were fetched. + +The explicit state machine pays off in reuse. The existing asynchronous +[`ParquetRecordBatchStream`] was [rewritten as a thin adapter][pr-8159] around +the push decoder: it fetches each `NeedsData` request through +`AsyncFileReader`, pushes +the buffers back, and exposes the familiar stream API. Applications that do +not need custom I/O still get a convenient reader, while format behavior lives +in one decoder core. + +The decoding loop shown earlier starts after the file metadata is available. Footer and metadata +decoding follow the same pattern through [`ParquetMetaDataPushDecoder`], so the +application can control I/O from the first footer read through the final data +page. + +## One decoder, many I/O policies + +To demonstrate what the separation enables, our [prototype driver] uses the +same Arrow Rust decoder with four deliberately different policies: + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PolicyPhysical readsPrimary benefitPrimary cost
Exact rangeRead exactly the requested rangesMinimum over-read and bufferingPotentially many requests
Request coalescingMerge adjacent or nearby rangesFewer physical requestsExtra bytes and buffering
Read-aheadExtend requested ranges and retain unused bytesFewer future stalls and more cache hitsMore bandwidth and memory
Full-file cacheFetch the file once and return zero-copy slicesZero-copy hits on subsequent requests (models a buffer manager or hot cache)Buffers the entire file
+
+ +These policies optimize different things; the point is not that one always +wins. The point is that none requires a second Parquet implementation. A local +backend can use `io_uring`; an object-store backend can coalesce or race reads; +a database cache can return already-resident buffers by reference. The decoder +continues to request the same logical ranges and decode the same format. + +This separation also makes measurement clearer. The application can account +independently for logical ranges requested by the decoder, physical storage +requests after policy decisions, bytes over-read, and peak buffered memory. + +## What existing readers expose + +We surveyed eight production-grade open-source analytic file readers. The +snapshot below reflects the versions examined for the paper in mid-2026. Every +reader offered an integrated I/O path; only Arrow Rust offered a production +public Sans I/O API, while cuDF exposed an experimental hybrid path. + +
+ + + + + + + + + + + + + + + + + + + +
ImplementationFormatPublic Sans I/O pathPrefetch / coalescing
Apache Arrow C++ParquetNoInternal and external
Apache Parquet JavaParquetNoInternal
Apache Arrow GoParquetNoNone
Apache Arrow RustParquetYesExternal
cuDF (RAPIDS)ParquetExperimentalInternal; experimental external path
hyparquetParquetNoInternal
VortexVortexNoInternal and external
LanceLanceNoInternal and external
+
+ +Many of these are well-designed readers with extensible storage interfaces, +zero-copy buffers, and sophisticated prefetchers. The distinction is about +control: even when the byte source is pluggable, an integrated decoder still +decides the read schedule. Some systems, including DuckDB and Polars, go +further and maintain embedded readers tightly coupled to their own runtimes and +buffer managers. Those readers can be very effective, but their format logic +is not available as a reusable public library. + +## Why streaming CSV and JSON already look familiar + +Sans I/O is not a new idea. The term came from the Python networking community, +where protocol libraries such as `h11` and `h2` operate as state machines over +caller-provided bytes. Streaming CSV and JSON parsers commonly use the same +shape: accept a byte chunk, emit records or tokens, and report when more input +is needed. + +These formats have an easier answer to "what bytes are needed next?": usually, +the next bytes in the stream. Analytic formats such as Parquet must request +non-contiguous footers, metadata blocks, indexes, and selected pages. The idea +is the same, but the API must communicate byte ranges rather than simply ask +for another sequential chunk. + +## Tradeoffs and design guidance + +Sans I/O moves complexity to an explicit boundary; it does not make that +complexity disappear. + +- **The decoder is harder to implement.** State that was implicit in a call + stack or async future must become an explicit, testable state machine. +- **The host must drive the feedback loop.** Applications with no special I/O + requirements should not have to write that loop themselves, so libraries + should ship synchronous and asynchronous adapters. +- **Performance is not automatic.** Sans I/O enables a system-specific policy; + the application still has to choose and implement a good one. +- **The boundary must preserve zero-copy operation.** Callers should be able to + provide reference-counted or otherwise borrowed resident buffers, including + data fetched speculatively. + +For new analytic formats, we recommend making the Sans I/O state machine the +decoder core, then layering blocking, async, memory-mapped, and prefetching +readers on top. Format-specific choices such as pruning remain inside that +core. Storage-specific choices such as request size, concurrency, caching, and +buffer ownership remain outside. + +## Conclusion + +A file reader API is more than a convenience wrapper. It determines whether a +database can combine shared, well-tested format logic with the I/O and memory +architecture that makes the database distinctive. + +Tightly integrated readers are easy to start with and remain useful as +adapters. But when they are the only API, applications must either accept the +reader's policy or build another decoder. Sans I/O provides a third option: one +reusable implementation of the format, with I/O orchestration owned by each +host system. That is a better fit for the increasingly composable data systems +being built around Arrow and Parquet. + +The paper *Implementing Analytic File Formats Sans I/O* is by Andrew Lamb, +Xudong Wang, Neil Conway, and Daniël Heres and appears at CIDR 2027. The paper +is licensed under [CC BY 4.0]. We thank InfluxData, Massive, and Coralogix for +supporting this work. + +[Apache Parquet]: https://parquet.apache.org/ +[arrow-rs-parquet]: https://github.com/apache/arrow-rs/tree/57.1.0/parquet +[`ParquetPushDecoder`]: https://docs.rs/parquet/57.1.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html +[`try_decode`]: https://docs.rs/parquet/57.1.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html#method.try_decode +[`push_ranges`]: https://docs.rs/parquet/57.1.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html#method.push_ranges +[`ParquetRecordBatchStream`]: https://docs.rs/parquet/57.1.0/parquet/arrow/async_reader/struct.ParquetRecordBatchStream.html +[pr-8159]: https://github.com/apache/arrow-rs/pull/8159 +[`ParquetMetaDataPushDecoder`]: https://docs.rs/parquet/57.1.0/parquet/file/metadata/struct.ParquetMetaDataPushDecoder.html +[prototype driver]: https://github.com/xudong963/push_decoder_policies +[CC BY 4.0]: https://creativecommons.org/licenses/by/4.0/ diff --git a/img/sans-io-analytic-formats/architecture.png b/img/sans-io-analytic-formats/architecture.png new file mode 100644 index 00000000000..36f1590bda2 Binary files /dev/null and b/img/sans-io-analytic-formats/architecture.png differ diff --git a/img/sans-io-analytic-formats/parquet-state-machine.png b/img/sans-io-analytic-formats/parquet-state-machine.png new file mode 100644 index 00000000000..58482119473 Binary files /dev/null and b/img/sans-io-analytic-formats/parquet-state-machine.png differ