Skip to content

perf(parquet): pre-size values in OffsetBuffer::extend_from_dictionary #10690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AarryaSaraf wants to merge 3 commits into from
Closed

perf(parquet): pre-size values in OffsetBuffer::extend_from_dictionary #10690

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6b4cc75
perf(parquet): pre-size values in OffsetBuffer::extend_from_dictionary
AarryaSaraf Aug 14, 2026
a5f6a21
Address review: drop the size-hint test, make the arithmetic total
AarryaSaraf Aug 18, 2026
294292b
Merge branch 'main' into dict-extend-reserve-values
etseidl Aug 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions parquet/src/arrow/buffer/offset_buffer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,15 @@ impl<I: OffsetSizeTrait> OffsetBuffer<I> {
) -> Result<()> {
self.offsets.reserve(keys.len());

// Pre-size `values` from the dictionary's average value length. Exact when
// values are uniformly sized, which is the large-value case this targets.
// `try_reserve` because a skewed dictionary can inflate the estimate above
// the real output size, and the dictionary comes from an untrusted file.
let dict_len = dict_offsets.len().saturating_sub(1);
if let Some(avg_len) = dict_values.len().checked_div(dict_len) {
let _ = self.values.try_reserve(avg_len.saturating_mul(keys.len()));
}

for key in keys {
let index = key.as_usize();
if index + 1 >= dict_offsets.len() {
Expand Down
Loading