Skip to content

Commit 19f771f

Browse files
connortsui20claude
andauthored
perf: allow cross-crate inlining of BitBuffer accessors (#9285)
Related: #9259 ## Rationale for this change `Buffer<T>` and `BufferMut<T>` are generic, so their MIR travels in the rlib and a downstream crate inlines them without LTO. `BitBuffer`, `BitBufferMut`, `BitBufferView` and `BitBufferMutView` are concrete, so a method without `#[inline]` reaches a downstream crate as a declaration only. The thin ones cost a real call per invocation, and the caller loses the offset and length constants it needs to fold the surrounding code. Note that `#[inline]` is not what enables inlining across codegen units inside a crate. MIR inlining runs before partitioning, and `profile.bench` sets `lto = false`, which is thin-local LTO rather than no LTO. Both already inline small functions across a codegen unit boundary within `vortex-buffer`. The attribute only matters across the crate boundary, which is where every measurement below was taken. ## What changes are included in this PR? `#[inline]` on the thin wrappers of those four types: constructors, iterator factories, and the methods that forward a slice, an offset and a length. Methods with a body worth outlining keep their current behavior, including `append_buffer`, whose bitvec fallback path is too large to justify inlining for the 1.6% it measured. Marginal instructions per iteration, measured with callgrind against a probe crate that calls `vortex-buffer` across a crate boundary, built at the `profile.bench` settings: | kernel | before | after | fat LTO | | --- | --- | --- | --- | | `BitBuffer::slice` | 8972 | 6415 | 5390 | | `BitBufferView::slice` | 3017 | 2186 | 1994 | `slice_vortex_buffer` measures 1.904us to 1.829us of wall time, and stops intermittently landing on a slower 2.12us mode. The wall-clock gain is much smaller than the instruction-count gain because these paths are bound by refcount atomics rather than by instruction issue. CodSpeed measures instruction counts, so expect its numbers to sit closer to the table than to the wall time. <details> <summary>Measurement method, and the parts of the LTO win this does not reach</summary> The probe is a separate crate that calls `vortex-buffer` over a real crate boundary, so the cross-crate path is the one under test. Each kernel runs at two iteration counts under `valgrind --tool=callgrind` and the totals are differenced, which cancels process startup, CPU feature warmup and setup allocations. This matches what CodSpeed's Simulation mode reports. `BitBuffer::set_indices` goes 77702 to 69503 at `codegen-units = 1`. At `codegen-units = 16` the baseline lands on the faster value about half the time depending on how thin-local LTO's import decisions fall, so the change makes a previously partition-dependent win reliable rather than producing a new one. The remaining fat LTO gap on these benchmarks is not cross-crate inlining, and `#[inline]` cannot reach it: - `set_slices` is 1.68x, and it is arrow's `BitSliceIterator`. Ten `#[inline]` attributes on the `BitSliceIterator` and `UnalignedBitChunk` chain recover 13729 to 10026 with no LTO. That belongs upstream in arrow-rs. - `from_iter` and `bitand_owned` are 1.9x. Nightly `-Zcross-crate-inline-threshold=always` does not move either one, so no amount of MIR availability explains them. Fat LTO is partially rescuing a per-bit read-modify-write loop by unrolling it. The real fix is that `BitBuffer::from_iter` costs 5.19 instructions per bit while `BitBufferMut::from(&[bool])` does the same job at 0.19 through the word-packing kernels in `pack.rs`. Follow-up. - `value_vortex_buffer` and `value_arrow_buffer` both reported +56.8% on #9259. The probe measures both at exactly 147467 instructions in every profile. That row is divan overhead. Thin LTO was measured as an alternative and rejected: 0 to 2.5% across these kernels for 2.5x the bench build time. </details> Signed-off-by: "Connor Tsui" <connor@spiraldb.com> Co-authored-by: Claude <noreply@anthropic.com>
1 parent fed7038 commit 19f771f

4 files changed

Lines changed: 55 additions & 0 deletions

File tree

vortex-buffer/src/bit/buf.rs

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,7 @@ impl BitBuffer {
9595
/// Create a new `BoolBuffer` backed by a [`ByteBuffer`] with `len` bits in view.
9696
///
9797
/// Panics if the buffer is not large enough to hold `len` bits.
98+
#[inline]
9899
pub fn new(buffer: ByteBuffer, len: usize) -> Self {
99100
assert!(
100101
buffer.len() * 8 >= len,
@@ -115,6 +116,7 @@ impl BitBuffer {
115116
/// the given `offset` (in bits).
116117
///
117118
/// Panics if the buffer is not large enough to hold `len` bits after the offset.
119+
#[inline]
118120
pub fn new_with_offset(buffer: ByteBuffer, len: usize, offset: usize) -> Self {
119121
assert!(
120122
len.saturating_add(offset) <= buffer.len().saturating_mul(8),
@@ -142,6 +144,7 @@ impl BitBuffer {
142144
}
143145

144146
/// Create a new `BoolBuffer` of length `len` where all bits are set (true).
147+
#[inline]
145148
pub fn new_set(len: usize) -> Self {
146149
let words = len.div_ceil(8);
147150
let buffer = buffer![0xFF; words];
@@ -154,6 +157,7 @@ impl BitBuffer {
154157
}
155158

156159
/// Create a new `BoolBuffer` of length `len` where all bits are unset (false).
160+
#[inline]
157161
pub fn new_unset(len: usize) -> Self {
158162
let words = len.div_ceil(8);
159163
let buffer = Buffer::zeroed(words);
@@ -171,11 +175,13 @@ impl BitBuffer {
171175
}
172176

173177
/// Create a new empty `BitBuffer`.
178+
#[inline]
174179
pub fn empty() -> Self {
175180
Self::new_set(0)
176181
}
177182

178183
/// Create a new `BitBuffer` of length `len` where all bits are set to `value`.
184+
#[inline]
179185
pub fn full(value: bool, len: usize) -> Self {
180186
if value {
181187
Self::new_set(len)
@@ -271,6 +277,7 @@ impl BitBuffer {
271277
}
272278

273279
/// Clear all bits in the buffer, preserving existing capacity.
280+
#[inline]
274281
pub fn clear(&mut self) {
275282
self.buffer.clear();
276283
self.len = 0;
@@ -345,6 +352,7 @@ impl BitBuffer {
345352
/// for `len` bits.
346353
///
347354
/// Panics if the slice would extend beyond the end of the buffer.
355+
#[inline]
348356
pub fn slice(&self, range: impl RangeBounds<usize>) -> Self {
349357
let (byte_offset, meta) = BitBufferMeta::new(self.offset, self.len).slice(range);
350358

@@ -376,13 +384,15 @@ impl BitBuffer {
376384
}
377385

378386
/// Access chunks of the buffer aligned to 8 byte boundary as [prefix, \<full chunks\>, suffix]
387+
#[inline]
379388
pub fn unaligned_chunks(&self) -> UnalignedBitChunk<'_> {
380389
UnalignedBitChunk::new(self.buffer.as_slice(), self.offset, self.len)
381390
}
382391

383392
/// Access chunks of the underlying buffer as 8 byte chunks with a final trailer
384393
///
385394
/// If you're performing operations on a single buffer, prefer [BitBuffer::unaligned_chunks]
395+
#[inline]
386396
pub fn chunks(&self) -> BitChunks<'_> {
387397
BitChunks::new(self.buffer.as_slice(), self.offset, self.len)
388398
}
@@ -413,6 +423,7 @@ impl BitBuffer {
413423
/// which logical bit position holds that rank.
414424
///
415425
/// Returns `None` if `nth` is greater than or equal to the number of set bits.
426+
#[inline]
416427
pub fn select(&self, nth: usize) -> Option<usize> {
417428
bit_select(self.buffer.as_slice(), self.offset, self.len, nth)
418429
}
@@ -424,16 +435,19 @@ impl BitBuffer {
424435
}
425436

426437
/// Iterator over bits in the buffer
438+
#[inline]
427439
pub fn iter(&self) -> BitIterator<'_> {
428440
BitIterator::new(self.buffer.as_slice(), self.offset, self.len)
429441
}
430442

431443
/// Iterator over set indices of the underlying buffer
444+
#[inline]
432445
pub fn set_indices(&self) -> BitIndexIterator<'_> {
433446
BitIndexIterator::new(self.buffer.as_slice(), self.offset, self.len)
434447
}
435448

436449
/// Iterator over set slices of the underlying buffer
450+
#[inline]
437451
pub fn set_slices(&self) -> BitSliceIterator<'_> {
438452
BitSliceIterator::new(self.buffer.as_slice(), self.offset, self.len)
439453
}
@@ -484,11 +498,13 @@ impl BitBuffer {
484498

485499
impl BitBuffer {
486500
/// Returns the offset, len and underlying buffer.
501+
#[inline]
487502
pub fn into_inner(self) -> (usize, usize, ByteBuffer) {
488503
(self.offset, self.len, self.buffer)
489504
}
490505

491506
/// Attempt to convert this `BitBuffer` into a mutable version.
507+
#[inline]
492508
pub fn try_into_mut(self) -> Result<BitBufferMut, Self> {
493509
match self.buffer.try_into_mut() {
494510
Ok(buffer) => Ok(BitBufferMut::from_buffer(buffer, self.offset, self.len)),
@@ -510,6 +526,7 @@ impl From<Vec<bool>> for BitBuffer {
510526
}
511527

512528
impl FromIterator<bool> for BitBuffer {
529+
#[inline]
513530
fn from_iter<T: IntoIterator<Item = bool>>(iter: T) -> Self {
514531
BitBufferMut::from_iter(iter).freeze()
515532
}

vortex-buffer/src/bit/buf_mut.rs

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@ pub struct BitBufferMut {
102102

103103
impl BitBufferMut {
104104
/// Create new bit buffer from given byte buffer and logical bit length
105+
#[inline]
105106
pub fn from_buffer(buffer: ByteBufferMut, offset: usize, len: usize) -> Self {
106107
assert!(
107108
len <= buffer.len() * 8,
@@ -125,6 +126,7 @@ impl BitBufferMut {
125126
}
126127

127128
/// Create a new empty mutable bit buffer with requested capacity (in bits).
129+
#[inline]
128130
pub fn with_capacity(capacity: usize) -> Self {
129131
Self {
130132
buffer: BufferMut::with_capacity(capacity.div_ceil(8)),
@@ -134,6 +136,7 @@ impl BitBufferMut {
134136
}
135137

136138
/// Create a new mutable buffer with requested `len` and all bits set to `true`.
139+
#[inline]
137140
pub fn new_set(len: usize) -> Self {
138141
Self {
139142
buffer: buffer_mut![0xFF; len.div_ceil(8)],
@@ -143,6 +146,7 @@ impl BitBufferMut {
143146
}
144147

145148
/// Create a new mutable buffer with requested `len` and all bits set to `false`.
149+
#[inline]
146150
pub fn new_unset(len: usize) -> Self {
147151
Self {
148152
buffer: BufferMut::zeroed(len.div_ceil(8)),
@@ -158,6 +162,7 @@ impl BitBufferMut {
158162
}
159163

160164
/// Create a new mutable buffer with requested `len` and all bits set to `value`.
165+
#[inline]
161166
pub fn full(value: bool, len: usize) -> Self {
162167
if value {
163168
Self::new_set(len)
@@ -247,11 +252,13 @@ impl BitBufferMut {
247252
}
248253

249254
/// Return the underlying byte buffer.
255+
#[inline]
250256
pub fn inner(&self) -> &ByteBufferMut {
251257
&self.buffer
252258
}
253259

254260
/// Consumes the buffer and return the underlying byte buffer.
261+
#[inline]
255262
pub fn into_inner(self) -> ByteBufferMut {
256263
self.buffer
257264
}
@@ -299,6 +306,7 @@ impl BitBufferMut {
299306
}
300307

301308
/// Reserve additional bit capacity for the buffer.
309+
#[inline]
302310
pub fn reserve(&mut self, additional: usize) {
303311
let required_bits = self.offset + self.len + additional;
304312
let required_bytes = required_bits.div_ceil(8); // Rounds up.
@@ -308,6 +316,7 @@ impl BitBufferMut {
308316
}
309317

310318
/// Clears the bit buffer (but keeps any allocated memory).
319+
#[inline]
311320
pub fn clear(&mut self) {
312321
// Also clear the byte buffer (not just `len`) so the "bits beyond len are zero"
313322
// invariant holds; `append_false` and `append_buffer` rely on it.
@@ -415,6 +424,7 @@ impl BitBufferMut {
415424
/// Truncate the buffer to the given length.
416425
///
417426
/// If the given length is greater than the current length, this is a no-op.
427+
#[inline]
418428
pub fn truncate(&mut self, len: usize) {
419429
if len > self.len {
420430
return;
@@ -609,16 +619,19 @@ impl BitBufferMut {
609619
}
610620

611621
/// Freeze the buffer in its current state into an immutable `BoolBuffer`.
622+
#[inline]
612623
pub fn freeze(self) -> BitBuffer {
613624
BitBuffer::new_with_offset(self.buffer.freeze(), self.len, self.offset)
614625
}
615626

616627
/// Get the underlying bytes as a slice
628+
#[inline]
617629
pub fn as_slice(&self) -> &[u8] {
618630
self.buffer.as_slice()
619631
}
620632

621633
/// Get the underlying bytes as a mutable slice
634+
#[inline]
622635
pub fn as_mut_slice(&mut self) -> &mut [u8] {
623636
self.buffer.as_mut_slice()
624637
}

vortex-buffer/src/bit/meta.rs

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ impl BitBufferMeta {
1919
///
2020
/// Panics if `offset >= 8`. Use [`from_raw_offset`](Self::from_raw_offset) to normalize a
2121
/// larger offset.
22+
#[inline]
2223
pub fn new(offset: usize, len: usize) -> Self {
2324
assert!(offset < 8, "BitBufferMeta offset must be < 8, got {offset}");
2425
Self { offset, len }
@@ -29,6 +30,7 @@ impl BitBufferMeta {
2930
///
3031
/// Returns `(byte_offset, meta)` so the caller can slice its backing buffer by `byte_offset`
3132
/// and store the remaining sub-byte offset in `meta`.
33+
#[inline]
3234
pub fn from_raw_offset(offset: usize, len: usize) -> (usize, Self) {
3335
(
3436
offset / 8,
@@ -44,6 +46,7 @@ impl BitBufferMeta {
4446
/// # Panics
4547
///
4648
/// Panics if the range is out of bounds or its end precedes its start.
49+
#[inline]
4750
pub fn slice(&self, range: impl RangeBounds<usize>) -> (usize, Self) {
4851
let start = match range.start_bound() {
4952
Bound::Included(&start) => start,

0 commit comments

Comments
 (0)