[Bugfix] Fix BatchLoad O_DIRECT alignment and EOF short-read handling#4
Merged
Merged
Conversation
The merged-pread path introduced in 606e4b3 ("store: merge adjacent preads in BucketStorageBackend::BatchLoad") was broken when buckets were opened via UringFile (i.e. when USE_URING + MOONCAKE_OFFLOAD_USE_URING both enabled), causing cache hit rate to collapse from 1.0 to ~0.12 in stage-4 SSD-offload tests. Two bugs, both in BucketStorageBackend::BatchLoad's merged-group branch: 1. O_DIRECT alignment violation: vector_read was called with seg_offset (key-boundary derived, generally not 4096-aligned), size = seg_size (unaligned), and buffer = std::vector<char> (16-byte aligned only). io_uring readv under O_DIRECT requires all three to be 4096-aligned, so every merged group CQE returned EINVAL. This silently dropped the whole batch when a merge group touched a UringFile. 2. EOF short-read false positive: even after alignment was fixed, the strict check `got == aligned_size` failed for merge groups whose aligned_end overshoots the bucket file by up to (kDirectIOAlignment - 1) bytes — a normal EOF condition. pread returns the real file remainder, which is less than aligned_size but still covers every key's actual data range. This caused FILE_READ_FAIL on ~2% of BatchGet calls, dropping stage-4 hit rate to ~0.86. Fix (mirror the single-key path's alignment strategy): - align_down(seg_offset, kDirectIOAlignment) for the read offset - align_up(seg_end, kDirectIOAlignment) for the read end - posix_memalign(kDirectIOAlignment, aligned_size) for the buffer (replaces std::vector<char>; matches the WriteBucket UringFile path) - Relax the size check from `got == aligned_size` to `got >= (seg_end - aligned_offset)`. Each key's memcpy stays in bounds because [key_offset, key_offset + dest_slice.size) is always a subset of [seg_offset, seg_end), which is itself a subset of the bytes actually read. - Scatter loop uses `key_offset - aligned_offset` (not seg_offset) to account for the alignment padding before seg_offset. Also add a defensive short-read check on the single-key UringFile path, which previously overwrote read_res with plan.dest_slice.size without verifying UringFile actually read enough bytes. This path did not manifest bugs in tested workloads but could silently read uninitialized memory if a bucket file were ever truncated. Verified in stage-4 SSD-offload test (log-ssd-store-ttft-0719-1729): - CQE errors: 0 (was 67/store pod before alignment fix) - merged read short of data: 0 (was 2/store pod before EOF fix) - inference-side SSD read failures: 0 (was 9) - cache hit rate: 1.00 (was 0.12 then 0.86) - store-side BatchGet p50: 60ms (was ~190ms under PosixFile buffered I/O) - BatchLoad throughput: 4.6-4.9 GB/s (was ~1.4 GB/s) Safe for buffered I/O (PosixFile): the extra aligned-bytes are simply ignored, and the relaxed size check still rejects genuine truncations.
pingzhuu
approved these changes
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The merged-pread path introduced in 606e4b3 ("store: merge adjacent preads in BucketStorageBackend::BatchLoad") was broken when buckets were opened via UringFile (i.e. when USE_URING + MOONCAKE_OFFLOAD_USE_URING both enabled), causing cache hit rate to collapse from 1.0 to ~0.12 in stage-4 SSD-offload tests.
Two bugs, both in BucketStorageBackend::BatchLoad's merged-group branch:
O_DIRECT alignment violation: vector_read was called with seg_offset (key-boundary derived, generally not 4096-aligned), size = seg_size (unaligned), and buffer = std::vector (16-byte aligned only). io_uring readv under O_DIRECT requires all three to be 4096-aligned, so every merged group CQE returned EINVAL. This silently dropped the whole batch when a merge group touched a UringFile.
EOF short-read false positive: even after alignment was fixed, the strict check
got == aligned_sizefailed for merge groups whose aligned_end overshoots the bucket file by up to (kDirectIOAlignmentFix (mirror the single-key path's alignment strategy):
got == aligned_sizetogot >= (seg_end - aligned_offset). Each key's memcpy stays in bounds because [key_offset, key_offset + dest_slice.size) is always a subset of [seg_offset, seg_end), which is itself a subset of the bytes actually read.key_offset - aligned_offset(not seg_offset) to account for the alignment padding before seg_offset.Also add a defensive short-read check on the single-key UringFile path, which previously overwrote read_res with plan.dest_slice.size without verifying UringFile actually read enough bytes. This path did not manifest bugs in tested workloads but could silently read uninitialized memory if a bucket file were ever truncated.
Verified in stage-4 SSD-offload test (log-ssd-store-ttft-0719-1729):
Safe for buffered I/O (PosixFile): the extra aligned-bytes are simply ignored, and the relaxed size check still rejects genuine truncations.