Skip to content

[Bugfix] Fix BatchLoad O_DIRECT alignment and EOF short-read handling#4

Merged
pingzhuu merged 1 commit into
opt/batchload-pread-mergefrom
fix/uring-merge-alignment
Jul 21, 2026
Merged

[Bugfix] Fix BatchLoad O_DIRECT alignment and EOF short-read handling#4
pingzhuu merged 1 commit into
opt/batchload-pread-mergefrom
fix/uring-merge-alignment

Conversation

@leaves-zwx

Copy link
Copy Markdown

The merged-pread path introduced in 606e4b3 ("store: merge adjacent preads in BucketStorageBackend::BatchLoad") was broken when buckets were opened via UringFile (i.e. when USE_URING + MOONCAKE_OFFLOAD_USE_URING both enabled), causing cache hit rate to collapse from 1.0 to ~0.12 in stage-4 SSD-offload tests.

Two bugs, both in BucketStorageBackend::BatchLoad's merged-group branch:

  1. O_DIRECT alignment violation: vector_read was called with seg_offset (key-boundary derived, generally not 4096-aligned), size = seg_size (unaligned), and buffer = std::vector (16-byte aligned only). io_uring readv under O_DIRECT requires all three to be 4096-aligned, so every merged group CQE returned EINVAL. This silently dropped the whole batch when a merge group touched a UringFile.

  2. EOF short-read false positive: even after alignment was fixed, the strict check got == aligned_size failed for merge groups whose aligned_end overshoots the bucket file by up to (kDirectIOAlignment

      1. bytes — a normal EOF condition. pread returns the real file remainder, which is less than aligned_size but still covers every key's actual data range. This caused FILE_READ_FAIL on ~2% of BatchGet calls, dropping stage-4 hit rate to ~0.86.

Fix (mirror the single-key path's alignment strategy):

  • align_down(seg_offset, kDirectIOAlignment) for the read offset
  • align_up(seg_end, kDirectIOAlignment) for the read end
  • posix_memalign(kDirectIOAlignment, aligned_size) for the buffer (replaces std::vector; matches the WriteBucket UringFile path)
  • Relax the size check from got == aligned_size to got >= (seg_end - aligned_offset). Each key's memcpy stays in bounds because [key_offset, key_offset + dest_slice.size) is always a subset of [seg_offset, seg_end), which is itself a subset of the bytes actually read.
  • Scatter loop uses key_offset - aligned_offset (not seg_offset) to account for the alignment padding before seg_offset.

Also add a defensive short-read check on the single-key UringFile path, which previously overwrote read_res with plan.dest_slice.size without verifying UringFile actually read enough bytes. This path did not manifest bugs in tested workloads but could silently read uninitialized memory if a bucket file were ever truncated.

Verified in stage-4 SSD-offload test (log-ssd-store-ttft-0719-1729):

  • CQE errors: 0 (was 67/store pod before alignment fix)
  • merged read short of data: 0 (was 2/store pod before EOF fix)
  • inference-side SSD read failures: 0 (was 9)
  • cache hit rate: 1.00 (was 0.12 then 0.86)
  • store-side BatchGet p50: 60ms (was ~190ms under PosixFile buffered I/O)
  • BatchLoad throughput: 4.6-4.9 GB/s (was ~1.4 GB/s)

Safe for buffered I/O (PosixFile): the extra aligned-bytes are simply ignored, and the relaxed size check still rejects genuine truncations.

The merged-pread path introduced in 606e4b3 ("store: merge adjacent
preads in BucketStorageBackend::BatchLoad") was broken when buckets
were opened via UringFile (i.e. when USE_URING + MOONCAKE_OFFLOAD_USE_URING
both enabled), causing cache hit rate to collapse from 1.0 to ~0.12 in
stage-4 SSD-offload tests.

Two bugs, both in BucketStorageBackend::BatchLoad's merged-group branch:

1. O_DIRECT alignment violation: vector_read was called with seg_offset
   (key-boundary derived, generally not 4096-aligned), size = seg_size
   (unaligned), and buffer = std::vector<char> (16-byte aligned only).
   io_uring readv under O_DIRECT requires all three to be 4096-aligned,
   so every merged group CQE returned EINVAL. This silently dropped the
   whole batch when a merge group touched a UringFile.

2. EOF short-read false positive: even after alignment was fixed, the
   strict check `got == aligned_size` failed for merge groups whose
   aligned_end overshoots the bucket file by up to (kDirectIOAlignment
   - 1) bytes — a normal EOF condition. pread returns the real file
   remainder, which is less than aligned_size but still covers every
   key's actual data range. This caused FILE_READ_FAIL on ~2% of BatchGet
   calls, dropping stage-4 hit rate to ~0.86.

Fix (mirror the single-key path's alignment strategy):
  - align_down(seg_offset, kDirectIOAlignment) for the read offset
  - align_up(seg_end, kDirectIOAlignment) for the read end
  - posix_memalign(kDirectIOAlignment, aligned_size) for the buffer
    (replaces std::vector<char>; matches the WriteBucket UringFile path)
  - Relax the size check from `got == aligned_size` to
    `got >= (seg_end - aligned_offset)`. Each key's memcpy stays in
    bounds because [key_offset, key_offset + dest_slice.size) is always
    a subset of [seg_offset, seg_end), which is itself a subset of the
    bytes actually read.
  - Scatter loop uses `key_offset - aligned_offset` (not seg_offset) to
    account for the alignment padding before seg_offset.

Also add a defensive short-read check on the single-key UringFile path,
which previously overwrote read_res with plan.dest_slice.size without
verifying UringFile actually read enough bytes. This path did not
manifest bugs in tested workloads but could silently read uninitialized
memory if a bucket file were ever truncated.

Verified in stage-4 SSD-offload test (log-ssd-store-ttft-0719-1729):
  - CQE errors: 0  (was 67/store pod before alignment fix)
  - merged read short of data: 0  (was 2/store pod before EOF fix)
  - inference-side SSD read failures: 0  (was 9)
  - cache hit rate: 1.00  (was 0.12 then 0.86)
  - store-side BatchGet p50: 60ms  (was ~190ms under PosixFile buffered I/O)
  - BatchLoad throughput: 4.6-4.9 GB/s  (was ~1.4 GB/s)

Safe for buffered I/O (PosixFile): the extra aligned-bytes are simply
ignored, and the relaxed size check still rejects genuine truncations.
@pingzhuu
pingzhuu merged commit ef9fe20 into opt/batchload-pread-merge Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants