Repository navigation
Align compacted data and indexes to 4 KiB - #474
Closed
liulanzheng wants to merge 1 commit into
Closed
liulanzheng wants to merge 1 commit into
liulanzheng wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No blocking issues were identified in offset accounting, ordinary and warp integration, or default-layout compatibility.
No open issues 🎉
What changed in this PR
Adds optional page-offset preservation to LSMT compaction, enabling direct mapping of complete logical 4 KiB pages from eligible uncompressed layers while keeping the default tightly packed layout.
Changes:
- Adds
--preserve-page-offsetand a matching commit option. - Pads local extents and aligns the index start in ordinary and warp commits.
- Documents the layout and adds tests for offsets, padding, zero extents, and reopened reads.
| File | Description |
|---|---|
| src/tools/overlaybd-commit.cpp | Exposes and forwards the new CLI flag. |
| src/overlaybd/lsmt/test/test.cpp | Tests ordinary and warp layouts with padding enabled and disabled. |
| src/overlaybd/lsmt/format_spec.md | Documents optional padding and unchanged format units. |
| src/overlaybd/lsmt/file.h | Adds the default-disabled commit option. |
| src/overlaybd/lsmt/file.cpp | Implements extent padding and index alignment in both commit paths. |
Review effort: Balanced
Member
|
We've also encountered a similar issue. We believe it's sufficient to simply align to the default 4K block size used by ext4 during mkfs, without needing to explicitly specify it via parameters. Using 4K padding effectively reduces the number of indices during compress_index. I'll be opening a PR shortly to fix this. |
Pad each non-zero local extent to a 4 KiB file offset during compaction and align the index start after the data. Apply the layout directly to ordinary and warp commits without a command-line option. Keep zero extents index-only, preserve remote mappings, and retain the existing 512-byte sector units and on-disk record format. Signed-off-by: Lanzheng Liu <liulanzheng@gmail.com>
liulanzheng
force-pushed
the
codex/compact-page-offset
branch
from
October 28, 2026 01:59
98e39d5 to
8b354eb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Align each non-zero local data extent and the index start in compacted LSMT output to 4 KiB. This gives block-aligned consumers a 4 KiB payload layout while retaining 512-byte logical I/O and mapping units.
The shared compaction path covers ordinary commits and merge/flatten output; warp commits apply the same alignment to local data. Explicit zero extents remain index-only, and remote mappings retain their original offsets. Zero padding costs at most 3584 bytes before each local extent, plus up to 3584 bytes before the index.
The on-disk records and reader addressing are unchanged. Existing layers remain readable; newly compacted output uses the aligned layout. Logical offsets are not rounded, so extents whose logical starts are not 4 KiB aligned do not necessarily share their physical offsets within a page.
For uncompressed layers, alignment applies to file offsets. For compressed output, it applies to offsets in the decoded LSMT stream, not the compressed backing file.
Validation:
Local tests used Debian 13 x86_64, Photon v0.9.6 and psync I/O (--nwrites=256 --layers=3 --vsize=8). The QEMU test runner used EPOLL/INIT_IO_NONE because io_setup is unsupported, and returned the GoogleTest exit status. These runner adjustments are outside the repository; local libaio validation was not performed.
Which issue(s) this PR fixes:
None.
Please check the following list: