Repository navigation
Conversation
overlaybd-apply wrote regular files through ext4, so in a LSMT layer
the file data was interleaved with the fs metadata that ext4 wrote
back whenever its cache flushed, scattering fsmeta all over the
data area.
A plain LSMT layer is now converted in two pas/ses. The tar
stream is read once, appending the padded data of every regular file to
the upper data file in tar order and spooling the headers to a
temporary file, with each file's data offset recorded in devmajor. Then
the ext4 fs is created and the headers are replayed metadata-only, as
TurboOCI does, mapping each file's blocks to its staged data through
RemoteData. The mappings are applied after the fs is synced, so a
metadata block reused for file data cannot be written back over it, and
the tail of a reused last block is zeroed. `overlaybd-commit --seal`
then appends the index and trailer in place, giving
Header | file data | fs metadata | index | Trailer
Signed-off-by: Yifan Yuan <tuji.yyf@outlook.com>
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open issues
discard()does not validate/clampoffset/countagainstm_size, andclear(block)indexes… · Newmem.data()is onlychar-aligned, but it’s used as storage for placement-new of… · NewerrnotoEIOeven whenpwritefails with a more specific error, which makes… · NewUnTarinstance allocated withnewis never deleted on either success or failure paths,… · New
What changed in this PR
Improves overlaybd-apply ext4 layer conversion by staging regular-file payload ahead of ext4 metadata, then replaying headers metadata-only and mapping file blocks to the staged payload to avoid metadata/data interleaving.
Changes:
- Add a two-pass “stage payload + replay metadata” extraction flow for plain LSMT append layers.
- Introduce
FsMetaFilefor sparse staging and compact appending of live ext4 metadata blocks. - Extend tar extraction and LSMT layer support to defer and then apply
RemoteDatamappings, plus add comprehensive tests and documentation.
| File | Description |
|---|---|
| src/tools/overlaybd-apply.cpp | Orchestrates staging/replay flow and temp header/fsmeta spooling. |
| src/overlaybd/tar/libtar.h | Adds staged extraction APIs and deferred mapping bookkeeping. |
| src/overlaybd/tar/libtar.cpp | Implements stage_tar, extract_staged, pending mappings, and block-tail handling. |
| src/overlaybd/tar/whiteout.cpp | Routes unlinks through new helper to keep pending mappings consistent. |
| src/overlaybd/tar/fsmeta_file.h | Declares FsMetaFile virtual file wrapper for fs metadata staging. |
| src/overlaybd/tar/fsmeta_file.cpp | Implements staging reads/writes, dirty tracking, and compact commit into base. |
| src/overlaybd/tar/test/test.cpp | Adds staged extraction + fsmeta tests and layout assertions. |
| src/overlaybd/lsmt/file.h | Exposes can_map_staged_data() capability query. |
| src/overlaybd/lsmt/file.cpp | Adds staged RemoteData validation/mapping and improves sealed index write padding. |
| src/overlaybd/lsmt/test/test.cpp | Adds staging/mapping behavior tests (reopen, seal, stacking, rejects). |
| docs/apply-data-layout.md | Documents the new ext4 userspace conversion layout and rationale (EN). |
| docs/apply-data-layout_zh.md | Documents the new ext4 userspace conversion layout and rationale (ZH). |
| docs/README.md | Mentions the ext4 optimized layout and links to the new doc. |
| CMake/Findphoton.cmake | Forces Photon to use shared liburing build and adds dependency ordering. |
Review effort: Lite
Comment on lines
+154
to
+155
| auto end = (offset + count + BLOCK_SIZE - 1) / BLOCK_SIZE; | ||
| for (size_t block = offset / BLOCK_SIZE; block < end; block++) { |
Comment on lines
+470
to
479
| photon::fs::fiemap probe(0, size, 0); | ||
| if (fout->fiemap(&probe) != 0) { | ||
| return -1; | ||
| } | ||
| std::vector<char> mem(sizeof(photon::fs::fiemap) + | ||
| probe.fm_mapped_extents * sizeof(photon::fs::fiemap_extent)); | ||
| auto fie = new (mem.data()) photon::fs::fiemap(0, size, probe.fm_mapped_extents); | ||
| if (fout->fiemap(fie) != 0) { | ||
| return -1; | ||
| } |
Comment on lines
+166
to
+168
| if (target->pwrite(buf, buffered, base + staged) != (ssize_t)buffered) { | ||
| LOG_ERRNO_RETURN(EIO, -1, "failed to stage ` bytes", buffered); | ||
| } |
Comment on lines
+241
to
+242
| auto tar = new UnTar(src_file, target, 0, 4096, base_file, gen_turboOCI); | ||
| if (tar->extract_all() < 0) { |
Copilot stopped reviewing on behalf of
BigVan due to an error
October 27, 2026 16:09
Member
Author
|
@WaberZhuang PTAL |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
overlaybd-apply wrote regular files through ext4, so in a LSMT layer the file data was interleaved with the fs metadata that ext4 wrote back whenever its cache flushed, scattering fsmeta all over the data area.
A plain LSMT layer is now converted in two pas/ses. The tar stream is read once, appending the padded data of every regular file to the upper data file in tar order and spooling the headers to a temporary file, with each file's data offset recorded in devmajor. Then the ext4 fs is created and the headers are replayed metadata-only, as TurboOCI does, mapping each file's blocks to its staged data through RemoteData. The mappings are applied after the fs is synced, so a metadata block reused for file data cannot be written back over it, and the tail of a reused last block is zeroed.
overlaybd-commit --sealthen appends the index and trailer in place, givingWhat this PR does / why we need it:
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #
Please check the following list: