Add optional sparse data sync to minimize image size

[martinp26]

Provide a runtime configuration key to let the data sync that installs the image payload turn sequences of null bytes in the source into holes on the target:

sync:
sparse: true

The key is off by default. It adds the --sparse option to the rsync call. It is skipped with a warning rather than failing the build if it cannot be used, which is the case for rsync releases before 3.1.3 because they reject --sparse together with --inplace, and for a caller that asked for --preallocate because that allocates every file in full before writing it.

The option alone is not enough on a filesystem that performs speculative preallocation. rsync only seeks forward across a null sequence, it does not punch a hole, and on xfs the skipped range falls inside the preallocation beyond the end of file and is recorded as an allocated but unwritten extent. Such an extent reads as zero and is reported as a hole by SEEK_HOLE, yet it still occupies space in the target image.

A filesystem implementation can therefore declare mount options that its install time mount needs for a sparse sync to be effective, via the new sparse_install_mount_options attribute. These apply to the mount the payload is synced into and are not passed on to the fstab of the image, so they do not become mount options of the running system. An explicit setting of the same option in the fsmountoptions attribute of the image description takes precedence.

The xfs implementation declares the minimum of allocsize=4k for maximal size savings.

Measured with a payload of 64k aligned binaries carrying a zero tail per segment, as produced for architectures with a 64k page size. The filesystem accounts 244MB used and the image file occupies 581MB without the key. With the key it is 119MB and 457MB, with no unwritten extents left. The extent count drops at the same time, because the unwritten bridges are extents of their own.

Assisted-by: Kiro:claude-opus-5

[schaefi]

@martinp26 Thanks much, the idea is nice and the implementation looks good. I have a few questions prior merge though:

1. This seems to be done with a testing focus set on XFS, correct ?
2. Can we be sure that allocsize=4k during a sparse enabled data sync is not causing trouble with the default allocsize (different) during runtime ? I'm asking in the context if that has been human tested ?
3. All other filesystem implementations would run the sparse sync with the same options during sync and during runtime. Was there any testing done that shows a significant performance improve ? If not can this be mentioned in the documentation somewhere
4. Was there any testing with sparse enabled sync on layered images like LVM, encryption (LUKS), raid, etc ... ? If not can this then also be mentioned in the docs. I believe there is a risk on sparse sync when additional layers are included. I also believe that a sparse sync has no benefit on encrypted images.

Your thoughts please.

Thanks much

[martinp26]

Hi @schaefi, thanks for looking into this.

A few quick answers:

1. Yes, I did most testing on XFS as our Amazon Linux AMIs use XFS primarily today. There was additional testing outside of kiwi to understand the pre-allocation behavior with xfs, ext4, and btrfs with the outcome that only xfs needs special handling. It did similar testing before for a librpm change and saw the preallocation limitations there as well (Sparse file handling to save space for ARM 64k binaries with small segments rpm-software-management/rpm#4258).
2. I did create images with this setting that did boot without trouble and where boot-speed did not regress, if that is the core of your question. If you are asking if the mount settings would leak into the target image, there is a dedicated tests for that.
3. I would not expect a perf. improvement from this change directly. Rather, the actual disk usage of the target image would be lower because the null runs would not be allocated in the image. Depending on your storage backend, that will lead to lower resource utilization or lower transfer costs etc. in the backend. I do not directly expect boot-speed benefits because our test systems are not IO bound during boot.
   Not sure I misread your question and you had image creation speed in mind? I saw a few numbers during testing and in my setup (~8GB images with ~1.5GB payload) the additional content scanning from rsync was in the noise.
4. I have not looked into that. Is there a set of existing test recipes available that I could create to test this? Regarding your expected benefits for layered or encrypted setups, if the sparse areas are landing in a filesystem that supports sparseness, I think it should lead to lower disk allocations in that layer. That might either lead to a smaller needed volume size or few executed IOs during operations, so there could be benefit. I don't think layering itself or encryption would prevent the benefit.
   If we use EROFS layers or some other form of compression, I would expect that compression and sparseness would eat each others' benefit as compression would also get rid of the zero runs with another mechanism.

Let me iterate a bit on the testing story.

[schaefi]

Not sure I misread your question and you had image creation speed in mind?

sorry my question was not very clear here. You already answered it. With "performance" I meant the disk usage on the storage backend. Thanks makes sense to me.

I have not looked into that. Is there a set of existing test recipes available that I could create to test this?

yes, it would be imho interesting what happens with sparse sync enabled on the following integration tests

• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-luks
• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-lvm
• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-raid

You find the respective image descriptions in source below kiwi/build-tests. I believe the distro doesn't matter much so you should be able to rebuild this type of image also with Amazon Linux or another OS of your choice

I believe sparse sync on encrypted will not have many null bytes to write as kiwi also randomize the target. For the others I agree with you that the storage backend will benefit in the same way as with less complex layouts.

Don't get me wrong, I don't want to burn your time in testing. I just wanted to make sure enabling the sparse sync feature is not causing some sort of corruption at the end of the sync chain.

The feature is great, let's merge when you believe we are good to go :)

Thanks much

[martinp26]

Thanks :-).

I understand this is a complex change and you need to protect kiwi's stability. Totally warranted!

I believe sparse sync on encrypted will not have many null bytes to write as kiwi also randomize the target. For the others I agree with you that the storage backend will benefit in the same way as with less complex layouts.

Right, just saw that you enforce randomizing. Unless I am mixing things up here, rsync would still see the zero runs in the source files to be synced and would create sparse representations in the target. Only the outer image would not see a size reduction as the unwritten areas would not manifest as sparse holes or zero runs in it. The only benefit would be within the file system, where the user has a little bit more free space now from the sparse savings. There are no direct savings on the out image itself.