Repository navigation
Support big-endian platforms (s390x) in the scalar codecs - #85
Merged
Merged
Conversation
The scalar encode/decode paths copied integers to and from the byte stream with memcpy, which silently assumes a little-endian host. On big-endian systems such as s390x this produced corrupted streams. Add small svb_load_le/svb_store_le helpers that keep using memcpy on little-endian hosts (no codegen change) and fall back to explicit byte shifts on big-endian hosts. Add an s390x CI job running under QEMU.
- windows-2025 no longer ships Visual Studio 2022; let CMake pick the installed Visual Studio instead of hardcoding the generator. - Replace leftover SIMDJSON_* options (ignored by this project) with a BUILD_SHARED_LIBS axis so the static/shared matrix tests something. - Bump actions/checkout to v7 and docker/setup-qemu-action to v4 (Node 20 deprecation).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #83.
The scalar encode/decode paths used
memcpyto move 2/3/4-byte values to and from the stream, which assumes a little-endian host. On big-endian systems (e.g. s390x, as reported for vbz_compression in Debian) this corrupts the data andunitfails.This PR:
src/streamvbyte_endian.hwithsvb_load_le/svb_store_le: plainmemcpyon little-endian hosts (so generated code is unchanged there) and explicit byte shifts when__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__;streamvbyte_{encode,decode}.c,streamvbytedelta_{encode,decode}.c,streamvbyte_0124_{encode,decode}.c). The patch in the issue editedstreamvbytedelta.c, which isn't built; the delta codec actually lives instreamvbytedelta_{encode,decode}.c;Verified locally under s390x emulation:
unitfails on master and passes with this change (-Wall -Werror). Encoded bytes for the standard, 0124 and delta formats are identical on s390x and on little-endian hosts, so streams can be moved between platforms. Thanks to @pranavkaruvally for the report and the initial patch.