Repository navigation
Produce reproducible native libraries for all language bindings - #2016
Draft
spacebear21 wants to merge 8 commits into
Draft
spacebear21 wants to merge 8 commits into
spacebear21 wants to merge 8 commits into
Conversation
Collaborator
Coverage Report for CI Build 40964389085Coverage remained the same at 87.547%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
spacebear21
force-pushed
the
native-build-contract
branch
from
October 29, 2026 22:07
4eb7f73 to
bd9af28
Compare
Each binding builds its native libraries in its own job, on its own
runner, with whatever toolchain that job sets up, so no shipped file
can be rebuilt bit for bit. Add one script that builds every shipped
target on Linux with one pinned toolchain:
contrib/native/build.sh <triple> <out-dir>
Nothing yet checks that build.sh produces the same bytes twice. Build each target twice, on separate runners. The second build runs from another checkout path, home and cargo home, all with a space in them, so any path that leaks into a library breaks the match, including one from a cache the toolchain keeps under the home directory. A compare job fails unless the second build matches the first one's hashes. Once they match, the compare job combines the two macOS slices with llvm-lipo into universal2-apple-darwin, the file Python ships. lipo output depends only on its inputs, so the derived file needs no second build. Every checked library is uploaded as one `payjoin-native` artifact, laid out by triple, for the bindings to package. The workflow runs in the daily bindings run and not on pull requests: two builds of eleven targets is 22 jobs, too many for every PR, so scheduled and release runs carry this check.
The Linux builds link against stubs derived from zig's macOS libSystem, not Apple's SDKs, so a symbol missing on the platform would link and then fail to load. Nothing has yet loaded these libraries on an Apple system. Add a macOS job that verifies the libraries and never produces any: * Xcode's linker resolves every import of each iOS and macOS library against the real SDK of its platform. * Each library, the universal2 one included, is renamed with install_name_tool and signed with codesign, as Flutter does when it wraps a dylib in a framework. * A small Flutter app runs a smoke test in the arm64 simulator. A stand-in build hook makes it bundle the Linux-built library, so the test goes through Flutter's own embed path with those exact bytes. The smoke test does not reuse the Dart unit tests, because those call helpers from the `_test-utils` feature, which the shipped library is built without.
The NuGet native assets were cross-built with their own toolchain: cargo-zigbuild and cargo-xwin from pip, clang-cl and llvm-lib from the runner image, Rust 1.85.0, and the `csharp` feature, which only the bindings generator needs but which changes the library's metadata hash. None of it followed the native build contract, so the shipped files could not be rebuilt bit for bit. Build each RID with contrib/native/build.sh in the native devshell, the same command the native workflow checks for reproducibility. The library is featureless and the toolchain comes from the flake, so the pip pins, the clang-cl symlinks and the cross branch of build_nuget_native.sh go away. The per-RID smoke tests still install the packed nupkg on each platform.
The wheel's libraries were built by build-wheel.sh itself: the Linux one with the python devshell's host cargo, so its glibc floor was whatever the runner's toolchain linked against, and the universal2 macOS one with cargo-zigbuild, which links with `-g` and needs a toolchain of its own. Neither followed the native build contract. Build the Linux library and both macOS slices with contrib/native/build.sh in the native devshell, and combine the slices with llvm-lipo as the native workflow does. build-wheel.sh now takes the library to bundle as its argument. It generates the bindings from a development build of the same featureless library, because uniffi reads its metadata from the static symbol table, which the zig-linked Linux library lacks. The Linux library needs at most glibc 2.17, so auditwheel tags the wheel manylinux_2_17, as the changelog notes. The python devshell no longer cross-links anything, so it drops cargo-zigbuild and the macOS targets on Linux.
native_toolchain_rust links Android libraries with the NDK the consumer's Flutter project selects, and accepts r27, the first NDK with the API 35 compiler wrappers it calls. r27 still aligns 64-bit libraries to 4 KB, which Google Play no longer accepts for apps that target Android 15 or later. NDK r28 and later align to 16 KB by default. Pass -z max-page-size=16384 to the linker for the 64-bit Android targets. Linking payjoin-ffi with NDK r27 changes its LOAD alignment from 0x1000 to 0x4000, and with NDK r29 the library is byte-identical with or without the flag, so no version check is needed. 32-bit ARM has no 16 KB requirement and is left alone.
The native libraries build with Rust 1.85.1, but the jobs that build the rest of each release still used the 1.85.0 MSRV toolchain. pack-npm compiles the published wasm and build-wheel generates the published bindings, in the javascript and python devshells. pack-nuget generates its bindings with 1.85.0 from the runner. Move those two devshells and pack-nuget to 1.85.1. The devshells' test jobs now build with 1.85.1 too. rust.yml's 1.85.0 leg keeps the MSRV for the core crates, and the C# and Kotlin test jobs keep it for payjoin-ffi. payjoin-ffi now declares rust-version 1.85 like the other crates, so Cargo names the floor to anyone building below it. The dart devshell stays on 1.85.0. Dart publishes generated Dart source and the Rust crate, which consumers compile with the package's own 1.85.1 toolchain file. Its build hook runs rustup's 1.85.1, which sets LD_LIBRARY_PATH to that toolchain's libraries while cargo still finds the devshell's rustc first in PATH. A nix 1.85.1 rustc then loads rustup's librustc_driver of the same name and fails to find libz. The C# Windows test job installed 1.85.0 but selected `1.85`, which rustup resolves to the newest 1.85 release, 1.85.1, so it did not test the MSRV like the Linux and macOS jobs. It now names 1.85.0.
The Kotlin workflow compiles production bindings after its tests by calling generate_bindings.sh directly, and only test.sh put Cargo-recent.lock in place. CI still built that step from the lock, but only because a fresh checkout has no Cargo.lock for test.sh to restore on exit. Calling the script on its own, or after test.sh restored a different lockfile, resolves the dependency graph afresh. Have generate_bindings.sh use the maintained lockfile itself, as the C# script does. When test.sh has already put it in place, the call does nothing.
spacebear21
force-pushed
the
native-build-contract
branch
from
October 29, 2026 22:14
bd9af28 to
2c7acf2
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of a 2-part stacked PR, checking that CI passes here before following up with part 2.
Pull Request Checklist
Please confirm the following before requesting review: