Android release_0.77.6 tarball does not reproduce from documented GitLab CI recipe

[xrviv]

Hi Blockstream team,

Following the suggestion from green_android issue #297, I tested the Android GDK release build directly in the GDK repo.

I am trying to reproduce the Android native library shipped in:

https://github.com/Blockstream/gdk/releases/download/release_0.77.6/gdk-release_0.77.6.tar.gz

Official tarball SHA256:

e86f164b77e690472f4cbdaf552d76ca847db885ced4502a49b5b0dfc96c9b8c

I used the public GitLab CI recipe from release_0.77.6.

Source:

release_0.77.6
fa2e267

Image:

docker.io/blockstream/gdk-android-builder@sha256:e08f5a77cc5472c9539cd0a83ee376efffab02eff4e768eae2c16c243f587f1e

Command:

./tools/build.sh --ndk arm64-v8a --buildtype release --no-deps-rebuild --external-deps-dir /prebuild/android-arm64-v8a --parallel 16 --install gdk-android-jni

I ran this twice in fresh clones.

Both local builds produced the same result:

b60ce8002dd7bb37d22809993a1e8a3a229a575f5e9c253b1a671fb84013bdf3 lib/arm64-v8a/libgreen_gdk_java.so

The official release tarball contains:

13d4ba62e75bebc4f8a9a50e2770a20def0469b1c470d321baadcadd407f07e7 lib/arm64-v8a/libgreen_gdk_java.so

The files have the same size:

26521752 bytes

But they differ substantially:

cmp -l local official | wc -l
6188322

Build IDs also differ:

local: 4507fd08da6325fe9af180846ecc571ffb4d2638
official: 50217d48b9f66acb2ab9e448b047e1a0fa3c8edd

So the public recipe appears deterministic locally, but it does not reproduce the official release_0.77.6 Android GDK artifact.

Question:

Was the official gdk-release_0.77.6.tar.gz built with this exact image digest and command?

If not, could you share the exact CI image/input that produced the published tarball?

This matters for Blockstream Green Android reproducibility because the app-layer build now reproduces, and the remaining blocking difference is libgreen_gdk_java.so.

This came from the Blockstream Green Android 5.5.1 WalletScrutiny verification:

https://walletscrutiny.com/mobile/com.greenaddress.greenbits_android_wallet/#verificationId=f61e6f43e7d4907fa94f3aa933f06ee7005e2e688435ec62d783550ed3dd34d3

For context, the F-Droid variant of Blockstream Green 5.5.1 is reproducible in our verification.

https://walletscrutiny.com/mobile/com.greenaddress.greenbits_android_wallet/#verificationId=18ea3b9ad35aec8dcebed8ed4dfe0463825219393895813bd6c3d8973ac03319

That build path compiles GDK from source and matches the official F-Droid APK.

So the issue appears specific to reproducing the prebuilt Android GDK release artifact used by the Play Store build, not to the Green Android application code in general.

[jgriffiths]

I can confirm the release binaries are built from our CI using the same commands you used, as specified in the gitlab config.

Running this locally, I get a different hash (440ca6ed22de6642e1c5edbf6178f6d396383b602401f1a76a96a20180236daa) from both you and the release binaries, but the same hash if I run the command twice.

I added the following change:

diff --git a/tools/buildgdk_rust.sh b/tools/buildgdk_rust.sh
index 1b0543c6..3183c2d8 100755
--- a/tools/buildgdk_rust.sh
+++ b/tools/buildgdk_rust.sh
@@ -39,6 +39,7 @@ CARGO_ARGS+=("--target-dir=${OUTPUT_DIR}")
 
 if [ "$BUILDTYPE" == "Release" ]; then
     CARGO_ARGS+=("--release")
+    CARGO_ARGS+=("--locked")
     ARTIFACT_PATH_HINT=${OUTPUT_DIR}"/release"
 else
     ARTIFACT_PATH_HINT=${OUTPUT_DIR}"/debug"

It did not change my hash, but should probably be added.

Its possible the difference is the build directory, since grepping shows its included in at least the C++ object files. Can you confirm that you built from the directory /root/gdk/gdk ? (The container should start in /root/gdk/ and then I cloned gdk and changed into the gdk git directory to build).

If you built from a different directory please try the one listed above and compare your ./gdk-android-jni/lib/arm64-v8a/libgreen_gdk_java.so sha256 to the one above.

If it differs, I suspect finding the differences probably needs to be done at the object file level. We might be able to initially determine if its the C++ or rust code at fault, can you also compare:

sha256sum build-android-arm64-v8a/src/libgreen_gdk.a  
052705163d5d3ad171c0d6e28e4b34665248351023ef00e7e76919bef30854d6  build-android-arm64-v8a/src/libgreen_gdk.a

With your build done from /root/gdk/gdk?

[xrviv]

Thanks @jgriffiths — the build directory does explain the difference between our original build and yours.

Our original clone was mounted directly at /root/gdk, so the repository root was /root/gdk, not /root/gdk/gdk. Rebuilding release_0.77.6 (fa2e267c511cc336aabc3350da56edfd4fa9b9a3) at your exact /root/gdk/gdk path, with the same pinned builder image and build command, reproduced both of your hashes byte-for-byte:

440ca6ed22de6642e1c5edbf6178f6d396383b602401f1a76a96a20180236daa  gdk-android-jni/lib/arm64-v8a/libgreen_gdk_java.so
052705163d5d3ad171c0d6e28e4b34665248351023ef00e7e76919bef30854d6  build-android-arm64-v8a/src/libgreen_gdk.a

That confirms the build is deterministic across our two machines when the absolute build path is held constant. Thanks also for the --locked addition; agreed that it is worth adding independently.

I then tried the apparent GitLab CI checkout path /builds/blockstream/gdk, since the tag's CI configuration does not set a custom GIT_CLONE_PATH. That produced a third distinct output:

fbcd5f119cc4add8c65cb09dafb8b59f425b5f6ae17756550d37d160ac0cd20b  gdk-android-jni/lib/arm64-v8a/libgreen_gdk_java.so

This still does not match the published release_0.77.6 library:

13d4ba62e75bebc4f8a9a50e2770a20def0469b1c470d321baadcadd407f07e7  lib/arm64-v8a/libgreen_gdk_java.so

Could you confirm the exact CI_PROJECT_DIR used by the release job? Depending on the self-hosted runner executor/configuration, the checkout path may include runner or concurrency-specific components, so /builds/blockstream/gdk may not be the actual release path.

Separately, would you consider mapping the build path out of the C++ output, for example with -ffile-prefix-map=$PWD=. and/or -fdebug-prefix-map=$PWD=.? That should make future artifacts independent of the runner checkout path. I would be happy to test a patch or open a small PR if useful.

[jgriffiths]

Hi @xrviv thanks for confirming. We'd be happy to use the remapping flags to remove the build path - it seems onerous to require that you replicate our paths to achieve reproducibility.

We are very busy at the moment so if you'd like to submit a PR that would be great - I'm happy to shepherd it towards merging.

[jgriffiths]

In the meantime, our CI uses /builds/blockstream/green/gdk/, with --external-deps-dir set to /prebuild/$CC for linux gcc/clang builds. If you grep the repo for prebuild you can see the deps directories used for each target.

[xrviv]

Hi Jon,

That did it — thank you. Building at /builds/blockstream/green/gdk reproduces the official
release_0.77.6 arm64 library byte-for-byte:

13d4ba62e75bebc4f8a9a50e2770a20def0469b1c470d321baadcadd407f07e7  gdk-android-jni/lib/arm64-v8a/libgreen_gdk_java.so

The extra green/ level was the piece we were missing. For the record, four paths gave four
distinct hashes, and only the last matches the release:

Build path	arm64 libgreen_gdk_java.so
/root/gdk	b60ce800…
/root/gdk/gdk	440ca6ed… (matches yours)
/builds/blockstream/gdk	fbcd5f11…
/builds/blockstream/green/gdk	13d4ba62… ← official

Same commit (fa2e267c), same pinned builder image, same command, with
--external-deps-dir /prebuild/android-arm64-v8a. Only the checkout path differed.

To be precise about scope: this is the arm64 libgreen_gdk_java.so specifically. We haven't
checked the other architectures or the rest of the release tarball.

This is also the case for the remapping flags in a nutshell: we reproduced your shipped binary
exactly, but only after asking you for your runner's directory layout, which isn't something a
third party can discover on their own.

I'll get that PR together. Since the build is mixed CMake and cargo, I'll plan on covering both
sides — -ffile-prefix-map for the C++ objects and --remap-path-prefix via RUSTFLAGS for
gdk_rust — so the result doesn't depend on the checkout path anywhere. If you'd rather it were
wired in differently, say so and I'll follow your preference; otherwise I'll propose something and
you can steer it in review. I'll keep --locked out of this one so the change stays focused.

No rush on a reply — I know you're busy, and there's nothing here you need to unblock.

Thanks again for the pointer; it saved a lot of guessing.

[jgriffiths]

Hi @xrviv freat news! Many thanks for confirming. Please do include the --locked change as we should be pinning our deps for security purposes in addition to ensuring reproducibility. Please also make a commit per build system - we try to keep the rust and c++ changes distinct for history purposes. Thanks!