diff --git a/.github/ffi.md b/.github/ffi.md index 3e9f5b3486e..de22438b1ec 100644 --- a/.github/ffi.md +++ b/.github/ffi.md @@ -6,11 +6,14 @@ It builds nine distinct default-feature targets, replacing fourteen native builds across those packages. Dart retains its eleven minimal-feature targets; Python retains its wheel builds and Python-specific environment. -C++ retains its host CMake builds and independent `cpp-v*` release tags. All -four sampled C++ static libraries have different hashes from the candidates -shared here. In particular, ARM Linux builds natively on Ubuntu 24.04 rather -than on the shared cross-compilation host. Sharing those packages needs -separate artifact/ABI proof and a settled prebuilt-library CMake interface. +C++ retains source builds as its default and independent `cpp-v*` release tags. +Nightly packages reuse the coordinator's validated same-run native archives. +The [historical C++ comparison](#c-comparison) records differing archive hashes, +including native versus cross-built ARM Linux inputs. The +[current installed-consumer proof](#current-installed-consumer-proof) separately +validates the prebuilt path on all four C++ hosts. Pinned C++ release reuse +still awaits real published default-feature assets and validation of their +exact version and checksum pins. The shared build uploads `default-lib-`. Binding jobs read the built Linux library with `just rs package-ffi --bindings-only --library PATH`, so @@ -77,3 +80,180 @@ That is about 90 runner-minutes saved per run. Binding generation now waits for every native build, since it reads the Linux library; Go's binding job spends most of its 12 minutes compiling `uniffi-bindgen-go`. The Go, Kotlin and Swift packagers each selected their own targets from the shared artifacts and passed. + +## C++ comparison + +This comparison uses the sampled 2026-10-14 Nightly linked above, at +`0d9e9ff78857003b305f11ebf07095af4c3cdd87`. Compare each `moq-cpp-` +artifact's installed static library with `go-lib-`; those Go libraries +are the default-feature inputs already proved equivalent to Kotlin and Swift. +Preserve this source revision when reproducing the comparison, rather than +mixing artifacts from different runs or rebuilding today's bindings. + +Both paths build `moq-ffi` with default `audio` and `video` features, +`--locked`, and the release profile: fat LTO, one codegen unit, and debuginfo +stripped for `moq-ffi`. The toolchain action installed stable Rust 1.99.0, +but the repository's `rust-toolchain.toml` selects Rust 1.98.1 (`48a229cea`) +for the actual Cargo builds on both paths. C++ invokes +`cargo build --locked --package moq-ffi --release` through +CMake; the shared candidate adds `--target ` through +`rs/moq-ffi/build.sh`. C++ is always a native host build. Only the shared ARM +Linux candidate crosses from an x86 Linux host. + +| Target | C++ host | Shared candidate host | Native archive comparison | +| --- | --- | --- | --- | +| x86\_64-unknown-linux-gnu | Ubuntu 24.04, image 20261004.327.1 | Same image and architecture | 772 common objects byte-identical; one differently named Rust LTO object | +| aarch64-unknown-linux-gnu | Ubuntu 24.04 ARM, image 20261012.149.1 | Ubuntu 24.04 x86, image 20261004.327.1, GCC cross toolchain | 879 common objects byte-identical; one differently named Rust LTO object | +| aarch64-apple-darwin | macOS 26 ARM, image 20260907.0351.1 | Same image and architecture | 847 common objects byte-identical; one differently named Rust LTO object | +| x86\_64-pc-windows-msvc | Windows 2025 VS2026, image 20260925.250.1 | Same image and architecture | 6,077 members each, preserving duplicate basenames; timestamps, debug paths and six assembly code sections differ, plus one differently named Rust LTO object | + +The Linux native and cross compilers are GCC 13.3.0. Both runners use Ubuntu +24.04; the ARM cross sysroot is Ubuntu's `libc6-dev-arm64-cross`. The sampled +shared ARM cdylib requires up to GLIBC\_2.38 and GLIBCXX\_3.4. The installed C++ +archive carries unversioned imports, so its glibc minimum cannot be read as a +linked executable's version requirement. Its native objects are nevertheless +byte-identical to the cross build, and the two Linux archive pairs have the +same sets of non-Rust undefined symbols. This rules out a native dependency +change in this sample, without proving an older-distro runtime baseline. + +The Windows C++ job identifies MSVC 19.51.36260.0; both jobs use the same +runner image. A sampled AWS-LC object requests `MSVCRT` and `OLDNAMES` in both +archives, preserving the release CRT. Preserving duplicate archive member +names gives 6,077 members per archive; the earlier 665 count was distinct +basename keys. Of 6,076 matched members, 5,685 are identical, 43 differ only +in timestamps, and 342 differ only in debug/discarded sections and their +associated auxiliary symbols, timestamps and offsets. Their `.debug$S` +records contain `target\release\build` versus +`target\x86_64-pc-windows-msvc\release\build` paths. + +Six OpenH264 assembly members have executable `.text` differences: `dct.o`, +`downsample_bilinear.o`, `matrix_transpose.o`, `mb_copy.o`, `mc_luma.o`, and +the second `vaa.o`. For `matrix_transpose.o`, `mb_copy.o`, and the second +`vaa.o`, normalized instruction listings match after accounting for +CDQE versus MOVSXD sign extension and alignment padding. The other three +remain unproven. The C++ job installed NASM 3.2.0; the shared assembler's +version was not logged, so a tool-version cause is plausible, not established. + +On each Linux target, all 427 `.text.uniffi_*` and 295 +`.rodata.UNIFFI_META*` payloads match. Their relocations and internal symbols +differ, so equal section payloads do not prove linked equivalence. None of +the four Rust fat-LTO objects has a complete executable-code equivalence +proof. No incompatible ABI difference has been established; installed +consumers must still execute against the actual selected prebuilt libraries. + +Static-library SHA-256 hashes, C++ first and shared candidate second: + +```text +x86_64-unknown-linux-gnu +f786f40bd739a7cf97471549bcb30f05f25c35acc14d0b2f826c34a80c9e9505 +d663534420b8bea820388c058a01b590eafc2531b8fe72961cdf1afe7ac630bc +aarch64-unknown-linux-gnu +09e0ea5e0ddfd72d4799febba21a22f4d83de4c2b16a08ffa58e9fbfa5ad4f1e +7fcd57f6c4a52332828daf0e6f2d64c9a81082a13df20da023ac3c1d0124577f +aarch64-apple-darwin +1c05406ca3bb8f13f45a9760b5e7eaf5e3fd7eddfeb72f6fa220cd4fb61c74ad +39eaa5f69770ecf6c20dedad43d10891e66a46984cdae3639875cad4235d78fa +x86_64-pc-windows-msvc +6cbbe795ebc6bf992f62ffc4dac98befac2b3f07f523c757f2c9848f395d4ffe +617eb2d358d17af2efe590d0185212fb15aec46e5f4d31318d884155dfe06b86 +``` + +Using the pinned `uniffi-bindgen-cpp` version +`v0.11.0-kixelated.4+v0.32.2` and this revision's `cpp/moq/uniffi.toml`, +all four candidates generate exactly the installed C++ bindings: `moq.hpp`, +`moq.cpp`, `moq_scaffolding.hpp`, and `uniffi_expected.hpp`. Linux and macOS +static archives are readable directly; Windows requires the shared candidate's +DLL because this generator cannot read its static archive's long member names. +Compare generated `moq.cpp` with the installed `include/moq/ffi/moq.cpp`, not +the separate handwritten `share/moq-cpp/moq.cpp` wrapper. + +The sampled revision's C++17 consumer probe passed on Linux x86 after linking +the installed headers and wrapper source against the shared candidate static +library. That manual experiment used the Nix development toolchain. ARM Linux, +macOS and Windows consumers were not run with that historical sample; it alone +did not justify reuse. The current-source proof below tests all four hosts +without claiming byte-for-byte executable equivalence. + +### Current installed-consumer proof + +The [2026-10-16 focused dry-run](https://github.com/moq-dev/moq/actions/runs/38969347067) +passed at implementation commit `cbbcf9b2c92aa88c5550d9175adcc317f8137f4a`. +All four shared default-feature release builds passed, followed by installed +C++17 and C++23 consumers on Linux x86, Linux ARM, macOS ARM and Windows MSVC. +Both C++23 forms ran their coroutine probes. Unix also passed pkg-config +consumers and compiled the documented examples. Each job then archived, +verified and extracted its checked input and produced its final C++ package. +The consumer and package jobs did not rebuild moq-ffi with Cargo. Release and +OBS publication jobs were skipped. + +This validates the same-run prebuilt path for that coherent source, generator +and configuration on those runners. No published release pin has been tested, +and this does not establish an older Linux runtime baseline. Documentation-only +evidence updates after that implementation commit do not change the tested code. + +### Release input decision + +`cpp-v*` and `moq-ffi-v*` remain separate release lifecycles. The shared build's +default-feature static libraries are Actions artifacts, which expire; an old +C++ release must not depend on those artifacts still existing. Current +`moq-ffi-v*` releases publish Dart's minimal-feature shared libraries and Swift +packages, not standalone default-feature static libraries for all four C++ +targets. The minimal-feature assets cannot replace these candidates. + +The maintainer approved an explicit `cpp/moq/FFI_VERSION` pin selecting +checksummed default-feature assets on the corresponding `moq-ffi-v*` release. +`FFI_SHA256.json` pins every selected archive by its complete filename and +SHA-256. There is no newest-release lookup. The pin is intentionally unset: +existing releases lack this four-target asset contract. Source builds remain +the default until a real release publishes all four validated assets and a +subsequent pin change supplies their exact hashes. A malformed pin, missing +asset, wrong hash, wrong feature/profile, or incompatible target fails. + +### Native input and release gates + +`MOQ_FFI_DIR` in CMake and `--ffi-dir DIR` in `just cpp package` select an +extracted default-feature native input. Its manifest names the FFI version, +source commit, target, feature set and profile; it carries the static library, +metadata cdylib and native linker inputs together. CMake generates bindings +from that cdylib without invoking Cargo. Direct directories are explicit local +or same-run inputs; release downloads additionally require the checked-in hash. +Without an explicit directory, `just cpp package` fetches exactly the pinned +release input. The pin selects package-release inputs; direct CMake builds and +`just cpp check` build the current checkout unless `MOQ_FFI_DIR` or `--ffi-dir` +is explicitly supplied. A failed selected release fetch never builds source. + +The shared coordinator downloads its four exact `default-lib-` +artifacts on Ubuntu x86, Ubuntu 24.04 ARM, macOS ARM and Windows MSVC. Each runs +`just cpp check --ffi-dir DIR`: install the package, then build and execute the +existing C++17 and C++23 consumers, including the standard set on the consumer +target, with exceptions/RTTI disabled. Unix also tests installed pkg-config +linking and compiles the documented examples. No prebuilt error invokes Cargo. +The cross-built ARM input therefore executes on an ARM host. This checks that +runner's baseline only; it does not establish an older Linux runtime baseline. + +Only after all four consumers succeed can the tag coordinator publish +`moq-ffi-default--.tar.gz` and one `.sha256` per archive. +The publisher refuses changed bytes under an existing asset name and never +clobbers assets. These names are separate from Dart's minimal-feature assets. +A first publication is an external condition for selecting a release pin; +this change creates no release or tag. After publication, validate the actual +pinned downloads through the same four-host consumer path before activation. + +A branch-only `release-cpp.yml` dispatch with `prebuilt-test=true` builds just +these four shared targets and exercises the prebuilt package and installed +consumers without publishing. Nightly waits for the shared coordinator, then +packages each target from its +`cpp-native-` artifact, which exists only after that target's installed +consumers succeed. It verifies the archive sidecar and manifest before CMake +uses it, with no source fallback. An unrelated coordinator failure does not +suppress a successful C++ target. Tag releases refuse same-run inputs and keep +the explicit release pin path. `just cpp check` and nightly `just gh test` also +cover hash refusal, +archive traversal/symlink refusal, reproducible packaging, publication refusal, +and CMake's build-free prebuilt path with malformed-input fixtures. + +PR #5727 independently changes the generator and overlaps the C++ checks and +release workflow. Today's matrix validates this checkout's coherent generator +and config. If that PR lands before this change or the selected release, rerun +the same gates on the final checkout. The old sampled binding matches alone +do not validate a new generator. diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index db74fda80c6..fc412ab28d0 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -338,7 +338,11 @@ jobs: release-cpp: name: Release Cpp + needs: [release-shared-ffi] + if: ${{ always() && !cancelled() }} uses: ./.github/workflows/release-cpp.yml + with: + reuse-native: true permissions: contents: write diff --git a/.github/workflows/release-cpp.yml b/.github/workflows/release-cpp.yml index f75d88d5428..c6bde6b28b3 100644 --- a/.github/workflows/release-cpp.yml +++ b/.github/workflows/release-cpp.yml @@ -10,14 +10,34 @@ on: # Dry-run: build everything, publish nothing. Nightly calls every release # workflow this way so a broken build surfaces before the tag is cut. workflow_call: + inputs: + reuse-native: + description: "Use this run's native input after its installed-consumer gate succeeds" + type: boolean + default: false workflow_dispatch: + inputs: + prebuilt-test: + description: "Validate the same-revision shared default-feature input on all four hosts" + type: boolean + default: false permissions: contents: read jobs: + native: + if: github.event_name == 'workflow_dispatch' && inputs.prebuilt-test + uses: ./.github/workflows/release-ffi.yml + with: + language: default + bindings: false + targets: '["x86_64-unknown-linux-gnu", "aarch64-unknown-linux-gnu", "aarch64-apple-darwin", "x86_64-pc-windows-msvc"]' + build: name: Build (${{ matrix.target }}) + needs: [native] + if: ${{ always() && !cancelled() }} runs-on: ${{ matrix.os }} strategy: @@ -69,11 +89,67 @@ jobs: exit 1 fi + - name: Refuse same-run input for a release tag + if: inputs.reuse-native && github.ref_type == 'tag' + run: | + echo "::error::cpp-v releases must use their immutable FFI_VERSION pin" + exit 1 + + - name: Download validated same-run input + if: inputs.reuse-native + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: cpp-native-${{ matrix.target }} + path: native-archive + + - name: Parse prebuilt FFI version + if: inputs.reuse-native || (github.event_name == 'workflow_dispatch' && inputs.prebuilt-test) + id: ffi-version + run: just gh release parse-version moq-ffi + + - name: Download shared input for compatibility check + if: github.event_name == 'workflow_dispatch' && inputs.prebuilt-test + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: default-lib-${{ matrix.target }} + path: native-library + + - name: Stage shared input + if: github.event_name == 'workflow_dispatch' && inputs.prebuilt-test + shell: bash + env: + TARGET: ${{ matrix.target }} + VERSION: ${{ needs.native.outputs.version }} + run: just cpp ffi stage --version "$VERSION" --target "$TARGET" --library native-library --output native-stage + + - name: Check installed consumers with shared input + if: github.event_name == 'workflow_dispatch' && inputs.prebuilt-test + shell: bash + run: just cpp check --ffi-dir "$GITHUB_WORKSPACE/native-stage" + + - name: Archive checked input + if: github.event_name == 'workflow_dispatch' && inputs.prebuilt-test + run: just cpp ffi archive --directory native-stage --output native-archive + + - name: Verify and extract validated input + if: inputs.reuse-native || (github.event_name == 'workflow_dispatch' && inputs.prebuilt-test) + shell: bash + env: + TARGET: ${{ matrix.target }} + VERSION: ${{ steps.ffi-version.outputs.version }} + run: just cpp ffi extract --version "$VERSION" --target "$TARGET" --directory native-archive --output native-input + - name: Build and package shell: bash env: TARGET: ${{ matrix.target }} - run: just cpp package --target "$TARGET" --output dist + PREBUILT_INPUT: ${{ inputs.reuse-native || (github.event_name == 'workflow_dispatch' && inputs.prebuilt-test) }} + run: | + args=(--target "$TARGET" --output dist) + if [[ "$PREBUILT_INPUT" == true ]]; then + args+=(--ffi-dir "$GITHUB_WORKSPACE/native-input") + fi + just cpp package "${args[@]}" - name: Upload artifact uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 diff --git a/.github/workflows/release-shared-ffi.yml b/.github/workflows/release-shared-ffi.yml index 8456bd9d380..27890c733df 100644 --- a/.github/workflows/release-shared-ffi.yml +++ b/.github/workflows/release-shared-ffi.yml @@ -65,3 +65,85 @@ jobs: permissions: contents: write secrets: inherit + + cpp-native: + name: C++ native input (${{ matrix.target }}) + needs: [native] + if: ${{ always() && !cancelled() }} + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + include: + - target: x86_64-unknown-linux-gnu + os: ubuntu-latest + - target: aarch64-unknown-linux-gnu + os: ubuntu-24.04-arm + - target: aarch64-apple-darwin + os: macos-latest + - target: x86_64-pc-windows-msvc + os: windows-latest + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: ./.github/actions/just + - name: Install Rust + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable + - name: Install uniffi-bindgen-cpp + run: cargo install uniffi-bindgen-cpp --locked --git https://github.com/kixelated/uniffi-bindgen-cpp --tag v0.11.0-kixelated.4+v0.32.2 + - name: Download default-feature input + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: default-lib-${{ matrix.target }} + path: native-library + - name: Parse FFI version + id: parse + run: just gh release parse-version moq-ffi + - name: Stage native input + shell: bash + env: + TARGET: ${{ matrix.target }} + VERSION: ${{ steps.parse.outputs.version }} + run: just cpp ffi stage --version "$VERSION" --target "$TARGET" --library native-library --output native-input + - name: Check installed consumers + shell: bash + run: just cpp check --ffi-dir "$GITHUB_WORKSPACE/native-input" + - name: Archive validated input + run: just cpp ffi archive --directory native-input --output native-dist + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: cpp-native-${{ matrix.target }} + path: native-dist/* + if-no-files-found: error + + cpp-native-release: + name: Publish C++ native inputs + needs: [cpp-native] + if: ${{ always() && !cancelled() && github.ref_type == 'tag' && startsWith(github.ref_name, 'moq-ffi-v') && needs.cpp-native.result == 'success' }} + runs-on: ubuntu-latest + permissions: + contents: write + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - uses: ./.github/actions/just + - name: Find previous tag + id: prev_tag + run: just gh release prev-tag moq-ffi + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + pattern: cpp-native-* + path: native-dist + merge-multiple: true + - name: Parse FFI version + id: parse + run: just gh release parse-version moq-ffi + - name: Publish without replacing assets + env: + GH_TOKEN: ${{ github.token }} + VERSION: ${{ steps.parse.outputs.version }} + PREV_TAG: ${{ steps.prev_tag.outputs.tag }} + run: just cpp ffi publish --version "$VERSION" --directory native-dist --prev-tag "$PREV_TAG" diff --git a/cpp/justfile b/cpp/justfile index 963443c8fbe..e5f9d681968 100644 --- a/cpp/justfile +++ b/cpp/justfile @@ -9,8 +9,10 @@ default: just check # Build the package and run the probe and doc samples against its install. -check: - ../sh/cpp/check.sh +[positional-arguments] +check *args: + python3 ../sh/cpp/ffi.test.py + ../sh/cpp/check.sh "$@" # Runs in the caller's directory, so a relative `--output` means what it would # to the script itself. @@ -24,3 +26,9 @@ package *args: # Remove the package builds. clean: rm -rf moq/build + +# Stage, archive, fetch, or publish the pinned default-feature native input. +[no-cd] +[positional-arguments] +ffi *args: + python3 {{ quote(justfile_directory() / "sh/cpp/ffi.py") }} "$@" diff --git a/cpp/moq/CMakeLists.txt b/cpp/moq/CMakeLists.txt index fe85ed35ec1..306c30007c8 100644 --- a/cpp/moq/CMakeLists.txt +++ b/cpp/moq/CMakeLists.txt @@ -8,22 +8,52 @@ endif() project(moq-cpp VERSION ${_version} LANGUAGES CXX) -# Builds moq-ffi as a staticlib with cargo and renders its bindings with uniffi-bindgen-cpp, -# the same no-Corrosion custom target rs/moq-c/CMakeLists.txt uses. The cargo profile follows -# CMAKE_BUILD_TYPE (release unless Debug), even on a multi-config generator (Visual Studio), -# whose --config never changes it: the staticlib links the release CRT either way. set(_workspace "${CMAKE_CURRENT_SOURCE_DIR}/../..") -if(CMAKE_BUILD_TYPE STREQUAL "Debug") - set(_profile debug) - set(_cargo_flags "") +set(MOQ_FFI_DIR "" CACHE PATH "Extracted default-feature moq-ffi native package") +if(MOQ_FFI_DIR) + get_filename_component(MOQ_FFI_DIR "${MOQ_FFI_DIR}" ABSOLUTE) + file(READ "${MOQ_FFI_DIR}/manifest.json" _manifest) + foreach(_field version target features profile source) + string(JSON _input_${_field} GET "${_manifest}" ${_field}) + endforeach() + if(NOT _input_features STREQUAL "default" OR NOT _input_profile STREQUAL "release") + message(FATAL_ERROR "Prebuilt moq-ffi requires default features and release profile") + endif() + if(NOT _input_version MATCHES "^[0-9]+\\.[0-9]+\\.[0-9]+([-+][0-9A-Za-z.+-]+)?$") + message(FATAL_ERROR "Malformed prebuilt moq-ffi version: ${_input_version}") + endif() + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + if(CMAKE_SYSTEM_PROCESSOR MATCHES "^(aarch64|arm64)$") + set(_ffi_target aarch64-unknown-linux-gnu) + elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "^(x86_64|AMD64)$") + set(_ffi_target x86_64-unknown-linux-gnu) + endif() + elseif(APPLE AND CMAKE_SYSTEM_PROCESSOR MATCHES "^(aarch64|arm64)$") + set(_ffi_target aarch64-apple-darwin) + elseif(MSVC AND CMAKE_SIZEOF_VOID_P EQUAL 8 AND CMAKE_SYSTEM_PROCESSOR MATCHES "^(AMD64|x86_64)$") + set(_ffi_target x86_64-pc-windows-msvc) + endif() + if(NOT DEFINED _ffi_target OR NOT _ffi_target STREQUAL _input_target) + message(FATAL_ERROR "Prebuilt moq-ffi target ${_input_target} does not match this host") + endif() else() - set(_profile release) - set(_cargo_flags --release) + if(CMAKE_BUILD_TYPE STREQUAL "Debug") + set(_profile debug) + set(_cargo_flags "") + else() + set(_profile release) + set(_cargo_flags --release) + endif() + if(DEFINED ENV{CARGO_TARGET_DIR}) + set(_cargo_target "$ENV{CARGO_TARGET_DIR}") + else() + set(_cargo_target "${_workspace}/target") + endif() endif() -if(DEFINED ENV{CARGO_TARGET_DIR}) - set(_cargo_target "$ENV{CARGO_TARGET_DIR}") -else() - set(_cargo_target "${_workspace}/target") + +if(MOQ_FFI_DIR) + set(_cargo_target "${MOQ_FFI_DIR}/lib") + set(_profile "") endif() if(WIN32) set(MOQ_FFI_LIB "${_cargo_target}/${_profile}/moq_ffi.lib") @@ -35,6 +65,17 @@ else() set(MOQ_FFI_LIB "${_cargo_target}/${_profile}/libmoq_ffi.a") set(_metadata "${_cargo_target}/${_profile}/libmoq_ffi.so") endif() +if(MOQ_FFI_DIR) + foreach(_input "${MOQ_FFI_LIB}" "${_metadata}" "${MOQ_FFI_DIR}/native-libs.txt") + if(NOT EXISTS "${_input}") + message(FATAL_ERROR "Missing prebuilt FFI input: ${_input}") + endif() + endforeach() + set(_build_ffi "") +else() + set(_build_ffi COMMAND cargo build --locked --package moq-ffi ${_cargo_flags}) + set(_ffi_byproducts "${MOQ_FFI_LIB}" "${_metadata}") +endif() find_program(UNIFFI_BINDGEN_CPP uniffi-bindgen-cpp REQUIRED) @@ -49,15 +90,15 @@ list(TRANSFORM _generated_files PREPEND "${_generated}/" OUTPUT_VARIABLE _genera set(_render "${CMAKE_CURRENT_BINARY_DIR}/render") list(TRANSFORM _generated_files PREPEND "${_render}/" OUTPUT_VARIABLE _render_paths) add_custom_target(moq_ffi ALL - COMMAND cargo build --locked --package moq-ffi ${_cargo_flags} + ${_build_ffi} COMMAND ${CMAKE_COMMAND} -E rm -rf "${_render}" COMMAND "${UNIFFI_BINDGEN_CPP}" --library "${_metadata}" --config "${CMAKE_CURRENT_SOURCE_DIR}/uniffi.toml" --out-dir "${_render}" COMMAND ${CMAKE_COMMAND} -E make_directory "${_generated}" COMMAND ${CMAKE_COMMAND} -E copy_if_different ${_render_paths} "${_generated}" WORKING_DIRECTORY "${_workspace}" - BYPRODUCTS "${MOQ_FFI_LIB}" "${_metadata}" ${_generated_paths} - COMMENT "Building moq-ffi and rendering its C++ bindings" + BYPRODUCTS ${_ffi_byproducts} ${_generated_paths} + COMMENT "Preparing moq-ffi and rendering its C++ bindings" VERBATIM ) # System libraries the staticlib needs from an external linker. moq-ffi and moq-c share one @@ -69,7 +110,12 @@ elseif(WIN32) else() set(_platform linux) endif() -file(STRINGS "${_workspace}/rs/moq-c/native-libs/${_platform}.txt" _lines) +if(MOQ_FFI_DIR) + set(_native_libs "${MOQ_FFI_DIR}/native-libs.txt") +else() + set(_native_libs "${_workspace}/rs/moq-c/native-libs/${_platform}.txt") +endif() +file(STRINGS "${_native_libs}" _lines) set(MOQ_NATIVE_LIBS "") foreach(_line IN LISTS _lines) string(STRIP "${_line}" _line) diff --git a/cpp/moq/FFI_SHA256.json b/cpp/moq/FFI_SHA256.json new file mode 100644 index 00000000000..0967ef424bc --- /dev/null +++ b/cpp/moq/FFI_SHA256.json @@ -0,0 +1 @@ +{} diff --git a/cpp/moq/FFI_VERSION b/cpp/moq/FFI_VERSION new file mode 100644 index 00000000000..e70bfcad5ba --- /dev/null +++ b/cpp/moq/FFI_VERSION @@ -0,0 +1,2 @@ +# Set one moq-ffi release version after all four default-feature assets publish +# and their SHA-256 hashes are pinned in FFI_SHA256.json. Until then, build source. diff --git a/cpp/moq/README.md b/cpp/moq/README.md index ca184c07690..fe357ad25d4 100644 --- a/cpp/moq/README.md +++ b/cpp/moq/README.md @@ -4,10 +4,10 @@ C++17 bindings for [rs/moq-ffi](../../rs/moq-ffi): the sources `uniffi-bindgen-c ## Layout -- `CMakeLists.txt` builds moq-ffi as a staticlib with cargo, renders the bindings into the build tree, and compiles them as the `moq::cpp` target. Use it with `add_subdirectory`, or install it for `find_package(moq-cpp)` and `moq-cpp.pc`. +- `CMakeLists.txt` builds moq-ffi as a staticlib with cargo, or consumes an explicit `MOQ_FFI_DIR` native input, renders the bindings into the build tree, and compiles them as the `moq::cpp` target. Use it with `add_subdirectory`, or install it for `find_package(moq-cpp)` and `moq-cpp.pc`. - `src/moq.cpp` compiles the generated `moq/ffi/moq.cpp`. It ships as source so it builds with the consumer's standard and flags. - `cmake/` holds the installed CMake package and pkg-config templates. -- `build.sh` builds the release archive `release-cpp.yml` publishes. +- `build.sh` builds the release archive `release-cpp.yml` publishes. `--ffi-dir DIR` selects an extracted default-feature native package; otherwise an `FFI_VERSION` pin fetches the release archive verified against `FFI_SHA256.json`. The pin remains unset until the first four-target release passes its installed-consumer checks. - `test/probe.cpp` exercises the installed package over a real QUIC session. ## Checking diff --git a/cpp/moq/build.sh b/cpp/moq/build.sh index 694e9a44b24..473145cc8f1 100755 --- a/cpp/moq/build.sh +++ b/cpp/moq/build.sh @@ -2,7 +2,7 @@ set -euo pipefail # Build the moq C++ package for the host and archive it for release. -# Usage: ./build.sh [--target TARGET] [--output DIR] +# Usage: ./build.sh [--target TARGET] [--output DIR] [--ffi-dir DIR] # # The archive holds what `cmake --install` lays out: the moq-ffi staticlib, the wrapper and # generated headers, the generated source, the CMake package, and moq-cpp.pc. Needs cargo, cmake, @@ -12,6 +12,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TARGET="" OUTPUT_DIR="dist" +FFI_DIR="" while [[ $# -gt 0 ]]; do case $1 in @@ -19,12 +20,16 @@ while [[ $# -gt 0 ]]; do TARGET="$2" shift 2 ;; + --ffi-dir) + FFI_DIR="$(cd "$2" && pwd)" + shift 2 + ;; --output) OUTPUT_DIR="$2" shift 2 ;; -h | --help) - echo "Usage: $0 [--target TARGET] [--output DIR]" + echo "Usage: $0 [--target TARGET] [--output DIR] [--ffi-dir DIR]" exit 0 ;; *) @@ -50,10 +55,16 @@ OUTPUT_DIR="$(cd "$OUTPUT_DIR" && pwd)" PACKAGE_DIR="$OUTPUT_DIR/$NAME" BUILD_DIR="$SCRIPT_DIR/build/release" +if [[ -z "$FFI_DIR" ]] && sed '/^[[:space:]]*#/d; /^[[:space:]]*$/d' "$SCRIPT_DIR/FFI_VERSION" | grep -q .; then + FFI_DIR="$BUILD_DIR/ffi" + rm -rf "$FFI_DIR" + python3 "$SCRIPT_DIR/../../sh/cpp/ffi.py" fetch --target "$TARGET" --output "$FFI_DIR" +fi + echo "Packaging $NAME..." rm -rf "$PACKAGE_DIR" # lib/, not the lib64/ GNUInstallDirs picks on some hosts, so every archive has one layout. -cmake -S "$SCRIPT_DIR" -B "$BUILD_DIR" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_LIBDIR=lib +cmake -S "$SCRIPT_DIR" -B "$BUILD_DIR" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_LIBDIR=lib -DMOQ_FFI_DIR="$FFI_DIR" cmake --build "$BUILD_DIR" --config Release cmake --install "$BUILD_DIR" --config Release --prefix "$PACKAGE_DIR" diff --git a/doc/lib/cpp/index.md b/doc/lib/cpp/index.md index dd3a7ea1aa6..249c89e27bb 100644 --- a/doc/lib/cpp/index.md +++ b/doc/lib/cpp/index.md @@ -57,8 +57,15 @@ source and to your code. From source, `add_subdirectory(cpp/moq)` in a checkout builds `moq-ffi` with cargo and renders the bindings with the pinned `uniffi-bindgen-cpp` (see [`cpp/moq`](https://github.com/moq-dev/moq/tree/main/cpp/moq)), then exposes -the same `moq::cpp` target. The package is `moq-cpp`, so it installs beside -the C package, `moq-c`, without colliding. +the same `moq::cpp` target. To build from an explicitly staged default-feature +FFI package instead, set `MOQ_FFI_DIR` to its extracted directory, or pass +`--ffi-dir DIR` to `just cpp package`. This renders bindings from the supplied +metadata library and links its static library without compiling Rust. Without +an explicit directory, a release pin in `cpp/moq/FFI_VERSION` selects the default +input; `just cpp package` fetches it with the checked-in SHA-256 hashes and fails +if it cannot verify it. Direct CMake builds and `just cpp check` build the +current checkout unless given an explicit native directory. The package is +`moq-cpp`, so it installs beside the C package, `moq-c`, without colliding. ## Example diff --git a/flake.nix b/flake.nix index e72811d3d34..6cedf2231bd 100644 --- a/flake.nix +++ b/flake.nix @@ -367,10 +367,11 @@ # # Like uniffi-bindgen-go, the tag pairs the generator's version with the # uniffi release it reads, so it moves with the `uniffi` dependency in - # rs/moq-ffi/Cargo.toml. Five other places name the same tag and must be + # rs/moq-ffi/Cargo.toml. The other places name the same tag and must be # bumped together: the `cargo install` lines in rs/moq-ffi/build.sh, # cpp/moq/README.md, .github/workflows/cpp.yml, - # .github/workflows/obs.yml, and .github/workflows/release-cpp.yml. + # .github/workflows/obs.yml, .github/workflows/platform.yml, + # .github/workflows/release-cpp.yml, and .github/workflows/release-shared-ffi.yml. # # This points at a fork of LiveKit's async branch (livekit/uniffi-bindgen-cpp # PR #1): neither LiveKit nor NordSecurity has a uniffi 0.32 generator, diff --git a/quest/m2/cpp-ffi-reuse.md b/quest/m2/cpp-ffi-reuse.md index 656f4ef446c..70413cd2342 100644 --- a/quest/m2/cpp-ffi-reuse.md +++ b/quest/m2/cpp-ffi-reuse.md @@ -11,21 +11,51 @@ it ships. ## Plan -Follow-up to [#5549](https://github.com/moq-dev/moq/pull/5549), which shares -the default FFI build across Go, Kotlin and Swift and records its evidence in -`.github/ffi.md`. The sampled C++ static libraries differed from the reuse -candidates, so C++ was left out. - -- Prove compatibility before changing a build. Compare, per C++ target: - target triple, features, profile, native versus cross host, glibc/CRT - version, and UniFFI metadata. Record the comparison in `.github/ffi.md` - beside the existing ones. A difference that matters ends the quest with - the reason recorded; one that does not gets normalized, as #5549 did for - Windows timestamps. -- Then design a prebuilt input for the CMake and build-script paths, and test - installed consumers against it. -- Keep `cpp-v*` releases separate from `moq-ffi-v*`; the C++ release - selects an FFI build explicitly rather than following the newest one. +The maintainer approved an explicit `cpp/moq/FFI_VERSION` pin selecting +immutable default-feature native assets on the corresponding `moq-ffi-v*` +release. `cpp/moq/FFI_SHA256.json` pins all four archives by complete filename +and SHA-256. Keep `cpp-v*` and `moq-ffi-v*` separate; never select the newest +FFI release or use expiring Actions artifacts as a release input. + +The prebuilt CMake/build-script paths and publication gates are implemented. +Nightly packages reuse each target's checked same-run archive after its +installed consumers pass, without rebuilding moq-ffi. Direct CMake builds and +`just cpp check` use the current source checkout unless an explicit native +directory is supplied. Package-release builds select the pin when it is set; +a failed selected input never falls back to source. + +The [four-host dry-run](https://github.com/moq-dev/moq/actions/runs/38969347067) +passed on implementation commit `cbbcf9b2c92aa88c5550d9175adcc317f8137f4a`. +Linux x86, Linux ARM, macOS ARM and Windows MSVC passed installed C++17 and +C++23 consumers, including both coroutine probes. Unix also passed pkg-config +and doc samples. All four jobs then verified/extracted their checked archives +and produced C++ packages. Later commits change only documentation and the +tag-only release-creation notes, which that dry-run does not exercise. + +The [FFI build evidence](/.github/ffi.md#c-comparison) separates historical +archive/metadata comparisons from this current executable consumer proof. +Historical Windows assembly and Rust LTO comparisons do not prove complete +byte-for-byte code equivalence. The current four-host consumers have passed; +no older Linux runtime baseline is claimed. If source, +generator or configuration changes before activation, rerun the same four-host +gates on the final checkout. + +Keep this quest open for the published release input: + +- A real `moq-ffi-v*` release must publish all four default-feature native + archives and their checksums after its installed-consumer gates pass. No + existing release has this complete asset contract; no release or tag was + created during this work. +- Propose that release's exact version and four archive hashes, then validate + the actual downloaded release inputs through all four installed-consumer + paths before landing the pin. Same-run artifacts and metadata comparisons + alone do not satisfy this release-pin activation condition. +- Until those conditions hold, leave `FFI_VERSION` unset. Nightly may use its + validated same-run inputs; standalone package builds retain source builds. + Missing, malformed, incompatible or checksum-mismatched selected inputs fail. + +Follow-up to [#5549](https://github.com/moq-dev/moq/pull/5549), which shared +the default FFI build across Go, Kotlin and Swift. ## Related diff --git a/sh/cpp/check.sh b/sh/cpp/check.sh index f75839a1cf5..ddc2d416517 100755 --- a/sh/cpp/check.sh +++ b/sh/cpp/check.sh @@ -5,13 +5,23 @@ # and compiles the doc samples. CXX picks the compiler on Unix; Windows uses MSVC. set -euo pipefail +ffi_dir="" +if [[ $# -gt 0 ]]; then + if [[ $# != 2 || "$1" != --ffi-dir ]]; then + echo "Usage: $0 [--ffi-dir DIR]" >&2 + exit 1 + fi + ffi_dir="$(cd "$2" && pwd)" +fi + cd "$(dirname "${BASH_SOURCE[0]}")/../../cpp" # Single-config generators build Debug. Visual Studio ignores CMAKE_BUILD_TYPE and gets # --config Release, since the Rust staticlib links the release CRT (/MD) either way. build=moq/build +if [[ -n "$ffi_dir" ]]; then build=moq/build/prebuilt; fi prefix="$PWD/$build/prefix" -cmake -S moq -B "$build/package" -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_LIBDIR=lib +cmake -S moq -B "$build/package" -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_LIBDIR=lib -DMOQ_FFI_DIR="$ffi_dir" cmake --build "$build/package" --config Release rm -rf "$prefix" cmake --install "$build/package" --config Release --prefix "$prefix" diff --git a/sh/cpp/ffi.py b/sh/cpp/ffi.py new file mode 100644 index 00000000000..de754bd6a49 --- /dev/null +++ b/sh/cpp/ffi.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 +"""Stage and fetch the default-feature native input used by C++ packages.""" + +import argparse +import gzip +import hashlib +import json +import pathlib +import re +import shutil +import subprocess +import tarfile +import tempfile +import urllib.request + +ROOT = pathlib.Path(__file__).resolve().parents[2] +TARGETS = { + "x86_64-unknown-linux-gnu": ("libmoq_ffi.a", "libmoq_ffi.so", "linux"), + "aarch64-unknown-linux-gnu": ("libmoq_ffi.a", "libmoq_ffi.so", "linux"), + "aarch64-apple-darwin": ("libmoq_ffi.a", "libmoq_ffi.dylib", "apple"), + "x86_64-pc-windows-msvc": ("moq_ffi.lib", "moq_ffi.dll", "windows"), +} + + +def version(value): + if not re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?", value): + raise ValueError(f"unsupported FFI version: {value!r}") + return value + + +def name(release, target): + if target not in TARGETS: + raise ValueError(f"unsupported native target: {target!r}") + return f"moq-ffi-default-{version(release)}-{target}" + + +def stage(release, target, library, output): + version(release) + static, metadata, platform = TARGETS[target] + for filename in (static, metadata): + if not (library / filename).is_file(): + raise ValueError(f"missing native library: {library / filename}") + output.mkdir(parents=True, exist_ok=False) + (output / "lib").mkdir() + for filename in (static, metadata): + shutil.copyfile(library / filename, output / "lib" / filename) + shutil.copyfile(ROOT / f"rs/moq-c/native-libs/{platform}.txt", output / "native-libs.txt") + (output / "manifest.json").write_text(json.dumps({ + "version": version(release), "target": target, "features": "default", + "profile": "release", "source": subprocess.check_output( + ["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip(), + }, indent=2) + "\n") + + +def archive(directory, output): + manifest = json.loads((directory / "manifest.json").read_text()) + filename = name(manifest["version"], manifest["target"]) + ".tar.gz" + output.mkdir(parents=True, exist_ok=True) + path = output / filename + # Deterministic headers let reruns refuse changed payloads without timestamp noise. + with tempfile.TemporaryFile() as raw: + with tarfile.open(fileobj=raw, mode="w") as tar: + for path_in in sorted(directory.rglob("*")): + info = tar.gettarinfo(str(path_in), str(path_in.relative_to(directory))) + info.uid = info.gid = info.mtime = 0 + info.uname = info.gname = "" + with path_in.open("rb") if path_in.is_file() else tempfile.TemporaryFile() as data: + tar.addfile(info, data if path_in.is_file() else None) + raw.seek(0) + with path.open("wb") as dest, gzip.GzipFile(filename="", mode="wb", fileobj=dest, mtime=0) as gz: + shutil.copyfileobj(raw, gz) + digest = hashlib.sha256(path.read_bytes()).hexdigest() + path.with_suffix(path.suffix + ".sha256").write_text(f"{digest} {filename}\n") + + +def fetch(target, output): + pins = [line.strip() for line in (ROOT / "cpp/moq/FFI_VERSION").read_text().splitlines() + if line.strip() and not line.strip().startswith("#")] + if len(pins) != 1: + raise ValueError("FFI_VERSION must select one published version before fetching") + release = version(pins[0]) + filename = name(release, target) + ".tar.gz" + hashes = json.loads((ROOT / "cpp/moq/FFI_SHA256.json").read_text()) + for native_target in TARGETS: + asset = name(release, native_target) + ".tar.gz" + if not re.fullmatch(r"[a-f0-9]{64}", hashes.get(asset, "")): + raise ValueError(f"FFI_SHA256.json lacks a SHA-256 pin for {asset}") + expected = hashes[filename] + if output.exists(): + raise ValueError(f"output already exists: {output}") + with tempfile.TemporaryDirectory() as tmp: + path = pathlib.Path(tmp) / filename + urllib.request.urlretrieve( + f"https://github.com/moq-dev/moq/releases/download/moq-ffi-v{release}/{filename}", path) + unpack(path, output, release, target, expected) + + +def unpack(path, output, release, target, expected): + if output.exists(): + raise ValueError(f"output already exists: {output}") + if hashlib.sha256(path.read_bytes()).hexdigest() != expected: + raise ValueError(f"checksum mismatch: {path.name}") + with tempfile.TemporaryDirectory() as tmp: + extracted = pathlib.Path(tmp) / "extracted" + # These archives contain only directories and regular files at their root. + with tarfile.open(path) as tar: + for member in tar.getmembers(): + rel = pathlib.PurePosixPath(member.name) + if rel.is_absolute() or ".." in rel.parts or "\\" in member.name or not (member.isfile() or member.isdir()): + raise ValueError(f"unsupported archive member: {member.name}") + extracted.mkdir() + tar.extractall(extracted, filter="data") + manifest = json.loads((extracted / "manifest.json").read_text()) + if (manifest["version"], manifest["target"], manifest["features"], manifest["profile"]) != ( + release, target, "default", "release" + ): + raise ValueError("downloaded native manifest disagrees with the selected release input") + output.parent.mkdir(parents=True, exist_ok=True) + shutil.move(str(extracted), output) + + +def extract(release, target, directory, output): + filename = name(release, target) + ".tar.gz" + sidecar = (directory / (filename + ".sha256")).read_text() + if not re.fullmatch(r"[a-f0-9]{64} " + re.escape(filename) + r"\n", sidecar): + raise ValueError(f"malformed native checksum: {filename}") + unpack(directory / filename, output, release, target, sidecar[:64]) + + +def publish(release, directory, prev_tag): + tag = f"moq-ffi-v{version(release)}" + files = sorted(directory.iterdir()) + if len(files) != 8 or {p.name for p in files} != { + name(release, target) + suffix for target in TARGETS for suffix in (".tar.gz", ".tar.gz.sha256") + }: + raise ValueError("publication requires all four native archives and checksums") + for target in TARGETS: + filename = name(release, target) + ".tar.gz" + expected = hashlib.sha256((directory / filename).read_bytes()).hexdigest() + if (directory / (filename + ".sha256")).read_text() != f"{expected} {filename}\n": + raise ValueError(f"native checksum disagrees with archive: {filename}") + # Never clobber release inputs. Identical reruns are harmless; changed bytes fail. + result = subprocess.run(["gh", "release", "view", tag], capture_output=True) + if result.returncode: + if b"release not found" not in result.stderr.lower(): + raise RuntimeError(result.stderr.decode()) + # Match the other moq-ffi publishers (sh/gh/release.sh) so the notes never depend on who wins the race. + notes = ["--generate-notes"] + (["--notes-start-tag", prev_tag] if prev_tag else []) + created = subprocess.run(["gh", "release", "create", tag, "--verify-tag", "--title", f"moq-ffi v{release}", *notes]) + if created.returncode: + # Another language publisher can create the same release concurrently. + subprocess.run(["gh", "release", "view", tag], check=True) + assets = json.loads(subprocess.check_output( + ["gh", "release", "view", tag, "--json", "assets"], text=True))["assets"] + existing = {asset["name"] for asset in assets} + with tempfile.TemporaryDirectory() as tmp: + for path in files: + if path.name in existing: + subprocess.run(["gh", "release", "download", tag, "--pattern", path.name, "--dir", tmp], check=True) + if (pathlib.Path(tmp) / path.name).read_bytes() != path.read_bytes(): + raise ValueError(f"refusing to replace published FFI asset: {path.name}") + for path in files: + if path.name not in existing: + subprocess.run(["gh", "release", "upload", tag, str(path)], check=True) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + commands = parser.add_subparsers(dest="command", required=True) + for command in ("stage", "fetch", "archive", "extract", "publish"): + sub = commands.add_parser(command) + if command in ("stage", "extract", "publish"): + sub.add_argument("--version", required=True) + if command in ("stage", "fetch", "extract"): + sub.add_argument("--target", required=True, choices=TARGETS) + if command == "stage": + sub.add_argument("--library", type=pathlib.Path, required=True) + if command in ("archive", "extract", "publish"): + sub.add_argument("--directory", type=pathlib.Path, required=True) + if command == "publish": + sub.add_argument("--prev-tag", default="") + if command != "publish": + sub.add_argument("--output", type=pathlib.Path, required=True) + args = parser.parse_args() + if args.command == "stage": + stage(args.version, args.target, args.library, args.output) + elif args.command == "archive": + archive(args.directory, args.output) + elif args.command == "fetch": + fetch(args.target, args.output) + elif args.command == "extract": + extract(args.version, args.target, args.directory, args.output) + else: + publish(args.version, args.directory, args.prev_tag) + + +if __name__ == "__main__": + main() diff --git a/sh/cpp/ffi.test.py b/sh/cpp/ffi.test.py new file mode 100644 index 00000000000..a84ed33f53c --- /dev/null +++ b/sh/cpp/ffi.test.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python3 +"""Exercise pinned native inputs without network access or Rust compilation.""" + +import hashlib +import importlib.util +import io +import json +import os +import pathlib +import platform +import shutil +import subprocess +import tarfile +import tempfile +import unittest +from unittest.mock import patch + +REPO = pathlib.Path(__file__).resolve().parents[2] +SPEC = importlib.util.spec_from_file_location("ffi", pathlib.Path(__file__).with_name("ffi.py")) +ffi = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(ffi) + + +class Inputs(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.root = pathlib.Path(self.tmp.name) + self.patch = patch.object(ffi, "ROOT", self.root) + self.patch.start() + self.addCleanup(self.patch.stop) + (self.root / "cpp/moq").mkdir(parents=True) + (self.root / "cpp/moq/FFI_VERSION").write_text("1.2.3\n") + (self.root / "cpp/moq/FFI_SHA256.json").write_text("{}") + + def pin(self, path): + (self.root / "cpp/moq/FFI_SHA256.json").write_text(json.dumps({ + ffi.name("1.2.3", target) + ".tar.gz": hashlib.sha256(path.read_bytes()).hexdigest() + for target in ffi.TARGETS + })) + + def test_fetch_requires_a_real_pin_before_network(self): + with patch.object(ffi.urllib.request, "urlretrieve") as download: + with self.assertRaisesRegex(ValueError, "SHA-256 pin"): + ffi.fetch("x86_64-unknown-linux-gnu", self.root / "out") + download.assert_not_called() + + def test_fetch_rejects_absent_or_malformed_selection_before_network(self): + for pin in ("", "# unset\n", "1.2.3\n4.5.6\n", "latest\n"): + (self.root / "cpp/moq/FFI_VERSION").write_text(pin) + with patch.object(ffi.urllib.request, "urlretrieve") as download: + with self.assertRaises(ValueError): + ffi.fetch("x86_64-unknown-linux-gnu", self.root / "out") + download.assert_not_called() + + def test_checksum_mismatch_never_extracts(self): + path = self.root / "bad" + path.write_bytes(b"changed") + self.pin(path) + path.write_bytes(b"replacement") + def download(url, dest): + self.assertIn("moq-ffi-v1.2.3/moq-ffi-default-1.2.3-", url) + shutil.copyfile(path, dest) + with patch.object(ffi.urllib.request, "urlretrieve", side_effect=download): + with self.assertRaisesRegex(ValueError, "checksum mismatch"): + ffi.fetch("x86_64-unknown-linux-gnu", self.root / "out") + self.assertFalse((self.root / "out").exists()) + + def test_checked_archive_refuses_paths_and_symlinks(self): + for member in ("../escape", "/escape", "link"): + path = self.root / "bad.tar.gz" + with tarfile.open(path, "w:gz") as tar: + info = tarfile.TarInfo(member) + if member == "link": + info.type = tarfile.SYMTYPE + info.linkname = "../escape" + tar.addfile(info, io.BytesIO(b"")) + self.pin(path) + with patch.object(ffi.urllib.request, "urlretrieve", side_effect=lambda url, dest: shutil.copyfile(path, dest)): + with self.assertRaisesRegex(ValueError, "archive member"): + ffi.fetch("x86_64-unknown-linux-gnu", self.root / "out") + self.assertFalse((self.root / "out").exists()) + + def test_reproducible_archive_roundtrip(self): + directory = self.root / "input" + directory.mkdir() + (directory / "manifest.json").write_text(json.dumps({"version": "1.2.3", "target": "x86_64-unknown-linux-gnu", "features": "default", "profile": "release"})) + (directory / "library").write_bytes(b"static native library") + ffi.archive(directory, self.root / "archives") + path = next((self.root / "archives").glob("*.tar.gz")) + before = path.read_bytes() + os.utime(directory / "library", (42, 42)) + ffi.archive(directory, self.root / "archives") + self.assertEqual(before, path.read_bytes()) + self.pin(path) + with patch.object(ffi.urllib.request, "urlretrieve", side_effect=lambda url, dest: shutil.copyfile(path, dest)): + ffi.fetch("x86_64-unknown-linux-gnu", self.root / "out") + self.assertEqual((self.root / "out/library").read_bytes(), b"static native library") + + def test_same_run_extract_checks_sidecar_and_manifest(self): + directory = self.root / "input" + directory.mkdir() + manifest = {"version": "1.2.3", "target": "x86_64-unknown-linux-gnu", "features": "default", "profile": "release"} + (directory / "manifest.json").write_text(json.dumps(manifest)) + ffi.archive(directory, self.root / "archives") + ffi.extract("1.2.3", manifest["target"], self.root / "archives", self.root / "good") + self.assertEqual(json.loads((self.root / "good/manifest.json").read_text()), manifest) + path = next((self.root / "archives").glob("*.tar.gz")) + path.write_bytes(b"changed") + with self.assertRaisesRegex(ValueError, "checksum mismatch"): + ffi.extract("1.2.3", manifest["target"], self.root / "archives", self.root / "bad") + self.assertFalse((self.root / "bad").exists()) + + def test_checksummed_archive_must_match_selected_manifest(self): + directory = self.root / "input" + directory.mkdir() + manifest = {"version": "9.9.9", "target": "x86_64-unknown-linux-gnu", "features": "default", "profile": "release"} + for field, value in (("version", "9.9.9"), ("target", "aarch64-unknown-linux-gnu"), ("features", "minimal"), ("profile", "debug")): + current = dict(manifest, version="1.2.3", **{field: value}) if field != "version" else manifest + (directory / "manifest.json").write_text(json.dumps(current)) + ffi.archive(directory, self.root / "archives") + path = self.root / "archives" / (ffi.name(current["version"], current["target"]) + ".tar.gz") + self.pin(path) + output = self.root / ("out-" + field) + with patch.object(ffi.urllib.request, "urlretrieve", side_effect=lambda url, dest: shutil.copyfile(path, dest)): + with self.assertRaisesRegex(ValueError, "manifest disagrees"): + ffi.fetch("x86_64-unknown-linux-gnu", output) + self.assertFalse(output.exists()) + + def test_publish_rejects_bad_sidecar_before_network(self): + directory = self.root / "assets" + directory.mkdir() + for target in ffi.TARGETS: + for suffix in (".tar.gz", ".tar.gz.sha256"): + (directory / (ffi.name("1.2.3", target) + suffix)).write_bytes(b"bad") + with patch.object(ffi.subprocess, "run") as command: + with self.assertRaisesRegex(ValueError, "checksum disagrees"): + ffi.publish("1.2.3", directory, "") + command.assert_not_called() + + def test_publish_refuses_changed_existing_asset(self): + directory = self.root / "assets" + directory.mkdir() + for target in ffi.TARGETS: + filename = ffi.name("1.2.3", target) + ".tar.gz" + (directory / filename).write_bytes(b"new") + (directory / (filename + ".sha256")).write_text(f"{hashlib.sha256(b'new').hexdigest()} {filename}\n") + existing = next(directory.iterdir()).name + def run(args, **kwargs): + if args[2] == "download": + (pathlib.Path(args[-1]) / existing).write_bytes(b"old") + return subprocess.CompletedProcess(args, 0) + with patch.object(ffi.subprocess, "run", side_effect=run) as calls, patch.object( + ffi.subprocess, "check_output", return_value=json.dumps({"assets": [{"name": existing}]})): + with self.assertRaisesRegex(ValueError, "refusing to replace"): + ffi.publish("1.2.3", directory, "") + self.assertFalse(any(call.args[0][2] == "upload" for call in calls.call_args_list)) + + + def test_publish_creates_release_like_other_ffi_publishers(self): + directory = self.root / "assets" + directory.mkdir() + for target in ffi.TARGETS: + filename = ffi.name("1.2.3", target) + ".tar.gz" + (directory / filename).write_bytes(b"new") + (directory / (filename + ".sha256")).write_text(f"{hashlib.sha256(b'new').hexdigest()} {filename}\n") + def run(args, **kwargs): + if args[2] == "view": + return subprocess.CompletedProcess(args, 1, b"", b"release not found") + return subprocess.CompletedProcess(args, 0) + with patch.object(ffi.subprocess, "run", side_effect=run) as calls, patch.object( + ffi.subprocess, "check_output", return_value=json.dumps({"assets": []})): + ffi.publish("1.2.3", directory, "moq-ffi-v1.2.2") + create = next(call.args[0] for call in calls.call_args_list if call.args[0][2] == "create") + self.assertEqual(create[-3:], ["--generate-notes", "--notes-start-tag", "moq-ffi-v1.2.2"]) + self.assertNotIn("--notes", create) + self.assertEqual(sum(call.args[0][2] == "upload" for call in calls.call_args_list), 8) + +@unittest.skipUnless(platform.system() == "Linux", "Linux CMake input fixtures") +class Configure(unittest.TestCase): + def test_prebuilt_generation_never_builds_rust_and_refuses_bad_manifest(self): + with tempfile.TemporaryDirectory() as tmp: + root = pathlib.Path(tmp) + source = root / "source" + shutil.copytree(REPO / "cpp/moq", source, ignore=shutil.ignore_patterns("build")) + native = root / "native" + (native / "lib").mkdir(parents=True) + for name in ("libmoq_ffi.a", "libmoq_ffi.so"): + (native / "lib" / name).touch() + (native / "native-libs.txt").write_text("# no linker inputs needed by this fixture\n") + manifest = {"version": "1.2.3", "target": "aarch64-unknown-linux-gnu" if platform.machine() == "aarch64" else "x86_64-unknown-linux-gnu", + "features": "default", "profile": "release", "source": "a" * 40} + tools = root / "tools" + tools.mkdir() + generator = tools / "uniffi-bindgen-cpp" + generator.write_text("#!/usr/bin/env python3\nimport pathlib, sys\nout=pathlib.Path(sys.argv[sys.argv.index('--out-dir')+1]); out.mkdir(parents=True)\nfor name in ('moq.hpp','moq.cpp','moq_scaffolding.hpp','uniffi_expected.hpp'): (out/name).touch()\n") + generator.chmod(0o755) + cargo = tools / "cargo" + cargo.write_text("#!/usr/bin/env bash\necho unexpected Rust build >&2\nexit 99\n") + cargo.chmod(0o755) + env = dict(os.environ, PATH=str(tools) + os.pathsep + os.environ["PATH"]) + def configure(): + return subprocess.run(["cmake", "-S", str(source), "-B", str(root / "good"), + f"-DMOQ_FFI_DIR={native}"], env=env, capture_output=True, text=True) + (native / "manifest.json").write_text(json.dumps(manifest)) + configured = configure() + self.assertEqual(configured.returncode, 0, configured.stdout + configured.stderr) + built = subprocess.run(["cmake", "--build", str(root / "good"), "--target", "moq_ffi"], + env=env, capture_output=True, text=True) + self.assertEqual(built.returncode, 0, built.stdout + built.stderr) + for field, value in (("target", "unsupported"), ("features", "minimal"), ("profile", "debug"), ("version", "unsupported")): + (native / "manifest.json").write_text(json.dumps(dict(manifest, **{field: value}))) + self.assertNotEqual(configure().returncode, 0) + + +if __name__ == "__main__": + unittest.main() diff --git a/sh/gh/nightly.test.py b/sh/gh/nightly.test.py index 2fbcb8eae6f..006df748bf9 100644 --- a/sh/gh/nightly.test.py +++ b/sh/gh/nightly.test.py @@ -18,7 +18,7 @@ def run_group(self, group, fail=""): for command in ("just", "cargo"): stub = scratch / command stub.write_text( - '#!/bin/bash\n' + '#!/usr/bin/env bash\n' 'printf "%s %s\n" "${0##*/}" "$*" >> "$NIGHTLY_CALLS"\n' '[[ "$*" != "$NIGHTLY_FAIL" ]]\n' )