diff --git a/.github/ffi.md b/.github/ffi.md new file mode 100644 index 00000000000..bffbfd4d6c9 --- /dev/null +++ b/.github/ffi.md @@ -0,0 +1,62 @@ +# Shared FFI releases + +`release-shared-ffi.yml` owns the existing `moq-ffi-v*` tag trigger for Go, +Kotlin and Swift. Nightly calls the same coordinator without publisher secrets. +It builds nine distinct default-feature targets, replacing fourteen native +builds across those packages. Dart retains its eleven minimal-feature targets; +Python retains its wheel builds and Python-specific environment. + +C++ retains its host CMake builds and independent `cpp-v*` release tags. All +four sampled C++ static libraries have different hashes from the candidates +shared here. In particular, ARM Linux builds natively on Ubuntu 24.04 rather +than on the shared cross-compilation host. Sharing those packages needs +separate artifact/ABI proof and a settled prebuilt-library CMake interface. + +The shared build uploads `default-lib-`. Binding jobs read the built +Linux library with `just rs package-ffi --bindings-only --library PATH`, so +binding generation does not compile moq-ffi again. Package workflows select +only their own targets, retain language binding names and package outputs, +and keep their existing publisher steps and credentials. Manual calls remain +standalone. Go's PR dry-run also remains standalone. + +A target failure must not suppress a package that does not consume that target. +The build matrix and binding matrix do not fail fast; packaging still runs +after a failed shared job and refuses missing artifacts. The wrapper workflows +accept only the existing trusted owner/push/tag inputs. Their read-only +preflight checks the completed coordinator's own language publisher and jobs, +so a failed Android build does not prevent a successful Go wrapper release. +The [jobs API](https://docs.github.com/en/rest/actions/workflow-jobs#list-jobs-for-a-workflow-run) +query includes all attempts and uses each job's latest execution. A publisher +counts only if it ran in the triggering attempt: a partial rerun copies earlier +successes with their original start time, and their wrappers already ran when +that attempt completed. Rerunning only Kotlin on an old tag therefore cannot +re-cut the Go wrapper from old sources after a newer tag published, while a +rerun publisher still releases its wrapper. Missing publisher names or an API +error fail loudly. No publication is polled. +The legacy per-language workflow names remain accepted for in-flight runs. + +## Evidence and validation + +[The sampled Nightly](https://github.com/moq-dev/moq/actions/runs/38564590314) +contains fourteen default-feature native builds. Downloaded library payloads +from all nine uploads for duplicated targets confirm: + +| Target | Consumers | Comparison | +| --- | --- | --- | +| x86\_64-unknown-linux-gnu | Go, Kotlin | static and shared libraries have identical SHA-256 hashes | +| aarch64-unknown-linux-gnu | Go, Kotlin | static and shared libraries have identical SHA-256 hashes | +| aarch64-apple-darwin | Go, Kotlin, Swift | static and shared libraries have identical SHA-256 hashes | +| x86\_64-pc-windows-msvc | Go, Kotlin | import library identical; DLL and static library differ only in timestamps and DLL debug GUID | + +The Windows DLL has 23 differing bytes, all in PE/COFF timestamps, debug-directory +timestamps and the CodeView GUID. The static archive has 85 differing bytes, +all in COFF timestamps. Zeroing those metadata fields makes both payloads +byte-identical; no executable code or binding metadata differs. The source +build command is the same target/default-feature Cargo invocation for every +consumer. Sharing preserves the same native code and library files. + +`just gh test` runs local fixtures for artifact subsets, +build-free binding generation, independent language outcomes, failed API +queries and workflow guards. `just gh check` validates workflow syntax and +alert coverage. The first Nightly after landing measures actual elapsed time; +the sampled five duplicate native builds cost about seventy runner-minutes. diff --git a/.github/justfile b/.github/justfile index 1bccfc53380..df327809b99 100644 --- a/.github/justfile +++ b/.github/justfile @@ -10,6 +10,8 @@ check: test: python3 sh/gh/install-formula.test.py sh/gh/release.test.sh + python3 sh/gh/ffi.test.py + bun test sh/gh/ffi.test.ts python3 sh/rs/package-rename.test.py sh/go/publish-wrapper.test.sh bash .claude/hooks/direnv.test.sh @@ -38,3 +40,8 @@ install-formula *args: [positional-arguments] trigger-repo-publish *args: sh/gh/trigger-repo-publish.sh "$@" + +# Select shared native artifacts or check an upstream FFI publisher. +[positional-arguments] +ffi *args: + python3 sh/gh/ffi.py "$@" diff --git a/.github/workflows/alert.yml b/.github/workflows/alert.yml index 2c4084d32e2..36769cf5b69 100644 --- a/.github/workflows/alert.yml +++ b/.github/workflows/alert.yml @@ -43,6 +43,7 @@ on: - release-brew - Release Dart - Release Dart FFI + - Release FFI - Release Go - Release Go FFI - Release JS diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index dca9159c37e..a39c2997942 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -280,24 +280,12 @@ jobs: contents: read id-token: write - release-kt-ffi: - name: Release Kotlin FFI - uses: ./.github/workflows/release-kt-ffi.yml - permissions: - contents: read - - release-swift-ffi: - name: Release Swift FFI - uses: ./.github/workflows/release-swift-ffi.yml + release-shared-ffi: + name: Release FFI + uses: ./.github/workflows/release-shared-ffi.yml permissions: contents: write - release-go-ffi: - name: Release Go FFI - uses: ./.github/workflows/release-go-ffi.yml - permissions: - contents: read - release-dart-ffi: name: Release Dart FFI uses: ./.github/workflows/release-dart-ffi.yml diff --git a/.github/workflows/release-ffi.yml b/.github/workflows/release-ffi.yml index ff7cfc85aee..44937e13f23 100644 --- a/.github/workflows/release-ffi.yml +++ b/.github/workflows/release-ffi.yml @@ -1,14 +1,8 @@ name: release-ffi -# Builds rs/moq-ffi for each requested target and generates its UniFFI -# bindings. The release-{dart,go,kt,swift}-ffi.yml callers package and publish; -# they keep the moq-ffi-v* tag trigger and the workflow name that alert.yml and -# the workflow_run chains match on. -# -# Uploads, all prefixed by `language` so callers sharing one nightly run never -# collide: -# -lib- the target's native libraries (the lib/ directory) -# -bindings dist/bindings//, when `bindings` is set +# Builds each requested target and generates bindings. The shared coordinator +# uploads default-lib-; standalone callers retain their language prefix. +# Binding artifacts retain their language names in either mode. on: workflow_call: @@ -29,7 +23,18 @@ on: description: "Build moq-ffi with its default features (audio and video)" type: boolean default: true + reuse-native: + description: "Consume the coordinator's native libraries and bindings" + type: boolean + default: false + binding-languages: + description: "JSON array of binding languages; empty means language" + type: string + default: "" outputs: + targets: + description: "Requested targets, for selecting and staging native artifacts" + value: ${{ jobs.parse-version.outputs.targets }} version: description: "moq-ffi version, from the tag or (off a tag) rs/moq-ffi/Cargo.toml" value: ${{ jobs.parse-version.outputs.version }} @@ -47,6 +52,7 @@ jobs: name: Parse version runs-on: ubuntu-latest outputs: + targets: ${{ inputs.targets }} version: ${{ steps.parse.outputs.version }} steps: @@ -63,6 +69,7 @@ jobs: build: name: Build moq-ffi (${{ matrix.target }}) needs: [parse-version] + if: ${{ !inputs.reuse-native }} runs-on: ${{ contains(matrix.target, '-apple-') && 'macos-latest' || contains(matrix.target, '-windows-') && 'windows-latest' || 'ubuntu-latest' }} strategy: @@ -120,10 +127,16 @@ jobs: path: dist/moq-ffi-${{ needs.parse-version.outputs.version }}-${{ matrix.target }}/lib/ bindings: - name: Generate bindings - needs: [parse-version] - if: inputs.bindings + name: Generate bindings (${{ matrix.language }}) + needs: [parse-version, build] + # An unrelated target failure must not suppress bindings for successful + # consumers. A missing Linux artifact fails its download below. + if: ${{ always() && !cancelled() && inputs.bindings && !inputs.reuse-native }} runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + language: ${{ fromJSON(inputs.binding-languages || format('["{0}"]', inputs.language)) }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -140,7 +153,7 @@ jobs: # Go uses a third-party generator; build.sh skips Go when it is absent. - name: Install uniffi-bindgen-go - if: inputs.language == 'go' + if: matrix.language == 'go' env: # A fork until NordSecurity ships a uniffi 0.32 generator; see flake.nix. # The revision is v0.10.0-kixelated.1+v0.32.0. @@ -151,19 +164,32 @@ jobs: --git "$UNIFFI_BINDGEN_GO_REPO" \ --rev "$UNIFFI_BINDGEN_GO_REV" + - name: Download binding metadata + if: contains(fromJSON(inputs.targets), 'x86_64-unknown-linux-gnu') + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: ${{ inputs.language }}-lib-x86_64-unknown-linux-gnu + path: binding-library + - name: Generate bindings env: BUILD_VERSION: ${{ needs.parse-version.outputs.version }} + REUSE_LIBRARY: ${{ contains(fromJSON(inputs.targets), 'x86_64-unknown-linux-gnu') }} + DEFAULT_FEATURES: ${{ inputs.default-features }} run: | - just rs package-ffi \ - --bindings-only \ - --version "$BUILD_VERSION" \ - --output dist + args=(--bindings-only --version "$BUILD_VERSION" --output dist) + if [[ "$REUSE_LIBRARY" == true ]]; then + args+=(--library binding-library/libmoq_ffi.so) + fi + if [[ "$DEFAULT_FEATURES" == false ]]; then + args+=(--no-default-features) + fi + just rs package-ffi "${args[@]}" - name: Upload bindings uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: - name: ${{ inputs.language }}-bindings - path: dist/bindings/${{ inputs.language }}/ + name: ${{ matrix.language }}-bindings + path: dist/bindings/${{ matrix.language }}/ # A language build.sh does not generate would otherwise upload nothing. if-no-files-found: error diff --git a/.github/workflows/release-go-ffi.yml b/.github/workflows/release-go-ffi.yml index e35e940046b..f3a31542f74 100644 --- a/.github/workflows/release-go-ffi.yml +++ b/.github/workflows/release-go-ffi.yml @@ -6,17 +6,20 @@ name: Release Go FFI # (release-go.yml), which also auto-bumps its require to each new ffi here. on: - push: - tags: - - "moq-ffi-v*" - # Dry-run: build everything, publish nothing. Nightly calls every release - # workflow this way so a broken build surfaces before the tag is cut. + # The tag coordinator supplies shared default-feature artifacts. Manual + # calls and Go PR dry-runs still build their own target subset. workflow_call: + inputs: + reuse-native: + type: boolean + default: false workflow_dispatch: pull_request: paths: - ".github/workflows/release-go-ffi.yml" - ".github/workflows/release-ffi.yml" + - ".github/workflows/release-shared-ffi.yml" + - "sh/gh/ffi*" - "sh/gh/release.sh" - "go/ffi/**" - "sh/go/package-ffi.sh" @@ -40,6 +43,7 @@ jobs: uses: ./.github/workflows/release-ffi.yml with: language: go + reuse-native: ${{ inputs.reuse-native || false }} targets: >- [ "x86_64-unknown-linux-gnu", @@ -60,20 +64,24 @@ jobs: - uses: ./.github/actions/just + - name: Select native artifacts + id: native + env: + LIB_PREFIX: ${{ inputs.reuse-native && 'default' || 'go' }} + BUILD_TARGETS: ${{ needs.ffi.outputs.targets }} + run: just gh ffi pattern + - name: Download per-target libs uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: path: libs-raw - pattern: go-lib-* + pattern: ${{ steps.native.outputs.pattern }} - name: Flatten lib layout - run: | - mkdir -p libs - for dir in libs-raw/go-lib-*; do - target="${dir#libs-raw/go-lib-}" - mv "$dir" "libs/$target" - done - ls -la libs + env: + LIB_PREFIX: ${{ inputs.reuse-native && 'default' || 'go' }} + BUILD_TARGETS: ${{ needs.ffi.outputs.targets }} + run: just gh ffi stage - name: Download bindings uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 diff --git a/.github/workflows/release-go.yml b/.github/workflows/release-go.yml index 759d4ad19c2..260ac06393d 100644 --- a/.github/workflows/release-go.yml +++ b/.github/workflows/release-go.yml @@ -14,7 +14,7 @@ name: Release Go # release is mid-flight) or rs/moq-ffi changed since # its tag, the job skips: the workflow_run below # republishes once the ffi lands. -# * Release Go FFI done — a new ffi just published to the moq-go-ffi mirror. +# * Release FFI done: its Go publisher succeeded on the moq-go-ffi mirror. # Re-cut the wrapper so `moq.dev/moq@latest` floats to # the newest native core (Go MVS won't float on its own). # Chaining off the upstream run guarantees the mirror @@ -28,7 +28,7 @@ on: branches: - release workflow_run: - workflows: ["Release Go FFI"] + workflows: ["Release FFI", "Release Go FFI"] types: - completed @@ -42,20 +42,44 @@ concurrency: cancel-in-progress: false jobs: - build: - name: Build wrapper module + ffi-release: + name: Check FFI publisher runs-on: ubuntu-latest - # Owner gate, plus: only react to a successful Release Go FFI run for a - # moq-ffi-v* tag. Require event == 'push' so a fork PR whose head branch is - # named `moq-ffi-v*` can't satisfy head_branch and reach the publish secrets - # (workflow_run runs in the base repo's privileged context). Pushes here are - # unconditional. + # workflow_run is privileged. Preserve the owner, same-repo, push and tag + # gates before reading the trusted upstream checkout or allowing wrapper + # publication. if: >- github.repository_owner == 'moq-dev' && (github.event_name == 'push' || - (github.event.workflow_run.conclusion == 'success' && + ((github.event.workflow_run.name == 'Release FFI' || github.event.workflow_run.conclusion == 'success') && github.event.workflow_run.event == 'push' && + github.event.workflow_run.head_repository.full_name == github.repository && startsWith(github.event.workflow_run.head_branch, 'moq-ffi-v'))) + permissions: + contents: read + actions: read + outputs: + ready: ${{ steps.publisher.outputs.ready }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.sha }} + persist-credentials: false + - uses: ./.github/actions/just + - name: Check independent publisher + id: publisher + env: + GH_TOKEN: ${{ github.token }} + UPSTREAM_NAME: ${{ github.event.workflow_run.name }} + UPSTREAM_RUN_ID: ${{ github.event.workflow_run.id }} + UPSTREAM_RUN_STARTED_AT: ${{ github.event.workflow_run.run_started_at }} + run: just gh ffi completed go + + build: + name: Build wrapper module + needs: [ffi-release] + if: needs.ffi-release.outputs.ready == 'true' + runs-on: ubuntu-latest outputs: skip: ${{ steps.gate.outputs.skip }} @@ -94,7 +118,7 @@ jobs: if [[ "$GITHUB_EVENT_NAME" == "workflow_run" ]]; then # Upstream tag ref, e.g. moq-ffi-v0.3.1. VERSION="${HEAD_BRANCH#moq-ffi-v}" - echo "Triggered by Release Go FFI: ffi v$VERSION" + echo "Triggered by FFI publisher: ffi v$VERSION" else VERSION=$(grep '^version' rs/moq-ffi/Cargo.toml | head -1 | sed 's/.*"\(.*\)".*/\1/') echo "Using ffi version from Cargo.toml: $VERSION" @@ -111,7 +135,7 @@ jobs: run: just gh release ffi-unreleased "$FFI_VERSION" # Skip on unreleased ffi changes, and when the pinned ffi's mirror tag is - # missing (release-plz merged but Release Go FFI hasn't published yet): + # missing (release-plz merged but the Go FFI publisher hasn't finished yet): # the workflow_run trigger republishes once the ffi lands. One existence # check, no polling. workflow_run runs are chained after that publish, so # the tag is guaranteed present and this gate doesn't run for them. @@ -135,7 +159,7 @@ jobs: exit 1 fi if [[ -z "$TAGS" ]]; then - echo "::warning::moq-go-ffi v${FFI_VERSION} not on the mirror yet; deferring to the Release Go FFI chain." + echo "::warning::moq-go-ffi v${FFI_VERSION} not on the mirror yet; deferring to the FFI release chain." echo "skip=true" >> "$GITHUB_OUTPUT" exit 0 fi @@ -153,7 +177,7 @@ jobs: if grep -qE '^module[[:space:]]+moq\.dev/moq-ffi$' <<< "$MOD"; then echo "moq-go-ffi v${FFI_VERSION} is published and declares moq.dev/moq-ffi; proceeding." else - echo "::warning::moq-go-ffi v${FFI_VERSION} predates the module rename; deferring to the Release Go FFI chain." + echo "::warning::moq-go-ffi v${FFI_VERSION} predates the module rename; deferring to the FFI release chain." echo "skip=true" >> "$GITHUB_OUTPUT" fi @@ -174,7 +198,8 @@ jobs: publish: name: Publish to Go module mirror - needs: [build] + # Directly on ffi-release: its gate is what makes the head_sha checkout trusted. + needs: [ffi-release, build] if: needs.build.outputs.skip != 'true' runs-on: ubuntu-latest diff --git a/.github/workflows/release-kt-ffi.yml b/.github/workflows/release-kt-ffi.yml index 0b48e10b929..cffcab9534d 100644 --- a/.github/workflows/release-kt-ffi.yml +++ b/.github/workflows/release-kt-ffi.yml @@ -5,12 +5,13 @@ name: Release Kotlin FFI # `dev.moq:moq` wrapper is published separately by release-kt-lib.yml. on: - push: - tags: - - "moq-ffi-v*" - # Dry-run: build everything, publish nothing. Nightly calls every release - # workflow this way so a broken build surfaces before the tag is cut. + # The tag coordinator supplies shared default-feature artifacts. Manual + # calls and Go PR dry-runs still build their own target subset. workflow_call: + inputs: + reuse-native: + type: boolean + default: false workflow_dispatch: permissions: @@ -26,6 +27,7 @@ jobs: uses: ./.github/workflows/release-ffi.yml with: language: kotlin + reuse-native: ${{ inputs.reuse-native || false }} # Android, then the desktop JVM. targets: >- [ @@ -72,20 +74,24 @@ jobs: - uses: gradle/actions/setup-gradle@9c971963bec38e04b3d30dcc455b5382be2fdbfb # v6.3.0 + - name: Select native artifacts + id: native + env: + LIB_PREFIX: ${{ inputs.reuse-native && 'default' || 'kotlin' }} + BUILD_TARGETS: ${{ needs.ffi.outputs.targets }} + run: just gh ffi pattern + - name: Download per-target libs uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: path: libs-raw - pattern: kotlin-lib-* + pattern: ${{ steps.native.outputs.pattern }} - name: Flatten lib layout - run: | - mkdir -p libs - for dir in libs-raw/kotlin-lib-*; do - target="${dir#libs-raw/kotlin-lib-}" - mv "$dir" "libs/$target" - done - ls -la libs + env: + LIB_PREFIX: ${{ inputs.reuse-native && 'default' || 'kotlin' }} + BUILD_TARGETS: ${{ needs.ffi.outputs.targets }} + run: just gh ffi stage - name: Download bindings uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 diff --git a/.github/workflows/release-kt-lib.yml b/.github/workflows/release-kt-lib.yml index 7df333c6b3a..a6a1db527fd 100644 --- a/.github/workflows/release-kt-lib.yml +++ b/.github/workflows/release-kt-lib.yml @@ -16,8 +16,8 @@ name: Release Kotlin lib # # Those bindings come from this checkout, so a push is deferred while rs/moq-ffi # has changes since its release tag: the wrapper could call FFI API that no -# published moq-ffi has. Release Kotlin FFI re-triggers this workflow once the -# next moq-ffi release lands, which then publishes from that tag's commit. +# published moq-ffi has. The FFI coordinator re-triggers this workflow after +# its Kotlin publisher succeeds, then publishes from that tag's commit. on: push: @@ -44,7 +44,7 @@ on: # dispatch on `release` publishes. workflow_dispatch: workflow_run: - workflows: ["Release Kotlin FFI"] + workflows: ["Release FFI", "Release Kotlin FFI"] types: - completed @@ -52,17 +52,44 @@ permissions: contents: read jobs: - check-version: - name: Check version + ffi-release: + name: Check FFI publisher runs-on: ubuntu-latest - # workflow_run has the base repository's privileges, so accept only a - # successful upstream tag-push run from the expected release workflow. + # workflow_run is privileged. Preserve the owner, same-repo, push and tag + # gates before reading the trusted upstream checkout or allowing wrapper + # publication. if: >- github.event_name != 'workflow_run' || (github.repository_owner == 'moq-dev' && - github.event.workflow_run.conclusion == 'success' && + (github.event.workflow_run.name == 'Release FFI' || github.event.workflow_run.conclusion == 'success') && github.event.workflow_run.event == 'push' && + github.event.workflow_run.head_repository.full_name == github.repository && startsWith(github.event.workflow_run.head_branch, 'moq-ffi-v')) + permissions: + contents: read + actions: read + outputs: + ready: ${{ steps.publisher.outputs.ready }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.sha }} + persist-credentials: false + - uses: ./.github/actions/just + - name: Check independent publisher + id: publisher + env: + GH_TOKEN: ${{ github.token }} + UPSTREAM_NAME: ${{ github.event.workflow_run.name }} + UPSTREAM_RUN_ID: ${{ github.event.workflow_run.id }} + UPSTREAM_RUN_STARTED_AT: ${{ github.event.workflow_run.run_started_at }} + run: just gh ffi completed kotlin + + check-version: + name: Check version + needs: [ffi-release] + if: needs.ffi-release.outputs.ready == 'true' + runs-on: ubuntu-latest outputs: version: ${{ steps.version.outputs.version }} should_publish: ${{ steps.decide.outputs.should_publish }} @@ -99,7 +126,7 @@ jobs: fi echo "version=$VERSION" >> "$GITHUB_OUTPUT" - # workflow_run is chained after Release Kotlin FFI, so its tag is + # workflow_run verifies the Kotlin FFI publisher, so its tag is # published. PRs run it too, so the dry-run shows whether merging would # publish. - name: Check for unreleased FFI changes @@ -142,7 +169,8 @@ jobs: publish: name: Publish wrapper - needs: [check-version] + # Directly on ffi-release: its gate is what makes the head_sha checkout trusted. + needs: [ffi-release, check-version] # PRs always run (dry-run). Pushes publish only when the version is new. if: >- github.event_name == 'pull_request' || diff --git a/.github/workflows/release-shared-ffi.yml b/.github/workflows/release-shared-ffi.yml new file mode 100644 index 00000000000..8456bd9d380 --- /dev/null +++ b/.github/workflows/release-shared-ffi.yml @@ -0,0 +1,67 @@ +name: Release FFI + +# Default-feature libraries are identical across Go, Kotlin and Swift: build +# each target once, then preserve each package's own artifacts and publishers. +on: + push: + tags: + - "moq-ffi-v*" + workflow_call: + +permissions: + contents: read + +concurrency: + group: release-shared-ffi-${{ github.ref_type == 'tag' && 'publish' || github.ref }} + cancel-in-progress: ${{ github.ref_type != 'tag' }} + +jobs: + native: + name: Shared moq-ffi + uses: ./.github/workflows/release-ffi.yml + with: + language: default + binding-languages: '["go", "kotlin", "swift"]' + targets: >- + [ + "x86_64-unknown-linux-gnu", + "aarch64-unknown-linux-gnu", + "aarch64-apple-darwin", + "x86_64-pc-windows-msvc", + "aarch64-linux-android", + "armv7-linux-androideabi", + "x86_64-linux-android", + "aarch64-apple-ios", + "aarch64-apple-ios-sim" + ] + + # A failed target must affect only packages that consume it. Each packager + # selects its own targets and fails if one of its libraries is missing. + go: + name: Release Go FFI + needs: [native] + if: ${{ always() && !cancelled() }} + uses: ./.github/workflows/release-go-ffi.yml + with: + reuse-native: true + secrets: inherit + + kotlin: + name: Release Kotlin FFI + needs: [native] + if: ${{ always() && !cancelled() }} + uses: ./.github/workflows/release-kt-ffi.yml + with: + reuse-native: true + secrets: inherit + + swift: + name: Release Swift FFI + needs: [native] + if: ${{ always() && !cancelled() }} + uses: ./.github/workflows/release-swift-ffi.yml + with: + reuse-native: true + permissions: + contents: write + secrets: inherit diff --git a/.github/workflows/release-swift-ffi.yml b/.github/workflows/release-swift-ffi.yml index afa2fbf175e..28a06528b7d 100644 --- a/.github/workflows/release-swift-ffi.yml +++ b/.github/workflows/release-swift-ffi.yml @@ -7,12 +7,13 @@ name: Release Swift FFI # (release-swift-lib.yml) and versions independently. on: - push: - tags: - - "moq-ffi-v*" - # Dry-run: build everything, publish nothing. Nightly calls every release - # workflow this way so a broken build surfaces before the tag is cut. + # The tag coordinator supplies shared default-feature artifacts. Manual + # calls and Go PR dry-runs still build their own target subset. workflow_call: + inputs: + reuse-native: + type: boolean + default: false workflow_dispatch: permissions: @@ -28,6 +29,7 @@ jobs: uses: ./.github/workflows/release-ffi.yml with: language: swift + reuse-native: ${{ inputs.reuse-native || false }} targets: '["aarch64-apple-ios", "aarch64-apple-ios-sim", "aarch64-apple-darwin"]' package: @@ -42,20 +44,24 @@ jobs: - uses: ./.github/actions/just + - name: Select native artifacts + id: native + env: + LIB_PREFIX: ${{ inputs.reuse-native && 'default' || 'swift' }} + BUILD_TARGETS: ${{ needs.ffi.outputs.targets }} + run: just gh ffi pattern + - name: Download per-target libs uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: path: libs-raw - pattern: swift-lib-* + pattern: ${{ steps.native.outputs.pattern }} - name: Flatten lib layout - run: | - mkdir -p libs - for dir in libs-raw/swift-lib-*; do - target="${dir#libs-raw/swift-lib-}" - mv "$dir" "libs/$target" - done - ls -la libs + env: + LIB_PREFIX: ${{ inputs.reuse-native && 'default' || 'swift' }} + BUILD_TARGETS: ${{ needs.ffi.outputs.targets }} + run: just gh ffi stage - name: Download bindings uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 diff --git a/.github/workflows/release-swift-lib.yml b/.github/workflows/release-swift-lib.yml index 7283b8dfc13..87db931a13a 100644 --- a/.github/workflows/release-swift-lib.yml +++ b/.github/workflows/release-swift-lib.yml @@ -9,8 +9,8 @@ name: Release Swift Lib # it isn't already tagged on the mirror. That tag check is the release gate (the # same model release-plz uses for crates), so there's no separate tag to # remember. When wrapper and FFI changes land together, publication waits for -# Release Swift FFI and resumes from its successful workflow run so the required -# generated API already exists on the mirror. +# the Swift FFI publisher in Release FFI and resumes once it succeeds, so the +# required generated API already exists on the mirror. on: push: @@ -38,7 +38,7 @@ on: - ".github/workflows/release-swift-lib.yml" - "sh/gh/release.sh" workflow_run: - workflows: ["Release Swift FFI"] + workflows: ["Release FFI", "Release Swift FFI"] types: - completed @@ -52,17 +52,44 @@ concurrency: cancel-in-progress: false jobs: - build: - name: Build wrapper package - runs-on: macos-latest - # workflow_run has the base repository's privileges, so accept only a - # successful upstream tag-push run from the expected release workflow. + ffi-release: + name: Check FFI publisher + runs-on: ubuntu-latest + # workflow_run is privileged. Preserve the owner, same-repo, push and tag + # gates before reading the trusted upstream checkout or allowing wrapper + # publication. if: >- github.repository_owner == 'moq-dev' && (github.event_name != 'workflow_run' || - (github.event.workflow_run.conclusion == 'success' && + ((github.event.workflow_run.name == 'Release FFI' || github.event.workflow_run.conclusion == 'success') && github.event.workflow_run.event == 'push' && + github.event.workflow_run.head_repository.full_name == github.repository && startsWith(github.event.workflow_run.head_branch, 'moq-ffi-v'))) + permissions: + contents: read + actions: read + outputs: + ready: ${{ steps.publisher.outputs.ready }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.sha }} + persist-credentials: false + - uses: ./.github/actions/just + - name: Check independent publisher + id: publisher + env: + GH_TOKEN: ${{ github.token }} + UPSTREAM_NAME: ${{ github.event.workflow_run.name }} + UPSTREAM_RUN_ID: ${{ github.event.workflow_run.id }} + UPSTREAM_RUN_STARTED_AT: ${{ github.event.workflow_run.run_started_at }} + run: just gh ffi completed swift + + build: + name: Build wrapper package + needs: [ffi-release] + if: needs.ffi-release.outputs.ready == 'true' + runs-on: macos-latest outputs: version: ${{ steps.version.outputs.version }} ffi_version: ${{ steps.version.outputs.ffi_version }} @@ -95,7 +122,7 @@ jobs: echo "ffi_version=$FFI_VERSION" >> "$GITHUB_OUTPUT" echo "wrapper $VERSION, moq-ffi pin $FFI_VERSION" - # workflow_run is chained after Release Swift FFI, so its tag is published. + # workflow_run verifies the Swift FFI publisher, so its tag is published. # PRs run it too, so the dry-run shows whether merging would publish. - name: Check for unreleased FFI changes id: ffi-diff @@ -114,7 +141,7 @@ jobs: if: steps.ffi-diff.outputs.unreleased != 'true' && steps.ffi-gate.outputs.exists != 'true' && github.event_name != 'pull_request' env: FFI_VERSION: ${{ steps.version.outputs.ffi_version }} - run: echo "::warning::moq-swift-ffi ${FFI_VERSION} is not published; deferring to the Release Swift FFI chain." + run: echo "::warning::moq-swift-ffi ${FFI_VERSION} is not published; deferring to the FFI release chain." # The release gate: skip publishing a version already tagged on the mirror. # Runs on PRs too, as an informational dry-run. @@ -142,7 +169,8 @@ jobs: # failed) when the pinned moq-ffi version isn't mirrored yet, so the first # bootstrap doesn't deadlock on a not-yet-published FFI package. name: Verify staged package resolves - needs: [build] + # Directly on ffi-release: its gate is what makes the head_sha checkout trusted. + needs: [ffi-release, build] runs-on: macos-latest steps: @@ -189,7 +217,8 @@ jobs: publish: name: Publish to Swift Package mirror - needs: [build, verify] + # Directly on ffi-release: its gate is what makes the head_sha checkout trusted. + needs: [ffi-release, build, verify] runs-on: ubuntu-latest if: ${{ needs.build.outputs.publish == 'true' }} diff --git a/flake.nix b/flake.nix index e0ac5182e2e..0ad9e149433 100644 --- a/flake.nix +++ b/flake.nix @@ -275,7 +275,7 @@ # understands metadata emitted by that uniffi, so it moves with the `uniffi` # dependency in rs/moq-ffi/Cargo.toml. Five other places name the same # generator version and must be bumped together: the repo and revision - # in release-go-ffi.yml, and the `cargo install` line in + # in release-ffi.yml, and the `cargo install` line in # rs/moq-ffi/build.sh, go/ffi/README.md, sh/go/check.sh, and # sh/go/stage.sh. # diff --git a/go/ffi/README.md b/go/ffi/README.md index 6edaf6937e2..e2fab932a8b 100644 --- a/go/ffi/README.md +++ b/go/ffi/README.md @@ -53,4 +53,4 @@ Compiled binaries never live in the source tree. `sh/go/check.sh` stages into th ## Release -The `release-go-ffi.yml` workflow fires on every `moq-ffi-v*` tag, builds per-target static libraries, runs `uniffi-bindgen-go`, calls `just go package-ffi` to assemble the module, and `just go publish-ffi` to push the result to `moq-dev/moq-go-ffi` with a bare-semver tag (e.g. `v0.2.18`) lockstep with the crate. Go's module proxy picks up the new tag automatically. +The `release-shared-ffi.yml` coordinator fires on every `moq-ffi-v*` tag and builds the default-feature native libraries once for Go, Kotlin, and Swift. It generates bindings from the built Linux library and calls `release-go-ffi.yml`, which calls `just go package-ffi` to assemble the module, and `just go publish-ffi` to push the result to `moq-dev/moq-go-ffi` with a bare-semver tag (e.g. `v0.2.18`) lockstep with the crate. Go's module proxy picks up the new tag automatically. diff --git a/kt/README.md b/kt/README.md index 7e0958fd691..fe1e76d2f94 100644 --- a/kt/README.md +++ b/kt/README.md @@ -58,7 +58,7 @@ The `dev.moq` package is intentionally thin: Kotlin has extension functions, so ## Versioning - `moqffi.version` (gradle.properties): the bindings version. CI overrides it from the `moq-ffi-v*` tag; only used for local dev otherwise. -- `moq.version` (gradle.properties): the wrapper version, the source of truth. **Bump this by hand** to ship a new wrapper. `release-kt-lib.yml` reads it, checks whether `dev.moq:moq:` is already on Maven Central, and publishes only if it isn't. If `rs/moq-ffi` changed since its last `moq-ffi-v*` tag, it waits and publishes after the next Release Kotlin FFI run. Must stay `>= 0.3.0` (the line continues from the pre-split `dev.moq:moq` releases). +- `moq.version` (gradle.properties): the wrapper version, the source of truth. **Bump this by hand** to ship a new wrapper. `release-kt-lib.yml` reads it, checks whether `dev.moq:moq:` is already on Maven Central, and publishes only if it isn't. If `rs/moq-ffi` changed since its last `moq-ffi-v*` tag, it waits and publishes after its Kotlin publisher succeeds in the next Release FFI run. Must stay `>= 0.3.0` (the line continues from the pre-split `dev.moq:moq` releases). ## Local development diff --git a/rs/moq-ffi/build.sh b/rs/moq-ffi/build.sh index ef8ef928f89..2fddb9b7927 100755 --- a/rs/moq-ffi/build.sh +++ b/rs/moq-ffi/build.sh @@ -2,7 +2,7 @@ set -euo pipefail # Build and package moq-ffi native libraries for release. -# Usage: ./build.sh [--target TARGET] [--version VERSION] [--output DIR] [--bindings-only] [--no-default-features] +# Usage: ./build.sh [--target TARGET] [--version VERSION] [--output DIR] [--bindings-only [--library PATH]] [--no-default-features] # # Examples: # ./build.sh # Build for host, detect version from Cargo.toml @@ -37,6 +37,7 @@ TARGET="" VERSION="" OUTPUT_DIR="dist" BINDINGS_ONLY=false +LIBRARY="" ARCHIVE=false NO_DEFAULT_FEATURES=false @@ -59,6 +60,10 @@ while [[ $# -gt 0 ]]; do BINDINGS_ONLY=true shift ;; + --library) + LIBRARY="$2" + shift 2 + ;; --archive) # Default leaves the staging dir alone for upload-artifact; # --archive recreates the legacy tar.gz/zip behavior. @@ -70,7 +75,7 @@ while [[ $# -gt 0 ]]; do shift ;; -h | --help) - echo "Usage: $0 [--target TARGET] [--version VERSION] [--output DIR] [--bindings-only] [--archive] [--no-default-features]" + echo "Usage: $0 [--target TARGET] [--version VERSION] [--output DIR] [--bindings-only [--library PATH]] [--archive] [--no-default-features]" exit 0 ;; *) @@ -80,6 +85,13 @@ while [[ $# -gt 0 ]]; do esac done +if [[ -n "$LIBRARY" ]]; then + if [[ "$BINDINGS_ONLY" != true || ! -f "$LIBRARY" ]]; then + echo "Error: --library requires --bindings-only and an existing library" >&2 + exit 1 + fi +fi + # Get version from Cargo.toml if not specified if [[ -z "$VERSION" ]]; then VERSION=$(grep '^version' "$SCRIPT_DIR/Cargo.toml" | head -1 | sed 's/.*"\(.*\)".*/\1/') @@ -205,8 +217,12 @@ mkdir -p "$OUTPUT_DIR" # --- Bindings-only mode --- if [[ "$BINDINGS_ONLY" == true ]]; then - build_target "$HOST_TARGET" - cdylib=$(find_cdylib "$HOST_TARGET") + if [[ -n "$LIBRARY" ]]; then + cdylib="$LIBRARY" + else + build_target "$HOST_TARGET" + cdylib=$(find_cdylib "$HOST_TARGET") + fi if [[ ! -f "$cdylib" ]]; then echo "Error: cdylib not found at $cdylib" >&2 exit 1 diff --git a/sh/gh/ffi.py b/sh/gh/ffi.py new file mode 100644 index 00000000000..38180b019bb --- /dev/null +++ b/sh/gh/ffi.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python3 +"""Select shared native libraries and check an independent FFI publisher.""" + +from datetime import datetime +import json +import os +from pathlib import Path +import re +import subprocess +import sys + +LANGUAGES = { + "go": ("Release Go FFI", "Publish to Go module mirror"), + "kotlin": ("Release Kotlin FFI", "Package and publish"), + "swift": ("Release Swift FFI", "Publish to Swift Package mirror"), +} + + +def output(key, value): + with open(os.environ["GITHUB_OUTPUT"], "a") as file: + file.write(f"{key}={value}\n") + + +def selection(): + prefix = os.environ["LIB_PREFIX"] + if prefix not in ("default", *LANGUAGES): + raise ValueError(f"unsupported native artifact prefix: {prefix}") + targets = json.loads(os.environ["BUILD_TARGETS"]) + if not isinstance(targets, list) or not targets: + raise ValueError("BUILD_TARGETS must be a nonempty JSON array") + if any(not isinstance(t, str) or not re.fullmatch(r"[a-z0-9_]+(?:-[a-z0-9_]+)+", t) for t in targets): + raise ValueError("invalid Rust target") + if len(set(targets)) != len(targets): + raise ValueError("duplicate Rust targets") + return prefix, targets + + +def pattern(): + prefix, targets = selection() + suffix = targets[0] if len(targets) == 1 else "{" + ",".join(targets) + "}" + output("pattern", f"{prefix}-lib-{suffix}") + + +def stage(): + prefix, targets = selection() + raw = Path("libs-raw") + sources = [raw / f"{prefix}-lib-{target}" for target in targets] + # download-artifact flattens a single matching artifact even without merge. + if len(targets) == 1 and raw.is_dir() and not sources[0].exists() and all(p.is_file() for p in raw.iterdir()): + sources = [raw] + for source in sources: + if not source.is_dir() or not any(source.iterdir()): + raise ValueError(f"missing native library artifact: {source}") + Path("libs").mkdir(exist_ok=True) + for source, target in zip(sources, targets): + source.rename(Path("libs") / target) + + +def published(jobs, language, started): + group, publisher = LANGUAGES[language] + prefix = group + " / " + # A partial rerun copies successful jobs into the new attempt, so judge the + # latest record of each own job. + latest = {} + for job in jobs: + name = job["name"] + if not name.startswith(prefix): + continue + previous = latest.get(name) + if previous and previous["run_attempt"] == job["run_attempt"]: + raise ValueError(f"duplicate job in one attempt: {name}") + if previous is None or job["run_attempt"] > previous["run_attempt"]: + latest[name] = job + own = list(latest.values()) + publishers = [job for job in own if job["name"] == prefix + publisher] + if len(publishers) != 1: + raise ValueError(f"expected one {group} publisher, found {len(publishers)}") + publisher = publishers[0] + if publisher["conclusion"] != "success": + return False + # A copied job keeps its original start time. Its wrapper already ran when + # that earlier attempt completed; re-releasing it after a newer tag published + # would roll the wrapper back to old sources. + if datetime.fromisoformat(publisher["started_at"]) < started: + return False + return all(job["status"] == "completed" and job["conclusion"] in ("success", "skipped") for job in own) + + +def completed(language): + group, _ = LANGUAGES[language] + # Existing direct triggers and the old per-language workflow retain their + # own gates. Only the shared coordinator needs per-language attribution. + if os.environ["GITHUB_EVENT_NAME"] != "workflow_run": + output("ready", "true") + return + name = os.environ["UPSTREAM_NAME"] + if name == group: + output("ready", "true") + return + if name != "Release FFI": + raise ValueError(f"unexpected upstream workflow: {name}") + run = os.environ["UPSTREAM_RUN_ID"] + if not re.fullmatch(r"[1-9][0-9]*", run): + raise ValueError("invalid upstream run ID") + started = datetime.fromisoformat(os.environ["UPSTREAM_RUN_STARTED_AT"]) + repo = os.environ["GITHUB_REPOSITORY"] + # A single query after workflow_run completion, never a publication poll. + result = subprocess.run( + ["gh", "api", f"repos/{repo}/actions/runs/{run}/jobs?filter=all", "--paginate", "--slurp"], + check=True, stdout=subprocess.PIPE, text=True, + ) + pages = json.loads(result.stdout) + jobs = [job for page in pages for job in page["jobs"]] + output("ready", str(published(jobs, language, started)).lower()) + + +def main(): + match sys.argv[1:]: + case ["pattern"]: + pattern() + case ["stage"]: + stage() + case ["completed", language] if language in LANGUAGES: + completed(language) + case _: + raise ValueError("usage: ffi.py pattern|stage|completed go|kotlin|swift") + + +if __name__ == "__main__": + main() diff --git a/sh/gh/ffi.test.py b/sh/gh/ffi.test.py new file mode 100644 index 00000000000..257e876f422 --- /dev/null +++ b/sh/gh/ffi.test.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +"""Exercise artifact selection, independent release gates, and build-free bindings.""" + +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parents[2] +HELPER = ROOT / "sh/gh/ffi.py" +ATTEMPT_1, ATTEMPT_2 = "2026-01-01T00:00:00Z", "2026-01-02T00:00:00Z" + + +class Ffi(unittest.TestCase): + def setUp(self): + self.directory = tempfile.TemporaryDirectory() + self.addCleanup(self.directory.cleanup) + self.root = Path(self.directory.name) + self.bin = self.root / "bin" + self.bin.mkdir() + self.env = dict(os.environ, BASH_ENV="/dev/null", PATH=f"{self.bin}:{os.environ['PATH']}", + GITHUB_OUTPUT=str(self.root / "output"), LIB_PREFIX="default", + BUILD_TARGETS='["x86_64-unknown-linux-gnu", "aarch64-apple-darwin"]', + GITHUB_EVENT_NAME="workflow_run", UPSTREAM_NAME="Release FFI", UPSTREAM_RUN_ID="42", + UPSTREAM_RUN_STARTED_AT=ATTEMPT_2, + GITHUB_REPOSITORY="moq-dev/moq", TEST_ROOT=str(self.root)) + + def tool(self, name, body): + path = self.bin / name + path.write_text("#!/usr/bin/env python3\n" + body) + path.chmod(0o755) + + def run_helper(self, *args, success=True): + result = subprocess.run([sys.executable, str(HELPER), *args], cwd=self.root, + env=self.env, capture_output=True, text=True) + self.assertEqual(result.returncode == 0, success, result.stderr) + return (self.root / "output").read_text() if (self.root / "output").exists() else "" + + def test_select_and_stage_only_requested_targets(self): + self.assertEqual(self.run_helper("pattern"), + "pattern=default-lib-{x86_64-unknown-linux-gnu,aarch64-apple-darwin}\n") + for target in ["x86_64-unknown-linux-gnu", "aarch64-apple-darwin", "aarch64-apple-ios"]: + directory = self.root / f"libs-raw/default-lib-{target}" + directory.mkdir(parents=True) + (directory / "library").write_bytes(target.encode()) + self.run_helper("stage") + self.assertEqual(sorted(p.name for p in (self.root / "libs").iterdir()), + ["aarch64-apple-darwin", "x86_64-unknown-linux-gnu"]) + self.assertEqual((self.root / "libs/aarch64-apple-darwin/library").read_bytes(), b"aarch64-apple-darwin") + self.assertTrue((self.root / "libs-raw/default-lib-aarch64-apple-ios/library").exists()) + + def test_missing_target_fails_before_staging(self): + directory = self.root / "libs-raw/default-lib-x86_64-unknown-linux-gnu" + directory.mkdir(parents=True) + (directory / "library").touch() + self.run_helper("stage", success=False) + self.assertFalse((self.root / "libs").exists()) + + def test_invalid_selection_is_refused(self): + for targets in ["[]", '{}', '["../outside"]', '["x86-linux","x86-linux"]', '[false]']: + with self.subTest(targets=targets): + self.env["BUILD_TARGETS"] = targets + self.run_helper("pattern", success=False) + + def test_single_target_pattern_has_no_brace_literal(self): + self.env["BUILD_TARGETS"] = '["x86_64-unknown-linux-gnu"]' + self.assertEqual(self.run_helper("pattern"), "pattern=default-lib-x86_64-unknown-linux-gnu\n") + + def test_single_artifact_download_is_staged_from_flat_layout(self): + self.env["BUILD_TARGETS"] = '["x86_64-unknown-linux-gnu"]' + raw = self.root / "libs-raw" + raw.mkdir() + (raw / "libmoq_ffi.a").write_bytes(b"native") + self.run_helper("stage") + self.assertEqual((self.root / "libs/x86_64-unknown-linux-gnu/libmoq_ffi.a").read_bytes(), b"native") + + def jobs(self, language, conclusion="success"): + group, publisher = { + "go": ("Release Go FFI", "Publish to Go module mirror"), + "kotlin": ("Release Kotlin FFI", "Package and publish"), + "swift": ("Release Swift FFI", "Publish to Swift Package mirror"), + }[language] + return [{"name": f"{group} / {publisher}", "status": "completed", "conclusion": conclusion}] + + def gh(self, jobs): + jobs = [dict(dict(run_attempt=2, started_at=ATTEMPT_2), **job) for job in jobs] + (self.root / "jobs").write_text(json.dumps([{"jobs": jobs[:1]}, {"jobs": jobs[1:]}])) + self.tool("gh", '''import os, pathlib, sys +assert sys.argv[1:] == ["api", "repos/moq-dev/moq/actions/runs/42/jobs?filter=all", "--paginate", "--slurp"] +root = pathlib.Path(os.environ["TEST_ROOT"]) +(root / "queried").touch() +if os.environ.get("GH_FAIL"): sys.exit(1) +print((root / "jobs").read_text()) +''') + + def test_unrelated_failure_does_not_suppress_own_release(self): + for language in ["go", "kotlin", "swift"]: + with self.subTest(language=language): + (self.root / "output").unlink(missing_ok=True) + jobs = self.jobs(language) + [{"name": "Another language / Build", "status": "completed", "conclusion": "failure"}, + {"name": self.jobs(language)[0]["name"].split(" / ")[0] + " / Publish dry-run", + "status": "completed", "conclusion": "skipped"}] + self.gh(jobs) + self.assertEqual(self.run_helper("completed", language), "ready=true\n") + + def test_own_failure_or_incomplete_job_prevents_release(self): + for jobs in [self.jobs("go", "failure"), self.jobs("go") + [ + {"name": "Release Go FFI / Package", "status": "in_progress", "conclusion": None}], + self.jobs("go") + [{"name": "Release Go FFI / Package", "status": "completed", "conclusion": "failure"}]]: + with self.subTest(jobs=jobs): + (self.root / "output").unlink(missing_ok=True) + self.gh(jobs) + self.assertEqual(self.run_helper("completed", "go"), "ready=false\n") + + def test_partial_reruns_use_the_latest_execution_of_each_own_job(self): + for old, new, ready in [("failure", "success", "true"), ("success", "failure", "false")]: + (self.root / "output").unlink(missing_ok=True) + self.gh([dict(self.jobs("go", old)[0], run_attempt=1, started_at=ATTEMPT_1), + dict(self.jobs("go", new)[0], run_attempt=2)]) + self.assertEqual(self.run_helper("completed", "go"), f"ready={ready}\n") + + def test_publisher_copied_from_an_earlier_attempt_does_not_release_again(self): + # Attempt 1 published Go and failed Kotlin, so Go's wrapper already ran. + # A newer tag may have published since; rerunning only Kotlin must not + # re-cut Go's wrapper from this older tag. GitHub copies the successful + # job into attempt 2 with its original start time. + go, kotlin = self.jobs("go")[0], self.jobs("kotlin")[0] + self.gh([dict(go, run_attempt=1, started_at=ATTEMPT_1), dict(go, run_attempt=2, started_at=ATTEMPT_1), + dict(kotlin, run_attempt=1, started_at=ATTEMPT_1, conclusion="failure"), kotlin]) + self.assertEqual(self.run_helper("completed", "go"), "ready=false\n") + (self.root / "output").unlink() + self.assertEqual(self.run_helper("completed", "kotlin"), "ready=true\n") + + def test_missing_publisher_and_api_error_fail_loudly(self): + self.gh(self.jobs("swift")) + self.run_helper("completed", "go", success=False) + self.env["GH_FAIL"] = "1" + self.run_helper("completed", "swift", success=False) + + def test_direct_and_legacy_triggers_need_no_api(self): + for event, name in [("push", ""), ("workflow_run", "Release Go FFI")]: + (self.root / "output").unlink(missing_ok=True) + self.env.update(GITHUB_EVENT_NAME=event, UPSTREAM_NAME=name) + self.assertEqual(self.run_helper("completed", "go"), "ready=true\n") + self.assertFalse((self.root / "queried").exists()) + + def test_bindings_from_existing_library_never_build_moq_ffi(self): + library = self.root / "libmoq_ffi.so" + library.write_bytes(b"metadata") + self.tool("cargo", '''import json, os, pathlib, sys +root = pathlib.Path(os.environ["TEST_ROOT"]) +with (root / "calls").open("a") as log: log.write(json.dumps(sys.argv[1:]) + "\\n") +if sys.argv[1] == "metadata": print(json.dumps({"target_directory": str(root / "target")})) +elif sys.argv[1] == "run": + assert sys.argv[sys.argv.index("--library") + 1] == str(root / "libmoq_ffi.so") + pathlib.Path(sys.argv[sys.argv.index("--out-dir") + 1]).mkdir(parents=True) +else: raise AssertionError("unexpected native build") +''') + self.tool("rustc", 'print("host: x86_64-unknown-linux-gnu")\n') + for tool in ["uniffi-bindgen-go", "uniffi_bindgen_dart", "uniffi-bindgen-cpp"]: + self.tool(tool, '''import os, pathlib, sys +assert sys.argv[sys.argv.index("--library") + 1] == str(pathlib.Path(os.environ["TEST_ROOT"]) / "libmoq_ffi.so") +pathlib.Path(sys.argv[sys.argv.index("--out-dir") + 1]).mkdir(parents=True) +''') + args = ["bash", str(ROOT / "rs/moq-ffi/build.sh"), "--bindings-only", "--library", str(library), + "--version", "1.2.3", "--output", str(self.root / "dist")] + result = subprocess.run(args, env=self.env, cwd=self.root, capture_output=True, text=True) + self.assertEqual(result.returncode, 0, result.stderr) + calls = [json.loads(line) for line in (self.root / "calls").read_text().splitlines()] + self.assertEqual([call[0] for call in calls], ["metadata", "run", "run", "run"]) + self.assertEqual(sorted(p.name for p in (self.root / "dist/bindings").iterdir()), + ["dart", "go", "kotlin", "python", "swift"]) + for suffix in [["--library", str(library)], ["--bindings-only", "--library", str(self.root / "missing")]]: + result = subprocess.run(["bash", str(ROOT / "rs/moq-ffi/build.sh"), *suffix], env=self.env, + cwd=self.root, capture_output=True, text=True) + self.assertNotEqual(result.returncode, 0) + self.assertIn("requires --bindings-only and an existing library", result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/sh/gh/ffi.test.ts b/sh/gh/ffi.test.ts new file mode 100644 index 00000000000..5a913eba3ef --- /dev/null +++ b/sh/gh/ffi.test.ts @@ -0,0 +1,112 @@ +// biome-ignore-all lint/suspicious/noTemplateCurlyInString: GitHub Actions expressions are literal strings. +import { expect, test } from "bun:test"; +import { YAML } from "bun"; + +type Job = { + name?: string; + if?: string; + uses?: string; + needs?: string[]; + permissions?: Record; + with?: Record; + strategy?: { "fail-fast": boolean }; + steps?: { name?: string; if?: string; env?: Record; with?: Record; run?: string }[]; +}; +type Workflow = { + on: { push?: { tags?: string[] }; workflow_call?: unknown; workflow_run?: { workflows: string[] } }; + permissions: Record; + jobs: Record; +}; +async function read(name: string): Promise { + return YAML.parse(await Bun.file(`.github/workflows/${name}.yml`).text()) as Workflow; +} + +test("shared default builds cover each consumer exactly once", async () => { + const shared = await read("release-shared-ffi"); + const targets = JSON.parse(shared.jobs.native.with?.targets as string) as string[]; + expect(targets.length).toBe(9); + expect(new Set(targets).size).toBe(targets.length); + const all: string[] = []; + for (const [job, file] of [ + ["go", "go"], + ["kotlin", "kt"], + ["swift", "swift"], + ]) { + const consumer = await read(`release-${file}-ffi`); + all.push(...JSON.parse(consumer.jobs.ffi.with?.targets as string)); + expect(consumer.on.push).toBeUndefined(); + expect(consumer.on.workflow_call).toBeDefined(); + expect(shared.jobs[job].with?.["reuse-native"]).toBe(true); + expect(shared.jobs[job].if).toContain("always()"); + expect(shared.jobs[job].if).toContain("!cancelled()"); + const select = consumer.jobs.package.steps?.find((step) => step.name === "Select native artifacts"); + expect(select?.env?.BUILD_TARGETS).toBe("${{ needs.ffi.outputs.targets }}"); + expect(select?.env?.LIB_PREFIX).toContain("'default'"); + } + expect([...new Set(all)].sort()).toEqual(targets.toSorted()); + expect(all.length - targets.length).toBe(5); + expect(shared.on.push?.tags).toEqual(["moq-ffi-v*"]); + const dart = await read("release-dart-ffi"); + expect(dart.jobs.ffi.with?.["default-features"]).toBe(false); + expect(dart.on.push?.tags).toEqual(["moq-ffi-v*"]); +}); + +test("bindings reuse Linux metadata even when an unrelated target fails", async () => { + const producer = await read("release-ffi"); + expect(producer.jobs.build.strategy?.["fail-fast"]).toBe(false); + expect(producer.jobs.bindings.strategy?.["fail-fast"]).toBe(false); + expect(producer.jobs.bindings.needs).toContain("build"); + expect(producer.jobs.bindings.if).toContain("always()"); + expect(producer.jobs.bindings.if).toContain("!inputs.reuse-native"); + expect(producer.jobs.build.if).toContain("!inputs.reuse-native"); + const steps = producer.jobs.bindings.steps ?? []; + expect(steps.find((s) => s.name === "Download binding metadata")?.with?.name).toBe( + "${{ inputs.language }}-lib-x86_64-unknown-linux-gnu", + ); + expect(steps.find((s) => s.name === "Generate bindings")?.run).toContain("--library binding-library/libmoq_ffi.so"); + expect(steps.find((s) => s.name === "Upload bindings")?.with?.name).toBe("${{ matrix.language }}-bindings"); +}); + +test("wrapper gates preserve trusted tag inputs and inspect only their publisher", async () => { + for (const [file, language, first] of [ + ["go", "go", "build"], + ["kt-lib", "kotlin", "check-version"], + ["swift-lib", "swift", "build"], + ]) { + const wrapper = await read(`release-${file}`); + expect(wrapper.on.workflow_run?.workflows).toContain("Release FFI"); + expect(wrapper.permissions).toEqual({ contents: "read" }); + const gate = wrapper.jobs["ffi-release"]; + expect(gate.permissions).toEqual({ contents: "read", actions: "read" }); + expect(gate.if).toContain("github.repository_owner == 'moq-dev'"); + expect(gate.if).toContain("github.event.workflow_run.event == 'push'"); + expect(gate.if).toContain("startsWith(github.event.workflow_run.head_branch, 'moq-ffi-v')"); + expect(gate.if).toContain("github.event.workflow_run.name == 'Release FFI'"); + expect(gate.steps?.[0]?.with?.ref).toBe("${{ github.sha }}"); + expect(gate.steps?.at(-1)?.run).toBe(`just gh ffi completed ${language}`); + expect(gate.steps?.at(-1)?.env?.UPSTREAM_RUN_STARTED_AT).toBe( + "${{ github.event.workflow_run.run_started_at }}", + ); + expect(wrapper.jobs[first].needs).toEqual(["ffi-release"]); + expect(wrapper.jobs[first].if).toBe("needs.ffi-release.outputs.ready == 'true'"); + } +}); + +test("Nightly and alert chains retain packaged outputs with one shared producer", async () => { + const nightly = await read("nightly"); + expect(nightly.jobs["release-shared-ffi"].uses).toBe("./.github/workflows/release-shared-ffi.yml"); + for (const old of ["release-go-ffi", "release-kt-ffi", "release-swift-ffi"]) + expect(nightly.jobs[old]).toBeUndefined(); + expect(nightly.jobs["release-py-ffi"]).toBeDefined(); + expect(nightly.jobs["release-dart-ffi"]).toBeDefined(); + // A called workflow's jobs cannot exceed the caller's grant, so a dry-run + // of a gated wrapper must pass on the gate's permissions. + for (const [name, job] of Object.entries(nightly.jobs)) { + const file = job.uses?.match(/^\.\/\.github\/workflows\/(.+)\.yml$/)?.[1]; + const gate = file ? (await read(file)).jobs["ffi-release"] : undefined; + if (!gate) continue; + for (const [scope, level] of Object.entries(gate.permissions ?? {})) + expect([name, scope, job.permissions?.[scope]]).toEqual([name, scope, level]); + } + expect((await read("alert")).on.workflow_run?.workflows).toContain("Release FFI"); +}); diff --git a/swift/README.md b/swift/README.md index 32cf1c96559..45fb7969f2a 100644 --- a/swift/README.md +++ b/swift/README.md @@ -107,8 +107,8 @@ Edit the templates when changing a released manifest; never copy the monolithic Two workflows, mirroring the two packages: -- **`release-swift-ffi.yml`** fires on each `moq-ffi-v*` tag (pushed by release-plz). It builds the per-target libs + bindings, assembles the `MoqFFI` package via `package-ffi.sh`, attaches `MoqFFI.xcframework.zip` to the `moq-ffi-v*` GitHub Release, verifies the staged package resolves (`verify-ffi.sh`), and mirrors it to [moq-dev/moq-swift-ffi](https://github.com/moq-dev/moq-swift-ffi) on a bare-semver tag (`publish-ffi.sh`). -- **`release-swift-lib.yml`** fires on push to `main`/`dev` when `swift/VERSION` (or the wrapper sources) change. It reads `swift/VERSION`, checks whether that tag already exists on the mirror (the release gate, the same model release-plz uses for crates), assembles the wrapper via `package.sh` (substituting the `moq-ffi` pin from `rs/moq-ffi/Cargo.toml`), verifies it resolves against the published `MoqFFI` (`verify.sh`), and publishes to [moq-dev/moq-swift](https://github.com/moq-dev/moq-swift) only when the version is new (`publish.sh`). +- **`release-shared-ffi.yml`** fires on each `moq-ffi-v*` tag (pushed by release-plz) and builds the default-feature libraries once for Go, Kotlin, and Swift. It generates bindings from the built Linux library, then calls **`release-swift-ffi.yml`**, which assembles the `MoqFFI` package via `package-ffi.sh`, attaches `MoqFFI.xcframework.zip` to the `moq-ffi-v*` GitHub Release, verifies the staged package resolves (`verify-ffi.sh`), and mirrors it to [moq-dev/moq-swift-ffi](https://github.com/moq-dev/moq-swift-ffi) on a bare-semver tag (`publish-ffi.sh`). +- **`release-swift-lib.yml`** fires on push to `release` when `swift/VERSION` (or the wrapper sources) change. It reads `swift/VERSION`, checks whether that tag already exists on the mirror (the release gate, the same model release-plz uses for crates), assembles the wrapper via `package.sh` (substituting the `moq-ffi` pin from `rs/moq-ffi/Cargo.toml`), verifies it resolves against the published `MoqFFI` (`verify.sh`), and publishes to [moq-dev/moq-swift](https://github.com/moq-dev/moq-swift) only when the version is new (`publish.sh`). Both `verify` jobs build a throwaway SPM consumer against the staged package before any mirror push, so a manifest SPM cannot resolve never reaches consumers. The `moq-bot` GitHub App mints a fresh installation token per run, scoped to the relevant mirror.