Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 102 additions & 48 deletions .github/actions/rust-cache/action.yml
Original file line number Diff line number Diff line change
@@ -1,65 +1,119 @@
name: Rust cache
description: Install mbx, put its Cargo shim on PATH, and restore the build cache the default branch warms.
description: Install mbx, put its Cargo shim on PATH, and back it with the R2 build cache that `main` and its merge queue write.

inputs:
scope:
description: Key space for the store. A workflow that builds a different set of artifacts needs its own.
default: shared
save-on-dispatch:
description: Let a trusted workflow_dispatch seed the cache (the warming workflow only).
default: "false"
access-key-id:
description: R2 access key id. Empty, as for a fork pull request, builds cold.
default: ""
secret-access-key:
description: R2 secret access key matching `access-key-id`.
default: ""

runs:
using: composite
steps:
# The pinned action saves in a post step guarded by success(), so failed
# builds cannot publish a cache even when dispatch saving is enabled.
- uses: jdx/mr-boxington-action@7234d3dd1a6ca8f6c381eea8e4dfb03f18fcf777 # v1.3.0
# Runs on `main` and in its merge queue write; everything else reads. A
# queued pull request is about to become `main`, so its code is trusted
# like `main`'s, and publishing from the queue keeps the cache current
# with what lands without building it again after the merge.
#
# The write key is the boundary. It lives in the `cache-write`
# Environment, whose branch policy admits exactly the refs `WRITE` names,
# and its secrets shadow the repository's read key under the same names.
# So every job that can run on those refs names the Environment there, and
# a pull request's code never sees the write key, even from this
# repository. Fork pull requests get no secrets at all and build cold.
#
# A writer without a key means the Environment is misconfigured, and
# building on would leave every reader cold with nothing to say why. Half
# a pair is a broken secret on any job, never a fork's empty one.
- name: Cache credentials
shell: bash
env:
KEY_ID: ${{ inputs.access-key-id }}
SECRET: ${{ inputs.secret-access-key }}
WRITE: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/gh-readonly-queue/main/') }}
run: |
if [ -z "$KEY_ID" ] && [ -z "$SECRET" ]; then
if [ "$WRITE" = "true" ]; then
echo "::error::$GITHUB_REF writes the cache but the cache-write Environment supplied no R2 key"
exit 1
fi
echo "::warning::No R2 cache credentials, as for a fork pull request; building cold"
exit 0
fi
if [ -z "$KEY_ID" ] || [ -z "$SECRET" ]; then
echo "::error::only one of the R2 access key id and secret is set; check the MBX_R2_* secrets"
exit 1
fi
{
echo "AWS_ACCESS_KEY_ID=$KEY_ID"
echo "AWS_SECRET_ACCESS_KEY=$SECRET"
echo "MBX_REMOTE_S3_ENDPOINT=https://dd618f5dbd5da77b8296f1613c301f5c.r2.cloudflarestorage.com"
echo "MBX_REMOTE_S3_REGION=auto"
if [ "$WRITE" = "true" ]; then
echo "MBX_REMOTE_MODE=read-write"
# mbx trusts only a push to a protected branch, which leaves out
# the merge queue and Nightly's schedule. Its guess is bypassed on
# purpose: the Environment already decided who holds the write key.
echo "MBX_REMOTE_TRUST_WRITES=true"
else
echo "MBX_REMOTE_MODE=read-only"
fi
} >> "$GITHUB_ENV"

# mbx keys every compilation on the compiler, target, and inputs, so one
# bucket serves every toolchain and workflow without a cache key of ours.
# The namespace splits it per runner platform anyway, so a platform's
# usage and misses can be read off its own prefix. The remote mode comes
# from MBX_REMOTE_MODE above.
- uses: jdx/mr-boxington-action@47ce49b32025d93a7a2f4570815afaa2b7092240 # v1.7.2
with:
version: 1.10.0
# `github-cache-mode` is left at its `target` default. That transports
# Cargo's pruned target tree, the same payload Swatinem carried, with
# mbx's per-compilation store on top. The `objects` closure is portable
# across differing target directories, which nothing here needs, and
# measured slower. It transports `./target` relative to the working
# directory and ignores CARGO_TARGET_DIR, so a workflow that sets one
# has to point it there.
#
# The key is spelled out because the generated one cannot name this
# repository's compiler. It keys on `rustc -vV` from PATH, which here is
# whatever the runner image ships: the compiler these jobs build with
# exists only inside `nix develop`. The `toolchain` input is no help
# either, since it names a rustup toolchain and probes
# `rustc +<name> -vV`, and nothing installs that channel through rustup.
#
# flake.lock pins that compiler, so hashing the flake identifies it more
# precisely than `rustc -vV` would, and rolls the cache when the dev
# shell changes. Build scripts link against Nix store paths, so a
# restored executable whose dynamic linker moved is worse than a cold
# cache. rust-toolchain.toml is in the hash because it picks the channel,
# even though it is kept in sync with the flake.
#
# Every segment the generated layout carries has to be carried here too:
# the runner pair, so a job on an x86 runner cannot restore the ARM64
# store, and `target`, so changing `github-cache-mode` cannot
# restore the other payload. The trailing run id rolls the key on every
# run, which is what lets a repeated `workflow_dispatch` seed a cold
# cache against GitHub's immutable entries.
cache-key: ${{ runner.os }}-${{ runner.arch }}-mbx-target-${{ inputs.scope }}-${{ hashFiles('flake.nix', 'flake.lock', 'rust-toolchain.toml') }}-${{ github.run_id }}-${{ github.run_attempt }}
restore-keys: ${{ runner.os }}-${{ runner.arch }}-mbx-target-${{ inputs.scope }}-${{ hashFiles('flake.nix', 'flake.lock', 'rust-toolchain.toml') }}-
save-on-workflow-dispatch: ${{ inputs.save-on-dispatch }}
version: 1.25.0
backend: ${{ inputs.access-key-id != '' && inputs.secret-access-key != '' && 'remote' || 'local' }}
remote-url: s3://mbx-moq-dev
namespace: ${{ runner.os }}-${{ runner.arch }}

# The action only warns when it cannot reach the remote, which is how a
# revoked or mistyped key would look. A cache that silently never hits
# costs every job its full build, so refuse instead. A writer also has to
# come out read-write, or nothing would publish and every reader would
# drift cold unnoticed.
- name: Check the cache
if: inputs.access-key-id != '' && inputs.secret-access-key != ''
shell: bash
run: |
report=$(mbx doctor --json || true)
remote=$(jq -r '.checks[] | select(.name == "remote") | "\(.severity): \(.detail)"' <<< "$report")
policy=$(jq -r '.checks[] | select(.name == "policy") | .detail' <<< "$report")
echo "remote: $remote"
echo "policy: $policy"
case "$remote" in
pass:*) ;;
*) echo "::error::R2 cache unusable: $remote"; exit 1 ;;
esac
if [ "$MBX_REMOTE_MODE" = "read-write" ] && [[ "$policy" != *"effective read-write" ]]; then
echo "::error::the cache writer cannot publish: $policy"
exit 1
fi

# The shim is how mbx reaches Cargo. The action above only puts `mbx` itself
# on PATH, which reaches a build that spells the wrapper out; nothing here
# does, and release-plz spawns Cargo where no environment variable could.
# `mbx setup` installs a `cargo` launcher instead, and flake.nix puts its
# directory in front of the dev shell's pinned toolchain.
# `mbx setup` installs a `cargo` launcher instead. Its directory goes on
# PATH for jobs that use the runner's rustup toolchain, and flake.nix puts
# it back in front of the dev shell's pinned toolchain.
#
# `--status` is the assertion, not decoration: the shim is the only thing
# routing Cargo through mbx, so a setup that quietly did nothing would build
# every job uncached and still report success.
# The shim existing is the assertion: it is the only thing routing Cargo
# through mbx, so a setup that quietly did nothing would build every job
# uncached and still report success.
- name: Cargo shim
shell: bash
run: |
mbx setup --yes
mbx setup --status
shim=$(mbx setup --status 2>/dev/null | sed -n '1s/.*: //p' || true)
if [ ! -e "$shim" ]; then
echo "::error::mbx setup installed no Cargo shim: $(mbx setup --status 2>&1)"
exit 1
fi
echo "${shim%[/\\]*}" >> "$GITHUB_PATH"
74 changes: 10 additions & 64 deletions .github/workflows/cache.yml
Original file line number Diff line number Diff line change
@@ -1,49 +1,27 @@
name: Cache

# Populates the Rust build cache that pull request jobs restore, and the dev
# shell closure that every nix job substitutes.
# Pushes the dev shell closure that every nix job substitutes.
#
# Actions scopes cache reads to the current branch plus the default branch, so
# only caches written here on `main` are reachable from a pull request. This
# workflow is therefore the single trusted writer. The action refuses to publish
# from a pull request regardless.
#
# The Rust store is ARM64 only, because every job that restores it runs there.
# Each store is about 6 GB against the repository's 10 GB budget, and Actions
# evicts least recently used first, so an x86 store nobody on a pull request
# read was enough to push out the one they all do. The x86 leg still pushes its
# dev shell, which the x86 jobs substitute.
#
# This runs the unscoped suite rather than the diff-scoped one: the point is to
# leave artifacts for whatever a future PR happens to touch, not to check `main`
# (the PR that became `main` was already checked). It blocks nobody.
# The Rust build cache is not warmed here: `main` and its merge queue publish
# what they compile as they run (see .github/actions/rust-cache).

permissions:
contents: read

on:
push:
branches: [main]
# A docs-only merge can't change the build graph, so don't spend a runner.
# Only the flake decides the dev shell, so don't spend a runner otherwise.
paths:
- "rs/**"
- "Cargo.toml"
- "Cargo.lock"
- "rust-toolchain.toml"
- "flake.nix"
- "flake.lock"
- "nix/**"
- ".github/workflows/cache.yml"
# The action pin and the cache key both live here, and either can roll
# the store. Without this the sole writer would not re-run after such a
# change, leaving every PR reader cold until something else landed.
- ".github/actions/rust-cache/**"
# Priming a cold cache by hand, e.g. after a toolchain bump rolls the key.
# Priming the shell by hand, e.g. after a failed push.
workflow_dispatch:

# Queue rather than cancel. A warm takes 30-50 minutes and `main` merges more
# often than that, so cancelling meant no warm ever finished and every pull
# request restored a store hours stale. Actions keeps only the newest pending
# run per group, so merges that land mid-warm still coalesce into one.
# Queue rather than cancel, so a run always finishes. Actions keeps only the
# newest pending run per group, so merges that land mid-run coalesce into one.
concurrency:
group: cache-${{ github.ref }}
cancel-in-progress: false
Expand All @@ -54,11 +32,7 @@ jobs:
strategy:
fail-fast: false
matrix:
include:
- runner: ubuntu-24.04-arm
rust: true
- runner: ubuntu-latest
rust: false
runner: [ubuntu-24.04-arm, ubuntu-latest]
runs-on: ${{ matrix.runner }}
timeout-minutes: 60

Expand All @@ -85,7 +59,7 @@ jobs:
# package, so the dev shell itself was never cached. Anything in it that
# nixpkgs doesn't serve prebuilt (uniffi-bindgen-go, a
# buildRustPackage from a git tag) was therefore compiled from source by
# every job that entered the shell. Push it once here so pull requests
# every job that entered the shell. Push it once here so later jobs
# substitute it instead.
- uses: cachix/cachix-action@38b082610b782e7e93e209c35fd730d399dee866 # v17
with:
Expand All @@ -99,31 +73,3 @@ jobs:
# out the build-time inputs.
nix develop --profile /tmp/devshell --command true
cachix push kixelated /tmp/devshell

# The single writer. `save-on-dispatch` is what lets the
# `workflow_dispatch` above prime a cold store; the action otherwise saves
# only on pushes to the default branch.
- name: Rust cache
if: matrix.rust
uses: ./.github/actions/rust-cache
with:
save-on-dispatch: true

# Both halves, so the saved cache carries the lint artifacts (clippy rmeta)
# and the test artifacts (codegen + linked test binaries). They share
# little, and a PR usually needs both.
#
# `continue-on-error` because the action saves only from a successful
# job, and the artifacts are worth saving whatever a lint or test on
# `main` says: a formatting slip or one flaky test otherwise left every
# pull request restoring a store days stale. The merge queue gates are
# what report those failures.
- name: Check
if: matrix.rust
continue-on-error: true
run: nix develop --command just ci check --all

- name: Test
if: ${{ matrix.rust && !cancelled() }}
continue-on-error: true
run: nix develop --command just ci test --all
20 changes: 14 additions & 6 deletions .github/workflows/check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ jobs:
# the concurrency group above.
if: github.event.action != 'closed' && (github.event_name != 'pull_request' || github.base_ref == 'release')
runs-on: ubuntu-24.04-arm
# `main` and its merge queue write the Rust cache; see .github/actions/rust-cache.
environment: ${{ (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/gh-readonly-queue/main/')) && 'cache-write' || '' }}
# Backstop a hang instead of holding the runner for the 6-hour default.
timeout-minutes: 60

Expand Down Expand Up @@ -71,14 +73,15 @@ jobs:
- uses: DeterminateSystems/flake-checker-action@786422608c7bded2bbc9741ad9f91356842bf520 # main
if: steps.scope.outputs.needed == 'true'

# cache.yml warms the default-branch store on `main`. Queue and release PR
# runs restore it; the action refuses to publish from them, and Actions
# would scope such an entry to its own ref anyway, where no later run
# could read it while it ate the repository's 10 GB budget. A selection
# of lints alone compiles no Rust, so it skips the restore.
# The merge queue writes what it compiles, so the cache follows what lands
# on `main` without a second build after it; release pull requests read.
# A selection of lints alone compiles no Rust, so it skips the cache.
- name: Rust cache
if: steps.scope.outputs.rust == 'true'
uses: ./.github/actions/rust-cache
with:
access-key-id: ${{ secrets.MBX_R2_ACCESS_KEY_ID }}
secret-access-key: ${{ secrets.MBX_R2_SECRET_ACCESS_KEY }}

# Half of the `just ci` a contributor runs before pushing, with the same
# MOQ_STRICT and nextest profile. In the queue it diffs against the merge
Expand All @@ -97,6 +100,8 @@ jobs:
name: Test
if: github.event.action != 'closed' && (github.event_name != 'pull_request' || github.base_ref == 'release')
runs-on: ubuntu-24.04-arm
# `main` and its merge queue write the Rust cache; see .github/actions/rust-cache.
environment: ${{ (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/gh-readonly-queue/main/')) && 'cache-write' || '' }}
timeout-minutes: 60

steps:
Expand Down Expand Up @@ -131,10 +136,13 @@ jobs:
extra-substituters = https://kixelated.cachix.org
extra-trusted-public-keys = kixelated.cachix.org-1:CmFcV0lyM6KuVM2m9mih0q4SrAa0XyCsiM7GHrz3KKk=

# Restore only, and only for selected Rust, same as the `check` job above.
# Reads and writes like the `check` job above.
- name: Rust cache
if: steps.scope.outputs.rust == 'true'
uses: ./.github/actions/rust-cache
with:
access-key-id: ${{ secrets.MBX_R2_ACCESS_KEY_ID }}
secret-access-key: ${{ secrets.MBX_R2_SECRET_ACCESS_KEY }}

- name: Test
if: steps.scope.outputs.needed == 'true'
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,8 @@ name: Dependabot
# calls one `gh` command against the pull request number.
#
# The moq-bot token rather than GITHUB_TOKEN: a merge enabled by GITHUB_TOKEN
# does not trigger further workflow runs, which would silently skip Cache
# (the Rust cache warm keyed on Cargo.lock, exactly what the cargo group
# changes), release-rs and release-js on every auto-merged bump.
# does not trigger further workflow runs, which would silently skip
# release-rs and release-js on every auto-merged bump.

permissions: {}

Expand Down
13 changes: 7 additions & 6 deletions .github/workflows/interop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ jobs:
interop:
name: Interop
runs-on: ubuntu-latest
# `main` and its merge queue write the Rust cache; see .github/actions/rust-cache.
environment: ${{ (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/gh-readonly-queue/main/')) && 'cache-write' || '' }}
# The matrix runs in real time (ffmpeg -re pacing + per-subscriber timeouts);
# this is a safety net against a hung relay/client plus a cold first build.
timeout-minutes: 60
Expand All @@ -68,14 +70,13 @@ jobs:
extra-substituters = https://kixelated.cachix.org
extra-trusted-public-keys = kixelated.cachix.org-1:CmFcV0lyM6KuVM2m9mih0q4SrAa0XyCsiM7GHrz3KKk=

# Reuse ~/.cargo + ./target across nightly runs so only changed crates
# recompile (the relay, cli, moq-ffi, and moq-c all build from source).
# The relay, cli, moq-ffi, and moq-c all build from source. The nightly
# run publishes them for the pull requests that touch the harness.
- name: Rust cache
uses: Swatinem/rust-cache@f0d9c3887740aee45f6153b24b3a6b815192ec16 # v2
uses: ./.github/actions/rust-cache
with:
# A pull request's entry is scoped to its own branch, where no later
# run reads it, while it spends the budget the shared store needs.
save-if: ${{ github.event_name != 'pull_request' }}
access-key-id: ${{ secrets.MBX_R2_ACCESS_KEY_ID }}
secret-access-key: ${{ secrets.MBX_R2_SECRET_ACCESS_KEY }}

# Playwright's Chromium isn't in the nix devShell. Install it plus its apt
# runtime libs (`--with-deps` needs the host package manager) so the
Expand Down
Loading