From b276639e7296479c91cc41f545f10cc2a97253d0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 27 Oct 2026 23:36:23 -0700 Subject: [PATCH 1/8] chore(quest): claim quest/m0/ci/nix-cache-verify.md Quest-Claim: 70923595651a7e0f From a575993f1f3f8a0142d87ecba5cb08a9ab8ba84d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 27 Oct 2026 23:40:55 -0700 Subject: [PATCH 2/8] docs(quest): record the verified Nix dev-shell cache Nightly run 40506938673 entered the dev shell in about a minute with nothing compiled. Narrow the quest to the release push, bucket sizes, and Cachix retirement, and note the smoke nix channel's cache gap. Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/README.md | 2 +- quest/m0/ci/nix-cache-verify.md | 57 ++++++++++++++++++--------------- 2 files changed, 32 insertions(+), 27 deletions(-) diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 46ae5afef89..5e5a007ca5f 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -129,7 +129,7 @@ The historical samples locate candidates; they are not controlled benchmarks. - [Land main through quest land](/quest/m0/ci/land.md) - one self-hosted runner tests every queued PR as a squash chain and fast-forwards `main` to the longest passing prefix - [Measure the parallel frontend](/quest/m0/ci/nightly-levers.md) - a pinned nightly's parallel frontend, adopted only if the numbers beat its risks; Cranelift is ruled out - [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measure the hosted Nightly jobs' hits, restore time, and expiry once the bump lands -- [Verify the Nix cache](/quest/m0/ci/nix-cache-verify.md) - confirm CI and releases use `nix.moq.dev` and retire Cachix +- [Verify the Nix cache](/quest/m0/ci/nix-cache-verify.md) - confirm releases use `nix.moq.dev`, check its bucket sizes, and retire Cachix - [Unify dependency features](/quest/m0/ci/hakari.md) - a cargo-hakari workspace-hack so every job compiles each dependency alike and mbx hits across jobs - [Replay full-run failures against the selected gate](/quest/m0/ci/select-audit.md) - every dependency edge `just ci` misses is found and fixed with a test - [First shared FFI release](/quest/m0/ci/nightly-ffi-reuse.md) - condition: the backport merges and the next `moq-ffi-v*` tag runs `Release FFI`; verify the wrappers diff --git a/quest/m0/ci/nix-cache-verify.md b/quest/m0/ci/nix-cache-verify.md index 2607b069630..4e24e5fef92 100644 --- a/quest/m0/ci/nix-cache-verify.md +++ b/quest/m0/ci/nix-cache-verify.md @@ -2,9 +2,8 @@ ## Goal -`nix.moq.dev` serves CI and releases: hosted Nix jobs enter the dev shell -without compiling it, release tags substitute on a clean machine, its bucket -sizes hold under the 30-day expiry, and Cachix is retired. +`nix.moq.dev` serves releases: release tags substitute on a clean machine, +its bucket sizes hold under the 30-day expiry, and Cachix is retired. ## Plan @@ -14,36 +13,42 @@ release tag ruleset, both buckets and the Worker, the `cache-write` Environment's tag policies and secrets, and the signing key). Check it with `curl https://nix.moq.dev/nix-cache-info`. -[#6588](https://github.com/moq-dev/moq/pull/6588) restored `cache.yml` -after [#6040](https://github.com/moq-dev/moq/pull/6040) deleted it: every -merge to `main` pushes the dev shell's build closure for x86 Linux, ARM Linux, -and ARM macOS to `nix-ci-moq-dev` (`sh/gh/cache.sh`), and `release-nix.yml` -pushes on release tags. So the dev-shell entry time can be measured on any -scheduled Nightly now, and the expiry check below applies. +[#6588](https://github.com/moq-dev/moq/pull/6588) restored `cache.yml`: +every merge to `main` pushes the dev shell's build closure for x86 Linux, ARM +Linux, and ARM macOS to `nix-ci-moq-dev` (`sh/gh/cache.sh`), and +`release-nix.yml` pushes on release tags. Hosted CI entering the dev shell is +verified: Nightly run 40506938673 (2026-10-27) substituted it in about a +minute with nothing compiled, against 12-14 minutes of source builds in the +2026-10-16 baseline. -Then confirm it, and record the numbers in the PR that closes this: +Then confirm the rest, and record the numbers in the PR that closes this: -- A hosted Nightly job enters `nix develop` without compiling - uniffi-bindgen-go. The baseline, measured 2026-10-16 in Nightly run - 38897397428: every workload spent 12-14 minutes building - `uniffi-bindgen-go`, `uniffi-bindgen-cpp` and `quest` before entering the - dev shell, because Cachix had no copy. - The first `moq-relay-v*` or `moq-cli-v*` tag cut after `release` carries `release-nix.yml` pushes from all three legs, and `nix run github:moq-dev/moq/#moq-relay --accept-flake-config` substitutes on a clean machine. Tags are cut from `release`, which still has `cachix.yml` and `cache.yml` - and no `release-nix.yml`, so the 2026-10-23 tags (`moq-relay-v0.17.8`, - `moq-cli-v0.14.8`) ran Cachix push and never reached `nix.moq.dev`. The - next cut from `main` fixes that; confirm its first tag's run. + and no `release-nix.yml`, so every tag so far, through `moq-relay-v0.17.9` + and `moq-cli-v0.14.9` on 2026-10-28, ran Cachix push and never reached + `nix.moq.dev`. Check with `gh run list --workflow release-nix.yml`; it stays + empty until a release cut merges `main` into `release`. - Both bucket sizes after a month, against the 30-day expiry in - `infra/nix/lifecycle.json`. A path expires 30 days after its upload even - while CI still reads it. If a dev-shell push is restored, look for jobs - that fell back to source builds once its paths expired, and refresh it on a - schedule or switch to a closure-aware collector (niks3) if they recur. -- Delete the `CACHIX_AUTH_TOKEN` secret and the `kixelated` Cachix cache once - the maintainer decides downstream flakes have had long enough to switch, - along with `smoke.yml`'s Cachix substituter and its push step for the nix - channel. + `infra/nix/lifecycle.json`, so no earlier than 2026-11-24: the first + `cache.yml` push to `nix-ci-moq-dev` ran 2026-10-24. Sizes need a + `wrangler login` to the Cloudflare account. A path expires 30 days after + its upload even while CI still reads it, and `nix copy` re-uploads it on the + next merge. Look for Nightly jobs that fell back to source builds once + paths expired (`building '/nix/store/...uniffi-bindgen` in a job log), and + refresh on a schedule or switch to a closure-aware collector (niks3) if + they recur. +- Retire Cachix once the maintainer decides downstream flakes have had long + enough to switch: delete the `CACHIX_AUTH_TOKEN` secret and the `kixelated` + Cachix cache, `smoke.yml`'s Cachix substituter and its push step for the + nix channel, and `CACHIX_SUB`/`CACHIX_KEY` in `test/smoke/justfile`. The + smoke nix channel builds `main` HEAD's `moq-relay` and `moq` packages, which + `cache.yml` never pushes, so without the Cachix push each new HEAD compiles + for 20-43 minutes there. Decide whether `cache.yml` also pushes those two + packages' runtime closures to `nix-ci-moq-dev`, smoke pushes them under the + `cache-write` Environment, or the channel accepts the compile. ## Related From 4e4e2e3234710be8de7fad5118c6b76454bee6bd Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 2 Nov 2026 20:41:46 -0800 Subject: [PATCH 3/8] chore(quest): claim quest/m0/ci/nix-cache-verify.md Quest-Claim: 40509a86543a98c9 From 3534f5d600d835f3b8be43b28ef9cf079b444bc8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 2 Nov 2026 20:44:31 -0800 Subject: [PATCH 4/8] ci: cache relay and CLI runtime closures on main Co-Authored-By: GPT-6.1 Sol --- .github/justfile | 2 +- .github/workflows/cache.yml | 4 ++-- flake.nix | 10 ++++---- infra/README.md | 8 ++++--- quest/m0/ci/nix-cache-verify.md | 41 ++++++++++++++++++--------------- sh/gh/cache.sh | 11 +++++++-- 6 files changed, 45 insertions(+), 31 deletions(-) diff --git a/.github/justfile b/.github/justfile index 7479ab77ce2..6e7772292dd 100644 --- a/.github/justfile +++ b/.github/justfile @@ -1,6 +1,6 @@ set working-directory := '..' -# Build and publish the development environment to nix.moq.dev. +# Build and publish the dev shell and Smoke's runtime packages to nix.moq.dev. cache: bash sh/gh/cache.sh diff --git a/.github/workflows/cache.yml b/.github/workflows/cache.yml index 6ffce7cb10d..16521aa2812 100644 --- a/.github/workflows/cache.yml +++ b/.github/workflows/cache.yml @@ -1,6 +1,6 @@ name: Cache -# Publish development environments independently of the landing worker's local caches. +# Publish dev shells and Smoke's runtime packages independently of the landing worker's local caches. permissions: contents: read @@ -27,7 +27,7 @@ jobs: persist-credentials: false - uses: ./.github/actions/nix - uses: ./.github/actions/just - - name: Cache the dev shell + - name: Cache the dev shell and runtime packages env: NIX_SIGNING_KEY: ${{ secrets.NIX_SIGNING_KEY }} AWS_ACCESS_KEY_ID: ${{ secrets.NIX_R2_ACCESS_KEY_ID }} diff --git a/flake.nix b/flake.nix index f291856ab86..fb802a933d8 100644 --- a/flake.nix +++ b/flake.nix @@ -1,16 +1,16 @@ { description = "MoQ - Media over QUIC"; - # Pre-built binaries live in our binary cache at nix.moq.dev. Of the - # packages, only tagged releases are pushed (moq-relay-v*, moq-cli-v*, etc.), - # so pin the flake ref to a recent tag to get a hit. The default branch HEAD - # builds from source, though its dev shell is cached: + # Pre-built binaries live in our binary cache at nix.moq.dev. Tagged + # releases are retained permanently. The default branch's dev shell, + # moq-relay, and moq are also cached, with a 30-day expiry. Pin the flake ref + # to a recent tag for a retained package: # nix run github:moq-dev/moq/moq-relay-v0.12.4#moq-relay --accept-flake-config # # --accept-flake-config opts into the nixConfig below for one command. To # trust the cache permanently instead, add these settings to nix.conf. # - # The dev shell paths expire 30 days after upload, possibly before a path + # The CI paths expire 30 days after upload, possibly before a path # that depends on them; `fallback` builds whatever is missing instead of # failing the substitution. nixConfig = { diff --git a/infra/README.md b/infra/README.md index 62d7ade9aaf..cf83c2890af 100644 --- a/infra/README.md +++ b/infra/README.md @@ -266,9 +266,11 @@ gh workflow run rpm-repo.yml --ref release -f tag=moq-relay-v1.2.3 `nix.moq.dev` serves two buckets as one binary cache. `nix-moq-dev` holds released packages, pushed by `release-nix.yml` on a release tag, and never -expires. `nix-ci-moq-dev` holds the dev shells `cache.yml` pushes after every -merge to `main`, for x86 Linux, ARM Linux, and ARM macOS. It publishes the full -build closure from `nix develop --profile`, uploading only missing paths. +expires. `nix-ci-moq-dev` holds the dev shells and the `moq-relay` and `moq` +packages `cache.yml` pushes after every merge to `main`, for x86 Linux, ARM +Linux, and ARM macOS. It publishes the full build closure from +`nix develop --profile` and the packages' runtime closures, uploading only +missing paths. Smoke's nix channel reads these packages without write keys. Its lifecycle rule (`infra/nix/lifecycle.json`) deletes each object 30 days after upload; the next warm re-uploads what it still needs. `flake.nix` sets `fallback = true` because that expiry can remove a path before one that diff --git a/quest/m0/ci/nix-cache-verify.md b/quest/m0/ci/nix-cache-verify.md index 3b67f7bc579..ab38fcb2a7c 100644 --- a/quest/m0/ci/nix-cache-verify.md +++ b/quest/m0/ci/nix-cache-verify.md @@ -21,18 +21,33 @@ verified: Nightly run 40506938673 (2026-10-27) substituted it in about a minute with nothing compiled, against 12-14 minutes of source builds in the 2026-10-16 baseline. -Then confirm the rest, and record the numbers in the PR that closes this: +`cache.yml` now also builds and pushes the `moq-relay` and `moq` runtime +closures to `nix-ci-moq-dev` on all three platforms (decided 2026-10-29). +Smoke's nix channel reads them without write credentials. Verify the first +post-merge Cache run uploads both and the next Smoke nix run substitutes +them; a local build does not prove hosted publication. -- The first `moq-relay-v*` or `moq-cli-v*` tag cut after `release` carries - `release-nix.yml` pushes from all three legs, and `nix run github:moq-dev/moq/#moq-relay - --accept-flake-config` substitutes on a clean machine. +[moq-dev/moq#7431](https://github.com/moq-dev/moq/pull/7431) dropped Cachix +from `main` (decided 2026-11-02): Smoke substitutes from `nix.moq.dev` and +no longer pushes. `curl https://nix.moq.dev/nix-cache-info` still succeeds +on 2026-11-02. `release-nix.yml` still has no runs that day. + +## Waiting + +- moq-dev/moq: After this cache coverage lands, record the first successful + Cache uploads of both runtime closures and a Smoke nix run substituting + them from `nix.moq.dev`. +- Maintainer: Cut a release from `main`, then record the first `moq-relay-v*` + or `moq-cli-v*` tag's `release-nix.yml` pushes from all three legs, and + `nix run github:moq-dev/moq/#moq-relay --accept-flake-config` + substituting on a clean machine (use `#moq` for a CLI tag). Tags are cut from `release`, which still has `cachix.yml` and `cache.yml` and no `release-nix.yml`, so every tag so far, through `moq-relay-v0.17.9` and `moq-cli-v0.14.9` on 2026-10-28, ran Cachix push and never reached `nix.moq.dev`. Check with `gh run list --workflow release-nix.yml`; it stays empty until a release cut merges `main` into `release`. -- Both bucket sizes after a month, against the 30-day expiry in - `infra/nix/lifecycle.json`, so no earlier than 2026-11-24: the first +- Maintainer: Record both bucket sizes after a month, against the CI bucket's + 30-day expiry in `infra/nix/lifecycle.json`, so no earlier than 2026-11-24: the first `cache.yml` push to `nix-ci-moq-dev` ran 2026-10-24. Sizes need a `wrangler login` to the Cloudflare account. A path expires 30 days after its upload even while CI still reads it, and `nix copy` re-uploads it on the @@ -40,22 +55,12 @@ Then confirm the rest, and record the numbers in the PR that closes this: paths expired (`building '/nix/store/...uniffi-bindgen` in a job log), and refresh on a schedule or switch to a closure-aware collector (niks3) if they recur. -- Retire Cachix once the first release cut from `main` shows +- Maintainer: Retire Cachix once the first release cut from `main` shows `release-nix.yml` succeeding on all three legs and the clean-machine `nix run` above substitutes (decided 2026-10-29): that is the first release a downstream user could fetch from `nix.moq.dev`. Then delete the `CACHIX_AUTH_TOKEN` secret and the `kixelated` Cachix cache. - [moq-dev/moq#7431](https://github.com/moq-dev/moq/pull/7431) already - dropped Cachix from `main` (decided 2026-11-02, in place of moving its token - into `cache-write`): Smoke's nix channel substitutes from `nix.moq.dev` and - no longer pushes. `release` keeps `cachix.yml` until that cut merges `main` - into it. -- Also push the `moq-relay` and `moq` runtime closures to `nix-ci-moq-dev` - (decided 2026-10-29). The smoke nix channel builds `main` HEAD's - `moq-relay` and `moq` packages, which nothing caches since Smoke stopped - pushing to Cachix, so each new HEAD compiles for 20-43 minutes there. - `cache.yml` already holds the write credentials on every merge, so smoke - needs no secret. + `release` keeps `cachix.yml` until that cut merges `main` into it. ## Related diff --git a/sh/gh/cache.sh b/sh/gh/cache.sh index 1beaed69496..dd64d4ff101 100644 --- a/sh/gh/cache.sh +++ b/sh/gh/cache.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Publish the dev shell's build closure, including its build-time inputs. +# Publish the dev shell's build closure and Smoke's runtime packages. set -euo pipefail : "${NIX_SIGNING_KEY:?not set; the job needs the cache-write Environment}" @@ -11,4 +11,11 @@ trap 'rm -rf "$profile"' EXIT # Building the devShell derivation alone omits inputs that nix develop needs. nix develop --profile "$profile/devshell" --command true -just infra nix push nix-ci-moq-dev "$profile/devshell" + +# Smoke's nix channel builds main HEAD, so it needs these runtime closures +# even before the next release tag publishes them to the release bucket. +for package in moq-relay moq; do + nix build ".#$package" --out-link "$profile/$package" --print-build-logs +done + +just infra nix push nix-ci-moq-dev "$profile/devshell" "$profile/moq-relay" "$profile/moq" From 1a66b76e6d8e464a6b86c173a19149e4e72f2601 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 2 Nov 2026 20:52:31 -0800 Subject: [PATCH 5/8] ci: preserve shell warming and cache the active Smoke platform Co-Authored-By: GPT-6.1 Sol --- .github/workflows/smoke.yml | 4 ++-- flake.nix | 6 +++--- infra/README.md | 11 ++++++----- quest/m0/ci/nix-cache-verify.md | 5 +++-- sh/gh/cache.sh | 15 ++++++++------- 5 files changed, 22 insertions(+), 19 deletions(-) diff --git a/.github/workflows/smoke.yml b/.github/workflows/smoke.yml index b63123e1f8d..471df1771bb 100644 --- a/.github/workflows/smoke.yml +++ b/.github/workflows/smoke.yml @@ -41,8 +41,8 @@ jobs: runs-on: ${{ matrix.os }} # The matrix runs in real time (ffmpeg -re pacing + per-subscriber timeouts); # this is just a safety net against a hung relay/client. The nix channel - # gets longer: nix.moq.dev serves the toolchain but not moq's packages at - # HEAD, and compiling them has taken 43 minutes. + # gets longer for a cold build when Cache has not finished publishing + # HEAD's packages yet; compiling them has taken 43 minutes. timeout-minutes: ${{ matrix.channel == 'nix' && 75 || 45 }} # smoke.sh resolves the latest libmoq release via the GitHub API. Without a diff --git a/flake.nix b/flake.nix index fb802a933d8..680b1a4682c 100644 --- a/flake.nix +++ b/flake.nix @@ -2,9 +2,9 @@ description = "MoQ - Media over QUIC"; # Pre-built binaries live in our binary cache at nix.moq.dev. Tagged - # releases are retained permanently. The default branch's dev shell, - # moq-relay, and moq are also cached, with a 30-day expiry. Pin the flake ref - # to a recent tag for a retained package: + # releases are retained permanently. The default branch's dev shell is + # also cached, along with moq-relay and moq on x86 Linux, with a 30-day + # expiry. Pin the flake ref to a recent tag for a retained package: # nix run github:moq-dev/moq/moq-relay-v0.12.4#moq-relay --accept-flake-config # # --accept-flake-config opts into the nixConfig below for one command. To diff --git a/infra/README.md b/infra/README.md index cf83c2890af..36b6d084290 100644 --- a/infra/README.md +++ b/infra/README.md @@ -266,11 +266,12 @@ gh workflow run rpm-repo.yml --ref release -f tag=moq-relay-v1.2.3 `nix.moq.dev` serves two buckets as one binary cache. `nix-moq-dev` holds released packages, pushed by `release-nix.yml` on a release tag, and never -expires. `nix-ci-moq-dev` holds the dev shells and the `moq-relay` and `moq` -packages `cache.yml` pushes after every merge to `main`, for x86 Linux, ARM -Linux, and ARM macOS. It publishes the full build closure from -`nix develop --profile` and the packages' runtime closures, uploading only -missing paths. Smoke's nix channel reads these packages without write keys. +expires. `nix-ci-moq-dev` holds the dev shells `cache.yml` pushes after every +merge to `main`, for x86 Linux, ARM Linux, and ARM macOS. It publishes the +full build closure from `nix develop --profile`, uploading only missing +paths. On x86 Linux it then builds and pushes the `moq-relay` and `moq` +runtime closures for Smoke's nix channel, which reads them without write +keys. The other platforms have no Smoke nix lane. Its lifecycle rule (`infra/nix/lifecycle.json`) deletes each object 30 days after upload; the next warm re-uploads what it still needs. `flake.nix` sets `fallback = true` because that expiry can remove a path before one that diff --git a/quest/m0/ci/nix-cache-verify.md b/quest/m0/ci/nix-cache-verify.md index ab38fcb2a7c..256dacb6687 100644 --- a/quest/m0/ci/nix-cache-verify.md +++ b/quest/m0/ci/nix-cache-verify.md @@ -22,8 +22,9 @@ minute with nothing compiled, against 12-14 minutes of source builds in the 2026-10-16 baseline. `cache.yml` now also builds and pushes the `moq-relay` and `moq` runtime -closures to `nix-ci-moq-dev` on all three platforms (decided 2026-10-29). -Smoke's nix channel reads them without write credentials. Verify the first +closures to `nix-ci-moq-dev` on x86 Linux, Smoke's only current nix platform +(runtime caching decided 2026-10-29). All three platforms still publish +their dev shells first. Smoke reads the packages without write credentials. Verify the first post-merge Cache run uploads both and the next Smoke nix run substitutes them; a local build does not prove hosted publication. diff --git a/sh/gh/cache.sh b/sh/gh/cache.sh index dd64d4ff101..7f5c70deb80 100644 --- a/sh/gh/cache.sh +++ b/sh/gh/cache.sh @@ -11,11 +11,12 @@ trap 'rm -rf "$profile"' EXIT # Building the devShell derivation alone omits inputs that nix develop needs. nix develop --profile "$profile/devshell" --command true +just infra nix push nix-ci-moq-dev "$profile/devshell" -# Smoke's nix channel builds main HEAD, so it needs these runtime closures -# even before the next release tag publishes them to the release bucket. -for package in moq-relay moq; do - nix build ".#$package" --out-link "$profile/$package" --print-build-logs -done - -just infra nix push nix-ci-moq-dev "$profile/devshell" "$profile/moq-relay" "$profile/moq" +# Smoke's nix channel currently runs only on x86 Linux and builds main HEAD. +# Publish its runtime closures without making the dev shell wait on Rust. +system=$(nix eval --impure --raw --expr builtins.currentSystem) +if [[ "$system" == x86_64-linux ]]; then + nix build .#moq-relay .#moq --out-link "$profile/runtime" --print-build-logs + just infra nix push nix-ci-moq-dev "$profile/runtime" "$profile/runtime-1" +fi From 80c81f6c77f8b0425178bc0ff974d3f91b81b17c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 2 Nov 2026 20:59:27 -0800 Subject: [PATCH 6/8] ci: reuse runtime dependency artifacts between cache warms Co-Authored-By: GPT-6.1 Sol --- .github/workflows/cache.yml | 6 ++++++ .github/workflows/smoke.yml | 3 ++- infra/README.md | 10 +++++++--- quest/m0/ci/nix-cache-verify.md | 4 ++++ sh/gh/cache.sh | 5 +++++ 5 files changed, 24 insertions(+), 4 deletions(-) diff --git a/.github/workflows/cache.yml b/.github/workflows/cache.yml index 16521aa2812..29f25f05651 100644 --- a/.github/workflows/cache.yml +++ b/.github/workflows/cache.yml @@ -9,6 +9,12 @@ on: branches: [main] workflow_dispatch: +# Finish the active publication; new pushes replace only the pending warm. +# Canceling active builds could starve a cold dependency upload on busy main. +concurrency: + group: cache-${{ github.ref }} + cancel-in-progress: false + jobs: warm: name: Warm (${{ matrix.runner }}) diff --git a/.github/workflows/smoke.yml b/.github/workflows/smoke.yml index 471df1771bb..070d68c99f2 100644 --- a/.github/workflows/smoke.yml +++ b/.github/workflows/smoke.yml @@ -74,7 +74,8 @@ jobs: - os: ubuntu-latest channel: docker # macOS cells temporarily disabled to cut CI spend (over budget this - # month). Re-enable by uncommenting; nothing else depends on them. + # month). Re-enable by uncommenting; the nix cell also needs its + # platform added to sh/gh/cache.sh's runtime publication. # - os: macos-latest # channel: cargo # - os: macos-latest diff --git a/infra/README.md b/infra/README.md index 36b6d084290..a3fe4ec9f49 100644 --- a/infra/README.md +++ b/infra/README.md @@ -266,12 +266,16 @@ gh workflow run rpm-repo.yml --ref release -f tag=moq-relay-v1.2.3 `nix.moq.dev` serves two buckets as one binary cache. `nix-moq-dev` holds released packages, pushed by `release-nix.yml` on a release tag, and never -expires. `nix-ci-moq-dev` holds the dev shells `cache.yml` pushes after every -merge to `main`, for x86 Linux, ARM Linux, and ARM macOS. It publishes the +expires. `nix-ci-moq-dev` holds the dev shells `cache.yml` pushes on `main`, +for x86 Linux, ARM Linux, and ARM macOS. It publishes the full build closure from `nix develop --profile`, uploading only missing paths. On x86 Linux it then builds and pushes the `moq-relay` and `moq` runtime closures for Smoke's nix channel, which reads them without write -keys. The other platforms have no Smoke nix lane. +keys. Their existing Crane `cargoArtifacts` outputs are published first so +source-only changes can reuse the dependency builds. The other platforms +have no Smoke nix lane. Cache runs serialize: the active warm finishes and +new pushes replace only the pending run, avoiding duplicate builds without +starving publication on a busy branch. Its lifecycle rule (`infra/nix/lifecycle.json`) deletes each object 30 days after upload; the next warm re-uploads what it still needs. `flake.nix` sets `fallback = true` because that expiry can remove a path before one that diff --git a/quest/m0/ci/nix-cache-verify.md b/quest/m0/ci/nix-cache-verify.md index 256dacb6687..7fee9265758 100644 --- a/quest/m0/ci/nix-cache-verify.md +++ b/quest/m0/ci/nix-cache-verify.md @@ -27,6 +27,10 @@ closures to `nix-ci-moq-dev` on x86 Linux, Smoke's only current nix platform their dev shells first. Smoke reads the packages without write credentials. Verify the first post-merge Cache run uploads both and the next Smoke nix run substitutes them; a local build does not prove hosted publication. +The runtime packages' existing Crane dependency artifacts are published +first so source-only changes can reuse them. Cache finishes the active +warm and keeps only the newest pending run rather than building every +intermediate commit or repeatedly canceling cold dependency builds. [moq-dev/moq#7431](https://github.com/moq-dev/moq/pull/7431) dropped Cachix from `main` (decided 2026-11-02): Smoke substitutes from `nix.moq.dev` and diff --git a/sh/gh/cache.sh b/sh/gh/cache.sh index 7f5c70deb80..d6f9f2fddde 100644 --- a/sh/gh/cache.sh +++ b/sh/gh/cache.sh @@ -17,6 +17,11 @@ just infra nix push nix-ci-moq-dev "$profile/devshell" # Publish its runtime closures without making the dev shell wait on Rust. system=$(nix eval --impure --raw --expr builtins.currentSystem) if [[ "$system" == x86_64-linux ]]; then + # These existing Crane outputs survive source-only changes. Publish them + # first so a later package build can substitute its dependency layer. + nix build .#moq-relay.cargoArtifacts .#moq.cargoArtifacts --out-link "$profile/deps" --print-build-logs + just infra nix push nix-ci-moq-dev "$profile/deps" "$profile/deps-1" + nix build .#moq-relay .#moq --out-link "$profile/runtime" --print-build-logs just infra nix push nix-ci-moq-dev "$profile/runtime" "$profile/runtime-1" fi From ddb70c383cfc1b4ec6f1b9b25f23f4831594dba0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 2 Nov 2026 21:04:00 -0800 Subject: [PATCH 7/8] ci: publish runtime caches independently of dev shells Co-Authored-By: GPT-6.1 Sol --- .github/justfile | 6 +++++- .github/workflows/cache.yml | 35 ++++++++++++++++++++++++++------- .github/workflows/smoke.yml | 2 +- infra/README.md | 15 +++++++------- quest/m0/ci/nix-cache-verify.md | 11 ++++++----- sh/gh/cache-runtime.sh | 18 +++++++++++++++++ sh/gh/cache.sh | 15 +------------- 7 files changed, 67 insertions(+), 35 deletions(-) create mode 100755 sh/gh/cache-runtime.sh diff --git a/.github/justfile b/.github/justfile index 6e7772292dd..eedc99fba46 100644 --- a/.github/justfile +++ b/.github/justfile @@ -1,9 +1,13 @@ set working-directory := '..' -# Build and publish the dev shell and Smoke's runtime packages to nix.moq.dev. +# Build and publish the development environment to nix.moq.dev. cache: bash sh/gh/cache.sh +# Publish Smoke's runtime packages and their build artifacts to nix.moq.dev. +cache-runtime: + bash sh/gh/cache-runtime.sh + # Build the runner image with a temporary policy on a disposable hosted runner. runner-vm-ci: #!/usr/bin/env bash diff --git a/.github/workflows/cache.yml b/.github/workflows/cache.yml index 29f25f05651..946f6b4572d 100644 --- a/.github/workflows/cache.yml +++ b/.github/workflows/cache.yml @@ -9,12 +9,6 @@ on: branches: [main] workflow_dispatch: -# Finish the active publication; new pushes replace only the pending warm. -# Canceling active builds could starve a cold dependency upload on busy main. -concurrency: - group: cache-${{ github.ref }} - cancel-in-progress: false - jobs: warm: name: Warm (${{ matrix.runner }}) @@ -33,9 +27,36 @@ jobs: persist-credentials: false - uses: ./.github/actions/nix - uses: ./.github/actions/just - - name: Cache the dev shell and runtime packages + - name: Cache the dev shell env: NIX_SIGNING_KEY: ${{ secrets.NIX_SIGNING_KEY }} AWS_ACCESS_KEY_ID: ${{ secrets.NIX_R2_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.NIX_R2_SECRET_ACCESS_KEY }} run: just gh cache + + runtime: + name: Runtime (ubuntu-24.04) + if: github.ref == 'refs/heads/main' + # Smoke's only current nix platform. Keep its Rust builds independent of + # the dev-shell matrix and use Smoke's cold-build budget. + runs-on: ubuntu-24.04 + timeout-minutes: 75 + environment: cache-write + # Finish the active publication; new pushes replace only the pending run. + # Canceling active builds could starve cold dependency uploads on busy main. + concurrency: + group: cache-runtime-${{ github.ref }} + cancel-in-progress: false + + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: ./.github/actions/nix + - uses: ./.github/actions/just + - name: Cache runtime packages and dependencies + env: + NIX_SIGNING_KEY: ${{ secrets.NIX_SIGNING_KEY }} + AWS_ACCESS_KEY_ID: ${{ secrets.NIX_R2_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.NIX_R2_SECRET_ACCESS_KEY }} + run: just gh cache-runtime diff --git a/.github/workflows/smoke.yml b/.github/workflows/smoke.yml index 070d68c99f2..dbc8513b25c 100644 --- a/.github/workflows/smoke.yml +++ b/.github/workflows/smoke.yml @@ -75,7 +75,7 @@ jobs: channel: docker # macOS cells temporarily disabled to cut CI spend (over budget this # month). Re-enable by uncommenting; the nix cell also needs its - # platform added to sh/gh/cache.sh's runtime publication. + # platform added to cache.yml's runtime job. # - os: macos-latest # channel: cargo # - os: macos-latest diff --git a/infra/README.md b/infra/README.md index a3fe4ec9f49..05d8a52ad90 100644 --- a/infra/README.md +++ b/infra/README.md @@ -269,13 +269,14 @@ released packages, pushed by `release-nix.yml` on a release tag, and never expires. `nix-ci-moq-dev` holds the dev shells `cache.yml` pushes on `main`, for x86 Linux, ARM Linux, and ARM macOS. It publishes the full build closure from `nix develop --profile`, uploading only missing -paths. On x86 Linux it then builds and pushes the `moq-relay` and `moq` -runtime closures for Smoke's nix channel, which reads them without write -keys. Their existing Crane `cargoArtifacts` outputs are published first so -source-only changes can reuse the dependency builds. The other platforms -have no Smoke nix lane. Cache runs serialize: the active warm finishes and -new pushes replace only the pending run, avoiding duplicate builds without -starving publication on a busy branch. +paths. An independent x86 Linux job builds and pushes the `moq-relay` and +`moq` runtime closures for Smoke's nix channel, which reads them without +write keys. Their existing Crane `cargoArtifacts` outputs are published +first so later builds can substitute them when their inputs are unchanged; +those inputs include the patched `kio` sources. The other platforms have no +Smoke nix lane. Runtime warms serialize: the active run finishes and new +pushes replace only the pending run, avoiding duplicate builds without +starving publication or delaying the dev-shell matrix. Its lifecycle rule (`infra/nix/lifecycle.json`) deletes each object 30 days after upload; the next warm re-uploads what it still needs. `flake.nix` sets `fallback = true` because that expiry can remove a path before one that diff --git a/quest/m0/ci/nix-cache-verify.md b/quest/m0/ci/nix-cache-verify.md index 7fee9265758..de7041c65c9 100644 --- a/quest/m0/ci/nix-cache-verify.md +++ b/quest/m0/ci/nix-cache-verify.md @@ -24,13 +24,14 @@ minute with nothing compiled, against 12-14 minutes of source builds in the `cache.yml` now also builds and pushes the `moq-relay` and `moq` runtime closures to `nix-ci-moq-dev` on x86 Linux, Smoke's only current nix platform (runtime caching decided 2026-10-29). All three platforms still publish -their dev shells first. Smoke reads the packages without write credentials. Verify the first +their dev shells independently. Smoke reads the packages without write credentials. Verify the first post-merge Cache run uploads both and the next Smoke nix run substitutes them; a local build does not prove hosted publication. -The runtime packages' existing Crane dependency artifacts are published -first so source-only changes can reuse them. Cache finishes the active -warm and keeps only the newest pending run rather than building every -intermediate commit or repeatedly canceling cold dependency builds. +The runtime job publishes the packages' existing Crane dependency artifacts +first, reusing them while their inputs (including patched `kio` sources) +are unchanged. It has Smoke's 75-minute cold-build budget and finishes the +active warm while keeping only the newest pending runtime run. The dev-shell +matrix does not queue behind it. [moq-dev/moq#7431](https://github.com/moq-dev/moq/pull/7431) dropped Cachix from `main` (decided 2026-11-02): Smoke substitutes from `nix.moq.dev` and diff --git a/sh/gh/cache-runtime.sh b/sh/gh/cache-runtime.sh new file mode 100755 index 00000000000..69f98a7cf9a --- /dev/null +++ b/sh/gh/cache-runtime.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Publish Smoke's runtime packages and their reusable dependency artifacts. +set -euo pipefail + +: "${NIX_SIGNING_KEY:?not set; the job needs the cache-write Environment}" +: "${AWS_ACCESS_KEY_ID:?not set}" +: "${AWS_SECRET_ACCESS_KEY:?not set}" + +roots=$(mktemp -d) +trap 'rm -rf "$roots"' EXIT + +# Publish Crane's existing dependency outputs first so later package builds +# can substitute them. Their inputs include the patched kio sources. +nix build .#moq-relay.cargoArtifacts .#moq.cargoArtifacts --out-link "$roots/deps" --print-build-logs +just infra nix push nix-ci-moq-dev "$roots/deps" "$roots/deps-1" + +nix build .#moq-relay .#moq --out-link "$roots/runtime" --print-build-logs +just infra nix push nix-ci-moq-dev "$roots/runtime" "$roots/runtime-1" diff --git a/sh/gh/cache.sh b/sh/gh/cache.sh index d6f9f2fddde..1beaed69496 100644 --- a/sh/gh/cache.sh +++ b/sh/gh/cache.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Publish the dev shell's build closure and Smoke's runtime packages. +# Publish the dev shell's build closure, including its build-time inputs. set -euo pipefail : "${NIX_SIGNING_KEY:?not set; the job needs the cache-write Environment}" @@ -12,16 +12,3 @@ trap 'rm -rf "$profile"' EXIT # Building the devShell derivation alone omits inputs that nix develop needs. nix develop --profile "$profile/devshell" --command true just infra nix push nix-ci-moq-dev "$profile/devshell" - -# Smoke's nix channel currently runs only on x86 Linux and builds main HEAD. -# Publish its runtime closures without making the dev shell wait on Rust. -system=$(nix eval --impure --raw --expr builtins.currentSystem) -if [[ "$system" == x86_64-linux ]]; then - # These existing Crane outputs survive source-only changes. Publish them - # first so a later package build can substitute its dependency layer. - nix build .#moq-relay.cargoArtifacts .#moq.cargoArtifacts --out-link "$profile/deps" --print-build-logs - just infra nix push nix-ci-moq-dev "$profile/deps" "$profile/deps-1" - - nix build .#moq-relay .#moq --out-link "$profile/runtime" --print-build-logs - just infra nix push nix-ci-moq-dev "$profile/runtime" "$profile/runtime-1" -fi From 54deb8cdf8e385ef1fddf5a85ac1e9e1bb07b16c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 2 Nov 2026 21:09:28 -0800 Subject: [PATCH 8/8] docs(nix): distinguish cached main from pre-cut release tags Co-Authored-By: GPT-6.1 Sol --- flake.nix | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/flake.nix b/flake.nix index 680b1a4682c..0902b269b8b 100644 --- a/flake.nix +++ b/flake.nix @@ -1,11 +1,11 @@ { description = "MoQ - Media over QUIC"; - # Pre-built binaries live in our binary cache at nix.moq.dev. Tagged - # releases are retained permanently. The default branch's dev shell is - # also cached, along with moq-relay and moq on x86 Linux, with a 30-day - # expiry. Pin the flake ref to a recent tag for a retained package: - # nix run github:moq-dev/moq/moq-relay-v0.12.4#moq-relay --accept-flake-config + # nix.moq.dev caches the default branch's dev shell, along with moq-relay + # and moq on x86 Linux, with a 30-day expiry: + # nix run github:moq-dev/moq#moq-relay --accept-flake-config + # Release Nix retains tagged packages permanently once a release cut + # carries that workflow from main; earlier tags still build from source. # # --accept-flake-config opts into the nixConfig below for one command. To # trust the cache permanently instead, add these settings to nix.conf.