diff --git a/.github/actions/rust-cache/action.yml b/.github/actions/rust-cache/action.yml index 216c6dd30b2..befa5eb23bd 100644 --- a/.github/actions/rust-cache/action.yml +++ b/.github/actions/rust-cache/action.yml @@ -1,5 +1,5 @@ name: Rust cache -description: Install mbx and its Cargo shim, using local storage in runner VMs and R2 on hosted runners. +description: Install mbx and its Cargo shim, using local storage on self-hosted runners and R2 on hosted runners. inputs: access-key-id: @@ -88,11 +88,13 @@ runs: remote-url: s3://mbx-moq-dev namespace: ${{ runner.os }}-${{ runner.arch }} - # A runner VM keeps managed targets and learned incremental state across - # jobs, and Cargo trusts whatever outputs it finds there. State another mbx - # version left is not trusted, since 1.26's was corrupt, so it goes before - # the first build on a new version. The shared store stays, so that build - # restores what it needs rather than compiling it. + # A self-hosted runner keeps managed targets and learned incremental state + # across jobs, and Cargo trusts whatever outputs it finds there. State + # another mbx version left is not trusted, since 1.26's was corrupt, so it + # goes before the first build on a new version. The shared store stays, so + # that build restores what it needs rather than compiling it. Runners on one + # host may share the home but never a workspace, so each workspace has its + # own stamp. - name: Drop another mbx version's private state if: runner.environment == 'self-hosted' shell: bash @@ -100,7 +102,7 @@ runs: VERSION: ${{ steps.mbx.outputs.mbx-version }} run: | : "${VERSION:?the mbx action reported no version}" - stamp="$HOME/.cache/moq-mbx-version" + stamp="$HOME/.cache/moq-mbx-versions$RUNNER_WORKSPACE" if [ "$(cat "$stamp" 2>/dev/null)" != "$VERSION" ]; then mbx clean --under "$RUNNER_WORKSPACE" mkdir -p "${stamp%/*}" diff --git a/.github/workflows/check.yml b/.github/workflows/check.yml index 5a6f7885604..4ebaf334566 100644 --- a/.github/workflows/check.yml +++ b/.github/workflows/check.yml @@ -75,8 +75,9 @@ jobs: access-key-id: ${{ secrets.MBX_R2_ACCESS_KEY_ID }} secret-access-key: ${{ secrets.MBX_R2_SECRET_ACCESS_KEY }} - # The pre-setup query and execution use the same impact map. A runner VM - # keeps target/, so drop the previous job's build timings first. + # The pre-setup query and execution use the same impact map. A + # self-hosted runner keeps target/, so drop the previous job's build + # timings first. - name: Check if: steps.scope.outputs.needed == 'true' run: | diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 67869442f67..2d81f07de27 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -53,15 +53,18 @@ Decisions (maintainer, 2026-10-13): - Self-hosted CI is dropped: hosted runners plus the R2 caches replace the spare-desktop runner, its cache server, and the cache release it waited on. Reopened 2026-10-19, conditionally, to compare runners once the pipeline - was optimized. Decided 2026-10-22 (maintainer): the queue runner is the - persistent `nix/runner` VM (no host mounts or tailnet, firewalled to public - egress), in use since 2026-10-20. Over 32 one-pass entries it took a median - 5.5 minutes per merged PR (p90 11.9, at most 24.2) against 21 minutes on a - hosted runner, with no per-minute bill. Blacksmith is not pursued. - Open (found 2026-10-23): the runner deployed on that host is a - systemd-nspawn container (`container@moq-ci`) with `/dev/kvm` and a host - cache directory bound in, not the VM this decision and `nix/runner/README.md` - describe. Confirm which is intended before changing either. + was optimized. Decided 2026-10-22 (maintainer): the queue runs on a + persistent self-hosted runner, in use since 2026-10-20. Over 32 one-pass + entries it took a median 5.5 minutes per merged PR (p90 11.9, at most 24.2) + against 21 minutes on a hosted runner, with no per-minute bill. Blacksmith + is not pursued. +- Decided 2026-10-25 (maintainer), superseding the 2026-10-22 choice of the + `nix/runner` VM: no VM. The queue runner is the deployed systemd-nspawn + container `container@moq-ci`, and new queue runners go into it. Reason: the + CI host should run mostly sandboxed through kernel primitives (namespaces, + cgroups and slices, seccomp), which the container already uses, so a VM is + not needed for isolation. The `nix/runner` VM module is not the deployment + target. Decided 2026-10-23 (maintainer), to speed up the serial queue: structural changes first (concurrent entries, parallel modules, one test build), then a diff --git a/quest/m0/ci/queue-concurrency.md b/quest/m0/ci/queue-concurrency.md index 9ed68f9f169..4d926161ec0 100644 --- a/quest/m0/ci/queue-concurrency.md +++ b/quest/m0/ci/queue-concurrency.md @@ -1,4 +1,4 @@ -# [M] Build queue entries concurrently +# [S] Build queue entries concurrently ## Goal @@ -15,16 +15,48 @@ Measured 2026-10-23 over 32 merge-queue runs: `max_entries_to_build: 1` with one `moq-queue` runner makes the queue fully serial, about 7 merged PRs per hour, with a median entry of 135 s (p90 516 s) and 3 to 12 s of runner wait. -- Add queue runner capacity (another instance or container) and raise - `max_entries_to_build` to match. The host also runs the `moq-pro` runner - under the same `ci.slice` quota and a desktop; split CPU and memory so - concurrent entries do not slow each other past the gain. -- Concurrent entries share the local mbx store; confirm that is safe, or give - each its own. Each job sweeps the store before it builds - (`.github/actions/rust-cache`), which can evict generated source trees that - a concurrent entry's next build reads. -- Measure merged PRs per hour and per-entry wall time before and after. -- Queue entries skip `quest gate`: `sh/local.sh` execs directly under GitHub - Actions, so the runner quota, not admission, has to keep entries apart. +Decided 2026-10-25 (maintainer): no VM. The second runner goes into the +deployed `container@moq-ci` systemd-nspawn container, sandboxed by kernel +primitives (see the [CI epic](/quest/m0/ci/README.md)), not into the +`nix/runner` VM module. The container is defined in the host's own flake, not +in this repository, so registering the runner is a host change. + +Each runner keeps its own work directory, since the nixpkgs runner service +empties it on start. The self-hosted cache step keeps its mbx version stamp per +workspace, so a runner sharing a home with another still drops its own +workspace's state after an mbx version change. + +Decided 2026-10-25 (maintainer): ✅ one home under one static user for both +runners. Concurrent entries share mbx's store, wait on a compilation the other +entry is running instead of repeating it, and draw on one machine-wide compile +budget, which mbx documents as supported +([how it works](https://github.com/jdx/mr-boxington/blob/v1.25.0/docs/how-it-works.md#machine-wide-scheduling)). +nixpkgs gives each runner its own `DynamicUser` by default, so both need the +same `user` and `HOME`. Rejected: a home per runner (two cold caches, double +the disk, two mbx schedulers each budgeting the whole container's CPUs) and a +second container (those costs plus another container). A NixOS container +builds through the host's Nix daemon by default, so the Nix store is shared +either way. + +Remaining (maintainer, on the host): + +- Register a second runner in `container@moq-ci` with its own name and work + directory, under the same static user and `HOME` as the first, with a fresh + registration token. +- Size the container's share of `ci.slice`, which it shares with the `moq-pro` + runner and a desktop, so concurrent entries do not slow each other past the + gain. Queue entries skip `quest gate` (`sh/local.sh` execs directly under + GitHub Actions), so the cgroup quota, not admission, keeps them apart. +- Keep `NEXTEST_TEST_THREADS=24` per entry (48 workers across two entries; + mbx budgets compiles, not tests) and watch the first day for timing flakes + rather than guess a lower number. +- Make the pre-build mbx sweep safe for a concurrent entry first. With + [moq-dev/moq#6709](https://github.com/moq-dev/moq/pull/6709), each job runs + `mbx gc` before it builds (`.github/actions/rust-cache`), and in a shared + home that sweep can evict generated source trees another entry's next + build reads. +- Raise the main ruleset's `max_entries_to_build` from 1 to 2. +- Measure merged PRs per hour and per-entry wall time for a day against the + baseline before going past 2. Public API: none. Wire: none. diff --git a/quest/m2/audit-ci.md b/quest/m2/audit-ci.md index 70dc21d6c3e..de8d7da94f7 100644 --- a/quest/m2/audit-ci.md +++ b/quest/m2/audit-ci.md @@ -140,7 +140,7 @@ verdicts: ran). The longest 2026-10-22 jobs: Audio quality 62 minutes, Loom 60 (cancelled), binding benchmark 52, Rust bench targets 48, relay profiles 45, Rust benchmark tests 44, the gate suite 40. -- The merge queue runs one `just ci` job on the persistent `nix/runner` VM: +- The merge queue runs one `just ci` job on the self-hosted `moq-queue` runner: median 5.5 minutes, p90 11.9, at most 24.2 over 32 entries (m0, 2026-10-21). Its job summary lists seconds per module, and the `cargo-timings` artifact is the per-crate drill-down. diff --git a/quest/m2/gpu-ci.md b/quest/m2/gpu-ci.md index 1b29133d2b8..339782be695 100644 --- a/quest/m2/gpu-ci.md +++ b/quest/m2/gpu-ci.md @@ -57,6 +57,11 @@ skipping inside the Nix shell. guest, and keep the runner registered only to this repository with approval required for every external contributor's workflow, as the README requires. +- Open (2026-10-25): the [CI epic](/quest/m0/ci/README.md) dropped the VM + for the CI host in favor of kernel sandboxing, which conflicts with VFIO + passthrough. Recommendation: a container runner with the NVIDIA device + nodes and the host driver's libraries bound in. Confirm with the + maintainer before starting. Public API: none. Wire: none.