From 238c965c5f1ad7ddf461a446a23d5be33fe88374 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 22:39:40 -0700 Subject: [PATCH 1/5] chore(quest): claim quest/m0/ci/queue-concurrency.md Quest-Claim: 1f53ab3dc41eaae9 From 5e876363e2de4f4fa647df7f7326d2ff14d59f2f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 22:55:29 -0700 Subject: [PATCH 2/5] feat(runner): run several queue runners in one guest `runner.count` registers that many runners in the runner VM. They share the home, so concurrent merge-queue entries share mbx's store, its dedup of in-flight compilations, and its compiler budget, while each keeps its own work directory (a runner empties its work directory on start). The first runner keeps its unit, name, and work directory, so an existing registration survives. The self-hosted cache step keyed its mbx version stamp on the shared home, so a second runner would skip dropping its own workspace's state after a version change. Key the stamp per workspace. The quest now holds what remains: pick the VM or the deployed container, deploy two runners, raise max_entries_to_build, and measure. Co-Authored-By: Claude Opus 5.5 --- .github/actions/rust-cache/action.yml | 5 +- nix/modules/github-runner-vm.nix | 5 + nix/runner/README.md | 11 +- nix/runner/checks.nix | 1 + nix/runner/guest.nix | 149 +++++++++++++++----------- nix/runner/image-test.nix | 2 +- nix/runner/image.nix | 5 +- nix/runner/test.nix | 33 +++++- quest/m0/ci/queue-concurrency.md | 25 +++-- 9 files changed, 159 insertions(+), 77 deletions(-) diff --git a/.github/actions/rust-cache/action.yml b/.github/actions/rust-cache/action.yml index a48c52f9288..6b65b2663b4 100644 --- a/.github/actions/rust-cache/action.yml +++ b/.github/actions/rust-cache/action.yml @@ -92,7 +92,8 @@ runs: # jobs, and Cargo trusts whatever outputs it finds there. State another mbx # version left is not trusted, since 1.26's was corrupt, so it goes before # the first build on a new version. The shared store stays, so that build - # restores what it needs rather than compiling it. + # restores what it needs rather than compiling it. Runners in one guest + # share the home but not their workspaces, so each workspace has a stamp. - name: Drop another mbx version's private state if: runner.environment == 'self-hosted' shell: bash @@ -100,7 +101,7 @@ runs: VERSION: ${{ steps.mbx.outputs.mbx-version }} run: | : "${VERSION:?the mbx action reported no version}" - stamp="$HOME/.cache/moq-mbx-version" + stamp="$HOME/.cache/moq-mbx-versions$RUNNER_WORKSPACE" if [ "$(cat "$stamp" 2>/dev/null)" != "$VERSION" ]; then mbx clean --under "$RUNNER_WORKSPACE" mkdir -p "${stamp%/*}" diff --git a/nix/modules/github-runner-vm.nix b/nix/modules/github-runner-vm.nix index 9b9e27178bb..faaf2098296 100644 --- a/nix/modules/github-runner-vm.nix +++ b/nix/modules/github-runner-vm.nix @@ -255,6 +255,11 @@ in default = [ ]; description = "Additional labels advertised to GitHub."; }; + count = lib.mkOption { + type = lib.types.ints.positive; + default = 1; + description = "Runners in the guest, each running one job at a time on shared caches."; + }; }; network.uplink = lib.mkOption { type = lib.types.strMatching "[a-zA-Z0-9_.:-]{1,15}"; diff --git a/nix/runner/README.md b/nix/runner/README.md index 57df71c15dc..3b6a2b8679f 100644 --- a/nix/runner/README.md +++ b/nix/runner/README.md @@ -75,6 +75,14 @@ Nix store. Keep the expired token file after registration. The guest retains its own long-lived credential; wiping its state requires a fresh registration token. Restart the host VM service after replacing the token. +Set `runner.count` to run several jobs at once in one guest. The runners share +the home, so concurrent jobs share mbx's store and compiler budget, and a +compilation one job is running is restored by the other instead of repeated. +Each runner has its own work directory. The first keeps `runner.name`; the others +append `-2`, `-3`, and so on. Raising the count registers new runners, so install +a fresh registration token and restart the host VM service; the existing runner +re-registers under its own name. Size `resources` for the concurrent jobs. + The host service and disk are named by the instance attribute: the example uses `queue-ci.service` and `/var/lib/queue-ci/root.raw`. Its disk is created once. Changing the configured image does not replace existing state, and changing `resources.disk` does not @@ -147,7 +155,8 @@ its build host must already support rootless user namespaces. startup without carrier, public egress for multiple guests, private-network and sibling-guest isolation including spoofed established replies, compatible host NAT settings, the runner's locked-memory budget, symlink-free home and work directories -with state migrated from the former DynamicUser layout, free page reporting and +with state migrated from the former DynamicUser layout, a second runner sharing the +home but not the work directory, free page reporting and idle cache drops, killing a job that fills memory before it thrashes the disk, and Nix/mbx persistence across runner and guest restarts. It also migrates a disposable ext4 disk, boots the resulting XFS image through diff --git a/nix/runner/checks.nix b/nix/runner/checks.nix index b057fb15c89..9f3a021fa12 100644 --- a/nix/runner/checks.nix +++ b/nix/runner/checks.nix @@ -5,6 +5,7 @@ let url = "https://github.com/example/queue"; name = "test-queue"; labels = [ "queue" ]; + count = 2; }; network = { hostAddress = "10.231.0.1"; diff --git a/nix/runner/guest.nix b/nix/runner/guest.nix index 1fe0bc8d774..23c2b8eb340 100644 --- a/nix/runner/guest.nix +++ b/nix/runner/guest.nix @@ -11,6 +11,23 @@ ... }: +let + # Each runner takes one job at a time. They share the home, so concurrent jobs + # share mbx's store and compiler budget, and a compilation one job is running + # is restored by the other instead of repeated. The first runner keeps the + # original unit, name, and work directory, so its registration survives. + runners = map ( + index: + let + suffix = lib.optionalString (index > 1) "-${toString index}"; + in + { + unit = "runner${suffix}"; + name = "${runner.name}${suffix}"; + work = "github-runner-work${suffix}"; + } + ) (lib.range 1 runner.count); +in { imports = [ (modulesPath + "/profiles/qemu-guest.nix") @@ -116,17 +133,23 @@ }; # Only a short-lived registration token enters the VM. The host's deployment - # key and private services are never mounted in the guest. - systemd.services.runner-token = { - wantedBy = [ "multi-user.target" ]; - before = [ "github-runner-runner.service" ]; - serviceConfig.Type = "oneshot"; - script = '' - install -D -m 0600 \ - /sys/firmware/qemu_fw_cfg/by_name/opt/io.systemd.credentials/runner-token/raw \ - /var/lib/secrets/github-runner.token - ''; - }; + # key and private services are never mounted in the guest. Every runner + # registers with the same token. + systemd.services.runner-token = + let + units = map (r: "github-runner-${r.unit}.service") runners; + in + { + wantedBy = [ "multi-user.target" ]; + requiredBy = units; + before = units; + serviceConfig.Type = "oneshot"; + script = '' + install -D -m 0600 \ + /sys/firmware/qemu_fw_cfg/by_name/opt/io.systemd.credentials/runner-token/raw \ + /var/lib/secrets/github-runner.token + ''; + }; # A static user, unlike DynamicUser, gets real cache directories instead of # symlinks into /var/cache/private. Builds resolve `..` lexically against @@ -141,57 +164,59 @@ }; users.groups.github-runner = { }; - services.github-runners.runner = { - enable = true; - user = "github-runner"; - group = "github-runner"; - url = runner.url; - name = runner.name; - extraLabels = runner.labels; - tokenFile = "/var/lib/secrets/github-runner.token"; - tokenType = "registration"; - replace = true; - workDir = "/var/cache/github-runner-work"; - # Tools a hosted runner's base image provides and jobs call directly. - extraPackages = with pkgs; [ - curl - just - openssh - procps - unzip - which - ]; - extraEnvironment = { - HOME = config.users.users.github-runner.home; - MBX_CACHE_DIR = "/var/cache/github-runner-home/.cache/mbx"; - CARGO_BUILD_JOBS = toString resources.cores; - }; - serviceOverrides = { - CacheDirectory = [ - "github-runner-home" - "github-runner-work" - ]; - CacheDirectoryMode = "0700"; - # The sandbox mounts cache directories noexec, but jobs execute checkout - # scripts, build outputs, and tools installed in their persistent home. - ExecPaths = [ - "/var/cache/github-runner-home" - "/var/cache/github-runner-work" - ]; - BindPaths = lib.mkForce [ ]; - ProtectHome = true; - RestrictNamespaces = false; - PrivateUsers = false; - SystemCallFilter = lib.mkForce [ ]; - TasksMax = 16384; - # Concurrent io_uring tests share the runner service's locked-memory budget. - LimitMEMLOCK = "64M"; - }; - }; - systemd.services.github-runner-runner = { - requires = [ "runner-token.service" ]; - after = [ "runner-token.service" ]; - }; + services.github-runners = lib.listToAttrs ( + map (r: { + name = r.unit; + value = { + enable = true; + user = "github-runner"; + group = "github-runner"; + url = runner.url; + name = r.name; + extraLabels = runner.labels; + tokenFile = "/var/lib/secrets/github-runner.token"; + tokenType = "registration"; + replace = true; + # A runner empties its work directory whenever it starts. + workDir = "/var/cache/${r.work}"; + # Tools a hosted runner's base image provides and jobs call directly. + extraPackages = with pkgs; [ + curl + just + openssh + procps + unzip + which + ]; + extraEnvironment = { + HOME = config.users.users.github-runner.home; + MBX_CACHE_DIR = "/var/cache/github-runner-home/.cache/mbx"; + CARGO_BUILD_JOBS = toString resources.cores; + }; + serviceOverrides = { + CacheDirectory = [ + "github-runner-home" + r.work + ]; + CacheDirectoryMode = "0700"; + # The sandbox mounts cache directories noexec, but jobs execute checkout + # scripts, build outputs, and tools installed in their persistent home. + ExecPaths = [ + "/var/cache/github-runner-home" + "/var/cache/${r.work}" + ]; + BindPaths = lib.mkForce [ ]; + ProtectHome = true; + RestrictNamespaces = false; + PrivateUsers = false; + SystemCallFilter = lib.mkForce [ ]; + TasksMax = 16384; + # Concurrent io_uring tests share the runner service's locked-memory budget. + LimitMEMLOCK = "64M"; + }; + }; + }) runners + ); # Free page reporting hands back only free pages. The disk bypasses the host's # cache, so the guest's page cache keeps every file a job touched resident on the diff --git a/nix/runner/image-test.nix b/nix/runner/image-test.nix index 166030fbf14..a5d5ed14bdb 100644 --- a/nix/runner/image-test.nix +++ b/nix/runner/image-test.nix @@ -6,7 +6,7 @@ pkgs.testers.runNixOSTest { imports = [ guestModule ]; environment.systemPackages = [ pkgs.python3 ]; system.switch.enable = true; - services.github-runners.runner.enable = lib.mkForce false; + services.github-runners = lib.mkForce { }; systemd.services.runner-token.enable = false; virtualisation = { directBoot.enable = false; diff --git a/nix/runner/image.nix b/nix/runner/image.nix index 576322e29c5..0533e1e34a4 100644 --- a/nix/runner/image.nix +++ b/nix/runner/image.nix @@ -1,5 +1,6 @@ { config, + lib, pkgs, modulesPath, ... @@ -53,8 +54,8 @@ wantedBy = [ "multi-user.target" ]; before = [ "nix-daemon.service" - "github-runner-runner.service" - ]; + ] + ++ map (runner: "github-runner-${runner}.service") (lib.attrNames config.services.github-runners); unitConfig.ConditionPathExists = "/nix-path-registration"; serviceConfig.Type = "oneshot"; environment.NIX_REMOTE = "local"; diff --git a/nix/runner/test.nix b/nix/runner/test.nix index cd74e6bfd53..0292e9578fd 100644 --- a/nix/runner/test.nix +++ b/nix/runner/test.nix @@ -1,6 +1,10 @@ { pkgs, guestModule }: let + runners = [ + "github-runner-runner.service" + "github-runner-runner-2.service" + ]; hostModule = { lib, ... }: { imports = [ ../modules/github-runner-vm.nix ]; services.github-runner-vms = @@ -104,8 +108,8 @@ pkgs.testers.runNixOSTest { "-device virtio-balloon-pci,free-page-reporting=on" ]; systemd.services.cache-seed-fixture = { - requiredBy = [ "github-runner-runner.service" ]; - before = [ "github-runner-runner.service" ]; + requiredBy = runners; + before = runners; unitConfig.ConditionPathExists = "!/var/lib/cache-seeded"; serviceConfig.Type = "oneshot"; # Lay out the cache and registration the way DynamicUser left them on @@ -155,6 +159,26 @@ pkgs.testers.runNixOSTest { '' ); }; + systemd.services.github-runner-runner-2.serviceConfig = { + ExecStartPre = lib.mkForce [ + "+${pkgs.coreutils}/bin/touch /var/lib/github-runner/runner-2/.current-token" + ]; + ExecStart = lib.mkForce ( + pkgs.writeShellScript "second-runner-probe" '' + set -euo pipefail + # The first runner empties its own work directory whenever it starts, + # so a second runner must not share it. + test "$PWD" = /var/cache/github-runner-work-2 + printf '#!${pkgs.runtimeShell}\necho executable\n' > exec-probe + chmod +x exec-probe + test "$(./exec-probe)" = executable + test "$(cat "$HOME/seeded")" = seeded + mkdir -p "$MBX_CACHE_DIR" + echo "$PWD" > "$MBX_CACHE_DIR/second" + exec ${pkgs.coreutils}/bin/sleep infinity + '' + ); + }; environment.systemPackages = [ pkgs.curl ]; }; other = { @@ -258,6 +282,11 @@ pkgs.testers.runNixOSTest { guest.succeed("test ! -L /var/lib/github-runner/runner") guest.succeed("test $(cat /var/lib/github-runner/runner/.runner) = registered") + with subtest("a second runner shares the home and mbx store but not the work directory"): + guest.wait_for_unit("github-runner-runner-2.service") + guest.wait_for_file("/var/cache/github-runner-home/.cache/mbx/second") + guest.succeed("test $(cat /var/cache/github-runner-home/.cache/mbx/second) = /var/cache/github-runner-work-2") + with subtest("the uplink stays IPv4-only"): guest.succeed("test $(cat /proc/sys/net/ipv6/conf/eth0/disable_ipv6) = 1") diff --git a/quest/m0/ci/queue-concurrency.md b/quest/m0/ci/queue-concurrency.md index 8c337c27973..050bf0603d6 100644 --- a/quest/m0/ci/queue-concurrency.md +++ b/quest/m0/ci/queue-concurrency.md @@ -1,4 +1,4 @@ -# [M] Build queue entries concurrently +# [S] Build queue entries concurrently ## Goal @@ -15,12 +15,23 @@ Measured 2026-10-23 over 32 merge-queue runs: `max_entries_to_build: 1` with one `moq-queue` runner makes the queue fully serial, about 7 merged PRs per hour, with a median entry of 135 s (p90 516 s) and 3 to 12 s of runner wait. -- Add queue runner capacity (another instance or container) and raise - `max_entries_to_build` to match. The host also runs the `moq-pro` runner - under the same `ci.slice` quota and a desktop; split CPU and memory so - concurrent entries do not slow each other past the gain. -- Concurrent entries share the local mbx store; confirm that is safe, or give - each its own. +The runner VM module's `runner.count` registers several runners in one guest. +They share the home, so concurrent entries share mbx's store, its dedup of +in-flight compilations, and its compiler CPU and memory budget; each has its own +work directory. mbx documents concurrent builds on one store as supported, and +the self-hosted cache step keeps a version stamp per workspace. + +Remaining: + +- Confirm which runner is intended: the documented `nix/runner` VM, or the + `container@moq-ci` systemd-nspawn container deployed on the host (see the + [CI epic](/quest/m0/ci/README.md)). Give whichever stays two runners sharing + its cache. +- Deploy with `runner.count = 2`, a fresh registration token, and a VM restart. + The host also runs the `moq-pro` runner under the same `ci.slice` quota and a + desktop; size the guest's `resources` so concurrent entries do not slow each + other past the gain. Each entry runs 24 nextest workers. +- Raise the main ruleset's `max_entries_to_build` to the runner count. - Measure merged PRs per hour and per-entry wall time before and after. Public API: none. Wire: none. From bc347219d4e2222dce8cc71160f0c28b9ddd722d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 25 Oct 2026 08:45:47 -0700 Subject: [PATCH 3/5] ci: plan queue runners in the moq-ci container, not the runner VM The maintainer dropped the VM for the CI host: it is sandboxed through kernel primitives, as the deployed container@moq-ci already is. Revert the runner VM module's runner.count and record the decision in the CI epic, superseding the 2026-10-22 VM choice. Keep the per-workspace mbx version stamp, so a second runner sharing a home still drops its own workspace's state after an mbx bump, and stop calling the self-hosted runner a VM in the cache comments. The quest now lists the host steps: register a second runner in the container, size its slice share, raise max_entries_to_build to 2, and measure a day before going further. How the runners share a home stays open with a recommendation. Co-Authored-By: Claude Opus 5.5 --- .github/actions/rust-cache/action.yml | 15 +-- .github/workflows/check.yml | 5 +- nix/modules/github-runner-vm.nix | 5 - nix/runner/README.md | 11 +- nix/runner/checks.nix | 1 - nix/runner/guest.nix | 149 +++++++++++--------------- nix/runner/image-test.nix | 2 +- nix/runner/image.nix | 5 +- nix/runner/test.nix | 33 +----- quest/m0/ci/README.md | 21 ++-- quest/m0/ci/queue-concurrency.md | 61 +++++++---- quest/m2/audit-ci.md | 2 +- quest/m2/gpu-ci.md | 5 + 13 files changed, 138 insertions(+), 177 deletions(-) diff --git a/.github/actions/rust-cache/action.yml b/.github/actions/rust-cache/action.yml index 6b65b2663b4..271a99c3231 100644 --- a/.github/actions/rust-cache/action.yml +++ b/.github/actions/rust-cache/action.yml @@ -1,5 +1,5 @@ name: Rust cache -description: Install mbx and its Cargo shim, using local storage in runner VMs and R2 on hosted runners. +description: Install mbx and its Cargo shim, using local storage on self-hosted runners and R2 on hosted runners. inputs: access-key-id: @@ -88,12 +88,13 @@ runs: remote-url: s3://mbx-moq-dev namespace: ${{ runner.os }}-${{ runner.arch }} - # A runner VM keeps managed targets and learned incremental state across - # jobs, and Cargo trusts whatever outputs it finds there. State another mbx - # version left is not trusted, since 1.26's was corrupt, so it goes before - # the first build on a new version. The shared store stays, so that build - # restores what it needs rather than compiling it. Runners in one guest - # share the home but not their workspaces, so each workspace has a stamp. + # A self-hosted runner keeps managed targets and learned incremental state + # across jobs, and Cargo trusts whatever outputs it finds there. State + # another mbx version left is not trusted, since 1.26's was corrupt, so it + # goes before the first build on a new version. The shared store stays, so + # that build restores what it needs rather than compiling it. Runners on one + # host may share the home but never a workspace, so each workspace has its + # own stamp. - name: Drop another mbx version's private state if: runner.environment == 'self-hosted' shell: bash diff --git a/.github/workflows/check.yml b/.github/workflows/check.yml index 5a6f7885604..4ebaf334566 100644 --- a/.github/workflows/check.yml +++ b/.github/workflows/check.yml @@ -75,8 +75,9 @@ jobs: access-key-id: ${{ secrets.MBX_R2_ACCESS_KEY_ID }} secret-access-key: ${{ secrets.MBX_R2_SECRET_ACCESS_KEY }} - # The pre-setup query and execution use the same impact map. A runner VM - # keeps target/, so drop the previous job's build timings first. + # The pre-setup query and execution use the same impact map. A + # self-hosted runner keeps target/, so drop the previous job's build + # timings first. - name: Check if: steps.scope.outputs.needed == 'true' run: | diff --git a/nix/modules/github-runner-vm.nix b/nix/modules/github-runner-vm.nix index faaf2098296..9b9e27178bb 100644 --- a/nix/modules/github-runner-vm.nix +++ b/nix/modules/github-runner-vm.nix @@ -255,11 +255,6 @@ in default = [ ]; description = "Additional labels advertised to GitHub."; }; - count = lib.mkOption { - type = lib.types.ints.positive; - default = 1; - description = "Runners in the guest, each running one job at a time on shared caches."; - }; }; network.uplink = lib.mkOption { type = lib.types.strMatching "[a-zA-Z0-9_.:-]{1,15}"; diff --git a/nix/runner/README.md b/nix/runner/README.md index 3b6a2b8679f..57df71c15dc 100644 --- a/nix/runner/README.md +++ b/nix/runner/README.md @@ -75,14 +75,6 @@ Nix store. Keep the expired token file after registration. The guest retains its own long-lived credential; wiping its state requires a fresh registration token. Restart the host VM service after replacing the token. -Set `runner.count` to run several jobs at once in one guest. The runners share -the home, so concurrent jobs share mbx's store and compiler budget, and a -compilation one job is running is restored by the other instead of repeated. -Each runner has its own work directory. The first keeps `runner.name`; the others -append `-2`, `-3`, and so on. Raising the count registers new runners, so install -a fresh registration token and restart the host VM service; the existing runner -re-registers under its own name. Size `resources` for the concurrent jobs. - The host service and disk are named by the instance attribute: the example uses `queue-ci.service` and `/var/lib/queue-ci/root.raw`. Its disk is created once. Changing the configured image does not replace existing state, and changing `resources.disk` does not @@ -155,8 +147,7 @@ its build host must already support rootless user namespaces. startup without carrier, public egress for multiple guests, private-network and sibling-guest isolation including spoofed established replies, compatible host NAT settings, the runner's locked-memory budget, symlink-free home and work directories -with state migrated from the former DynamicUser layout, a second runner sharing the -home but not the work directory, free page reporting and +with state migrated from the former DynamicUser layout, free page reporting and idle cache drops, killing a job that fills memory before it thrashes the disk, and Nix/mbx persistence across runner and guest restarts. It also migrates a disposable ext4 disk, boots the resulting XFS image through diff --git a/nix/runner/checks.nix b/nix/runner/checks.nix index 9f3a021fa12..b057fb15c89 100644 --- a/nix/runner/checks.nix +++ b/nix/runner/checks.nix @@ -5,7 +5,6 @@ let url = "https://github.com/example/queue"; name = "test-queue"; labels = [ "queue" ]; - count = 2; }; network = { hostAddress = "10.231.0.1"; diff --git a/nix/runner/guest.nix b/nix/runner/guest.nix index 23c2b8eb340..1fe0bc8d774 100644 --- a/nix/runner/guest.nix +++ b/nix/runner/guest.nix @@ -11,23 +11,6 @@ ... }: -let - # Each runner takes one job at a time. They share the home, so concurrent jobs - # share mbx's store and compiler budget, and a compilation one job is running - # is restored by the other instead of repeated. The first runner keeps the - # original unit, name, and work directory, so its registration survives. - runners = map ( - index: - let - suffix = lib.optionalString (index > 1) "-${toString index}"; - in - { - unit = "runner${suffix}"; - name = "${runner.name}${suffix}"; - work = "github-runner-work${suffix}"; - } - ) (lib.range 1 runner.count); -in { imports = [ (modulesPath + "/profiles/qemu-guest.nix") @@ -133,23 +116,17 @@ in }; # Only a short-lived registration token enters the VM. The host's deployment - # key and private services are never mounted in the guest. Every runner - # registers with the same token. - systemd.services.runner-token = - let - units = map (r: "github-runner-${r.unit}.service") runners; - in - { - wantedBy = [ "multi-user.target" ]; - requiredBy = units; - before = units; - serviceConfig.Type = "oneshot"; - script = '' - install -D -m 0600 \ - /sys/firmware/qemu_fw_cfg/by_name/opt/io.systemd.credentials/runner-token/raw \ - /var/lib/secrets/github-runner.token - ''; - }; + # key and private services are never mounted in the guest. + systemd.services.runner-token = { + wantedBy = [ "multi-user.target" ]; + before = [ "github-runner-runner.service" ]; + serviceConfig.Type = "oneshot"; + script = '' + install -D -m 0600 \ + /sys/firmware/qemu_fw_cfg/by_name/opt/io.systemd.credentials/runner-token/raw \ + /var/lib/secrets/github-runner.token + ''; + }; # A static user, unlike DynamicUser, gets real cache directories instead of # symlinks into /var/cache/private. Builds resolve `..` lexically against @@ -164,59 +141,57 @@ in }; users.groups.github-runner = { }; - services.github-runners = lib.listToAttrs ( - map (r: { - name = r.unit; - value = { - enable = true; - user = "github-runner"; - group = "github-runner"; - url = runner.url; - name = r.name; - extraLabels = runner.labels; - tokenFile = "/var/lib/secrets/github-runner.token"; - tokenType = "registration"; - replace = true; - # A runner empties its work directory whenever it starts. - workDir = "/var/cache/${r.work}"; - # Tools a hosted runner's base image provides and jobs call directly. - extraPackages = with pkgs; [ - curl - just - openssh - procps - unzip - which - ]; - extraEnvironment = { - HOME = config.users.users.github-runner.home; - MBX_CACHE_DIR = "/var/cache/github-runner-home/.cache/mbx"; - CARGO_BUILD_JOBS = toString resources.cores; - }; - serviceOverrides = { - CacheDirectory = [ - "github-runner-home" - r.work - ]; - CacheDirectoryMode = "0700"; - # The sandbox mounts cache directories noexec, but jobs execute checkout - # scripts, build outputs, and tools installed in their persistent home. - ExecPaths = [ - "/var/cache/github-runner-home" - "/var/cache/${r.work}" - ]; - BindPaths = lib.mkForce [ ]; - ProtectHome = true; - RestrictNamespaces = false; - PrivateUsers = false; - SystemCallFilter = lib.mkForce [ ]; - TasksMax = 16384; - # Concurrent io_uring tests share the runner service's locked-memory budget. - LimitMEMLOCK = "64M"; - }; - }; - }) runners - ); + services.github-runners.runner = { + enable = true; + user = "github-runner"; + group = "github-runner"; + url = runner.url; + name = runner.name; + extraLabels = runner.labels; + tokenFile = "/var/lib/secrets/github-runner.token"; + tokenType = "registration"; + replace = true; + workDir = "/var/cache/github-runner-work"; + # Tools a hosted runner's base image provides and jobs call directly. + extraPackages = with pkgs; [ + curl + just + openssh + procps + unzip + which + ]; + extraEnvironment = { + HOME = config.users.users.github-runner.home; + MBX_CACHE_DIR = "/var/cache/github-runner-home/.cache/mbx"; + CARGO_BUILD_JOBS = toString resources.cores; + }; + serviceOverrides = { + CacheDirectory = [ + "github-runner-home" + "github-runner-work" + ]; + CacheDirectoryMode = "0700"; + # The sandbox mounts cache directories noexec, but jobs execute checkout + # scripts, build outputs, and tools installed in their persistent home. + ExecPaths = [ + "/var/cache/github-runner-home" + "/var/cache/github-runner-work" + ]; + BindPaths = lib.mkForce [ ]; + ProtectHome = true; + RestrictNamespaces = false; + PrivateUsers = false; + SystemCallFilter = lib.mkForce [ ]; + TasksMax = 16384; + # Concurrent io_uring tests share the runner service's locked-memory budget. + LimitMEMLOCK = "64M"; + }; + }; + systemd.services.github-runner-runner = { + requires = [ "runner-token.service" ]; + after = [ "runner-token.service" ]; + }; # Free page reporting hands back only free pages. The disk bypasses the host's # cache, so the guest's page cache keeps every file a job touched resident on the diff --git a/nix/runner/image-test.nix b/nix/runner/image-test.nix index a5d5ed14bdb..166030fbf14 100644 --- a/nix/runner/image-test.nix +++ b/nix/runner/image-test.nix @@ -6,7 +6,7 @@ pkgs.testers.runNixOSTest { imports = [ guestModule ]; environment.systemPackages = [ pkgs.python3 ]; system.switch.enable = true; - services.github-runners = lib.mkForce { }; + services.github-runners.runner.enable = lib.mkForce false; systemd.services.runner-token.enable = false; virtualisation = { directBoot.enable = false; diff --git a/nix/runner/image.nix b/nix/runner/image.nix index 0533e1e34a4..576322e29c5 100644 --- a/nix/runner/image.nix +++ b/nix/runner/image.nix @@ -1,6 +1,5 @@ { config, - lib, pkgs, modulesPath, ... @@ -54,8 +53,8 @@ wantedBy = [ "multi-user.target" ]; before = [ "nix-daemon.service" - ] - ++ map (runner: "github-runner-${runner}.service") (lib.attrNames config.services.github-runners); + "github-runner-runner.service" + ]; unitConfig.ConditionPathExists = "/nix-path-registration"; serviceConfig.Type = "oneshot"; environment.NIX_REMOTE = "local"; diff --git a/nix/runner/test.nix b/nix/runner/test.nix index 0292e9578fd..cd74e6bfd53 100644 --- a/nix/runner/test.nix +++ b/nix/runner/test.nix @@ -1,10 +1,6 @@ { pkgs, guestModule }: let - runners = [ - "github-runner-runner.service" - "github-runner-runner-2.service" - ]; hostModule = { lib, ... }: { imports = [ ../modules/github-runner-vm.nix ]; services.github-runner-vms = @@ -108,8 +104,8 @@ pkgs.testers.runNixOSTest { "-device virtio-balloon-pci,free-page-reporting=on" ]; systemd.services.cache-seed-fixture = { - requiredBy = runners; - before = runners; + requiredBy = [ "github-runner-runner.service" ]; + before = [ "github-runner-runner.service" ]; unitConfig.ConditionPathExists = "!/var/lib/cache-seeded"; serviceConfig.Type = "oneshot"; # Lay out the cache and registration the way DynamicUser left them on @@ -159,26 +155,6 @@ pkgs.testers.runNixOSTest { '' ); }; - systemd.services.github-runner-runner-2.serviceConfig = { - ExecStartPre = lib.mkForce [ - "+${pkgs.coreutils}/bin/touch /var/lib/github-runner/runner-2/.current-token" - ]; - ExecStart = lib.mkForce ( - pkgs.writeShellScript "second-runner-probe" '' - set -euo pipefail - # The first runner empties its own work directory whenever it starts, - # so a second runner must not share it. - test "$PWD" = /var/cache/github-runner-work-2 - printf '#!${pkgs.runtimeShell}\necho executable\n' > exec-probe - chmod +x exec-probe - test "$(./exec-probe)" = executable - test "$(cat "$HOME/seeded")" = seeded - mkdir -p "$MBX_CACHE_DIR" - echo "$PWD" > "$MBX_CACHE_DIR/second" - exec ${pkgs.coreutils}/bin/sleep infinity - '' - ); - }; environment.systemPackages = [ pkgs.curl ]; }; other = { @@ -282,11 +258,6 @@ pkgs.testers.runNixOSTest { guest.succeed("test ! -L /var/lib/github-runner/runner") guest.succeed("test $(cat /var/lib/github-runner/runner/.runner) = registered") - with subtest("a second runner shares the home and mbx store but not the work directory"): - guest.wait_for_unit("github-runner-runner-2.service") - guest.wait_for_file("/var/cache/github-runner-home/.cache/mbx/second") - guest.succeed("test $(cat /var/cache/github-runner-home/.cache/mbx/second) = /var/cache/github-runner-work-2") - with subtest("the uplink stays IPv4-only"): guest.succeed("test $(cat /proc/sys/net/ipv6/conf/eth0/disable_ipv6) = 1") diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 2f0da3af34e..235494f5cea 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -53,15 +53,18 @@ Decisions (maintainer, 2026-10-13): - Self-hosted CI is dropped: hosted runners plus the R2 caches replace the spare-desktop runner, its cache server, and the cache release it waited on. Reopened 2026-10-19, conditionally, to compare runners once the pipeline - was optimized. Decided 2026-10-22 (maintainer): the queue runner is the - persistent `nix/runner` VM (no host mounts or tailnet, firewalled to public - egress), in use since 2026-10-20. Over 32 one-pass entries it took a median - 5.5 minutes per merged PR (p90 11.9, at most 24.2) against 21 minutes on a - hosted runner, with no per-minute bill. Blacksmith is not pursued. - Open (found 2026-10-23): the runner deployed on that host is a - systemd-nspawn container (`container@moq-ci`) with `/dev/kvm` and a host - cache directory bound in, not the VM this decision and `nix/runner/README.md` - describe. Confirm which is intended before changing either. + was optimized. Decided 2026-10-22 (maintainer): the queue runs on a + persistent self-hosted runner, in use since 2026-10-20. Over 32 one-pass + entries it took a median 5.5 minutes per merged PR (p90 11.9, at most 24.2) + against 21 minutes on a hosted runner, with no per-minute bill. Blacksmith + is not pursued. +- Decided 2026-10-25 (maintainer), superseding the 2026-10-22 choice of the + `nix/runner` VM: no VM. The queue runner is the deployed systemd-nspawn + container `container@moq-ci`, and new queue runners go into it. Reason: the + CI host should run mostly sandboxed through kernel primitives (namespaces, + cgroups and slices, seccomp), which the container already uses, so a VM is + not needed for isolation. The `nix/runner` VM module is not the deployment + target. Decided 2026-10-23 (maintainer), to speed up the serial queue: structural changes first (concurrent entries, parallel modules, one test build), then a diff --git a/quest/m0/ci/queue-concurrency.md b/quest/m0/ci/queue-concurrency.md index b504b49665e..5ef15f3a3d9 100644 --- a/quest/m0/ci/queue-concurrency.md +++ b/quest/m0/ci/queue-concurrency.md @@ -15,25 +15,46 @@ Measured 2026-10-23 over 32 merge-queue runs: `max_entries_to_build: 1` with one `moq-queue` runner makes the queue fully serial, about 7 merged PRs per hour, with a median entry of 135 s (p90 516 s) and 3 to 12 s of runner wait. -The runner VM module's `runner.count` registers several runners in one guest. -They share the home, so concurrent entries share mbx's store, its dedup of -in-flight compilations, and its compiler CPU and memory budget; each has its own -work directory. mbx documents concurrent builds on one store as supported, and -the self-hosted cache step keeps a version stamp per workspace. - -Remaining: - -- Confirm which runner is intended: the documented `nix/runner` VM, or the - `container@moq-ci` systemd-nspawn container deployed on the host (see the - [CI epic](/quest/m0/ci/README.md)). Give whichever stays two runners sharing - its cache. -- Deploy with `runner.count = 2`, a fresh registration token, and a VM restart. - The host also runs the `moq-pro` runner under the same `ci.slice` quota and a - desktop; size the guest's `resources` so concurrent entries do not slow each - other past the gain. Each entry runs 24 nextest workers. -- Raise the main ruleset's `max_entries_to_build` to the runner count. -- Measure merged PRs per hour and per-entry wall time before and after. -- Queue entries skip `quest gate`: `sh/local.sh` execs directly under GitHub - Actions, so the runner quota, not admission, has to keep entries apart. +Decided 2026-10-25 (maintainer): no VM. The second runner goes into the +deployed `container@moq-ci` systemd-nspawn container, sandboxed by kernel +primitives (see the [CI epic](/quest/m0/ci/README.md)), not into the +`nix/runner` VM module. The container is defined in the host's own flake, not +in this repository, so registering the runner is a host change. + +Each runner keeps its own work directory, since the nixpkgs runner service +empties it on start. The self-hosted cache step keeps its mbx version stamp per +workspace, so a runner sharing a home with another still drops its own +workspace's state after an mbx version change. + +Open: how the two runners share state. A NixOS container builds through the +host's Nix daemon by default, so the Nix store is shared in every option. + +1. One home under one static user (recommended): concurrent entries share + mbx's store, wait on a compilation the other entry is running instead of + repeating it, and draw on one machine-wide compile budget, which mbx + documents as supported + ([how it works](https://github.com/jdx/mr-boxington/blob/v1.25.0/docs/how-it-works.md#machine-wide-scheduling)). + nixpkgs gives each runner its own `DynamicUser` by default, so both need + the same `user` and `HOME`. +2. A home per runner: no mutable state crosses entries, but each cache warms + separately, disk use doubles, and two mbx schedulers each budget the whole + container's CPUs. +3. A second container: the strongest separation and its own cgroup, with the + costs of option 2 plus a second container to maintain. + +Remaining (maintainer, on the host): + +- Register a second runner in `container@moq-ci` with its own name and work + directory, under the chosen layout, with a fresh registration token. +- Size the container's share of `ci.slice`, which it shares with the `moq-pro` + runner and a desktop, so concurrent entries do not slow each other past the + gain. Queue entries skip `quest gate` (`sh/local.sh` execs directly under + GitHub Actions), so the cgroup quota, not admission, keeps them apart. +- Keep `NEXTEST_TEST_THREADS=24` per entry (48 workers across two entries; + mbx budgets compiles, not tests) and watch the first day for timing flakes + rather than guess a lower number. +- Raise the main ruleset's `max_entries_to_build` from 1 to 2. +- Measure merged PRs per hour and per-entry wall time for a day against the + baseline before going past 2. Public API: none. Wire: none. diff --git a/quest/m2/audit-ci.md b/quest/m2/audit-ci.md index b1e6ddb2720..fe331a78580 100644 --- a/quest/m2/audit-ci.md +++ b/quest/m2/audit-ci.md @@ -140,7 +140,7 @@ verdicts: ran). The longest 2026-10-22 jobs: Audio quality 62 minutes, Loom 60 (cancelled), binding benchmark 52, Rust bench targets 48, relay profiles 45, Rust benchmark tests 44, the gate suite 40. -- The merge queue runs one `just ci` job on the persistent `nix/runner` VM: +- The merge queue runs one `just ci` job on the self-hosted `moq-queue` runner: median 5.5 minutes, p90 11.9, at most 24.2 over 32 entries (m0, 2026-10-21). Its job summary lists seconds per module, and the `cargo-timings` artifact is the per-crate drill-down. diff --git a/quest/m2/gpu-ci.md b/quest/m2/gpu-ci.md index 1b29133d2b8..339782be695 100644 --- a/quest/m2/gpu-ci.md +++ b/quest/m2/gpu-ci.md @@ -57,6 +57,11 @@ skipping inside the Nix shell. guest, and keep the runner registered only to this repository with approval required for every external contributor's workflow, as the README requires. +- Open (2026-10-25): the [CI epic](/quest/m0/ci/README.md) dropped the VM + for the CI host in favor of kernel sandboxing, which conflicts with VFIO + passthrough. Recommendation: a container runner with the NVIDIA device + nodes and the host driver's libraries bound in. Confirm with the + maintainer before starting. Public API: none. Wire: none. From 1f08e6fc2d1de8c23217855d5921f09d38e83695 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 25 Oct 2026 09:16:33 -0700 Subject: [PATCH 4/5] quest: record one home under one static user for the queue runners Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/queue-concurrency.md | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/quest/m0/ci/queue-concurrency.md b/quest/m0/ci/queue-concurrency.md index 5ef15f3a3d9..fadf0593926 100644 --- a/quest/m0/ci/queue-concurrency.md +++ b/quest/m0/ci/queue-concurrency.md @@ -26,26 +26,23 @@ empties it on start. The self-hosted cache step keeps its mbx version stamp per workspace, so a runner sharing a home with another still drops its own workspace's state after an mbx version change. -Open: how the two runners share state. A NixOS container builds through the -host's Nix daemon by default, so the Nix store is shared in every option. - -1. One home under one static user (recommended): concurrent entries share - mbx's store, wait on a compilation the other entry is running instead of - repeating it, and draw on one machine-wide compile budget, which mbx - documents as supported - ([how it works](https://github.com/jdx/mr-boxington/blob/v1.25.0/docs/how-it-works.md#machine-wide-scheduling)). - nixpkgs gives each runner its own `DynamicUser` by default, so both need - the same `user` and `HOME`. -2. A home per runner: no mutable state crosses entries, but each cache warms - separately, disk use doubles, and two mbx schedulers each budget the whole - container's CPUs. -3. A second container: the strongest separation and its own cgroup, with the - costs of option 2 plus a second container to maintain. +Decided 2026-10-25 (maintainer): ✅ one home under one static user for both +runners. Concurrent entries share mbx's store, wait on a compilation the other +entry is running instead of repeating it, and draw on one machine-wide compile +budget, which mbx documents as supported +([how it works](https://github.com/jdx/mr-boxington/blob/v1.25.0/docs/how-it-works.md#machine-wide-scheduling)). +nixpkgs gives each runner its own `DynamicUser` by default, so both need the +same `user` and `HOME`. Rejected: a home per runner (two cold caches, double +the disk, two mbx schedulers each budgeting the whole container's CPUs) and a +second container (those costs plus another container). A NixOS container +builds through the host's Nix daemon by default, so the Nix store is shared +either way. Remaining (maintainer, on the host): - Register a second runner in `container@moq-ci` with its own name and work - directory, under the chosen layout, with a fresh registration token. + directory, under the same static user and `HOME` as the first, with a fresh + registration token. - Size the container's share of `ci.slice`, which it shares with the `moq-pro` runner and a desktop, so concurrent entries do not slow each other past the gain. Queue entries skip `quest gate` (`sh/local.sh` execs directly under From e243574db773406808bbb598ef07827aa4247161 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 25 Oct 2026 09:27:04 -0700 Subject: [PATCH 5/5] quest: make the pre-build mbx sweep safe before concurrent entries Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/queue-concurrency.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/quest/m0/ci/queue-concurrency.md b/quest/m0/ci/queue-concurrency.md index fadf0593926..4d926161ec0 100644 --- a/quest/m0/ci/queue-concurrency.md +++ b/quest/m0/ci/queue-concurrency.md @@ -50,6 +50,11 @@ Remaining (maintainer, on the host): - Keep `NEXTEST_TEST_THREADS=24` per entry (48 workers across two entries; mbx budgets compiles, not tests) and watch the first day for timing flakes rather than guess a lower number. +- Make the pre-build mbx sweep safe for a concurrent entry first. With + [moq-dev/moq#6709](https://github.com/moq-dev/moq/pull/6709), each job runs + `mbx gc` before it builds (`.github/actions/rust-cache`), and in a shared + home that sweep can evict generated source trees another entry's next + build reads. - Raise the main ruleset's `max_entries_to_build` from 1 to 2. - Measure merged PRs per hour and per-entry wall time for a day against the baseline before going past 2.