Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 40 additions & 2 deletions .github/workflows/bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ jobs:
- name: Diff
id: diff
run: |
# The trigger's paths, less the flake's.
if ! git diff --quiet HEAD^ HEAD -- bench/build .github/workflows/bench.yml ':!*.md'; then
# The trigger's paths, less the flake's and the Nix comparison's, which runs nightly only.
if ! git diff --quiet HEAD^ HEAD -- bench/build .github/workflows/bench.yml ':!*.md' ':!bench/build/nix*'; then
echo "bench=true" >>"$GITHUB_OUTPUT"
exit 0
fi
Expand Down Expand Up @@ -113,3 +113,41 @@ jobs:
name: bench-${{ strategy.job-index }}
path: bench-results
if-no-files-found: ignore

# moq's Nix release builds, each way they could share compilations. It takes
# hours and informs a choice rather than gating a change, so it runs nightly only.
nix:
name: Bench (moq nix)
if: github.event_name != 'pull_request'
runs-on: ubuntu-24.04
timeout-minutes: 330

steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: ./.github/actions/nix

# Each variant builds in a private Nix store, which sandboxes its builds in
# user namespaces that Ubuntu reserves for root.
- name: Allow user namespaces
run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

- name: Allow perf counters
run: sudo sysctl -w kernel.perf_event_paranoid=2

- name: Bench
run: nix develop .#build-bench --command just bench-nix --out bench-results

- name: Summary
run: cat bench-results/report.md >>"$GITHUB_STEP_SUMMARY"

- name: Upload
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: bench-nix
path: bench-results
if-no-files-found: ignore
52 changes: 51 additions & 1 deletion bench/build/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Build-speed benchmark

`just bench` measures Rust build time on the pinned projects and build configurations in [corpus.toml](corpus.toml).
It runs nightly in CI ([bench.yml](../../.github/workflows/bench.yml)), and on a PR that changes this harness, its workflow, the `build-bench` shell, or the `bench-ra-jit` package.
It runs nightly in CI ([bench.yml](../../.github/workflows/bench.yml)), and on a PR that changes this harness (apart from the [Nix comparison](#nix-builds)), its workflow, the `build-bench` shell, or the `bench-ra-jit` package.
Every build-speed change here states its effect with its numbers.

```bash
Expand Down Expand Up @@ -73,6 +73,56 @@ A busy host adds an occasional slow run, which on an edit of a few seconds can m
Instructions barely move with machine load, so they are the signal on a busy machine, where wall time is only a smoke test; they read `n/a` where the host exposes no performance counters.
No threshold is set: the numbers inform a decision, they do not gate one.

## Nix builds

`just bench-nix` measures moq's Nix release builds (`moq-relay`, then `moq-cli`) in CPU time, built each way they could share compilations:

- `crane`: moq's flake as it is, a dependency layer and a workspace layer per package.
- `crane-mbx`: the same packages with no dependency layer, every compilation going through mbx on a store mounted into the sandbox with `extra-sandbox-paths`, as nixpkgs' ccache support does.
- `crate2nix`: a derivation per crate, from `crate2nix generate`, so the Nix store itself is the cache.
- `cargo-mbx`: cargo through mbx outside Nix, as moq's CI builds today, with a fresh target directory per build.

```bash
just bench-nix # every variant; takes hours
just bench-nix --variant crane --variant crate2nix
```

[nix.toml](nix.toml) pins moq, the packages, the edit, and the lockfile bump; [nix.nix](nix.nix) holds the builds, with the toolchain, features, and `RUSTFLAGS` of moq's overlay throughout.
Results land in `target/bench/nix/`, newest at `target/bench/nix/latest/report.md`.
It runs nightly in CI, never on a pull request: it informs a choice rather than gating a change.

Each variant runs four scenarios in order on a private Nix store and mbx store, so only the first is cold:

- `clean`: the first package from nothing.
- `second-package`: the second package, which shows what the two share.
- `edit-hub`: the first package after the moq-net body edit the cargo scenarios make.
- `lock-bump`: the first package after `cargo update` of one dependency, the edit reverted.

A cold edit or bump costs what `clean` does, so only the warm ones run.
Each Nix build runs in a store of its own, which substitutes only from cache.nixos.org, so nothing the host or moq's cache built leaks in; before the timed build, it realises everything the build needs that does not compile Rust (toolchain, sources, vendored crates).
That store runs its builds as this user in user namespaces, which makes their CPU time the `nix` process's own; Ubuntu reserves those namespaces for root unless `kernel.apparmor_restrict_unprivileged_userns` is 0.
Evaluation is timed apart from the build, since crate2nix evaluates a derivation per crate.
The report compares retired instructions, which barely move with machine load, and shows CPU time beside them.
crate2nix reads no Cargo profile, so nix.nix gives it moq's `[profile.release]`: one codegen unit, and fat LTO and symbol stripping on the linked packages. `crate2nix generate`, which a project commits, is timed but not counted.

### Nix builds for moq

Decided on 2026-10-26: moq explores crate2nix, and keeps its crane layers as they are.

Measured on 2026-10-25 and 2026-10-26 on a busy 32-CPU host, in billions of instructions, at the pin in nix.toml:

| scenario | crane | crane-mbx | crate2nix | cargo-mbx |
| --- | ---: | ---: | ---: | ---: |
| `clean` | 3681 | 3377 | 4672 | 3378 |
| `second-package` | 5926 | 3945 | 6999 | 3924 |
| `edit-hub` | 1365 | 1351 | 1509 | 1276 |
| `lock-bump` | 3681 | 1123 | 1255 | 1113 |
| total, evaluation included | 14756 | 9894 | 14538 | 9692 |

- A lockfile bump costs crane a full dependency rebuild; the per-crate caches rebuild hyper and the 12 crates above it, about a third of crane's cost, most of it moq-relay's fat-LTO link.
- moq-cli shares 285 of its 548 compilations with moq-relay; the rest are crates or feature sets moq-relay does not build. crane shares none, and its dependency layer runs a `cargo check` of every dependency before building them.
- crate2nix costs 27% more than crane on a clean build and 18% more on the second package. Cargo compiles a dependency that only fat LTO reads to bitcode alone, and build scripts and proc macros at opt-level 0; buildRustCrate compiles every crate to machine code and bitcode at opt-level 3.

## Cranelift for moq's CI

Recommended on 2026-10-25: moq's CI keeps building its tests with LLVM, since 42 of its 5418 tests fail under Cranelift.
Expand Down
69 changes: 69 additions & 0 deletions bench/build/nix-report.jq
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Renders a `just bench-nix` run as Markdown. Inputs: $host (host.json),
# $results (results.jsonl), and $generate (crate2nix's generate timings). Run
# with `jq -n -r`.

def fixed(p): (. * pow(10; p) | round) / pow(10; p) | tostring;
def secs: if . == null then "-" elif . < 10 then fixed(2) else fixed(1) end;
def giga: if . == null then "n/a" else (. / 1e9 | round | tostring) + "G" end;
def pct: (. * 100 | fixed(1)) as $p | if . > 0 then "+\($p)%" else "\($p)%" end;
def vs($base): if $base == null or $base == 0 or . == null then "-" else . / $base - 1 | pct end;
def count: if . == null then "-" else tostring end;
def row: "| " + join(" | ") + " |";

$host[0] as $h
| ["clean", "second-package", "edit-hub", "lock-bump"] as $scenarios
| ($results | map({key: "\(.scenario)/\(.variant)", value: .}) | from_entries) as $by
| ($results | map(.variant) | unique) as $variants
| [
"# Nix build benchmark",
"",
"\($h.date) on \($h.cpu) (\($h.cpus) CPUs, \($h.memory)), \($h.os), runner `\($h.runner)`.",
"rust2 `\($h.rust2[:12])`; moq `\($h.moq[:12])`; \($h.nix); \($h.mbx); crate2nix \($h.crate2nix).",
"",
"One build per row: CPU seconds (user plus system time of every process it ran) and retired instructions.",
"Instructions barely move with machine load, so they are the comparison on a busy host; `vs crane` compares them.",
"Each variant runs the scenarios in this order on one store, so every row after `clean` starts warm.",
"Evaluation is timed apart, and the totals add it.",
"`derivations` counts the Nix derivations that compiled Rust; `compiled` counts rustc runs, as cargo's `Compiling` and `Checking` lines for crane, a derivation per crate for crate2nix, and mbx's misses for mbx; `restored` counts mbx's hits.",
"",
(["scenario", "variant", "CPU", "instructions", "vs crane", "wall", "eval CPU", "derivations", "compiled", "restored"] | row),
"| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |",
(
$scenarios[] as $s
| $variants[] as $v
| $by["\($s)/\($v)"] // empty
| [
$s,
$v,
(.build.cpu | secs),
(.build.instructions | giga),
(if $v == "crane" then "base" else .build.instructions | vs($by["\($s)/crane"].build.instructions) end),
(.build.wall | secs),
(.eval.cpu | secs),
(.derivations | count),
(.compiled | count),
(.restored | count)
]
| row
),
"",
(["variant", "total CPU", "total instructions", "vs crane"] | row),
"| --- | ---: | ---: | ---: |",
(
# A variant with no evaluation adds none; a measurement a phase lacks leaves the total unknown.
def total($v; f): $results | map(select(.variant == $v) | [(.build | f), (if .eval == null then 0 else .eval | f end)])
| flatten | if any(. == null) then null else add end;
total("crane"; .instructions) as $base
| $variants[] as $v
| [$v, (total($v; .cpu) | secs), (total($v; .instructions) | giga),
(if $v == "crane" then "base" else total($v; .instructions) | vs($base) end)]
| row
),
(
select($generate | length > 0)
| "",
"`crate2nix generate`, which a project reruns and commits when its lockfile changes, took "
+ ($generate | map("\(.cpu | secs) CPU seconds at the \(.lock) lockfile") | join(" and ")) + "."
)
]
| .[]
111 changes: 111 additions & 0 deletions bench/build/nix.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# moq's Nix packages, built each way nix.sh compares. Evaluated with --impure,
# since the checkout is a path the run edits.
{
# moq's checkout, as an absolute path string: a path literal would copy it to
# the store first. crate2nix's Cargo.nix sits at its root, beside the crates
# it names by relative path.
src,
# mbx's store path, and the sandbox path its store is mounted at.
mbx,
mbxStore,
}:
let
system = builtins.currentSystem;
# moq's own nixpkgs, rust-overlay, and crane, from its lockfile.
lock = builtins.fromJSON (builtins.readFile "${src}/flake.lock");
locked = name: lock.nodes.${lock.nodes.root.inputs.${name}}.locked;
pkgs = import (builtins.fetchTree (locked "nixpkgs")) {
inherit system;
overlays = [ (import (builtins.fetchTree (locked "rust-overlay"))) ];
};
inherit (pkgs) lib;
crane' = with locked "crane"; builtins.getFlake "github:${owner}/${repo}/${rev}";

# The toolchain moq's overlay builds with, which pins the same channel.
channel = (lib.importTOML "${src}/rust-toolchain.toml").toolchain.channel;
toolchain = pkgs.rust-bin.stable.${channel}.default;

# What nix/overlay.nix passes cargo for each package; crate2nix and cargo
# resolve the same features from it.
packages = {
moq-relay = [ "jemalloc" ];
moq-cli = [ "jemalloc" ];
};
rustflags = [
"-C"
"force-frame-pointers=yes"
];

# crane as moq builds it: its flake's overlay, keyed by cargo package.
moq = pkgs.extend (import "${src}/nix/overlay.nix" { crane = crane'; });
crane = lib.mapAttrs (name: _: moq.${name}) packages;

# crane with no dependency layer: every compilation goes through mbx, on a
# store outside the sandbox. `cargo` on PATH is mbx's shim, which finds the
# real cargo after its own directory, so that directory goes on PATH as is.
craneMbx = lib.mapAttrs (
_: drv:
drv.overrideAttrs (old: {
cargoArtifacts = null;
preConfigure = ''
export PATH=${builtins.storePath mbx}/libexec/mbx:$PATH
''
+ (old.preConfigure or "");
# mbx would move a target directory it finds into its store.
CARGO_TARGET_DIR = "target";
MBX_CACHE_DIR = mbxStore;
MBX_GC_AUTO = "0";
# The report lands in the output, where nix.sh counts hits and misses.
preBuild = (old.preBuild or "") + ''
export MBX_STATS_REPORT=$NIX_BUILD_TOP/mbx-report.json
'';
postInstall = (old.postInstall or "") + ''
install -Dm644 $MBX_STATS_REPORT $out/share/mbx/report.json
'';
})
) crane;

# One derivation per crate, with crate2nix's generated build plan. It reads no
# Cargo profile, so the packages take moq's `[profile.release]` here:
# buildRustCrate's one codegen unit already matches, and fat LTO and symbol
# stripping apply where cargo applies them, to the linked packages.
crate2nix =
let
buildRustCrate = pkgs.buildRustCrate.override {
rustc = toolchain;
cargo = toolchain;
};
release = [
"-C"
"lto=fat"
"-C"
"strip=symbols"
];
plan = pkgs.callPackage "${src}/Cargo.nix" {
buildRustCrateForPkgs =
_: crate:
buildRustCrate (
crate
// {
extraRustcOpts =
rustflags ++ lib.optionals (packages ? ${crate.crateName}) release ++ (crate.extraRustcOpts or [ ]);
}
);
};
in
# Like cargo's `--features`, these add to the package's defaults.
lib.mapAttrs (
name: features:
plan.workspaceMembers.${name}.build.override { features = [ "default" ] ++ features; }
) packages;
in
{
inherit
crane
craneMbx
crate2nix
toolchain
packages
rustflags
;
}
Loading
Loading