From b0c6467dea8270b3d98fdda3156a36acc79c04af Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Mon, 26 Oct 2026 09:27:39 -0700 Subject: [PATCH 1/2] quest: crate2nix replaces the crane layering quest Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn --- quest/m0/ci/README.md | 2 +- quest/m0/ci/crane-layers.md | 48 ------------------------------------ quest/m0/ci/crate2nix.md | 49 +++++++++++++++++++++++++++++++++++++ quest/m0/ci/hakari.md | 2 +- 4 files changed, 51 insertions(+), 50 deletions(-) delete mode 100644 quest/m0/ci/crane-layers.md create mode 100644 quest/m0/ci/crate2nix.md diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 881b908829a..2da16163ed9 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -118,7 +118,7 @@ The historical samples locate candidates; they are not controlled benchmarks. - [Bump to the manifest fix](/quest/m0/ci/mbx-bump.md) - condition: an mbx release contains jdx/mr-boxington#798; then bump the Rust cache's pin - [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measure the hosted Nightly jobs' hits, restore time, and expiry once the bump lands - [Verify the Nix cache](/quest/m0/ci/nix-cache-verify.md) - confirm CI and releases use `nix.moq.dev` and retire Cachix -- [Share crane's dependency layer](/quest/m0/ci/crane-layers.md) - Nix packages build their dependencies once, and a workspace edit rebuilds none of them +- [Build Nix packages with crate2nix](/quest/m0/ci/crate2nix.md) - one derivation per crate cached in nix.moq.dev, with `Cargo.nix` regenerated by CI so contributors never need Nix - [Unify dependency features](/quest/m0/ci/hakari.md) - a cargo-hakari workspace-hack so every job compiles each dependency alike and mbx hits across jobs - [Audit CI's package selection](/quest/m0/ci/select-audit.md) - how often the queue runs everything, and whether each rule that forces it is needed - [First shared FFI release](/quest/m0/ci/nightly-ffi-reuse.md) - condition: the backport merges and the next `moq-ffi-v*` tag runs `Release FFI`; verify the wrappers diff --git a/quest/m0/ci/crane-layers.md b/quest/m0/ci/crane-layers.md deleted file mode 100644 index af44263f843..00000000000 --- a/quest/m0/ci/crane-layers.md +++ /dev/null @@ -1,48 +0,0 @@ -# [S] Share crane's dependency layer - -## Goal - -moq's Nix packages build their dependencies once, in one crane dependency layer that -every package reuses as its `cargoArtifacts`. Editing a workspace crate, kio -included, rebuilds no dependency, and `nix build` of a second package after the -first compiles only that package's own crates plus any dependency whose resolved -features differ from the layer's. Record in the PR how many crates each case -compiles, before and after, and which dependencies still differ. - -## Plan - -Decided with the maintainer on 2026-10-25, as the first step of making each crate -configuration compile once across moq's CI and release builds, measured in runner -CPU time. Today `nix/overlay.nix` builds a separate dependency layer per package, -and any change to that package's features, `cargoExtraArgs`, or `RUSTFLAGS` -rebuilds all of its roughly 330 dependencies. - -- One dependency layer, passed to every package as `cargoArtifacts`. Cargo - fingerprints each dependency by the features its own invocation resolves, and - a `-p` build resolves only that package's, so a union layer is reused only - where a package resolves the same features. Build the layer from the packages' - own invocations (each package's `cargoExtraArgs`, in one derivation) so every - package finds its exact compilations, and count what each package adds. - [Unifying dependency features](/quest/m0/ci/hakari.md) later makes those - resolutions identical. -- Move moq-relay's `-C force-frame-pointers=yes` into `.cargo/config.toml` so - every build, Nix or cargo, compiles with the same flags and shares mbx keys. - Frame pointers cost little on x86_64; measure aarch64 too. -- The dependency layer copies kio's real source into crane's dummy tree, because - `[patch.crates-io] kio` points registry crates at the workspace member. That - makes every kio edit rebuild every dependency. Restructure it so kio builds in - the workspace layer, for example by giving the dummy tree a stub kio that - exposes its API, or by building the crates that depend on the patched kio in - the workspace layer. - -kixelated/rust2's -[Nix build benchmark](https://github.com/kixelated/rust2/blob/main/quest/m1/bench-nix.md) -then compares this layering with crane plus mbx, crate2nix, and cargo with mbx, -and moq adopts the winner in a follow-up. - -Public API: none. Release artifacts: built with frame pointers on every target, -not only moq-relay. - -## Related - -- [Unify dependency features](/quest/m0/ci/hakari.md) - makes every package resolve the layer's features diff --git a/quest/m0/ci/crate2nix.md b/quest/m0/ci/crate2nix.md new file mode 100644 index 00000000000..538a9ca9d51 --- /dev/null +++ b/quest/m0/ci/crate2nix.md @@ -0,0 +1,49 @@ +# [M] Build moq's Nix packages with crate2nix + +## Goal + +moq's Nix packages build one derivation per crate through crate2nix instead of crane, +so `nix.moq.dev` serves every compiled crate to any runner and a lockfile bump or a +workspace edit rebuilds only the crates it changes. Contributors without Nix are +unaffected: cargo stays the build they use, and nothing they run needs Nix. Record +in the PR the instructions or CPU time for a clean build, a second package, the +moq-net edit, and a lockfile bump, before and after. + +## Plan + +Decided with the maintainer on 2026-10-26, from kixelated/rust2's +[Nix build benchmark](https://github.com/kixelated/rust2/pull/63). That benchmark +measured billions of instructions across a clean build, a second package, an edit +and a lockfile bump: + +| | crane | crane with mbx | crate2nix | cargo with mbx | +| --- | ---: | ---: | ---: | ---: | +| total | 14,756 | 9,894 | 12,461 | 9,692 | +| edit | 1,365 | 1,351 | 1,008 | 1,276 | +| lockfile bump | 3,681 | 1,123 | 672 | 1,113 | + +On the builds CI repeats, crate2nix was cheapest, and it caches through the Nix store +with no builder configuration. crane with mbx needs `extra-sandbox-paths` and a +persistent store on every builder. + +- **Explore first.** Prototype crate2nix on moq-relay and moq-cli before switching + every package, and keep crane until the numbers hold on moq's own CI. +- **No Nix required.** The generated `Cargo.nix` is committed so Nix users build + without generating it. CI regenerates it and fails when it is stale. Prefer + regenerating it automatically, for example a CI job that commits the update on a + Renovate or lockfile PR, so a contributor without Nix never has to. +- **Cold build.** crate2nix builds build scripts and proc macros at opt-level 3, + where cargo uses 0, which makes a clean build 29% dearer. Match cargo. +- **Flags.** Move moq-relay's `-C force-frame-pointers=yes` into + `.cargo/config.toml` so Nix and cargo builds use the same flags; measure aarch64. +- **Also measure** `codegen-units = 1` for the release profile, which cut crate2nix's + builds 20% to 45% in the benchmark. +- Replaces the crane layering quest, which only improved the second-package case: + with crane, a lockfile bump still rebuilds every dependency. + +Public API: none. Release artifacts: built by crate2nix, with frame pointers on +every target. + +## Related + +- [Unify dependency features](/quest/m0/ci/hakari.md) - lets packages share more crate derivations diff --git a/quest/m0/ci/hakari.md b/quest/m0/ci/hakari.md index a70298dab26..24afd699c2b 100644 --- a/quest/m0/ci/hakari.md +++ b/quest/m0/ci/hakari.md @@ -31,7 +31,7 @@ feature set in another job is a miss, however warm the cache. - Unifying features means some packages build features they don't use. Measure whether that costs more CPU than the cache hits save; target, profile, and `RUSTFLAGS` differences still split keys, and Nix's flags are unified in - [crane-layers](/quest/m0/ci/crane-layers.md). + [crate2nix](/quest/m0/ci/crate2nix.md). - Start from the hit report in [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md), which lists the compilations that miss only because their features or flags differ. From 75a1e45c27c41ecd516d8a05b548cbff2e87fee6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 26 Oct 2026 09:50:42 -0700 Subject: [PATCH 2/2] quest: match crate2nix to the release profile; regenerate Cargo.nix on every PR The benchmark built crate2nix with 16 codegen units and no LTO while crane and cargo used moq's release profile, so the quest now matches that profile before comparing. Cargo.nix is regenerated on every PR, not only lockfile ones, with a patch for forks CI cannot push to. Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/README.md | 13 +++++----- quest/m0/ci/crate2nix.md | 53 ++++++++++++++++++++++++++-------------- 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 2da16163ed9..609dac871aa 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -91,13 +91,12 @@ Decided 2026-10-25 (maintainer): across moq's many CI and release builds compiles once and is reused through the cache, and each run builds and tests only what its diff can affect, measured in runner CPU time. Wall clock for its own sake and replacing cargo are non-goals. mbx already caches each compilation, -so the work is making builds identical enough to hit it (one crane dependency -layer, flags in `.cargo/config.toml`, cargo-hakari), and auditing the package -selection `just ci` already does. Nix builds run sandboxed and cannot reach R2: -kixelated/rust2's -[Nix build benchmark](https://github.com/kixelated/rust2/blob/main/quest/m1/bench-nix.md) -decides between crane with mbx, crate2nix, and cargo with mbx after the crane -fixes land. +so the work is making builds identical enough to hit it (flags in +`.cargo/config.toml`, cargo-hakari), and auditing the package selection `just ci` +already does. Nix builds run sandboxed and cannot reach R2; decided 2026-10-26 +from kixelated/rust2's +[Nix build benchmark](https://github.com/kixelated/rust2/pull/63), they move to +crate2nix, whose per-crate derivations `nix.moq.dev` caches. Decided 2026-10-25 (maintainer), from a profile of moq-net: monomorphized generics dominate its build (53% of the library's CPU), and its test target diff --git a/quest/m0/ci/crate2nix.md b/quest/m0/ci/crate2nix.md index 538a9ca9d51..7a1fdc2aaee 100644 --- a/quest/m0/ci/crate2nix.md +++ b/quest/m0/ci/crate2nix.md @@ -12,32 +12,47 @@ moq-net edit, and a lockfile bump, before and after. ## Plan Decided with the maintainer on 2026-10-26, from kixelated/rust2's -[Nix build benchmark](https://github.com/kixelated/rust2/pull/63). That benchmark -measured billions of instructions across a clean build, a second package, an edit -and a lockfile bump: +[Nix build benchmark](https://github.com/kixelated/rust2/pull/63), in billions of +instructions. Each scenario builds moq-relay, except the second package (moq-cli +after moq-relay); the total sums the four. -| | crane | crane with mbx | crate2nix | cargo with mbx | -| --- | ---: | ---: | ---: | ---: | -| total | 14,756 | 9,894 | 12,461 | 9,692 | -| edit | 1,365 | 1,351 | 1,008 | 1,276 | -| lockfile bump | 3,681 | 1,123 | 672 | 1,113 | +| | crane | crane with mbx | crate2nix | crate2nix, 1 codegen unit | cargo with mbx | +| --- | ---: | ---: | ---: | ---: | ---: | +| clean | 3,681 | 3,377 | 4,736 | 3,695 | 3,378 | +| second package | 5,926 | 3,945 | 5,951 | 4,902 | 3,924 | +| moq-net edit | 1,365 | 1,351 | 1,008 | 753 | 1,276 | +| lockfile bump | 3,681 | 1,123 | 672 | 496 | 1,113 | +| total | 14,756 | 9,894 | 12,461 | 9,846 | 9,692 | -On the builds CI repeats, crate2nix was cheapest, and it caches through the Nix store -with no builder configuration. crane with mbx needs `extra-sandbox-paths` and a -persistent store on every builder. +Neither crate2nix column matches moq's `[profile.release]`, which every other +column builds with: `codegen-units = 1` and fat LTO. buildRustCrate reads no Cargo +profile. The benchmark gave it 16 codegen units, and the separate run used +buildRustCrate's default of 1; neither ran LTO, which every crane and cargo build +pays at the final link, edits and bumps included. The 29% cold-build penalty +vanishes at one codegen unit, so it came from that mismatch. +crate2nix was still cheapest on the edits and lockfile bumps CI repeats, and it +caches through the Nix store with no builder configuration. crane with mbx needs +`extra-sandbox-paths` and a persistent store on every builder. + +- **Match the release profile first.** crate2nix's builds match `[profile.release]` + before any comparison: one codegen unit, fat LTO on the final link, stripped + symbols, the per-package overrides, and cargo's opt-level 0 for build scripts and + proc macros where buildRustCrate uses 3. The before and after numbers come from + those matched builds; LTO will raise crate2nix's edit and bump costs above the + table's. - **Explore first.** Prototype crate2nix on moq-relay and moq-cli before switching - every package, and keep crane until the numbers hold on moq's own CI. + every package, and keep crane until the matched numbers hold on moq's own CI. - **No Nix required.** The generated `Cargo.nix` is committed so Nix users build - without generating it. CI regenerates it and fails when it is stale. Prefer - regenerating it automatically, for example a CI job that commits the update on a - Renovate or lockfile PR, so a contributor without Nix never has to. -- **Cold build.** crate2nix builds build scripts and proc macros at opt-level 3, - where cargo uses 0, which makes a clean build 29% dearer. Match cargo. + without generating it. Any change to a manifest, the lockfile, or a crate's + targets can make it stale, so CI regenerates it on every PR and fails when it + differs. A bot commits the regenerated file to the PR's branch whenever it can + push there, Renovate's included; for a fork that blocks maintainer edits, the + failure carries the update as a patch to `git apply`. The PR's manifests are + untrusted, so generation runs without write access, and the job that pushes + takes only `Cargo.nix` from it. - **Flags.** Move moq-relay's `-C force-frame-pointers=yes` into `.cargo/config.toml` so Nix and cargo builds use the same flags; measure aarch64. -- **Also measure** `codegen-units = 1` for the release profile, which cut crate2nix's - builds 20% to 45% in the benchmark. - Replaces the crane layering quest, which only improved the second-package case: with crane, a lockfile bump still rebuilds every dependency.