From 5fe7614531269f8fbae2aab952aa38326f99d4c6 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 23 Oct 2026 02:45:50 -0700 Subject: [PATCH 1/3] chore(quest): claim quest/m1/audit-compile-time.md Quest-Claim: db811a3d8bf24666 From 32ccaaf55add3d5f7fe1422ffc3b769b0af7df2f Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 23 Oct 2026 03:03:54 -0700 Subject: [PATCH 2/3] perf(mux): decode fMP4 fragments through a slice cursor `Cursor<&Bytes>` was the only fMP4 decode not reading through `Cursor<&[u8]>`, so mp4-atom's `Any::decode_atom` compiled a fourth time. Dev IR for the moq-mux lib drops from 1,006,004 to 996,345 lines. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01EeyiEy5FWz3z44DYZVo7vU --- rs/moq-mux/src/container/fmp4/mod.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rs/moq-mux/src/container/fmp4/mod.rs b/rs/moq-mux/src/container/fmp4/mod.rs index c0aa07bdee5..941c186c092 100644 --- a/rs/moq-mux/src/container/fmp4/mod.rs +++ b/rs/moq-mux/src/container/fmp4/mod.rs @@ -491,7 +491,9 @@ pub(crate) fn decode(data: Bytes, timestamp: Option, track: Track) -> defaults, } = track; - let mut cursor = std::io::Cursor::new(&data); + // A slice cursor like the other fMP4 decodes: mp4-atom's `Any` decoder is large and compiles + // once per reader type. + let mut cursor = std::io::Cursor::new(data.as_ref()); let mut moof = None; let mut mdat = None; From 6666403d9c3ee6d00f28e850d8775e3ec8c539a2 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 23 Oct 2026 03:21:37 -0700 Subject: [PATCH 3/3] quest: finish the m1 compile time audit and plan the m2 run Records the m1 baselines (clean dev, release, incremental, and llvm-lines per target) in the successor, which waits on the release cut and checks moq-net compiles once (#6176) first. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01EeyiEy5FWz3z44DYZVo7vU --- quest/m1/README.md | 1 - quest/m1/audit-compile-time.md | 169 --------------------------------- quest/m2/README.md | 1 + quest/m2/audit-compile-time.md | 129 +++++++++++++++++++++++++ 4 files changed, 130 insertions(+), 170 deletions(-) delete mode 100644 quest/m1/audit-compile-time.md create mode 100644 quest/m2/audit-compile-time.md diff --git a/quest/m1/README.md b/quest/m1/README.md index f99627545fe..c897a80a886 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -155,7 +155,6 @@ tracks, so per-group stats stop costing cluster announces. - [Shared watch and publish UI](/quest/m1/js-ui.md) - a new `@moq/ui` package holds the graph, buttons, and support banner that watch and publish duplicate - [Simplification audit](/quest/m1/audit-simplify.md) - repeat the simplification audit, starting with the m0 leftovers, before the next major release - [Artifact size audit](/quest/m1/audit-size.md) - binary, FFI, WASM, and bundle growth since the m0 run is explained, fixed, or planned -- [Compile time audit](/quest/m1/audit-compile-time.md) - build-time hotspots are measured against the m0 baselines and the worst are fixed or planned - [Peer count bounds](/quest/m1/peer-count-bounds.md) - namespace interests, a legacy announce list, a WebSocket upgrade, and public HTTP admission are bounded by what the peer sent or by an existing budget - [Caption store bounds](/quest/m1/caption-store-bounds.md) - a hostile caption track cannot grow a viewer's cue and region store or its per-frame work without bound - [Auth contract](/quest/m1/auth-contract.md) - the auth server receives the certificate-selecting SNI, documented as client-chosen, and a certificate grants nothing alone diff --git a/quest/m1/audit-compile-time.md b/quest/m1/audit-compile-time.md deleted file mode 100644 index aecb99b11f8..00000000000 --- a/quest/m1/audit-compile-time.md +++ /dev/null @@ -1,169 +0,0 @@ -# [M] Compile time audit - -## Goal - -The crates, dependencies, and code that dominate build time are known, with -recorded baselines, and the worst are fixed or planned. - -## Plan - -Recurs once per major release (decided 2026-10-19). The PR that finishes -this run also adds the next one with the same file name at the end of the next -milestone's `Required` (m0 to m1, m1 to m2). The successor keeps the Goal -and this paragraph; the rest of its Plan is rewritten with what this run -learned: baselines, results, and what to check first. A successor placed in m2 -also lists the [release cut](/quest/m2/release/cut.md) under `Required`, so it -waits for the release in flight to ship. Only `/quest-delete` ends the -recurrence. Fixes land in the run's PR, one commit each, rather than as follow-ups. -Only a change that needs a maintainer decision (public API, scope) or is -independently landable and too large to review alongside the rest becomes a -follow-up through `/quest-plan`. - -Scope: this audit owns per-crate build cost in one build (dev, release, and -incremental). The [CI workflow audit](/quest/m2/audit-ci.md) owns job-level CI -cost and caching; the [dependency audit](/quest/m2/audit-deps.md) owns -deduplicating and upgrading dependencies. - -What to check first: whether [moq-net compiles once](/quest/m1/net-erase-transport.md) -landed. If so, rerun `cargo llvm-lines` on the moq-tokio lib and compare the -transport share below, then rerun the thin-LTO size comparison against the -smaller archive. Then whether `cargo-llvm-lines` joined the pinned shell. - -### Baselines (m0 run, 2026-10-19 and 2026-10-20) - -Pinned Nix shell (Rust 1.98.1, LLVM 22.1.8), Linux x86_64, Ryzen 9 5900XT, -32 logical CPUs, Cargo, native tools and Nextest bounded to two jobs. The host -was shared with other builds; load is the one-minute average. Only the -llvm-lines counts are deterministic. Every time is load-sensitive and a single -observation: compare shares within a run, never seconds across runs. - -Clean builds (`MBX_DISABLE=1`, fresh target dir, registry and shell present): - -| Build | Source | Load | Wall | Longest unit | -| --- | --- | --- | ---: | --- | -| `cargo build --locked --timings` (default members) | `cdff3c8f9` | 0.7 to 2.7 | 274 s | aws-lc-sys build script 87.6 s | -| `cargo build --locked --release --timings` (default members) | `cdff3c8f9` | up to 6.1 | 821 s | moq-cli bin 342 s | -| `cargo build --locked --release -p moq-cli --timings` | `d55407256` | 21 to 89, median 55 | 1,213 s | moq-cli bin 644 s | - -The final `moq` binary unit was 42% and 53% of those two release walls. It is -serial: fat LTO with one codegen unit. - -`cargo llvm-lines` 0.4.48 (`nix shell nixpkgs#cargo-llvm-lines`), dev profile, -default features, on `d55407256`: - -| Target | LLVM IR lines | Copies | Notable share | -| --- | ---: | ---: | --- | -| moq-tokio lib | 1,661,503 | 54,069 | moq-net code generic over the transport session 43.2%; usage-rs derive 5.2% | -| moq-cli `moq` bin | 1,037,734 | 41,758 | tokio 30.1%; usage-rs derive 2.8% | -| moq-mux lib | 908,532 | 25,751 | serde 19.4%; mp4-atom `Any::decode_atom` over four reader types 4.2% | -| moq-relay lib | 711,268 | 27,162 | serde 28.0% (TOML config); usage-rs derive 8.6% | -| moq-net lib | 686,443 | 22,660 | BTreeMap internals 24.1% | -| moq-relay bin | 384,813 | 15,113 | tokio 29.8% | - -In moq-tokio the transport-generic share splits into -`transport::Session` and -`transport::Session`, each again under moq-net's IETF -`ControlStreamAdapter`, so the IETF half compiles four times. Dev builds share -generics, so downstream crates pay little of it (0.3% of the relay lib). - -`RUSTC_BOOTSTRAP=moq cargo rustc --locked --release -p moq-cli --bin moq -- -Z time-passes` -on `d55407256` (load 25 to 52, median 36; 429 s): - -| Pass | Seconds | Share | -| --- | ---: | ---: | -| `LLVM_fatlto` | 317.8 | 74.1% | -| `LLVM_passes` | 98.9 | 23.1% | -| `codegen_crate` (monomorphization and IR emission) | 7.2 | 1.7% | -| everything else (expansion, type and borrow checking) | 3.9 | 0.9% | -| `link` | 1.2 | 0.3% | - -Naming only the `moq` crate in `RUSTC_BOOTSTRAP` keeps build-script nightly -probes stable, so the dependency build is the ordinary one. Shares should hold -under load, since every pass is single-threaded CPU work in one process. - -Incremental: ordinary cache-enabled `just check origin/main` with one -formatted comment line appended to `rs/moq-net/src/lib.rs`, fresh target dir, -mbx 1.25.0, on `d55407256` (load 20 to 44, median 34; 646 s, failed). It -selected 29 Rust packages (moq-net and every dependent, including C, FFI, -GStreamer and WASM). Phases: rustdoc 1m41s (678 hits, 76 misses, 34 bypassed), -WASM clippy 18.8 s, io-uring clippy 33.5 s, test build 5m40s (524 hits, 83 -misses, 5 bypassed). Nextest started 7,890 tests and stopped after 663 on the -known failover flake, fixed by -[moq-dev/moq#6113](https://github.com/moq-dev/moq/pull/6113). -Cargo rebuilds every dependent when an upstream crate's source changes at all, -even a comment, so the test build is the bulk of the compile cost. - -### Thin-LTO experiment (2026-10-20) - -The root `[profile.release]` (`lto = "fat"`, one codegen unit) builds every -shipped native artifact; `bench/ffi-gen` mirrors it but ships nothing, and -Cargo refuses `lto` in per-package overrides, so the choice is workspace-wide. -The Go mirror commits `libmoq_ffi.a` for four targets and fails its push at -100 MiB per file. Thin ran as `CARGO_PROFILE_RELEASE_LTO=thin` on `d55407256`, -x86_64 Linux host build, `nproc` 24, two jobs. - -Sizes (deterministic): - -| Artifact | Fat | Thin | Change | -| --- | ---: | ---: | ---: | -| `libmoq_ffi.a` (Go mirror, `--features openh264`) | 71.23 MiB | 81.07 MiB | +13.8% | -| `libmoq_ffi.so` | 33.88 MiB | 37.16 MiB | +9.7% | -| `moq` CLI | 48.62 MiB | 52.93 MiB | +9.0% | - -Both archives fit, but thin cuts the headroom under 100 MiB from 28.8 to -18.9 MiB. The mirror's v0.4.17 archives (fat, cross-built) are 62.3 MiB -(linux_amd64), 61.5 (linux_arm64), 54.4 (windows_amd64) and 40.5 -(darwin_arm64); the other targets were not built here. - -Build time, load 20 to 51 (load-sensitive; compare ratios): - -| Build | Fat wall | Thin wall | Thin/fat wall | Thin/fat CPU | -| --- | ---: | ---: | ---: | ---: | -| Clean `-p moq-ffi`, both modes at once | 544 s | 469 s | 0.86 | 1.00 | -| Then `-p moq-cli --bin moq`, both modes at once | 643 s | 545 s | 0.85 | 1.02 | -| `moq_ffi` final unit, interleaved, two rounds | 226, 256 s | 199, 158 s | 0.74 | 1.11 | -| `moq` final unit, interleaved, two rounds | 416, 404 s | 246, 240 s | 0.59 | 0.97 | - -Two jobs cap thin LTO's parallel backend at two threads, so a runner with -more cores would save more wall time than this; CPU time stays about even. - -Runtime (deterministic instruction counts, valgrind cachegrind on criterion -`--test`, which runs each case once; `perf stat` is blocked by -`perf_event_paranoid` 4): moq-net `session_delivery_viewers` (lite-06, -lite-07, moq-transport-22) 2,621M fat against 2,677M thin (+2.1%), and -`track_fanout` 414M against 421M (+1.6%). Repeats varied by 0.4% at most. - -Not measured: thin sizes for linux_arm64, darwin_arm64 and windows_amd64; -build time above two jobs; wall-clock runtime. - -Decision (maintainer, 2026-10-20): keep fat LTO, and revisit once -[moq-net compiles once](/quest/m1/net-erase-transport.md) shrinks the archive. -Thin saves about 40% of the final unit's wall time on two jobs, but release -builds run only nightly and on release PRs, off the merge-queue gate. It costs -10 MiB of the Go archive's headroom under a hard publish limit while that -archive is growing, and 2% more instructions. Revisit sooner if release -builds join the PR gate. - -### Results of the m0 run - -- Planned: [moq-net compiles once](/quest/m1/net-erase-transport.md) removes - the per-transport copies, the largest attributable dev IR (43% of moq-tokio) - and a share of the release fat-LTO module. -- Explained, no action: the release `moq` unit is all LLVM backend work on one - fat-LTO module; linking is 1 s. Thin LTO would cut that unit by about 40% - at a size and instruction cost (above); the maintainer kept fat LTO. - aws-lc-sys's build script is the longest dev unit; the crypto provider - stays. usage-rs derive output is external code. serde visitors per - deserializer are inherent to TOML config. -- Candidates for next run: mp4-atom's `Any::decode_atom` instantiated per - reader type (38k lines in moq-mux, an upstream change to decode from one - slice type); rerun the thin-LTO comparison if release builds join a PR gate. - -### Method - -Per target, `CARGO_TARGET_DIR=.scratch/target-lines CARGO_BUILD_JOBS=2 -MBX_DISABLE=1 cargo llvm-lines --locked -p --lib` (or `--bin`), then -group rows by the first path segment after stripping crate hashes, and by -substrings for the transport session types. Release: `cargo build --release --p moq-cli --timings` into its own target dir, then the time-passes command -above. Sample `/proc/loadavg` every 30 s during each run. diff --git a/quest/m2/README.md b/quest/m2/README.md index ec76e91552b..ab5a5914144 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -232,3 +232,4 @@ ranked just below deterministic TS legs, which it requires. - [Quest tree audit](/quest/m2/audit-quests.md) - repeat the whole-tree audit after the release ships, against the m1 baseline - [CI workflow audit](/quest/m2/audit-ci.md) - repeat the per-workflow and per-job review after the release ships, against the m1 baselines - [Performance audit](/quest/m2/audit-perf.md) - benchmarks that regressed since the m1 run are explained, fixed, or planned, and unbenched hot paths get coverage +- [Compile time audit](/quest/m2/audit-compile-time.md) - build-time hotspots are measured against the m1 baselines, starting with the transport share once moq-net compiles once, and the worst are fixed or planned diff --git a/quest/m2/audit-compile-time.md b/quest/m2/audit-compile-time.md new file mode 100644 index 00000000000..d7eaec71c9d --- /dev/null +++ b/quest/m2/audit-compile-time.md @@ -0,0 +1,129 @@ +# [M] Compile time audit + +## Goal + +The crates, dependencies, and code that dominate build time are known, with +recorded baselines, and the worst are fixed or planned. + +## Plan + +Recurs once per major release (decided 2026-10-19). The PR that finishes +this run also adds the next one with the same file name at the end of the next +milestone's `Required` (m0 to m1, m1 to m2). The successor keeps the Goal +and this paragraph; the rest of its Plan is rewritten with what this run +learned: baselines, results, and what to check first. A successor placed in m2 +also lists the [release cut](/quest/m2/release/cut.md) under `Required`, so it +waits for the release in flight to ship. Only `/quest-delete` ends the +recurrence. Fixes land in the run's PR, one commit each, rather than as follow-ups. +Only a change that needs a maintainer decision (public API, scope) or is +independently landable and too large to review alongside the rest becomes a +follow-up through `/quest-plan`. + +Scope: this audit owns per-crate build cost in one build (dev, release, and +incremental). The [CI workflow audit](/quest/m2/audit-ci.md) owns job-level CI +cost and caching; the [dependency audit](/quest/m2/audit-deps.md) owns +deduplicating and upgrading dependencies. + +What to check first: + +- [moq-dev/moq#6176](https://github.com/moq-dev/moq/pull/6176) (moq-net + compiles once) was still open at the m1 run. Once it lands, rerun + `cargo llvm-lines` on the moq-tokio lib and compare the transport rows + below, then rerun the thin-LTO comparison against the smaller + `libmoq_ffi.a`. On `d55407256` (m0, method in git history) thin grew the Go + mirror's x86_64 archive from 71.23 to 81.07 MiB under its 100 MiB per-file + limit, cut the final `moq` and `moq_ffi` units' wall by 26 to 41% at two + jobs with CPU time even, and cost 1.6 to 2.1% more instructions in the + moq-net session benches. Maintainer decision 2026-10-20: keep fat LTO + until the archive shrinks, and revisit sooner if release builds join the + PR gate. +- The last per-reader `Any::decode_atom` copy in moq-mux: `fmp4/text.rs` + decodes from a bare `&[u8]` (13,307 lines, 1.3% of the lib) while every + other fMP4 path uses `Cursor<&[u8]>`. It was skipped for open PR + [moq-dev/moq#6379](https://github.com/moq-dev/moq/pull/6379). +- Whether `cargo-llvm-lines` joined the pinned shell. `flake.nix` had four + open PRs at the m1 run, so it was not added. + +### Baselines (m1 run, 2026-10-23) + +Pinned Nix shell (Rust 1.98.1), Linux x86_64, 32 logical CPUs, Cargo bounded +to two jobs, `MBX_DISABLE=1`, one fresh target dir, registry present. The +host's `~/.cargo/config.toml` sets `incremental = false`, so every touched +crate rebuilds whole. The host was shared; load is the one-minute average +sampled every 30 s. Times are single observations: compare shares within a +run, never seconds across runs. Only the llvm-lines counts are deterministic. + +Builds on `0d3b3e084` (release with the moq-mux fix below): + +| Build | Load (median) | Wall | Unit time | Longest unit | +| --- | --- | ---: | ---: | --- | +| `cargo build --locked --timings` (default members, clean) | 4.1 to 9.0 (5.6) | 308 s | 614 s | aws-lc-sys build script 114 s, 18.6% | +| `cargo build --locked --release -p moq-cli --timings` (clean deps) | 3.1 to 17.7 (5.4) | 750 s | 1,127 s | `moq` bin 369 s, 49% of wall | +| Default members after a comment appended to `rs/moq-net/src/lib.rs` | 5.0 to 8.4 | 70 s | 126 s | moq-tokio 21.8 s, 17.4% | + +Clean dev: build scripts are 31% of unit time (aws-lc-sys alone 18.6%, ring +1.7%) and workspace crates 18.6%; the two-job queue stays full, so total +unit time sets the wall, not the critical path. Release: the final `moq` +unit is still fat LTO on one codegen unit (m0: 42 to 53% of the wall), and +aws-lc-sys is 11% of unit time. Incremental: a moq-net comment rebuilds +every dependent; moq-tokio (17.4%), the `moq` bin (15.6%), moq-net (10.9%), +moq-mux (9.3%) and moq-relay (8.6%) lead. + +`cargo llvm-lines` 0.4.48 (`nix shell nixpkgs#cargo-llvm-lines`), dev profile, +default features: + +| Target | LLVM IR lines | Copies | m0 lines | Notable share | +| --- | ---: | ---: | ---: | --- | +| moq-tokio lib | 1,679,222 | 48,553 | 1,661,503 | moq-net rows instantiated over a transport session 39.1% (any row naming one 55.7%); usage-rs derive 5.2% | +| moq-mux lib | 996,345 | 28,525 | 908,532 | serde 18.6%; mp4-atom 11.9% | +| moq-cli `moq` bin | 1,108,118 | 39,327 | 1,037,734 | tokio 16.6%; serde 5.2%; usage-rs derive 2.7% | +| moq-ffi lib | 811,228 | 32,309 | not run | tokio 18.6%; uniffi_core 9.6%; moq-mux generics 7.3% | +| moq-net lib | 799,404 | 25,306 | 686,443 | BTreeMap internals 29.5%; rows naming `moq_net::model` 47.7% | +| moq-relay lib | 667,136 | 24,636 | 711,268 | serde 30.4% (toml 9.3%); usage-rs derive 9.2% | +| moq-relay bin | 311,790 | 11,313 | 384,813 | tokio 18.9%; transport-session rows 3.0% | +| qmux lib | 12,528 | 524 | not run | generic, instantiated downstream | + +Transport rows: in moq-tokio, `qmux::session::Session` names 28.6% and +`moq_tokio::transport::Session` 28.2%, each again under +moq-net's IETF `ControlStreamAdapter` (20.0%), so the IETF half compiles four +times. The m0 share (43.2%) used an unrecorded grouping; compare against the +39.1% definition above (first path segment `moq_net`, name contains +`qmux::session::Session` or `moq_tokio::transport::Session`). + +moq-net grew 16.5% since m0, mostly in `model`; its BTreeMap share rose from +24.1% to 29.5% across about 30 distinct key and value types, each +instantiating the whole map. + +### Results of the m1 run + +- Fixed: moq-mux's fragment decode read through `Cursor<&Bytes>`, a fourth + instantiation of mp4-atom's `Any::decode_atom`; it now uses + `Cursor<&[u8]>` like the rest (moq-mux lib 1,006,004 to 996,345 lines). +- Waiting: the transport-generic share waits on + [moq-dev/moq#6176](https://github.com/moq-dev/moq/pull/6176), and the + thin-LTO comparison with it. +- Explained, no action: aws-lc-sys's build script remains the longest unit + in clean builds; the crypto provider stays (m0). usage-rs derive output + scales with config fields and is external code. serde visitors per + deserializer are inherent to TOML config. moq-rtmp, moq-srt and moq-rtc + entry points generic over `impl AsPath` instantiate their async bodies in + the `moq` bin (20,327 lines with drop glue, 1.8%); a non-generic inner + function would move them, but adds code for little. The catalog JSON diff instantiated + per crate is [catalog publish once](/quest/m2/catalog-publish-once.md). +- Not run this time: `-Z time-passes` on the `moq` unit (m0: fat LTO 74%, + LLVM passes 23%); the release share did not move enough to warrant it. + +### Method + +Per target, `CARGO_TARGET_DIR=.scratch/target CARGO_BUILD_JOBS=2 +MBX_DISABLE=1 cargo llvm-lines --locked -p --lib` (or `--bin`; name +`qmux@` since a dev-dependency pins an older qmux), then group rows +by the first path segment after stripping crate hashes, and by substrings for +the transport session types. Clean dev and release builds use `--timings` +into the same fresh target dir; the incremental probe appends one comment +line to `rs/moq-net/src/lib.rs` after a no-op catch-up build. Sample +`/proc/loadavg` every 30 s during each run. + +## Required + +- [Release cut](/quest/m2/release/cut.md) - the release in flight ships before this run starts