Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .config/nextest.toml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,12 @@ slow-timeout = { period = "60s", terminate-after = 3 }
io-uring = { max-threads = 4 }
v4l2 = { max-threads = 1 }

# Nightly's ignored benchmark tests (`just gh nightly rust-test`) keep each test's
# printed tables in target/nextest/bench/junit.xml for the benchmark report.
[profile.bench.junit]
path = "junit.xml"
store-success-output = true

# CI has noisier neighbours and cold caches, so give a test longer before
# calling it wedged.
[profile.ci]
Expand Down
20 changes: 19 additions & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,9 @@ jobs:
include:
- name: Rust benchmark tests
recipe: gh nightly rust-test
artifact: bench-rust-test
artifact-path: target/bench/nightly/rust-test
benchmark: true
- name: Binding benchmark runners
recipe: gh nightly bindings
- name: Rust bench targets
Expand Down Expand Up @@ -244,7 +247,7 @@ jobs:
# Explicit Pages deployment works with GITHUB_TOKEN; a branch push alone does not.
benchmark-site:
name: Benchmark site
needs: nightly
needs: [nightly, sizes]
if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
runs-on: ubuntu-24.04-arm
timeout-minutes: 15
Expand Down Expand Up @@ -671,6 +674,21 @@ jobs:
size-report/js/*.metafile.json
retention-days: 45
if-no-files-found: error
# The benchmark table records tonight's sizes even when the comparison fails on growth.
- name: Convert sizes for the benchmark table
id: size-table
if: ${{ !cancelled() && hashFiles('size-report/sizes.json') != '' }}
run: nix develop --command just gh benchmark-report size size-report target/bench/size/report.json
- name: Record benchmark outcome
if: ${{ !cancelled() }}
run: nix develop --command just gh benchmark-report status "${{ steps.size-table.outcome }}" target/bench/size
- name: Preserve benchmark report
if: ${{ !cancelled() }}
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: bench-size
path: target/bench/size
retention-days: 45

fuzz:
name: Fuzz (${{ matrix.target }})
Expand Down
27 changes: 19 additions & 8 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,8 @@ throttled one:
nix develop --command just rs test -p moq-tokio --release -E 'test(benchmark_datagram_delivery)' --run-ignored only --success-output immediate
```

It reports offered, received and lost datagrams, bottleneck queue drops, reordered
arrivals and append-to-receive latency per viewer, and fails if the loopback
It reports offered and lost datagrams, bottleneck queue drops, QUIC DATAGRAM frames
that delivered no datagram, reordered arrivals and append-to-receive latency per viewer, and fails if the loopback
viewer loses any. It runs nightly.

Measure 50 fps audio at per-packet, 100 and 200 ms grouping, plus 400 fps
Expand Down Expand Up @@ -620,16 +620,27 @@ The table includes these measurements:
time elapsed time. The Rust bench target jobs measure them with a short warmup
and Criterion's quick mode. Per-group benchmark settings can override the default warmup. Quick estimates
trade precision for nightly runtime; the paired comparison below assesses a change.
- JS benchmark CSV time columns (`ns`, `us`, `ms`, and their per-operation
variants) and `size_bytes` columns, retaining every other column as a workload
dimension in each series name.
- The CSV tables the JS benchmarks and the Rust benchmark tests print. A header
line names the columns; the time columns (`ns`, `us`, `ms`, and their
per-operation variants such as `ns_per_frame` or `plain_ns_per_read`),
`size_bytes` and `peak_bytes`, `mib_per_s`, and `*_packets` columns are
measurements, and every other column is a workload dimension in each series
name, so print outcomes only in measured columns. The Rust benchmark tests
(`just gh nightly rust-test`) run in release builds, one at a time, under
nextest's `bench` profile, which keeps each test's output in a JUnit report;
their series are named by package and test path, such as
`rust-test/moq-congestion/gcc/bench/benchmark_per_packet`.
- Shipping sizes, in bytes: the release artifacts, minimal FFI build, and JS
bundles the [size report](../sh/size/README.md) measures, recorded even on a
night its growth check fails. Release build times stay out.

Hosted runners vary across nights, so a trend flag prompts a look at the paired
comparison rather than proving a regression.
Malformed, empty, nonfinite, or duplicate measurements fail validation. The
nightly artifacts retain the reports, raw JS output, and each Criterion run's
metadata and estimates for 45 days. Relay load, audio quality, and the Rust
benchmark tests job's nextest output do not enter the table.
nightly artifacts retain the reports, raw JS output, nextest JUnit reports, and
each Criterion run's metadata and estimates for 45 days. Relay load, audio
quality, and the binding runners (measured in rust2, above) do not enter the
table.

Render the table from the published history:

Expand Down
35 changes: 17 additions & 18 deletions quest/m2/bench-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,23 @@ is a smoke run that measures nothing.
## Plan

Split from the benchmark trend site on 2026-10-20, when that quest became
the m0 table. Sources decided with the maintainer on 2026-10-15: the binding
benchmark's JSON and the "Rust benchmark tests", "Rust bench targets" and "JS
benchmarks" nightly jobs, converted to the history input format where they
don't already emit one.
the m0 table. The Criterion targets, the JS benchmarks, the Rust benchmark
tests and the nightly shipping sizes feed it now
([moq-dev/moq#7510](https://github.com/moq-dev/moq/pull/7510) added the last
two). The binding runners stay an unmeasured smoke run: rust2 measures them,
and [moq-dev/moq#6629](https://github.com/moq-dev/moq/pull/6629) retired their
series. The relay load benches left Nightly in
[moq-dev/moq#7603](https://github.com/moq-dev/moq/pull/7603). What remains joins
through the same input format; the sources decided with the maintainer on
2026-10-15 left it for later:

- Criterion targets now measure through `just gh criterion` in
`sh/gh/nightly.sh` (`rust-bench`, `rust-shared`, `rust-extra`, `rust-uring`), and the
comparison harness uses quick mode. Check their published histories
rather than converting them again; the unreported sources below remain.
- The "Rust benchmark tests" job is not recorded yet: its nextest benchmarks
print no structured measurements.
- Check each JS bench reports a measurement before recording it.
- Nightly size history (today a 45-day artifact) is a natural extra series
if it fits cheaply.
- The audio-quality job's benchmarks can join later through the same input
format.
- The paired nightly comparison publishes beside the table from the same
`gh-pages` branch (`bench/paired.json`); a new Rust source joins it through
`just gh benchmark-report rust`.
- The audio-quality job's benchmarks: `just js qmux-bench`, `serve-bench` and
`ring-bench`. Check each prints a CSV table the `js` converter reads before
recording it.

A source joins as a `bench-*` artifact with a recorded outcome, which the
benchmark-site job collects. The paired nightly comparison publishes beside the
table from the same `gh-pages` branch (`bench/paired.json`); a new Rust source
joins it through `just gh benchmark-report rust`.

Public API: none. Wire: none.
5 changes: 4 additions & 1 deletion rs/moq-audio/src/encode/producer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -707,6 +707,7 @@ mod tests {
#[tokio::test]
#[ignore = "measures real codec CPU; run with just rs bench-playback"]
async fn benchmark_raw_audio() {
println!("tracks,retention_secs,always_on,idle_ms,catchup_ms");
for tracks in [1, 8] {
for seconds in [1, 30] {
for active in [true, false] {
Expand Down Expand Up @@ -747,7 +748,9 @@ mod tests {
}
let catchup = start.elapsed();
println!(
"tracks={tracks} retention={seconds}s always_on={active}: idle={idle:?} catchup={catchup:?}"
"{tracks},{seconds},{active},{:.3},{:.3}",
idle.as_secs_f64() * 1e3,
catchup.as_secs_f64() * 1e3
);
for producer in &mut producers {
producer.finish().unwrap();
Expand Down
6 changes: 2 additions & 4 deletions rs/moq-audio/src/playback/mixer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -497,6 +497,7 @@ mod tests {
reason = "the benchmark measures mixer callbacks on the host clock"
)]
fn benchmark_mixer_callbacks() {
println!("sinks,frames,ns_per_callback");
for sinks in [1, 8, 32, MAX_SINKS] {
for frames in [128, 480, 1024, 2048] {
let mut harness = Harness::with_depth(Layout::Stereo, MAX_SINKS);
Expand All @@ -514,10 +515,7 @@ mod tests {
elapsed += start.elapsed();
std::hint::black_box(&out);
}
println!(
"sinks={sinks} frames={frames}: {:.1} ns/callback",
elapsed.as_nanos() as f64 / 1000.0
);
println!("{sinks},{frames},{:.1}", elapsed.as_nanos() as f64 / 1000.0);
}
}
}
Expand Down
3 changes: 2 additions & 1 deletion rs/moq-audio/src/resample.rs
Original file line number Diff line number Diff line change
Expand Up @@ -539,6 +539,7 @@ mod tests {
#[test]
#[ignore = "run with just rs bench-playback"]
fn benchmark_resampler() {
println!("input_rate,output_rate,channels,chunk,packet,ns_per_input_frame");
for (input_rate, output_rate) in [(44_100, 48_000), (48_000, 44_100), (96_000, 48_000)] {
for channels in [1, 2, 6] {
let mut chunks = vec![input_rate as usize / 50, (input_rate as usize / 50).clamp(1, 960)];
Expand Down Expand Up @@ -567,7 +568,7 @@ mod tests {
assert_eq!(count, output_rate as usize * channels as usize);
}
println!(
"{input_rate}->{output_rate} channels={channels} chunk={chunk} packet={packet}: {:.1} ns/input-frame",
"{input_rate},{output_rate},{channels},{chunk},{packet},{:.1}",
elapsed.as_nanos() as f64 / (10 * frames) as f64
);
}
Expand Down
3 changes: 2 additions & 1 deletion rs/moq-bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -292,4 +292,5 @@ to the results.

`just rs bench-latency` measures latency recording with 1, 4 and 16 concurrent
writers and 0, 10 and 100 snapshots per run. It checks final sample conservation
and prints throughput for each case. The nightly workflow runs the same matrix.
and prints the cost per sample for each case. The nightly workflow runs the same
matrix and records it on the benchmark table.
6 changes: 2 additions & 4 deletions rs/moq-bench/src/connection.rs
Original file line number Diff line number Diff line change
Expand Up @@ -766,6 +766,7 @@ mod tests {
reason = "the benchmark measures real establishment time"
)]
fn benchmark_readiness_establishment() {
println!("connections,broadcasts_per_connection,elapsed_ns");
for connections in [1, 16, 128] {
for broadcasts in [1, 16, 128] {
let count = connections * broadcasts;
Expand Down Expand Up @@ -793,10 +794,7 @@ mod tests {
}
}
assert!(std::hint::black_box(stats.ready()));
println!(
"readiness connections={connections} broadcasts_per_connection={broadcasts} elapsed_ns={}",
start.elapsed().as_nanos()
);
println!("{connections},{broadcasts},{}", start.elapsed().as_nanos());
assert_eq!(stats.broadcasts_established.load(Ordering::Relaxed), count);
}
}
Expand Down
7 changes: 3 additions & 4 deletions rs/moq-bench/src/stats.rs
Original file line number Diff line number Diff line change
Expand Up @@ -788,6 +788,7 @@ mod tests {
use std::sync::Barrier;
use std::time::Instant;

println!("writers,snapshots,ns_per_sample");
for writers in [1, 4, 16] {
for snapshots in [0, 10, 100] {
let stats = Stats::default();
Expand Down Expand Up @@ -821,10 +822,8 @@ mod tests {
let elapsed = elapsed.elapsed();
assert_eq!(Snapshot::take(&stats).latency_samples, (writers * samples) as u64);
println!(
"writers={writers} snapshots={snapshots} samples={} elapsed_ns={} samples_per_second={:.0}",
writers * samples,
elapsed.as_nanos(),
(writers * samples) as f64 / elapsed.as_secs_f64()
"{writers},{snapshots},{:.2}",
elapsed.as_nanos() as f64 / (writers * samples) as f64
);
}
}
Expand Down
5 changes: 3 additions & 2 deletions rs/moq-c/src/id.rs
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ mod tests {
fn benchmark_handles() {
use std::hint::black_box;

println!("live,wrap,ns_per_iteration");
for live in [0, 64, 4096] {
for wrap in [false, true] {
let next = AtomicU32::new(live + 1);
Expand All @@ -166,8 +167,8 @@ mod tests {
let id = slab.insert_next(black_box(42), &next).unwrap();
black_box(slab.remove(id).unwrap());
}
eprintln!(
"handles live={live} wrap={wrap}: {:.0} ns/iteration",
println!(
"{live},{wrap},{:.0}",
start.elapsed().as_nanos() as f64 / f64::from(iterations)
);
}
Expand Down
6 changes: 2 additions & 4 deletions rs/moq-cli/src/publish.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1988,6 +1988,7 @@ pub(crate) mod tests {
let mut null = [0xff; 188];
null[..4].copy_from_slice(&[0x47, 0x1f, 0xff, 0x10]);
let padding = null.repeat(INPUT.len() / 188);
println!("copies,ns_per_byte");
for copies in [1, 16, 128] {
let mut file = tempfile::NamedTempFile::new().unwrap();
file.write_all(INPUT).unwrap();
Expand All @@ -2012,10 +2013,7 @@ pub(crate) mod tests {
decode(decoder, catalog, input, Publication::default()).await.unwrap();
let elapsed = start.elapsed();
let bytes = INPUT.len() * copies;
println!(
"bytes={bytes}: {elapsed:?}, {:.2} Mbit/s",
bytes as f64 * 8.0 / elapsed.as_secs_f64() / 1e6
);
println!("{copies},{:.3}", elapsed.as_nanos() as f64 / bytes as f64);
}
}

Expand Down
11 changes: 5 additions & 6 deletions rs/moq-congestion/src/bbr3/bench.rs
Original file line number Diff line number Diff line change
Expand Up @@ -90,11 +90,10 @@ fn measure(flight: usize, batch: usize, steady: bool) -> (Duration, usize) {
#[test]
#[ignore = "controller benchmark: run with --release and --success-output immediate"]
fn benchmark_per_packet() {
println!(
"bbr_packet controller_stack_bytes={} packet_state_bytes={}",
size_of::<Bbr3>(),
size_of::<Packet>()
);
println!("state,size_bytes");
println!("controller_stack,{}", size_of::<Bbr3>());
println!("packet,{}", size_of::<Packet>());
println!("steady,flight,batch,cpu_ns,peak_bytes");
for steady in [false, true] {
for flight in [16, 64, 1000, 4000, 16000] {
for batch in [1, 8, 32] {
Expand All @@ -107,7 +106,7 @@ fn benchmark_per_packet() {
}
samples.sort();
println!(
"bbr_packet steady={steady} flight={flight} batch={batch} cpu_ns={} peak_bytes={bytes}",
"{steady},{flight},{batch},{},{bytes}",
samples[1].as_nanos()
);
}
Expand Down
9 changes: 4 additions & 5 deletions rs/moq-congestion/src/gcc/bench.rs
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,9 @@ fn measure(flight: usize, batch: usize, steady: bool) -> Duration {
#[test]
#[ignore = "controller benchmark: run with --release and --success-output immediate"]
fn benchmark_per_packet() {
println!("gcc_packet controller_stack_bytes={}", size_of::<Gcc>());
println!("state,size_bytes");
println!("controller_stack,{}", size_of::<Gcc>());
println!("steady,flight,batch,cpu_ns");
for steady in [false, true] {
for flight in [16, 64, 1000, 4000, 16000] {
for batch in [1, 8, 32] {
Expand All @@ -84,10 +86,7 @@ fn benchmark_per_packet() {
*sample = measure(flight, batch, steady);
}
samples.sort();
println!(
"gcc_packet steady={steady} flight={flight} batch={batch} cpu_ns={}",
samples[1].as_nanos()
);
println!("{steady},{flight},{batch},{}", samples[1].as_nanos());
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions rs/moq-ffi/src/media.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2086,7 +2086,7 @@ mod timebase_tests {
#[test]
#[ignore = "benchmark, runs in Nightly"]
fn benchmark_timebase_reservation_fanout() {
eprintln!("timebases,containers,ns_per_container");
println!("timebases,containers,ns_per_container");
for entries in [1, 32, 1024] {
let owner = MoqBroadcastProducer::new().unwrap();
let timebases: Vec<_> = (0..entries)
Expand Down Expand Up @@ -2115,7 +2115,7 @@ mod timebase_tests {
}
}
let nanos = start.elapsed().as_nanos() as f64 / (1000 * touched) as f64;
eprintln!("{entries},{touched},{nanos:.1}");
println!("{entries},{touched},{nanos:.1}");
}
owner.close().unwrap();
}
Expand Down
4 changes: 2 additions & 2 deletions rs/moq-mux/src/container/ts/damage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ mod tests {
)]
async fn benchmark_damage_pid_fanout() {
let error = anyhow::anyhow!("broken PES");
eprintln!("pids,damaged_units,ns_per_unit");
println!("pids,damaged_units,ns_per_unit");
for pids in [1_u16, 64, 8192] {
for units in [1, 16, 128] {
let mut log = Log::default();
Expand All @@ -108,7 +108,7 @@ mod tests {
}
}
let nanos = start.elapsed().as_nanos() as f64 / (10_000 * units) as f64;
eprintln!("{pids},{units},{nanos:.1}");
println!("{pids},{units},{nanos:.1}");
}
}
}
Expand Down
Loading
Loading