Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,10 @@ jobs:
x11:
uses: ./.github/workflows/x11.yml

# Each independent workload gets its own hour. Rust and JS benchmark groups
# keep one build environment each, so splitting the feature compile does not
# turn every benchmark into a fresh compile of the same dependencies.
# Each independent workload gets its own hour. Each benchmark group keeps one
# build environment, so a benchmark reuses the dependencies its group already
# compiled. The Rust benchmarks outgrew one hour, so they split by harness:
# nextest's ignored benchmark tests and cargo's bench targets share few builds.
nightly:
name: Nightly (${{ matrix.name }})
runs-on: ubuntu-24.04-arm
Expand All @@ -74,8 +75,10 @@ jobs:
- name: Relay lock-wait profile
recipe: bench-lock fanout-heavy 4
perf: true
- name: Rust benchmarks
recipe: gh nightly rust
- name: Rust benchmark tests
recipe: gh nightly rust-test
- name: Rust bench targets
recipe: gh nightly rust-bench
- name: JS benchmarks
recipe: gh nightly js
uncached: true
Expand Down
1 change: 0 additions & 1 deletion quest/m0/ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,5 @@ The historical samples locate candidates; they are not controlled benchmarks.
- [Require platform gates](/quest/m0/ci/platform-queue.md) - activate the consolidated platform checks in the main and release rulesets after the workflow lands
- [mbx on R2](/quest/m0/ci/mbx-r2.md) - Rust builds on every OS read an R2 cache that only `main` writes, escaping the 10 GB Actions cache
- [Nix cache](/quest/m0/ci/nix-cache.md) - `nix.moq.dev` on R2 replaces Cachix for CI and releases
- [Rust benchmark group fits its hour](/quest/m0/ci/nightly-rust-bench.md) - Nightly's Rust benchmark workload finishes inside its job limit
- [Nightly workloads pass](/quest/m0/ci/nightly-green.md) - the Nightly failures found on 2026-10-16 are fixed
- [First shared FFI release](/quest/m0/ci/nightly-ffi-reuse.md) - condition: the first `moq-ffi-v*` tag that runs `Release FFI`, and a `Shipping sizes` run; verify the wrappers and size keys
1 change: 0 additions & 1 deletion quest/m0/ci/nightly-green.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,5 +39,4 @@ have not run yet; triage them from the next Nightly too.

## Related

- [Rust benchmark group fits its hour](/quest/m0/ci/nightly-rust-bench.md) - the cancelled Rust group
- [First shared FFI release](/quest/m0/ci/nightly-ffi-reuse.md) - the check that found these
37 changes: 0 additions & 37 deletions quest/m0/ci/nightly-rust-bench.md

This file was deleted.

6 changes: 3 additions & 3 deletions quest/m2/bench-site.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ Decided with the maintainer on 2026-10-15:
- **GitHub Pages via github-action-benchmark**: the nightly pushes each
run's results to a `gh-pages` branch, and the action's built-in charts are
the site. Its own alerting stays off; tracking is informational.
- **Sources**: the binding benchmark's JSON and the "Rust benchmarks" and
"JS benchmarks" nightly jobs, converted to the action's input format where
they don't already emit one. Several are smoke runs today: most Rust
- **Sources**: the binding benchmark's JSON and the "Rust benchmark tests",
"Rust bench targets" and "JS benchmarks" nightly jobs, converted to the
action's input format where they don't already emit one. Several are smoke runs today: most Rust
Criterion targets run with `-- --test`, which measures nothing. Switch the
charted targets to a measuring mode (or add a measurement pass), weighing
the added nightly runtime, and check each JS bench reports a measurement
Expand Down
24 changes: 14 additions & 10 deletions sh/gh/nightly.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Keep benchmark builds together while unrelated Nightly workloads run independently.
set -euo pipefail

usage='usage: just gh nightly rust|js'
usage='usage: just gh nightly rust-test|rust-bench|js'
if [[ $# != 1 ]]; then
echo "$usage" >&2
exit 2
Expand All @@ -15,16 +15,27 @@ run() {
}

case "${1:-}" in
rust)
# Ignored benchmark tests under nextest: the release builds first, then the test-profile ones.
rust-test)
run just rs bench-latency
run just rs test -p moq-bench --release --run-ignored only \
-E 'test(benchmark_readiness_establishment)' --success-output immediate
run just rs bench-playback
run just rs test --release -p moq-quic -E 'test(benchmark_ack_cleanup)' \
--run-ignored only --success-output immediate
run just rs bench-feed-fetch
run just rs test -p moq-relay -E 'test(benchmark_stun_sources)' --run-ignored only --success-output immediate
run just rs test -p moq-quic --release -E 'test(benchmark_receive_credit)' --run-ignored only --success-output immediate
run just rs test -p moq-relay -E 'test(benchmark_stun_sources)' --run-ignored only --success-output immediate
run just rs test -p moq-archive -p moq-mux -p moq-hls --no-fail-fast \
-E 'test(benchmark_timeline_fetch_fanout) | test(benchmark_damage_pid_fanout) | test(benchmark_pes_staging) | test(benchmark_reference_handoff_fanout)' \
--run-ignored only --success-output immediate
# Codecs are irrelevant to timebase reservation, so skip building them.
run just rs test -p moq-ffi --no-default-features -E 'test(benchmark_timebase_reservation_fanout)' \
--run-ignored only --success-output immediate
run just rs test -p moq-cli -E 'test(benchmark_stream_import)' --run-ignored only --success-output immediate
;;
# Cargo bench targets, all in the bench profile.
rust-bench)
# Fuzz exposes moq-net's codec benches; it is not a feature of the other crates.
run cargo bench --locked --no-fail-fast -p moq-net --features fuzz --bench '*' -- --test
# Exercise the optional thread-CPU clock as well as the normal elapsed-time smoke.
Expand All @@ -39,15 +50,8 @@ case "${1:-}" in
--bench catalog --bench ts_schedule --bench export --bench containers --bench matching \
--bench chunk_interleaving --bench worker_turns \
--bench streams --bench receive --bench endpoint --bench wire -- --test
run just rs test -p moq-archive -p moq-mux -p moq-hls --no-fail-fast \
-E 'test(benchmark_timeline_fetch_fanout) | test(benchmark_damage_pid_fanout) | test(benchmark_pes_staging) | test(benchmark_reference_handoff_fanout)' \
--run-ignored only --success-output immediate
# Codecs are irrelevant to timebase reservation, so skip building them.
run just rs test -p moq-ffi --no-default-features -E 'test(benchmark_timebase_reservation_fanout)' \
--run-ignored only --success-output immediate
# moq-gst's catalog bench shares a target name with moq-mux's, so it runs on its own.
run just rs bench-gst-catalog -- --quick
run just rs test -p moq-cli -E 'test(benchmark_stream_import)' --run-ignored only --success-output immediate
;;
js)
for benchmark in broadcasts request-ids requests forward reader frames varint lite-varint track interests priority; do
Expand Down
21 changes: 15 additions & 6 deletions sh/gh/nightly.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ def test_failed_js_benchmark_does_not_hide_later_work(self):
self.assertEqual(calls[-1], "just js signals-bench")
self.assertIn("failed: just js bench reader", result.stderr)

def test_release_benchmarks_share_one_invocation(self):
result, calls = self.run_group("rust")
def test_bench_targets_share_one_invocation(self):
result, calls = self.run_group("rust-bench")
self.assertEqual(result.returncode, 0, result.stderr)
release = [call for call in calls if call.startswith("cargo bench")]
self.assertEqual(len(release), 4) # moq-net, its CPU clock, kio, then the shared build.
Expand All @@ -53,16 +53,25 @@ def test_release_benchmarks_share_one_invocation(self):
self.assertIn("--bench containers", shared[0])
self.assertIn("-p moq-quic", shared[0])
self.assertIn("--bench wire", shared[0])
self.assertIn("just rs bench-gst-catalog -- --quick", calls)
self.assertEqual(calls[-1], "just rs bench-gst-catalog -- --quick")

def test_benchmark_tests_stay_out_of_cargo_bench(self):
result, calls = self.run_group("rust-test")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(any(call.startswith("cargo bench") for call in calls))
# Release builds run before the test-profile ones.
profiles = ["--release" in call or " bench-" in call for call in calls]
self.assertEqual(profiles, sorted(profiles, reverse=True))
self.assertIn("just rs bench-feed-fetch", calls)
self.assertTrue(calls[-1].startswith("just rs test -p moq-cli"))

def test_failed_release_build_does_not_hide_later_work(self):
_, calls = self.run_group("rust")
_, calls = self.run_group("rust-bench")
build = next(call.removeprefix("cargo ") for call in calls
if call.startswith("cargo bench") and "-p moq-relay" in call)
result, calls = self.run_group("rust", build)
result, calls = self.run_group("rust-bench", build)
self.assertEqual(result.returncode, 1)
self.assertTrue(calls[-1].startswith("just rs test -p moq-cli"))
self.assertEqual(calls[-1], "just rs bench-gst-catalog -- --quick")

def test_unknown_group_fails_before_running_anything(self):
result, calls = self.run_group("unknown")
Expand Down
Loading