Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,8 @@ source files. The retained path is printed at the end, including on failure.
Choose an evidence parent outside disposable worktrees when the report must survive
worktree removal. Criterion case discovery uses isolated short measurements and
their full JSON IDs, since `--list` truncates display names; discovery samples
are retained separately and never used in the comparison.
are retained separately and never used in the comparison. A target that writes
no Criterion samples, such as one that prints its own table, is listed as skipped.

Each comparison reports the median of the paired percentage deltas, their median
absolute deviation and range, and flags fewer than three observations or a sign
Expand Down
5 changes: 5 additions & 0 deletions bench/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -435,6 +435,11 @@ criterion_cases() {
CRITERION_ARGS=(--warm-up-time 0.001 --measurement-time 0.001 --sample-size 10 --nresamples 1 --noplot)
CRITERION_HOME=$discovery run_criterion_target "$checkout" "$manifest" "$target" --save-baseline discovery
) >&2 || return 1
# A harness = false target without Criterion, such as moq-cli's play_buffer, writes no samples.
if [[ -z $(find "$discovery" -name benchmark.json -print -quit 2>/dev/null) ]]; then
printf 'Not a Criterion target, skipped: %s %s\n' "$manifest" "$target" >&2
continue
fi
python3 "$ROOT/bench/compare.py" --criterion-cases "$discovery" >"$RUN/criterion-listing" || return 1
while IFS= read -r benchmark; do
printf '%s\t%s\t%s\n' "$manifest" "$target" "$benchmark"
Expand Down
98 changes: 83 additions & 15 deletions quest/m1/release-perf/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# [M] Main is no slower than release
# Main is no slower than release

## Goal

Expand All @@ -7,12 +7,11 @@ Before the next release cut, every benchmark where `main` is slower than
is fixed. Cost that pays for a feature `main` deliberately added is accepted,
and the reason is written here.

This epic's own work: once every child has landed, run the full main against
release comparison again with `just bench origin/release`, plus the JS frame
path in Chromium with `bench/browser/micro.ts`, and confirm nothing new
regressed. Then add a release-baseline step to the recurring
[performance audit](/quest/m2/audit-perf.md), so every later cut
repeats this comparison.
This epic's own work: once every child has landed, confirm the sparse rows
and the relay fanout load against `release` (see the 2026-10-25 decisions).
The browser frame path needs no repeat; the
[performance audit](/quest/m2/audit-perf.md) now pairs every later cut with
`release`.

## Plan

Expand Down Expand Up @@ -74,8 +73,7 @@ noise.
one-frame import's instructions are its zero-fill), and the encoders already
build that state lazily on first use. It is paid once per broadcast: the
import ratio falls from 3.97x at one frame to 1.49x at 64. TS export's
growth to 2.09x at `4x64` is in the export reap that
[moq-dev/moq#6281](https://github.com/moq-dev/moq/pull/6281) changes.
growth with track count is its T-STD schedule (see the final run).
- The model's remaining per-group and per-frame cost is accepted, after
[moq-dev/moq#6294](https://github.com/moq-dev/moq/pull/6294) cut the
accidental part (a second track upgrade per first frame, a 128-bit division
Expand Down Expand Up @@ -137,7 +135,7 @@ the moq-net and kio rows include the fixes above.
| `track_fanout_frame/1` | 1.42x | 1.86x | accepted (above) |
| `track_empty_run/{fresh/1,stale/8}/512` | median 1.36x | 1.53x | accepted (above) |
| `track_datagram_subscriber_join/0d/1`, `64d/64` | median 2.3x | 2.12x, 3.55x | accepted (above) |
| `track_sparse_read/timed/4096` | 4.3x | | [moq-dev/moq#6287](https://github.com/moq-dev/moq/pull/6287) |
| `track_sparse_read/timed/4096` | 4.3x | | fixed in [moq-dev/moq#6287](https://github.com/moq-dev/moq/pull/6287), regressed again (final run) |
| `origin/announce_duplicate`, `1d_1s` to `240d_240s` | 4.3x | 1.06x to 1.16x | noise |
| `session_delivery_{relays,paced,scale}` | 1.43x to 1.56x | 1.28x to 1.77x | accepted (above) |
| moq-stats `aggregate`, live=16, departed=4096 | 2.1x | 1.05x | noise |
Expand All @@ -156,11 +154,81 @@ releasing a finished stream's reader lock, which built a `TypeError` per
group stream. Every row then sits within the A/A spread of release.
The group write/read microbench stays 3 to 12% above release against an
A/A of 3 to 8% on the same host. Decided 2026-10-23 (maintainer): browser
parity accepted on this evidence; the final re-run above checks it again.

The raw report lived in a scratch directory that has since been removed.
`just bench <base>` now pairs full Criterion IDs; point `BENCH_ARTIFACTS`
outside the worktree to keep the next one.
parity accepted on this evidence; the 2026-10-25 run confirmed it.

### Final run (2026-10-25)

`release` (`3db33f6715`) against `main` (`018f1c46e5`): 1,019 paired
Criterion cases, three relay load pairings, and nine Chromium frame rounds
with an A/A pair, at host load 20 to 60. Each slower case was re-run in
alternating rounds and counted in callgrind instructions.

Fixed in [moq-dev/moq#6784](https://github.com/moq-dev/moq/pull/6784):

- The MKV and FLV exports rebuilt the jitter clock's expected track set on
every frame; they now set it once per catalog update.
`container-export/mkv/16x64` went from 27.1M to 17.6M instructions (2.27x to
1.47x release, flat across the sweep), `flv/16x64` 14.5M to 12.9M.
- The TS export and its T-STD schedule hash PIDs with FxHash:
`container-export/ts/16x64` 30.8M to 25.5M (2.70x to 2.24x).
- Benches that broke since release now run: the moq-mux catalog bench caps its
renditions, the relay auth bench reads its observed state directly, moq-stats
`producer_tick` skips tiers above one at 50,000 paths, and `bench/run.sh`
skips targets that are not Criterion.

Accepted, flat per operation:

- e2ee `protect` is 1.5x release on grouped frames and 1.21x on datagrams. The
crypto is unchanged; the rest is the per-group model cost accepted above.
Both branches expand the AES-GCM key on every frame, about a quarter of
`protect`, which the [performance audit](/quest/m2/audit-perf.md) owns.
- gst `moqsink/eos` is +12% instructions per event, from
[#5512](https://github.com/moq-dev/moq/pull/5512) and the gstreamer-rs 0.25
bump in [#6169](https://github.com/moq-dev/moq/pull/6169).
- `group_write_frames` (1.67x) and `track_stamp_groups/sparse` (1.48x, down
from 4.4x) are the per-frame and per-group costs accepted above, as are the
session join, burst, and delivery families (1.2x to 1.76x).
`session_dash_relays` is the same per-track cost: 1.50x instructions at 768
tracks over 2 or 3 relays, and 0.63x over 33 relays, so it falls with relay
count. The import fixed costs above (`annexb-import`, `cmaf-import` at one
frame) and the IETF `/fetch` delivery rows also reappear unchanged.
- The TS export still grows with track count (1.50x at `1x1` to 2.24x at
`16x64`). The T-STD schedule scans and sorts every queued unit per PCR slot,
which the [performance audit](/quest/m2/audit-perf.md) owns.

Noise: `frame_recv/upfront/size=262144/streams=64` is 3 to 4x in wall time at
0.96x instructions. Its 1.6M page faults (against 192k) are glibc returning
the freed 256 KiB buffers between iterations; with
`GLIBC_TUNABLES=glibc.malloc.mmap_threshold=1048576:glibc.malloc.trim_threshold=1073741824`
both sides fault 4.6k times and time the same, and the relay runs jemalloc.
`session_idle` IETF at 1024 sessions, `origin/resume_overlap`,
`origin/retire`, `origin/reprice_duplicate` (0.55x instructions), kio
`contended_notify`, and moq-stats `producer/merge` are at parity in
alternating rounds or instructions.

Relay load: video used 7.8% less relay CPU with the same delivery. Fanout used
16.4% more, but its paired deltas were -1.5%, +16.4%, and +26.1% (MAD 9.7%)
at load 25, and the 2026-10-21 run measured 12% less, so it needs a quiet
re-run. Browser frames: lite rows 1.09 to 1.14x and IETF 0.95x, the same as
`main` at the 2026-10-23 decision (1.13x lite).

Decided 2026-10-25 (maintainer), in
[moq-dev/moq#6784](https://github.com/moq-dev/moq/pull/6784):

- The sparse read slope from
[dynamic range requests](https://github.com/moq-dev/moq/pull/6171) is the
child below, and it blocks the cut.
- The TS schedule's growth is left to the
[performance audit](/quest/m2/audit-perf.md); it does not block the cut.
- Once the child lands, re-run the relay fanout pairing on a quiet host,
recording `/proc/loadavg`, with
`BENCH_ARTIFACTS=<scratch> just bench origin/release` (it also re-checks
the sparse rows). Fix it if the relay
CPU still rises with consistent sign; otherwise this epic is done.

## Required

- [Sparse reads judge misses incrementally](/quest/m1/release-perf/sparse-misses.md) - `track_sparse_read` stays flat across gaps, as before range requests

## Related

Expand Down
33 changes: 33 additions & 0 deletions quest/m1/release-perf/sparse-misses.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# [M] Sparse reads judge misses incrementally

## Goal

A reader of a sparse track pays O(log gaps) per group handed out, as it did
before [dynamic range requests](https://github.com/moq-dev/moq/pull/6171).
`track_sparse_read` and the setup of `track_gap_flush` stay flat across their
gap sweeps.

## Plan

Found by the 2026-10-25 main against release run. Every hand-out goes through
`Cursor::probe` (`model/track.rs`), and `miss::Misses::judge`
(`model/track/miss.rs`) rebuilds its clipped span from the whole missable
window and re-judges every deferred gap, so each read walks every gap below
it. Instructions per write and read pair of `track_sparse_read/timed`: 152k
at 16 gaps, 2.57M at 256, and 51.4M at 4096, which is 1079x release in wall
time (2.75 ms against 2.5 us). The untimed rows match. `track_gap_flush`
builds its gaps through the same path, so its untimed setup is quadratic
(50.8G instructions at 4096 gaps).

Judge only what changed since the last probe: the groups the read passed,
gaps the ledger resolved, and a window or budget change. A probe with
nothing new must not touch the deferred set. Keep the fills and drops
that range requests serve; the sweep, not a release ratio, is the acceptance signal.
[Drop set](/quest/m1/drop-set.md) reshapes the same state, so whichever lands
second rebases onto the other.

Public API: none. Wire: none.

## Related

- [Drop set](/quest/m1/drop-set.md) - per-cursor miss judgement over the same ledger
40 changes: 36 additions & 4 deletions quest/m2/audit-perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,45 @@

Every benchmark that regressed since the previous run is explained, fixed, or
planned, and every hot or fan-out path without a benchmark gets one or a
follow-up.
follow-up. `main` ships no slower than `release`: every case where the cut
this run gates would be slower is fixed or accepted with the feature it pays
for.

## Plan

Recurs once per major release (decided 2026-10-19). The PR that finishes
this run also adds the next one with the same file name at the end of the next
milestone's `Required` (m0 to m1, m1 to m2). The successor keeps the Goal
and this paragraph; the rest of its Plan is rewritten with what this run
learned: baselines, results, and what to check first. A successor placed in m2
milestone's `Required` (m0 to m1, m1 to m2). The successor keeps the Goal,
this paragraph, and the release baseline section; the rest of its Plan is
rewritten with what this run learned: baselines, results, and what to check
first. A successor placed in m2
also lists the [release cut](/quest/m2/release/cut.md) under `Required`, so it
waits for the release in flight to ship. Only `/quest-delete` ends the
recurrence. Fixes land in the run's PR, one commit each, rather than as follow-ups.
Only a change that needs a maintainer decision (public API, scope) or is
independently landable and too large to review alongside the rest becomes a
follow-up through `/quest-plan`.

### Release baseline

The run's last step, once the rest of its milestone has merged, so `main` is
what the cut ships. Pair it with the last cut, keeping the evidence outside
the checkout, and record `/proc/loadavg` around each run:

```bash
BENCH_ARTIFACTS=<scratch> just bench origin/release
bun bench/browser/micro.ts --bench frames --out <scratch>/frames.json base=<release> current=. aa=.
```

`<release>` is an export of `origin/release` with its own
`bun install --frozen-lockfile`. Re-run every slower case in alternating
rounds and count instructions for any that holds (see Method). A ratio
against `release` shows where to look, not what to fix: `main` carries
features `release` lacks, so a flat per-operation cost usually pays for one,
while a cost that grows with `main`'s own sweep is accidental. Fix accidental
cost in this run, and record each accepted cost in its results with the
feature and PR it pays for. A browser row inside the `aa` spread is noise.

### What the m1 run (2026-10-22) fixed

Instruction counts are one optimized `--test --exact` iteration under callgrind,
Expand Down Expand Up @@ -89,6 +112,12 @@ did not move.
catalog on every estimate to validate it before retaining it (62% of
`catalog_estimate_fanout/hang/tracks_4/4`). Validating only the edited entry
would weaken that guarantee for custom configs, so it needs a decision.
- The TS export grows with track count (1.50x release at `container-export/ts/1x1`,
2.24x at `16x64`): `Schedule::index` and `Schedule::admit`
(`moq-mux/src/container/ts/schedule.rs`) scan and sort every queued unit
per PCR slot.
- e2ee expands the AES-GCM key on every frame (`aead` in
`moq-e2ee/src/protect.rs`), about a quarter of `protect`; release does too.

### Method: attribute with instruction counts, not wall time

Expand All @@ -108,6 +137,9 @@ wall-clock bisect blames the wrong commit. Count instructions instead:
`finish` for `track_stamp_groups`) and divide by the count.
- `callgrind_annotate --inclusive=yes` diffs between two commits point at the
step; workspace sources show as `${workspace}`, so read them by name.
- A wall-time hold at equal instructions may be page faults: compare
`/usr/bin/time -f %R`, then pin glibc with `GLIBC_TUNABLES` (see
[the m1 release run](/quest/m1/release-perf/README.md)) before blaming code.
- A bench that is not CPU-bound is not attributable this way:
`relay_auth_refresh` runs real loopback sockets and moved +27-60% on CI with
+0.4% instructions.
Expand Down
9 changes: 5 additions & 4 deletions quest/m2/performance-survey/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,10 +143,11 @@ No duplicate quest or workflow edit belongs here.

### Existing ownership and deliberately deferred work

- [Release performance](/quest/m1/release-perf/README.md) fixed sparse gaps,
resume duplicates, idle GC, stats drain, FETCH fan-out, JS frame regressions,
and paired comparison correctness; its remaining work is the final
main-against-release comparison.
- [Release performance](/quest/m1/release-perf/README.md) owns the main
against release comparison for this cut, including the sparse read slope;
it already fixed or accepted resume duplicates, idle GC, stats drain, FETCH
fan-out, JS frame regressions, and paired comparison correctness. The
[performance audit](/quest/m2/audit-perf.md) repeats it before later cuts.
- [Perf](/quest/m2/perf/README.md) owns measured channel contention, relay
allocations, ingest batching, worker quiescence and runtime selection.
- The [release audit](/quest/m1/release-audit/README.md) bounded the media
Expand Down
1 change: 1 addition & 0 deletions rs/moq-mux/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ mp4-atom = { version = "0.16.2", features = ["tokio", "bytes", "serde"] }
mpeg2ts = "0.6.0"
num_enum = { workspace = true }
roxmltree = "0.21.1"
rustc-hash = { workspace = true }
scuffle-av1 = { version = "0.1.4" }
scuffle-h265 = { version = "0.2.2" }
serde = { workspace = true }
Expand Down
4 changes: 2 additions & 2 deletions rs/moq-mux/benches/catalog.rs
Original file line number Diff line number Diff line change
Expand Up @@ -222,8 +222,8 @@ fn bench_unresolved_teardown(c: &mut Criterion) {
group.finish();
}

/// Tracks interleaving frames on one catalog.
const ORDERED: [usize; 3] = [1, 16, 256];
/// Tracks interleaving frames on one catalog, up to the most renditions it can hold.
const ORDERED: [usize; 3] = [1, 16, hang::catalog::MAX_RENDITIONS];

/// Frames each track writes per iteration.
const FRAMES: u64 = 100;
Expand Down
11 changes: 6 additions & 5 deletions rs/moq-mux/src/container/flv/export.rs
Original file line number Diff line number Diff line change
Expand Up @@ -401,11 +401,6 @@ impl<S: Stream> Export<S> {
}
}

self.jitter.expect(
(0..self.video.len())
.map(|i| (0, i))
.chain((0..self.audio.len()).map(|i| (1, i))),
);
self.fill(waiter)?;

// 3. Emit the header once every track's codec config has resolved.
Expand Down Expand Up @@ -600,6 +595,12 @@ impl<S: Stream> Export<S> {
// publisher retires a rendition as its track finishes or drops, and the catalog
// update races that end on another stream, so only the track's end can say which it
// was: a finish ends it cleanly, and a drop is the error the export reports.
// Only binding adds tracks, so the clock waits on the bound set as of this update.
self.jitter.expect(
(0..self.video.len())
.map(|i| (0, i))
.chain((0..self.audio.len()).map(|i| (1, i))),
);
Ok(())
}

Expand Down
3 changes: 2 additions & 1 deletion rs/moq-mux/src/container/mkv/export.rs
Original file line number Diff line number Diff line change
Expand Up @@ -391,7 +391,6 @@ impl<S: Stream> Export<S> {
}
}

self.jitter.expect(self.tracks.keys().cloned());
self.fill(waiter)?;

// 3. Before the header is emitted: keep pulling until every video
Expand Down Expand Up @@ -661,6 +660,8 @@ impl<S: Stream> Export<S> {
next_track_number += 1;
}

// Tracks change only before the header, so the clock waits on this set from here on.
self.jitter.expect(self.tracks.keys().cloned());
self.catalog_snapshot = Some(catalog);
Ok(())
}
Expand Down
5 changes: 3 additions & 2 deletions rs/moq-mux/src/container/ts/export.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ use mpeg2ts::ts::{
};

use moq_net::Timestamp;
use rustc_hash::FxHashMap;

use super::catalog::Catalog as _;
use crate::catalog::Stream;
Expand Down Expand Up @@ -147,7 +148,7 @@ where
/// The program tables changed, so the next frame muxed is a tune-in point that leads with them.
revised: bool,
/// The next continuity counter per PID, numbered as packets go out.
counters: HashMap<u16, u8>,
counters: FxHashMap<u16, u8>,
/// PMT program-level descriptors captured on import, re-emitted in the PMT.
program_descriptors: Vec<catalog::Descriptor>,
/// Transport/service identity captured on import, used to rebuild a consistent
Expand Down Expand Up @@ -1249,7 +1250,7 @@ where
held: None,
generation: 0,
tracks: HashMap::new(),
counters: HashMap::new(),
counters: FxHashMap::default(),
program_descriptors: Vec::new(),
program: None,
si: BTreeMap::new(),
Expand Down
Loading
Loading