diff --git a/quest/m1/release-perf/README.md b/quest/m1/release-perf/README.md index 6b92b4e8cc9..0bdaaf5a476 100644 --- a/quest/m1/release-perf/README.md +++ b/quest/m1/release-perf/README.md @@ -1,4 +1,4 @@ -# [M] Main is no slower than release +# Main is no slower than release ## Goal @@ -7,11 +7,11 @@ Before the next release cut, every benchmark where `main` is slower than is fixed. Cost that pays for a feature `main` deliberately added is accepted, and the reason is written here. -This epic's own work: once every child has landed, confirm the sparse rows -and the relay fanout load against `release` (see the 2026-10-25 decisions). -The browser frame path needs no repeat; the -[performance audit](/quest/m2/audit-perf.md) now pairs every later cut with -`release`. +This epic's own work: once its child has landed, re-pair the relay `fanout` +workload against `release` and confirm main's relay CPU is no higher in every +round. The sparse rows are accepted (2026-10-30). The browser frame path +needs no repeat; the [performance audit](/quest/m2/audit-perf.md) now pairs +every later cut with `release`. ## Plan @@ -227,6 +227,51 @@ Decided 2026-10-25 (maintainer), in the sparse rows). Fix it if the relay CPU still rises with consistent sign; otherwise this epic is done. +### Re-run (2026-10-30) + +`release` (`de0a2b0534`) against `main` (`7c8e255304`). The host was shared +with other agents: 1-minute load 11 to 29 on 24 cores. Rather than the full +`just bench origin/release`, this run paired only the relay workloads (the +tail of `bench/run.sh`, seven alternating rounds) and `track_sparse_read` +(three alternating rounds). Release's `track` bench has no `async` feature, so +`run.sh --criterion` cannot build both sides with one set of arguments. + +Relay fanout is bimodal on `main`. Release's relay used 0.062 to 0.082 cores +in all seven rounds. Main's used 0.062 to 0.072 in three rounds and 0.157 to +0.241 in four, with the same delivery (640 frames/s, p99 within 2%). The +comparator calls the sign mixed (paired deltas -15% to +218%, median +119%), +but the high runs share a mechanism. In them, 21 to 24 Tokio workers are busy +and the relay makes 9,400 to 12,600 voluntary context switches/s, against 2 +to 6 busy workers and 400 to 1,300 switches/s otherwise. Release never +exceeded 4 busy workers or 2,800 switches/s. Eight CPU-profiled runs of +`main` reproduced it in five. In a high run, 54% of samples end in a syscall +(3 of 227 in a low run). Most are futex waits on the track's shared +`kio::Lock`, taken by every subscriber's `read` in +`poll_recv_next → poll_peek_group` and +`poll_forward_drop → poll_drop`, plus the wakes that follow. When the 64 +subscriber tasks land on many workers, every frame makes them contend that +one lock; packed onto a few workers, they never contend. Video used 9.7% less +CPU (MAD 7.1%), with 1.4% more frames delivered. + +`track_sparse_read/timed` is flat now that +[moq-dev/moq#6921](https://github.com/moq-dev/moq/pull/6921) has landed: +release 2.42, 2.49, 2.64 us and main 5.33, 5.98, 6.42 us at 16, 256, 4096 +gaps (medians of three rounds). From 16 to 4096 gaps, main grows 20% and +release 9%. The 2.2 to 2.4x gap is per iteration, not per gap. Each iteration +creates, writes, finishes, and receives two groups, so this is the per-group +model cost accepted above. + +Decided 2026-10-30 (maintainer), in +[moq-dev/moq#7299](https://github.com/moq-dev/moq/pull/7299): + +- The sparse rows are accepted as that per-group model cost. +- The fanout mode is accidental and is fixed before the cut, in + [Fanout lock contention](/quest/m1/release-perf/fanout-lock.md). + +## Required + +- [Fanout lock contention](/quest/m1/release-perf/fanout-lock.md) - relay fanout subscribers stop contending one track lock, so main's relay CPU matches release's in every round + ## Related - [Performance audit](/quest/m2/audit-perf.md) - compares nightly to nightly; this epic compares main to release diff --git a/quest/m1/release-perf/fanout-lock.md b/quest/m1/release-perf/fanout-lock.md new file mode 100644 index 00000000000..b94c52ee01e --- /dev/null +++ b/quest/m1/release-perf/fanout-lock.md @@ -0,0 +1,59 @@ +# [M] Relay fanout subscribers stop contending one track lock + +## Goal + +The relay's `fanout` workload (`bench/relay.sh`, 64 subscribers of one track) +uses no more relay CPU on `main` than on `release` in every alternating +round, with the same delivery. Subscribers of one track no longer serialize +on the track's shared `kio::Lock` for every frame, so spreading their tasks +across Tokio workers no longer multiplies the relay's CPU. + +## Plan + +Found by the 2026-10-30 re-run in the +[release-perf epic](/quest/m1/release-perf/README.md) +([moq-dev/moq#7299](https://github.com/moq-dev/moq/pull/7299)), `release` +`de0a2b0534` against `main` `7c8e255304`. Decided 2026-10-30 (maintainer): +this is accidental cost, fixed before the cut. + +What the profile found: + +- `main`'s relay is bimodal under `fanout`. Release used 0.062 to 0.082 + cores in all seven rounds; main used 0.062 to 0.072 in three and 0.157 to + 0.241 in four, at the same 640 frames/s and p99. +- In the high mode 21 to 24 Tokio workers are busy and the relay makes 9,400 + to 12,600 voluntary context switches/s, against 2 to 6 workers and 400 to + 1,300 switches/s in the low mode. Release never exceeded 4 busy workers or + 2,800 switches/s. Eight CPU-profiled runs of `main` (frame pointers, the + default shared runtime) reproduced the high mode in five. +- In a high run 54% of samples end in a syscall (3 of 227 in a low run), + mostly futex waits on the track's shared `kio::Lock` and the wakes after + them. Every subscriber's read takes it, in the lite publisher's + `poll_recv_next` → `poll_group` / `poll_peek_group` and `poll_forward_drop` + → `poll_drop` (`rs/moq-net/src/lite/publisher.rs`), which reach the track + state through `rs/moq-net/src/model/resume.rs` and + `rs/moq-net/src/model/track.rs`. +- Packed onto a few workers, the 64 tasks never contend. Spread over many, + each frame wakes them all onto one mutex. + +Approach: + +- First find what `release`'s read path did differently, since release ran in + the same rounds and never spread. Compare how many times one subscriber's + wake takes the track lock on each branch. +- Then cut the per-wake lock work: take the lock once per wake rather than + once per poll step, or keep per-reader state outside the producer's mutex + so a reader only locks when a new group or drop is published. +- Add a Criterion case that reproduces the contention without the relay + harness (many readers of one track on a multi-threaded runtime), and keep + it in the release pairing. +- Verify with seven alternating `fanout` rounds against `origin/release`, + recording `/proc/loadavg`, busy workers, and voluntary context switches + per run; a quiet host is preferred. `just bench-profile` cannot yet run the + default shared runtime, where the mode shows up. + +Public API: none expected. Wire: none. + +## Related + +- [Log lock profile](/quest/m1/unbounded-groups/log-lock-profile.md) - lock contention profiling of `GroupState`, not the track lock