From 27837d2682b51a25e5231d26a26ce4011fdb3715 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 29 Oct 2026 23:50:10 -0700 Subject: [PATCH 1/3] chore(quest): claim quest/m1/release-perf/README.md Quest-Claim: 1f0778c2e697bcdf From 2ba27b68480653c9c273a316f5c9c49a662763b1 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 30 Oct 2026 00:59:13 -0700 Subject: [PATCH 2/3] quest(release-perf): record the 2026-10-30 release re-run The relay fanout pairing is bimodal on main: in four of seven rounds the relay spreads its subscriber tasks over every Tokio worker and contends the track's shared lock, doubling or tripling its CPU. The sparse read rows are flat after #6921, leaving the accepted per-group cost. Co-Authored-By: Claude Opus 5.5 --- quest/m1/release-perf/README.md | 43 +++++++++++++++++++++++++++++---- 1 file changed, 38 insertions(+), 5 deletions(-) diff --git a/quest/m1/release-perf/README.md b/quest/m1/release-perf/README.md index 6b92b4e8cc9..56042222183 100644 --- a/quest/m1/release-perf/README.md +++ b/quest/m1/release-perf/README.md @@ -7,11 +7,10 @@ Before the next release cut, every benchmark where `main` is slower than is fixed. Cost that pays for a feature `main` deliberately added is accepted, and the reason is written here. -This epic's own work: once every child has landed, confirm the sparse rows -and the relay fanout load against `release` (see the 2026-10-25 decisions). -The browser frame path needs no repeat; the -[performance audit](/quest/m2/audit-perf.md) now pairs every later cut with -`release`. +This epic's own work: settle the relay fanout regression and the sparse rows +that the 2026-10-30 re-run found (see that section). The browser frame path +needs no repeat; the [performance audit](/quest/m2/audit-perf.md) now pairs +every later cut with `release`. ## Plan @@ -227,6 +226,40 @@ Decided 2026-10-25 (maintainer), in the sparse rows). Fix it if the relay CPU still rises with consistent sign; otherwise this epic is done. +### Re-run (2026-10-30) + +`release` (`de0a2b0534`) against `main` (`7c8e255304`). The host was shared +with other agents: 1-minute load 11 to 29 on 24 cores. Rather than the full +`just bench origin/release`, this run paired only the relay workloads (the +tail of `bench/run.sh`, seven alternating rounds) and `track_sparse_read` +(three alternating rounds). Release's `track` bench has no `async` feature, so +`run.sh --criterion` cannot build both sides with one set of arguments. + +Relay fanout is bimodal on `main`. Release's relay used 0.062 to 0.082 cores +in all seven rounds. Main's used 0.062 to 0.072 in three rounds and 0.157 to +0.241 in four, with the same delivery (640 frames/s, p99 within 2%). The +comparator calls the sign mixed (paired deltas -15% to +218%, median +119%), +but the high runs share a mechanism. In them, 21 to 24 Tokio workers are busy +and the relay makes 9,400 to 12,600 voluntary context switches/s, against 2 +to 6 busy workers and 400 to 1,300 switches/s otherwise. Release never +exceeded 4 busy workers or 2,800 switches/s. Eight CPU-profiled runs of +`main` reproduced it in five. In a high run, 54% of samples end in a syscall +(3 of 227 in a low run). Most are futex waits on the track's shared +`kio::Lock`, taken by every subscriber's `read` in +`poll_recv_next → poll_peek_group` and +`poll_forward_drop → poll_drop`, plus the wakes that follow. When the 64 +subscriber tasks land on many workers, every frame makes them contend that +one lock; packed onto a few workers, they never contend. Video used 9.7% less +CPU (MAD 7.1%), with 1.4% more frames delivered. + +`track_sparse_read/timed` is flat now that +[moq-dev/moq#6921](https://github.com/moq-dev/moq/pull/6921) has landed: +release 2.42, 2.49, 2.64 us and main 5.33, 5.98, 6.42 us at 16, 256, 4096 +gaps (medians of three rounds). From 16 to 4096 gaps, main grows 20% and +release 9%. The 2.2 to 2.4x gap is per iteration, not per gap. Each iteration +creates, writes, finishes, and receives two groups, so this is the per-group +model cost accepted above. + ## Related - [Performance audit](/quest/m2/audit-perf.md) - compares nightly to nightly; this epic compares main to release From 90b93fd9cb1e27acd106f1c6f900c39a539366ea Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 30 Oct 2026 13:16:41 -0700 Subject: [PATCH 3/3] quest(release-perf): accept the sparse rows, fix fanout lock contention as a child Co-Authored-By: Claude Opus 5.5 --- quest/m1/release-perf/README.md | 18 +++++++-- quest/m1/release-perf/fanout-lock.md | 59 ++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+), 3 deletions(-) create mode 100644 quest/m1/release-perf/fanout-lock.md diff --git a/quest/m1/release-perf/README.md b/quest/m1/release-perf/README.md index 56042222183..0bdaaf5a476 100644 --- a/quest/m1/release-perf/README.md +++ b/quest/m1/release-perf/README.md @@ -1,4 +1,4 @@ -# [M] Main is no slower than release +# Main is no slower than release ## Goal @@ -7,8 +7,9 @@ Before the next release cut, every benchmark where `main` is slower than is fixed. Cost that pays for a feature `main` deliberately added is accepted, and the reason is written here. -This epic's own work: settle the relay fanout regression and the sparse rows -that the 2026-10-30 re-run found (see that section). The browser frame path +This epic's own work: once its child has landed, re-pair the relay `fanout` +workload against `release` and confirm main's relay CPU is no higher in every +round. The sparse rows are accepted (2026-10-30). The browser frame path needs no repeat; the [performance audit](/quest/m2/audit-perf.md) now pairs every later cut with `release`. @@ -260,6 +261,17 @@ release 9%. The 2.2 to 2.4x gap is per iteration, not per gap. Each iteration creates, writes, finishes, and receives two groups, so this is the per-group model cost accepted above. +Decided 2026-10-30 (maintainer), in +[moq-dev/moq#7299](https://github.com/moq-dev/moq/pull/7299): + +- The sparse rows are accepted as that per-group model cost. +- The fanout mode is accidental and is fixed before the cut, in + [Fanout lock contention](/quest/m1/release-perf/fanout-lock.md). + +## Required + +- [Fanout lock contention](/quest/m1/release-perf/fanout-lock.md) - relay fanout subscribers stop contending one track lock, so main's relay CPU matches release's in every round + ## Related - [Performance audit](/quest/m2/audit-perf.md) - compares nightly to nightly; this epic compares main to release diff --git a/quest/m1/release-perf/fanout-lock.md b/quest/m1/release-perf/fanout-lock.md new file mode 100644 index 00000000000..b94c52ee01e --- /dev/null +++ b/quest/m1/release-perf/fanout-lock.md @@ -0,0 +1,59 @@ +# [M] Relay fanout subscribers stop contending one track lock + +## Goal + +The relay's `fanout` workload (`bench/relay.sh`, 64 subscribers of one track) +uses no more relay CPU on `main` than on `release` in every alternating +round, with the same delivery. Subscribers of one track no longer serialize +on the track's shared `kio::Lock` for every frame, so spreading their tasks +across Tokio workers no longer multiplies the relay's CPU. + +## Plan + +Found by the 2026-10-30 re-run in the +[release-perf epic](/quest/m1/release-perf/README.md) +([moq-dev/moq#7299](https://github.com/moq-dev/moq/pull/7299)), `release` +`de0a2b0534` against `main` `7c8e255304`. Decided 2026-10-30 (maintainer): +this is accidental cost, fixed before the cut. + +What the profile found: + +- `main`'s relay is bimodal under `fanout`. Release used 0.062 to 0.082 + cores in all seven rounds; main used 0.062 to 0.072 in three and 0.157 to + 0.241 in four, at the same 640 frames/s and p99. +- In the high mode 21 to 24 Tokio workers are busy and the relay makes 9,400 + to 12,600 voluntary context switches/s, against 2 to 6 workers and 400 to + 1,300 switches/s in the low mode. Release never exceeded 4 busy workers or + 2,800 switches/s. Eight CPU-profiled runs of `main` (frame pointers, the + default shared runtime) reproduced the high mode in five. +- In a high run 54% of samples end in a syscall (3 of 227 in a low run), + mostly futex waits on the track's shared `kio::Lock` and the wakes after + them. Every subscriber's read takes it, in the lite publisher's + `poll_recv_next` → `poll_group` / `poll_peek_group` and `poll_forward_drop` + → `poll_drop` (`rs/moq-net/src/lite/publisher.rs`), which reach the track + state through `rs/moq-net/src/model/resume.rs` and + `rs/moq-net/src/model/track.rs`. +- Packed onto a few workers, the 64 tasks never contend. Spread over many, + each frame wakes them all onto one mutex. + +Approach: + +- First find what `release`'s read path did differently, since release ran in + the same rounds and never spread. Compare how many times one subscriber's + wake takes the track lock on each branch. +- Then cut the per-wake lock work: take the lock once per wake rather than + once per poll step, or keep per-reader state outside the producer's mutex + so a reader only locks when a new group or drop is published. +- Add a Criterion case that reproduces the contention without the relay + harness (many readers of one track on a multi-threaded runtime), and keep + it in the release pairing. +- Verify with seven alternating `fanout` rounds against `origin/release`, + recording `/proc/loadavg`, busy workers, and voluntary context switches + per run; a quiet host is preferred. `just bench-profile` cannot yet run the + default shared runtime, where the mode shows up. + +Public API: none expected. Wire: none. + +## Related + +- [Log lock profile](/quest/m1/unbounded-groups/log-lock-profile.md) - lock contention profiling of `GroupState`, not the track lock