diff --git a/.github/release-perf.md b/.github/release-perf.md index 52e2bcb8e87..2696d8e0fb7 100644 --- a/.github/release-perf.md +++ b/.github/release-perf.md @@ -62,9 +62,98 @@ that allocated it, in KiB per connection (main minus release): The same pairing measured `chat` and `fanout` relay CPU per delivered frame +43% and +35% on `main` (every pair slower), the opposite of the 2026-11-05 -result. [Relay CPU](/quest/m2/relay-cpu.md) owns explaining that before the -cut. The run directories stayed in the measuring checkout's disposable -`.tmp/bench`; the numbers above are the record. +result; [Relay CPU](#relay-cpu-2026-11-16) explains it. The run directories +stayed in the measuring checkout's disposable `.tmp/bench`; the numbers above +are the record. + +## Relay CPU (2026-11-16) + +`main`'s extra relay CPU in `chat` and `fanout` is work under a track's +shared state lock, and contention for it. Every subscriber of a track wakes on +each group at once, and on each pass over the track the relay's publisher +forwards drops ([moq-dev/moq#5691](https://github.com/moq-dev/moq/pull/5691)) +and judges misses for fills +([moq-dev/moq#6171](https://github.com/moq-dev/moq/pull/6171)) for that +subscriber, taking the track lock several times. `release` has neither, so its +readers rarely meet on the lock. The contention grows with the workers that +meet there, so the gap depends on the host: the 2026-11-05 pairing ran on 8 +pinned CPUs, this one on 32. + +[moq-dev/moq#8867](https://github.com/moq-dev/moq/pull/8867) added one more +lock per drop poll, reading the revision that says whether a stale run +received from upstream may be gone even when the reader held none; +[moq-dev/moq#8979](https://github.com/moq-dev/moq/pull/8979) skips that read +when nothing was received. The rest is accepted for the cut as the cost of drop +forwarding and fills; [Track lock passes](/quest/m3/relay-track-lock.md) +plans cutting it. + +All runs on `fedora` (32 CPUs, shared: one-minute load 24 to 66), `release` +at [ca3b801a6c](https://github.com/moq-dev/moq/commit/ca3b801a6c), `main` +at [ba9c4c6533](https://github.com/moq-dev/moq/commit/ba9c4c6533). By owner, +in instructions per delivered frame +(`just bench-profile instructions 8 tokio-workers`, relay on CPUs +0-7, three alternating rounds; `chat` at load 27 to 35, `fanout` 25 to 60): + +| Owner | `chat` release | `chat` main | `fanout` release | `fanout` main | +| --- | ---: | ---: | ---: | ---: | +| transport (QUIC, TLS, sockets) | 79.1k | 49.8k | 77.9k | 51.0k | +| session (`moq_net` protocol, `kio`) | 57.7k | 49.5k | 64.8k | 66.1k | +| origin (`moq_net::model`) | 26.3k | 88.5k | 29.2k | 102.4k | +| total | 163.2k | 187.8k (+15%) | 171.9k | 219.4k (+28%) | + +- **The lock.** In `chat`, futex syscalls rose from 2.4k to 19.7k + instructions per frame and user-side lock and unlock from 32.4k to 39.6k, + the whole net growth; cycles per frame rose 25%. In `fanout`, futex rose + from 12.6k to 33.8k and lock and unlock from 35.0k to 46.7k, most of it. + On `main`, nearly every contended acquire is the track's (`TrackState`), + from the drop poll, its miss judgement, the drift edge, and the read path. +- **The passes.** Each subscriber's track side runs three passes per group + (its arrival, the pass after serving it, its finish), forwarding drops five + times and judging misses six (callgrind, `just bench-fanout`, `session` at + 512 subscribers). The forward-drop path is 56k instructions per frame in + `chat` and 65k in `fanout`, 30% of `main`'s relay in each. +- **Workers.** Futex instructions per frame in `chat` on `main` were 3.2k at + 2 workers and 8.7k at 8 in one run (load 31 to 66), against 2.3k for + `release` at both. + +With the fix, alternating rounds of the same count: + +| | release | main | main with the fix | +| --- | ---: | ---: | ---: | +| `chat` instructions per frame (4 rounds, load 24 to 53) | 161.2k | 164.8k | 164.0k | +| `chat` futex instructions per frame | 2.3k | 8.5k | 4.7k | +| `fanout` instructions per frame (3 rounds, load 25 to 60) | 170.7k | 219.7k | 218.7k | +| `fanout` futex instructions per frame | 12.6k | 33.8k | 32.9k | + +The fix halves `chat`'s futex cost and barely moves `fanout`'s, whose 64 +subscribers per track contend in the passes themselves. + +Median relay CPU per delivered frame from `just bench-relay quic` +(the default shared runtime), with the range of paired changes: + +| Workload | main to the fix (5 rounds) | release to the fix (5 rounds) | +| --- | --- | --- | +| `chat` | -11.5% (-31.8 to +1.6) | +8.7% (-5.3 to +25.3) | +| `fanout` | +4.1% (-10.1 to +18.5) | +18.8% (-6.8 to +38.2) | +| `video` | 0.0% (-14.3 to +17.3) | -15.2% (-36.3 to -12.5) | + +Four of five `chat` pairs read the fix lower than `main`; the first column +also kept the spare chunk of sent packets below, about 0.4% of a `chat` +frame. Against `release`, two of the fix's `chat` rounds delivered below the +offered 20,000 frames/s in the window; the three at the offered rate read ++8.7%, +22.1%, and +25.3%. The fix lost 0.3% to 1.1% of `chat`'s groups +over each run's lifetime, all during the startup ramp, and `release` none. +Earlier runs in the same session saw `main` lose up to 11.9% there, with the +window still draining the backlog, which read the largest CPU per frame of +any pair. [Chat startup loss](/quest/m3/chat-startup-loss.md) owns that loss. + +The transport lead was measured and not taken. A connection with every packet +acknowledged before it sends the next frees its 8 KiB chunk of sent packets +and allocates another per packet +([moq-dev/moq#6936](https://github.com/moq-dev/moq/pull/6936)). Keeping one +spare chunk cut that from 803 to 200 instructions per packet (jemalloc), +about 0.4% of a `chat` frame, but holds 8 KiB on every idle connection (16 +MiB across `chat`'s 2,001), so the memory finding above outweighs it. ## Final relay pairing (2026-11-05) diff --git a/bench/README.md b/bench/README.md index 25a81ae5290..342706b915f 100644 --- a/bench/README.md +++ b/bench/README.md @@ -426,7 +426,7 @@ hides steps of a few percent. `just bench-fanout` counts the chat fan-out without a relay process or sockets, so the counts are the code's alone: ```bash -nix develop --command just bench-fanout "cdff3c8f95 21d6da9fc1" +nix develop --command just bench-fanout origin/main nix develop --command just bench-fanout "" "model" "1 512" ``` @@ -436,7 +436,8 @@ run by default: - `model`: one finished group a cycle through the track model, which every subscriber takes before parking again, as moq-net's `track_fanout_group` bench does. It builds against every revision since - [cdff3c8f95](https://github.com/moq-dev/moq/commit/cdff3c8f952b1d6bd2fd40c3f64eb0fd53b97e95). + [moq-dev/moq#8961](https://github.com/moq-dev/moq/pull/8961) started groups at a + timestamp; an older revision reports it unsupported. - `session`: one publisher session into a relay origin and one viewer session out of it per subscriber, over `transport::memory`, on one current-thread runtime. It counts the publisher, relay, and viewer sessions together, but no diff --git a/bench/fanout/model/src/main.rs b/bench/fanout/model/src/main.rs index 8ffd20f532e..38030f3023a 100644 --- a/bench/fanout/model/src/main.rs +++ b/bench/fanout/model/src/main.rs @@ -30,9 +30,10 @@ fn main() { } let payload = Bytes::from(vec![0; 64]); // Warm up past the first groups' one-time work, so both runs share it. - for _ in 0..(16 + cycles) { - let mut group = track.append_group().unwrap(); - group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + for start in 0..(16 + cycles) as u64 { + let at = Timestamp::from_millis(start).unwrap(); + let mut group = track.append_group(at).unwrap(); + group.write_frame(at, payload.clone()).unwrap(); group.finish().unwrap(); for (s, w) in subs.iter_mut().zip(&waiters) { match s.poll_recv_group(w) { diff --git a/bench/fanout/session/src/main.rs b/bench/fanout/session/src/main.rs index 03e7f3d27fb..5cf1845d0c8 100644 --- a/bench/fanout/session/src/main.rs +++ b/bench/fanout/session/src/main.rs @@ -8,12 +8,12 @@ use std::hint::black_box; use bytes::Bytes; -use moq_net::{Client, Hop, Server, Timestamp, Version, cache, origin, transport::memory}; +use moq_net::{Client, Server, Timestamp, Version, cache, origin, transport::memory}; /// An origin whose cache holds what the model scenario's does, so neither the relay's nor a /// viewer's grows with the groups a run delivers. fn origin(hop: u64) -> origin::Producer { - let mut config = origin::Config::new(Hop::new(hop).unwrap()); + let mut config = origin::Config::new(std::num::NonZeroU64::new(hop).unwrap()); config.pool = cache::Pool::new(cache::Config::default().with_capacity(64 * 1024)); let (producer, driver) = origin::Producer::new(config); tokio::spawn(moq_net::time::run(driver)); @@ -96,9 +96,10 @@ fn main() { let payload = Bytes::from(vec![0; 64]); // Past the first groups' one-time work, so the zero-cycle run shares it. - for _ in 0..(16 + cycles) { - let mut group = track.append_group().unwrap(); - group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + for start in 0..(16 + cycles) as u64 { + let at = Timestamp::from_millis(start).unwrap(); + let mut group = track.append_group(at).unwrap(); + group.write_frame(at, payload.clone()).unwrap(); group.finish().unwrap(); for (_, _, subscriber) in &mut audience { let mut group = subscriber.recv_group().await.unwrap().expect("a group"); diff --git a/quest/m2/README.md b/quest/m2/README.md index 60307943054..865c16d2f5c 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -41,8 +41,7 @@ removal, a rename, or a wire, catalog, or format change that is free until demos return to their earlier format. - **Unused API.** [One sweep per area](/quest/m2/unused-api/README.md), each item re-checked for a downstream caller first. -- **Relay and transports.** [CPU](/quest/m2/relay-cpu.md), which the - memory pairing found; [congestion names](/quest/m2/congestion-names.md); +- **Relay and transports.** [Congestion names](/quest/m2/congestion-names.md); [the 0.14 leftovers](/quest/m2/relay-014-leftovers.md); and [transport names](/quest/m2/transport-names.md). - **Auth and crypto.** [Claims](/quest/m2/auth-claims.md). End-to-end encryption diff --git a/quest/m2/relay-cpu.md b/quest/m2/relay-cpu.md deleted file mode 100644 index 00d3e92a4cd..00000000000 --- a/quest/m2/relay-cpu.md +++ /dev/null @@ -1,59 +0,0 @@ -# [M] The relay's chat and fanout CPU growth since release is explained before the cut - -## Goal - -The relay's extra CPU per delivered frame on `main` against `release` in the -`chat` and `fanout` workloads has a named cause, and the cut ships with it -either fixed or written down as the price of a feature, in -`.github/release-perf.md` beside the -[memory finding](/.github/release-perf.md#relay-memory-2026-11-16). Nothing -here changes an API or the wire. - -## Plan - -Found 2026-11-16 by [relay memory](https://github.com/moq-dev/moq/pull/8943), -whose pairing reported CPU beside RSS. Placed in m2, like the memory quest, -so the cut waits on it: the 2026-11-05 pairing said `main` used less relay -CPU in every workload, and 0.22 should not be announced on that claim while -two workloads now say the opposite. - -Facts: - -- Five alternating rounds of `just bench-relay origin/release quic` - (`release` at `ca3b801a6c`, `main` at `6488ce6b87`, host load 24 to 28 on - 32 cores): `cpu_us_per_frame` +43% in `chat` (range +24% to +59%) and +35% - in `fanout` (range +3% to +65%), every pair slower; `video` -24%. Delivered - frames per second matched. The heap-profile controls on 8 Tokio workers - agree: `chat` relay CPU 1.40 to 1.52 cores on `release`, 1.96 to 2.15 on - `main`. -- `main`'s relay allocates 2.3 times the bytes per second in `chat` (about - 300 MiB/s against 133), though fewer allocations. Half of it is - `moq_quic::connection::slots::Slots::insert`: a connection that has every - packet acknowledged before it sends the next frees its 64-slot chunk - (about 8 KiB of `Option`) and allocates a fresh one for the - next packet ([moq-dev/moq#6936](https://github.com/moq-dev/moq/pull/6936)). - That is one 8 KiB allocation per packet in `chat`. A lead, not a measured - cause: count instructions before and after keeping one spare chunk. -- The lifetime `loss_pct` also rose (about 1.5% against 0.1%), all of it - during the startup ramp: `groups_lost` stops growing once the load is - ready, so steady delivery is unchanged. It may be groups `main` now - reports dropped as a subscriber joins; confirm that before reading it as - a regression. - -For the PR: attribute the CPU with `just bench-profile cpu chat 8 -tokio-workers origin/release`, counted in instructions as the -[performance audit's Method](/quest/m3/audit-perf.md#method) does, by -owner: transport, session, origin, other. Fix what is accidental; record a -feature's cost with its PR. - -## Waiting - -- Maintainer: whether the cut waits on this. Recommended: yes, as it does on - memory, since the release comparison now claims the opposite of what - `main` measures. Otherwise move it under [m3](/quest/m3/README.md) with - `quest mv`, where the performance audit's release run picks it up after - the cut. - -## Related - -- [Performance audit](/quest/m3/audit-perf.md) - reruns the release pairing after the cut diff --git a/quest/m3/chat-startup-loss.md b/quest/m3/chat-startup-loss.md new file mode 100644 index 00000000000..970ff3816f5 --- /dev/null +++ b/quest/m3/chat-startup-loss.md @@ -0,0 +1,35 @@ +# [M] The relay's chat startup loss is explained + +## Goal + +The groups `main`'s relay loses while `chat`'s 2,000 viewers join have a named +cause: a real delivery loss, fixed or accepted with what it pays for, or +groups now reported dropped that `release` never reported, documented in +`.github/release-perf.md` beside the relay CPU finding. + +## Plan + +Found by [Relay CPU](https://github.com/moq-dev/moq/pull/8979) +([release-perf.md](/.github/release-perf.md#relay-cpu-2026-11-16)). + +Facts: + +- The relay memory pairing saw `chat`'s lifetime `loss_pct` at about 1.5% on + `main` against 0.1% on `release`, all during the startup ramp: + `groups_lost` stops growing once the load is ready. +- In the relay CPU runs (`just bench-relay origin/release quic`, `fedora`, + load 24 to 66), three of sixteen `main` `chat` rounds, with and without its + fix, lost 6.7% to 11.9% of groups and still delivered above the offered + 20,000 frames/s in the measured window, draining a backlog. The other + thirteen lost 0.04% to 1.8%; `release`'s six lost at most 0.06%. The lossy + rounds read the highest relay CPU per frame. + +Start from one lossy round's `load.jsonl`: whether the lost groups are drops +the relay reported to viewers that joined after them (lite-05+ drop +forwarding, which `release` lacks) or groups a viewer waited for and never +got. The generator's per-connection reports say which viewers lost them and +when. + +## Related + +- [Track lock passes](/quest/m3/relay-track-lock.md) - the same subscribers' steady-state CPU diff --git a/quest/m3/relay-track-lock.md b/quest/m3/relay-track-lock.md new file mode 100644 index 00000000000..ca5f809883e --- /dev/null +++ b/quest/m3/relay-track-lock.md @@ -0,0 +1,60 @@ +# [L] A live subscriber's track passes stop meeting on the track lock + +## Goal + +The relay's publisher serves a live subscriber's track side without +contending for the track's shared state lock: in `chat` and `fanout`, futex +instructions per delivered frame fall to `release`'s level (about 2.3k and +12.6k on 8 workers) at 2, 8, and 16 workers, and relay CPU per frame is no +higher than `release`'s. Drop forwarding and fills behave as before. Nothing +here changes an API or the wire. + +## Plan + +Found by [Relay CPU](https://github.com/moq-dev/moq/pull/8979), which +recorded the numbers in +[release-perf.md](/.github/release-perf.md#relay-cpu-2026-11-16) and accepted +the rest of the cost for the cut. + +Decided 2026-11-17 (maintainer, on +[moq-dev/moq#8979](https://github.com/moq-dev/moq/pull/8979#issuecomment-6937624164)): +the cut does not wait on this; it follows the cut in m3. Rejected: moving it +to m2. + +Facts: + +- Every subscriber of a track wakes on each group at once. Its + `TrackRun::poll_track` (`rs/moq-net/src/lite/publisher.rs`) then runs three + passes per group (the arrival, the pass after serving it, the finish), + forwarding drops five times (`Subscriber::poll_drop_probing`) and judging + misses six (`Cursor::probe`, `Misses::judge`). Each takes the track lock + separately for the drift edge (`Cursor::poll_drift`), the judgement, and the + drop cursor (`DropCursor::next`), and the read path takes it again + (`poll_peek_group`, `take`, `forget_past_budget`). +- On 8 tokio workers, `chat`'s futex instructions per frame were 2.3k on + `release`, 8.5k on `main`, and 4.7k once the drop poll stopped reading the + stale revision it rarely needs. `fanout`'s were 12.6k, 33.8k, and 32.9k: + its 64 subscribers per track contend in the passes themselves, and its + forward-drop path is 65k instructions per frame, 30% of the relay. + +Approach, in order of reach: + +- A pass that changed nothing judges nothing: the drop cursor keeps the track + version (`kio::Channel::version`, as `kio::Watch` does) its last pending + judgement saw, with the cursor inputs it read (`next_key`, the floor, the + window's revision, the record, cuts, the subscription's budget), and parks + again at an unchanged version without the lock. A pending deadline (the + misses' deferred gaps, `watch_wall`) wakes without a version change, so a + cursor waiting on one judges again. +- One pass takes the lock once: the drift edge, the judgement, and the drop + cursor's next run share one guard. + +Measure futex and total instructions per frame in both workloads with +`just bench-profile instructions chat "2 8 16" tokio-workers origin/main`, +and relay CPU with `just bench-relay origin/release quic`, both before and +after; `just bench-fanout` checks the single-threaded count did not grow. + +## Related + +- [kio channel contention](/quest/m9/kio-channel-contention/README.md) - the general lock-wait work this narrows to one channel +- [Performance audit](/quest/m3/audit-perf.md) - reruns the release pairing after the cut diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index bedaed5d033..bc38084aaa8 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -1949,14 +1949,18 @@ impl Reader { // A received stale run queued here that new terms superseded since is unknown again: // the copy no longer holds it, and its upstream re-offers or drops it again. Checked // only once the copy let a stale run go, never on every poll: each poll reports one - // run, so a check per poll would cost the received runs squared to drain them. - let current = sub.stale_revision(); - if self.received_checked != Some(current) { - for run in self.received.iter() { - superseded.extend(sub.stale_held(run.clone()).gaps(run)); + // run, so a check per poll would cost the received runs squared to drain them. With + // none received there is nothing to check, so a live reader never reads the revision + // through the track lock every subscriber shares. + if !self.received.is_empty() { + let current = sub.stale_revision(); + if self.received_checked != Some(current) { + for run in self.received.iter() { + superseded.extend(sub.stale_held(run.clone()).gaps(run)); + } } + self.received_checked = Some(current); } - self.received_checked = Some(current); loop { let polled = match probe { true => sub.poll_drop(waiter),