Skip to content
95 changes: 92 additions & 3 deletions .github/release-perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,98 @@ that allocated it, in KiB per connection (main minus release):

The same pairing measured `chat` and `fanout` relay CPU per delivered frame
+43% and +35% on `main` (every pair slower), the opposite of the 2026-11-05
result. [Relay CPU](/quest/m2/relay-cpu.md) owns explaining that before the
cut. The run directories stayed in the measuring checkout's disposable
`.tmp/bench`; the numbers above are the record.
result; [Relay CPU](#relay-cpu-2026-11-16) explains it. The run directories
stayed in the measuring checkout's disposable `.tmp/bench`; the numbers above
are the record.

## Relay CPU (2026-11-16)

`main`'s extra relay CPU in `chat` and `fanout` is work under a track's
shared state lock, and contention for it. Every subscriber of a track wakes on
each group at once, and on each pass over the track the relay's publisher
forwards drops ([moq-dev/moq#5691](https://github.com/moq-dev/moq/pull/5691))
and judges misses for fills
([moq-dev/moq#6171](https://github.com/moq-dev/moq/pull/6171)) for that
subscriber, taking the track lock several times. `release` has neither, so its
readers rarely meet on the lock. The contention grows with the workers that
meet there, so the gap depends on the host: the 2026-11-05 pairing ran on 8
pinned CPUs, this one on 32.

[moq-dev/moq#8867](https://github.com/moq-dev/moq/pull/8867) added one more
lock per drop poll, reading the revision that says whether a stale run
received from upstream may be gone even when the reader held none;
[moq-dev/moq#8979](https://github.com/moq-dev/moq/pull/8979) skips that read
when nothing was received. The rest is accepted for the cut as the cost of drop
forwarding and fills; [Track lock passes](/quest/m3/relay-track-lock.md)
plans cutting it.

All runs on `fedora` (32 CPUs, shared: one-minute load 24 to 66), `release`
at [ca3b801a6c](https://github.com/moq-dev/moq/commit/ca3b801a6c), `main`
at [ba9c4c6533](https://github.com/moq-dev/moq/commit/ba9c4c6533). By owner,
in instructions per delivered frame
(`just bench-profile instructions <workload> 8 tokio-workers`, relay on CPUs
0-7, three alternating rounds; `chat` at load 27 to 35, `fanout` 25 to 60):

| Owner | `chat` release | `chat` main | `fanout` release | `fanout` main |
| --- | ---: | ---: | ---: | ---: |
| transport (QUIC, TLS, sockets) | 79.1k | 49.8k | 77.9k | 51.0k |
| session (`moq_net` protocol, `kio`) | 57.7k | 49.5k | 64.8k | 66.1k |
| origin (`moq_net::model`) | 26.3k | 88.5k | 29.2k | 102.4k |
| total | 163.2k | 187.8k (+15%) | 171.9k | 219.4k (+28%) |

- **The lock.** In `chat`, futex syscalls rose from 2.4k to 19.7k
instructions per frame and user-side lock and unlock from 32.4k to 39.6k,
the whole net growth; cycles per frame rose 25%. In `fanout`, futex rose
from 12.6k to 33.8k and lock and unlock from 35.0k to 46.7k, most of it.
On `main`, nearly every contended acquire is the track's (`TrackState`),
from the drop poll, its miss judgement, the drift edge, and the read path.
- **The passes.** Each subscriber's track side runs three passes per group
(its arrival, the pass after serving it, its finish), forwarding drops five
times and judging misses six (callgrind, `just bench-fanout`, `session` at
512 subscribers). The forward-drop path is 56k instructions per frame in
`chat` and 65k in `fanout`, 30% of `main`'s relay in each.
- **Workers.** Futex instructions per frame in `chat` on `main` were 3.2k at
2 workers and 8.7k at 8 in one run (load 31 to 66), against 2.3k for
`release` at both.

With the fix, alternating rounds of the same count:

| | release | main | main with the fix |
| --- | ---: | ---: | ---: |
| `chat` instructions per frame (4 rounds, load 24 to 53) | 161.2k | 164.8k | 164.0k |
| `chat` futex instructions per frame | 2.3k | 8.5k | 4.7k |
| `fanout` instructions per frame (3 rounds, load 25 to 60) | 170.7k | 219.7k | 218.7k |
| `fanout` futex instructions per frame | 12.6k | 33.8k | 32.9k |

The fix halves `chat`'s futex cost and barely moves `fanout`'s, whose 64
subscribers per track contend in the passes themselves.

Median relay CPU per delivered frame from `just bench-relay <base> quic`
(the default shared runtime), with the range of paired changes:

| Workload | main to the fix (5 rounds) | release to the fix (5 rounds) |
| --- | --- | --- |
| `chat` | -11.5% (-31.8 to +1.6) | +8.7% (-5.3 to +25.3) |
| `fanout` | +4.1% (-10.1 to +18.5) | +18.8% (-6.8 to +38.2) |
| `video` | 0.0% (-14.3 to +17.3) | -15.2% (-36.3 to -12.5) |

Four of five `chat` pairs read the fix lower than `main`; the first column
also kept the spare chunk of sent packets below, about 0.4% of a `chat`
frame. Against `release`, two of the fix's `chat` rounds delivered below the
offered 20,000 frames/s in the window; the three at the offered rate read
+8.7%, +22.1%, and +25.3%. The fix lost 0.3% to 1.1% of `chat`'s groups
over each run's lifetime, all during the startup ramp, and `release` none.
Earlier runs in the same session saw `main` lose up to 11.9% there, with the
window still draining the backlog, which read the largest CPU per frame of
any pair. [Chat startup loss](/quest/m3/chat-startup-loss.md) owns that loss.

The transport lead was measured and not taken. A connection with every packet
acknowledged before it sends the next frees its 8 KiB chunk of sent packets
and allocates another per packet
([moq-dev/moq#6936](https://github.com/moq-dev/moq/pull/6936)). Keeping one
spare chunk cut that from 803 to 200 instructions per packet (jemalloc),
about 0.4% of a `chat` frame, but holds 8 KiB on every idle connection (16
MiB across `chat`'s 2,001), so the memory finding above outweighs it.

## Final relay pairing (2026-11-05)

Expand Down
5 changes: 3 additions & 2 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -426,7 +426,7 @@ hides steps of a few percent. `just bench-fanout` counts the chat fan-out
without a relay process or sockets, so the counts are the code's alone:

```bash
nix develop --command just bench-fanout "cdff3c8f95 21d6da9fc1"
nix develop --command just bench-fanout origin/main
nix develop --command just bench-fanout "" "model" "1 512"
```

Expand All @@ -436,7 +436,8 @@ run by default:
- `model`: one finished group a cycle through the track model, which every
subscriber takes before parking again, as moq-net's `track_fanout_group`
bench does. It builds against every revision since
[cdff3c8f95](https://github.com/moq-dev/moq/commit/cdff3c8f952b1d6bd2fd40c3f64eb0fd53b97e95).
[moq-dev/moq#8961](https://github.com/moq-dev/moq/pull/8961) started groups at a
timestamp; an older revision reports it unsupported.
- `session`: one publisher session into a relay origin and one viewer session
out of it per subscriber, over `transport::memory`, on one current-thread
runtime. It counts the publisher, relay, and viewer sessions together, but no
Expand Down
7 changes: 4 additions & 3 deletions bench/fanout/model/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,10 @@ fn main() {
}
let payload = Bytes::from(vec![0; 64]);
// Warm up past the first groups' one-time work, so both runs share it.
for _ in 0..(16 + cycles) {
let mut group = track.append_group().unwrap();
group.write_frame(Timestamp::ZERO, payload.clone()).unwrap();
for start in 0..(16 + cycles) as u64 {
let at = Timestamp::from_millis(start).unwrap();
let mut group = track.append_group(at).unwrap();
group.write_frame(at, payload.clone()).unwrap();
group.finish().unwrap();
for (s, w) in subs.iter_mut().zip(&waiters) {
match s.poll_recv_group(w) {
Expand Down
11 changes: 6 additions & 5 deletions bench/fanout/session/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@
use std::hint::black_box;

use bytes::Bytes;
use moq_net::{Client, Hop, Server, Timestamp, Version, cache, origin, transport::memory};
use moq_net::{Client, Server, Timestamp, Version, cache, origin, transport::memory};

/// An origin whose cache holds what the model scenario's does, so neither the relay's nor a
/// viewer's grows with the groups a run delivers.
fn origin(hop: u64) -> origin::Producer {
let mut config = origin::Config::new(Hop::new(hop).unwrap());
let mut config = origin::Config::new(std::num::NonZeroU64::new(hop).unwrap());
config.pool = cache::Pool::new(cache::Config::default().with_capacity(64 * 1024));
let (producer, driver) = origin::Producer::new(config);
tokio::spawn(moq_net::time::run(driver));
Expand Down Expand Up @@ -96,9 +96,10 @@ fn main() {

let payload = Bytes::from(vec![0; 64]);
// Past the first groups' one-time work, so the zero-cycle run shares it.
for _ in 0..(16 + cycles) {
let mut group = track.append_group().unwrap();
group.write_frame(Timestamp::ZERO, payload.clone()).unwrap();
for start in 0..(16 + cycles) as u64 {
let at = Timestamp::from_millis(start).unwrap();
let mut group = track.append_group(at).unwrap();
group.write_frame(at, payload.clone()).unwrap();
group.finish().unwrap();
for (_, _, subscriber) in &mut audience {
let mut group = subscriber.recv_group().await.unwrap().expect("a group");
Expand Down
3 changes: 1 addition & 2 deletions quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,7 @@ removal, a rename, or a wire, catalog, or format change that is free until
demos return to their earlier format.
- **Unused API.** [One sweep per area](/quest/m2/unused-api/README.md), each
item re-checked for a downstream caller first.
- **Relay and transports.** [CPU](/quest/m2/relay-cpu.md), which the
memory pairing found; [congestion names](/quest/m2/congestion-names.md);
- **Relay and transports.** [Congestion names](/quest/m2/congestion-names.md);
[the 0.14 leftovers](/quest/m2/relay-014-leftovers.md); and
[transport names](/quest/m2/transport-names.md).
- **Auth and crypto.** [Claims](/quest/m2/auth-claims.md). End-to-end encryption
Expand Down
59 changes: 0 additions & 59 deletions quest/m2/relay-cpu.md

This file was deleted.

35 changes: 35 additions & 0 deletions quest/m3/chat-startup-loss.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# [M] The relay's chat startup loss is explained

## Goal

The groups `main`'s relay loses while `chat`'s 2,000 viewers join have a named
cause: a real delivery loss, fixed or accepted with what it pays for, or
groups now reported dropped that `release` never reported, documented in
`.github/release-perf.md` beside the relay CPU finding.

## Plan

Found by [Relay CPU](https://github.com/moq-dev/moq/pull/8979)
([release-perf.md](/.github/release-perf.md#relay-cpu-2026-11-16)).

Facts:

- The relay memory pairing saw `chat`'s lifetime `loss_pct` at about 1.5% on
`main` against 0.1% on `release`, all during the startup ramp:
`groups_lost` stops growing once the load is ready.
- In the relay CPU runs (`just bench-relay origin/release quic`, `fedora`,
load 24 to 66), three of sixteen `main` `chat` rounds, with and without its
fix, lost 6.7% to 11.9% of groups and still delivered above the offered
20,000 frames/s in the measured window, draining a backlog. The other
thirteen lost 0.04% to 1.8%; `release`'s six lost at most 0.06%. The lossy
rounds read the highest relay CPU per frame.

Start from one lossy round's `load.jsonl`: whether the lost groups are drops
the relay reported to viewers that joined after them (lite-05+ drop
forwarding, which `release` lacks) or groups a viewer waited for and never
got. The generator's per-connection reports say which viewers lost them and
when.

## Related

- [Track lock passes](/quest/m3/relay-track-lock.md) - the same subscribers' steady-state CPU
60 changes: 60 additions & 0 deletions quest/m3/relay-track-lock.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# [L] A live subscriber's track passes stop meeting on the track lock

## Goal

The relay's publisher serves a live subscriber's track side without
contending for the track's shared state lock: in `chat` and `fanout`, futex
instructions per delivered frame fall to `release`'s level (about 2.3k and
12.6k on 8 workers) at 2, 8, and 16 workers, and relay CPU per frame is no
higher than `release`'s. Drop forwarding and fills behave as before. Nothing
here changes an API or the wire.

## Plan

Found by [Relay CPU](https://github.com/moq-dev/moq/pull/8979), which
recorded the numbers in
[release-perf.md](/.github/release-perf.md#relay-cpu-2026-11-16) and accepted
the rest of the cost for the cut.

Decided 2026-11-17 (maintainer, on
[moq-dev/moq#8979](https://github.com/moq-dev/moq/pull/8979#issuecomment-6937624164)):
the cut does not wait on this; it follows the cut in m3. Rejected: moving it
to m2.

Facts:

- Every subscriber of a track wakes on each group at once. Its
`TrackRun::poll_track` (`rs/moq-net/src/lite/publisher.rs`) then runs three
passes per group (the arrival, the pass after serving it, the finish),
forwarding drops five times (`Subscriber::poll_drop_probing`) and judging
misses six (`Cursor::probe`, `Misses::judge`). Each takes the track lock
separately for the drift edge (`Cursor::poll_drift`), the judgement, and the
drop cursor (`DropCursor::next`), and the read path takes it again
(`poll_peek_group`, `take`, `forget_past_budget`).
- On 8 tokio workers, `chat`'s futex instructions per frame were 2.3k on
`release`, 8.5k on `main`, and 4.7k once the drop poll stopped reading the
stale revision it rarely needs. `fanout`'s were 12.6k, 33.8k, and 32.9k:
its 64 subscribers per track contend in the passes themselves, and its
forward-drop path is 65k instructions per frame, 30% of the relay.

Approach, in order of reach:

- A pass that changed nothing judges nothing: the drop cursor keeps the track
version (`kio::Channel::version`, as `kio::Watch` does) its last pending
judgement saw, with the cursor inputs it read (`next_key`, the floor, the
window's revision, the record, cuts, the subscription's budget), and parks
again at an unchanged version without the lock. A pending deadline (the
misses' deferred gaps, `watch_wall`) wakes without a version change, so a
cursor waiting on one judges again.
- One pass takes the lock once: the drift edge, the judgement, and the drop
cursor's next run share one guard.

Measure futex and total instructions per frame in both workloads with
`just bench-profile instructions chat "2 8 16" tokio-workers origin/main`,
and relay CPU with `just bench-relay origin/release quic`, both before and
after; `just bench-fanout` checks the single-threaded count did not grow.

## Related

- [kio channel contention](/quest/m9/kio-channel-contention/README.md) - the general lock-wait work this narrows to one channel
- [Performance audit](/quest/m3/audit-perf.md) - reruns the release pairing after the cut
16 changes: 10 additions & 6 deletions rs/moq-net/src/model/resume.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1949,14 +1949,18 @@ impl Reader {
// A received stale run queued here that new terms superseded since is unknown again:
// the copy no longer holds it, and its upstream re-offers or drops it again. Checked
// only once the copy let a stale run go, never on every poll: each poll reports one
// run, so a check per poll would cost the received runs squared to drain them.
let current = sub.stale_revision();
if self.received_checked != Some(current) {
for run in self.received.iter() {
superseded.extend(sub.stale_held(run.clone()).gaps(run));
// run, so a check per poll would cost the received runs squared to drain them. With
// none received there is nothing to check, so a live reader never reads the revision
// through the track lock every subscriber shares.
if !self.received.is_empty() {
let current = sub.stale_revision();
if self.received_checked != Some(current) {
for run in self.received.iter() {
superseded.extend(sub.stale_held(run.clone()).gaps(run));
}
}
self.received_checked = Some(current);
}
self.received_checked = Some(current);
loop {
let polled = match probe {
true => sub.poll_drop(waiter),
Expand Down