Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions .github/release-perf.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,71 @@
# Release performance comparison

## Relay memory (2026-11-16)

The 2026-11-05 pairing below recorded relay RSS but never reported it: `chat`,
with 2,001 connections, used 17% more on `main`. The cause was the relay's own
per-connection task, and [moq-dev/moq#8943](https://github.com/moq-dev/moq/pull/8943)
fixed it; `main` now uses less memory than `release` in every workload. The
comparison reports `rss_bytes` beside CPU from now on.

Five alternating rounds of `just bench-relay origin/release quic` on
`fedora` (32 CPUs, shared: the 1-minute load read 13 to 26 at each
snapshot), `release` at
[ca3b801a6c](https://github.com/moq-dev/moq/commit/ca3b801a6c), `main` at
[6488ce6b87](https://github.com/moq-dev/moq/commit/6488ce6b87), median peak
relay RSS with its range:

| Workload | release | main | Change | main with the fix | Change |
| --- | --- | --- | --- | --- | --- |
| `chat` | 427.0 MiB | 495.6 MiB | +16.1% (+14.1 to +29.4) | 392.1 MiB | -7.2% (-13.4 to -6.8) |
| `fanout` | 37.2 MiB | 41.1 MiB | +10.3% (+5.5 to +23.5) | 37.4 MiB | -2.7% (-5.2 to +4.0) |
| `video` | 30.5 MiB | 29.1 MiB | -3.5% (-14.1 to +0.7) | 27.4 MiB | -6.0% (-13.2 to -0.4) |

Each fix column is its own five-round run against `release`. `chat` RSS
before the fix was +35 KiB per connection, as the audit found (+37).

A jemalloc heap profile of `chat` on both sides
(`just bench-profile heap chat 8 tokio-workers origin/release`) attributes
the retained heap at the end of the steady window, by the innermost crate
that allocated it, in KiB per connection (main minus release):

| Owner | main | main with the fix |
| --- | --- | --- |
| relay (`moq-relay`'s per-connection tasks) | +65.3 | -13.1 |
| origin (`moq_net::model`) | +12.4 | +9.7 |
| session (`moq_net` protocol, `kio`) | -1.3 | +1.9 |
| transport (QUIC, TLS, the Tokio driver) | -38.3 | -38.6 |
| total | +38.8 | -39.2 |

- **Relay, fixed.** `Connection::run` passed its startup future by value
through `Lifecycle::start`, `Lifecycle::run` and `Lifecycle::isolate`, and
an async fn keeps its arguments for its whole life. So the connection's
task held 44.8 KiB (startup's largest state, twice) and a box beside it
held 20.4 KiB more, until the connection closed; serving needs 6.2 KiB.
Boxing the startup future before the task is built frees it once the
session starts, and building the supervision with combinators instead of
an async block drops its second copy of the session's state. The task is
now 1.4 KiB plus the 6.2 KiB supervision (`-Zprint-type-sizes`).
Holding startup's state for the session's life is as old as the task
(`release`'s holds about 27 KiB);
[moq-dev/moq#5761](https://github.com/moq-dev/moq/pull/5761) moved its box
behind by-value arguments, which put a second copy in the task itself.
- **Transport, a saving.** The in-tree QUIC stack holds about 38 KiB less
per connection than `noq` did.
- **Origin, accepted.** About 10 KiB per connection, one reader each in
`chat`, spread across bookkeeping `release` lacks. The largest, 4 KiB, is the record of what a reader was
told about each group ([moq-dev/moq#8440](https://github.com/moq-dev/moq/pull/8440));
the one-delivery record ([moq-dev/moq#5446](https://github.com/moq-dev/moq/pull/5446)),
demand preferences ([moq-dev/moq#5670](https://github.com/moq-dev/moq/pull/5670))
and routes are each about 1 KiB. Both records are billed to the cache
pool, so a peer cannot grow them past it.

The same pairing measured `chat` and `fanout` relay CPU per delivered frame
+43% and +35% on `main` (every pair slower), the opposite of the 2026-11-05
result. [Relay CPU](/quest/m2/relay-cpu.md) owns explaining that before the
cut. The run directories stayed in the measuring checkout's disposable
`.tmp/bench`; the numbers above are the record.

## Final relay pairing (2026-11-05)

The remaining fanout check passes: main uses less relay CPU in all seven
Expand Down
5 changes: 3 additions & 2 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,9 @@ nix develop --command just bench origin/release "quic ws wss tcp"
```

Beside relay CPU in cores, each case
reports CPU microseconds per delivered frame and cores per delivered Gbps; base
comparisons pair those too.
reports CPU microseconds per delivered frame, cores per delivered Gbps, and the
relay's peak RSS over the window; base comparisons pair those too. Only `chat`,
with 2,001 connections, shows a per-connection memory change.

The qmux micro-bench isolates the transport from the relay. It connects
client/server pairs over loopback TCP, each echoing one message at a time on one
Expand Down
9 changes: 6 additions & 3 deletions bench/compare.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,11 @@
"generator_cpu_cores",
"loss_pct",
"latency_p99_ms",
# The relay's peak resident memory over the window; per-connection state shows here, not in CPU.
"rss_bytes",
)
CPU_METRICS = ("cpu_cores", "cpu_us_per_frame", "cpu_cores_per_gbps", "generator_cpu_cores")
# Read from the host sampler, which runs only on Linux.
HOST_METRICS = ("cpu_cores", "cpu_us_per_frame", "cpu_cores_per_gbps", "generator_cpu_cores", "rss_bytes")


def value(row, metric):
Expand Down Expand Up @@ -54,10 +57,10 @@ def reduce(pairs):
"metrics": {},
}
for metric in METRICS:
if metric in CPU_METRICS and all(
if metric in HOST_METRICS and all(
value(pair[label], metric) is None for pair in pairs for label in ("base", "current")
):
result["metrics"][metric] = {"delta_pct": None, "status": "unavailable: no host CPU samples"}
result["metrics"][metric] = {"delta_pct": None, "status": "unavailable: no host samples"}
continue
deltas = []
absolute = []
Expand Down
13 changes: 9 additions & 4 deletions bench/compare.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ def test_stable_aa(self):
def test_known_degradation(self):
self.assertAlmostEqual(compare.reduce([pair(-20)] * 3)["metrics"]["recv_mbps"]["delta_pct"], -20)

def test_memory_is_compared(self):
self.assertAlmostEqual(compare.reduce([pair(17)] * 3)["metrics"]["rss_bytes"]["delta_pct"], 17)

def test_noisy(self):
metric = compare.reduce([pair(-10), pair(), pair(10)])["metrics"]["recv_mbps"]
self.assertEqual(metric["status"], "sign varies across pairs")
Expand All @@ -34,12 +37,14 @@ def test_noisy(self):
def test_insufficient(self):
self.assertEqual(compare.reduce([pair()])["metrics"]["recv_mbps"]["status"], "insufficient samples")

def test_cpu_unavailable_on_both_sides_is_explicit(self):
def test_host_metrics_unavailable_on_both_sides_are_explicit(self):
p = pair()
del p["base"]["cpu_cores"]
del p["current"]["cpu_cores"]
for metric in ("cpu_cores", "rss_bytes"):
del p["base"][metric]
del p["current"][metric]
result = compare.reduce([p] * 3)
self.assertEqual(result["metrics"]["cpu_cores"]["status"], "unavailable: no host CPU samples")
for metric in ("cpu_cores", "rss_bytes"):
self.assertEqual(result["metrics"][metric]["status"], "unavailable: no host samples")
self.assertEqual(result["metrics"]["recv_mbps"]["delta_pct"], 0)

def test_missing_invalid(self):
Expand Down
4 changes: 2 additions & 2 deletions quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ removal, a rename, or a wire, catalog, or format change that is free until
demos return to their earlier format.
- **Unused API.** [One sweep per area](/quest/m2/unused-api/README.md), each
item re-checked for a downstream caller first.
- **Relay and transports.** [Memory](/quest/m2/relay-memory.md), which the
cut waits on by name; [congestion names](/quest/m2/congestion-names.md);
- **Relay and transports.** [CPU](/quest/m2/relay-cpu.md), which the
memory pairing found; [congestion names](/quest/m2/congestion-names.md);
[config names](/quest/m2/relay-config-names.md),
[the two dial keys](/quest/m2/relay-connect-keys.md), and
[the 0.14 leftovers](/quest/m2/relay-014-leftovers.md);
Expand Down
59 changes: 59 additions & 0 deletions quest/m2/relay-cpu.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# [M] The relay's chat and fanout CPU growth since release is explained before the cut

## Goal

The relay's extra CPU per delivered frame on `main` against `release` in the
`chat` and `fanout` workloads has a named cause, and the cut ships with it
either fixed or written down as the price of a feature, in
`.github/release-perf.md` beside the
[memory finding](/.github/release-perf.md#relay-memory-2026-11-16). Nothing
here changes an API or the wire.

## Plan

Found 2026-11-16 by [relay memory](https://github.com/moq-dev/moq/pull/8943),
whose pairing reported CPU beside RSS. Placed in m2, like the memory quest,
so the cut waits on it: the 2026-11-05 pairing said `main` used less relay
CPU in every workload, and 0.22 should not be announced on that claim while
two workloads now say the opposite.

Facts:

- Five alternating rounds of `just bench-relay origin/release quic`
(`release` at `ca3b801a6c`, `main` at `6488ce6b87`, host load 24 to 28 on
32 cores): `cpu_us_per_frame` +43% in `chat` (range +24% to +59%) and +35%
in `fanout` (range +3% to +65%), every pair slower; `video` -24%. Delivered
frames per second matched. The heap-profile controls on 8 Tokio workers
agree: `chat` relay CPU 1.40 to 1.52 cores on `release`, 1.96 to 2.15 on
`main`.
- `main`'s relay allocates 2.3 times the bytes per second in `chat` (about
300 MiB/s against 133), though fewer allocations. Half of it is
`moq_quic::connection::slots::Slots::insert`: a connection that has every
packet acknowledged before it sends the next frees its 64-slot chunk
(about 8 KiB of `Option<SentPacket>`) and allocates a fresh one for the
next packet ([moq-dev/moq#6936](https://github.com/moq-dev/moq/pull/6936)).
That is one 8 KiB allocation per packet in `chat`. A lead, not a measured
cause: count instructions before and after keeping one spare chunk.
- The lifetime `loss_pct` also rose (about 1.5% against 0.1%), all of it
during the startup ramp: `groups_lost` stops growing once the load is
ready, so steady delivery is unchanged. It may be groups `main` now
reports dropped as a subscriber joins; confirm that before reading it as
a regression.

For the PR: attribute the CPU with `just bench-profile cpu chat 8
tokio-workers origin/release`, counted in instructions as the
[performance audit's Method](/quest/m3/audit-perf.md#method) does, by
owner: transport, session, origin, other. Fix what is accidental; record a
feature's cost with its PR.

## Waiting

- Maintainer: whether the cut waits on this. Recommended: yes, as it does on
memory, since the release comparison now claims the opposite of what
`main` measures. Otherwise move it under [m3](/quest/m3/README.md) with
`quest mv`, where the performance audit's release run picks it up after
the cut.

## Related

- [Performance audit](/quest/m3/audit-perf.md) - reruns the release pairing after the cut
77 changes: 0 additions & 77 deletions quest/m2/relay-memory.md

This file was deleted.

7 changes: 5 additions & 2 deletions quest/m3/audit-perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,11 @@ bun bench/browser/micro.ts --bench frames --out <scratch>/frames.json base=<rele
```

`<release>` is an export of `origin/release` with its own
`bun install --frozen-lockfile`. Re-run every slower case in alternating
rounds and count instructions for any that holds. `main` carries features
`bun install --frozen-lockfile`. The relay comparison reports relay RSS
(`rss_bytes`) beside CPU, and the report gives both per workload; memory that
grows with connections shows only in `chat`. Re-run every slower or larger case in alternating
rounds and count instructions for any that holds, or take a heap profile
(`just bench-profile heap chat 1 tokio-workers origin/release`) for memory. `main` carries features
`release` lacks, so a flat per-operation cost usually pays for one, while a
cost that grows with `main`'s own sweep is accidental. Fix accidental cost in
this run, and record each accepted cost with the feature and PR it pays for.
Expand Down
5 changes: 0 additions & 5 deletions quest/m3/release/cut.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,10 +211,6 @@ with the suites above.

Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit):

- ✅ The cut waits on [the relay's memory growth](/quest/m2/relay-memory.md)
by name: the chat scenario's relay uses about 17% more memory on `main`
than on `release`, and 0.22 is not announced with that unexplained
(rejected: cutting and investigating after).
- ✅ The cut waits on [the rind 0.4 adapters](/quest/m0/rind-adapters.md)
([moq-dev/moq#8432](https://github.com/moq-dev/moq/pull/8432)), so the
`UnsupportedCpu` error that exists only for rind 0.3 is never published.
Expand All @@ -235,7 +231,6 @@ Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit):
## Required

- [Breaking release readiness](/quest/m2/README.md) - code, protocol and package proofs
- [Relay memory](/quest/m2/relay-memory.md) - the relay's memory growth since `release` is fixed or recorded before 0.22 is announced
- [rind 0.4 adapters](/quest/m0/rind-adapters.md) - `UnsupportedCpu`, which exists only for rind 0.3, never publishes
- [Migration audit](/quest/m3/release/migrate-audit.md) - every break on the cut PR has its MIGRATE.md entry
- [API shape review](/quest/m3/release/api-review.md) - every new or changed public API was reviewed for a better shape, and approved changes landed
Expand Down
Loading