diff --git a/.github/release-perf.md b/.github/release-perf.md index e83880aa563..52e2bcb8e87 100644 --- a/.github/release-perf.md +++ b/.github/release-perf.md @@ -1,5 +1,71 @@ # Release performance comparison +## Relay memory (2026-11-16) + +The 2026-11-05 pairing below recorded relay RSS but never reported it: `chat`, +with 2,001 connections, used 17% more on `main`. The cause was the relay's own +per-connection task, and [moq-dev/moq#8943](https://github.com/moq-dev/moq/pull/8943) +fixed it; `main` now uses less memory than `release` in every workload. The +comparison reports `rss_bytes` beside CPU from now on. + +Five alternating rounds of `just bench-relay origin/release quic` on +`fedora` (32 CPUs, shared: the 1-minute load read 13 to 26 at each +snapshot), `release` at +[ca3b801a6c](https://github.com/moq-dev/moq/commit/ca3b801a6c), `main` at +[6488ce6b87](https://github.com/moq-dev/moq/commit/6488ce6b87), median peak +relay RSS with its range: + +| Workload | release | main | Change | main with the fix | Change | +| --- | --- | --- | --- | --- | --- | +| `chat` | 427.0 MiB | 495.6 MiB | +16.1% (+14.1 to +29.4) | 392.1 MiB | -7.2% (-13.4 to -6.8) | +| `fanout` | 37.2 MiB | 41.1 MiB | +10.3% (+5.5 to +23.5) | 37.4 MiB | -2.7% (-5.2 to +4.0) | +| `video` | 30.5 MiB | 29.1 MiB | -3.5% (-14.1 to +0.7) | 27.4 MiB | -6.0% (-13.2 to -0.4) | + +Each fix column is its own five-round run against `release`. `chat` RSS +before the fix was +35 KiB per connection, as the audit found (+37). + +A jemalloc heap profile of `chat` on both sides +(`just bench-profile heap chat 8 tokio-workers origin/release`) attributes +the retained heap at the end of the steady window, by the innermost crate +that allocated it, in KiB per connection (main minus release): + +| Owner | main | main with the fix | +| --- | --- | --- | +| relay (`moq-relay`'s per-connection tasks) | +65.3 | -13.1 | +| origin (`moq_net::model`) | +12.4 | +9.7 | +| session (`moq_net` protocol, `kio`) | -1.3 | +1.9 | +| transport (QUIC, TLS, the Tokio driver) | -38.3 | -38.6 | +| total | +38.8 | -39.2 | + +- **Relay, fixed.** `Connection::run` passed its startup future by value + through `Lifecycle::start`, `Lifecycle::run` and `Lifecycle::isolate`, and + an async fn keeps its arguments for its whole life. So the connection's + task held 44.8 KiB (startup's largest state, twice) and a box beside it + held 20.4 KiB more, until the connection closed; serving needs 6.2 KiB. + Boxing the startup future before the task is built frees it once the + session starts, and building the supervision with combinators instead of + an async block drops its second copy of the session's state. The task is + now 1.4 KiB plus the 6.2 KiB supervision (`-Zprint-type-sizes`). + Holding startup's state for the session's life is as old as the task + (`release`'s holds about 27 KiB); + [moq-dev/moq#5761](https://github.com/moq-dev/moq/pull/5761) moved its box + behind by-value arguments, which put a second copy in the task itself. +- **Transport, a saving.** The in-tree QUIC stack holds about 38 KiB less + per connection than `noq` did. +- **Origin, accepted.** About 10 KiB per connection, one reader each in + `chat`, spread across bookkeeping `release` lacks. The largest, 4 KiB, is the record of what a reader was + told about each group ([moq-dev/moq#8440](https://github.com/moq-dev/moq/pull/8440)); + the one-delivery record ([moq-dev/moq#5446](https://github.com/moq-dev/moq/pull/5446)), + demand preferences ([moq-dev/moq#5670](https://github.com/moq-dev/moq/pull/5670)) + and routes are each about 1 KiB. Both records are billed to the cache + pool, so a peer cannot grow them past it. + +The same pairing measured `chat` and `fanout` relay CPU per delivered frame ++43% and +35% on `main` (every pair slower), the opposite of the 2026-11-05 +result. [Relay CPU](/quest/m2/relay-cpu.md) owns explaining that before the +cut. The run directories stayed in the measuring checkout's disposable +`.tmp/bench`; the numbers above are the record. + ## Final relay pairing (2026-11-05) The remaining fanout check passes: main uses less relay CPU in all seven diff --git a/bench/README.md b/bench/README.md index 887f644848e..25a81ae5290 100644 --- a/bench/README.md +++ b/bench/README.md @@ -168,8 +168,9 @@ nix develop --command just bench origin/release "quic ws wss tcp" ``` Beside relay CPU in cores, each case -reports CPU microseconds per delivered frame and cores per delivered Gbps; base -comparisons pair those too. +reports CPU microseconds per delivered frame, cores per delivered Gbps, and the +relay's peak RSS over the window; base comparisons pair those too. Only `chat`, +with 2,001 connections, shows a per-connection memory change. The qmux micro-bench isolates the transport from the relay. It connects client/server pairs over loopback TCP, each echoing one message at a time on one diff --git a/bench/compare.py b/bench/compare.py index 4652de1418a..0b1a0d3922e 100644 --- a/bench/compare.py +++ b/bench/compare.py @@ -20,8 +20,11 @@ "generator_cpu_cores", "loss_pct", "latency_p99_ms", + # The relay's peak resident memory over the window; per-connection state shows here, not in CPU. + "rss_bytes", ) -CPU_METRICS = ("cpu_cores", "cpu_us_per_frame", "cpu_cores_per_gbps", "generator_cpu_cores") +# Read from the host sampler, which runs only on Linux. +HOST_METRICS = ("cpu_cores", "cpu_us_per_frame", "cpu_cores_per_gbps", "generator_cpu_cores", "rss_bytes") def value(row, metric): @@ -54,10 +57,10 @@ def reduce(pairs): "metrics": {}, } for metric in METRICS: - if metric in CPU_METRICS and all( + if metric in HOST_METRICS and all( value(pair[label], metric) is None for pair in pairs for label in ("base", "current") ): - result["metrics"][metric] = {"delta_pct": None, "status": "unavailable: no host CPU samples"} + result["metrics"][metric] = {"delta_pct": None, "status": "unavailable: no host samples"} continue deltas = [] absolute = [] diff --git a/bench/compare.test.py b/bench/compare.test.py index fc200c215d5..939d458a069 100644 --- a/bench/compare.test.py +++ b/bench/compare.test.py @@ -26,6 +26,9 @@ def test_stable_aa(self): def test_known_degradation(self): self.assertAlmostEqual(compare.reduce([pair(-20)] * 3)["metrics"]["recv_mbps"]["delta_pct"], -20) + def test_memory_is_compared(self): + self.assertAlmostEqual(compare.reduce([pair(17)] * 3)["metrics"]["rss_bytes"]["delta_pct"], 17) + def test_noisy(self): metric = compare.reduce([pair(-10), pair(), pair(10)])["metrics"]["recv_mbps"] self.assertEqual(metric["status"], "sign varies across pairs") @@ -34,12 +37,14 @@ def test_noisy(self): def test_insufficient(self): self.assertEqual(compare.reduce([pair()])["metrics"]["recv_mbps"]["status"], "insufficient samples") - def test_cpu_unavailable_on_both_sides_is_explicit(self): + def test_host_metrics_unavailable_on_both_sides_are_explicit(self): p = pair() - del p["base"]["cpu_cores"] - del p["current"]["cpu_cores"] + for metric in ("cpu_cores", "rss_bytes"): + del p["base"][metric] + del p["current"][metric] result = compare.reduce([p] * 3) - self.assertEqual(result["metrics"]["cpu_cores"]["status"], "unavailable: no host CPU samples") + for metric in ("cpu_cores", "rss_bytes"): + self.assertEqual(result["metrics"][metric]["status"], "unavailable: no host samples") self.assertEqual(result["metrics"]["recv_mbps"]["delta_pct"], 0) def test_missing_invalid(self): diff --git a/quest/m2/README.md b/quest/m2/README.md index 42df4d4cfa6..fc639cfdcf0 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -42,8 +42,8 @@ removal, a rename, or a wire, catalog, or format change that is free until demos return to their earlier format. - **Unused API.** [One sweep per area](/quest/m2/unused-api/README.md), each item re-checked for a downstream caller first. -- **Relay and transports.** [Memory](/quest/m2/relay-memory.md), which the - cut waits on by name; [congestion names](/quest/m2/congestion-names.md); +- **Relay and transports.** [CPU](/quest/m2/relay-cpu.md), which the + memory pairing found; [congestion names](/quest/m2/congestion-names.md); [config names](/quest/m2/relay-config-names.md), [the two dial keys](/quest/m2/relay-connect-keys.md), and [the 0.14 leftovers](/quest/m2/relay-014-leftovers.md); diff --git a/quest/m2/relay-cpu.md b/quest/m2/relay-cpu.md new file mode 100644 index 00000000000..00d3e92a4cd --- /dev/null +++ b/quest/m2/relay-cpu.md @@ -0,0 +1,59 @@ +# [M] The relay's chat and fanout CPU growth since release is explained before the cut + +## Goal + +The relay's extra CPU per delivered frame on `main` against `release` in the +`chat` and `fanout` workloads has a named cause, and the cut ships with it +either fixed or written down as the price of a feature, in +`.github/release-perf.md` beside the +[memory finding](/.github/release-perf.md#relay-memory-2026-11-16). Nothing +here changes an API or the wire. + +## Plan + +Found 2026-11-16 by [relay memory](https://github.com/moq-dev/moq/pull/8943), +whose pairing reported CPU beside RSS. Placed in m2, like the memory quest, +so the cut waits on it: the 2026-11-05 pairing said `main` used less relay +CPU in every workload, and 0.22 should not be announced on that claim while +two workloads now say the opposite. + +Facts: + +- Five alternating rounds of `just bench-relay origin/release quic` + (`release` at `ca3b801a6c`, `main` at `6488ce6b87`, host load 24 to 28 on + 32 cores): `cpu_us_per_frame` +43% in `chat` (range +24% to +59%) and +35% + in `fanout` (range +3% to +65%), every pair slower; `video` -24%. Delivered + frames per second matched. The heap-profile controls on 8 Tokio workers + agree: `chat` relay CPU 1.40 to 1.52 cores on `release`, 1.96 to 2.15 on + `main`. +- `main`'s relay allocates 2.3 times the bytes per second in `chat` (about + 300 MiB/s against 133), though fewer allocations. Half of it is + `moq_quic::connection::slots::Slots::insert`: a connection that has every + packet acknowledged before it sends the next frees its 64-slot chunk + (about 8 KiB of `Option`) and allocates a fresh one for the + next packet ([moq-dev/moq#6936](https://github.com/moq-dev/moq/pull/6936)). + That is one 8 KiB allocation per packet in `chat`. A lead, not a measured + cause: count instructions before and after keeping one spare chunk. +- The lifetime `loss_pct` also rose (about 1.5% against 0.1%), all of it + during the startup ramp: `groups_lost` stops growing once the load is + ready, so steady delivery is unchanged. It may be groups `main` now + reports dropped as a subscriber joins; confirm that before reading it as + a regression. + +For the PR: attribute the CPU with `just bench-profile cpu chat 8 +tokio-workers origin/release`, counted in instructions as the +[performance audit's Method](/quest/m3/audit-perf.md#method) does, by +owner: transport, session, origin, other. Fix what is accidental; record a +feature's cost with its PR. + +## Waiting + +- Maintainer: whether the cut waits on this. Recommended: yes, as it does on + memory, since the release comparison now claims the opposite of what + `main` measures. Otherwise move it under [m3](/quest/m3/README.md) with + `quest mv`, where the performance audit's release run picks it up after + the cut. + +## Related + +- [Performance audit](/quest/m3/audit-perf.md) - reruns the release pairing after the cut diff --git a/quest/m2/relay-memory.md b/quest/m2/relay-memory.md deleted file mode 100644 index 5e96a40648f..00000000000 --- a/quest/m2/relay-memory.md +++ /dev/null @@ -1,77 +0,0 @@ -# [M] The relay's memory growth since release is explained before the cut - -## Goal - -The relay's extra memory per connection on `main` against `release` has a -named cause, and the cut ships with it either fixed or written down as the -price of a feature. The release comparison reports memory beside CPU, so the -next one cannot miss it. Nothing here changes an API or the wire. - -## Plan - -Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit): - -- ✅ Profile it and fix or record it before the cut, and the cut waits - (rejected: writing the number into the report and shipping; leaving it to - the [performance audit](/quest/m3/audit-perf.md), whose release run comes - after the cut by its own `Required`). A regression found after 0.22 is - announced costs a patch release and a correction. -- ✅ The release comparison reports memory (rejected: CPU only, as today). - -Facts found by the audit: - -- `.github/release-perf.json` is the 2026-11-05 pairing of `release` - (`dc3478a863`) against `main` (`56d7989c78`), seven alternating rounds. - The median of `summaries[*].rss_bytes` per workload, relay process: - - | Workload | release | main | Change | - | --- | --- | --- | --- | - | `video` | 31.0 MiB | 28.4 MiB | -8% | - | `fanout` | 35.8 MiB | 35.5 MiB | -1% | - | `chat` (2,001 connections) | 428.3 MiB | 501.8 MiB | +17% | - - The `chat` ranges never overlap (428.1 to 429.0 against 499.6 to 503.1), - and the difference is about 37 KiB per connection. -- `.github/release-perf.md`, the report written from that file, never - mentions RSS, memory, or MiB (`grep -n -i 'rss\|memory\|MiB'` finds - nothing), so nobody read the number. -- The only earlier memory evidence for the in-tree QUIC stack is - [moq-dev/moq#5664](https://github.com/moq-dev/moq/pull/5664): 256 - sessions, measured on a host at load 20 to 56. A downstream deployment's - condition for relying on the stack is that relay memory matches the old - one. -- The comparison is of whole relays, so the growth may be the QUIC stack - (per-connection buffers, the 8 MiB socket buffers), moq-net (per-session - route and announce state), or the allocator. `grep -rni rss quest/` finds - only benchmark plumbing: no quest owned it. - -For the PR: - -1. Reproduce on today's `main` and `origin/release` with - `bench/workloads/chat.toml`, alternating rounds, recording - `/proc/loadavg`, with the evidence kept outside the checkout. `main` has - moved about 850 commits since the pairing, so the number comes first. -2. Attribute it with a heap profile of the relay on both sides - (`moq_tokio::jemalloc` is on-demand heap profiling; the pairing itself - ran the system allocator). Report bytes per connection by owner: - transport, session, origin, other. -3. Fix what is accidental in this PR. A cost that pays for a feature stays, - recorded in `.github/release-perf.md` with the feature and its PR, as - the performance audit records accepted CPU costs. A fix too large to - review here is a follow-up quest in m2, and this quest lists it under - `Required` until it lands. -4. Report RSS per workload in the release comparison. Each run's summary - already records `rss_bytes` (`bench/summary.py:91`), but - `bench/compare.py` never compares it, and the Release baseline section - of `quest/m3/audit-perf.md` never asks for it. It gets the same - accept-or-fix rule as CPU. - -Do not set a threshold: the outcome is a cause and a decision, not a number. -If the cause turns out to be a public default (a buffer size, a window), -changing it is the maintainer's call; bring the profile. - -Public API: none. Wire: none. Config: none expected. - -## Related - -- [Performance audit](/quest/m3/audit-perf.md) - reruns the release pairing after the cut and inherits the memory column diff --git a/quest/m3/audit-perf.md b/quest/m3/audit-perf.md index abcf3acca8a..bf1f5281fb2 100644 --- a/quest/m3/audit-perf.md +++ b/quest/m3/audit-perf.md @@ -36,8 +36,11 @@ bun bench/browser/micro.ts --bench frames --out /frames.json base=` is an export of `origin/release` with its own -`bun install --frozen-lockfile`. Re-run every slower case in alternating -rounds and count instructions for any that holds. `main` carries features +`bun install --frozen-lockfile`. The relay comparison reports relay RSS +(`rss_bytes`) beside CPU, and the report gives both per workload; memory that +grows with connections shows only in `chat`. Re-run every slower or larger case in alternating +rounds and count instructions for any that holds, or take a heap profile +(`just bench-profile heap chat 1 tokio-workers origin/release`) for memory. `main` carries features `release` lacks, so a flat per-operation cost usually pays for one, while a cost that grows with `main`'s own sweep is accidental. Fix accidental cost in this run, and record each accepted cost with the feature and PR it pays for. diff --git a/quest/m3/release/cut.md b/quest/m3/release/cut.md index 17530884aa7..9c8268a70b1 100644 --- a/quest/m3/release/cut.md +++ b/quest/m3/release/cut.md @@ -211,10 +211,6 @@ with the suites above. Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit): -- ✅ The cut waits on [the relay's memory growth](/quest/m2/relay-memory.md) - by name: the chat scenario's relay uses about 17% more memory on `main` - than on `release`, and 0.22 is not announced with that unexplained - (rejected: cutting and investigating after). - ✅ The cut waits on [the rind 0.4 adapters](/quest/m0/rind-adapters.md) ([moq-dev/moq#8432](https://github.com/moq-dev/moq/pull/8432)), so the `UnsupportedCpu` error that exists only for rind 0.3 is never published. @@ -235,7 +231,6 @@ Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit): ## Required - [Breaking release readiness](/quest/m2/README.md) - code, protocol and package proofs -- [Relay memory](/quest/m2/relay-memory.md) - the relay's memory growth since `release` is fixed or recorded before 0.22 is announced - [rind 0.4 adapters](/quest/m0/rind-adapters.md) - `UnsupportedCpu`, which exists only for rind 0.3, never publishes - [Migration audit](/quest/m3/release/migrate-audit.md) - every break on the cut PR has its MIGRATE.md entry - [API shape review](/quest/m3/release/api-review.md) - every new or changed public API was reviewed for a better shape, and approved changes landed diff --git a/rs/moq-relay/src/connection.rs b/rs/moq-relay/src/connection.rs index ee68dfaece6..0d0479dc8a2 100644 --- a/rs/moq-relay/src/connection.rs +++ b/rs/moq-relay/src/connection.rs @@ -81,20 +81,23 @@ impl Connection { /// Admits and serves this connection until it closes, then logs the close. A panic /// ends only this connection, as an error. - pub async fn run(self) { + pub fn run(self) -> impl std::future::Future { let Self { request, lifecycle } = self; - match request.shared().cloned() { - Some(shared) => lifecycle.start(&shared, Self::start(request, &lifecycle)).await, - None => { - lifecycle - .run(async { Self::start(request, &lifecycle).await?.await }) - .await + let shared = request.shared().cloned(); + // Boxed before the task is built, so it is freed once the session starts: the task holds + // what it captures, and any future it awaits inline, for the connection's whole life, + // and starting takes several times the memory serving does. + let start = Box::pin(Self::start(request, lifecycle.clone())); + async move { + match lifecycle.isolate(start).await { + Ok(supervision) => lifecycle.serve(shared.as_ref(), supervision).await, + Err(err) => lifecycle.closed(Err(err)), } } } /// Admit the request and answer its SETUP, returning the session's supervision. - async fn start(mut request: R, lifecycle: &Lifecycle) -> anyhow::Result { + async fn start(mut request: R, lifecycle: Lifecycle) -> anyhow::Result { // Only this relay's origin holds its identity, so a session declaring it is this // relay dialing itself under a spelling its self filter missed. Every route // from it would be dropped as reflected, so refuse it rather than serve it. @@ -177,18 +180,19 @@ impl Connection { reply = request.ok() => reply?, }; + tracing::info!(version = %session.version(), %transport, "negotiated"); // Joined rather than awaited in turn, for a driver that runs only when polled. A // spawned driver's panic still surfaces here, so it cannot look like cancellation. - let lifecycle = lifecycle.clone(); - Ok(async move { - let (result, driven) = futures::join!( - lifecycle.supervise(serving, session, lease, registration, transport), - R::drive(driver), - ); - driven?; - result - } - .boxed()) + // Mapped, not awaited in an async block, which would hold a second copy of the + // session's state beside the one `supervise` holds. + Ok( + futures::future::join(supervise(session, lease, serving, registration), R::drive(driver)) + .map(|(result, driven)| { + driven?; + result + }) + .boxed(), + ) } } @@ -228,33 +232,34 @@ impl Lifecycle { self } - /// Run `task` in this connection's span, then log its close; a panic closes it as an error. - async fn run(&self, task: impl std::future::Future>) { - let result = self.isolate(task).await; - self.closed(result); - } - - /// Run the connection task's half, then supervise the started session on `runtime`, which logs - /// the close there: a worker stopped at shutdown drops its tasks, and the line with them. - async fn start( - &self, - runtime: &tokio::runtime::Handle, - start: impl std::future::Future>, - ) where - S: std::future::Future> + Send + 'static, - { - match self.isolate(start).await { - Ok(supervise) => { + /// Supervise a started session, on `runtime` when given, then log its close there: a + /// worker stopped at shutdown drops its tasks, and the close line with them. + async fn serve(&self, runtime: Option<&tokio::runtime::Handle>, supervision: Supervision) { + match runtime { + Some(runtime) => { let lifecycle = self.clone(); - runtime.spawn(async move { lifecycle.run(supervise).await }.with_current_subscriber()); + runtime.spawn(async move { lifecycle.run(supervision).await }.with_current_subscriber()); } - Err(err) => self.closed(Err(err)), + None => self.run(supervision).await, } } - async fn isolate(&self, task: impl std::future::Future>) -> anyhow::Result { - // Boxing keeps large protocol futures off nested debug-build stack frames. - moq_tokio::panic::isolate(Box::pin(task)) + /// Supervise a started session in this connection's span, then log its close; a panic + /// closes it as an error. + async fn run(&self, supervision: Supervision) { + let result = self.isolate(supervision).await; + self.closed(result); + } + + /// Run `task` in this connection's span, returning a panic as an error. + /// + /// The task comes boxed: an async fn keeps its arguments for its whole life, so a task + /// passed inline would stay in the caller's state after it finished. + async fn isolate(&self, task: std::pin::Pin>) -> anyhow::Result + where + F: std::future::Future> + ?Sized, + { + moq_tokio::panic::isolate(task) .instrument(self.span.clone()) .await .unwrap_or_else(|| Err(anyhow::anyhow!("connection panicked"))) @@ -383,18 +388,6 @@ impl Lifecycle { let registration = self.sessions.as_ref().map(|sessions| sessions.register(request)); Ok((admitted, registration)) } - - async fn supervise( - &self, - serving: crate::shutdown::Serving, - session: moq_net::Session, - lease: auth::Lease, - registration: Option, - transport: impl std::fmt::Display, - ) -> anyhow::Result<()> { - tracing::info!(version = %session.version(), %transport, "negotiated"); - supervise(session, lease, serving, registration).await - } } /// Hold an accepted session open for as long as its lease allows. @@ -504,17 +497,16 @@ mod tests { let cluster = cluster::Cluster::spawn(cluster::Options::default()).unwrap(); let (auth, _admissions) = auth::Auth::embedded("test"); let lifecycle = Lifecycle::new(cluster, auth, 42, None); + let supervision = async { + panic!("injected supervision panic"); + #[expect( + unreachable_code, + reason = "the panic is the point; the `Ok` names the future's output type" + )] + Ok::<_, anyhow::Error>(()) + }; lifecycle - .start(&tokio::runtime::Handle::current(), async { - Ok::<_, anyhow::Error>(async { - panic!("injected supervision panic"); - #[expect( - unreachable_code, - reason = "the panic is the point; the `Ok` names the future's output type" - )] - Ok::<_, anyhow::Error>(()) - }) - }) + .serve(Some(&tokio::runtime::Handle::current()), supervision.boxed()) .await; logs.wait_for("connection closed").await; let text = logs.text(); diff --git a/rs/moq-relay/src/incoming.rs b/rs/moq-relay/src/incoming.rs index d065b759781..0dbf2c55960 100644 --- a/rs/moq-relay/src/incoming.rs +++ b/rs/moq-relay/src/incoming.rs @@ -95,7 +95,7 @@ pub trait Sealed: Sized { async fn ok(self) -> anyhow::Result<(moq_net::Session, Self::Driver)>; /// Run the session's protocol until it ends, alongside its supervision. - fn drive(driver: Self::Driver) -> impl std::future::Future> + Send; + fn drive(driver: Self::Driver) -> impl std::future::Future> + Send + 'static; } impl Sealed for moq_tokio::server::Request {