From 7983a0bb6e781dfcdbf29c0550fb6900de506e75 Mon Sep 17 00:00:00 2001 From: Zac Shenker Date: Thu, 5 Nov 2026 21:19:32 -0700 Subject: [PATCH 1/5] quest: add the sendmmsg prototype's end-to-end measurements Co-Authored-By: Claude Opus 5.5 --- quest/m5/quic-egress-profile.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/quest/m5/quic-egress-profile.md b/quest/m5/quic-egress-profile.md index 710bb85a5ab..06c2af08a33 100644 --- a/quest/m5/quic-egress-profile.md +++ b/quest/m5/quic-egress-profile.md @@ -86,6 +86,33 @@ Then, only where the profile shows a cost: last byte): p50 rose from 36 to 72 µs to 258 to 352 µs. On the small-object and audio points, p99 rose from 104 to 705 µs to 606 to 770 µs. + **End to end, through a QUIC reverse proxy.** The same prototype, measured as + one server shard behind a QUIC reverse proxy. Clients connect to the proxy, + and the proxy connects to the server over loopback with an 8,192-byte UDP + payload. Clients are moq-bench fan-out viewers: audio-rate, then 3 Mbit/s + video, 5 min each. The four runs ran in ABBA order (off, on, on, off). Server + CPU is in cores, as mean / max. + + | Load (audio / video viewers) | Batching | Audio p50 / p99, lost groups | Audio CPU | Video p99 | + |---|---|---|---|---| + | 300 / 150 | off | 11 / 38 ms, 6,090 | 0.492 / 0.596 | 21 ms | + | 300 / 150 | on | 13 / 54 ms, 9,781 | 0.490 / 0.560 | 26 ms | + | 400 / 200 | on | 27 / 110 ms, 150,424 | 0.596 / 0.695 | 79 ms | + | 400 / 200 | off | 27 / 212 ms, 325,694 | 0.688 / 0.779 | 31 ms | + + - **Below the server's ceiling, batching saved no CPU and raised p99.** The + loopback harness's 8–22% saving did not carry over, and why hasn't been + measured. + - **Only past the ceiling (400 audio) did it cut CPU, by 13%,** and the losses + with it. But the shard was overloaded either way, and video's p99 rose from + 31 to 79 ms. + - **The losses at 300 came from the proxy's client-facing socket buffer**, not + the server. With a 1 MiB receive buffer on the proxy, that point loses + nothing. + + So on this path the latency cost shows up and the CPU saving doesn't. That + argues for keeping batching opt-in and off by default, if it ships. + Its shape: - `UdpSocketState::try_send_many` on Linux sends up to 64 transmits, one `mmsghdr` each, in one `sendmmsg`, each with its own GSO segment size, From c2d6c2a9a366bdcf566900771989d881be592f01 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 7 Nov 2026 12:09:05 -0800 Subject: [PATCH 2/5] quest: qualify the sendmmsg end-to-end conclusions Co-Authored-By: Claude Opus 5.5 --- quest/m5/quic-egress-profile.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/quest/m5/quic-egress-profile.md b/quest/m5/quic-egress-profile.md index 06c2af08a33..566e58b7652 100644 --- a/quest/m5/quic-egress-profile.md +++ b/quest/m5/quic-egress-profile.md @@ -90,8 +90,9 @@ Then, only where the profile shows a cost: one server shard behind a QUIC reverse proxy. Clients connect to the proxy, and the proxy connects to the server over loopback with an 8,192-byte UDP payload. Clients are moq-bench fan-out viewers: audio-rate, then 3 Mbit/s - video, 5 min each. The four runs ran in ABBA order (off, on, on, off). Server - CPU is in cores, as mean / max. + video, 5 min each. The four runs ran in ABBA order (off, on, on, off), one + run per cell, so no load's comparison is counterbalanced and there's no + variance estimate. Server CPU is in cores, as mean / max. | Load (audio / video viewers) | Batching | Audio p50 / p99, lost groups | Audio CPU | Video p99 | |---|---|---|---|---| @@ -102,16 +103,19 @@ Then, only where the profile shows a cost: - **Below the server's ceiling, batching saved no CPU and raised p99.** The loopback harness's 8–22% saving did not carry over, and why hasn't been - measured. - - **Only past the ceiling (400 audio) did it cut CPU, by 13%,** and the losses - with it. But the shard was overloaded either way, and video's p99 rose from - 31 to 79 ms. + measured. Both arms lost groups at this point, batching more (9,781 against + 6,090), so the p99 rise is measured under unequal loss, not drop-free. + - **Only at 400 audio did it cut CPU, by 13%,** and the lost groups with it. + The shard was overloaded either way, though no control at 400 separates the + server's ceiling from proxy-side loss. Audio p99 fell from 212 to 110 ms, + while video p99 rose from 31 to 79 ms. - **The losses at 300 came from the proxy's client-facing socket buffer**, not the server. With a 1 MiB receive buffer on the proxy, that point loses nothing. - So on this path the latency cost shows up and the CPU saving doesn't. That - argues for keeping batching opt-in and off by default, if it ships. + So on this path, below the ceiling p99 rose with no CPU saving; past it, CPU + and audio losses fell but video p99 rose. That leans toward keeping batching + opt-in and off by default, if it ships, on single runs. Its shape: - `UdpSocketState::try_send_many` on Linux sends up to 64 transmits, one From a2b0fcab671ed9a08bd3a7264c90dfc6b93497fc Mon Sep 17 00:00:00 2001 From: Zac Shenker Date: Sat, 7 Nov 2026 17:27:38 -0700 Subject: [PATCH 3/5] quest: record the end-to-end runs' build, proxy topology, and drops Co-Authored-By: Claude Opus 5.5 --- quest/m5/quic-egress-profile.md | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/quest/m5/quic-egress-profile.md b/quest/m5/quic-egress-profile.md index 566e58b7652..b85c7ceb402 100644 --- a/quest/m5/quic-egress-profile.md +++ b/quest/m5/quic-egress-profile.md @@ -91,8 +91,18 @@ Then, only where the profile shows a cost: and the proxy connects to the server over loopback with an 8,192-byte UDP payload. Clients are moq-bench fan-out viewers: audio-rate, then 3 Mbit/s video, 5 min each. The four runs ran in ABBA order (off, on, on, off), one - run per cell, so no load's comparison is counterbalanced and there's no - variance estimate. Server CPU is in cores, as mean / max. + run per cell, so no load's comparison is counterbalanced. A near-repeat of + off-300 put CPU within about 1% and audio p99 about 8 ms apart. Server CPU is + in cores, as mean / max. + - Build: the server ran the prototype in a private downstream build, not + public, with batching switched on per run through an opt-in on its endpoint + (`Endpoint::batch_sends` in the linked port). The clients were moq-bench at + 337f794 plus one local patch, with each run's first 30 s skipped. + - Proxy: it terminates QUIC and MoQ, and opens one upstream session per client + session to the server, each on its own UDP socket (602 upstream sessions for + 600 viewers at 400/200). So the server's egress is still per-viewer fan-out, + the cost this lever targets. Only the server batched; the proxy was + unchanged. | Load (audio / video viewers) | Batching | Audio p50 / p99, lost groups | Audio CPU | Video p99 | |---|---|---|---|---| @@ -105,16 +115,20 @@ Then, only where the profile shows a cost: loopback harness's 8–22% saving did not carry over, and why hasn't been measured. Both arms lost groups at this point, batching more (9,781 against 6,090), so the p99 rise is measured under unequal loss, not drop-free. - - **Only at 400 audio did it cut CPU, by 13%,** and the lost groups with it. - The shard was overloaded either way, though no control at 400 separates the - server's ceiling from proxy-side loss. Audio p99 fell from 212 to 110 ms, - while video p99 rose from 31 to 79 ms. + - **Only at 400 audio did it cut CPU, by 13%.** Lost groups fell too (150,424 + against 325,694), but the proxy's listener dropped about 1.1 M datagrams in + both runs, so that difference can't be credited to batching. The server was + past its limit: with batching off its own socket dropped 3,100 datagrams, + against none with it on, and CPU peaked at 0.78. Audio p99 fell from 212 to + 110 ms, while video p99 rose from 31 to 79 ms. - **The losses at 300 came from the proxy's client-facing socket buffer**, not - the server. With a 1 MiB receive buffer on the proxy, that point loses - nothing. + the server: it dropped 242,019 datagrams with batching off and 232,662 with + it on, at its 208 KiB default. With a 1 MiB receive buffer, batching off + loses nothing there (audio p50 / p99 11 / 29 ms, CPU 0.499 / 0.609, video + p99 22 ms); batching on was not re-run with it. So on this path, below the ceiling p99 rose with no CPU saving; past it, CPU - and audio losses fell but video p99 rose. That leans toward keeping batching + fell and audio p99 halved but video p99 rose. That leans toward keeping batching opt-in and off by default, if it ships, on single runs. Its shape: From 58210cded4f86cd80986c6c6a8c5005902bf3418 Mon Sep 17 00:00:00 2001 From: Zac Shenker Date: Sat, 7 Nov 2026 17:39:06 -0700 Subject: [PATCH 4/5] quest: scope the end-to-end loss attributions to what the runs show Co-Authored-By: Claude Opus 5.5 --- quest/m5/quic-egress-profile.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/quest/m5/quic-egress-profile.md b/quest/m5/quic-egress-profile.md index b85c7ceb402..3dbef559379 100644 --- a/quest/m5/quic-egress-profile.md +++ b/quest/m5/quic-egress-profile.md @@ -117,15 +117,18 @@ Then, only where the profile shows a cost: 6,090), so the p99 rise is measured under unequal loss, not drop-free. - **Only at 400 audio did it cut CPU, by 13%.** Lost groups fell too (150,424 against 325,694), but the proxy's listener dropped about 1.1 M datagrams in - both runs, so that difference can't be credited to batching. The server was - past its limit: with batching off its own socket dropped 3,100 datagrams, - against none with it on, and CPU peaked at 0.78. Audio p99 fell from 212 to - 110 ms, while video p99 rose from 31 to 79 ms. - - **The losses at 300 came from the proxy's client-facing socket buffer**, not - the server: it dropped 242,019 datagrams with batching off and 232,662 with - it on, at its 208 KiB default. With a 1 MiB receive buffer, batching off - loses nothing there (audio p50 / p99 11 / 29 ms, CPU 0.499 / 0.609, video - p99 22 ms); batching on was not re-run with it. + both runs, so that difference can't be credited to batching. The server's own + socket also dropped 3,100 datagrams with batching off, against none with it + on, and its CPU peaked at 0.78 cores; both runs were measured under the + proxy's overflow too, so neither number isolates the server's ceiling. Audio + p99 fell from 212 to 110 ms, while video p99 rose from 31 to 79 ms. + - **At 300, both arms lost datagrams at the proxy's client-facing socket + buffer**, at its 208 KiB default: 242,019 with batching off and 232,662 with + it on. The server's socket dropped none. With a 1 MiB proxy buffer, batching + off loses nothing there (audio p50 / p99 11 / 29 ms, CPU 0.499 / 0.609, video + p99 22 ms), so its loss was the proxy's. Batching on was not re-run with it, + and what accounts for its extra lost groups on fewer proxy drops hasn't been + measured. So on this path, below the ceiling p99 rose with no CPU saving; past it, CPU fell and audio p99 halved but video p99 rose. That leans toward keeping batching From ccf8dedc5341983697ce88e61ea9540066183ebc Mon Sep 17 00:00:00 2001 From: Zac Shenker Date: Sat, 7 Nov 2026 20:07:55 -0700 Subject: [PATCH 5/5] quest: name the end-to-end loads instead of an unisolated ceiling Co-Authored-By: Claude Opus 5.5 --- quest/m5/quic-egress-profile.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/quest/m5/quic-egress-profile.md b/quest/m5/quic-egress-profile.md index 3dbef559379..e6f4abe1ec8 100644 --- a/quest/m5/quic-egress-profile.md +++ b/quest/m5/quic-egress-profile.md @@ -111,7 +111,7 @@ Then, only where the profile shows a cost: | 400 / 200 | on | 27 / 110 ms, 150,424 | 0.596 / 0.695 | 79 ms | | 400 / 200 | off | 27 / 212 ms, 325,694 | 0.688 / 0.779 | 31 ms | - - **Below the server's ceiling, batching saved no CPU and raised p99.** The + - **At 300, batching saved no CPU and raised p99.** The loopback harness's 8–22% saving did not carry over, and why hasn't been measured. Both arms lost groups at this point, batching more (9,781 against 6,090), so the p99 rise is measured under unequal loss, not drop-free. @@ -130,8 +130,8 @@ Then, only where the profile shows a cost: and what accounts for its extra lost groups on fewer proxy drops hasn't been measured. - So on this path, below the ceiling p99 rose with no CPU saving; past it, CPU - fell and audio p99 halved but video p99 rose. That leans toward keeping batching + So on this path, at 300 p99 rose with no CPU saving; at 400 CPU fell and + audio p99 halved but video p99 rose. That leans toward keeping batching opt-in and off by default, if it ships, on single runs. Its shape: