From f7ae776ef663c663ee15cb934856f58947fe19c0 Mon Sep 17 00:00:00 2001 From: Zac Shenker Date: Mon, 2 Nov 2026 21:35:13 -0700 Subject: [PATCH 1/5] quest: record a sendmmsg prototype's measurements in the egress profile Co-Authored-By: Claude Opus 5.5 --- quest/m2/quic-egress-profile.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/quest/m2/quic-egress-profile.md b/quest/m2/quic-egress-profile.md index 1d35c388071..1df6f55a283 100644 --- a/quest/m2/quic-egress-profile.md +++ b/quest/m2/quic-egress-profile.md @@ -39,7 +39,28 @@ Then, only where the profile shows a cost: - `sendmmsg` in `moq_sock::udp` behind the existing GSO path, submitting every train ready across connections in one call, falling back to per-train - `sendmsg` on partial failure the way the GSO fallback does. + `sendmsg` on partial failure the way the GSO fallback does. A prototype on + the tokio driver gives the profile a starting point. At a 1280-byte payload + on one core, about 17 trains went per `sendmmsg`, server CPU fell 8 to 13% + for small-object sessions and 16 to 21% for audio-rate ones, and median + latency rose 0.3 ms. Its shape: + - `UdpSocketState::try_send_many` on Linux sends up to 64 transmits in one + call, each with its own segment size, ECN, and source-address cmsgs. It + returns how many the kernel took, and retries an `EIO` or `EINVAL` on the + first transmit alone through `try_send`, so GSO halting still applies. + - The driver copies each train into a per-socket queue, bounded in bytes, + and one task per socket yields once and then flushes the queue. + - Handshake packets bypass the queue. `handle_unreachable` acts only during + a handshake, and a queued send's error never reaches it, so a dial to an + unreachable address would wait out the handshake timeout instead of + failing with `NO_VIABLE_PATH`. + - A rebind wraps the new socket the same way. Dropping the queue flushes it + and closes the socket synchronously, so `close` still sends its + CONNECTION_CLOSE frames and frees the port. + + Open: whether batching is opt-in in the driver's config or on by default, + given the latency cost, and whether moq-uring, which already submits every + train in one ring enter, needs any of it. - A `BufFactory`-style seam on `SendBuffer` in `moq-quic` that takes buffers from a pool sized from the send window and returns them on ACK. Design it with [the I/O boundary redesign](/quest/m2/quic-io-boundary.md), which From fb664afa7c00f97beacd457840f01776d6f39c25 Mon Sep 17 00:00:00 2001 From: Zac Shenker Date: Mon, 2 Nov 2026 22:03:44 -0700 Subject: [PATCH 2/5] quest: give the sendmmsg prototype's numbers their setup, samples, and code Co-Authored-By: Claude Opus 5.5 --- quest/m2/quic-egress-profile.md | 45 +++++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 10 deletions(-) diff --git a/quest/m2/quic-egress-profile.md b/quest/m2/quic-egress-profile.md index 1df6f55a283..f6120aba7bc 100644 --- a/quest/m2/quic-egress-profile.md +++ b/quest/m2/quic-egress-profile.md @@ -40,14 +40,39 @@ Then, only where the profile shows a cost: - `sendmmsg` in `moq_sock::udp` behind the existing GSO path, submitting every train ready across connections in one call, falling back to per-train `sendmsg` on partial failure the way the GSO fallback does. A prototype on - the tokio driver gives the profile a starting point. At a 1280-byte payload - on one core, about 17 trains went per `sendmmsg`, server CPU fell 8 to 13% - for small-object sessions and 16 to 21% for audio-rate ones, and median - latency rose 0.3 ms. Its shape: - - `UdpSocketState::try_send_many` on Linux sends up to 64 transmits in one - call, each with its own segment size, ECN, and source-address cmsgs. It - returns how many the kernel took, and retries an `EIO` or `EINVAL` on the - first transmit alone through `try_send`, so GSO halting still applies. + the tokio driver gives the profile a starting point. It was measured before + being [ported onto main](https://github.com/zshenker/moq/commit/040df9948bd110a83463f35cd4bd86245db03373) + (opt-in through `Endpoint::batch_sends`), and comes with its + [raw samples](https://github.com/zshenker/moq/blob/eae2ca6941ce67b8e14ba6afdb76c1c2240c45a0/bench/driver-batch-sends/samples.jsonl), + so the profile should re-measure the port. + - Setup: server and client on one host over loopback, the server on one + tokio worker pinned to one physical core, and the client on 6 pinned workers + over 12 sockets. 1,280-byte payloads, MTU discovery off, GSO and GRO on, + BBR3. Only the server batches. Each point is one 15 s run after a 5 s + warmup, on a dual-socket Broadwell Xeon with Linux 6.12. On and off ran back + to back on the same binary, in alternating order across runs. + - Server CPU (`getrusage`, % of one core): + - small objects (1,024 to 4,096 B at 30/s, 1,000 sessions): 61.7% off + against 54.9% on, n=3 each, so 8 to 15% lower; + - audio-rate (100 to 200 B at 50/s, 1,000 sessions): 66.9% against 53.9%, + n=3 each, so 16 to 22% lower; + - 1 and 3 Mbit/s video: 10% and 8% lower, n=1 each. + + The system share fell from 31 to 38% to 22 to 34%. A single run varies by + about ±3%. + - Calls: over 5 s of the small-object point, `strace` counted 7,214 + `sendmmsg` against 135,497 `sendmsg`, about 17 trains per call. `strace` + slows the server, so only the ratio holds. + - Latency (per object, from the server's write to the client's read of the + last byte): p50 rose from 36 to 72 µs to 258 to 352 µs. On the small-object + and audio points, p99 rose from 104 to 705 µs to 606 to 770 µs. + + Its shape: + - `UdpSocketState::try_send_many` on Linux sends up to 64 transmits, one + `mmsghdr` each, in one `sendmmsg`, each with its own GSO segment size, + ECN, and source-address cmsgs. It returns how many the kernel took, and + retries an `EIO` or `EINVAL` on the first transmit alone through + `try_send`, so GSO halting still applies. - The driver copies each train into a per-socket queue, bounded in bytes, and one task per socket yields once and then flushes the queue. - Handshake packets bypass the queue. `handle_unreachable` acts only during @@ -59,8 +84,8 @@ Then, only where the profile shows a cost: CONNECTION_CLOSE frames and frees the port. Open: whether batching is opt-in in the driver's config or on by default, - given the latency cost, and whether moq-uring, which already submits every - train in one ring enter, needs any of it. + given the added p50 and p99 latency, and whether moq-uring, which already + submits every train in one ring enter, needs any of it. - A `BufFactory`-style seam on `SendBuffer` in `moq-quic` that takes buffers from a pool sized from the send window and returns them on ACK. Design it with [the I/O boundary redesign](/quest/m2/quic-io-boundary.md), which From 776d6941d16bdbec5363e4084eaa8527796cdefa Mon Sep 17 00:00:00 2001 From: Zac Shenker Date: Tue, 3 Nov 2026 08:08:42 -0700 Subject: [PATCH 3/5] quest: record the sendmmsg prototype's measured revision and invocation Co-Authored-By: Claude Opus 5.5 --- quest/m2/quic-egress-profile.md | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/quest/m2/quic-egress-profile.md b/quest/m2/quic-egress-profile.md index f6120aba7bc..6b082a5c37c 100644 --- a/quest/m2/quic-egress-profile.md +++ b/quest/m2/quic-egress-profile.md @@ -45,6 +45,21 @@ Then, only where the profile shows a cost: (opt-in through `Endpoint::batch_sends`), and comes with its [raw samples](https://github.com/zshenker/moq/blob/eae2ca6941ce67b8e14ba6afdb76c1c2240c45a0/bench/driver-batch-sends/samples.jsonl), so the profile should re-measure the port. + - Revision: measured on e072cf604 with #7458, #7418, #7394, #7543, and a + GSO byte cap equivalent to #7649 applied. The prototype revisions measured + predate the handshake bypass, and the last round ran after the drop and + rebind fixes. The steady-state send path matches the port, which differs + in its opt-in (`Endpoint::batch_sends`), its flush task's spawner, the + handshake bypass, and its base. Those revisions and the harness are not + public. + - Invocation: one run per point of the harness's + `bench --sessions 1000 --rate 30 --object-size 1024-4096 --objects-per-stream 1 --client-threads 6 --client-endpoints 12 --pin --batch-sends {false|true}`, + with audio at `--rate 50 --object-size 100-200`, 1 Mbit/s video at + `--object-size 3125-5208 --objects-per-stream 30`, and 3 Mbit/s video at + `--sessions 500 --object-size 9375-15625 --objects-per-stream 30`. Its + defaults are a 1,280-byte payload, GSO on, one tokio worker, a 5 s warmup, + a 15 s measure, 128 handshakes in flight, and a 15 s keep-alive, all under + `taskset` on physical cores. - Setup: server and client on one host over loopback, the server on one tokio worker pinned to one physical core, and the client on 6 pinned workers over 12 sockets. 1,280-byte payloads, MTU discovery off, GSO and GRO on, @@ -60,9 +75,11 @@ Then, only where the profile shows a cost: The system share fell from 31 to 38% to 22 to 34%. A single run varies by about ±3%. - - Calls: over 5 s of the small-object point, `strace` counted 7,214 - `sendmmsg` against 135,497 `sendmsg`, about 17 trains per call. `strace` - slows the server, so only the ratio holds. + - Calls: in separate 5 s `strace` windows of the small-object point, the + unbatched server made 135,497 `sendmsg` and the batched one 7,214 + `sendmmsg` plus 506 `sendmsg`, so about 18 to 19 trains per `sendmmsg`. + That is an estimate across runs with no direct counter, and `strace` slows + each arm differently. - Latency (per object, from the server's write to the client's read of the last byte): p50 rose from 36 to 72 µs to 258 to 352 µs. On the small-object and audio points, p99 rose from 104 to 705 µs to 606 to 770 µs. From 453399fa39cbb5bd01be106348eafa94db91061b Mon Sep 17 00:00:00 2001 From: Zac Shenker Date: Tue, 3 Nov 2026 09:31:47 -0700 Subject: [PATCH 4/5] quest: describe the sendmmsg prototype's drop flush as best-effort Co-Authored-By: Claude Opus 5.5 --- quest/m2/quic-egress-profile.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/quest/m2/quic-egress-profile.md b/quest/m2/quic-egress-profile.md index 6b082a5c37c..c2ac16a9de2 100644 --- a/quest/m2/quic-egress-profile.md +++ b/quest/m2/quic-egress-profile.md @@ -96,9 +96,11 @@ Then, only where the profile shows a cost: a handshake, and a queued send's error never reaches it, so a dial to an unreachable address would wait out the handshake timeout instead of failing with `NO_VIABLE_PATH`. - - A rebind wraps the new socket the same way. Dropping the queue flushes it - and closes the socket synchronously, so `close` still sends its - CONNECTION_CLOSE frames and frees the port. + - A rebind wraps the new socket the same way. Dropping the queue sends what + the socket has room for and closes it synchronously, freeing the port. The + flush is best-effort: it stops at `WouldBlock`, and a batch the flush task + already took is dropped once the socket closes, so a CONNECTION_CLOSE can + be lost. A shipped version should drain that batch too. Open: whether batching is opt-in in the driver's config or on by default, given the added p50 and p99 latency, and whether moq-uring, which already From e88c4f4175f007d3ebc9d351d7ac90d668a74d7a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 15:52:13 -0800 Subject: [PATCH 5/5] quest: qualify the sendmmsg prototype's handshake bypass and label it out of tree The port queues a handshake packet on WouldBlock, so its send error never reaches handle_unreachable; say so and that a shipped version should keep backpressure instead. State that none of the prototype's code is in-tree and that the numbers are the prototype's, and call each run a sample so it reads consistently with n=3. Co-Authored-By: Claude Opus 5.5 --- quest/m2/quic-egress-profile.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/quest/m2/quic-egress-profile.md b/quest/m2/quic-egress-profile.md index c2ac16a9de2..b16bda86a55 100644 --- a/quest/m2/quic-egress-profile.md +++ b/quest/m2/quic-egress-profile.md @@ -39,9 +39,11 @@ Then, only where the profile shows a cost: - `sendmmsg` in `moq_sock::udp` behind the existing GSO path, submitting every train ready across connections in one call, falling back to per-train - `sendmsg` on partial failure the way the GSO fallback does. A prototype on - the tokio driver gives the profile a starting point. It was measured before - being [ported onto main](https://github.com/zshenker/moq/commit/040df9948bd110a83463f35cd4bd86245db03373) + `sendmsg` on partial failure the way the GSO fallback does. A contributor's + prototype on the tokio driver gives the profile a starting point; none of + its code is in-tree, and its numbers below are the prototype's, not the + profile's. It was measured before being + [ported onto main in a fork](https://github.com/zshenker/moq/commit/040df9948bd110a83463f35cd4bd86245db03373) (opt-in through `Endpoint::batch_sends`), and comes with its [raw samples](https://github.com/zshenker/moq/blob/eae2ca6941ce67b8e14ba6afdb76c1c2240c45a0/bench/driver-batch-sends/samples.jsonl), so the profile should re-measure the port. @@ -52,7 +54,7 @@ Then, only where the profile shows a cost: in its opt-in (`Endpoint::batch_sends`), its flush task's spawner, the handshake bypass, and its base. Those revisions and the harness are not public. - - Invocation: one run per point of the harness's + - Invocation: one run per sample of the harness's `bench --sessions 1000 --rate 30 --object-size 1024-4096 --objects-per-stream 1 --client-threads 6 --client-endpoints 12 --pin --batch-sends {false|true}`, with audio at `--rate 50 --object-size 100-200`, 1 Mbit/s video at `--object-size 3125-5208 --objects-per-stream 30`, and 3 Mbit/s video at @@ -63,7 +65,7 @@ Then, only where the profile shows a cost: - Setup: server and client on one host over loopback, the server on one tokio worker pinned to one physical core, and the client on 6 pinned workers over 12 sockets. 1,280-byte payloads, MTU discovery off, GSO and GRO on, - BBR3. Only the server batches. Each point is one 15 s run after a 5 s + BBR3. Only the server batches. Each sample is one 15 s run after a 5 s warmup, on a dual-socket Broadwell Xeon with Linux 6.12. On and off ran back to back on the same binary, in alternating order across runs. - Server CPU (`getrusage`, % of one core): @@ -92,10 +94,13 @@ Then, only where the profile shows a cost: `try_send`, so GSO halting still applies. - The driver copies each train into a per-socket queue, bounded in bytes, and one task per socket yields once and then flushes the queue. - - Handshake packets bypass the queue. `handle_unreachable` acts only during - a handshake, and a queued send's error never reaches it, so a dial to an - unreachable address would wait out the handshake timeout instead of - failing with `NO_VIABLE_PATH`. + - Handshake packets try the socket directly, since `handle_unreachable` + acts only during a handshake and a queued send's error never reaches it; + without that, a dial to an unreachable address would wait out the + handshake timeout instead of failing with `NO_VIABLE_PATH`. The bypass is + not unconditional: on `WouldBlock` the port queues the handshake packet, + and that packet's send error is lost the same way. A shipped version + should keep backpressure on handshake sends instead of queueing them. - A rebind wraps the new socket the same way. Dropping the queue sends what the socket has room for and closes it synchronously, freeing the port. The flush is best-effort: it stops at `WouldBlock`, and a batch the flush task