Repository navigation
quest: add the sendmmsg prototype's end-to-end measurements #8184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
7983a0b
214ea40
c2d6c2a
a2b0fca
58210cd
ccf8ded
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -86,6 +86,54 @@ Then, only where the profile shows a cost: | |||||||||||
| last byte): p50 rose from 36 to 72 µs to 258 to 352 µs. On the small-object | ||||||||||||
| and audio points, p99 rose from 104 to 705 µs to 606 to 770 µs. | ||||||||||||
|
|
||||||||||||
| **End to end, through a QUIC reverse proxy.** The same prototype, measured as | ||||||||||||
| one server shard behind a QUIC reverse proxy. Clients connect to the proxy, | ||||||||||||
| and the proxy connects to the server over loopback with an 8,192-byte UDP | ||||||||||||
| payload. Clients are moq-bench fan-out viewers: audio-rate, then 3 Mbit/s | ||||||||||||
| video, 5 min each. The four runs ran in ABBA order (off, on, on, off), one | ||||||||||||
| run per cell, so no load's comparison is counterbalanced. A near-repeat of | ||||||||||||
| off-300 put CPU within about 1% and audio p99 about 8 ms apart. Server CPU is | ||||||||||||
| in cores, as mean / max. | ||||||||||||
| - Build: the server ran the prototype in a private downstream build, not | ||||||||||||
| public, with batching switched on per run through an opt-in on its endpoint | ||||||||||||
| (`Endpoint::batch_sends` in the linked port). The clients were moq-bench at | ||||||||||||
| 337f794 plus one local patch, with each run's first 30 s skipped. | ||||||||||||
| - Proxy: it terminates QUIC and MoQ, and opens one upstream session per client | ||||||||||||
| session to the server, each on its own UDP socket (602 upstream sessions for | ||||||||||||
| 600 viewers at 400/200). So the server's egress is still per-viewer fan-out, | ||||||||||||
| the cost this lever targets. Only the server batched; the proxy was | ||||||||||||
| unchanged. | ||||||||||||
|
|
||||||||||||
| | Load (audio / video viewers) | Batching | Audio p50 / p99, lost groups | Audio CPU | Video p99 | | ||||||||||||
| |---|---|---|---|---| | ||||||||||||
| | 300 / 150 | off | 11 / 38 ms, 6,090 | 0.492 / 0.596 | 21 ms | | ||||||||||||
| | 300 / 150 | on | 13 / 54 ms, 9,781 | 0.490 / 0.560 | 26 ms | | ||||||||||||
| | 400 / 200 | on | 27 / 110 ms, 150,424 | 0.596 / 0.695 | 79 ms | | ||||||||||||
| | 400 / 200 | off | 27 / 212 ms, 325,694 | 0.688 / 0.779 | 31 ms | | ||||||||||||
|
|
||||||||||||
| - **At 300, batching saved no CPU and raised p99.** The | ||||||||||||
| loopback harness's 8–22% saving did not carry over, and why hasn't been | ||||||||||||
| measured. Both arms lost groups at this point, batching more (9,781 against | ||||||||||||
| 6,090), so the p99 rise is measured under unequal loss, not drop-free. | ||||||||||||
| - **Only at 400 audio did it cut CPU, by 13%.** Lost groups fell too (150,424 | ||||||||||||
| against 325,694), but the proxy's listener dropped about 1.1 M datagrams in | ||||||||||||
| both runs, so that difference can't be credited to batching. The server's own | ||||||||||||
| socket also dropped 3,100 datagrams with batching off, against none with it | ||||||||||||
| on, and its CPU peaked at 0.78 cores; both runs were measured under the | ||||||||||||
| proxy's overflow too, so neither number isolates the server's ceiling. Audio | ||||||||||||
| p99 fell from 212 to 110 ms, while video p99 rose from 31 to 79 ms. | ||||||||||||
|
Comment on lines
+123
to
+124
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. At 400 this bullet says the lost-group gap (325,694 off against 150,424 on) can't be credited to batching. It then reports audio p99 falling 212→110 ms with no qualifier, even though p50 is 27/27 and that p99 was measured under the same unequal loss. The 300 bullet already qualifies its p99 rise as "measured under unequal loss". The verdict at L133-134 repeats "audio p99 halved" as one of 400's results, so a reader takes it as a batching benefit that the runs don't isolate.
Suggested change
|
||||||||||||
| - **At 300, both arms lost datagrams at the proxy's client-facing socket | ||||||||||||
| buffer**, at its 208 KiB default: 242,019 with batching off and 232,662 with | ||||||||||||
| it on. The server's socket dropped none. With a 1 MiB proxy buffer, batching | ||||||||||||
| off loses nothing there (audio p50 / p99 11 / 29 ms, CPU 0.499 / 0.609, video | ||||||||||||
| p99 22 ms), so its loss was the proxy's. Batching on was not re-run with it, | ||||||||||||
| and what accounts for its extra lost groups on fewer proxy drops hasn't been | ||||||||||||
| measured. | ||||||||||||
|
|
||||||||||||
| So on this path, at 300 p99 rose with no CPU saving; at 400 CPU fell and | ||||||||||||
| audio p99 halved but video p99 rose. That leans toward keeping batching | ||||||||||||
| opt-in and off by default, if it ships, on single runs. | ||||||||||||
|
|
||||||||||||
| Its shape: | ||||||||||||
| - `UdpSocketState::try_send_many` on Linux sends up to 64 transmits, one | ||||||||||||
| `mmsghdr` each, in one `sendmmsg`, each with its own GSO segment size, | ||||||||||||
|
|
||||||||||||
Uh oh!
There was an error while loading. Please reload this page.