Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 71 additions & 1 deletion quest/m2/quic-egress-profile.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,77 @@ Then, only where the profile shows a cost:

- `sendmmsg` in `moq_sock::udp` behind the existing GSO path, submitting every
train ready across connections in one call, falling back to per-train
`sendmsg` on partial failure the way the GSO fallback does.
`sendmsg` on partial failure the way the GSO fallback does. A contributor's
prototype on the tokio driver gives the profile a starting point; none of
its code is in-tree, and its numbers below are the prototype's, not the
profile's. It was measured before being
[ported onto main in a fork](https://github.com/zshenker/moq/commit/040df9948bd110a83463f35cd4bd86245db03373)
(opt-in through `Endpoint::batch_sends`), and comes with its
[raw samples](https://github.com/zshenker/moq/blob/eae2ca6941ce67b8e14ba6afdb76c1c2240c45a0/bench/driver-batch-sends/samples.jsonl),
so the profile should re-measure the port.
- Revision: measured on e072cf604 with #7458, #7418, #7394, #7543, and a
GSO byte cap equivalent to #7649 applied. The prototype revisions measured
predate the handshake bypass, and the last round ran after the drop and
rebind fixes. The steady-state send path matches the port, which differs
in its opt-in (`Endpoint::batch_sends`), its flush task's spawner, the
handshake bypass, and its base. Those revisions and the harness are not
public.
- Invocation: one run per sample of the harness's
`bench --sessions 1000 --rate 30 --object-size 1024-4096 --objects-per-stream 1 --client-threads 6 --client-endpoints 12 --pin --batch-sends {false|true}`,
with audio at `--rate 50 --object-size 100-200`, 1 Mbit/s video at
`--object-size 3125-5208 --objects-per-stream 30`, and 3 Mbit/s video at
`--sessions 500 --object-size 9375-15625 --objects-per-stream 30`. Its
defaults are a 1,280-byte payload, GSO on, one tokio worker, a 5 s warmup,
a 15 s measure, 128 handshakes in flight, and a 15 s keep-alive, all under
`taskset` on physical cores.
- Setup: server and client on one host over loopback, the server on one
tokio worker pinned to one physical core, and the client on 6 pinned workers
over 12 sockets. 1,280-byte payloads, MTU discovery off, GSO and GRO on,
BBR3. Only the server batches. Each sample is one 15 s run after a 5 s
warmup, on a dual-socket Broadwell Xeon with Linux 6.12. On and off ran back
to back on the same binary, in alternating order across runs.
- Server CPU (`getrusage`, % of one core):
- small objects (1,024 to 4,096 B at 30/s, 1,000 sessions): 61.7% off
against 54.9% on, n=3 each, so 8 to 15% lower;
- audio-rate (100 to 200 B at 50/s, 1,000 sessions): 66.9% against 53.9%,
n=3 each, so 16 to 22% lower;
- 1 and 3 Mbit/s video: 10% and 8% lower, n=1 each.

The system share fell from 31 to 38% to 22 to 34%. A single run varies by
about ±3%.
- Calls: in separate 5 s `strace` windows of the small-object point, the
unbatched server made 135,497 `sendmsg` and the batched one 7,214
`sendmmsg` plus 506 `sendmsg`, so about 18 to 19 trains per `sendmmsg`.
That is an estimate across runs with no direct counter, and `strace` slows
each arm differently.
- Latency (per object, from the server's write to the client's read of the
last byte): p50 rose from 36 to 72 µs to 258 to 352 µs. On the small-object
and audio points, p99 rose from 104 to 705 µs to 606 to 770 µs.

Its shape:
- `UdpSocketState::try_send_many` on Linux sends up to 64 transmits, one
`mmsghdr` each, in one `sendmmsg`, each with its own GSO segment size,
ECN, and source-address cmsgs. It returns how many the kernel took, and
retries an `EIO` or `EINVAL` on the first transmit alone through
`try_send`, so GSO halting still applies.
- The driver copies each train into a per-socket queue, bounded in bytes,
and one task per socket yields once and then flushes the queue.
- Handshake packets try the socket directly, since `handle_unreachable`
acts only during a handshake and a queued send's error never reaches it;
without that, a dial to an unreachable address would wait out the
handshake timeout instead of failing with `NO_VIABLE_PATH`. The bypass is
not unconditional: on `WouldBlock` the port queues the handshake packet,
and that packet's send error is lost the same way. A shipped version
should keep backpressure on handshake sends instead of queueing them.
- A rebind wraps the new socket the same way. Dropping the queue sends what
the socket has room for and closes it synchronously, freeing the port. The
flush is best-effort: it stops at `WouldBlock`, and a batch the flush task
already took is dropped once the socket closes, so a CONNECTION_CLOSE can
be lost. A shipped version should drain that batch too.

Open: whether batching is opt-in in the driver's config or on by default,
given the added p50 and p99 latency, and whether moq-uring, which already
submits every train in one ring enter, needs any of it.
- A `BufFactory`-style seam on `SendBuffer` in `moq-quic` that takes buffers
from a pool sized from the send window and returns them on ACK. Design it
with [the I/O boundary redesign](/quest/m2/quic-io-boundary.md), which
Expand Down
Loading