Repository navigation
quest(m2): plan wire stats, per-root transport bytes in Presence #5422
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
02de2a7
cd26d08
715fe10
182bf72
9a347c5
f8b0dee
95d00e2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| # Wire stats | ||
|
|
||
| ## Goal | ||
|
|
||
| The relay's session stats carry `wire_tx` and `wire_rx` in `Presence`: the | ||
| bytes each root's connections put on and took off the wire, on each tier. | ||
| They appear in the `totals.json` track per tier and in the per-root | ||
| `presence.json` track ([Stats](/doc/concept/stats.md#tracks)). An operator | ||
| reads what each tenant root costs in transport bytes, not only the payload | ||
| `Traffic.bytes` counts. | ||
|
|
||
| Wire bytes are the transport's payload, with IP, UDP, and TCP headers | ||
| excluded: | ||
|
|
||
| - **QUIC and WebTransport:** `udp_tx` / `udp_rx`, which include handshake, | ||
| ACKs, and retransmits. | ||
| - **WebSocket:** the TCP stream's bytes, which include TLS, WebSocket, and | ||
| qmux framing and retransmits. | ||
| - **Gateways an embedder terminates:** SRT and WebRTC through moq-srt's and | ||
| moq-rtc's sources. An embedder terminating RTMP on its own socket | ||
| registers that socket's `TCP_INFO` counts; moq-rtmp's own `Server` and | ||
| client expose no source until a consumer needs one. | ||
| - **TCP receive:** bytes TCP delivered, so a peer's duplicate retransmits | ||
| are not counted. | ||
|
|
||
| Non-goals: | ||
|
|
||
| - **Splitting a shared connection's bytes across roots.** A cluster | ||
| connection's wire bytes count under its own root, which is empty. | ||
| - **Per-broadcast wire bytes.** Wire bytes belong to connections, not tracks. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-10-13. Payload undercounts what delivery costs, and by how much | ||
| depends on the workload. Measured wire/payload in moq.pro's bench: | ||
|
|
||
| | Workload | Wire/payload | | ||
| |---|---| | ||
| | Video | 1.04x | | ||
| | Audio | 1.23x | | ||
| | Small chat-sized messages | 1.29-1.56x | | ||
| | Announce-only | 14.4x | | ||
|
|
||
| A provider bills the wire, so a relay operator needs per-root wire bytes to | ||
| see cost per tenant. | ||
|
|
||
| - **`Presence`, not `Traffic`.** `Presence` is already per (tier, root) and | ||
| per session. `Traffic.bytes` stays payload, and its docs say so. | ||
| - **Named `wire_tx` / `wire_rx`.** Rejected: `bytes_sent`, which reads like | ||
| payload beside `Traffic.bytes`. | ||
| - **Sampled on the stats tick.** Rejected: a once-only count at session end, | ||
| which loses a crashed session and lags a long one. | ||
| - **One source registry for sessions and embedders.** An embedder registers | ||
| a gateway connection's wire source against a (tier, root) on its stats | ||
| handle, the same path the relay's own sessions use, like | ||
| [session outcomes](/quest/m1/session-outcomes.md)' embedder refusals. | ||
|
|
||
| ## Required | ||
|
|
||
| - [Presence wire counters](/quest/m2/wire-stats/presence.md) - QUIC, | ||
| WebTransport, and WebSocket sessions and embedder sources count per | ||
| (tier, root) | ||
| - [qmux byte counters](/quest/m2/wire-stats/qmux.md) - a WebSocket session | ||
| reports its TCP bytes | ||
| - [moq-srt wire source](/quest/m2/wire-stats/srt.md) - an SRT connection | ||
| reports its wire bytes | ||
| - [moq-rtc wire source](/quest/m2/wire-stats/rtc.md) - a WebRTC peer reports | ||
| its wire bytes | ||
|
|
||
| ## Related | ||
|
|
||
| - [Session outcomes](/quest/m1/session-outcomes.md) - the other per-root | ||
| `Presence` counters and the embedder stats-handle pattern | ||
| - [HTTP fetch stats](/quest/m2/http-fetch-stats.md) - HTTP `/fetch`, which | ||
| has no session and stays payload-only |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| # [M] Presence wire counters | ||
|
|
||
| ## Goal | ||
|
|
||
| `Presence` gains cumulative `wire_tx` and `wire_rx`, under the same reset | ||
| contract as its other cumulative fields: counted since the stats group's | ||
| epoch began (a group returning from idle starts a new epoch at zero), and | ||
| reset with a reclaimed and re-held presence track, so the aggregator never | ||
| re-adds bytes an earlier epoch already reported. On every stats tick the | ||
| relay adds the delta of each live session's transport bytes since its last | ||
| reading. A closing session adds its final reading. An embedder registers a connection's wire source against a | ||
| (tier, root) on its stats handle, and the same tick reads it. | ||
|
|
||
| ## Plan | ||
|
|
||
| - **Sources:** | ||
| - **QUIC and WebTransport:** `bytes_sent` / `bytes_received` from | ||
| `moq_net::session::Stats`. noq maps them to `udp_tx` / `udp_rx`. | ||
| - **WebSocket:** the qmux counters from | ||
| [qmux byte counters](/quest/m2/wire-stats/qmux.md). Until those land, a | ||
| WebSocket session reports 0: an undercount, never a guess. | ||
| - **Read the transport directly.** The tick must not re-arm the per-session | ||
| 100 ms sampler that moq#5382 stopped for idle sessions. Read the | ||
| connection's counters once per tick, cheaply. The tick's work must scale | ||
| with sources plus held roots, never their product: benchmark idle-relay | ||
| CPU sweeping both the source (session) count and the held per-root | ||
| presence track count, before and after. | ||
| - **Ticks, tier changes, and closes:** | ||
| - A session's per-connection counters reset with the connection, so keep | ||
| each session's last reading and add only non-negative deltas. | ||
| - `Session::set_tier` moves later deltas to the new tier. | ||
| - A close takes one final reading. A slightly stale final sample | ||
| under-counts, which is accepted. | ||
| - **Embedder API:** a source is a closure returning cumulative | ||
| `(tx, rx)`, registered with a guard. Dropping the guard takes its final | ||
| reading and unregisters it. | ||
| - **Serialization:** `#[serde(default)]` on both fields, so an older | ||
| consumer and an older relay interoperate. `Presence` stays `Copy`. The | ||
| stats-feed sum (`Presence::add`), the delta encoding, and the aggregator's | ||
| reset check (`Mergeable::regressed` in `rs/moq-stats/src/aggregate.rs`) | ||
| carry them, so a node whose wire totals drop is a reset. | ||
| - **Docs:** `doc/concept/stats.md` defines both fields, says | ||
| `Traffic.bytes` is payload, and says WebSocket wire bytes need a host | ||
| whose `TCP_INFO` reports them (Linux 4.19+), counting 0 elsewhere. | ||
| - **Tests (on controlled time):** | ||
| - a session's wire bytes land under its root and tier; | ||
| - a tier change splits them; | ||
| - a reconnecting source never lowers a counter; | ||
| - a stats group that idles out and returns starts its new epoch at zero; | ||
| - the aggregator treats a wire-only drop as a reset; | ||
| - a dropped embedder guard adds its last reading. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| # [S] qmux byte counters | ||
|
|
||
| ## Goal | ||
|
|
||
| A qmux session over TCP (WebSocket or raw TLS) reports cumulative bytes sent | ||
| and received in its `SessionStats`, so moq-net's `Stats.bytes_sent` / | ||
| `bytes_received` is nonzero on WebSocket. Sent bytes include TLS, WebSocket, | ||
| and qmux framing and TCP retransmits, and exclude TCP/IP headers. Received | ||
| bytes are what TCP delivered (`tcpi_bytes_received` is RFC 4898's | ||
| application-throughput counter), so a peer's duplicate retransmitted payload | ||
| is not counted: an accepted undercount on the receive side, documented | ||
| beside the field. | ||
|
|
||
| ## Plan | ||
|
|
||
| - moq-dev/web-transport: qmux's `TcpStats` already reads `TCP_INFO` for rtt | ||
| and delivery rate. Add `tcpi_bytes_sent` and `tcpi_bytes_received` from | ||
| the same read. A platform without them (non-Linux, or Linux before 4.19) | ||
| reports `None`, which moq-net maps to 0. | ||
|
Comment on lines
+18
to
+19
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
On every non-Linux host, and on older Linux kernels, this converts an unavailable counter into a real zero in AGENTS.md reference: AGENTS.md:L17-L17 Useful? React with 👍 / 👎.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Partly agreed. I won't refuse WebSocket or add an unavailable state: (Written by Claude Opus 5.5) |
||
| - Release qmux on every line moq builds against, then bump it here and map | ||
| the fields in moq-net's WebSocket `Stats` and in the relay's | ||
| `SocketStats` wiring (`rs/moq-relay/src/websocket.rs`). | ||
| - Test: a loopback WebSocket session's `bytes_sent` covers at least the | ||
| payload it wrote. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # [S] moq-rtc wire source | ||
|
|
||
| ## Goal | ||
|
|
||
| moq-rtc gives its embedder each WebRTC peer's cumulative wire bytes, both | ||
| directions, for WHIP and WHEP: every STUN, DTLS, SRTP, and SRTCP datagram's | ||
| UDP payload. These counts exclude the IP and UDP headers, the same | ||
| definition as QUIC's `udp_tx`. | ||
|
|
||
| ## Plan | ||
|
|
||
| - Every outbound datagram leaves through one site, `Output::Transmit` → | ||
| `socket.send_to` (`rs/moq-rtc/src/session.rs`). Sum `contents.len()` | ||
| there. Count inbound datagrams where the mux hands them to the peer. This | ||
| needs no str0m stats interval. | ||
| - Hand the embedder a cumulative `(tx, rx)` source per peer, in the shape | ||
| [Presence wire counters](/quest/m2/wire-stats/presence.md) registers. | ||
| - Test: a WHEP session's reported tx covers at least the RTP payload it | ||
| sent. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # [S] moq-srt wire source | ||
|
|
||
| ## Goal | ||
|
|
||
| moq-srt gives its embedder each SRT connection's cumulative wire bytes, both | ||
| directions, for publish and subscribe. These are the SRT packets (headers, | ||
| control packets, and retransmits included) without the IP and UDP headers, | ||
| the same definition as QUIC's `udp_tx`. | ||
|
|
||
| ## Plan | ||
|
|
||
| - srt-protocol's `SocketStatistics` carries `tx_all_bytes` / | ||
| `rx_all_bytes`, which include control packets but also a fixed 20 B IPv4 + | ||
| 8 B UDP estimate per packet. Subtract that estimate per packet so the | ||
| count is SRT packet bytes. The stats arrive on the socket's | ||
| `statistics()` watch. | ||
| - `serve_subscribe` and the publish path own the socket today and expose | ||
| nothing. Hand the embedder a cumulative `(tx, rx)` source per connection, | ||
| in the shape [Presence wire counters](/quest/m2/wire-stats/presence.md) | ||
| registers. | ||
| - Target the shared-socket integration that drives `srt-protocol` directly, | ||
| not `srt-tokio` socket handling. | ||
| - Test: an SRT pull's reported tx covers at least the TS payload it sent. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On Linux,
tcpi_bytes_receivedis the RFC 4898 application-throughput counter, so it counts bytes delivered to the TCP consumer rather than duplicate payload received in retransmitted segments. Under packet loss, the plannedwire_rxwill therefore omit retransmitted TCP payload even though the stated wire definition includes retransmits. Use receive-side instrumentation that observes every TCP data segment, or explicitly change the receive definition instead of relying on this field.Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. I changed the definition rather than adding segment-level instrumentation: TCP receive counts what TCP delivered, an accepted and documented undercount. Ingress is not where this metering matters.
(Written by Claude Opus 5.5)