diff --git a/quest/m2/README.md b/quest/m2/README.md index fe11c1948e0..fb19de9213c 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -39,6 +39,7 @@ lite-07; bitrate enforcement is not part of this release. - [Discover media headroom](/quest/m2/quic-probe.md) - test useful-media pacing before adding redundant probe traffic - [Viewer up-switch](/quest/m2/viewer-upswitch.md) - a viewer capped by its small rendition finds headroom through PROBE and moves up - [HTTP fetch stats](/quest/m2/http-fetch-stats.md) - each HTTP `/fetch` counts one `requests` and its bytes under the token's tier and root, not a session +- [Wire stats](/quest/m2/wire-stats/README.md) - `Presence` counts each root's transport bytes on the wire, `wire_tx` and `wire_rx`, from QUIC, WebSocket, and embedder gateway sources - [Media stats](/quest/m2/stats/README.md) - publishers announce a stats track in the catalog, viewers answer a soliciting catalog through a per-catalog `.echo` broadcast, and one model turns both into a health verdict and a preflight report - [BBR ACK cleanup](/quest/m2/bbr-ack-cleanup.md) - packet bookkeeping scales with completed entries instead of scanning the flight on every ACK - [Perf](/quest/m2/perf/README.md) - eliminate measured hot-path costs across moq-uring, kio, and the moq-net model diff --git a/quest/m2/wire-stats/README.md b/quest/m2/wire-stats/README.md new file mode 100644 index 00000000000..52727f99525 --- /dev/null +++ b/quest/m2/wire-stats/README.md @@ -0,0 +1,75 @@ +# Wire stats + +## Goal + +The relay's session stats carry `wire_tx` and `wire_rx` in `Presence`: the +bytes each root's connections put on and took off the wire, on each tier. +They appear in the `totals.json` track per tier and in the per-root +`presence.json` track ([Stats](/doc/concept/stats.md#tracks)). An operator +reads what each tenant root costs in transport bytes, not only the payload +`Traffic.bytes` counts. + +Wire bytes are the transport's payload, with IP, UDP, and TCP headers +excluded: + +- **QUIC and WebTransport:** `udp_tx` / `udp_rx`, which include handshake, + ACKs, and retransmits. +- **WebSocket:** the TCP stream's bytes, which include TLS, WebSocket, and + qmux framing and retransmits. +- **Gateways an embedder terminates:** SRT and WebRTC through moq-srt's and + moq-rtc's sources. An embedder terminating RTMP on its own socket + registers that socket's `TCP_INFO` counts; moq-rtmp's own `Server` and + client expose no source until a consumer needs one. +- **TCP receive:** bytes TCP delivered, so a peer's duplicate retransmits + are not counted. + +Non-goals: + +- **Splitting a shared connection's bytes across roots.** A cluster + connection's wire bytes count under its own root, which is empty. +- **Per-broadcast wire bytes.** Wire bytes belong to connections, not tracks. + +## Plan + +Decided 2026-10-13. Payload undercounts what delivery costs, and by how much +depends on the workload. Measured wire/payload in moq.pro's bench: + +| Workload | Wire/payload | +|---|---| +| Video | 1.04x | +| Audio | 1.23x | +| Small chat-sized messages | 1.29-1.56x | +| Announce-only | 14.4x | + +A provider bills the wire, so a relay operator needs per-root wire bytes to +see cost per tenant. + +- **`Presence`, not `Traffic`.** `Presence` is already per (tier, root) and + per session. `Traffic.bytes` stays payload, and its docs say so. +- **Named `wire_tx` / `wire_rx`.** Rejected: `bytes_sent`, which reads like + payload beside `Traffic.bytes`. +- **Sampled on the stats tick.** Rejected: a once-only count at session end, + which loses a crashed session and lags a long one. +- **One source registry for sessions and embedders.** An embedder registers + a gateway connection's wire source against a (tier, root) on its stats + handle, the same path the relay's own sessions use, like + [session outcomes](/quest/m1/session-outcomes.md)' embedder refusals. + +## Required + +- [Presence wire counters](/quest/m2/wire-stats/presence.md) - QUIC, + WebTransport, and WebSocket sessions and embedder sources count per + (tier, root) +- [qmux byte counters](/quest/m2/wire-stats/qmux.md) - a WebSocket session + reports its TCP bytes +- [moq-srt wire source](/quest/m2/wire-stats/srt.md) - an SRT connection + reports its wire bytes +- [moq-rtc wire source](/quest/m2/wire-stats/rtc.md) - a WebRTC peer reports + its wire bytes + +## Related + +- [Session outcomes](/quest/m1/session-outcomes.md) - the other per-root + `Presence` counters and the embedder stats-handle pattern +- [HTTP fetch stats](/quest/m2/http-fetch-stats.md) - HTTP `/fetch`, which + has no session and stays payload-only diff --git a/quest/m2/wire-stats/presence.md b/quest/m2/wire-stats/presence.md new file mode 100644 index 00000000000..6f1f569b278 --- /dev/null +++ b/quest/m2/wire-stats/presence.md @@ -0,0 +1,51 @@ +# [M] Presence wire counters + +## Goal + +`Presence` gains cumulative `wire_tx` and `wire_rx`, under the same reset +contract as its other cumulative fields: counted since the stats group's +epoch began (a group returning from idle starts a new epoch at zero), and +reset with a reclaimed and re-held presence track, so the aggregator never +re-adds bytes an earlier epoch already reported. On every stats tick the +relay adds the delta of each live session's transport bytes since its last +reading. A closing session adds its final reading. An embedder registers a connection's wire source against a +(tier, root) on its stats handle, and the same tick reads it. + +## Plan + +- **Sources:** + - **QUIC and WebTransport:** `bytes_sent` / `bytes_received` from + `moq_net::session::Stats`. noq maps them to `udp_tx` / `udp_rx`. + - **WebSocket:** the qmux counters from + [qmux byte counters](/quest/m2/wire-stats/qmux.md). Until those land, a + WebSocket session reports 0: an undercount, never a guess. +- **Read the transport directly.** The tick must not re-arm the per-session + 100 ms sampler that moq#5382 stopped for idle sessions. Read the + connection's counters once per tick, cheaply. The tick's work must scale + with sources plus held roots, never their product: benchmark idle-relay + CPU sweeping both the source (session) count and the held per-root + presence track count, before and after. +- **Ticks, tier changes, and closes:** + - A session's per-connection counters reset with the connection, so keep + each session's last reading and add only non-negative deltas. + - `Session::set_tier` moves later deltas to the new tier. + - A close takes one final reading. A slightly stale final sample + under-counts, which is accepted. +- **Embedder API:** a source is a closure returning cumulative + `(tx, rx)`, registered with a guard. Dropping the guard takes its final + reading and unregisters it. +- **Serialization:** `#[serde(default)]` on both fields, so an older + consumer and an older relay interoperate. `Presence` stays `Copy`. The + stats-feed sum (`Presence::add`), the delta encoding, and the aggregator's + reset check (`Mergeable::regressed` in `rs/moq-stats/src/aggregate.rs`) + carry them, so a node whose wire totals drop is a reset. +- **Docs:** `doc/concept/stats.md` defines both fields, says + `Traffic.bytes` is payload, and says WebSocket wire bytes need a host + whose `TCP_INFO` reports them (Linux 4.19+), counting 0 elsewhere. +- **Tests (on controlled time):** + - a session's wire bytes land under its root and tier; + - a tier change splits them; + - a reconnecting source never lowers a counter; + - a stats group that idles out and returns starts its new epoch at zero; + - the aggregator treats a wire-only drop as a reset; + - a dropped embedder guard adds its last reading. diff --git a/quest/m2/wire-stats/qmux.md b/quest/m2/wire-stats/qmux.md new file mode 100644 index 00000000000..eed111243e1 --- /dev/null +++ b/quest/m2/wire-stats/qmux.md @@ -0,0 +1,24 @@ +# [S] qmux byte counters + +## Goal + +A qmux session over TCP (WebSocket or raw TLS) reports cumulative bytes sent +and received in its `SessionStats`, so moq-net's `Stats.bytes_sent` / +`bytes_received` is nonzero on WebSocket. Sent bytes include TLS, WebSocket, +and qmux framing and TCP retransmits, and exclude TCP/IP headers. Received +bytes are what TCP delivered (`tcpi_bytes_received` is RFC 4898's +application-throughput counter), so a peer's duplicate retransmitted payload +is not counted: an accepted undercount on the receive side, documented +beside the field. + +## Plan + +- moq-dev/web-transport: qmux's `TcpStats` already reads `TCP_INFO` for rtt + and delivery rate. Add `tcpi_bytes_sent` and `tcpi_bytes_received` from + the same read. A platform without them (non-Linux, or Linux before 4.19) + reports `None`, which moq-net maps to 0. +- Release qmux on every line moq builds against, then bump it here and map + the fields in moq-net's WebSocket `Stats` and in the relay's + `SocketStats` wiring (`rs/moq-relay/src/websocket.rs`). +- Test: a loopback WebSocket session's `bytes_sent` covers at least the + payload it wrote. diff --git a/quest/m2/wire-stats/rtc.md b/quest/m2/wire-stats/rtc.md new file mode 100644 index 00000000000..1f200adf3a1 --- /dev/null +++ b/quest/m2/wire-stats/rtc.md @@ -0,0 +1,19 @@ +# [S] moq-rtc wire source + +## Goal + +moq-rtc gives its embedder each WebRTC peer's cumulative wire bytes, both +directions, for WHIP and WHEP: every STUN, DTLS, SRTP, and SRTCP datagram's +UDP payload. These counts exclude the IP and UDP headers, the same +definition as QUIC's `udp_tx`. + +## Plan + +- Every outbound datagram leaves through one site, `Output::Transmit` → + `socket.send_to` (`rs/moq-rtc/src/session.rs`). Sum `contents.len()` + there. Count inbound datagrams where the mux hands them to the peer. This + needs no str0m stats interval. +- Hand the embedder a cumulative `(tx, rx)` source per peer, in the shape + [Presence wire counters](/quest/m2/wire-stats/presence.md) registers. +- Test: a WHEP session's reported tx covers at least the RTP payload it + sent. diff --git a/quest/m2/wire-stats/srt.md b/quest/m2/wire-stats/srt.md new file mode 100644 index 00000000000..46700997348 --- /dev/null +++ b/quest/m2/wire-stats/srt.md @@ -0,0 +1,23 @@ +# [S] moq-srt wire source + +## Goal + +moq-srt gives its embedder each SRT connection's cumulative wire bytes, both +directions, for publish and subscribe. These are the SRT packets (headers, +control packets, and retransmits included) without the IP and UDP headers, +the same definition as QUIC's `udp_tx`. + +## Plan + +- srt-protocol's `SocketStatistics` carries `tx_all_bytes` / + `rx_all_bytes`, which include control packets but also a fixed 20 B IPv4 + + 8 B UDP estimate per packet. Subtract that estimate per packet so the + count is SRT packet bytes. The stats arrive on the socket's + `statistics()` watch. +- `serve_subscribe` and the publish path own the socket today and expose + nothing. Hand the embedder a cumulative `(tx, rx)` source per connection, + in the shape [Presence wire counters](/quest/m2/wire-stats/presence.md) + registers. +- Target the shared-socket integration that drives `srt-protocol` directly, + not `srt-tokio` socket handling. +- Test: an SRT pull's reported tx covers at least the TS payload it sent.