Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ lite-07; bitrate enforcement is not part of this release.
- [Discover media headroom](/quest/m2/quic-probe.md) - test useful-media pacing before adding redundant probe traffic
- [Viewer up-switch](/quest/m2/viewer-upswitch.md) - a viewer capped by its small rendition finds headroom through PROBE and moves up
- [HTTP fetch stats](/quest/m2/http-fetch-stats.md) - each HTTP `/fetch` counts one `requests` and its bytes under the token's tier and root, not a session
- [Wire stats](/quest/m2/wire-stats/README.md) - `Presence` counts each root's transport bytes on the wire, `wire_tx` and `wire_rx`, from QUIC, WebSocket, and embedder gateway sources
- [Media stats](/quest/m2/stats/README.md) - publishers announce a stats track in the catalog, viewers answer a soliciting catalog through a per-catalog `.echo` broadcast, and one model turns both into a health verdict and a preflight report
- [BBR ACK cleanup](/quest/m2/bbr-ack-cleanup.md) - packet bookkeeping scales with completed entries instead of scanning the flight on every ACK
- [Perf](/quest/m2/perf/README.md) - eliminate measured hot-path costs across moq-uring, kio, and the moq-net model
Expand Down
75 changes: 75 additions & 0 deletions quest/m2/wire-stats/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Wire stats

## Goal

The relay's session stats carry `wire_tx` and `wire_rx` in `Presence`: the
bytes each root's connections put on and took off the wire, on each tier.
They appear in the `totals.json` track per tier and in the per-root
`presence.json` track ([Stats](/doc/concept/stats.md#tracks)). An operator
reads what each tenant root costs in transport bytes, not only the payload
`Traffic.bytes` counts.

Wire bytes are the transport's payload, with IP, UDP, and TCP headers
excluded:

- **QUIC and WebTransport:** `udp_tx` / `udp_rx`, which include handshake,
ACKs, and retransmits.
- **WebSocket:** the TCP stream's bytes, which include TLS, WebSocket, and
qmux framing and retransmits.
- **Gateways an embedder terminates:** SRT and WebRTC through moq-srt's and
moq-rtc's sources. An embedder terminating RTMP on its own socket
registers that socket's `TCP_INFO` counts; moq-rtmp's own `Server` and
client expose no source until a consumer needs one.
- **TCP receive:** bytes TCP delivered, so a peer's duplicate retransmits
are not counted.

Non-goals:

- **Splitting a shared connection's bytes across roots.** A cluster
connection's wire bytes count under its own root, which is empty.
- **Per-broadcast wire bytes.** Wire bytes belong to connections, not tracks.

## Plan

Decided 2026-10-13. Payload undercounts what delivery costs, and by how much
depends on the workload. Measured wire/payload in moq.pro's bench:

| Workload | Wire/payload |
|---|---|
| Video | 1.04x |
| Audio | 1.23x |
| Small chat-sized messages | 1.29-1.56x |
| Announce-only | 14.4x |

A provider bills the wire, so a relay operator needs per-root wire bytes to
see cost per tenant.

- **`Presence`, not `Traffic`.** `Presence` is already per (tier, root) and
per session. `Traffic.bytes` stays payload, and its docs say so.
- **Named `wire_tx` / `wire_rx`.** Rejected: `bytes_sent`, which reads like
payload beside `Traffic.bytes`.
- **Sampled on the stats tick.** Rejected: a once-only count at session end,
which loses a crashed session and lags a long one.
- **One source registry for sessions and embedders.** An embedder registers
a gateway connection's wire source against a (tier, root) on its stats
handle, the same path the relay's own sessions use, like
[session outcomes](/quest/m1/session-outcomes.md)' embedder refusals.

## Required

- [Presence wire counters](/quest/m2/wire-stats/presence.md) - QUIC,
WebTransport, and WebSocket sessions and embedder sources count per
(tier, root)
- [qmux byte counters](/quest/m2/wire-stats/qmux.md) - a WebSocket session
reports its TCP bytes
- [moq-srt wire source](/quest/m2/wire-stats/srt.md) - an SRT connection
reports its wire bytes
- [moq-rtc wire source](/quest/m2/wire-stats/rtc.md) - a WebRTC peer reports
its wire bytes

## Related

- [Session outcomes](/quest/m1/session-outcomes.md) - the other per-root
`Presence` counters and the embedder stats-handle pattern
- [HTTP fetch stats](/quest/m2/http-fetch-stats.md) - HTTP `/fetch`, which
has no session and stays payload-only
51 changes: 51 additions & 0 deletions quest/m2/wire-stats/presence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# [M] Presence wire counters

## Goal

`Presence` gains cumulative `wire_tx` and `wire_rx`, under the same reset
contract as its other cumulative fields: counted since the stats group's
epoch began (a group returning from idle starts a new epoch at zero), and
reset with a reclaimed and re-held presence track, so the aggregator never
re-adds bytes an earlier epoch already reported. On every stats tick the
relay adds the delta of each live session's transport bytes since its last
reading. A closing session adds its final reading. An embedder registers a connection's wire source against a
(tier, root) on its stats handle, and the same tick reads it.

## Plan

- **Sources:**
- **QUIC and WebTransport:** `bytes_sent` / `bytes_received` from
`moq_net::session::Stats`. noq maps them to `udp_tx` / `udp_rx`.
- **WebSocket:** the qmux counters from
[qmux byte counters](/quest/m2/wire-stats/qmux.md). Until those land, a
WebSocket session reports 0: an undercount, never a guess.
- **Read the transport directly.** The tick must not re-arm the per-session
100 ms sampler that moq#5382 stopped for idle sessions. Read the
connection's counters once per tick, cheaply. The tick's work must scale
with sources plus held roots, never their product: benchmark idle-relay
CPU sweeping both the source (session) count and the held per-root
presence track count, before and after.
- **Ticks, tier changes, and closes:**
- A session's per-connection counters reset with the connection, so keep
each session's last reading and add only non-negative deltas.
- `Session::set_tier` moves later deltas to the new tier.
- A close takes one final reading. A slightly stale final sample
under-counts, which is accepted.
- **Embedder API:** a source is a closure returning cumulative
`(tx, rx)`, registered with a guard. Dropping the guard takes its final
reading and unregisters it.
- **Serialization:** `#[serde(default)]` on both fields, so an older
consumer and an older relay interoperate. `Presence` stays `Copy`. The
stats-feed sum (`Presence::add`), the delta encoding, and the aggregator's
reset check (`Mergeable::regressed` in `rs/moq-stats/src/aggregate.rs`)
carry them, so a node whose wire totals drop is a reset.
- **Docs:** `doc/concept/stats.md` defines both fields, says
`Traffic.bytes` is payload, and says WebSocket wire bytes need a host
whose `TCP_INFO` reports them (Linux 4.19+), counting 0 elsewhere.
- **Tests (on controlled time):**
- a session's wire bytes land under its root and tier;
- a tier change splits them;
- a reconnecting source never lowers a counter;
- a stats group that idles out and returns starts its new epoch at zero;
- the aggregator treats a wire-only drop as a reset;
- a dropped embedder guard adds its last reading.
24 changes: 24 additions & 0 deletions quest/m2/wire-stats/qmux.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# [S] qmux byte counters

## Goal

A qmux session over TCP (WebSocket or raw TLS) reports cumulative bytes sent
and received in its `SessionStats`, so moq-net's `Stats.bytes_sent` /
`bytes_received` is nonzero on WebSocket. Sent bytes include TLS, WebSocket,
and qmux framing and TCP retransmits, and exclude TCP/IP headers. Received
bytes are what TCP delivered (`tcpi_bytes_received` is RFC 4898's
application-throughput counter), so a peer's duplicate retransmitted payload
is not counted: an accepted undercount on the receive side, documented
beside the field.

## Plan

- moq-dev/web-transport: qmux's `TcpStats` already reads `TCP_INFO` for rtt
and delivery rate. Add `tcpi_bytes_sent` and `tcpi_bytes_received` from
the same read. A platform without them (non-Linux, or Linux before 4.19)
Comment on lines +16 to +18

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Count retransmitted data in WebSocket receive totals

On Linux, tcpi_bytes_received is the RFC 4898 application-throughput counter, so it counts bytes delivered to the TCP consumer rather than duplicate payload received in retransmitted segments. Under packet loss, the planned wire_rx will therefore omit retransmitted TCP payload even though the stated wire definition includes retransmits. Use receive-side instrumentation that observes every TCP data segment, or explicitly change the receive definition instead of relying on this field.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I changed the definition rather than adding segment-level instrumentation: TCP receive counts what TCP delivered, an accepted and documented undercount. Ingress is not where this metering matters.

(Written by Claude Opus 5.5)

reports `None`, which moq-net maps to 0.
Comment on lines +18 to +19

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Refuse unavailable TCP wire counters instead of publishing zero

On every non-Linux host, and on older Linux kernels, this converts an unavailable counter into a real zero in Presence, so billing and monitoring consumers cannot distinguish an idle connection from completely unsupported WebSocket accounting and silently undercount all of its traffic. Preserve the unavailable state or refuse to enable WebSocket wire accounting on these platforms rather than publishing fabricated zero totals.

AGENTS.md reference: AGENTS.md:L17-L17

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly agreed. I won't refuse WebSocket or add an unavailable state: Presence sums across sessions and nodes, so a per-field "unknown" does not compose (a root mixing QUIC and WebSocket sessions is partly known either way), and refusing WebSocket over a metering counter trades a working fallback for a stat. The gap is narrow (non-Linux hosts, or Linux before 4.19), so 182bf72 makes it explicit instead: doc/concept/stats.md states the TCP_INFO requirement and that such hosts count 0.

(Written by Claude Opus 5.5)

- Release qmux on every line moq builds against, then bump it here and map
the fields in moq-net's WebSocket `Stats` and in the relay's
`SocketStats` wiring (`rs/moq-relay/src/websocket.rs`).
- Test: a loopback WebSocket session's `bytes_sent` covers at least the
payload it wrote.
19 changes: 19 additions & 0 deletions quest/m2/wire-stats/rtc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# [S] moq-rtc wire source

## Goal

moq-rtc gives its embedder each WebRTC peer's cumulative wire bytes, both
directions, for WHIP and WHEP: every STUN, DTLS, SRTP, and SRTCP datagram's
UDP payload. These counts exclude the IP and UDP headers, the same
definition as QUIC's `udp_tx`.

## Plan

- Every outbound datagram leaves through one site, `Output::Transmit` →
`socket.send_to` (`rs/moq-rtc/src/session.rs`). Sum `contents.len()`
there. Count inbound datagrams where the mux hands them to the peer. This
needs no str0m stats interval.
- Hand the embedder a cumulative `(tx, rx)` source per peer, in the shape
[Presence wire counters](/quest/m2/wire-stats/presence.md) registers.
- Test: a WHEP session's reported tx covers at least the RTP payload it
sent.
23 changes: 23 additions & 0 deletions quest/m2/wire-stats/srt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# [S] moq-srt wire source

## Goal

moq-srt gives its embedder each SRT connection's cumulative wire bytes, both
directions, for publish and subscribe. These are the SRT packets (headers,
control packets, and retransmits included) without the IP and UDP headers,
the same definition as QUIC's `udp_tx`.

## Plan

- srt-protocol's `SocketStatistics` carries `tx_all_bytes` /
`rx_all_bytes`, which include control packets but also a fixed 20 B IPv4 +
8 B UDP estimate per packet. Subtract that estimate per packet so the
count is SRT packet bytes. The stats arrive on the socket's
`statistics()` watch.
- `serve_subscribe` and the publish path own the socket today and expose
nothing. Hand the embedder a cumulative `(tx, rx)` source per connection,
in the shape [Presence wire counters](/quest/m2/wire-stats/presence.md)
registers.
- Target the shared-socket integration that drives `srt-protocol` directly,
not `srt-tokio` socket handling.
- Test: an SRT pull's reported tx covers at least the TS payload it sent.
Loading