Skip to content

feat: report per-connection usage - #310

Merged
englishm merged 3 commits into
cloudflare:mainfrom
itzmanish:feat/connection-usage
Nov 10, 2026
Merged

englishm merged 3 commits into
cloudflare:mainfrom
itzmanish:feat/connection-usage

Conversation

@itzmanish

Copy link
Copy Markdown
Contributor

Summary

Adds opt-in, per-connection usage reporting to the relay. An embedder registers a ConnectionUsageObserver with Relay::with_connection_usage, and the relay hands it a ConnectionUsage handle for every accepted connection. The handle exposes running totals that the embedder can read at any time, plus the connection's identity and tenant. When no observer is registered, nothing is read and the counting hooks are the only cost.

This targets draft-18 (main). A draft-16 backport will follow separately.

What is reported

Field Source
Connection ID QUIC connection ID
Interface public or internal (SessionInterface)
Transport webtransport or raw_quic
Scope Coordinator::metrics_scope_id, or none (see below)
Ingress / egress bytes UDP bytes from quinn's connection statistics
Ingress / egress objects MoQT objects on subgroup streams, datagrams, and FETCH records
Ingress / egress groups Distinct MoQT groups, counted once when their first object crosses

Totals run from when the connection was accepted. When the connection ends, its totals are frozen, so a final read after close still gets everything.

How counting works

moq-transport gains a TrafficCounter per session half, exposed through Session::traffic() as SessionTraffic / TrafficSnapshot. Counting happens where objects actually cross the wire:

  • Ingress: recv_subgroup, datagram receive, and Fetch::next_record.
  • Egress: the subgroup object forwarder, serve_datagrams, FetchWriter::write_record, and FetchRequested::proxy_body, so FETCH responses the relay proxies from upstream are counted at every hop.

A group can be split across several subgroup streams, and a late subgroup can arrive after newer groups. So each track keeps a 64-group bitmap behind its newest group, and a group counts only the first time it is seen inside that window. A connection tracks at most 4,096 tracks per direction and evicts the least recently used beyond that, which bounds memory for a client that touches many tracks.

Consecutive datagrams of one group skip the counter's lock. The receive loop remembers the last group it counted, so a burst of datagrams takes the lock once.

Tenant attribution

The reported scope comes only from the new Coordinator::metrics_scope_id. It defaults to None, so existing coordinators report nothing. ScopeInfo::scope_id is never reported directly, because its format is coordinator-defined and can contain credentials. The relay asks for metrics_scope_id only after the session is authorized.

The usage handle is created once MoQT setup completes, before scope resolution and authorization. A connection refused at either step is still reported, but without a scope, so a refused session's traffic is never attributed to a tenant. AGENTS.md records this rule.

API changes

  • moq-transport: Session::traffic(), SessionTraffic, TrafficSnapshot.
  • moq-native-ietf: Server::accept_with_stats, ConnectionStatistics, ConnectionStatisticsSnapshot; re-exports rustls, whose types appear in its public API.
  • moq-relay-ietf: Relay::with_connection_usage, ConnectionUsage, ConnectionUsageObserver, UsageSnapshot, Coordinator::metrics_scope_id (default method, so not breaking); re-exports moq_native_ietf, whose types appear in RelayConfig and Coordinator::lookup.

Known gaps

  • Very late groups. A group whose first object arrives more than 64 groups behind its track's newest is not counted, rather than risk counting it twice. A track that returns after LRU eviction may count a group again.
  • Setup failures and outbound links. Connections that fail MoQT setup, and the relay's own outbound connections to upstream relays, are not reported.
  • Byte counts include QUIC overhead. Bytes are UDP bytes, so they include framing, retransmissions and acknowledgements, not just MoQT payload.

Testing

  • End to end: a session test publishes across several subgroups and datagrams, and asserts exact counts on both sides (6 objects, 4 groups).
  • FETCH through a proxy: a proxied FETCH is asserted to count at every hop.
  • Full relay: a coordinator that refuses scope resolution confirms the connection is still reported, with no scope.
  • Units: the group window (late group inside the window, group older than the window) and LRU eviction.
  • Coordinator default: metrics_scope_id does not expose the routing scope.
  • Mutation checks: each new counting hook and the attribution rule were removed in turn to confirm a test goes red.

itzmanish and others added 3 commits November 4, 2026 11:43
Add an opt-in per-connection usage handle. With
Relay::with_connection_usage(observer), the relay hands the embedder a
ConnectionUsage for every accepted connection once its scope is
resolved and its interface classified. It carries the QUIC connection
ID, interface, transport, reported scope, and totals since the
connection was accepted: UDP bytes, MoQT objects and MoQT groups in
each direction. The embedder reads it on its own schedule; the relay
keeps no timer, registry or lock shared between connections. When the
connection ends, the totals are frozen and the connection is released.

moq-transport counts objects and groups as they cross a session, on
subgroup streams, as datagrams and as FETCH records, including FETCH
responses relayed by FetchRequested::proxy, and exposes them through
Session::traffic(). A group counts once, when its first object crosses,
even when it is split across several subgroup streams. Consecutive
datagrams of one group skip the per-session lookup.

The reported scope comes only from the new
Coordinator::metrics_scope_id(), which defaults to None;
ScopeInfo::scope_id is never reported because its format is
coordinator-defined and may carry credentials. It is set only after
authorization succeeds, so a refused session is never reported under a
tenant. A connection refused at scope resolution is not reported.

moq-native-ietf gains Server::accept_with_stats and re-exports rustls;
moq-relay-ietf re-exports moq_native_ietf. Their types appear in the
public APIs.
Report a connection refused at scope resolution. The connection tagger
needs only the peer address, TLS SNI and path, so it now runs before
scope resolution and the usage handle is created then; a refused
connection is reported without a tenant, as one refused at
authorization already was.

Count a group whose first object arrives after a newer group's. Each
track now remembers which of the 64 groups below its newest it has seen,
so a late group inside that window counts once. A group first seen
further behind is not counted, rather than risk counting twice one whose
mark has left the window.

Forget the least recently used track at the 4,096-track cap instead of
every track, so groups in flight on active tracks are not counted twice.
…its tenant

Only the call order kept a session refused at authorization from being
reported under its tenant: reported_scope took an authorized flag that
the relay always passed as true, so its test covered a branch the relay
never took. Drop the flag and test the rule through the relay instead,
with a coordinator that names a tenant and then denies the session.
Reporting the tenant before authorization now fails that test.

Create the usage guard before handing the usage to the observer, so an
observer that panics still closes it.
@englishm
englishm merged commit 0bc7e9e into cloudflare:main Nov 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants