diff --git a/doc/concept/stats.md b/doc/concept/stats.md index 324ef6227af..33e4a1a1301 100644 --- a/doc/concept/stats.md +++ b/doc/concept/stats.md @@ -162,6 +162,13 @@ A presence track's entries are cumulative from when it was first held: a session that ends stays counted, and so does every one after it, while anyone holds the track. A track held again starts over. +Wire bytes start at zero on every hold, including the first. Other presence +fields seed from the root's current live-session counters; they do not recover +closed roots from before the hold. Node totals keep the full group epoch's +history. Each recreated track starts a fresh snapshot group using the +broadcast's shared, increasing group sequence, so relay caches can distinguish +it from an earlier hold. Plain and compressed tracks share that allocator. + ### Prefix ```json @@ -236,7 +243,8 @@ should prefer the canonical name and fall back to the legacy one. "sessions": 12, "sessions_closed": 10, "sessions_refused": { "refused": 0, "unavailable": 0, "request": 0, "forbidden": 0, "lan": 0 }, "sessions_failed": { "idle_timeout": 1, "transport": 0, "application": 0, "expired": 0 }, - "announces_peak": 40, "subscriptions_peak": 900 + "announces_peak": 40, "subscriptions_peak": 900, + "wire_tx": 2048, "wire_rx": 1024 } } ``` @@ -248,6 +256,21 @@ group on the tier for `totals.json`. `sessions` and `sessions_closed` are their legacy spellings. A session moved to a new tier ends on the old one and starts on the new. +`wire_tx` and `wire_rx` count transport bytes sent and received, separately +from `Traffic.bytes`, which counts media payload. QUIC and WebTransport count +UDP bytes, including retransmissions and transport overhead. The transport +updates a connection's counters as it sends and receives, and a stats tick reads +them without polling the session or activating its periodic sampler. Tier +changes flush the old tier first; closing a session records its final delta. +An embedder creates a `transport::Wire`, registers it with `stats::Handle::wire` +or `stats::Session::wire`, records its connection's cumulative totals on it, and +holds the returned guard. Dropping the guard records the final reading. + +Unsupported transports, including browsers and the current WebSocket/qmux +adapter, contribute zero. WebSocket wire accounting requires the separate +qmux counter source; unavailable counters are never estimated from payload. +Readers default missing wire fields to zero for older relays. + `sessions_refused` counts sessions turned away before admission, by reason. They never start a session. The relay's own refusals have no resolved root, so they count under the empty root on the default tier; an embedder can count one diff --git a/quest/m2/wire-stats/README.md b/quest/m2/wire-stats/README.md index b910cc7d16d..50490a99137 100644 --- a/quest/m2/wire-stats/README.md +++ b/quest/m2/wire-stats/README.md @@ -57,9 +57,6 @@ see cost per tenant. ## Required -- [Presence wire counters](/quest/m2/wire-stats/presence.md) - QUIC, - WebTransport, and WebSocket sessions and embedder sources count per - (tier, root) - [qmux byte counters](/quest/m2/wire-stats/qmux.md) - a WebSocket session reports its TCP bytes - [moq-srt wire source](/quest/m2/wire-stats/srt.md) - an SRT connection diff --git a/quest/m2/wire-stats/presence.md b/quest/m2/wire-stats/presence.md deleted file mode 100644 index 273e90ebd71..00000000000 --- a/quest/m2/wire-stats/presence.md +++ /dev/null @@ -1,51 +0,0 @@ -# [M] Presence wire counters - -## Goal - -`Presence` gains cumulative `wire_tx` and `wire_rx`, under the same reset -contract as its other cumulative fields: counted since the stats group's -epoch began (a group returning from idle starts a new epoch at zero), and -reset with a reclaimed and re-held presence track, so the aggregator never -re-adds bytes an earlier epoch already reported. On every stats tick the -relay adds the delta of each live session's transport bytes since its last -reading. A closing session adds its final reading. An embedder registers a connection's wire source against a -(tier, root) on its stats handle, and the same tick reads it. - -## Plan - -- **Sources:** - - **QUIC and WebTransport:** `bytes_sent` / `bytes_received` from - `moq_net::session::Stats`. `moq-quic` maps them to `udp_tx` / `udp_rx`. - - **WebSocket:** the qmux counters from - [qmux byte counters](/quest/m2/wire-stats/qmux.md). Until those land, a - WebSocket session reports 0: an undercount, never a guess. -- **Read the transport directly.** The tick must not re-arm the per-session - 100 ms sampler that moq#5382 stopped for idle sessions. Read the - connection's counters once per tick, cheaply. The tick's work must scale - with sources plus held roots, never their product: benchmark idle-relay - CPU sweeping both the source (session) count and the held per-root - presence track count, before and after. -- **Ticks, tier changes, and closes:** - - A session's per-connection counters reset with the connection, so keep - each session's last reading and add only non-negative deltas. - - `Session::set_tier` moves later deltas to the new tier. - - A close takes one final reading. A slightly stale final sample - under-counts, which is accepted. -- **Embedder API:** a source is a closure returning cumulative - `(tx, rx)`, registered with a guard. Dropping the guard takes its final - reading and unregisters it. -- **Serialization:** `#[serde(default)]` on both fields, so an older - consumer and an older relay interoperate. `Presence` stays `Copy`. The - stats-feed sum (`Presence::add`), the delta encoding, and the aggregator's - reset check (`Mergeable::regressed` in `rs/moq-stats/src/aggregate.rs`) - carry them, so a node whose wire totals drop is a reset. -- **Docs:** `doc/concept/stats.md` defines both fields, says - `Traffic.bytes` is payload, and says WebSocket wire bytes need a host - whose `TCP_INFO` reports them (Linux 4.19+), counting 0 elsewhere. -- **Tests (on controlled time):** - - a session's wire bytes land under its root and tier; - - a tier change splits them; - - a reconnecting source never lowers a counter; - - a stats group that idles out and returns starts its new epoch at zero; - - the aggregator treats a wire-only drop as a reset; - - a dropped embedder guard adds its last reading. diff --git a/quest/m2/wire-stats/qmux.md b/quest/m2/wire-stats/qmux.md index eed111243e1..d5507f67589 100644 --- a/quest/m2/wire-stats/qmux.md +++ b/quest/m2/wire-stats/qmux.md @@ -2,9 +2,9 @@ ## Goal -A qmux session over TCP (WebSocket or raw TLS) reports cumulative bytes sent -and received in its `SessionStats`, so moq-net's `Stats.bytes_sent` / -`bytes_received` is nonzero on WebSocket. Sent bytes include TLS, WebSocket, +A qmux session over TCP (WebSocket or raw TLS) keeps its connection's +`moq_net::transport::Wire` counters current, so the stats tick counts +WebSocket wire bytes instead of zero. Sent bytes include TLS, WebSocket, and qmux framing and TCP retransmits, and exclude TCP/IP headers. Received bytes are what TCP delivered (`tcpi_bytes_received` is RFC 4898's application-throughput counter), so a peer's duplicate retransmitted payload @@ -16,9 +16,14 @@ beside the field. - moq-dev/web-transport: qmux's `TcpStats` already reads `TCP_INFO` for rtt and delivery rate. Add `tcpi_bytes_sent` and `tcpi_bytes_received` from the same read. A platform without them (non-Linux, or Linux before 4.19) - reports `None`, which moq-net maps to 0. -- Release qmux on every line moq builds against, then bump it here and map - the fields in moq-net's WebSocket `Stats` and in the relay's + reports `None`, which moq-net maps to 0. Upstream source work: + moq-dev/web-transport#435 and #436. +- The transport pushes its counters: the stats tick never polls a session, + and there is no per-connection timer. `TCP_INFO` is read on demand, so + decide how qmux's I/O path refreshes the `Wire` (for example, after each + socket write batch) without a syscall per write, then return it from + moq-tokio's `Grouped::wire` for `qmux::Session`, including the relay's `SocketStats` wiring (`rs/moq-relay/src/websocket.rs`). -- Test: a loopback WebSocket session's `bytes_sent` covers at least the +- Release qmux on every line moq builds against, then bump it here. +- Test: a loopback WebSocket session's `sent_bytes` covers at least the payload it wrote. diff --git a/quest/m2/wire-stats/srt.md b/quest/m2/wire-stats/srt.md index 6a0debce67d..b464008a1bd 100644 --- a/quest/m2/wire-stats/srt.md +++ b/quest/m2/wire-stats/srt.md @@ -15,9 +15,9 @@ the same definition as QUIC's `udp_tx`. count is SRT packet bytes. The stats arrive on the socket's `statistics()` watch. - `serve_subscribe` and the publish path own the socket today and expose - nothing. Hand the embedder a cumulative `(tx, rx)` source per connection, - in the shape [Presence wire counters](/quest/m2/wire-stats/presence.md) - registers. + nothing. Update a `moq_net::transport::Wire` per connection + from each statistics update and hand it to the embedder, which registers it + with `moq_net::stats::Handle::wire`. - Target the shared-socket integration that drives `srt-protocol` directly, not `srt-tokio` socket handling. - Test: an SRT pull's reported tx covers at least the TS payload it sent. diff --git a/rs/moq-net/src/client.rs b/rs/moq-net/src/client.rs index 00e3d42a6d8..5d51a3f8f4b 100644 --- a/rs/moq-net/src/client.rs +++ b/rs/moq-net/src/client.rs @@ -227,6 +227,7 @@ impl Client { start.recv_bandwidth, crate::driver::Protocol::Lite(Box::new(start.driver)), crate::session::Handles { + stats: self.stats.clone(), goaway: start.goaway, auth: start.auth, estimate: Some(start.estimate), @@ -320,6 +321,7 @@ impl Client { None, crate::driver::Protocol::Ietf(protocol), crate::session::Handles { + stats: self.stats.clone(), goaway, auth, estimate: None, @@ -493,7 +495,12 @@ impl Client { version, recv_bw, protocol, - crate::session::Handles { goaway, auth, estimate }, + crate::session::Handles { + stats: self.stats.clone(), + goaway, + auth, + estimate, + }, )) } } diff --git a/rs/moq-net/src/server.rs b/rs/moq-net/src/server.rs index 50fe8bf01e1..b23061cfcf4 100644 --- a/rs/moq-net/src/server.rs +++ b/rs/moq-net/src/server.rs @@ -194,6 +194,7 @@ impl Server { start.recv_bandwidth, crate::driver::Protocol::Lite(Box::new(start.driver)), crate::session::Handles { + stats: self.stats.clone(), goaway: start.goaway, auth: start.auth, estimate: Some(start.estimate), @@ -649,6 +650,7 @@ where None, crate::driver::Protocol::Ietf(protocol), crate::session::Handles { + stats: server.stats.clone(), goaway, auth, estimate: None, @@ -784,7 +786,12 @@ where version, recv_bw, protocol, - crate::session::Handles { goaway, auth, estimate }, + crate::session::Handles { + stats: server.stats.clone(), + goaway, + auth, + estimate, + }, )) } .maybe_boxed() diff --git a/rs/moq-net/src/session.rs b/rs/moq-net/src/session.rs index a8082c3da84..6062ed379d4 100644 --- a/rs/moq-net/src/session.rs +++ b/rs/moq-net/src/session.rs @@ -289,6 +289,7 @@ impl Session { /// The handles a protocol driver shares with its [`Session`]. pub(super) struct Handles { + pub stats: crate::stats::Session, pub goaway: goaway::Handle, pub auth: auth::Handle, /// Where the driver publishes [`Estimate`] changes, for a protocol that reports @@ -308,7 +309,13 @@ impl Session { where S: crate::transport::poll::Session, { - let Handles { goaway, auth, estimate } = handles; + let Handles { + goaway, + auth, + estimate, + stats: context, + } = handles; + let wire = session.wire().map(|counters| context.wire(counters)); let sample = snapshot(&session); if let Some(estimate) = &estimate && let Ok(mut current) = estimate.write() @@ -341,6 +348,7 @@ impl Session { close: Some(close.consume()), closed, stats: stats.clone(), + wire, send_bandwidth: send_producer, estimate, sampled: None, @@ -401,6 +409,8 @@ pub(crate) struct Supervisor { sampled: Option, mode: SamplerMode, drain: Drain, + /// The transport's wire counters, registered under the session's root; dropping records the last reading. + wire: Option, } /// A [`Session::close`] waiting for the protocol to deliver what it queued. @@ -436,6 +446,7 @@ impl Supervisor { // this cell, so leave it holding the session's final counters // rather than whichever sample the last demand happened to catch. self.stats.lock().sample = snapshot(&self.session); + self.wire.take(); let drain = match std::mem::replace(&mut self.drain, Drain::Idle) { Drain::Done(res) => Some(res), _ => None, diff --git a/rs/moq-net/src/stats.rs b/rs/moq-net/src/stats.rs index 3b773c3c05d..542f4d58053 100644 --- a/rs/moq-net/src/stats.rs +++ b/rs/moq-net/src/stats.rs @@ -62,6 +62,12 @@ //! Driven by [`Handle::session`] (the [`Session`] context); a //! [`Session::set_tier`] ends the session on the old tier and starts it on //! the new one. +//! * `wire_tx` / `wire_rx` ([`Presence`]): cumulative transport bytes, including +//! overhead and retransmissions. The transport updates its +//! [`crate::transport::Wire`] counters as it sends and receives, and +//! [`Registry::report`] and [`Registry::snapshot`] read each registered [`Wire`] +//! without polling the session. A tier move flushes its old tier; dropping the guard flushes +//! its final nonnegative delta. Unsupported transports contribute zero. //! * `announces_peak` / `subscriptions_peak` ([`Presence`]): the most //! announcements, or subscriptions, any one session under the auth root held //! at once, to compare against its `crate::session::Limits`. A peak never @@ -280,6 +286,8 @@ struct SessionCounters { sessions_ended: AtomicU64, announces_peak: AtomicU64, subscriptions_peak: AtomicU64, + wire_tx: AtomicU64, + wire_rx: AtomicU64, } impl SessionCounters { @@ -301,6 +309,8 @@ impl SessionCounters { sessions_refused: self.sessions_refused.snapshot(), sessions_failed, requests: self.requests.load(Ordering::Relaxed), + wire_tx: self.wire_tx.load(Ordering::Relaxed), + wire_rx: self.wire_rx.load(Ordering::Relaxed), sessions_started, sessions_ended, announces_peak: self.announces_peak.load(Ordering::Relaxed), @@ -561,6 +571,10 @@ pub struct Presence { pub announces_peak: u64, /// The most subscriptions any one session has held on our broadcasts at once. pub subscriptions_peak: u64, + /// Cumulative transport bytes sent, including transport framing and retransmits. + pub wire_tx: u64, + /// Cumulative transport bytes received, including transport framing. + pub wire_rx: u64, } #[derive(Serialize)] @@ -574,6 +588,8 @@ struct PresenceSer { sessions_closed: u64, announces_peak: u64, subscriptions_peak: u64, + wire_tx: u64, + wire_rx: u64, } impl From for PresenceSer { @@ -582,6 +598,8 @@ impl From for PresenceSer { sessions_refused: p.sessions_refused, sessions_failed: p.sessions_failed, requests: p.requests, + wire_tx: p.wire_tx, + wire_rx: p.wire_rx, sessions_started: p.sessions_started, sessions: p.sessions_started, sessions_ended: p.sessions_ended, @@ -604,6 +622,10 @@ struct PresenceDe { sessions_closed: Edge, announces_peak: u64, subscriptions_peak: u64, + #[serde(default)] + wire_tx: u64, + #[serde(default)] + wire_rx: u64, } impl From for Presence { @@ -612,6 +634,8 @@ impl From for Presence { sessions_refused: d.sessions_refused, sessions_failed: d.sessions_failed, requests: d.requests, + wire_tx: d.wire_tx, + wire_rx: d.wire_rx, sessions_started: counter_edge(d.sessions_started, d.sessions), sessions_ended: counter_edge(d.sessions_ended, d.sessions_closed), announces_peak: d.announces_peak, @@ -638,6 +662,8 @@ impl Presence { self.sessions_refused.add(other.sessions_refused); self.sessions_failed.add(other.sessions_failed); self.requests += other.requests; + self.wire_tx += other.wire_tx; + self.wire_rx += other.wire_rx; self.sessions_started += other.sessions_started; self.sessions_ended += other.sessions_ended; self.announces_peak = self.announces_peak.max(other.announces_peak); @@ -875,6 +901,7 @@ struct Shared { /// presence tracks. A tier's inner map is /// created the first time a session records under it. sessions: Lock>>>, + wire: Lock>>, } /// Per-broadcast counters, lazily split by tier. A tier's [`TierCounters`] is @@ -923,6 +950,7 @@ impl Registry { retired: Lock::default(), entries: Lock::default(), sessions: Default::default(), + wire: Lock::default(), })), } } @@ -1019,6 +1047,7 @@ impl Registry { /// node lifetime totals (what a `/metrics`-style scrape wants) and never prunes. /// Retired entries remain in these totals after [`Self::report`] prunes them. pub fn snapshot(&self) -> Snapshot { + self.collect_wire(); let Some(shared) = self.shared.as_ref() else { return Snapshot::default(); }; @@ -1058,6 +1087,7 @@ impl Registry { /// sees traffic again restarts from zero; see the module docs on counter /// resets. Leaves the report empty for a disabled registry. pub fn report(&self, report: &mut Report) { + self.collect_wire(); report.traffic.clear(); report.sessions.clear(); let Some(shared) = self.shared.as_ref() else { @@ -1110,6 +1140,15 @@ impl Registry { !roots.is_empty() }); } + + /// Sample each source once before taking the counter maps, independently of held tracks. + fn collect_wire(&self) { + let Some(shared) = &self.shared else { return }; + shared.wire.lock().retain(|source| { + let Some(source) = source.upgrade() else { return false }; + source.sample(false) + }); + } } impl Default for Registry { @@ -1155,29 +1194,38 @@ impl Handle { /// track for this tier, keyed by `root`, in its totals, and in `root`'s /// presence track. pub fn session(&self, root: impl AsPath) -> Session { - self.context(root, true) + self.context(root, Context::Session) } /// Count one admitted request under `root` and meter its traffic without session presence. pub fn request(&self, root: impl AsPath) -> Session { - self.context(root, false) + self.context(root, Context::Request) + } + + /// Count an external connection's wire counters under `root` without counting an admission. + pub fn wire(&self, root: impl AsPath, counters: crate::transport::Wire) -> Wire { + self.context(root, Context::Wire).wire(counters) } - fn context(&self, root: impl AsPath, connected: bool) -> Session { + fn context(&self, root: impl AsPath, context: Context) -> Session { let root = root.as_path().to_owned(); let presence = self.stats.session_counters(&self.tier, &root); if let Some(presence) = &presence { - if connected { - presence.sessions_started.fetch_add(1, Ordering::Relaxed); - } else { - presence.requests.fetch_add(1, Ordering::Relaxed); + match context { + Context::Session => { + presence.sessions_started.fetch_add(1, Ordering::Relaxed); + } + Context::Request => { + presence.requests.fetch_add(1, Ordering::Relaxed); + } + Context::Wire => {} } } Session { inner: Some(Arc::new(SessionInner { registry: self.stats.clone(), root, - connected, + connected: matches!(context, Context::Session), current: Mutex::new(Current { failure: None, tier: self.tier.clone(), @@ -1186,6 +1234,7 @@ impl Handle { generation: AtomicU64::new(0), viewers: Mutex::new(HashMap::new()), held: Mutex::new(Vec::new()), + wire: Mutex::new(Vec::new()), })), } } @@ -1198,6 +1247,70 @@ impl Default for Handle { } } +enum Context { + Session, + Request, + Wire, +} + +/// Registered wire counters; dropping this adds their final reading and unregisters them. +#[derive(Default)] +pub struct Wire { + inner: Option>, +} + +struct WireInner { + session: Arc, + counters: crate::transport::Wire, + last: Mutex, +} + +#[derive(Default)] +struct WireState { + sent_bytes: u64, + received_bytes: u64, + closed: bool, +} + +impl WireInner { + fn sample(&self, close: bool) -> bool { + let current = self.session.current.lock().unwrap_or_else(PoisonError::into_inner); + self.sample_on(¤t, close) + } + + /// Current always precedes last: tier changes and a tick cannot attribute one reading twice. + fn sample_on(&self, current: &Current, close: bool) -> bool { + let mut last = self.last.lock().unwrap_or_else(PoisonError::into_inner); + if last.closed { + return false; + }; + let sent_bytes = self.counters.sent_bytes(); + let received_bytes = self.counters.received_bytes(); + if let Some(presence) = ¤t.presence { + presence + .wire_tx + .fetch_add(sent_bytes.saturating_sub(last.sent_bytes), Ordering::Relaxed); + presence + .wire_rx + .fetch_add(received_bytes.saturating_sub(last.received_bytes), Ordering::Relaxed); + } + *last = WireState { + sent_bytes, + received_bytes, + closed: close, + }; + !close + } +} + +impl Drop for Wire { + fn drop(&mut self) { + if let Some(inner) = &self.inner { + inner.sample(true); + } + } +} + /// Which side of a [`TierCounters`] a bump lands on: publisher (egress) or /// subscriber (ingress). `Default` is `Publisher`, chosen only so an empty /// [`Meter`] / [`Scope`] has one; it never records because its counters are `None`. @@ -1271,6 +1384,8 @@ struct SessionInner { /// What this session holds against each cap right now, so a tier change can seed the /// new tier's peaks with slots it still holds. held: Mutex)>>, + /// Weak, because a wire guard retains this context until its final reading. + wire: Mutex>>, } /// The tier a [`Session`] records under right now. @@ -1289,6 +1404,28 @@ struct Viewer { } impl Session { + /// Count a connection's wire counters under this context's root, following its tier. + pub fn wire(&self, counters: crate::transport::Wire) -> Wire { + let Some(inner) = &self.inner else { + return Wire::default(); + }; + let Some(shared) = &inner.registry.shared else { + return Wire::default(); + }; + let source = Arc::new(WireInner { + session: inner.clone(), + counters, + last: Mutex::new(WireState::default()), + }); + { + let mut wire = inner.wire.lock().unwrap_or_else(PoisonError::into_inner); + wire.retain(|source| source.strong_count() > 0); + wire.push(Arc::downgrade(&source)); + } + shared.wire.lock().push(Arc::downgrade(&source)); + Wire { inner: Some(source) } + } + /// Record the first abnormal end on this connected session, shared across clones. pub fn fail(&self, kind: Failure) { let Some(inner) = &self.inner else { return }; @@ -1308,6 +1445,14 @@ impl Session { if current.tier == tier { return; } + inner + .wire + .lock() + .unwrap_or_else(PoisonError::into_inner) + .retain(|source| { + let Some(source) = source.upgrade() else { return false }; + source.sample_on(¤t, false) + }); let presence = inner.registry.session_counters(&tier, &inner.root); if inner.connected && let Some(presence) = &presence @@ -1691,6 +1836,120 @@ mod tests { use super::*; + fn counters(sent: u64, received: u64) -> crate::transport::Wire { + let wire = crate::transport::Wire::new(); + wire.set_sent_bytes(sent); + wire.set_received_bytes(received); + wire + } + + #[test] + fn wire_deltas_follow_root_and_tier_changes() { + let registry = test_stats(); + let old = Tier::new("old"); + let new = Tier::new("new"); + let session = registry.tier(old.clone()).session("room"); + let bytes = counters(10, 20); + let _wire = session.wire(bytes.clone()); + let report = drain(®istry); + assert_eq!(report.sessions.len(), 1); + assert_eq!(report.sessions[0].root.as_str(), "room"); + assert_eq!(report.sessions[0].tier, old); + assert_eq!( + (report.sessions[0].presence.wire_tx, report.sessions[0].presence.wire_rx), + (10, 20) + ); + bytes.set_sent_bytes(15); + bytes.set_received_bytes(27); + session.set_tier(new.clone()); + bytes.set_sent_bytes(25); + bytes.set_received_bytes(50); + let report = drain(®istry); + for (tier, expected) in [(old, (15, 27)), (new, (10, 23))] { + let row = report.sessions.iter().find(|row| row.tier == tier).unwrap(); + assert_eq!((row.presence.wire_tx, row.presence.wire_rx), expected); + } + } + + #[test] + fn a_wire_counter_reset_never_lowers_presence() { + let registry = test_stats(); + let session = registry.tier(Tier::default()).session("room"); + let bytes = crate::transport::Wire::new(); + let _wire = session.wire(bytes.clone()); + for (reading, expected) in [((20, 30), (20, 30)), ((3, 4), (20, 30)), ((8, 10), (25, 36))] { + bytes.set_sent_bytes(reading.0); + bytes.set_received_bytes(reading.1); + let report = drain(®istry); + assert_eq!( + (report.sessions[0].presence.wire_tx, report.sessions[0].presence.wire_rx), + expected + ); + } + } + + #[test] + fn dropping_a_wire_guard_records_once_and_unregisters_without_an_admission() { + let registry = test_stats(); + let bytes = counters(10, 3); + let wire = registry.tier(Tier::default()).wire("room", bytes.clone()); + assert_eq!(drain(®istry).sessions[0].presence.wire_tx, 10); + bytes.set_sent_bytes(17); + drop(wire); + let final_row = drain(®istry).sessions.pop().unwrap(); + assert!(final_row.closed); + assert_eq!((final_row.presence.wire_tx, final_row.presence.wire_rx), (17, 3)); + assert_eq!( + ( + final_row.presence.requests, + final_row.presence.sessions_started, + final_row.presence.sessions_ended + ), + (0, 0, 0) + ); + bytes.set_sent_bytes(40); + assert!(drain(®istry).sessions.is_empty()); + assert_eq!( + registry.snapshot().sessions()[0].1.wire_tx, + 17, + "a dropped guard is never read again" + ); + } + + #[test] + fn disabled_wire_collection_reports_nothing() { + let registry = Registry::disabled(); + let wire = registry.tier(Tier::default()).wire("root", counters(5, 7)); + assert!(drain(®istry).sessions.is_empty()); + drop(wire); + assert!(drain(®istry).sessions.is_empty()); + } + + #[test] + fn wire_guards_hold_the_session_until_the_final_reading() { + let registry = test_stats(); + let session = registry.tier(Tier::default()).session("room"); + let wire = session.wire(counters(19, 23)); + drop(session); + assert_eq!(drain(®istry).sessions[0].presence.active(), 1); + drop(wire); + let report = drain(®istry); + assert!(report.sessions[0].closed); + assert_eq!(report.sessions[0].presence.active(), 0); + assert_eq!( + (report.sessions[0].presence.wire_tx, report.sessions[0].presence.wire_rx), + (19, 23) + ); + } + + #[test] + fn presence_wire_fields_survive_serialization() { + let value: Presence = serde_json::from_str(r#"{"wire_tx":71,"wire_rx":29}"#).unwrap(); + let output = serde_json::to_value(value).unwrap(); + assert_eq!(output["wire_tx"], 71); + assert_eq!(output["wire_rx"], 29); + } + #[test] fn outcomes_survive_clones_tier_changes_and_pruning() { let stats = test_stats(); diff --git a/rs/moq-net/src/transport.rs b/rs/moq-net/src/transport.rs index 6d0375e9203..107e1f7fc5d 100644 --- a/rs/moq-net/src/transport.rs +++ b/rs/moq-net/src/transport.rs @@ -47,6 +47,47 @@ pub enum Binding { InBand, } +/// One connection's cumulative wire counters, which its transport updates as it sends and receives. +/// +/// Clones share the counters, so stats read them without polling the session or keeping the +/// connection alive, and the final totals stay readable after it closes. Each counter is +/// independent: a transport records the ones it tracks, and the rest stay zero. +#[derive(Clone, Debug, Default)] +pub struct Wire(std::sync::Arc); + +#[derive(Debug, Default)] +struct WireCounters { + sent_bytes: std::sync::atomic::AtomicU64, + received_bytes: std::sync::atomic::AtomicU64, +} + +impl Wire { + /// Create counters that start at zero. + pub fn new() -> Self { + Self::default() + } + + /// The bytes sent so far, including transport framing and retransmissions. + pub fn sent_bytes(&self) -> u64 { + self.0.sent_bytes.load(std::sync::atomic::Ordering::Relaxed) + } + + /// The bytes received so far, including transport framing. + pub fn received_bytes(&self) -> u64 { + self.0.received_bytes.load(std::sync::atomic::Ordering::Relaxed) + } + + /// Record the cumulative bytes sent so far. + pub fn set_sent_bytes(&self, total: u64) { + self.0.sent_bytes.store(total, std::sync::atomic::Ordering::Relaxed); + } + + /// Record the cumulative bytes received so far. + pub fn set_received_bytes(&self, total: u64) { + self.0.received_bytes.store(total, std::sync::atomic::Ordering::Relaxed); + } +} + /// Connection-level statistics. /// /// Methods return `Option`: `None` means the implementation doesn't track @@ -237,6 +278,11 @@ pub mod poll { /// rather than defaulted for the same reason as [`protocol`](Self::protocol). fn stats(&self) -> impl Stats; + /// Return the wire counters this transport updates, or `None` when it tracks none. + fn wire(&self) -> Option { + None + } + /// Accept the next unidirectional stream opened by the peer. fn accept_uni(&mut self) -> AcceptUni<'_, Self> { AcceptUni(self) diff --git a/rs/moq-net/tests/integration/idle_session.rs b/rs/moq-net/tests/integration/idle_session.rs index 4e910e1f595..b5f01fa54c2 100644 --- a/rs/moq-net/tests/integration/idle_session.rs +++ b/rs/moq-net/tests/integration/idle_session.rs @@ -123,3 +123,40 @@ async fn ietf_stats_refresh_costs_one_driver_poll() { assert_eq!(polls, reads, "{version}: one poll per refresh"); } } + +/// A stats tick reads the transport's wire counters and adds no sampler timer or session wake. +#[moq_clock::test] +async fn wire_collection_never_arms_the_idle_sampler() { + for version in ["moq-lite-06", "moq-lite-07-wip", "moq-transport-16", "moq-transport-22"] { + let registry = moq_net::stats::Registry::new(Default::default()); + let polls = Arc::new(AtomicUsize::new(0)); + let mut options = MockConnectOptions::new(version.parse().unwrap()); + options.server_stats = registry.tier(Default::default()).session("room"); + options.server_polls = Some(polls.clone()); + let pair = connect_mock(options).await; + moq_clock::sleep(Duration::from_secs(1)).await; + let start_polls = polls.load(Ordering::Relaxed); + let start_reads = pair.server_transport.stats_reads(); + for tick in 1..=60 { + pair.server_transport.wire().set_sent_bytes(tick); + pair.server_transport.wire().set_received_bytes(tick * 2); + let mut report = moq_net::stats::Report::default(); + registry.report(&mut report); + assert_eq!( + (report.sessions[0].presence.wire_tx, report.sessions[0].presence.wire_rx), + (tick, tick * 2) + ); + moq_clock::sleep(Duration::from_secs(1)).await; + } + assert_eq!( + polls.load(Ordering::Relaxed), + start_polls, + "{version}: a tick woke the session" + ); + assert_eq!( + pair.server_transport.stats_reads(), + start_reads, + "{version}: a tick armed the sampler" + ); + } +} diff --git a/rs/moq-net/tests/integration/support/harness.rs b/rs/moq-net/tests/integration/support/harness.rs index dca98dd7851..677e2bc5ad7 100644 --- a/rs/moq-net/tests/integration/support/harness.rs +++ b/rs/moq-net/tests/integration/support/harness.rs @@ -81,6 +81,8 @@ pub struct MockConnectOptions { pub cost: Option, /// What both transports report as their connection statistics. pub stats: MockStats, + /// The context the server records under. + pub server_stats: moq_net::stats::Session, /// Count server driver polls into this counter when supplied. pub server_polls: Option>, } @@ -101,6 +103,7 @@ impl MockConnectOptions { server_extensions: Default::default(), cost: None, stats: MockStats::default(), + server_stats: Default::default(), server_polls: None, } } @@ -150,6 +153,7 @@ pub async fn connect_mock(opts: MockConnectOptions) -> MockPair { } let mut server = Server::new() + .with_stats(opts.server_stats) .with_versions(opts.version.into()) .with_extensions(opts.server_extensions); if let Some(publish) = &opts.server_publish { diff --git a/rs/moq-net/tests/integration/support/mock.rs b/rs/moq-net/tests/integration/support/mock.rs index 2301a706073..1ba0ec693ba 100644 --- a/rs/moq-net/tests/integration/support/mock.rs +++ b/rs/moq-net/tests/integration/support/mock.rs @@ -469,6 +469,8 @@ struct Side { stats_reads: AtomicUsize, /// Whether reading the stats panics; see [`MockSession::panic_on_stats`]. panic_on_stats: AtomicBool, + /// The wire counters [`poll::Session::wire`] exposes, which a test updates as a transport would. + wire: moq_net::transport::Wire, } /// The connection statistics a [`MockSession`] reports. Unavailable by default. @@ -644,6 +646,10 @@ impl poll::Session for MockSession { self.side.stats_reads.fetch_add(1, Ordering::Relaxed); *self.side.stats.lock().unwrap() } + + fn wire(&self) -> Option { + Some(self.side.wire.clone()) + } } // Only some test binaries steer delivery. @@ -683,6 +689,11 @@ impl MockSession { self.side.stats_reads.load(Ordering::Relaxed) } + /// The wire counters this side's transport exposes. + pub fn wire(&self) -> &moq_net::transport::Wire { + &self.side.wire + } + /// Panic on the driver's next stats sample, which a [`moq_net::Session::stats`] /// read requests. pub fn panic_on_stats(&self) { @@ -863,6 +874,7 @@ pub fn create_mock_session_pair(config: memory::Config) -> (MockSession, MockSes stats: Default::default(), stats_reads: Default::default(), panic_on_stats: Default::default(), + wire: Default::default(), }), accept_uni: Default::default(), accept_bi: Default::default(), diff --git a/rs/moq-relay/tests/integration/main.rs b/rs/moq-relay/tests/integration/main.rs index a21f62cd43c..8e8d85d9afe 100644 --- a/rs/moq-relay/tests/integration/main.rs +++ b/rs/moq-relay/tests/integration/main.rs @@ -25,3 +25,4 @@ mod shutdown_signal; mod smoke; mod stack; mod upstream_links; +mod wire_stats; diff --git a/rs/moq-relay/tests/integration/wire_stats.rs b/rs/moq-relay/tests/integration/wire_stats.rs new file mode 100644 index 00000000000..cf97416540d --- /dev/null +++ b/rs/moq-relay/tests/integration/wire_stats.rs @@ -0,0 +1,108 @@ +//! Transport wire counters across the shared-runtime and io_uring listener boundary. +#![cfg(feature = "_quic")] + +use moq_net::stats::{Presence, Registry, Tier}; +use moq_relay::{Config, Relay}; +use std::time::Duration; + +const TIMEOUT: Duration = Duration::from_secs(10); + +fn presence(registry: &Registry) -> Presence { + registry + .snapshot() + .sessions() + .into_iter() + .find(|(tier, _)| *tier == Tier::default()) + .map_or(Presence::default(), |(_, value)| value) +} + +async fn wire_sources(io_uring: bool) { + moq_tokio::crypto::install_default().unwrap(); + let dir = tempfile::tempdir().unwrap(); + let key = rcgen::KeyPair::generate().unwrap(); + let cert = rcgen::CertificateParams::new(vec!["localhost".into()]) + .unwrap() + .self_signed(&key) + .unwrap(); + let cert_path = dir.path().join("cert.pem"); + let key_path = dir.path().join("key.pem"); + std::fs::write(&cert_path, cert.pem()).unwrap(); + std::fs::write(&key_path, key.serialize_pem()).unwrap(); + let mut config = Config::default(); + config.listen.bind = Some("127.0.0.1:0".parse().unwrap()); + config.listen.tls.cert = vec![cert_path]; + config.listen.tls.key = vec![key_path]; + config.auth.public = vec![moq_auth::Pattern::all()]; + config.stats.enabled = true; + config.stats.interval = 3600; + config.drain_timeout = Duration::ZERO; + if io_uring { + config.runtime.workers = Some(1); + config.runtime.pin = false; + config.runtime.io_uring = true; + } + let relay = Relay::load(config).await.unwrap().with_signals(false); + let addr = relay.addr().unwrap(); + let registry = relay.cluster().stats.clone(); + let trigger = relay.shutdown_trigger().clone(); + let running = tokio::spawn(relay.run()); + // Both adapters must expose UDP totals without asking Session::stats(), + // which would arm a periodic sampler. Snapshot reads the source directly. + for scheme in ["moql", "https"] { + let mut client = moq_tokio::connect::Config::default(); + client.tls.insecure = Some(true); + client.bind = Some("127.0.0.1:0".parse().unwrap()); + let client = client.init(Default::default()).unwrap(); + let url: url::Url = format!("{scheme}://{addr}/wire").parse().unwrap(); + let connection = tokio::time::timeout(TIMEOUT, client.with_reconnect(false).connect(url).established()) + .await + .unwrap() + .unwrap(); + let before = tokio::time::timeout(TIMEOUT, async { + loop { + let row = presence(®istry); + if row.active() == 1 && row.wire_tx > 0 && row.wire_rx > 0 { + break row; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("admitted session has direct wire counters"); + drop(connection); + let after = tokio::time::timeout(TIMEOUT, async { + loop { + let row = presence(®istry); + if row.active() == 0 { + break row; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("dropping the client closes its stats context"); + assert!(after.wire_tx >= before.wire_tx && after.wire_rx >= before.wire_rx); + assert_eq!(presence(®istry), after, "a closed source is unregistered"); + } + trigger.start(); + tokio::time::timeout(TIMEOUT, running).await.unwrap().unwrap().unwrap(); +} + +#[tokio::test] +async fn native_sessions_supply_direct_wire_counters() { + wire_sources(false).await; +} + +#[cfg(all(target_os = "linux", feature = "_uring"))] +#[tokio::test] +async fn uring_sessions_supply_direct_wire_counters() { + match moq_uring::Worker::new(Default::default()) { + Ok(_) => (), + Err(moq_uring::Error::Unsupported(reason)) if std::env::var_os("MOQ_STRICT").is_none() => { + eprintln!("skipping io_uring wire stats test: {reason}"); + return; + } + Err(error) => panic!("io_uring worker setup failed: {error}"), + } + wire_sources(true).await; +} diff --git a/rs/moq-stats/benches/producer_tick.rs b/rs/moq-stats/benches/producer_tick.rs index 58087ea0df2..169e5955c33 100644 --- a/rs/moq-stats/benches/producer_tick.rs +++ b/rs/moq-stats/benches/producer_tick.rs @@ -273,6 +273,7 @@ struct Presence { _origin: origin::Producer, _origin_driver: origin::Driver, _sessions: Vec, + _wire: Vec, _held: Vec, } @@ -281,10 +282,15 @@ impl Presence { let registry = stats::Registry::new(stats::Config::new()); let (origin, origin_driver) = origin::Producer::new(origin::Config::default()); let names: Vec = (0..roots).map(|index| format!("room/{index:05}")).collect(); - let sessions = names + let sessions: Vec<_> = names .iter() .map(|root| registry.tier(stats::Tier::default()).session(root.as_str())) .collect(); + // One set of wire counters per connection, independently of held presence tracks. + let wire = sessions + .iter() + .map(|session| session.wire(moq_net::transport::Wire::new())) + .collect(); let mut driver = Driver::new(registry.clone(), origin.clone()).expect("stats broadcast"); driver.tick(); let held = names.iter().take(held).map(|root| driver.hold_presence(root)).collect(); @@ -294,6 +300,7 @@ impl Presence { _origin: origin, _origin_driver: origin_driver, _sessions: sessions, + _wire: wire, _held: held, } } diff --git a/rs/moq-stats/src/aggregate.rs b/rs/moq-stats/src/aggregate.rs index 6cfde68ac2d..f374acb580d 100644 --- a/rs/moq-stats/src/aggregate.rs +++ b/rs/moq-stats/src/aggregate.rs @@ -386,7 +386,9 @@ impl Mergeable for Presence { } fn regressed(&self, prev: &Self) -> bool { - self.sessions_refused.regressed(&prev.sessions_refused) + self.wire_tx < prev.wire_tx + || self.wire_rx < prev.wire_rx + || self.sessions_refused.regressed(&prev.sessions_refused) || self.sessions_failed.regressed(&prev.sessions_failed) || self.requests < prev.requests || self.sessions_started < prev.sessions_started @@ -1266,6 +1268,29 @@ mod tests { assert_eq!(traffic.inner.nodes.len(), 3, "one entry per epoch"); } + #[test] + fn a_wire_only_drop_starts_a_new_presence_segment() { + for received in [false, true] { + let (mut merged, key) = hand_merged::(); + let value = |wire| { + let mut presence = Presence::default(); + presence.sessions_started = 1; + if received { + presence.wire_rx = wire; + } else { + presence.wire_tx = wire; + } + presence + }; + let node = merged.nodes.get_mut(&key).unwrap(); + node.observe(BTreeMap::from([(String::new(), value(10))])); + node.observe(BTreeMap::from([(String::new(), value(3))])); + let combined = &merged.merged()[""]; + assert_eq!(if received { combined.wire_rx } else { combined.wire_tx }, 13); + assert_eq!(combined.active(), 1, "retiring closes the earlier segment"); + } + } + #[test] fn outgoing_epoch_metadata_migrates_without_choosing_a_grace_entry() { let origin = produce_origin(); diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index 5030f4b163a..b2565e6fe3f 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -980,6 +980,22 @@ struct PresenceTrack { name: String, pair: TrackPair, sums: Cumulative, + /// Wire collection starts at the current reading on every hold. Other presence + /// fields keep their existing live-session seeding behavior. + wire_start: HashMap, +} + +impl PresenceTrack { + fn publish(&mut self) { + self.sums.fill(&mut self.pair.frame); + for (tier, presence) in &mut self.pair.frame.entries { + if let Some(&(tx, rx)) = self.wire_start.get(tier) { + presence.wire_tx = presence.wire_tx.saturating_sub(tx); + presence.wire_rx = presence.wire_rx.saturating_sub(rx); + } + } + self.pair.publish(&self.name); + } } /// One broadcast prefix's requested `traffic.json` pair and its per-tier sums, @@ -1270,8 +1286,7 @@ impl GroupPublisher { self.totals.sums.fill(&mut self.totals.pair.frame); self.totals.pair.publish(crate::TOTALS); for track in self.presence.values_mut() { - track.sums.fill(&mut track.pair.frame); - track.pair.publish(&track.name); + track.publish(); } for track in self.prefixes.values_mut() { track.publish(tick); @@ -1391,15 +1406,18 @@ impl GroupPublisher { name, pair, sums: Cumulative::new(), + wire_start: HashMap::new(), }; for &i in &self.session_rows { let entry = &report.sessions[i]; if entry.root == root { track.sums.add(&entry.tier, entry.closed, |sum| sum.add(entry.presence)); + let start = track.wire_start.entry(entry.tier.label().clone()).or_default(); + start.0 += entry.presence.wire_tx; + start.1 += entry.presence.wire_rx; } } - track.sums.fill(&mut track.pair.frame); - track.pair.publish(&track.name); + track.publish(); self.presence.insert(root, track); } @@ -1609,6 +1627,177 @@ mod tests { } } + /// Real protocol relays keep their idle track cache while upstream presence + /// is reclaimed. A re-hold must refresh that cache before exposing any frame. + #[tokio::test(start_paused = true)] + async fn reheld_wire_presence_refreshes_two_relay_caches() { + fn relay() -> (origin::Producer, moq_net::cache::Pool) { + let config = origin::Config::default(); + let pool = config.pool.clone(); + let (origin, driver) = origin::Producer::new(config); + tokio::spawn(moq_net::time::run(driver)); + (origin, pool) + } + async fn pumping(drain: &mut Drain, future: impl std::future::Future) -> T { + let mut future = std::pin::pin!(future); + for _ in 0..1000 { + drain.collect(); + drain.publish(Instant::now()); + if let Some(result) = futures::FutureExt::now_or_never(future.as_mut()) { + return result; + } + tokio::task::yield_now().await; + } + panic!("protocol request did not settle while pumping the stats producer"); + } + async fn link( + upstream: &origin::Producer, + downstream: &origin::Producer, + version: moq_net::Version, + ) -> [moq_net::Session; 2] { + let mut config = moq_net::transport::memory::Config::default(); + config.protocol = Some(version.to_string()); + let (client_transport, server_transport) = moq_net::transport::memory::pair(config); + let client = moq_net::Client::new() + .with_versions(version.into()) + .with_subscriber(downstream.clone().peer()); + let server = moq_net::Server::new() + .with_versions(version.into()) + .with_publisher(upstream.consume().with_hidden(true)); + let client = async { + let (session, driver) = client.connect(moq_clock::now(), client_transport).await.unwrap(); + tokio::spawn(moq_net::time::run(driver)); + session + }; + let server = async { + let (session, driver) = server.accept(moq_clock::now(), server_transport).await.unwrap(); + tokio::spawn(moq_net::time::run(driver)); + session + }; + let (a, b) = tokio::join!(client, server); + [a, b] + } + for version in ["moq-lite-06", "moq-lite-07-wip"].map(|value| value.parse().unwrap()) { + for compressed in [false, true] { + let registry = Registry::new(moq_net::stats::Config::new()); + let origin = produce_origin(); + let mut drain = Drain::new(Task { + registry: registry.clone(), + origin: origin.clone(), + prefix: ".stats".into(), + node: None, + depth: 0, + linger: DEFAULT_LINGER, + interval: Duration::from_secs(1), + }) + .unwrap(); + let (middle, middle_cache) = relay(); + let (edge, edge_cache) = relay(); + let _links = [ + link(&origin, &middle, version).await, + link(&middle, &edge, version).await, + ]; + let session = registry.tier(Tier::default()).session("room"); + let counters = moq_net::transport::Wire::new(); + let _wire = session.wire(counters.clone()); + let store = |n: u64| { + counters.set_sent_bytes(n); + counters.set_received_bytes(n * 2); + }; + store(40); + drain.collect(); + drain.publish(Instant::now()); + let consumer = edge.consume().with_hidden(true); + let mut events = consumer.announced(); + let (route, active) = next_update(&mut events).await.unwrap(); + assert!(active); + assert_eq!(route.prefix.as_str(), ".stats/node"); + let broadcast = consumer.request_broadcast(".stats/node", None).await.unwrap(); + let name = presence_track("room", compressed); + let mut config = moq_json::snapshot::consumer::Config::default(); + if compressed { + config.compression = moq_json::Compression::Deflate; + } + let mut previous = None; + for (reading, delta) in [(40, 11), (1_000, 7)] { + store(reading); + let subscriber = pumping(&mut drain, broadcast.track(&name).unwrap().subscribe(None)) + .await + .unwrap(); + let mut cursor = pumping(&mut drain, broadcast.track(&name).unwrap().subscribe(None)) + .await + .unwrap(); + let group = pumping(&mut drain, cursor.recv_group()).await.unwrap().unwrap(); + if let Some(old) = previous { + assert!(group.sequence > old, "re-hold must have a fresh group identity"); + } + drop(group); + drop(cursor); + let mut reader = moq_json::snapshot::Consumer::::new(subscriber, config.clone()); + let initial = pumping(&mut drain, reader.next()).await.unwrap().unwrap(); + assert_eq!( + (initial[""].wire_tx, initial[""].wire_rx), + (0, 0), + "a cache exposed an earlier hold's bytes" + ); + store(reading + delta); + drain.collect(); + drain.publish(Instant::now()); + let next = pumping(&mut drain, reader.next()).await.unwrap().unwrap(); + assert_eq!((next[""].wire_tx, next[""].wire_rx), (delta, delta * 2)); + if compressed { + // Cross the codec's group budget naturally, leaving a completed + // group cached when unsubscribe aborts the still-open last group. + for extra in 1..=300 { + store(reading + delta + extra); + let next = pumping(&mut drain, reader.next()).await.unwrap().unwrap(); + assert_eq!( + (next[""].wire_tx, next[""].wire_rx), + (delta + extra, (delta + extra) * 2) + ); + } + } + let mut cursor = pumping(&mut drain, broadcast.track(&name).unwrap().subscribe(None)) + .await + .unwrap(); + previous = Some( + pumping(&mut drain, cursor.recv_group()) + .await + .unwrap() + .unwrap() + .sequence, + ); + drop(cursor); + drop(reader); + for _ in 0..100 { + tokio::task::yield_now().await; + drain.collect(); + drain.publish(Instant::now()); + if drain.groups[""].presence.is_empty() { + break; + } + } + assert!( + drain.groups[""].presence.is_empty(), + "relays must release upstream demand while retaining their idle cache" + ); + assert!( + middle_cache.used() > 0 && edge_cache.used() > 0, + "idle cache: middle={} edge={}", + middle_cache.used(), + edge_cache.used() + ); + } + let total = drain.groups[""].totals.sums.tiers[&Tier::default()].current.sessions; + assert_eq!( + (total.wire_tx, total.wire_rx), + if compressed { (1_307, 2_614) } else { (1_007, 2_014) }, + "epoch totals never reset on re-hold" + ); + } + } + } + /// A flavor whose frame outgrows the cache ends with `FrameTooLarge` instead /// of leaving readers on its last snapshot, while its sibling, which DEFLATE /// shrinks under the limit, keeps updating. @@ -2180,6 +2369,46 @@ mod tests { const ACME: &str = ".stats/acme/node/sjc"; const FEED: &str = ".stats/feed/node/sjc"; + #[tokio::test(start_paused = true)] + async fn wire_totals_restart_after_a_group_epoch_ends() { + let (producer, origin, mut events) = grouped_producer(Duration::ZERO); + let registry = producer.registry(); + let session = registry.tier(Tier::default()).session("acme/live"); + let counters = moq_net::transport::Wire::new(); + counters.set_sent_bytes(71); + counters.set_received_bytes(29); + let wire = session.wire(counters); + drive_tick().await; + let [(path, true, old_epoch)] = &take_epochs(&mut events)[..] else { + panic!("expected the wire root's group"); + }; + let broadcast = origin.consume().request_broadcast(path, None).await.expect("resolve"); + let totals = default_totals(&broadcast).await.sessions; + assert_eq!((totals.wire_tx, totals.wire_rx), (71, 29)); + drop(wire); + drop(session); + for _ in 0..3 { + drive_tick().await; + } + assert_eq!(take_epochs(&mut events), [(path.clone(), false, old_epoch.clone())]); + + let session = registry.tier(Tier::default()).session("acme/live"); + let _wire = session.wire(moq_net::transport::Wire::new()); + drive_tick().await; + let [(new_path, true, new_epoch)] = &take_epochs(&mut events)[..] else { + panic!("expected a new wire group epoch"); + }; + assert_eq!(new_path, path); + assert!(new_epoch > old_epoch); + let broadcast = origin + .consume() + .request_broadcast(new_path, None) + .await + .expect("resolve"); + let totals = default_totals(&broadcast).await.sessions; + assert_eq!((totals.wire_tx, totals.wire_rx), (0, 0)); + } + #[tokio::test(start_paused = true)] async fn empty_group_lingers_then_unannounces() { let (producer, origin, mut events) = grouped_producer(Duration::from_secs(10)); diff --git a/rs/moq-tokio/src/quic/driver/connection.rs b/rs/moq-tokio/src/quic/driver/connection.rs index 8de19c76139..fa6c9d954e7 100644 --- a/rs/moq-tokio/src/quic/driver/connection.rs +++ b/rs/moq-tokio/src/quic/driver/connection.rs @@ -296,6 +296,11 @@ impl Future for ConnectionDriver { conn.forward_endpoint_events(); conn.forward_app_events(&self.conn.shared); + // Every datagram sent or received passes through this poll. + let stats = conn.inner.stats(); + self.conn.shared.wire.set_sent_bytes(stats.udp_tx.bytes); + self.conn.shared.wire.set_received_bytes(stats.udp_rx.bytes); + if !conn.inner.is_drained() { if keep_going { // If the connection hasn't processed all tasks, schedule it again @@ -647,6 +652,11 @@ impl Connection { self.0.state.lock("stats").inner.stats() } + /// The wire counters the driver updates as it sends and receives. + pub(crate) fn wire(&self) -> moq_net::transport::Wire { + self.0.shared.wire.clone() + } + /// Current state of the congestion control algorithm, for debugging purposes pub fn congestion_state(&self) -> Box { self.0 @@ -1043,6 +1053,8 @@ pub(crate) struct Shared { connected: Arc, /// Number of live handles that can used to initiate or handle I/O; excludes the driver ref_count: AtomicUsize, + /// Refreshed by every driver poll, so readers never wake the driver to sample. + wire: moq_net::transport::Wire, } pub(crate) struct State { diff --git a/rs/moq-tokio/src/transport.rs b/rs/moq-tokio/src/transport.rs index 2244aa9b8e5..9ca2498336b 100644 --- a/rs/moq-tokio/src/transport.rs +++ b/rs/moq-tokio/src/transport.rs @@ -862,6 +862,10 @@ impl Drop for RecvStream { #[cfg(feature = "quic")] impl Grouped for crate::quic::web::Session { + fn wire(&self) -> Option { + Some(crate::quic::driver::Connection::wire(self)) + } + fn send_group(&self, priority: i64) -> Option { Some(Self::send_group(self, priority)) } diff --git a/rs/moq-tokio/src/transport/grouped.rs b/rs/moq-tokio/src/transport/grouped.rs index 42751e5fcbd..97878edb257 100644 --- a/rs/moq-tokio/src/transport/grouped.rs +++ b/rs/moq-tokio/src/transport/grouped.rs @@ -1,4 +1,4 @@ -//! Send-group hooks for the async backends. +//! Send-group and wire-counter hooks for the async backends. //! //! The traits are `pub` in this private module, so other crates can neither name nor //! implement them: `pub(crate)` is refused (E0446) because the public adapter's @@ -6,10 +6,16 @@ /// A backend session whose send streams may join send groups, as native QUIC's do. /// -/// The async interface has no groups, so moq-net's -/// [`send_group`](moq_net::transport::poll::Session::send_group) reaches a backend through -/// this. A backend without groups implements it empty. +/// The async interface has no groups or wire counters, so moq-net's +/// [`send_group`](moq_net::transport::poll::Session::send_group) and +/// [`wire`](moq_net::transport::poll::Session::wire) reach a backend through this. A +/// backend without either implements it empty. pub trait Grouped: web_transport_trait::Session { + /// The wire counters the backend updates as it sends and receives, or `None` when it has none. + fn wire(&self) -> Option { + None + } + /// Open a send group at `priority`, or `None` when the backend has none. fn send_group(&self, priority: i64) -> Option<::Group> { let _ = priority; diff --git a/rs/moq-tokio/src/transport/owned.rs b/rs/moq-tokio/src/transport/owned.rs index 433022db697..85361972b41 100644 --- a/rs/moq-tokio/src/transport/owned.rs +++ b/rs/moq-tokio/src/transport/owned.rs @@ -95,6 +95,9 @@ impl> poll::Session fn stats(&self) -> impl moq::Stats { Stats(backend::Session::stats(self)) } + fn wire(&self) -> Option { + super::Grouped::wire(&self.session) + } } impl poll::SendStream for super::SendStream { type Error = Error; diff --git a/rs/moq-uring/src/quic/driver/connection.rs b/rs/moq-uring/src/quic/driver/connection.rs index 3f8f063eaae..60a35199fe1 100644 --- a/rs/moq-uring/src/quic/driver/connection.rs +++ b/rs/moq-uring/src/quic/driver/connection.rs @@ -29,6 +29,8 @@ const TRAIN_SEGMENTS: usize = 63; /// waiter without ever holding both. pub(crate) struct Inner { pub(crate) conn: RefCell, + /// Updated by packet ingress and egress, so readers never wake the worker to sample. + pub(crate) wire: moq_net::transport::Wire, pub(crate) state: RefCell, /// The worker driving this connection, which anything layered on it /// (the WebTransport handshake, say) runs on too. @@ -399,6 +401,11 @@ impl Connection { pub fn stats(&self) -> moq_quic::ConnectionStats { self.shared.conn.borrow().stats() } + + /// The wire counters this connection updates as it sends and receives. + pub(crate) fn wire(&self) -> moq_net::transport::Wire { + self.shared.wire.clone() + } } impl Clone for Connection { @@ -427,7 +434,13 @@ pub(crate) fn launch( key: ConnectionHandle, conn: moq_quic::Connection, ) -> (Shared, impl Future + use<>) { + // The handshake's first datagrams arrived before this driver existed. + let stats = conn.stats(); + let wire = moq_net::transport::Wire::new(); + wire.set_sent_bytes(stats.udp_tx.bytes); + wire.set_received_bytes(stats.udp_rx.bytes); let shared = Rc::new(Inner { + wire, conn: RefCell::new(conn), state: RefCell::new(State::new()), owner: owner.clone(), @@ -898,6 +911,7 @@ impl Driver { let Some(transmit) = conn.poll_transmit(moq_clock::Instant::now(), segments, &mut tx) else { return Poll::Pending; }; + self.shared.wire.set_sent_bytes(conn.stats().udp_tx.bytes); // Once closed, moq-quic transmits nothing but the CONNECTION_CLOSE. (transmit, conn.is_closed()) }; diff --git a/rs/moq-uring/src/quic/driver/endpoint.rs b/rs/moq-uring/src/quic/driver/endpoint.rs index 1b45c53b313..fbfc8ad678c 100644 --- a/rs/moq-uring/src/quic/driver/endpoint.rs +++ b/rs/moq-uring/src/quic/driver/endpoint.rs @@ -290,6 +290,7 @@ impl Inner { // we can feed. if let Some(conn) = conn { conn.conn.borrow_mut().handle_event(event); + conn.wire.set_received_bytes(conn.conn.borrow().stats().udp_rx.bytes); if !fed.contains(&key) { fed.push(key); } diff --git a/rs/moq-uring/src/quic/web.rs b/rs/moq-uring/src/quic/web.rs index b688ece4c32..90f49b3f587 100644 --- a/rs/moq-uring/src/quic/web.rs +++ b/rs/moq-uring/src/quic/web.rs @@ -445,6 +445,11 @@ pub struct Session { } impl Session { + /// The wire counters the connection updates as it sends and receives. + pub(crate) fn wire(&self) -> moq_net::transport::Wire { + self.conn.wire() + } + /// Open a send group at `priority`, for streams to join with /// [`SendStream::set_group`]. The HTTP/3 streams stay outside every /// group, so they go first. diff --git a/rs/moq-uring/src/transport.rs b/rs/moq-uring/src/transport.rs index e9e961bec77..150f429fa48 100644 --- a/rs/moq-uring/src/transport.rs +++ b/rs/moq-uring/src/transport.rs @@ -11,6 +11,10 @@ use crate::quic; #[cfg(feature = "quic")] impl Grouped for quic::web::Session { + fn wire(&self) -> Option { + Some(quic::web::Session::wire(self)) + } + fn send_group(&mut self, priority: i64) -> Option { Some(quic::web::Session::send_group(self, priority)) } diff --git a/rs/moq-uring/src/transport/adapter.rs b/rs/moq-uring/src/transport/adapter.rs index 0f4ff4dc4ce..c37beba426f 100644 --- a/rs/moq-uring/src/transport/adapter.rs +++ b/rs/moq-uring/src/transport/adapter.rs @@ -53,10 +53,15 @@ impl moq::Stats for Stats { /// through [`GroupedSend`]. The module is private and the trait is not re-exported, so /// other crates can neither name nor implement it. /// -/// The backend interface has no groups, so moq-net's -/// [`send_group`](poll::Session::send_group) reaches a backend through this. A backend -/// without groups implements it empty. +/// The backend interface has no groups or wire counters, so moq-net's +/// [`send_group`](poll::Session::send_group) and [`wire`](poll::Session::wire) reach a +/// backend through this. A backend without either implements it empty. pub trait Grouped: backend::Session { + /// The wire counters the backend updates as it sends and receives, or `None` when it has none. + fn wire(&self) -> Option { + None + } + /// Open a send group at `priority`, or `None` when the backend has none. fn send_group(&mut self, priority: i64) -> Option<::Group> { let _ = priority; @@ -161,6 +166,9 @@ impl + Clone + 'static> pol fn stats(&self) -> impl moq::Stats { Stats(backend::Session::stats(&self.0)) } + fn wire(&self) -> Option { + Grouped::wire(&self.0) + } } /// An outgoing WebTransport stream adapted to moq-net. pub struct SendStream(S);