From 8f6738c20176129afb00c8bca054946db10836a9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 16:21:17 -0800 Subject: [PATCH 1/5] chore(quest): claim quest/m1/route-mesh-sweep.md Quest-Claim: f0856249f64b3734 From ef52655451512f41b3c2e49d57a72f1d748133bf Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 17:41:54 -0800 Subject: [PATCH 2/5] wip: route mesh sweep --- .github/workflows/nightly.yml | 3 +- doc/bin/relay/http.md | 3 + doc/lib/rs/kio.md | 5 +- quest/m1/README.md | 1 - quest/m1/route-mesh-model.md | 9 +- quest/m1/route-mesh-sweep.md | 31 - quest/m2/cluster-routing/routes/README.md | 1 - rs/kio/src/clock/sim.rs | 31 +- rs/moq-net/src/coding/reader.rs | 15 +- rs/moq-net/src/coding/writer.rs | 6 +- rs/moq-net/src/stats.rs | 53 +- rs/moq-net/tests/integration/main.rs | 1 + rs/moq-net/tests/integration/mesh_sweep.rs | 794 ++++++++++++++++++ rs/moq-net/tests/integration/mesh_withdraw.rs | 20 +- rs/moq-net/tests/integration/support/mod.rs | 25 + rs/moq-relay/src/internal.rs | 14 +- sh/gh/bench-report.py | 4 +- sh/gh/bench-report.test.py | 2 + sh/gh/nightly.sh | 1 + test/justfile | 4 + test/route-sweep.sh | 12 + 21 files changed, 948 insertions(+), 87 deletions(-) delete mode 100644 quest/m1/route-mesh-sweep.md create mode 100644 rs/moq-net/tests/integration/mesh_sweep.rs create mode 100755 test/route-sweep.sh diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index d8ed5cbd0cc..ba1c5f04005 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -340,6 +340,7 @@ jobs: # `rs quic-sim` sweeps moq-quic's simulated workloads from a fresh seed; PRs run a fixed 64. # `rs rung-disable` proves FFI simulcast disable and recovery on a shaped BBR3 uplink. # `test resume-model` sweeps resume against its model from a fresh seed; PRs run a fixed 64. + # `test route-sweep` checks random lite-07 meshes against the route rules from a fresh seed; PRs run a fixed 8. # It also runs the bandwidth probe's acceptance suite, of which PRs run the short cases. # `test drill impaired` runs the relay drills over a freshly seeded shaper; PRs run loopback. # `rs cli-export` runs moq-cli's export process tests, which take tens of seconds of real time. @@ -355,7 +356,7 @@ jobs: # `rs v4l2` streams moq-video's V4L2 capture from the kernel's vivid # virtual camera, a module load no pull request job should make. # `js worklet-matrix` loads `?worklet` output through the bundlers pinned in test/worklet. - recipe: ["gate", "gh test-integration", "rs doctest --workspace", "rs send-depth", "test drill-sensitivity", "rs uring", "test json-properties", "test resume-model", "rs quic-sim", "rs rung-disable", "test drill impaired", "rs cli-export", "rs hls-conformance", "rs quic-stress", "rs audio-devices", "rs multicast", "rs render-sw", "test qmux", "rs v4l2", "js worklet-matrix"] + recipe: ["gate", "gh test-integration", "rs doctest --workspace", "rs send-depth", "test drill-sensitivity", "rs uring", "test json-properties", "test resume-model", "test route-sweep", "rs quic-sim", "rs rung-disable", "test drill impaired", "rs cli-export", "rs hls-conformance", "rs quic-stress", "rs audio-devices", "rs multicast", "rs render-sw", "test qmux", "rs v4l2", "js worklet-matrix"] include: # Miri interprets kio's and moq-net's in-memory unsafe; too slow for the merge gate. - recipe: rs miri diff --git a/doc/bin/relay/http.md b/doc/bin/relay/http.md index 5edf2d4b45c..b289fc6a4aa 100644 --- a/doc/bin/relay/http.md +++ b/doc/bin/relay/http.md @@ -73,6 +73,9 @@ count control-message bytes by `tier` and `class` (`session`, `announce`, `track `probe`, `auth`, `goaway`, `other`) to spot control-heavy sessions. They include message size prefixes but leave out media payload and transport overhead such as QUIC framing, ACKs, and retransmissions. +`moq_relay_control_tx_messages_total` and `moq_relay_control_rx_messages_total` +count the same messages by number, so an announce storm shows as a rate +rather than as bytes. `moq_relay_sessions_connected` is a gauge of connected sessions by `tier` and `state`: `active` if media payload moved through the session, either way, diff --git a/doc/lib/rs/kio.md b/doc/lib/rs/kio.md index 644fa225b06..a288499fc8e 100644 --- a/doc/lib/rs/kio.md +++ b/doc/lib/rs/kio.md @@ -94,8 +94,9 @@ timer is armed panics as a deadlock instead of hanging. `sim::advance()` and Draw jitter and other randomness from `kio::clock::rng()`. Inside the simulator it comes from one seed per run, picked at random or set with `MOQ_SIM_SEED`. A run that panics prints its seed, so rerunning the test with `MOQ_SIM_SEED` set to -that seed replays it bit-exact. Outside the simulator `rng()` is rand's thread -generator. +that seed replays it bit-exact. A sweep that derives each case's seed itself +passes it to `sim::run_seeded(seed, future)` instead, so one setting replays a +case. Outside the simulator `rng()` is rand's thread generator. ## Driving a tokio runtime by hand diff --git a/quest/m1/README.md b/quest/m1/README.md index bfcdf92eb96..5f903e788c3 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -47,7 +47,6 @@ moves what remains of them once their claims merge. - [A screen share publishes only while joined](/quest/m1/room-screen-joined.md) - `@moq/room` publishes no screen share before joining or after leaving - [The TLA+ models run nightly and reviews resync them](/quest/m1/model-checks-nightly.md) - every model config runs in Nightly, open findings must still fail, and review reruns a model whose covered code changed - [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - every ordering on 4-5 relay meshes with two origins and clients, with an announce budget and convergence -- [Seeded lite-07 meshes hold the route rules and an announce budget](/quest/m1/route-mesh-sweep.md) - the route rules and message budget checked on real sessions over random meshes - [A model of a relay chain's failover delivers every group once](/quest/m1/failover-chain-model.md) - route, resume, and subscribe composed across a relay chain whose hops fail - [A reused moq-transport alias refuses its track at once](/quest/m1/ietf-alias-refuse.md) - a SUBSCRIBE_OK on an alias already bound fails its readers and removes its entry without waiting on the peer's FIN acknowledgement - [Unused crates, scripts, and files are deleted](/quest/m1/unused-code.md) - moq-room, the moq-native tombstone, demo/throttle, sh/dart/publish.sh, demo/web's Solid deps, and unpublished packages' release notes go diff --git a/quest/m1/route-mesh-model.md b/quest/m1/route-mesh-model.md index cc0e93fc9b5..d7cfa26df8d 100644 --- a/quest/m1/route-mesh-model.md +++ b/quest/m1/route-mesh-model.md @@ -12,15 +12,18 @@ routing goes quiet after the last change. Decided 2026-11-10 (maintainer, /quest-plan), after the triangle model in [moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447) found count to infinity. Routing, announce spam, and resume failover have caused many -production issues; this and [the lite-07 mesh sweep](/quest/m1/route-mesh-sweep.md) -are routing's confidence gates for finalizing lite-07. +production issues; this and the lite-07 mesh sweep +(`rs/moq-net/tests/integration/mesh_sweep.rs`) are routing's confidence gates +for finalizing lite-07. - Grow `RouteHandoff.tla` (or a sibling spec sharing its operators) to 4 and 5 relays in a line, a ring, and a ring with a chord; two origin identities; and client leaves that advertise nothing but echo every ROUTE_END. - An invariant bounds the ROUTE, batch, and request messages sent between external changes (an origin's claim change, a session open or close) by a - budget derived from the links, so announce spam is a counterexample. + budget derived from the links, so announce spam is a counterexample. The + sweep's `budget` states 8 messages per session per route identity and per + claim; the model states the same one, or changes both. - A liveness property, under weak fairness of delivery: after the last external change, the queues empty and every relay shows what `QuietIsSettled` requires. diff --git a/quest/m1/route-mesh-sweep.md b/quest/m1/route-mesh-sweep.md deleted file mode 100644 index f78238e55ad..00000000000 --- a/quest/m1/route-mesh-sweep.md +++ /dev/null @@ -1,31 +0,0 @@ -# [L] Seeded lite-07 meshes hold the route rules and an announce budget - -## Goal - -A seeded sweep of random lite-07 meshes, real sessions over the memory -transport on a mocked clock, checks the route model's rules on the -implementation at every quiet point and counts announce messages against a -budget. Nightly sweeps fresh seeds, and a failing seed replays bit-exact. - -## Plan - -Decided 2026-11-10 (maintainer, /quest-plan). The model -([mesh route model](/quest/m1/route-mesh-model.md)) checks every ordering on -small meshes; this checks the real stack, sessions included, on larger random -ones. `rs/moq-net/tests/integration/mesh_withdraw.rs` already sweeps 200 -seeded meshes, but on lite-06; [route layer -acceptance](/quest/m2/cluster-routing/routes/acceptance.md) ports specific -fixtures, not a random sweep. - -- Each case: a random mesh of relays (sizes swept) with clients, publishing - and watching broadcasts; random link flaps, relay restarts, origin restarts - under a fresh identity, and client joins and leaves. -- At every quiet point, assert the `verify-route` rules on what each client - and relay sees: truthful, no revival, no false end, no regression, loop-free, - and settled. Count ROUTE and announce messages per external change against - the same budget the model states, and report the counts as a benchmark swept - over mesh size and churn. -- Seeds follow the repo's `MOQ_*_SEED` convention; the merge gate runs a few - fixed seeds only if they stay fast, and Nightly sweeps fresh ones. - -Public API: none. Wire: none. diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index e77823753b9..2a3f0c546a6 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -547,7 +547,6 @@ benchmarks. - [Playwright bundles Chromium's spec exporter](/quest/m2/cluster-routing/routes/playwright-pin.md) - the toolchain's Playwright bundles Chromium 158+, so the route key lane proves the Chromium tab - [A ROUTE_END echo arrives behind the echoer's routes](/quest/m2/cluster-routing/routes/echo-order.md) - no node drops its feasibility distance while a route derived from its own can still arrive, so routes never count to infinity - [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - every ordering on 4-5 relay meshes with two origins and clients, with an announce budget and convergence -- [Seeded lite-07 meshes hold the route rules and an announce budget](/quest/m1/route-mesh-sweep.md) - the route rules and message budget checked on real sessions over random meshes ## Related diff --git a/rs/kio/src/clock/sim.rs b/rs/kio/src/clock/sim.rs index cf63cc2a3e6..9601d239cdf 100644 --- a/rs/kio/src/clock/sim.rs +++ b/rs/kio/src/clock/sim.rs @@ -270,19 +270,18 @@ impl Wake for Flag { } /// Clears the thread's runtime when [`run`] returns or unwinds, printing the seed -/// that replays an unwinding run. +/// that replays an unwinding run when the environment picked it. struct Enter { - seed: u64, + replay: Option, } impl Drop for Enter { fn drop(&mut self) { CURRENT.with(|current| current.borrow_mut().take()); - if std::thread::panicking() { - eprintln!( - "kio::clock::sim run panicked; replay it with MOQ_SIM_SEED={}", - self.seed - ); + if let Some(seed) = self.replay + && std::thread::panicking() + { + eprintln!("kio::clock::sim run panicked; replay it with MOQ_SIM_SEED={seed}"); } } } @@ -298,12 +297,24 @@ impl Drop for Enter { /// exceeds the ready kio poll limit, a detached task panics, or `MOQ_SIM_SEED` is /// not a `u64`. pub fn run(future: F) -> F::Output { - run_seeded(seed_from_env(), future) + let seed = seed_from_env(); + enter(seed, Some(seed), future) +} + +/// [`run`] with [`rng`](crate::clock::rng) and [`seed`](crate::clock::seed) drawn from +/// `seed`, ignoring `MOQ_SIM_SEED`: for a sweep that derives each case's seed itself, +/// so one setting replays a case. +/// +/// # Panics +/// +/// As [`run`] does, except on `MOQ_SIM_SEED`. +pub fn run_seeded(seed: u64, future: F) -> F::Output { + enter(seed, None, future) } -fn run_seeded(seed: u64, future: F) -> F::Output { +fn enter(seed: u64, replay: Option, future: F) -> F::Output { let runtime = Runtime::enter(seed); - let enter = Enter { seed }; + let enter = Enter { replay }; let mut future = pin!(future.into_future()); let mut output = None; diff --git a/rs/moq-net/src/coding/reader.rs b/rs/moq-net/src/coding/reader.rs index 3c8526bbc53..376bd959393 100644 --- a/rs/moq-net/src/coding/reader.rs +++ b/rs/moq-net/src/coding/reader.rs @@ -615,7 +615,17 @@ mod tests { let size = bytes.len() as u64; let (head, tail) = bytes.split_at(2); let counted = || registry.snapshot().control(); - let probes = |n: u64| vec![(Tier::default(), Message::Probe, Totals { tx: 0, rx: size * n })]; + let probes = |n: u64| { + vec![( + Tier::default(), + Message::Probe, + Totals { + rx: size * n, + rx_messages: n, + ..Default::default() + }, + )] + }; let mut reader = queue(&[bytes, head], version).with_stats(&session); let waiter = &kio::Waiter::noop(); @@ -650,8 +660,9 @@ mod tests { Tier::default(), Message::Auth, Totals { - tx: 0, rx: reply.len() as u64, + rx_messages: 1, + ..Default::default() }, ); assert_eq!(counted(), [vec![auth.clone()], probes(2)].concat()); diff --git a/rs/moq-net/src/coding/writer.rs b/rs/moq-net/src/coding/writer.rs index c6de21f4b79..6ad1c962d9a 100644 --- a/rs/moq-net/src/coding/writer.rs +++ b/rs/moq-net/src/coding/writer.rs @@ -502,7 +502,11 @@ mod tests { vec![( Tier::new(crate::Path::from_static("internal")), Message::Probe, - Totals { tx: size, rx: 0 } + Totals { + tx: size, + tx_messages: 1, + ..Default::default() + } )] ); writer diff --git a/rs/moq-net/src/stats.rs b/rs/moq-net/src/stats.rs index 6d7c6fd962c..d5fe43ffcdf 100644 --- a/rs/moq-net/src/stats.rs +++ b/rs/moq-net/src/stats.rs @@ -298,7 +298,8 @@ struct SessionCounters { bytes: Mutex, wire_tx: AtomicU64, wire_rx: AtomicU64, - control: [[AtomicU64; 2]; Message::ALL.len()], + /// Per class: bytes sent and received, then messages sent and received. + control: [[AtomicU64; 4]; Message::ALL.len()], } impl SessionCounters { @@ -337,11 +338,12 @@ impl SessionCounters { (presence, activity, empty) } - fn control(&self, totals: &mut [[u64; 2]; Message::ALL.len()]) { - for (total, counters) in totals.iter_mut().zip(&self.control) { - for (total, counter) in total.iter_mut().zip(counters) { - *total += counter.load(Ordering::Relaxed); - } + fn control(&self, totals: &mut [Totals; Message::ALL.len()]) { + for (total, [tx, rx, tx_messages, rx_messages]) in totals.iter_mut().zip(&self.control) { + total.tx += tx.load(Ordering::Relaxed); + total.rx += rx.load(Ordering::Relaxed); + total.tx_messages += tx_messages.load(Ordering::Relaxed); + total.rx_messages += rx_messages.load(Ordering::Relaxed); } } @@ -916,7 +918,8 @@ impl Role { } } -/// Cumulative bytes sent and received, returned per [`Message`] class by [`Snapshot::control`]. +/// Cumulative bytes and messages sent and received, returned per [`Message`] class by +/// [`Snapshot::control`]. #[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] #[non_exhaustive] pub struct Totals { @@ -924,6 +927,10 @@ pub struct Totals { pub tx: u64, /// Bytes of complete messages decoded and consumed. pub rx: u64, + /// Messages successfully encoded for sending, counted like [`Self::tx`]. + pub tx_messages: u64, + /// Complete messages decoded and consumed, counted like [`Self::rx`]. + pub rx_messages: u64, } /// The bounded class of a successfully encoded or decoded control message. @@ -1003,7 +1010,7 @@ pub struct Snapshot { sessions: HashMap, /// Connected-session activity per tier; read via [`Self::activity`]. activity: HashMap, - control: HashMap, + control: HashMap, } impl Snapshot { @@ -1014,9 +1021,9 @@ impl Snapshot { let mut rows = Vec::new(); for (tier, totals) in &self.control { for class in Message::ALL { - let [tx, rx] = totals[class as usize]; - if tx != 0 || rx != 0 { - rows.push((tier.clone(), class, Totals { tx, rx })); + let totals = totals[class as usize]; + if totals != Totals::default() { + rows.push((tier.clone(), class, totals)); } } } @@ -1734,7 +1741,9 @@ impl Session { let Some(inner) = &self.inner else { return }; let current = inner.current.lock().unwrap_or_else(PoisonError::into_inner); if let Some(counters) = ¤t.presence { - counters.control[class as usize][usize::from(received)].fetch_add(bytes as u64, Ordering::Relaxed); + let counters = &counters.control[class as usize]; + counters[usize::from(received)].fetch_add(bytes as u64, Ordering::Relaxed); + counters[2 + usize::from(received)].fetch_add(1, Ordering::Relaxed); } } @@ -2722,8 +2731,24 @@ mod tests { context.retag(second.clone(), crate::Path::from_static("")); context.control(Message::TrackInfo, 23, true); let expected = vec![ - (second, Message::TrackInfo, Totals { tx: 0, rx: 23 }), - (first, Message::Announce, Totals { tx: 17, rx: 0 }), + ( + second, + Message::TrackInfo, + Totals { + rx: 23, + rx_messages: 1, + ..Default::default() + }, + ), + ( + first, + Message::Announce, + Totals { + tx: 17, + tx_messages: 1, + ..Default::default() + }, + ), ]; assert_eq!(registry.snapshot().control(), expected); drop(context); diff --git a/rs/moq-net/tests/integration/main.rs b/rs/moq-net/tests/integration/main.rs index e87fc5e9185..cef93ec104f 100644 --- a/rs/moq-net/tests/integration/main.rs +++ b/rs/moq-net/tests/integration/main.rs @@ -42,6 +42,7 @@ mod implementation; mod instance; mod late_lower_group; mod lazy_discovery; +mod mesh_sweep; mod mesh_withdraw; mod model; mod mount; diff --git a/rs/moq-net/tests/integration/mesh_sweep.rs b/rs/moq-net/tests/integration/mesh_sweep.rs new file mode 100644 index 00000000000..57114500087 --- /dev/null +++ b/rs/moq-net/tests/integration/mesh_sweep.rs @@ -0,0 +1,794 @@ +#![cfg(feature = "async")] + +//! Seeded random lite-07 meshes, real sessions over the memory transport on the +//! simulated clock, checked against the route rules at every quiet point. +//! +//! Each case builds a random mesh of relays with clients publishing and watching +//! broadcasts, then applies one external change at a time: a link flap, a relay +//! restart, a publisher restarting under a fresh identity, a client joining or +//! leaving, or a broadcast published or withdrawn. Once the mesh goes quiet, every +//! relay and client must have followed the rules the `verify-route` skill states on +//! what it showed, and must now show exactly what it can reach. The announce +//! messages each change cost are counted against [`budget`]. +//! +//! `MOQ_ROUTE_SEED` and `MOQ_ROUTE_CASES` pick the cases; a failure prints the seed +//! and case that replay it bit-exact. Nightly sweeps a fresh seed (`just test +//! route-sweep`), and `benchmark_route_mesh` reports the counts per mesh size and +//! change. + +use std::collections::{BTreeMap, BTreeSet}; +use std::fmt::Write as _; +use std::time::Duration; + +use futures::channel::mpsc; + +use moq_net::{Instance, Timescale, Timestamp, Version, cache, origin, stats, track}; +use support::harness::{MockConnectOptions, MockPair, connect_mock}; + +use crate::support::{self, Rng}; + +const VERSION: &str = "moq-lite-07-wip"; + +/// The only track each broadcast carries, holding one frame. +const TRACK: &str = "t"; + +/// What one case runs. +#[derive(Clone, Copy, Debug)] +struct Shape { + /// Relays in the mesh. + relays: usize, + /// External changes applied, one per quiet point. + changes: usize, +} + +/// One external change. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +enum Change { + /// A link between two relays goes down. + LinkDown, + /// A link that went down comes back. + LinkUp, + /// A relay restarts under a fresh identity, and its links and clients reconnect. + RelayRestart, + /// A publishing client restarts under a fresh identity, attaches to a random relay, + /// and publishes its broadcasts again as new instances. + OriginRestart, + /// A client joins at a random relay. + Join, + /// A client's session closes. + Leave, + /// A client publishes a new broadcast. + Publish, + /// A client withdraws one of its broadcasts. + Withdraw, +} + +impl Change { + const ALL: [Self; 8] = [ + Self::LinkDown, + Self::LinkUp, + Self::RelayRestart, + Self::OriginRestart, + Self::Join, + Self::Leave, + Self::Publish, + Self::Withdraw, + ]; + + fn as_str(self) -> &'static str { + match self { + Self::LinkDown => "link_down", + Self::LinkUp => "link_up", + Self::RelayRestart => "relay_restart", + Self::OriginRestart => "origin_restart", + Self::Join => "join", + Self::Leave => "leave", + Self::Publish => "publish", + Self::Withdraw => "withdraw", + } + } +} + +/// A route event as a node's reader saw it. +#[derive(Clone, Debug)] +struct Seen { + path: String, + kind: origin::RouteKind, + instance: Instance, + available: bool, +} + +/// Watch `routes` from its own task, so every intermediate event is recorded. +fn observe(mut routes: origin::Routes) -> mpsc::UnboundedReceiver { + let (tx, rx) = mpsc::unbounded(); + kio::clock::sim::spawn(async move { + while let Some(event) = routes.next().await { + let seen = Seen { + path: event.announce.prefix.to_string(), + kind: event.kind, + instance: event.announce.route.instance().clone(), + available: event.announce.route.available(), + }; + if tx.unbounded_send(seen).is_err() { + break; + } + } + }); + rx +} + +/// A broadcast as one node currently shows it. +#[derive(Clone, Debug)] +struct Shown { + instance: Instance, + available: bool, +} + +/// What one node's reader has been shown, checked event by event. +struct Reader { + events: mpsc::UnboundedReceiver, + shown: BTreeMap, + /// The newest generation shown per path, which must never go back. + newest: BTreeMap, + /// The instances this reader saw END for. + ended: BTreeSet, +} + +impl Reader { + fn new(origin: &origin::Producer) -> Self { + Self { + events: observe(origin.consume().routes()), + shown: BTreeMap::new(), + newest: BTreeMap::new(), + ended: BTreeSet::new(), + } + } +} + +/// One publication of a broadcast: a path served by one instance. +struct Publication { + path: String, + /// Counts the path's publications, so a newer instance compares higher. + generation: u64, + /// Whether it still serves; once over it never serves again. + live: bool, + /// The client publishing it. + client: usize, +} + +/// A client's own broadcast, held so it stays published. +struct Held { + route: origin::Route, + _track: track::Producer, +} + +struct Relay { + origin: origin::Producer, + reader: Reader, +} + +struct Client { + relay: usize, + origin: origin::Producer, + pair: MockPair, + reader: Reader, + held: BTreeMap, +} + +/// A link between two relays, up or down. +struct Link { + ends: (usize, usize), + latency: Duration, + pair: Option, +} + +/// Every external change's announce-message count, by kind. +type Counts = BTreeMap>; + +struct Mesh { + rng: Rng, + version: Version, + registry: stats::Registry, + next_hop: u64, + relays: Vec, + links: Vec, + clients: Vec>, + publications: BTreeMap, + /// Publications per path so far. + generations: BTreeMap, + /// Broadcasts published so far, so each gets a fresh path. + published: u64, + /// What each change has done, for the failure report. + log: String, +} + +impl Mesh { + async fn new(rng: Rng, relays: usize) -> Self { + let mut mesh = Self { + rng, + version: VERSION.parse().unwrap(), + registry: stats::Registry::new(Default::default()), + next_hop: 0, + relays: Vec::new(), + links: Vec::new(), + clients: Vec::new(), + publications: BTreeMap::new(), + generations: BTreeMap::new(), + published: 0, + log: String::new(), + }; + for _ in 0..relays { + let origin = mesh.origin(true); + let reader = Reader::new(&origin); + mesh.relays.push(Relay { origin, reader }); + } + // A random spanning tree, then up to as many extra links. + let mut linked = BTreeSet::new(); + for b in 1..relays { + let a = mesh.rng.pick(b); + linked.insert((a, b)); + } + for _ in 0..mesh.rng.range(0, relays as u64) { + let a = mesh.rng.pick(relays); + let b = mesh.rng.pick(relays); + if a != b { + linked.insert((a.min(b), a.max(b))); + } + } + for ends in linked { + let latency = Duration::from_millis(mesh.rng.range(1, 150)); + mesh.links.push(Link { + ends, + latency, + pair: None, + }); + let link = mesh.links.len() - 1; + mesh.link_up(link).await; + } + for _ in 0..mesh.rng.range(1, relays as u64) { + mesh.join().await; + } + mesh + } + + /// A fresh origin under a new identity: a proxying relay, or a client. + fn origin(&mut self, proxy: bool) -> origin::Producer { + self.next_hop += 1; + let mut config = origin::Config::new(std::num::NonZeroU64::new(self.next_hop).unwrap()); + // Nothing expires, so no expiry sweep keeps a timer armed and the mesh can go idle. + config.pool = cache::Pool::unbounded(); + let (origin, driver) = origin::Producer::new(config); + support::harness::spawn(driver); + origin.with_proxy(proxy) + } + + fn stats(&self) -> moq_net::stats::Session { + self.registry + .tier(stats::Tier::new(moq_net::Path::from_static("mesh"))) + .session(stats::SessionInfo::new(moq_net::Path::from_static(""))) + } + + fn size(&self) -> Size { + let clients = self.live_clients().len() as u64; + Size { + sessions: self.links.iter().filter(|link| link.pair.is_some()).count() as u64 + clients, + identities: self.relays.len() as u64 + clients, + claims: self.publications.values().filter(|publication| publication.live).count() as u64, + } + } + + /// Announce messages sent so far over every session, both directions. + fn announce_messages(&self) -> u64 { + self.registry + .snapshot() + .control() + .iter() + .filter(|(_, class, _)| *class == stats::Message::Announce) + .map(|(_, _, totals)| totals.tx_messages + totals.rx_messages) + .sum() + } + + async fn link_up(&mut self, link: usize) { + let (a, b) = self.links[link].ends; + let a = self.relays[a].origin.clone().with_link(origin::Link::Peer); + let b = self.relays[b].origin.clone().with_link(origin::Link::Peer); + let mut options = MockConnectOptions::new(self.version); + options.client_publish = Some(a.consume().with_hidden(true)); + options.client_subscribe = Some(a); + options.server_publish = Some(b.consume().with_hidden(true)); + options.server_subscribe = Some(b); + options.latency = self.links[link].latency; + options.server_stats = self.stats(); + self.links[link].pair = Some(connect_mock(options).await); + } + + fn link_down(&mut self, link: usize) { + if let Some(pair) = self.links[link].pair.take() { + close(pair); + } + } + + async fn connect_client(&mut self, relay: usize, origin: &origin::Producer) -> MockPair { + let mut options = MockConnectOptions::new(self.version); + options.client_publish = Some(origin.consume()); + options.client_subscribe = Some(origin.clone()); + options.server_publish = Some(self.relays[relay].origin.consume()); + options.server_subscribe = Some(self.relays[relay].origin.clone()); + options.latency = Duration::from_millis(self.rng.range(1, 50)); + options.server_stats = self.stats(); + connect_mock(options).await + } + + async fn join(&mut self) -> usize { + let relay = self.rng.pick(self.relays.len()); + let origin = self.origin(false); + let reader = Reader::new(&origin); + let pair = self.connect_client(relay, &origin).await; + self.clients.push(Some(Client { + relay, + origin, + pair, + reader, + held: BTreeMap::new(), + })); + let client = self.clients.len() - 1; + for _ in 0..self.rng.range(0, 2) { + self.publish(client); + } + client + } + + /// Publish a new broadcast from `client`. + fn publish(&mut self, client: usize) { + self.published += 1; + let path = format!("room/{}", self.published); + self.announce(client, path); + } + + /// Announce `path` from `client` as a new instance, with one frame on its track. + fn announce(&mut self, client: usize, path: String) { + let held = { + let client = self.clients[client].as_ref().unwrap(); + let route = client + .origin + .route(path.as_str(), origin::RouteOptions::hidden()) + .unwrap(); + let track = route + .broadcast() + .create_track(TRACK, track::Info::default().with_timescale(Timescale::MILLI)) + .unwrap(); + let mut group = track.append_group().unwrap(); + group + .write_frame(Timestamp::ZERO, bytes::Bytes::from(path.clone())) + .unwrap(); + group.finish().unwrap(); + route.announce().unwrap(); + Held { route, _track: track } + }; + let generation = self.generations.entry(path.clone()).or_default(); + *generation += 1; + self.publications.insert( + held.route.instance(), + Publication { + path: path.clone(), + generation: *generation, + live: true, + client, + }, + ); + self.clients[client].as_mut().unwrap().held.insert(path, held); + } + + /// Mark every publication `held` serves as over. + fn over(&mut self, held: &BTreeMap) { + for held in held.values() { + self.publications.get_mut(&held.route.instance()).unwrap().live = false; + } + } + + fn live_clients(&self) -> Vec { + (0..self.clients.len()).filter(|&c| self.clients[c].is_some()).collect() + } + + /// Apply one random external change, returning which, or `None` when the drawn + /// kind has nothing to act on. + async fn change(&mut self) -> Option { + let change = Change::ALL[self.rng.pick(Change::ALL.len())]; + match change { + Change::LinkDown | Change::LinkUp => { + let up = change == Change::LinkUp; + let links: Vec = (0..self.links.len()) + .filter(|&l| self.links[l].pair.is_none() == up) + .collect(); + let link = *links.get(self.rng.pick(links.len().max(1)))?; + let _ = writeln!(self.log, "{change:?} {:?}", self.links[link].ends); + match up { + true => self.link_up(link).await, + false => self.link_down(link), + } + } + Change::RelayRestart => { + let relay = self.rng.pick(self.relays.len()); + let _ = writeln!(self.log, "{change:?} {relay}"); + let touching: Vec = (0..self.links.len()) + .filter(|&l| { + let (a, b) = self.links[l].ends; + (a == relay || b == relay) && self.links[l].pair.is_some() + }) + .collect(); + for &link in &touching { + self.link_down(link); + } + let attached: Vec = (self.live_clients().into_iter()) + .filter(|&c| self.clients[c].as_ref().unwrap().relay == relay) + .collect(); + let mut held = Vec::new(); + for &client in &attached { + let client = self.clients[client].take().unwrap(); + close(client.pair); + held.push((client.origin, client.reader, client.held)); + } + let origin = self.origin(true); + let reader = Reader::new(&origin); + self.relays[relay] = Relay { origin, reader }; + for link in touching { + self.link_up(link).await; + } + for (client, (origin, reader, held)) in attached.into_iter().zip(held) { + let pair = self.connect_client(relay, &origin).await; + self.clients[client] = Some(Client { + relay, + origin, + pair, + reader, + held, + }); + } + } + Change::OriginRestart => { + let publishers: Vec = (self.live_clients().into_iter()) + .filter(|&c| !self.clients[c].as_ref().unwrap().held.is_empty()) + .collect(); + let client = *publishers.get(self.rng.pick(publishers.len().max(1)))?; + let old = self.clients[client].take().unwrap(); + close(old.pair); + self.over(&old.held); + let paths: Vec = old.held.keys().cloned().collect(); + drop(old.held); + drop(old.origin); + let relay = self.rng.pick(self.relays.len()); + let _ = writeln!(self.log, "{change:?} client {client} to relay {relay}: {paths:?}"); + let origin = self.origin(false); + let reader = Reader::new(&origin); + let pair = self.connect_client(relay, &origin).await; + self.clients[client] = Some(Client { + relay, + origin, + pair, + reader, + held: BTreeMap::new(), + }); + for path in paths { + self.announce(client, path); + } + } + Change::Join => { + let client = self.join().await; + let relay = self.clients[client].as_ref().unwrap().relay; + let _ = writeln!(self.log, "{change:?} client {client} at relay {relay}"); + } + Change::Leave => { + let clients = self.live_clients(); + let client = *clients.get(self.rng.pick(clients.len().max(1)))?; + let _ = writeln!(self.log, "{change:?} client {client}"); + let old = self.clients[client].take().unwrap(); + close(old.pair); + self.over(&old.held); + } + Change::Publish => { + let clients = self.live_clients(); + let client = *clients.get(self.rng.pick(clients.len().max(1)))?; + self.publish(client); + let _ = writeln!(self.log, "{change:?} client {client}: room/{}", self.published); + } + Change::Withdraw => { + let publishers: Vec = (self.live_clients().into_iter()) + .filter(|&c| !self.clients[c].as_ref().unwrap().held.is_empty()) + .collect(); + let client = *publishers.get(self.rng.pick(publishers.len().max(1)))?; + let held = &self.clients[client].as_ref().unwrap().held; + let path = held.keys().nth(self.rng.pick(held.len())).unwrap().clone(); + let _ = writeln!(self.log, "{change:?} client {client}: {path}"); + let held = self.clients[client].as_mut().unwrap().held.remove(&path).unwrap(); + self.over(&BTreeMap::from([(path, held)])); + } + } + Some(change) + } + + /// The relays each relay reaches over links that are up. + fn components(&self) -> Vec { + let mut component: Vec = (0..self.relays.len()).collect(); + fn find(component: &mut [usize], x: usize) -> usize { + let mut root = x; + while component[root] != root { + root = component[root]; + } + component[x] = root; + root + } + for link in self.links.iter().filter(|link| link.pair.is_some()) { + let (a, b) = link.ends; + let (a, b) = (find(&mut component, a), find(&mut component, b)); + component[a] = b; + } + (0..self.relays.len()).map(|r| find(&mut component, r)).collect() + } + + /// Check every reader's events since the last quiet point, then that each shows + /// exactly the live broadcasts it reaches. + fn check(&mut self) -> Result<(), String> { + let components = self.components(); + let mut failures = Vec::new(); + let publications = &self.publications; + let attached: BTreeMap = (self.clients.iter().enumerate()) + .filter_map(|(c, client)| Some((c, client.as_ref()?.relay))) + .collect(); + let mut readers: Vec<(String, Option, &mut Reader)> = Vec::new(); + for (r, relay) in self.relays.iter_mut().enumerate() { + readers.push((format!("relay {r}"), Some(components[r]), &mut relay.reader)); + } + for (c, client) in self.clients.iter_mut().enumerate() { + if let Some(client) = client { + readers.push((format!("client {c}"), Some(components[client.relay]), &mut client.reader)); + } + } + + for (name, component, reader) in readers { + while let Ok(seen) = reader.events.try_recv() { + if let Err(err) = apply(reader, publications, &seen) { + failures.push(format!("{name}: {err}: {seen:?}")); + } + } + // Settled: exactly the live broadcasts whose publisher this node reaches. + let expected: BTreeMap<&str, &Instance> = (publications.iter()) + .filter(|(_, publication)| publication.live) + .filter(|(_, publication)| { + let relay = attached[&publication.client]; + component == Some(components[relay]) + }) + .map(|(instance, publication)| (publication.path.as_str(), instance)) + .collect(); + let shown: BTreeMap<&str, &Instance> = (reader.shown.iter()) + .map(|(path, shown)| (path.as_str(), &shown.instance)) + .collect(); + if shown != expected { + failures.push(format!("{name}: not settled: shows {shown:?}, reaches {expected:?}")); + } + for (path, shown) in &reader.shown { + if !shown.available { + failures.push(format!("{name}: {path} still shown unavailable")); + } + } + } + match failures.is_empty() { + true => Ok(()), + false => Err(failures.join("\n")), + } + } + + /// Read the frame on one shown broadcast from each client, through its relay. + async fn read(&mut self) -> Result<(), String> { + let mut failures = Vec::new(); + for c in self.live_clients() { + let client = self.clients[c].as_ref().unwrap(); + let remote: Vec<(&String, &Shown)> = (client.reader.shown.iter()) + .filter(|(path, _)| !client.held.contains_key(*path)) + .collect(); + let Some(&(path, shown)) = remote.get(self.rng.pick(remote.len().max(1))) else { + continue; + }; + let read = async { + let broadcast = (client.origin.consume()) + .request(path.as_str(), shown.instance.clone()) + .await + .map_err(|err| format!("request: {err}"))?; + let mut track = (broadcast.track(TRACK).map_err(|err| format!("track: {err}"))?) + .subscribe(None) + .await + .map_err(|err| format!("subscribe: {err}"))?; + let mut group = (track.recv_group().await) + .map_err(|err| format!("group: {err}"))? + .ok_or("no group")?; + let frame = (group.read_frame().await) + .map_err(|err| format!("frame: {err}"))? + .ok_or("no frame")?; + match frame.payload == path.as_bytes() { + true => Ok(()), + false => Err(format!("read {:?}", frame.payload)), + } + }; + match kio::clock::timeout(Duration::from_secs(30), read).await { + Ok(Ok(())) => {} + Ok(Err(err)) => failures.push(format!("client {c} reading {path}: {err}")), + Err(_) => failures.push(format!("client {c} reading {path}: no frame in 30 s")), + } + } + match failures.is_empty() { + true => Ok(()), + false => Err(failures.join("\n")), + } + } +} + +/// Close both ends of a session, as a lost connection does. +fn close(pair: MockPair) { + pair.client.abort(moq_net::Error::Cancel); + pair.server.abort(moq_net::Error::Cancel); +} + +/// Apply one event to what `reader` shows, checking the per-event rules. +fn apply(reader: &mut Reader, publications: &BTreeMap, seen: &Seen) -> Result<(), String> { + use origin::RouteKind::*; + + // Truthful: every instance shown is one some client published at that path. + let publication = publications.get(&seen.instance).ok_or("an instance nobody published")?; + if publication.path != seen.path { + return Err(format!("the instance of {}", publication.path)); + } + let current = reader.shown.get(&seen.path); + match (seen.kind, current) { + (Start, Some(_)) => return Err("START for a shown broadcast".into()), + (Update | Restart | End | Offline, None) => return Err("an event for a broadcast not shown".into()), + (Update | End | Offline, Some(shown)) if shown.instance != seen.instance => { + return Err(format!("the event names another instance than {:?}", shown.instance)); + } + (Restart, Some(shown)) if shown.instance == seen.instance => { + return Err("RESTART to the same instance".into()); + } + _ => {} + } + match seen.kind { + Start | Update | Restart => { + // NoRevival: an instance this reader saw end never starts again. + if reader.ended.contains(&seen.instance) { + return Err("revived after END".into()); + } + // NoRegression: a reader never goes back to an older publication. + let newest = reader.newest.entry(seen.path.clone()).or_default(); + if publication.generation < *newest { + return Err(format!("generation {} after {newest}", publication.generation)); + } + *newest = publication.generation; + reader.shown.insert( + seen.path.clone(), + Shown { + instance: seen.instance.clone(), + available: seen.available, + }, + ); + } + End => { + // NoFalseEnd: END only once the publication stopped. + if publication.live { + return Err("END while its publisher still serves it".into()); + } + reader.ended.insert(seen.instance.clone()); + reader.shown.remove(&seen.path); + } + Offline => { + reader.shown.remove(&seen.path); + } + } + Ok(()) +} + +/// What a change's announce messages scale with: sessions up, route identities (each +/// relay and client origin), and live broadcasts. +#[derive(Clone, Copy, Debug)] +struct Size { + sessions: u64, + identities: u64, + claims: u64, +} + +/// The most announce messages one change may cost, from the mesh's size before and +/// after it: each session carries at most 4 messages each way per route identity and +/// per claim. Converging takes one advertisement of each route over each session; +/// path hunting or counting to infinity takes one per stale path or hop, which grows +/// past any such constant. +fn budget(before: Size, after: Size) -> u64 { + let sessions = before.sessions.max(after.sessions); + let identities = before.identities.max(after.identities); + let claims = before.claims.max(after.claims); + 8 * sessions * (identities + claims) +} + +/// What one case found: each change's announce-message count, or the failure. +async fn run_case(rng: Rng, shape: Shape) -> Result { + let mut mesh = Mesh::new(rng, shape.relays).await; + kio::clock::sim::idle().await; + mesh.check().map_err(|err| format!("after building the mesh:\n{err}"))?; + let mut counts = Counts::new(); + for _ in 0..shape.changes { + let before = (mesh.announce_messages(), mesh.size()); + let Some(change) = mesh.change().await else { + continue; + }; + kio::clock::sim::idle().await; + let sent = mesh.announce_messages() - before.0; + counts.entry(change).or_default().push(sent); + let budget = budget(before.1, mesh.size()); + let result = match sent <= budget { + true => mesh.check(), + false => Err(format!("{sent} announce messages, over the budget of {budget}")), + }; + let result = match result { + Ok(()) => mesh.read().await, + err => err, + }; + if let Err(err) = result { + return Err(format!("{}\n{err}", mesh.log)); + } + // The reads' own requests settle before the next change is counted. + kio::clock::sim::idle().await; + } + Ok(counts) +} + +/// The case `index` of a sweep from `seed`: its own rng and simulator seed, and a shape. +fn case(seed: u64, index: u64) -> (u64, Shape) { + let case_seed = Rng(seed ^ index.wrapping_mul(0xA24B_AED4_963E_E407)).next(); + let mut rng = Rng(case_seed); + let shape = Shape { + relays: rng.range(2, 10) as usize, + changes: rng.range(5, 30) as usize, + }; + (case_seed, shape) +} + +fn setting(name: &str, fallback: u64) -> u64 { + std::env::var(name).map_or(fallback, |value| value.parse().expect(name)) +} + +/// Runs the cases `MOQ_ROUTE_SEED` and `MOQ_ROUTE_CASES` pick. +#[test] +fn seeded_meshes_hold_the_route_rules() { + let seed = setting("MOQ_ROUTE_SEED", 0x6d6f71); + let cases = setting("MOQ_ROUTE_CASES", 8); + let first = setting("MOQ_ROUTE_CASE", 0); + eprintln!("MOQ_ROUTE_SEED={seed} MOQ_ROUTE_CASES={cases}"); + for index in first..first + cases { + let (case_seed, shape) = case(seed, index); + // A mesh that never goes quiet hangs rather than fails; the last of these names it. + eprintln!("MOQ_ROUTE_CASE={index} {shape:?}"); + let result = kio::clock::sim::run_seeded(case_seed, run_case(Rng(case_seed), shape)); + if let Err(err) = result { + panic!("{shape:?} failed; replay it with MOQ_ROUTE_SEED={seed} MOQ_ROUTE_CASE={index} MOQ_ROUTE_CASES=1\n{err}"); + } + } +} + +/// Announce messages per external change over fixed seeds, by mesh size and change. +#[test] +#[ignore = "a benchmark; Nightly reports it"] +fn benchmark_route_mesh() { + println!("relays,change,mean_messages,max_messages"); + for relays in [3, 6, 12, 24] { + let mut all = Counts::new(); + for index in 0..16 { + let (case_seed, _) = case(0x6d6f71, index); + let shape = Shape { relays, changes: 24 }; + let counts = kio::clock::sim::run_seeded(case_seed, run_case(Rng(case_seed), shape)) + .unwrap_or_else(|err| panic!("{shape:?} case {index}: {err}")); + for (change, sent) in counts { + all.entry(change).or_default().extend(sent); + } + } + for (change, sent) in all { + let mean = sent.iter().sum::() as f64 / sent.len() as f64; + let max = sent.iter().max().unwrap(); + println!("{relays},{},{mean:.1},{max}", change.as_str()); + } + } +} diff --git a/rs/moq-net/tests/integration/mesh_withdraw.rs b/rs/moq-net/tests/integration/mesh_withdraw.rs index c0d665f2687..b1c3d775ecc 100644 --- a/rs/moq-net/tests/integration/mesh_withdraw.rs +++ b/rs/moq-net/tests/integration/mesh_withdraw.rs @@ -15,7 +15,7 @@ use futures::channel::mpsc; use moq_net::{Version, cache, origin}; use support::harness::{MockConnectOptions, MockPair, connect_mock, peer, peer_with_latency}; -use support::{Kind, drain, watch}; +use support::{Kind, Rng, drain, watch}; fn produce_origin(hop: u64) -> origin::Producer { produce(hop, origin::Config::default().update_hold) @@ -529,24 +529,6 @@ async fn assert_capped_withdrawal(edges: &[Link], publisher: usize, watcher: usi } } -/// splitmix64, so every platform builds the same meshes. -struct Rng(u64); - -impl Rng { - fn next(&mut self) -> u64 { - self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); - let mut z = self.0; - z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); - z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); - z ^ (z >> 31) - } - - /// A value in `lo..=hi`. - fn range(&mut self, lo: u64, hi: u64) -> u64 { - lo + self.next() % (hi - lo + 1) - } -} - /// A connected mesh of 8 to 17 relays (a random spanning tree plus up to as /// many extra links), each link 1 to 150 ms, and the relay publishing on it. fn random_mesh(seed: u64) -> (Vec, usize) { diff --git a/rs/moq-net/tests/integration/support/mod.rs b/rs/moq-net/tests/integration/support/mod.rs index 5cbad38e63f..e354ec2a85d 100644 --- a/rs/moq-net/tests/integration/support/mod.rs +++ b/rs/moq-net/tests/integration/support/mod.rs @@ -137,3 +137,28 @@ impl Draining for goaway::Consumer { pub fn every_version() -> Vec { moq_net::Version::ALL.to_vec() } + +/// splitmix64, so every platform builds the same seeded cases. +#[allow(dead_code, reason = "shared by several test binaries, each using only part of it")] +pub struct Rng(pub u64); + +#[allow(dead_code, reason = "shared by several test binaries, each using only part of it")] +impl Rng { + pub fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + + /// A value in `lo..=hi`. + pub fn range(&mut self, lo: u64, hi: u64) -> u64 { + lo + self.next() % (hi - lo + 1) + } + + /// An index into a slice of `len` items. + pub fn pick(&mut self, len: usize) -> usize { + (self.next() % len as u64) as usize + } +} diff --git a/rs/moq-relay/src/internal.rs b/rs/moq-relay/src/internal.rs index af890edbce1..4c45957b0d4 100644 --- a/rs/moq-relay/src/internal.rs +++ b/rs/moq-relay/src/internal.rs @@ -762,7 +762,7 @@ impl Scrape { Ok(()) } - /// The per-tier session counters, plus control bytes split by message class. + /// The per-tier session counters, plus control bytes and messages split by message class. fn encode_sessions(&self, encoder: &mut DescriptorEncoder) -> std::fmt::Result { use moq_net::stats::{Presence, Totals}; @@ -836,6 +836,16 @@ impl Scrape { "Control-message bytes decoded, including codec prefixes; excludes transport overhead.", |b| b.rx, ), + ( + "moq_relay_control_tx_messages", + "Control messages encoded for sending.", + |b| b.tx_messages, + ), + ( + "moq_relay_control_rx_messages", + "Control messages decoded.", + |b| b.rx_messages, + ), ] { let rows = control .iter() @@ -2083,7 +2093,9 @@ mod tests { # TYPE moq_relay_announces_started_total counter # TYPE moq_relay_bytes_total counter # TYPE moq_relay_control_rx_bytes_total counter +# TYPE moq_relay_control_rx_messages_total counter # TYPE moq_relay_control_tx_bytes_total counter +# TYPE moq_relay_control_tx_messages_total counter # TYPE moq_relay_datagrams_total counter # TYPE moq_relay_draining_sessions gauge # TYPE moq_relay_fetches_total counter diff --git a/sh/gh/bench-report.py b/sh/gh/bench-report.py index 41cfb85b5c2..e4c52efe55d 100644 --- a/sh/gh/bench-report.py +++ b/sh/gh/bench-report.py @@ -16,7 +16,7 @@ from urllib.parse import quote BIGGER = {"MB/s", "MiB/s", "B/s", "ops/s", "frames/s", "frames"} -SMALLER = {"ns", "us", "µs", "ms", "s", "ms/MB", "x", "bytes", "B", "MiB", "packets", "streams", "instructions"} +SMALLER = {"ns", "us", "µs", "ms", "s", "ms/MB", "x", "bytes", "B", "MiB", "packets", "streams", "instructions", "messages"} ROOT = Path(__file__).resolve().parents[2] @@ -150,6 +150,8 @@ def measurement_unit(column): return "MiB/s" if column.endswith("_packets"): return "packets" + if column.endswith("_messages"): + return "messages" for unit in ("ns", "us", "ms"): if re.search(f"(^|_){unit}($|_per_)", column): return unit diff --git a/sh/gh/bench-report.test.py b/sh/gh/bench-report.test.py index 18436fd0a0c..ea782a35785 100644 --- a/sh/gh/bench-report.test.py +++ b/sh/gh/bench-report.test.py @@ -63,6 +63,7 @@ def test_measured_columns(self): "peak_bytes", "mib_per_s", "overflowed_packets", + "max_messages", "payload_bytes", "bytes", "frames_per_chunk", @@ -81,6 +82,7 @@ def test_measured_columns(self): "peak_bytes": "bytes", "mib_per_s": "MiB/s", "overflowed_packets": "packets", + "max_messages": "messages", # Workload dimensions, as the JS benchmarks name them. "payload_bytes": None, "bytes": None, diff --git a/sh/gh/nightly.sh b/sh/gh/nightly.sh index b79053c4e52..edd3ff95e57 100755 --- a/sh/gh/nightly.sh +++ b/sh/gh/nightly.sh @@ -101,6 +101,7 @@ case "$1" in nextest just rs test -p moq-tokio --release -E 'test(benchmark_receive_windows)' --run-ignored only --success-output immediate nextest just rs test -p moq-uring --release -E 'test(benchmark_socket_queues)' --run-ignored only --success-output immediate nextest just rs test -p moq-tokio --release -E 'test(benchmark_datagram_delivery)' --run-ignored only --success-output immediate + nextest just rs test -p moq-net --release --features async --test integration -E 'test(=mesh_sweep::benchmark_route_mesh)' --run-ignored only --success-output immediate # One build for the crates whose benchmarks need no features of their own. nextest just rs test --release --no-fail-fast -p moq-relay -p moq-c -p moq-mux -p moq-cli \ -E 'test(benchmark_stun_sources) | test(benchmark_handles) | test(benchmark_damage_pid_fanout) | test(benchmark_pes_staging) | test(benchmark_stream_import)' \ diff --git a/test/justfile b/test/justfile index d7a8b4deaaf..756594e0166 100644 --- a/test/justfile +++ b/test/justfile @@ -294,6 +294,10 @@ json-properties: resume-model: bash ../sh/local.sh bash resume-model.sh +# Check random lite-07 meshes against the route rules from a fresh replayable seed. +route-sweep: + bash ../sh/local.sh bash route-sweep.sh + # The TypeScript qmux peer as client and server against the in-tree Rust qmux # interop examples, per draft. The manifest path names the in-tree crate, which # shares its package name with the 0.6 release its tests interoperate with. diff --git a/test/route-sweep.sh b/test/route-sweep.sh new file mode 100755 index 00000000000..9c9d12c718a --- /dev/null +++ b/test/route-sweep.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +# Nightly sweeps fresh meshes; the printed settings replay a failing case bit-exact. +set -euo pipefail +cd "$(git rev-parse --show-toplevel)" + +MOQ_ROUTE_SEED=${MOQ_ROUTE_SEED:-$(python3 -c 'import secrets; print(secrets.randbits(31))')} +MOQ_ROUTE_CASES=${MOQ_ROUTE_CASES:-500} +export MOQ_ROUTE_SEED MOQ_ROUTE_CASES +echo "Route sweep: MOQ_ROUTE_SEED=$MOQ_ROUTE_SEED MOQ_ROUTE_CASES=$MOQ_ROUTE_CASES" +# Release keeps the sweep inside nextest's hang budget; the checks stay on so it still catches what a debug run would. +export CARGO_PROFILE_RELEASE_DEBUG_ASSERTIONS=true CARGO_PROFILE_RELEASE_OVERFLOW_CHECKS=true +just rs test -p moq-net --release --features async --test integration -E 'test(=mesh_sweep::seeded_meshes_hold_the_route_rules)' From 1ca193696613ebd427c883b4c1f9b9692cbfb6ae Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 17:51:21 -0800 Subject: [PATCH 3/5] wip: fmt --- rs/moq-net/tests/integration/mesh_sweep.rs | 16 +++++++++++++--- rs/moq-relay/src/internal.rs | 8 +++----- sh/gh/bench-report.py | 17 ++++++++++++++++- 3 files changed, 32 insertions(+), 9 deletions(-) diff --git a/rs/moq-net/tests/integration/mesh_sweep.rs b/rs/moq-net/tests/integration/mesh_sweep.rs index 57114500087..b8fde25b825 100644 --- a/rs/moq-net/tests/integration/mesh_sweep.rs +++ b/rs/moq-net/tests/integration/mesh_sweep.rs @@ -273,7 +273,11 @@ impl Mesh { Size { sessions: self.links.iter().filter(|link| link.pair.is_some()).count() as u64 + clients, identities: self.relays.len() as u64 + clients, - claims: self.publications.values().filter(|publication| publication.live).count() as u64, + claims: self + .publications + .values() + .filter(|publication| publication.live) + .count() as u64, } } @@ -540,7 +544,11 @@ impl Mesh { } for (c, client) in self.clients.iter_mut().enumerate() { if let Some(client) = client { - readers.push((format!("client {c}"), Some(components[client.relay]), &mut client.reader)); + readers.push(( + format!("client {c}"), + Some(components[client.relay]), + &mut client.reader, + )); } } @@ -764,7 +772,9 @@ fn seeded_meshes_hold_the_route_rules() { eprintln!("MOQ_ROUTE_CASE={index} {shape:?}"); let result = kio::clock::sim::run_seeded(case_seed, run_case(Rng(case_seed), shape)); if let Err(err) = result { - panic!("{shape:?} failed; replay it with MOQ_ROUTE_SEED={seed} MOQ_ROUTE_CASE={index} MOQ_ROUTE_CASES=1\n{err}"); + panic!( + "{shape:?} failed; replay it with MOQ_ROUTE_SEED={seed} MOQ_ROUTE_CASE={index} MOQ_ROUTE_CASES=1\n{err}" + ); } } } diff --git a/rs/moq-relay/src/internal.rs b/rs/moq-relay/src/internal.rs index 4c45957b0d4..1c168633a48 100644 --- a/rs/moq-relay/src/internal.rs +++ b/rs/moq-relay/src/internal.rs @@ -841,11 +841,9 @@ impl Scrape { "Control messages encoded for sending.", |b| b.tx_messages, ), - ( - "moq_relay_control_rx_messages", - "Control messages decoded.", - |b| b.rx_messages, - ), + ("moq_relay_control_rx_messages", "Control messages decoded.", |b| { + b.rx_messages + }), ] { let rows = control .iter() diff --git a/sh/gh/bench-report.py b/sh/gh/bench-report.py index e4c52efe55d..dfa6cba34ef 100644 --- a/sh/gh/bench-report.py +++ b/sh/gh/bench-report.py @@ -16,7 +16,22 @@ from urllib.parse import quote BIGGER = {"MB/s", "MiB/s", "B/s", "ops/s", "frames/s", "frames"} -SMALLER = {"ns", "us", "µs", "ms", "s", "ms/MB", "x", "bytes", "B", "MiB", "packets", "streams", "instructions", "messages"} +SMALLER = { + "ns", + "us", + "µs", + "ms", + "s", + "ms/MB", + "x", + "bytes", + "B", + "MiB", + "packets", + "streams", + "instructions", + "messages", +} ROOT = Path(__file__).resolve().parents[2] From d11034d8b57cdad1ffa01e67a907f36dc8206f45 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 18:10:56 -0800 Subject: [PATCH 4/5] test(net): seeded lite-07 meshes hold the route rules and an announce budget A seeded sweep of random lite-07 meshes over real sessions checks every relay's and client's route events against the route rules at each quiet point, reads through each route, and counts announce messages per change against a budget. stats::Totals counts messages beside bytes, exported by the relay; kio::clock::sim::run_seeded replays a case from one seed. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/tests/integration/mesh_sweep.rs | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/rs/moq-net/tests/integration/mesh_sweep.rs b/rs/moq-net/tests/integration/mesh_sweep.rs index b8fde25b825..e50a2bea0ca 100644 --- a/rs/moq-net/tests/integration/mesh_sweep.rs +++ b/rs/moq-net/tests/integration/mesh_sweep.rs @@ -538,17 +538,13 @@ impl Mesh { let attached: BTreeMap = (self.clients.iter().enumerate()) .filter_map(|(c, client)| Some((c, client.as_ref()?.relay))) .collect(); - let mut readers: Vec<(String, Option, &mut Reader)> = Vec::new(); + let mut readers: Vec<(String, usize, &mut Reader)> = Vec::new(); for (r, relay) in self.relays.iter_mut().enumerate() { - readers.push((format!("relay {r}"), Some(components[r]), &mut relay.reader)); + readers.push((format!("relay {r}"), components[r], &mut relay.reader)); } for (c, client) in self.clients.iter_mut().enumerate() { if let Some(client) = client { - readers.push(( - format!("client {c}"), - Some(components[client.relay]), - &mut client.reader, - )); + readers.push((format!("client {c}"), components[client.relay], &mut client.reader)); } } @@ -563,7 +559,7 @@ impl Mesh { .filter(|(_, publication)| publication.live) .filter(|(_, publication)| { let relay = attached[&publication.client]; - component == Some(components[relay]) + component == components[relay] }) .map(|(instance, publication)| (publication.path.as_str(), instance)) .collect(); From 1e4f6dde677166dfcad9a265925ce0a5b7211629 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 06:58:32 -0800 Subject: [PATCH 5/5] docs: the bench guide lists the messages unit, and a quest stops linking the finished sweep Co-Authored-By: Claude --- bench/README.md | 6 +++--- quest/m1/route-seqno-bumps.md | 3 +-- rs/kio/src/clock/sim.rs | 2 +- 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/bench/README.md b/bench/README.md index 690385e783c..32260c5f47f 100644 --- a/bench/README.md +++ b/bench/README.md @@ -839,8 +839,8 @@ The table includes these measurements: - The CSV tables the JS benchmarks and the Rust benchmark tests print. A header line names the columns; the time columns (`ns`, `us`, `ms`, and their per-operation variants such as `ns_per_frame` or `plain_ns_per_read`), - `size_bytes` and `peak_bytes`, `mib_per_s`, and `*_packets` columns are - measurements, and every other column is a workload dimension in each series + `size_bytes` and `peak_bytes`, `mib_per_s`, and `*_packets` and `*_messages` + columns are measurements, and every other column is a workload dimension in each series name, so print outcomes only in measured columns. The Rust benchmark tests (`just gh nightly rust-test`) run in release builds, one at a time, under nextest's `bench` profile, which keeps each test's output in a JUnit report; @@ -912,7 +912,7 @@ A new source uploads a `bench-*` artifact containing `report.json`, a nonempty JSON array of `{ "name": "case/dimensions", "unit": "ns", "value": 123 }`, and `outcome.json` containing `{ "status": "success" }`. Names must be unique within a source; dimensions belong in the name. Supported units are `ns`, `us`, -`µs`, `ms`, `s`, `ms/MB`, `x`, `bytes`, `B`, `MiB`, `packets`, `streams`, +`µs`, `ms`, `s`, `ms/MB`, `x`, `bytes`, `B`, `MiB`, `packets`, `messages`, `streams`, `instructions`, `MB/s`, `MiB/s`, `B/s`, `ops/s`, `frames/s`, and `frames`. Optional `range` and `extra` fields retain uncertainty and context. Validate and split sources into cost and throughput histories with `just gh benchmark-report prepare INPUT_DIRECTORY OUTPUT_DIRECTORY`; extending diff --git a/quest/m1/route-seqno-bumps.md b/quest/m1/route-seqno-bumps.md index 1987501db87..cc040af565c 100644 --- a/quest/m1/route-seqno-bumps.md +++ b/quest/m1/route-seqno-bumps.md @@ -45,12 +45,11 @@ new link lengthens no path) also keeps them passing. Mirror any change to `bump`'s callers in `RouteHandoff.tla` as the `verify-route` skill says, and state the ROUTE budget per session event so -[the mesh sweep](/quest/m1/route-mesh-sweep.md) can assert it. +the mesh sweep (`rs/moq-net/tests/integration/mesh_sweep.rs`) can assert it. Public API: none. Wire: none. ## Related - [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the contract's seqno rule this narrows -- [Seeded lite-07 meshes hold the route rules and an announce budget](/quest/m1/route-mesh-sweep.md) - the budget that would have caught this - [Each side says everything about routing on one stream](/quest/m2/cluster-routing/routes/route-stream.md) - registers one neighbour per session, which changes where the bump is called diff --git a/rs/kio/src/clock/sim.rs b/rs/kio/src/clock/sim.rs index 9601d239cdf..fa2e19358a9 100644 --- a/rs/kio/src/clock/sim.rs +++ b/rs/kio/src/clock/sim.rs @@ -270,7 +270,7 @@ impl Wake for Flag { } /// Clears the thread's runtime when [`run`] returns or unwinds, printing the seed -/// that replays an unwinding run when the environment picked it. +/// that replays an unwinding run when [`run`] seeded it. struct Enter { replay: Option, }