Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,7 @@ jobs:
# `rs quic-sim` sweeps moq-quic's simulated workloads from a fresh seed; PRs run a fixed 64.
# `rs rung-disable` proves FFI simulcast disable and recovery on a shaped BBR3 uplink.
# `test resume-model` sweeps resume against its model from a fresh seed; PRs run a fixed 64.
# `test route-sweep` checks random lite-07 meshes against the route rules from a fresh seed; PRs run a fixed 8.
# It also runs the bandwidth probe's acceptance suite, of which PRs run the short cases.
# `test drill impaired` runs the relay drills over a freshly seeded shaper; PRs run loopback.
# `rs cli-export` runs moq-cli's export process tests, which take tens of seconds of real time.
Expand All @@ -355,7 +356,7 @@ jobs:
# `rs v4l2` streams moq-video's V4L2 capture from the kernel's vivid
# virtual camera, a module load no pull request job should make.
# `js worklet-matrix` loads `?worklet` output through the bundlers pinned in test/worklet.
recipe: ["gate", "gh test-integration", "rs doctest --workspace", "rs send-depth", "test drill-sensitivity", "rs uring", "test json-properties", "test resume-model", "rs quic-sim", "rs rung-disable", "test drill impaired", "rs cli-export", "rs hls-conformance", "rs quic-stress", "rs audio-devices", "rs multicast", "rs render-sw", "test qmux", "rs v4l2", "js worklet-matrix"]
recipe: ["gate", "gh test-integration", "rs doctest --workspace", "rs send-depth", "test drill-sensitivity", "rs uring", "test json-properties", "test resume-model", "test route-sweep", "rs quic-sim", "rs rung-disable", "test drill impaired", "rs cli-export", "rs hls-conformance", "rs quic-stress", "rs audio-devices", "rs multicast", "rs render-sw", "test qmux", "rs v4l2", "js worklet-matrix"]
include:
# Miri interprets kio's and moq-net's in-memory unsafe; too slow for the merge gate.
- recipe: rs miri
Expand Down
6 changes: 3 additions & 3 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -839,8 +839,8 @@ The table includes these measurements:
- The CSV tables the JS benchmarks and the Rust benchmark tests print. A header
line names the columns; the time columns (`ns`, `us`, `ms`, and their
per-operation variants such as `ns_per_frame` or `plain_ns_per_read`),
`size_bytes` and `peak_bytes`, `mib_per_s`, and `*_packets` columns are
measurements, and every other column is a workload dimension in each series
`size_bytes` and `peak_bytes`, `mib_per_s`, and `*_packets` and `*_messages`
columns are measurements, and every other column is a workload dimension in each series
name, so print outcomes only in measured columns. The Rust benchmark tests
(`just gh nightly rust-test`) run in release builds, one at a time, under
nextest's `bench` profile, which keeps each test's output in a JUnit report;
Expand Down Expand Up @@ -912,7 +912,7 @@ A new source uploads a `bench-*` artifact containing `report.json`, a nonempty
JSON array of `{ "name": "case/dimensions", "unit": "ns", "value": 123 }`,
and `outcome.json` containing `{ "status": "success" }`. Names must be unique
within a source; dimensions belong in the name. Supported units are `ns`, `us`,
`µs`, `ms`, `s`, `ms/MB`, `x`, `bytes`, `B`, `MiB`, `packets`, `streams`,
`µs`, `ms`, `s`, `ms/MB`, `x`, `bytes`, `B`, `MiB`, `packets`, `messages`, `streams`,
`instructions`, `MB/s`, `MiB/s`, `B/s`, `ops/s`, `frames/s`, and `frames`. Optional `range` and `extra` fields retain uncertainty
and context. Validate and split sources into cost and throughput histories with
`just gh benchmark-report prepare INPUT_DIRECTORY OUTPUT_DIRECTORY`; extending
Expand Down
3 changes: 3 additions & 0 deletions doc/bin/relay/http.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,9 @@ count control-message bytes by `tier` and `class` (`session`, `announce`, `track
`probe`, `auth`, `goaway`, `other`) to spot control-heavy sessions. They
include message size prefixes but leave out media payload and transport
overhead such as QUIC framing, ACKs, and retransmissions.
`moq_relay_control_tx_messages_total` and `moq_relay_control_rx_messages_total`
count the same messages by number, so an announce storm shows as a rate
rather than as bytes.

`moq_relay_sessions_connected` is a gauge of connected sessions by `tier` and
`state`: `active` if media payload moved through the session, either way,
Expand Down
5 changes: 3 additions & 2 deletions doc/lib/rs/kio.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,9 @@ timer is armed panics as a deadlock instead of hanging. `sim::advance()` and
Draw jitter and other randomness from `kio::clock::rng()`. Inside the simulator
it comes from one seed per run, picked at random or set with `MOQ_SIM_SEED`. A
run that panics prints its seed, so rerunning the test with `MOQ_SIM_SEED` set to
that seed replays it bit-exact. Outside the simulator `rng()` is rand's thread
generator.
that seed replays it bit-exact. A sweep that derives each case's seed itself
passes it to `sim::run_seeded(seed, future)` instead, so one setting replays a
case. Outside the simulator `rng()` is rand's thread generator.

## Driving a tokio runtime by hand

Expand Down
9 changes: 6 additions & 3 deletions quest/m1/route-mesh-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,18 @@ routing goes quiet after the last change.
Decided 2026-11-10 (maintainer, /quest-plan), after the triangle model in
[moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447) found count to
infinity. Routing, announce spam, and resume failover have caused many
production issues; this and [the lite-07 mesh sweep](/quest/m1/route-mesh-sweep.md)
are routing's confidence gates for finalizing lite-07.
production issues; this and the lite-07 mesh sweep
(`rs/moq-net/tests/integration/mesh_sweep.rs`) are routing's confidence gates
for finalizing lite-07.

- Grow `RouteHandoff.tla` (or a sibling spec sharing its operators) to 4 and
5 relays in a line, a ring, and a ring with a chord; two origin identities;
and client leaves that advertise nothing but echo every ROUTE_END.
- An invariant bounds the ROUTE, batch, and request messages sent between
external changes (an origin's claim change, a session open or close) by a
budget derived from the links, so announce spam is a counterexample.
budget derived from the links, so announce spam is a counterexample. The
sweep's `budget` states 8 messages per session per route identity and per
claim; the model states the same one, or changes both.
- A liveness property, under weak fairness of delivery: after the last
external change, the queues empty and every relay shows what
`QuietIsSettled` requires.
Expand Down
31 changes: 0 additions & 31 deletions quest/m1/route-mesh-sweep.md

This file was deleted.

3 changes: 1 addition & 2 deletions quest/m1/route-seqno-bumps.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,11 @@ new link lengthens no path) also keeps them passing.

Mirror any change to `bump`'s callers in `RouteHandoff.tla` as the
`verify-route` skill says, and state the ROUTE budget per session event so
[the mesh sweep](/quest/m1/route-mesh-sweep.md) can assert it.
the mesh sweep (`rs/moq-net/tests/integration/mesh_sweep.rs`) can assert it.

Public API: none. Wire: none.

## Related

- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the contract's seqno rule this narrows
- [Seeded lite-07 meshes hold the route rules and an announce budget](/quest/m1/route-mesh-sweep.md) - the budget that would have caught this
- [Each side says everything about routing on one stream](/quest/m2/cluster-routing/routes/route-stream.md) - registers one neighbour per session, which changes where the bump is called
1 change: 0 additions & 1 deletion quest/m2/cluster-routing/routes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -556,7 +556,6 @@ benchmarks.
## Required

- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - every ordering on 4-5 relay meshes with two origins and clients, with an announce budget and convergence
- [Seeded lite-07 meshes hold the route rules and an announce budget](/quest/m1/route-mesh-sweep.md) - the route rules and message budget checked on real sessions over random meshes

## Related

Expand Down
31 changes: 21 additions & 10 deletions rs/kio/src/clock/sim.rs
Original file line number Diff line number Diff line change
Expand Up @@ -270,19 +270,18 @@ impl Wake for Flag {
}

/// Clears the thread's runtime when [`run`] returns or unwinds, printing the seed
/// that replays an unwinding run.
/// that replays an unwinding run when [`run`] seeded it.
struct Enter {
seed: u64,
replay: Option<u64>,
}

impl Drop for Enter {
fn drop(&mut self) {
CURRENT.with(|current| current.borrow_mut().take());
if std::thread::panicking() {
eprintln!(
"kio::clock::sim run panicked; replay it with MOQ_SIM_SEED={}",
self.seed
);
if let Some(seed) = self.replay
&& std::thread::panicking()
{
eprintln!("kio::clock::sim run panicked; replay it with MOQ_SIM_SEED={seed}");
}
}
}
Expand All @@ -298,12 +297,24 @@ impl Drop for Enter {
/// exceeds the ready kio poll limit, a detached task panics, or `MOQ_SIM_SEED` is
/// not a `u64`.
pub fn run<F: IntoFuture>(future: F) -> F::Output {
run_seeded(seed_from_env(), future)
let seed = seed_from_env();
enter(seed, Some(seed), future)
}

/// [`run`] with [`rng`](crate::clock::rng) and [`seed`](crate::clock::seed) drawn from
/// `seed`, ignoring `MOQ_SIM_SEED`: for a sweep that derives each case's seed itself,
/// so one setting replays a case.
///
/// # Panics
///
/// As [`run`] does, except on `MOQ_SIM_SEED`.
pub fn run_seeded<F: IntoFuture>(seed: u64, future: F) -> F::Output {
enter(seed, None, future)
}

fn run_seeded<F: IntoFuture>(seed: u64, future: F) -> F::Output {
fn enter<F: IntoFuture>(seed: u64, replay: Option<u64>, future: F) -> F::Output {
let runtime = Runtime::enter(seed);
let enter = Enter { seed };
let enter = Enter { replay };

let mut future = pin!(future.into_future());
let mut output = None;
Expand Down
15 changes: 13 additions & 2 deletions rs/moq-net/src/coding/reader.rs
Original file line number Diff line number Diff line change
Expand Up @@ -615,7 +615,17 @@ mod tests {
let size = bytes.len() as u64;
let (head, tail) = bytes.split_at(2);
let counted = || registry.snapshot().control();
let probes = |n: u64| vec![(Tier::default(), Message::Probe, Totals { tx: 0, rx: size * n })];
let probes = |n: u64| {
vec![(
Tier::default(),
Message::Probe,
Totals {
rx: size * n,
rx_messages: n,
..Default::default()
},
)]
};

let mut reader = queue(&[bytes, head], version).with_stats(&session);
let waiter = &kio::Waiter::noop();
Expand Down Expand Up @@ -650,8 +660,9 @@ mod tests {
Tier::default(),
Message::Auth,
Totals {
tx: 0,
rx: reply.len() as u64,
rx_messages: 1,
..Default::default()
},
);
assert_eq!(counted(), [vec![auth.clone()], probes(2)].concat());
Expand Down
6 changes: 5 additions & 1 deletion rs/moq-net/src/coding/writer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -502,7 +502,11 @@ mod tests {
vec![(
Tier::new(crate::Path::from_static("internal")),
Message::Probe,
Totals { tx: size, rx: 0 }
Totals {
tx: size,
tx_messages: 1,
..Default::default()
}
)]
);
writer
Expand Down
53 changes: 39 additions & 14 deletions rs/moq-net/src/stats.rs
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,8 @@ struct SessionCounters {
bytes: Mutex<SessionBytes>,
wire_tx: AtomicU64,
wire_rx: AtomicU64,
control: [[AtomicU64; 2]; Message::ALL.len()],
/// Per class: bytes sent and received, then messages sent and received.
control: [[AtomicU64; 4]; Message::ALL.len()],
}

impl SessionCounters {
Expand Down Expand Up @@ -337,11 +338,12 @@ impl SessionCounters {
(presence, activity, empty)
}

fn control(&self, totals: &mut [[u64; 2]; Message::ALL.len()]) {
for (total, counters) in totals.iter_mut().zip(&self.control) {
for (total, counter) in total.iter_mut().zip(counters) {
*total += counter.load(Ordering::Relaxed);
}
fn control(&self, totals: &mut [Totals; Message::ALL.len()]) {
for (total, [tx, rx, tx_messages, rx_messages]) in totals.iter_mut().zip(&self.control) {
total.tx += tx.load(Ordering::Relaxed);
total.rx += rx.load(Ordering::Relaxed);
total.tx_messages += tx_messages.load(Ordering::Relaxed);
total.rx_messages += rx_messages.load(Ordering::Relaxed);
}
}

Expand Down Expand Up @@ -916,14 +918,19 @@ impl Role {
}
}

/// Cumulative bytes sent and received, returned per [`Message`] class by [`Snapshot::control`].
/// Cumulative bytes and messages sent and received, returned per [`Message`] class by
/// [`Snapshot::control`].
#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
#[non_exhaustive]
pub struct Totals {
/// Bytes successfully encoded for sending, counted even if the stream later fails.
pub tx: u64,
/// Bytes of complete messages decoded and consumed.
pub rx: u64,
/// Messages successfully encoded for sending, counted like [`Self::tx`].
pub tx_messages: u64,
/// Complete messages decoded and consumed, counted like [`Self::rx`].
pub rx_messages: u64,
}

/// The bounded class of a successfully encoded or decoded control message.
Expand Down Expand Up @@ -1003,7 +1010,7 @@ pub struct Snapshot {
sessions: HashMap<Tier, Presence>,
/// Connected-session activity per tier; read via [`Self::activity`].
activity: HashMap<Tier, Activity>,
control: HashMap<Tier, [[u64; 2]; Message::ALL.len()]>,
control: HashMap<Tier, [Totals; Message::ALL.len()]>,
}

impl Snapshot {
Expand All @@ -1014,9 +1021,9 @@ impl Snapshot {
let mut rows = Vec::new();
for (tier, totals) in &self.control {
for class in Message::ALL {
let [tx, rx] = totals[class as usize];
if tx != 0 || rx != 0 {
rows.push((tier.clone(), class, Totals { tx, rx }));
let totals = totals[class as usize];
if totals != Totals::default() {
rows.push((tier.clone(), class, totals));
}
}
}
Expand Down Expand Up @@ -1734,7 +1741,9 @@ impl Session {
let Some(inner) = &self.inner else { return };
let current = inner.current.lock().unwrap_or_else(PoisonError::into_inner);
if let Some(counters) = &current.presence {
counters.control[class as usize][usize::from(received)].fetch_add(bytes as u64, Ordering::Relaxed);
let counters = &counters.control[class as usize];
counters[usize::from(received)].fetch_add(bytes as u64, Ordering::Relaxed);
counters[2 + usize::from(received)].fetch_add(1, Ordering::Relaxed);
}
}

Expand Down Expand Up @@ -2722,8 +2731,24 @@ mod tests {
context.retag(second.clone(), crate::Path::from_static(""));
context.control(Message::TrackInfo, 23, true);
let expected = vec![
(second, Message::TrackInfo, Totals { tx: 0, rx: 23 }),
(first, Message::Announce, Totals { tx: 17, rx: 0 }),
(
second,
Message::TrackInfo,
Totals {
rx: 23,
rx_messages: 1,
..Default::default()
},
),
(
first,
Message::Announce,
Totals {
tx: 17,
tx_messages: 1,
..Default::default()
},
),
];
assert_eq!(registry.snapshot().control(), expected);
drop(context);
Expand Down
1 change: 1 addition & 0 deletions rs/moq-net/tests/integration/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ mod implementation;
mod instance;
mod late_lower_group;
mod lazy_discovery;
mod mesh_sweep;
mod mesh_withdraw;
mod model;
mod mount;
Expand Down
Loading