diff --git a/quest/m1/README.md b/quest/m1/README.md index bd8fa80fa4a..691c39204a3 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -56,7 +56,7 @@ m4 default, so changes are proven before they ship: the [QUIC Interop Runner](/quest/m1/quic-interop.md), [sim rule checks](/quest/m1/quic-sim-rules.md), [relay parser fuzz targets](/quest/m1/fuzz-relay-parsers.md), and from m4 -[the relay on the sim](/quest/m1/quic-sim.md), +the relay on the sim (landed), [mutation testing](/quest/m1/mutants.md), and [peer-reachable unsafe](/quest/m1/wire-fuzz-coverage.md). OSS-Fuzz stays in m4 (maintainer, same round): it mostly waits on google/oss-fuzz. diff --git a/quest/m1/quic-sim-rules.md b/quest/m1/quic-sim-rules.md index d6e7b5f470a..f58febac7d7 100644 --- a/quest/m1/quic-sim-rules.md +++ b/quest/m1/quic-sim-rules.md @@ -35,10 +35,13 @@ Agent's choices, for the implementer to revisit: such patch does not land. - List the rules considered and left out, with the reason. - The 64-case gate run must not slow measurably; say by how much it did. +- The relay and moq-tokio's `Client` and `Server` already run over moq-tokio's + `quic::driver::sim::Network` (`rs/moq-relay/tests/integration/quic_sim.rs`, + `rs/moq-tokio/tests/integration/sim_client.rs`), whose paths cross the + shared `Link`, so a checker there judges those runs too. Public API: none. Wire: none. ## Related - [QUIC Interop Runner](/quest/m1/quic-interop.md) - the same rules judged by other stacks -- [The relay over QUIC on the sim](/quest/m1/quic-sim.md) - its runs gain the checker through the shared `Link` diff --git a/quest/m1/quic-sim.md b/quest/m1/quic-sim.md deleted file mode 100644 index e92ef5be5eb..00000000000 --- a/quest/m1/quic-sim.md +++ /dev/null @@ -1,51 +0,0 @@ -# [M] The relay over QUIC on the sim - -## Goal - -The relay, and moq-tokio's `Client` and `Server`, run over `moq-quic` on the -`kio::clock` sim with configurable latency, loss, reordering, and bandwidth, -so relay hops, the transport race, and reconnects over real QUIC are tested -deterministically with no sockets. - -## Plan - -Already landed: MoQ sessions over moq-quic on the sim. The QUIC driver -spawns on the simulator inside `kio::clock::sim::run`, under moq-tokio's -`sim` feature. `quic::driver::sim::Network` binds a socket per address -and carries datagrams across moq-quic's seeded `Link` (now -`moq_quic::sim`), so one seed replays a run. -`rs/moq-tokio/tests/integration/sim.rs` builds both endpoints on the raw -driver and seeds their `rng_seed`, which also seeds each connection's -congestion controller. - -What remains: - -- moq-tokio's `Client` and `Server` bind their QUIC endpoint from an address - (`crate::bind::udp`), so a sim test cannot reach them yet. Pick how a - caller hands either one a `driver::endpoint::Socket`, such as a - documented `sim`-only hook (`just check` bans `#[doc(hidden)]`). That API - choice belongs to the maintainer (see Waiting). `Client::with_network` - (`rs/moq-tokio/src/sim.rs`) already gives a client an in-memory network - under the `sim` feature, and the relay's integration tests run hops and - reconnects over it (`rs/moq-relay/tests/integration/memory.rs`), though - not over QUIC. -- Route the session-level spawns that `server.rs` and `origin.rs` still - send through `crate::runtime::spawn` through - `runtime::Spawner`, as `transport.rs` and the WebTransport session already - do. -- Then run `moq_relay` between a publisher and a subscriber, and between two - relays, on one `Network`. Also run the client's Happy Eyeballs race across - addresses with different paths, and a WebTransport (`https`) dial beside - raw QUIC. - -Keep random loss for recovery correctness. Any acceptance or performance -consumer uses a rate below its media bitrate behind a deep but finite queue, -and checks that the queue overflowed. A stalled sender that never filled the -queue is not evidence about congestion loss. - -Decided 2026-11-13 (maintainer, /quest-plan on what testing to add): ✅ m1 -(rejected: staying in m4). It is the base for [fault schedules](/quest/m4/cluster-fault-schedules.md). - -## Waiting - -- Maintainer: how moq-tokio's `Client` and `Server` take a simulated QUIC socket. Options: (a) extend the `sim`-feature `Client::with_network` path, already documented as tracking internals that can change in any release, to carry `quic::driver::sim::Network`, with a matching hook on `Server`; (b) a new `sim`-only `with_socket(driver::endpoint::Socket)` on each; (c) drop this quest, since the memory network already covers relay hops and reconnects. Recommended: (a) if a failure only real QUIC would show is named, otherwise (c). diff --git a/quest/m4/cluster-fault-schedules.md b/quest/m4/cluster-fault-schedules.md index 6ef27de1db5..98e43e8a5dc 100644 --- a/quest/m4/cluster-fault-schedules.md +++ b/quest/m4/cluster-fault-schedules.md @@ -38,15 +38,15 @@ Agent's choices, for the implementer to revisit: - Prove the harness finds things: `test/drill/mutations`'s patches, applied one at a time, must each fail some seed within the nightly budget. Report which do not, since those are the drills this cannot replace. +- Build on `rs/moq-relay/tests/integration/quic_sim.rs`: a relay serves a + moq-tokio `Server` whose `server::Config::sim` binds it on a + `quic::driver::sim::Host`, peers dial through `cluster::Options::sim`, and + clients through `client::Config::sim`, all on one `Network`. - No turmoil or madsim: `kio::clock::sim` and `moq_quic::sim` already give a seeded scheduler and network with no shims. Public API: none. Wire: none. -## Required - -- [The relay over QUIC on the sim](/quest/m1/quic-sim.md) - relays and clients on one simulated `Network` - ## Related - [Sim rule checks](/quest/m1/quic-sim-rules.md) - QUIC's rules checked on the same runs diff --git a/quest/m4/quic-realtime.md b/quest/m4/quic-realtime.md index 8ee7caa14ed..9f74a4e56c3 100644 --- a/quest/m4/quic-realtime.md +++ b/quest/m4/quic-realtime.md @@ -172,7 +172,3 @@ through `bandwidth_estimate` → `estimated_send_rate` → `set_bitrate`. the upstream encoder, so a GStreamer pipeline at a fixed bitrate still queues when capacity drops; [its own quest](/quest/m4/moqsink-feedback.md) owns that feedback. The bake-off and shipping `Realtime` do not wait on it. - -## Related - -- [QUIC on the sim](/quest/m1/quic-sim.md) - deterministic successor to the moq-shaper harness once it lands diff --git a/quest/m4/quic-sim-mtu.md b/quest/m4/quic-sim-mtu.md index 90f7f1610e3..e02476b9017 100644 --- a/quest/m4/quic-sim-mtu.md +++ b/quest/m4/quic-sim-mtu.md @@ -32,7 +32,3 @@ fixed by [moq-dev/moq#7429](https://github.com/moq-dev/moq/pull/7429), and the 0 report is moot ([0-RTT leftovers](https://github.com/moq-dev/moq/pull/7492)). Public API: none (test harness only). Wire: none. - -## Related - -- [QUIC on the sim](/quest/m1/quic-sim.md) - reuses this `Pair` for MoQ sessions diff --git a/quest/m5/bench-transport-sweep.md b/quest/m5/bench-transport-sweep.md index 00902477d16..a0ea2a2ac7c 100644 --- a/quest/m5/bench-transport-sweep.md +++ b/quest/m5/bench-transport-sweep.md @@ -28,4 +28,3 @@ Public API: none. Wire: none. ## Related - [Benchmark sources](/quest/m4/bench-sources.md) - feeds these series into the regression table -- [QUIC on the sim](/quest/m1/quic-sim.md) - an in-memory QUIC link the sweep could reuse diff --git a/rs/moq-relay/Cargo.toml b/rs/moq-relay/Cargo.toml index f61470ee170..956e0e3c60c 100644 --- a/rs/moq-relay/Cargo.toml +++ b/rs/moq-relay/Cargo.toml @@ -36,7 +36,8 @@ doc = false [features] # A programmatic relay fixture with generated TLS and ephemeral ports, and -# `cluster::Options::network`, which routes peer dials over in-memory links. +# `cluster::Options::network`, which routes peer dials over in-memory links, and with +# `quic`, `cluster::Options::sim`, which binds them on a simulated QUIC host. test-support = ["moq-tokio/sim"] # The optional public STUN Binding listener (`--stun-bind`). stun = ["dep:stun-types"] diff --git a/rs/moq-relay/src/cluster.rs b/rs/moq-relay/src/cluster.rs index 844ee1e1ba6..8bdb62f6db6 100644 --- a/rs/moq-relay/src/cluster.rs +++ b/rs/moq-relay/src/cluster.rs @@ -1046,6 +1046,11 @@ pub struct Options { /// `None` dials over sockets. #[cfg(feature = "test-support")] pub network: Option, + + /// The simulated host peer dials bind on, for tests over QUIC on `kio::clock::sim`; + /// see [`moq_tokio::client::Config::sim`]. `None` binds UDP sockets. + #[cfg(all(feature = "test-support", feature = "quic"))] + pub sim: Option, } impl Options { @@ -1321,6 +1326,9 @@ pub struct Cluster { /// Where peer dials go in place of the network; see [`Options::network`]. #[cfg(feature = "test-support")] network: Option, + /// The simulated host peer dials bind on; see [`Options::sim`]. + #[cfg(all(feature = "test-support", feature = "quic"))] + sim: Option, /// The client static and API peers share when the template binds a fixed /// `[connect]` port, so that port stays the cluster's egress port. Otherwise /// `None`, and every session builds its own client on an ephemeral port. Built @@ -1443,6 +1451,8 @@ impl Cluster { stats, #[cfg(feature = "test-support")] network, + #[cfg(all(feature = "test-support", feature = "quic"))] + sim, } = options; let deprecated = config.deprecated(); anyhow::ensure!(deprecated.is_empty(), "{deprecated}"); @@ -1500,6 +1510,8 @@ impl Cluster { quic: None, #[cfg(feature = "test-support")] network, + #[cfg(all(feature = "test-support", feature = "quic"))] + sim, dial: None, dialers: Dialers::new(), advertise: None, @@ -2350,7 +2362,14 @@ impl Cluster { } let mut quic = self.quic.as_deref().cloned().unwrap_or_default(); self.peer_quic.dial(&mut quic); - let client = connect.init(quic)?; + let mut config = moq_tokio::client::Config::default(); + config.connect = connect; + config.quic = quic; + #[cfg(all(feature = "test-support", feature = "quic"))] + { + config.sim = self.sim.clone(); + } + let client = config.init()?; #[cfg(feature = "test-support")] let client = match &self.network { Some(network) => client.with_network(network.clone()), diff --git a/rs/moq-relay/tests/integration/main.rs b/rs/moq-relay/tests/integration/main.rs index 49c21c23910..63182023c65 100644 --- a/rs/moq-relay/tests/integration/main.rs +++ b/rs/moq-relay/tests/integration/main.rs @@ -30,6 +30,8 @@ mod logs; mod memory; mod peer_limits; mod public_rules; +#[cfg(all(feature = "test-support", feature = "quic"))] +mod quic_sim; mod refusals; mod released_cli; mod runtime_uring; diff --git a/rs/moq-relay/tests/integration/quic_sim.rs b/rs/moq-relay/tests/integration/quic_sim.rs new file mode 100644 index 00000000000..b0da89ecf9b --- /dev/null +++ b/rs/moq-relay/tests/integration/quic_sim.rs @@ -0,0 +1,223 @@ +//! The relay over QUIC on kio's simulator: each relay accepts on a moq-tokio +//! [`Server`](moq_tokio::Server) bound on a simulated host, its peers dial over moq-tokio +//! [`Client`](moq_tokio::Client)s bound there too, and every datagram crosses a seeded, +//! impaired path. No sockets and no host clock. + +use std::{ + net::{IpAddr, Ipv4Addr, SocketAddr}, + time::Duration, +}; + +use moq_relay::{Connection, auth, cluster}; +use moq_tokio::quic::driver::sim::{Link, Network, Path}; + +use crate::memory; + +const PORT: u16 = 443; +/// Where each client binds, on its own host +const CLIENT_PORT: u16 = 4433; +const PUBLISHER: IpAddr = ip(2); +const SUBSCRIBER: IpAddr = ip(3); + +/// The groups the publisher writes, one every [`INTERVAL`] +const GROUPS: u64 = 20; +const INTERVAL: Duration = Duration::from_millis(100); +const FRAME: usize = 4_000; + +/// How long a run may take before it counts as stalled, in simulated time +const TIMEOUT: Duration = Duration::from_secs(60); + +const fn ip(last: u8) -> IpAddr { + IpAddr::V4(Ipv4Addr::new(10, 0, 0, last)) +} + +/// A publisher and a subscriber on either side of one relay, across lossy paths. +#[test] +fn one_relay() { + memory::run(|| async { + for seed in 0..2 { + let network = Network::new(seed); + let relay = SocketAddr::new(ip(1), PORT); + let auth = memory::public(&["**"], &[], &[]); + let cluster = memory::start(memory::cluster(), moq_tokio::connect::Config::default()).await; + let relay = serve(&network, relay, &cluster, &auth).await; + for client in [PUBLISHER, SUBSCRIBER] { + lossy(&network, SocketAddr::new(client, CLIENT_PORT), relay.addr); + } + + let groups = kio::clock::timeout(TIMEOUT, deliver(&network, &relay, &relay)) + .await + .unwrap_or_else(|_| panic!("seed {seed}: the track never finished")); + assert_eq!(groups, (0..GROUPS).collect::>(), "seed {seed}"); + } + }); +} + +/// A publisher on relay A and a subscriber on relay B, which dials A as its cluster peer over +/// QUIC, every hop across a lossy path. +#[test] +fn two_relays() { + memory::run(|| async { + let network = Network::new(1); + let auth = memory::public(&["**"], &[], &[]); + + let cluster_a = memory::start(memory::cluster(), moq_tokio::connect::Config::default()).await; + let a = serve(&network, SocketAddr::new(ip(1), PORT), &cluster_a, &auth).await; + + let b = SocketAddr::new(ip(4), PORT); + let mut config = cluster::Config::default(); + config.connect = vec![cluster::Peer::new(format!("moqt://{}/", a.addr))]; + let mut cache = moq_relay::cache::Config::default(); + // Past what the pool counts in milliseconds, so it never expires idle groups. + cache.duration = Some(Duration::MAX.into()); + let mut options = cluster::Options::new(config).with_cache(cache.init().expect("cache")); + options.sim = Some(network.host(b.ip())); + // A fixed port, which the static peer dials share, so the test can impair that path. + let peer = SocketAddr::new(b.ip(), CLIENT_PORT); + let mut connect = moq_tokio::connect::Config::default(); + connect.bind = Some(peer); + connect.tls.fingerprint = a.fingerprints.clone(); + let cluster_b = memory::start(memory::spawn_cluster(options), connect).await; + let b = serve(&network, b, &cluster_b, &auth).await; + + lossy(&network, peer, a.addr); + lossy(&network, SocketAddr::new(PUBLISHER, CLIENT_PORT), a.addr); + lossy(&network, SocketAddr::new(SUBSCRIBER, CLIENT_PORT), b.addr); + + let groups = kio::clock::timeout(TIMEOUT, deliver(&network, &a, &b)) + .await + .expect("the track never finished"); + assert_eq!(groups, (0..GROUPS).collect::>()); + }); +} + +/// A relay serving on a simulated host +struct Relay { + addr: SocketAddr, + fingerprints: Vec, +} + +/// Serve `cluster` on `addr` through `auth`: each session the server accepts runs a relay +/// [`Connection`] on its own simulator task. +async fn serve(network: &Network, addr: SocketAddr, cluster: &cluster::Cluster, auth: &auth::Auth) -> Relay { + let _ = rind_rustls::default_provider().unwrap().install_default(); + let mut config = moq_tokio::server::Config::default(); + config.listen.bind = Some(moq_tokio::listen::Bind::Addr(addr)); + config.listen.tls.generate = vec!["localhost".into()]; + config.sim = Some(network.host(addr.ip())); + let mut listener = config.init().expect("init server").listen().await.expect("listen"); + let fingerprints = listener.certificates().fingerprints(); + let (cluster, auth) = (cluster.clone(), auth.clone()); + drop(kio::clock::sim::spawn(async move { + while let Ok(request) = listener.accept().await { + let (cluster, auth) = (cluster.clone(), auth.clone()); + drop(kio::clock::sim::spawn(Connection::new(request, cluster, auth).run())); + } + })); + Relay { addr, fingerprints } +} + +/// Publish [`GROUPS`] groups to `publish` from [`PUBLISHER`], subscribe from `subscribe` on +/// [`SUBSCRIBER`], and return the sequence of every group that arrived whole. +async fn deliver(network: &Network, publish: &Relay, subscribe: &Relay) -> Vec { + let origin = moq_tokio::origin::spawn(); + let broadcast = origin + .route("live", moq_net::origin::RouteOptions::default()) + .expect("route"); + let track = broadcast + .broadcast() + .create_track( + "video", + moq_net::track::Info::default() + .with_timescale(moq_net::Timescale::MILLI) + .with_max_age(None), + ) + .expect("track"); + broadcast.announce().expect("announce"); + let _publisher = client(network, PUBLISHER, publish) + .with_publisher(&origin) + .connect(url(publish)); + + let viewer = moq_tokio::origin::spawn(); + let _subscriber = client(network, SUBSCRIBER, subscribe) + .with_subscriber(viewer.clone()) + .connect(url(subscribe)); + + let write = async { + let mut source = track.clone(); + while source.subscription_changed().await.unwrap().is_none() {} + for sequence in 0..GROUPS { + let mut group = source.create_group(moq_net::group::Info { sequence }).unwrap(); + let timestamp = moq_net::Timestamp::from_millis(sequence * INTERVAL.as_millis() as u64).unwrap(); + group.write_frame(timestamp, vec![sequence as u8; FRAME]).unwrap(); + group.finish().unwrap(); + kio::clock::sleep(INTERVAL).await; + } + let _source = source.finish().unwrap(); + }; + let read = async { + let remote = loop { + match viewer.consume().request("live", None).await { + Ok(remote) => break remote, + // Not announced across the relays yet. + Err(_) => kio::clock::sleep(Duration::from_millis(50)).await, + } + }; + let mut reader = remote + .track("video") + .unwrap() + // A budget spanning the run, so recovery has to deliver every group, however late. + .subscribe( + moq_net::track::Subscription::default().with_delay(moq_net::track::Delay::default().with_max(TIMEOUT)), + ) + .await + .unwrap(); + let mut groups = Vec::new(); + while let Some(mut group) = reader.recv_group().await.unwrap() { + let sequence = group.sequence; + let frame = group.read_frame().await.unwrap().expect("a group is cut short"); + assert_eq!(frame.payload.len(), FRAME, "group {sequence} is cut short"); + assert!( + frame.payload.iter().all(|&byte| byte == sequence as u8), + "group {sequence} is corrupted" + ); + groups.push(sequence); + } + groups.sort(); + groups + }; + let ((), groups) = futures::join!(write, read); + drop(broadcast); + groups +} + +/// A client on the host at `ip`, trusting `relay`'s certificate +fn client(network: &Network, ip: IpAddr, relay: &Relay) -> moq_tokio::Client { + let mut connect = moq_tokio::connect::Config::default(); + connect.bind = Some(SocketAddr::new(ip, CLIENT_PORT)); + connect.tls.fingerprint = relay.fingerprints.clone(); + let mut config = moq_tokio::client::Config::default(); + config.connect = connect; + config.sim = Some(network.host(ip)); + config.init().expect("init client") +} + +fn url(relay: &Relay) -> url::Url { + format!("moqt://{}/", relay.addr).parse().unwrap() +} + +/// Make both directions between `a` and `b` lossy, with latency +fn lossy(network: &Network, a: SocketAddr, b: SocketAddr) { + let mut link = Link::default(); + link.loss = 0.05; + link.duplicate = 0.01; + link.jitter = Duration::from_millis(5); + link.reorder = 0.02; + link.reorder_delay = Duration::from_millis(20); + let path = Path { + latency: Duration::from_millis(20), + link, + }; + network.path(a, b, |p| *p = path.clone()); + network.path(b, a, |p| *p = path); +} diff --git a/rs/moq-tokio/Cargo.toml b/rs/moq-tokio/Cargo.toml index 7702744cf5e..edb17466745 100644 --- a/rs/moq-tokio/Cargo.toml +++ b/rs/moq-tokio/Cargo.toml @@ -50,8 +50,9 @@ qmux = ["_transport", "dep:qmux"] uds = ["tcp", "qmux/uds"] # Inside `kio::clock::sim::run`, detached tasks spawn on kio's deterministic simulator, # and with `driver`, `quic::driver::sim` carries datagrams over seeded, impaired -# `moq_quic::sim::Link`s with no sockets. With `net`, `sim::Network` routes a client's dials -# over in-memory links instead. Turns on no transport of its own. Hidden from the +# `moq_quic::sim::Link`s with no sockets; with `quic` too, `client::Config::sim` and +# `server::Config::sim` bind a `Client` or `Server` on one of its hosts. With `net`, +# `sim::Network` routes a client's dials over in-memory links instead. Turns on no transport of its own. Hidden from the # docs. Meant for dev-dependencies, so production builds never compile it. sim = ["kio/sim", "moq-quic?/sim"] diff --git a/rs/moq-tokio/src/client.rs b/rs/moq-tokio/src/client.rs index 14a5aced765..680a0d62324 100644 --- a/rs/moq-tokio/src/client.rs +++ b/rs/moq-tokio/src/client.rs @@ -26,6 +26,11 @@ pub struct Config { /// QUIC socket and transport settings, shared with [`crate::Server`]. pub quic: crate::quic::Config, + + /// The simulated host the QUIC endpoint binds on, at the configured bind port, in place of + /// a UDP socket. For tests on `kio::clock::sim`; see [`crate::quic::driver::sim`]. + #[cfg(all(feature = "sim", feature = "quic"))] + pub sim: Option, } impl Config { @@ -122,7 +127,10 @@ impl Client { #[cfg(feature = "_transport")] pub fn new(config: Config) -> crate::Result { let Config { - connect: config, quic, .. + connect: config, + quic, + #[cfg(all(feature = "sim", feature = "quic"))] + sim, } = config; // Refuse here rather than in `init`, so a caller that skipped its own check @@ -143,8 +151,15 @@ impl Client { #[cfg(any(feature = "websocket", feature = "tcp"))] let qmux = crate::qmux_config(&quic.resolve()); + #[cfg(all(feature = "quic", not(feature = "sim")))] + let quic = crate::quic::Client::new(&config, &quic); + #[cfg(all(feature = "quic", feature = "sim"))] + let quic = match sim { + Some(host) => crate::quic::Client::simulated(&config, &quic, host), + None => crate::quic::Client::new(&config, &quic), + }; #[cfg(feature = "quic")] - let quic = Some(crate::quic::Client::new(&config, &quic)?); + let quic = Some(quic?); let versions = config.versions(); // Read before the struct literal below moves fields out of `config`. @@ -333,6 +348,10 @@ impl Client { // An explicitly closed QUIC endpoint cannot open another socket. #[cfg(feature = "quic")] let quic = self.quic.as_ref().is_some_and(|quic| !quic.is_closed()); + #[cfg(all(feature = "sim", feature = "quic"))] + if self.simulated() { + return quic && simulated_scheme(scheme); + } #[cfg(not(feature = "quic"))] let quic = false; #[cfg(feature = "websocket")] @@ -349,6 +368,12 @@ impl Client { } } + /// Whether QUIC binds on a simulated host, where no other transport can dial. + #[cfg(all(feature = "sim", feature = "quic"))] + fn simulated(&self) -> bool { + self.quic.as_ref().is_some_and(crate::quic::Client::is_simulated) + } + #[cfg(feature = "quic")] pub(crate) fn fresh_port(&mut self) -> crate::Result<()> { if let Some(quic) = &mut self.quic { @@ -453,6 +478,10 @@ impl Client { if self.client_cert && matches!(addr.url().scheme(), "http" | "ws" | "tcp" | "unix") { return Err(Error::ClientCertScheme(addr.url().scheme().to_string())); } + #[cfg(all(feature = "sim", feature = "quic"))] + if self.simulated() && !simulated_scheme(addr.url().scheme()) { + return Err(Error::SimScheme(addr.url().scheme().to_string())); + } #[cfg(test)] if let Some(dialer) = &self.dialer { return dialer(addr, client.moq.clone()).await; @@ -586,17 +615,15 @@ impl Client { .map_err(Error::from) }; + // A simulated host has no TCP for the fallback to race. #[cfg(feature = "websocket")] - { + if !quic.is_simulated() { return self.race_moq_connect(&moq, addr, quic_handle).await; } - #[cfg(not(feature = "websocket"))] - { - let session = quic_handle.await?; - let session = connect_session(&moq, session).await?; - return Ok(Dialed::new(session, quic_transport(&url))); - } + let session = quic_handle.await?; + let session = connect_session(&moq, session).await?; + return Ok(Dialed::new(session, quic_transport(&url))); } #[cfg(feature = "websocket")] @@ -762,6 +789,13 @@ where } } +/// Whether a client on a simulated host can dial `scheme`: only QUIC's, without the `http://` +/// bootstrap's fetch over TCP. +#[cfg(all(feature = "sim", feature = "quic"))] +fn simulated_scheme(scheme: &str) -> bool { + matches!(scheme, "https" | "moqt" | "moql") +} + /// The error for a race both arms lost. /// /// Auth is terminal only when both arms refused. A WebTransport-only endpoint @@ -786,7 +820,8 @@ async fn connect_session( ) -> Result { let (session, driver) = client.connect(moq_net::time::Instant::now(), transport).await?; use tracing::Instrument; - tokio::spawn(crate::panic::isolate(moq_net::time::run(driver)).instrument(tracing::Span::current())); + crate::runtime::Spawner::current() + .spawn(crate::panic::isolate(moq_net::time::run(driver)).instrument(tracing::Span::current())); Ok(session) } diff --git a/rs/moq-tokio/src/error.rs b/rs/moq-tokio/src/error.rs index abf657c2ceb..17796055334 100644 --- a/rs/moq-tokio/src/error.rs +++ b/rs/moq-tokio/src/error.rs @@ -151,6 +151,12 @@ pub enum Error { #[error("{0}:// can't present the client certificate; dial https://, wss://, moqt://, moql://, or tls:// instead")] ClientCertScheme(String), + /// A client on a simulated host dials only QUIC, since every other transport opens a real + /// socket. The payload is the scheme. + #[cfg(feature = "sim")] + #[error("{0}:// can't be dialed from a simulated host; dial https://, moqt://, or moql:// instead")] + SimScheme(String), + /// The server rejected the connection with an auth status. See [`crate::ConnectError`]. #[cfg(feature = "net")] #[error(transparent)] diff --git a/rs/moq-tokio/src/lib.rs b/rs/moq-tokio/src/lib.rs index 6a1767ad08d..3e575ae1a75 100644 --- a/rs/moq-tokio/src/lib.rs +++ b/rs/moq-tokio/src/lib.rs @@ -103,7 +103,7 @@ fn qmux_default() -> qmux::Config { fn spawn_qmux( (session, driver): (qmux::Session, qmux::Driver), ) -> qmux::Session { - runtime::spawn(driver); + runtime::Spawner::current().spawn(driver); session } diff --git a/rs/moq-tokio/src/origin.rs b/rs/moq-tokio/src/origin.rs index 3761922dae5..069028fdd7f 100644 --- a/rs/moq-tokio/src/origin.rs +++ b/rs/moq-tokio/src/origin.rs @@ -1,7 +1,8 @@ //! Tokio construction helpers for [`moq_net::origin`]. /// Build an origin producer and spawn its [`moq_net::origin::Driver`] on the -/// ambient tokio runtime, else moq-tokio's fallback runtime. +/// ambient tokio runtime, else moq-tokio's fallback runtime, or on the `kio::clock` +/// simulator when called inside one. /// /// The tokio shorthand for [`moq_net::origin::Producer::new`]: the driver runs /// as its own task, finishing once every producer handle drops and the @@ -20,7 +21,7 @@ pub fn spawn_config(config: moq_net::origin::Config) -> moq_net::origin::Produce use futures::FutureExt; let (producer, driver) = moq_net::origin::Producer::new(config); - crate::runtime::spawn(async move { + crate::runtime::Spawner::current().spawn(async move { if let Err(payload) = std::panic::AssertUnwindSafe(moq_net::time::run(driver)) .catch_unwind() .await diff --git a/rs/moq-tokio/src/quic/backend.rs b/rs/moq-tokio/src/quic/backend.rs index ad95eb60412..7d8e264d1f0 100644 --- a/rs/moq-tokio/src/quic/backend.rs +++ b/rs/moq-tokio/src/quic/backend.rs @@ -89,6 +89,12 @@ pub enum Error { #[error("failed to bind UDP socket")] BindSocket(#[source] std::io::Error), + /// A client on a simulated host was asked to look a name up, which only a real resolver can. + /// The payload is the name. + #[cfg(feature = "sim")] + #[error("{0} can't be resolved from a simulated host; dial an IP literal or pinned addresses")] + SimResolve(String), + /// The QUIC endpoint could not be created around the bound socket. #[error("failed to create QUIC endpoint")] CreateEndpoint(#[source] driver::endpoint::Error), @@ -294,7 +300,7 @@ pub(crate) struct Client { /// Each reconnect loop may replace its endpoint without moving other sessions. /// Weak handles let close reach all of them without retaining abandoned sockets. endpoints: Arc>, - bind: net::SocketAddr, + bind: Bind, pub transport: driver::transport::Config, /// Whether an `http://` URL may bootstrap a pin (see [crate::tls::Connect::allows_http_bootstrap]). pub http_bootstrap: bool, @@ -313,6 +319,51 @@ pub(crate) struct Client { keylog: Option>, } +/// Where a client's endpoints get their sockets +#[derive(Clone)] +enum Bind { + /// A UDP socket at this address + Udp(net::SocketAddr), + /// This port, or an ephemeral one for 0, on a simulated host + #[cfg(feature = "sim")] + Sim(driver::sim::Host, u16), +} + +impl Bind { + fn port(&self) -> u16 { + match self { + Self::Udp(addr) => addr.port(), + #[cfg(feature = "sim")] + Self::Sim(_, port) => *port, + } + } + + /// A new endpoint, and whether its socket reaches IPv4 destinations as well as its own family + #[cfg_attr( + not(feature = "sim"), + allow(unused_mut, reason = "only a simulated socket seeds the endpoint") + )] + fn endpoint(&self) -> Result<(driver::endpoint::Endpoint, bool)> { + let mut config = driver::endpoint::Config::default(); + let (socket, dual_stack): (driver::endpoint::Socket, _) = match self { + Self::Udp(addr) => { + let socket = crate::bind::udp(crate::bind::Udp::new(*addr)).map_err(Error::BindSocket)?; + let dual_stack = crate::bind::udp_is_dual_stack(&socket); + (socket.into(), dual_stack) + } + #[cfg(feature = "sim")] + Self::Sim(host, port) => { + let (socket, seed) = host.bind(*port).map_err(Error::BindSocket)?; + config.rng_seed = Some(seed); + // A host has one concrete address, so it reaches only its own family. + (socket.into(), false) + } + }; + let endpoint = driver::endpoint::Endpoint::new(config, socket).map_err(Error::CreateEndpoint)?; + Ok((endpoint, dual_stack)) + } +} + struct Endpoints { closed: bool, all: Vec, @@ -323,10 +374,28 @@ impl Client { self.endpoints.lock().unwrap().closed } + /// Whether each endpoint binds on a simulated host rather than a UDP socket. + pub fn is_simulated(&self) -> bool { + match self.bind { + Bind::Udp(_) => false, + #[cfg(feature = "sim")] + Bind::Sim(..) => true, + } + } + pub fn new(config: &connect::Config, quic: &crate::quic::Config) -> Result { + Self::bound(config, quic, Bind::Udp(config.resolve().bind)) + } + + /// [`Self::new`], binding each endpoint on `host` at the configured port. + #[cfg(feature = "sim")] + pub fn simulated(config: &connect::Config, quic: &crate::quic::Config, host: driver::sim::Host) -> Result { + Self::bound(config, quic, Bind::Sim(host, config.resolve().bind.port())) + } + + fn bound(config: &connect::Config, quic: &crate::quic::Config, bind: Bind) -> Result { let resolved = config.resolve(); - let socket = crate::bind::udp(crate::bind::Udp::new(resolved.bind)).map_err(Error::BindSocket)?; - let dual_stack = crate::bind::udp_is_dual_stack(&socket); + let (endpoint, dual_stack) = bind.endpoint()?; let quic = quic.resolve(); let transport = transport(&quic, "client")?; @@ -337,8 +406,7 @@ impl Client { .transpose()? .map(Arc::new); - let quic = driver::endpoint::Endpoint::new(driver::endpoint::Config::default(), socket) - .map_err(Error::CreateEndpoint)?; + let quic = endpoint; let endpoints = Arc::new(Mutex::new(Endpoints { closed: false, all: vec![quic.downgrade()], @@ -347,7 +415,7 @@ impl Client { Ok(Self { quic, endpoints, - bind: resolved.bind, + bind, transport, http_bootstrap: config.tls.allows_http_bootstrap(), host_name: config.tls.host_name.clone(), @@ -368,10 +436,7 @@ impl Client { return Ok(()); } // Bind before dropping the old endpoint so the kernel cannot reuse its port. - let socket = crate::bind::udp(crate::bind::Udp::new(self.bind)).map_err(Error::BindSocket)?; - let dual_stack = crate::bind::udp_is_dual_stack(&socket); - let quic = driver::endpoint::Endpoint::new(driver::endpoint::Config::default(), socket) - .map_err(Error::CreateEndpoint)?; + let (quic, dual_stack) = self.bind.endpoint()?; endpoints.all.retain(|endpoint| endpoint.upgrade().is_some()); endpoints.all.push(quic.downgrade()); tracing::info!(old = %self.quic.local_addr(), new = %quic.local_addr(), "redialing from a fresh UDP port"); @@ -422,6 +487,14 @@ impl Client { // answers Happy Eyeballs style as they land, so neither a broken family nor a // lookup still waiting on its AAAA record can stall the connect. let local = self.quic.local_addr(); + #[cfg(feature = "sim")] + if self.is_simulated() + && addr.addresses().is_none() + && let url::Host::Domain(domain) = &target + && domain.parse::().is_err() + { + return Err(Error::SimResolve(domain.to_string())); + } let candidates = match addr.addresses() { Some(addrs) => crate::resolve::Candidates::fixed(addrs.iter().copied()), None => crate::resolve::Candidates::resolve(target, port, self.resolution_delay), @@ -595,6 +668,9 @@ pub(crate) struct Options { pub member: Option, pub tls: crate::tls::Serve, pub demux: super::demux::Demux, + /// Bind on this simulated host instead of a UDP socket. + #[cfg(feature = "sim")] + pub sim: Option, } impl Server { @@ -605,6 +681,8 @@ impl Server { member, tls: served, demux, + #[cfg(feature = "sim")] + sim, } = options; let quic = quic.resolve(); let mut transport = transport(&quic, "server")?; @@ -706,12 +784,27 @@ impl Server { // A group socket was released only after every member bound and the // steering filter covered the final array. - let socket = match member { - Some(member) => member.into_inner(), - None => crate::bind::udp(crate::bind::Udp::new(listen)).map_err(Error::BindSocket)?, + #[cfg(feature = "sim")] + let simulated = match &sim { + Some(host) => { + let (socket, seed) = host.bind(listen.port()).map_err(Error::BindSocket)?; + endpoint_config.rng_seed = Some(seed); + Some(Box::new(socket) as Box) + } + None => None, + }; + #[cfg(not(feature = "sim"))] + let simulated = None; + let socket = match simulated { + Some(socket) => socket, + None => { + let socket = match member { + Some(member) => member.into_inner(), + None => crate::bind::udp(crate::bind::Udp::new(listen)).map_err(Error::BindSocket)?, + }; + driver::wrap_udp_socket(socket).map_err(|err| Error::CreateEndpoint(err.into()))? + } }; - - let socket = driver::wrap_udp_socket(socket).map_err(|err| Error::CreateEndpoint(err.into()))?; let socket = demux.wrap(socket); let (quic, peers) = match forward { Some(Forwarding { forward, watcher }) => ( @@ -1011,6 +1104,8 @@ mod tests { member: None, tls, demux, + #[cfg(feature = "sim")] + sim: None, }) .expect("server init"); let port = server.local_addr().port(); @@ -1101,6 +1196,8 @@ mod tests { member: None, tls, demux: Default::default(), + #[cfg(feature = "sim")] + sim: None, }) .expect("server init"); let addr = server.local_addr(); @@ -1203,6 +1300,8 @@ mod tests { member: None, tls, demux: Default::default(), + #[cfg(feature = "sim")] + sim: None, }) .expect("server init"); let addr = server.local_addr(); @@ -1295,6 +1394,8 @@ mod tests { member: None, tls, demux: Default::default(), + #[cfg(feature = "sim")] + sim: None, }) .expect("server init"); let addr = server.local_addr(); diff --git a/rs/moq-tokio/src/quic/driver/sim.rs b/rs/moq-tokio/src/quic/driver/sim.rs index 417e6569f2d..0b75922f392 100644 --- a/rs/moq-tokio/src/quic/driver/sim.rs +++ b/rs/moq-tokio/src/quic/driver/sim.rs @@ -1,7 +1,9 @@ //! Test support, enabled by the `sim` feature: QUIC with no sockets, endpoints on kio's //! simulator joined by seeded, impaired paths //! -//! A [`Network`] binds a [`Socket`] per address for [`Endpoint::new`](super::endpoint::Endpoint::new). +//! A [`Network`] binds a [`Socket`] per address for [`Endpoint::new`](super::endpoint::Endpoint::new), +//! and a [`Host`] is one machine's address on it, where a [`Client`](crate::Client) or +//! [`Server`](crate::Server) binds its QUIC endpoint in place of a UDP socket. //! Each datagram crosses the [`Path`] from its sender to its destination: the [`Link`]'s //! bottleneck queue, a fixed one-way latency, then its loss, duplication, jitter, and //! reordering, all drawn from the network's seed. With each endpoint's `rng_seed` set too, a @@ -15,14 +17,14 @@ use std::{ collections::{HashMap, VecDeque}, fmt, io::{self, IoSliceMut}, - net::SocketAddr, + net::{IpAddr, SocketAddr}, sync::{Arc, Mutex}, task::{Context, Poll, Waker}, }; pub use moq_quic::sim::{Bottleneck, Link, QueueStats}; use moq_sock::udp::{EcnCodepoint, RecvMeta, Transmit}; -use rand::{SeedableRng, rngs::StdRng}; +use rand::{RngExt, SeedableRng, rngs::StdRng}; use super::{Duration, endpoint::AsyncUdpSocket}; @@ -42,14 +44,19 @@ pub struct Network(Arc>); struct State { /// Draws every impairment, so the seed replays them rng: StdRng, - hosts: HashMap, + hosts: HashMap, /// Keyed by sender, then destination. A pair never set is a clean path with no latency. paths: HashMap<(SocketAddr, SocketAddr), Path>, + /// Where the search for a free ephemeral port starts, so a port is not reused soon after it closes + ephemeral: u16, } +/// The first port [`Host::bind`] hands out for port 0, the start of the IANA dynamic range +const EPHEMERAL: u16 = 49152; + /// A bound address #[derive(Default)] -struct Host { +struct Bound { /// Datagrams that arrived and await a receive, in arrival order inbound: VecDeque, /// The endpoint waiting for the next arrival @@ -69,16 +76,25 @@ impl Network { rng: StdRng::seed_from_u64(seed), hosts: HashMap::new(), paths: HashMap::new(), + ephemeral: EPHEMERAL, }))) } + /// The machine at `ip`, which binds its endpoints' sockets on this network + pub fn host(&self, ip: IpAddr) -> Host { + Host { + network: self.clone(), + ip, + } + } + /// Bind `addr`, which receives until the socket drops /// /// # Panics /// /// If `addr` is already bound. pub fn bind(&self, addr: SocketAddr) -> Socket { - let previous = self.0.lock().unwrap().hosts.insert(addr, Host::default()); + let previous = self.0.lock().unwrap().hosts.insert(addr, Bound::default()); assert!(previous.is_none(), "{addr} is already bound"); Socket { network: self.clone(), @@ -133,6 +149,53 @@ impl Network { } } +/// One machine's address on a [`Network`] +/// +/// Set it as [`client::Config::sim`](crate::client::Config::sim) or +/// [`server::Config::sim`](crate::server::Config::sim) to bind that side's QUIC endpoint +/// here, on the configured bind port, instead of opening a UDP socket. Each endpoint it binds +/// draws its random seed from the network's, so a seed replays the endpoints too. +#[derive(Clone, Debug)] +pub struct Host { + network: Network, + ip: IpAddr, +} + +impl Host { + /// The machine's address + pub fn ip(&self) -> IpAddr { + self.ip + } + + /// Bind `port`, or a free ephemeral one for 0, and draw a seed for the endpoint using it + pub(crate) fn bind(&self, port: u16) -> io::Result<(Socket, [u8; 32])> { + let state = &mut *self.network.0.lock().unwrap(); + let port = match port { + 0 => { + let range = u16::MAX - EPHEMERAL + 1; + let free = (0..range) + .map(|offset| EPHEMERAL + (state.ephemeral - EPHEMERAL + offset) % range) + .find(|&port| !state.hosts.contains_key(&SocketAddr::new(self.ip, port))) + .ok_or_else(|| io::Error::from(io::ErrorKind::AddrInUse))?; + state.ephemeral = free.checked_add(1).unwrap_or(EPHEMERAL); + free + } + port => port, + }; + let addr = SocketAddr::new(self.ip, port); + if state.hosts.contains_key(&addr) { + return Err(io::ErrorKind::AddrInUse.into()); + } + state.hosts.insert(addr, Bound::default()); + let seed = state.rng.random(); + let socket = Socket { + network: self.network.clone(), + addr, + }; + Ok((socket, seed)) + } +} + impl fmt::Debug for Network { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { f.debug_struct("Network").finish_non_exhaustive() diff --git a/rs/moq-tokio/src/server.rs b/rs/moq-tokio/src/server.rs index bc429ec99f3..6a8015de5f9 100644 --- a/rs/moq-tokio/src/server.rs +++ b/rs/moq-tokio/src/server.rs @@ -201,6 +201,11 @@ pub struct Config { /// take its [`webrtc`](crate::quic::demux::Demux::webrtc) feed or read its counters. #[cfg(feature = "quic")] pub demux: crate::quic::demux::Demux, + + /// The simulated host the QUIC endpoint binds on, at the configured bind port, in place of + /// a UDP socket. For tests on `kio::clock::sim`; see [`crate::quic::driver::sim`]. + #[cfg(all(feature = "sim", feature = "quic"))] + pub sim: Option, } impl Config { @@ -229,6 +234,8 @@ impl Config { stats: self.stats.clone(), admission: self.admission.clone(), demux: self.demux.clone(), + #[cfg(feature = "sim")] + sim: self.sim.clone(), } } } @@ -283,6 +290,8 @@ impl Server { stats, #[cfg(feature = "quic")] demux, + #[cfg(all(feature = "sim", feature = "quic"))] + sim, } = config; // Refuse here rather than in `init`, so a caller that skipped its own check @@ -389,6 +398,8 @@ impl Server { member, tls: tls.clone().expect("QUIC needs TLS"), demux: demux.clone(), + #[cfg(feature = "sim")] + sim: sim.clone(), }) }) .transpose()?; @@ -482,7 +493,7 @@ impl Server { } loop { let request = listener.accept().await?; - crate::runtime::spawn(crate::panic::isolate(async move { + crate::runtime::Spawner::current().spawn(crate::panic::isolate(async move { if let Err(err) = serve_session(request).await { tracing::warn!(%err, "session ended with error"); } @@ -1687,7 +1698,7 @@ impl Request { } let (session, driver) = crate::runtime::scope(handshake.ok()).await?; use tracing::Instrument; - crate::runtime::spawn( + crate::runtime::Spawner::current().spawn( async move { crate::panic::isolate(async move { let _active = active; diff --git a/rs/moq-tokio/tests/integration/main.rs b/rs/moq-tokio/tests/integration/main.rs index 7b8245dc69b..302aa1b86dc 100644 --- a/rs/moq-tokio/tests/integration/main.rs +++ b/rs/moq-tokio/tests/integration/main.rs @@ -20,6 +20,7 @@ mod reset_secret; mod runtime; mod scream; mod sim; +mod sim_client; mod stream_code; mod stream_limits; mod subscription_end_integrity; diff --git a/rs/moq-tokio/tests/integration/sim_client.rs b/rs/moq-tokio/tests/integration/sim_client.rs new file mode 100644 index 00000000000..4d8ecec11f6 --- /dev/null +++ b/rs/moq-tokio/tests/integration/sim_client.rs @@ -0,0 +1,334 @@ +//! moq-tokio's [`Client`](moq_tokio::Client) and [`Server`](moq_tokio::Server) over QUIC on kio's +//! simulator: each binds its endpoint on a simulated host instead of a UDP socket, so the whole +//! dial, from the Happy Eyeballs race through the reconnect loop, runs with no sockets and no +//! host clock. + +#![cfg(feature = "quic")] + +use std::{ + net::{IpAddr, Ipv4Addr, SocketAddr}, + time::Duration, +}; + +use kio::clock::Instant; +use moq_net::{Timescale, Timestamp, track}; +use moq_tokio::quic::driver::sim::{Link, Network, Path}; + +const CLIENT: IpAddr = IpAddr::V4(Ipv4Addr::new(10, 0, 0, 2)); +const PORT: u16 = 443; + +/// The groups the publisher writes, one every [`INTERVAL`] +const GROUPS: u64 = 20; +const INTERVAL: Duration = Duration::from_millis(100); +const FRAME: usize = 4_000; + +/// How long a run may take before it counts as stalled, in simulated time +const TIMEOUT: Duration = Duration::from_secs(60); + +fn server_addr(last: u8) -> SocketAddr { + SocketAddr::new(IpAddr::V4(Ipv4Addr::new(10, 0, 0, last)), PORT) +} + +/// A publisher dials a server over raw QUIC and over WebTransport, across a lossy path, and the +/// server receives every group whole. +#[test] +fn transports() { + for (scheme, transport) in [ + ("moqt", moq_tokio::Transport::Quic), + ("https", moq_tokio::Transport::WebTransport), + ] { + for seed in 0..2 { + kio::clock::sim::run(async move { + let network = Network::new(seed); + let server = server_addr(1); + let mut server = Server::start(&network, server).await; + let (connection, publisher) = + publish(&network, &[server.addr], scheme, &server.fingerprints, |client| { + let lossy = Path { + latency: Duration::from_millis(30), + link: lossy_link(), + }; + network.path(client, server.addr, |path| *path = lossy.clone()); + network.path(server.addr, client, |path| *path = lossy); + }); + let groups = kio::clock::timeout(TIMEOUT, async { + let session = server.sessions.recv().await.expect("the server stopped"); + let ((), groups) = futures::join!(publisher.run(), read(&server.origin)); + drop(session); + groups + }) + .await + .unwrap_or_else(|_| panic!("{scheme} seed {seed}: the track never finished")); + assert_eq!(groups, (0..GROUPS).collect::>(), "{scheme} seed {seed}"); + assert_eq!(connection.transport(), Some(transport), "{scheme} seed {seed}"); + }); + } + } +} + +/// Happy Eyeballs across two addresses: the first answers over a path whose handshake takes +/// seconds, so the client dials the second after the race delay, and that one wins. +#[test] +fn happy_eyeballs() { + kio::clock::sim::run(async { + let network = Network::new(1); + let (slow, fast) = (server_addr(1), server_addr(3)); + let mut slow_server = Server::start(&network, slow).await; + let mut fast_server = Server::start(&network, fast).await; + let mut fingerprints = slow_server.fingerprints.clone(); + fingerprints.extend(fast_server.fingerprints.clone()); + + let start = Instant::now(); + let (connection, _publisher) = publish(&network, &[slow, fast], "moqt", &fingerprints, |client| { + for server in [slow, fast] { + let latency = match server == slow { + true => Duration::from_secs(2), + false => Duration::from_millis(20), + }; + let path = Path { + latency, + link: Link::default(), + }; + network.path(client, server, |p| *p = path.clone()); + network.path(server, client, |p| *p = path); + } + }); + let connection = kio::clock::timeout(TIMEOUT, connection.established()) + .await + .expect("never connected") + .expect("the dial failed"); + let elapsed = Instant::now() - start; + // The second address is dialed only once the race delay lapses, then needs a handshake. + assert!( + elapsed >= Duration::from_millis(250) && elapsed < Duration::from_secs(2), + "connected after {elapsed:?}" + ); + let _session = fast_server.sessions.recv().await.expect("the fast server stopped"); + // The slow handshake never finished before the winner's cancelled it. + kio::clock::sleep(Duration::from_secs(10)).await; + assert!( + slow_server.sessions.try_recv().is_err(), + "the slow address also connected" + ); + assert_eq!(connection.connects(), 1); + }); +} + +/// The server restarts on the same address: the client's reconnect loop redials it over QUIC +/// and the broadcast reaches the new server. +#[test] +fn reconnect() { + kio::clock::sim::run(async { + let network = Network::new(1); + let addr = server_addr(1); + // One identity across the restart, so the pinned fingerprint still matches. + let tls = identity(); + let mut server = Server::with_identity(&network, addr, tls.clone()).await; + let (connection, publisher) = publish(&network, &[addr], "moqt", &server.fingerprints.clone(), |_| {}); + let _publisher = publisher; + let session = kio::clock::timeout(TIMEOUT, server.sessions.recv()) + .await + .expect("never connected") + .expect("the server stopped"); + announced(&server.origin).await; + assert_eq!(connection.connects(), 1); + + server.stop().await; + drop(session); + let mut server = Server::with_identity(&network, addr, tls).await; + let _session = kio::clock::timeout(TIMEOUT, server.sessions.recv()) + .await + .unwrap_or_else(|_| panic!("never reconnected")) + .expect("the server stopped"); + announced(&server.origin).await; + assert_eq!(connection.connects(), 2); + }); +} + +/// A server accepting on a simulated host, collecting what its sessions publish +struct Server { + addr: SocketAddr, + fingerprints: Vec, + origin: moq_net::origin::Producer, + sessions: futures::channel::mpsc::UnboundedReceiver, + stop: Option>, + stopped: Option>, +} + +/// A fresh served identity for `localhost` +fn identity() -> moq_tokio::tls::Serve { + let _ = rind_rustls::default_provider().unwrap().install_default(); + let mut listen = moq_tokio::tls::Listen::default(); + listen.generate = vec!["localhost".into()]; + listen.load().expect("generate identity") +} + +impl Server { + /// A server with a fresh identity + async fn start(network: &Network, addr: SocketAddr) -> Self { + Self::with_identity(network, addr, identity()).await + } + + async fn with_identity(network: &Network, addr: SocketAddr, tls: moq_tokio::tls::Serve) -> Self { + let mut config = moq_tokio::server::Config::default(); + config.listen.bind = Some(moq_tokio::listen::Bind::Addr(addr)); + config.tls = Some(tls); + config.sim = Some(network.host(addr.ip())); + let mut listener = config.init().expect("init server").listen().await.expect("listen"); + assert_eq!(listener.local_addr().expect("bound"), addr); + let fingerprints = listener.certificates().fingerprints(); + + let origin = moq_tokio::origin::spawn(); + let (accepted, sessions) = futures::channel::mpsc::unbounded(); + let (stop, mut stopping) = futures::channel::oneshot::channel(); + let (done, stopped) = futures::channel::oneshot::channel(); + let subscriber = origin.clone(); + drop(kio::clock::sim::spawn(async move { + loop { + futures::select_biased! { + _ = stopping => break, + request = futures::FutureExt::fuse(listener.accept()) => { + let Ok(request) = request else { break }; + if let Ok(session) = request.with_subscriber(subscriber.clone()).ok().await { + let _ = accepted.unbounded_send(session); + } + } + } + } + listener.close().await; + let _ = done.send(()); + })); + Self { + addr, + fingerprints, + origin, + sessions, + stop: Some(stop), + stopped: Some(stopped), + } + } + + /// Close every connection and release the address. + async fn stop(&mut self) { + let _ = self.stop.take().expect("running").send(()); + let _ = self.stopped.take().expect("running").await; + } +} + +/// A publisher on [`CLIENT`] dialing `servers` in order over `scheme`, trusting `fingerprints`. +/// +/// The client binds a fixed address, which `paths` is handed to set the paths from and to it. +fn publish( + network: &Network, + servers: &[SocketAddr], + scheme: &str, + fingerprints: &[String], + paths: impl FnOnce(SocketAddr), +) -> (moq_tokio::Connection, Publisher) { + let mut connect = moq_tokio::connect::Config::default(); + connect.bind = Some(SocketAddr::new(CLIENT, 4433)); + connect.tls.fingerprint = fingerprints.to_vec(); + let mut config = moq_tokio::client::Config::default(); + config.connect = connect; + config.sim = Some(network.host(CLIENT)); + paths(SocketAddr::new(CLIENT, 4433)); + + let origin = moq_tokio::origin::spawn(); + let broadcast = origin + .route("live", moq_net::origin::RouteOptions::default()) + .expect("route"); + let track = broadcast + .broadcast() + .create_track( + "video", + track::Info::default() + .with_timescale(Timescale::MILLI) + .with_max_age(None), + ) + .expect("track"); + broadcast.announce().expect("announce"); + + let url: url::Url = format!("{scheme}://localhost:{PORT}/").parse().unwrap(); + let addr = moq_tokio::connect::Addr::pinned(url, servers.iter().copied()).expect("pinned"); + let client = config.init().expect("init client").with_publisher(&origin); + let connection = client.connect(addr); + ( + connection, + Publisher { + _broadcast: broadcast, + track, + }, + ) +} + +struct Publisher { + _broadcast: moq_net::origin::Route, + track: moq_net::track::Producer, +} + +impl Publisher { + /// Write [`GROUPS`] one-frame groups once someone subscribes, then finish the track. + async fn run(&self) { + let mut source = self.track.clone(); + while source.subscription_changed().await.unwrap().is_none() {} + for sequence in 0..GROUPS { + let mut group = source.create_group(moq_net::group::Info { sequence }).unwrap(); + let timestamp = Timestamp::from_millis(sequence * INTERVAL.as_millis() as u64).unwrap(); + group.write_frame(timestamp, vec![sequence as u8; FRAME]).unwrap(); + group.finish().unwrap(); + kio::clock::sleep(INTERVAL).await; + } + let _source = source.finish().unwrap(); + } +} + +/// Wait until `origin` holds the publisher's broadcast. +async fn announced(origin: &moq_net::origin::Producer) { + let mut announced = origin.consume().announced(); + let present = async { + while let Some(event) = announced.next().await { + if let moq_net::announce::Event::Present(route) = event + && route.prefix.as_str() == "live" + { + return; + } + } + panic!("the origin closed"); + }; + kio::clock::timeout(TIMEOUT, present).await.expect("never announced"); +} + +/// Subscribe to the publisher's track through `origin` and return each whole group's sequence. +async fn read(origin: &moq_net::origin::Producer) -> Vec { + let remote = origin.consume().request("live", None).await.unwrap(); + let mut reader = remote + .track("video") + .unwrap() + // A budget spanning the run, so recovery has to deliver every group, however late. + .subscribe(track::Subscription::default().with_delay(track::Delay::default().with_max(TIMEOUT))) + .await + .unwrap(); + let mut groups = Vec::new(); + while let Some(mut group) = reader.recv_group().await.unwrap() { + let sequence = group.sequence; + let frame = group.read_frame().await.unwrap().expect("a group is cut short"); + assert_eq!(frame.payload.len(), FRAME, "group {sequence} is cut short"); + assert!( + frame.payload.iter().all(|&byte| byte == sequence as u8), + "group {sequence} is corrupted" + ); + groups.push(sequence); + } + groups.sort(); + groups +} + +fn lossy_link() -> Link { + let mut link = Link::default(); + link.loss = 0.05; + link.duplicate = 0.01; + link.jitter = Duration::from_millis(5); + link.reorder = 0.02; + link.reorder_delay = Duration::from_millis(20); + link +}