From 1029e7428149b1ac4c51d709d854094f14f10291 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 10:21:55 -0800 Subject: [PATCH 01/10] chore(quest): claim quest/m1/session-policy/connect.md Quest-Claim: 813cecbae86d83b8 From 3aaf14af6ff7a2d1ea11f3f917aeb7a6793ada46 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 11:39:45 -0800 Subject: [PATCH 02/10] feat(net): connect policy lives in moq-net as a poll machine moq_net::Reconnect owns the connect deadline, the address walk, backoff, GOAWAY handling and token refresh, doing no I/O. moq-tokio's Connection dials what it asks for and publishes what it does. Co-Authored-By: Claude Opus 5.5 --- doc/lib/rs/moq-net.md | 1 + quest/m1/session-policy/README.md | 1 - quest/m1/session-policy/connect.md | 32 - quest/m1/session-policy/tokio-handoff.md | 1 - quest/m3/uring-cluster-dial.md | 5 +- rs/moq-net/src/lib.rs | 4 + rs/moq-net/src/reconnect.rs | 951 ++++++++++++ rs/moq-net/src/reconnect/redirect.rs | 515 +++++++ rs/moq-net/src/reconnect/tests.rs | 676 +++++++++ rs/moq-net/src/session.rs | 37 +- rs/moq-tokio/src/client.rs | 112 +- rs/moq-tokio/src/connect.rs | 10 + rs/moq-tokio/src/connection.rs | 1765 +++------------------- rs/moq-tokio/src/error.rs | 23 +- 14 files changed, 2458 insertions(+), 1675 deletions(-) delete mode 100644 quest/m1/session-policy/connect.md create mode 100644 rs/moq-net/src/reconnect.rs create mode 100644 rs/moq-net/src/reconnect/redirect.rs create mode 100644 rs/moq-net/src/reconnect/tests.rs diff --git a/doc/lib/rs/moq-net.md b/doc/lib/rs/moq-net.md index fe0dbd9c88c..dee520378ed 100644 --- a/doc/lib/rs/moq-net.md +++ b/doc/lib/rs/moq-net.md @@ -27,6 +27,7 @@ What you use it for, beyond what the concept page already describes: - **Demand says whether anyone is watching.** `demand()` on a track, group, or broadcast says whether a subscriber is attached. First-party capture does not wait for it: video encodes while enabled and audio keeps raw PCM to encode when someone listens, so a late subscription's `delay.max` or a FETCH finds the recent past. Demand gates bandwidth reservations, transcode rungs, and on-demand video where a publisher opts in, which saves the encode but leaves nothing from before the first subscriber. Holding a broadcast or track consumer is not demand: a track consumer requests nothing until it queries, subscribes, or fetches, and those count until dropped. A shared fetch stays up until its last reader leaves. Demand rolls up: a frame being read keeps its group wanted, a group a subscription or fetch handed out keeps its track wanted until read to its end, and any wanted track keeps its broadcast wanted. - **Use the model without a runtime.** Disable default features on `moq-net` to use its `poll_*` APIs and origin driver with caller-supplied time. The default `async` feature adds async helpers, `time::run`, and both wire sessions. - **You drive the session, or `moq-tokio` does.** `connect` and `accept` return a session plus a driver that never reads the clock itself. `moq_net::time::run` polls it on tokio. `moq-tokio` hides that. A custom transport implements `moq_net::transport::poll`, and `moq_net::transport::memory::pair` links two sessions in one process with no sockets, for tests or in-process links. +- **Reconnect policy comes with it.** `moq_net::Reconnect` decides which of a peer's addresses to dial, how long each dial and handshake may take, when to back off and give up, and how to follow a GOAWAY, without doing I/O: you dial each `Attempt` it asks for and report back. `moq-tokio`'s `Connection` runs it, so a client on another runtime gets the same policy rather than a copy. - **A session bounds what its peer can hold.** Each request holds a stream, so the transport's stream credit bounds subscriptions and FETCHes. `session::Limits` (via `Client::with_limits` and `Server::with_limits`) bounds the announcements per session. The default suits a relay mesh, so lower it for untrusted peers. Past it, the session closes with `TOO_MANY_REQUESTS`. Peer-declared lengths are capped before they are buffered. - **Authorization is in band.** On moq-lite 07 (`moq-lite-07-wip`, opt-in) and on moq-transport draft-18+ with the [MoQ Auth extension](/draft/moq-auth), `session.auth()` presents tokens without reconnecting and watches the grant the peer sent back. `add(token)` resolves with the token and how the peer took it: `Answer::Granted` on its AUTH stream, or `Answer::Requests` on moq-transport without AUTH, where only a request that names it carries it. A handle passed to `Client::with_auth` outlives each session, so its tokens, and the routes naming them, carry across your own reconnects. A server verifies tokens itself by taking `handshake.requests()` before `ok()`. `authorize(&grant)` narrows or widens a live session on any version: what falls outside resets with `Unauthorized` and the session stays up. A client that publishes outside its grant closes the session, naming the path. - **A graceful close waits.** `session.close().await` withdraws announcements and drains for up to one second. `moq-lite-05` and newer wait for the subscriber to receive finished tails; older Lite versions and IETF wait only for transport ACKs. `abort` ends immediately. diff --git a/quest/m1/session-policy/README.md b/quest/m1/session-policy/README.md index 62ba3e31c82..e47f1eccb28 100644 --- a/quest/m1/session-policy/README.md +++ b/quest/m1/session-policy/README.md @@ -64,6 +64,5 @@ Research (2026-11-04, `origin/main` at e6be13d5f8): ## Required - [Accept policy in moq-net](/quest/m1/session-policy/accept.md) - one accept deadline, admission bound and window widening for every transport -- [Connect policy in moq-net](/quest/m1/session-policy/connect.md) - the connect deadline and reconnect as a poll machine - [The io_uring relay path uses moq-net's policy](/quest/m1/session-policy/uring.md) - deletes `uring.rs`'s copy - [moq-tokio becomes a thin shell](/quest/m1/session-policy/tokio-handoff.md) - sockets, TLS, accept loops and spawning, plus the transport handoffs WebRTC needs diff --git a/quest/m1/session-policy/connect.md b/quest/m1/session-policy/connect.md deleted file mode 100644 index aab876ced54..00000000000 --- a/quest/m1/session-policy/connect.md +++ /dev/null @@ -1,32 +0,0 @@ -# [L] Connect policy in moq-net - -## Goal - -moq-net owns client connect policy as poll machines: the connect deadline over -dial plus handshake, and reconnect with backoff, GOAWAY handling and the -candidate walk. moq-net does no I/O: the caller supplies each dial attempt and -the machine decides when to start, abandon, back off or move to the next -candidate. moq-tokio's `Client`/`Connection` drive these machines instead of -implementing the policy. - -## Plan - -Part of [Session policy lives in moq-net](/quest/m1/session-policy/README.md); -see it for the decisions and research. Decided 2026-11-04 (maintainer): -reconnect moves too, not just the deadline. - -- Today's policy: the connect deadline in moq-tokio's `client.rs` (default 30 s - in `connect.rs`), and `Backoff`, `Goaway`, `run`, `attempt_timeout`, - `settles_walk` and `refreshed` in `connection.rs`. All of it runs on - `kio::clock` and `select!`; only the dialing (DNS, Happy Eyeballs, sockets, - TLS) is tokio-specific and stays in moq-tokio. -- Keep the current behaviour and defaults; port the existing - `connection.rs` tests to the moq-net machines on kio's simulator, so the same - timings are asserted without tokio. -- This also gives a future io_uring client - ([uring-cluster-dial](/quest/m3/uring-cluster-dial.md)) the policy to reuse - rather than copy. - -Public API: breaking in moq-net's `Client` and new reconnect types; moq-tokio's -`Connection` API keeps its shape where possible, with MIGRATE notes otherwise. -Wire: none. diff --git a/quest/m1/session-policy/tokio-handoff.md b/quest/m1/session-policy/tokio-handoff.md index efcdec89f3d..0f0849d2c4a 100644 --- a/quest/m1/session-policy/tokio-handoff.md +++ b/quest/m1/session-policy/tokio-handoff.md @@ -37,4 +37,3 @@ it re-exported, with MIGRATE notes. Wire: none. ## Required - [Accept policy in moq-net](/quest/m1/session-policy/accept.md) - the accept policy the handoff runs -- [Connect policy in moq-net](/quest/m1/session-policy/connect.md) - the connect policy the handoff runs diff --git a/quest/m3/uring-cluster-dial.md b/quest/m3/uring-cluster-dial.md index 0234f2084d6..12823291add 100644 --- a/quest/m3/uring-cluster-dial.md +++ b/quest/m3/uring-cluster-dial.md @@ -13,8 +13,9 @@ lands only if the measurement shows a win. serves inbound traffic, so a ring relay forwards its cluster media through tokio. A port needs the ring client to grow what `moq-tokio` provides: DNS and address racing, reconnect, and fingerprint pinning (the raw ring -client takes one `SocketAddr`). Share that policy with `moq-tokio` rather -than duplicate it. +client takes one `SocketAddr`). The reconnect half already lives in +moq-net as `moq_net::Reconnect`, which a ring client drives with its own +dials; share the rest with `moq-tokio` rather than duplicate it. Measure first with a prototype dialing a fixed address from a worker: CPU per Gbps and latency on the relay-to-relay fanout shape, ring against diff --git a/rs/moq-net/src/lib.rs b/rs/moq-net/src/lib.rs index 96c10f71c14..adf31ab639c 100644 --- a/rs/moq-net/src/lib.rs +++ b/rs/moq-net/src/lib.rs @@ -133,6 +133,8 @@ mod test_interop; mod util; mod version; +#[cfg(feature = "async")] +pub mod reconnect; #[cfg(feature = "async")] pub mod server; #[cfg(feature = "async")] @@ -151,6 +153,8 @@ pub use instance::{Instance, InvalidInstance}; pub use model::*; pub use path::{InvalidPattern, Path, Pattern, Patterns}; #[cfg(feature = "async")] +pub use reconnect::Reconnect; +#[cfg(feature = "async")] pub use server::Server; #[cfg(feature = "async")] pub use session::Session; diff --git a/rs/moq-net/src/reconnect.rs b/rs/moq-net/src/reconnect.rs new file mode 100644 index 00000000000..08c05f25c49 --- /dev/null +++ b/rs/moq-net/src/reconnect.rs @@ -0,0 +1,951 @@ +//! Client connect policy: which of a peer's addresses to dial, how long each dial may +//! take, when to back off and when to give up, and how to follow a GOAWAY. +//! +//! [`Reconnect`] is a poll machine that does no I/O. The caller dials each [`Attempt`] +//! it asks for, reports the result with [`Reconnect::connected`] or +//! [`Reconnect::failed`], and otherwise polls it with the current time. The machine +//! watches each live session for its close and GOAWAY itself, keeps a drained +//! predecessor serving through its handover, and drops neither on the caller's behalf +//! until it reports [`Action::Done`]. + +mod redirect; +#[cfg(test)] +mod tests; + +pub use redirect::Redirect; + +use std::{collections::VecDeque, task::Poll, time::Duration}; + +use rand::RngExt; +use url::Url; + +use crate::{Error, Session, SessionError, auth, goaway, time::Instant}; +use redirect::{Redirects, Shown, endpoint}; + +/// How long one address gets before the walk moves to the peer's next one. +/// +/// A black-holed address answers nothing rather than refusing, so without a bound the +/// transport's idle timeout would decide how long the remaining addresses wait. Only an +/// attempt with a later address to fall back on is worth bounding: cutting the last one +/// short just turns a slow connect into a failed one, on every reconnect cycle. +const ATTEMPT: Duration = Duration::from_secs(5); + +/// When to redial after a failure, and when to give up. +/// +/// `initial` and `multiplier` set the wait before each retry, `max` caps it, and +/// `healthy` decides which sessions count as failed attempts. The delays carry jitter, +/// so a fleet knocked offline together does not reconnect in lockstep. `timeout` bounds +/// every failure; a settled rejection, malformed protocol input, or invalid local +/// configuration stops retrying. A GOAWAY naming no new URI is no failure: a restarting +/// peer sends one to each session until it goes, and each of those serves, so it +/// restarts the timeout instead. +/// +/// A GOAWAY off a healthy session migrates at once, unless it sends the connection back +/// to an endpoint that already redirected it elsewhere and the session lived less than +/// `max`. Two peers redirecting to each other thus wait out this backoff and count +/// against the timeout. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct Backoff { + /// Delay before the first redial. Later delays grow from it, so zero turns retry + /// waiting off entirely; an immediate close still counts as a failed attempt and + /// spends the timeout. Defaults to 1 s. + pub initial: Duration, + /// Growth of the delay after each failure. 0 or 1 keeps it constant. Defaults to 2. + pub multiplier: u32, + /// Cap on the delay, which caps `initial` too. Zero leaves it uncapped. Defaults to 5 s. + pub max: Duration, + /// How long a session must stay up to count as healthy. Must be nonzero. Defaults to 1 s. + /// + /// A healthy session's end resets the delay and the timeout. One that ends sooner is a + /// failed attempt, so a peer that accepts and then drops every session still escalates + /// the delay and exhausts the timeout. + pub healthy: Duration, + /// How long to keep retrying before giving up. Zero retries forever. Defaults to 10 s. + /// + /// No new walk of the peer's addresses starts once it has passed, but one under way + /// finishes: each address but the last gets up to 5 s, and the last runs to + /// [`Config::timeout`], so a handshake slower than this still lands. Giving up can + /// therefore take up to one whole walk longer. Resets after a healthy session outside a + /// redirect loop. + pub timeout: Duration, +} + +impl Default for Backoff { + fn default() -> Self { + Self { + initial: Duration::from_secs(1), + multiplier: 2, + max: Duration::from_secs(5), + healthy: Duration::from_secs(1), + timeout: Duration::from_secs(10), + } + } +} + +/// How a connection reacts to a peer's GOAWAY. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct Goaway { + /// What to do with the URI the GOAWAY names. Defaults to [`Redirect::SameHost`]. + pub redirect: Redirect, + /// The longest the old session keeps serving after a GOAWAY. Defaults to 10 s. + /// + /// It closes sooner, once its replacement is up and it carries no subscription or + /// fetch. A GOAWAY naming a shorter deadline wins, since the peer force-closes at its + /// own deadline regardless, but a longer one does not extend it. + pub handover: Duration, +} + +impl Default for Goaway { + fn default() -> Self { + Self { + redirect: Redirect::default(), + handover: Duration::from_secs(10), + } + } +} + +impl Goaway { + /// The handover window for a GOAWAY naming `deadline`: whichever is sooner. + fn handover(&self, deadline: Option) -> Duration { + deadline.map_or(self.handover, |deadline| deadline.min(self.handover)) + } +} + +/// What to do when a peer's GOAWAY names no URI, meaning it is restarting where we reached it. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +#[non_exhaustive] +pub enum Restart { + /// Redial the same addresses at once while the drained session keeps serving. Right for + /// an address that can lead to another node, such as a fleet name a fresh resolve moves + /// off the drained one. The default. + #[default] + Redial, + /// Keep serving on the drained session until the peer closes it, then reconnect as after + /// any drop. Right for an address that names one node, where a redial only reaches the + /// draining process again. The peer must close the session itself, as a relay does at + /// its drain deadline. + Wait, +} + +/// The whole connect policy for one peer. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct Config { + /// When to redial, and when to give up. + pub backoff: Backoff, + /// How to follow a peer's GOAWAY. + pub goaway: Goaway, + /// What a GOAWAY naming no URI does. + pub restart: Restart, + /// The deadline for one dial and its handshake. Zero waits forever. Defaults to 30 s. + pub timeout: Duration, + /// End when the first session ends instead of redialing. Defaults to false. + /// + /// A GOAWAY then ends it cleanly: with no replacement to dial, leaving is the answer. + pub once: bool, + /// Whether the dial pins a certificate fingerprint, which verifies only the host it + /// was configured for, so a GOAWAY may not move an encrypted dial to another host. + pub pinned: bool, +} + +impl Default for Config { + fn default() -> Self { + Self { + backoff: Backoff::default(), + goaway: Goaway::default(), + restart: Restart::default(), + timeout: Duration::from_secs(30), + once: false, + pinned: false, + } + } +} + +/// One of a peer's addresses, as the caller dials it. +pub trait Target: Clone + From { + /// The URL a dial reaches, which a GOAWAY redirect is judged against. + fn url(&self) -> &Url; + + /// Whether the dial skips resolving the URL for fixed socket addresses, which a + /// GOAWAY redirect cannot carry over. + fn fixed(&self) -> bool; +} + +impl Target for Url { + fn url(&self) -> &Url { + self + } + + fn fixed(&self) -> bool { + false + } +} + +/// What a failed dial means for the rest of the policy, judged by the caller. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub enum Verdict { + /// This address failed; try the next, then back off and retry. + Retry, + /// This address can never work, such as an unsupported scheme. Give up unless + /// another address fails transiently. + Unusable, + /// The peer's answer for every address, such as an HTTP 404: stop the walk, then + /// back off and retry. + Settled, + /// The peer refused the credential: give up unless the tokens changed since the dial. + Refused, + /// Nothing a retry can repair, such as malformed protocol input: give up. + Fatal, +} + +impl From<&Error> for Verdict { + /// Judge an error from the MoQ handshake or a session's close. + fn from(err: &Error) -> Self { + match SessionError::from(err) { + SessionError::Unauthorized | SessionError::Expired | SessionError::MalformedAuthToken => Self::Refused, + SessionError::ProtocolViolation | SessionError::KeyValueFormatting | SessionError::InvalidRequestId => { + Self::Fatal + } + _ => Self::Retry, + } + } +} + +/// A failure the policy itself decides on. +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] +#[non_exhaustive] +pub enum Failure { + /// An address with another to fall back on did not connect in time. + #[error("timed out connecting to {0}")] + Attempt(String), + /// A dial and its handshake outlasted [`Config::timeout`]. + #[error("connect timed out after {0:?}")] + Timeout(Duration), + /// The retry window ran out; carries the last failure's message. + #[error("reconnect timed out after {after:?}{}", last.as_ref().map(|last| format!(": {last}")).unwrap_or_default())] + Exhausted { + /// The retry window. + after: Duration, + /// Why the last attempt failed, if one did. + last: Option, + }, + /// Peers redirected the connection around in a loop. + #[error("redirect loop")] + Loop, + /// The peer redirected a session elsewhere before it was healthy. + #[error("peer redirected immediately")] + Redirected, + /// The redirect a GOAWAY named breaks the [`Redirect`] policy; carries why. + #[error("refused GOAWAY redirect: {0}")] + Refused(&'static str), + /// A GOAWAY named a redirect for a target dialing fixed socket addresses. + #[error("a GOAWAY redirected a target dialing fixed socket addresses")] + Fixed, +} + +/// Invalid input to [`Reconnect::new`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +#[non_exhaustive] +pub enum ConfigError { + /// There is no address to dial. + #[error("no target to dial")] + Empty, + /// [`Backoff::healthy`] is zero, which would count every session as healthy and reset + /// the timeout forever under a peer that drops each one on sight. + #[error("the healthy threshold must be nonzero")] + Healthy, +} + +/// One dial the caller runs for [`Reconnect`]. +#[derive(Clone, Debug)] +pub struct Attempt { + target: A, + url: Url, + token: Option, +} + +impl Attempt { + /// The address to dial. + pub fn target(&self) -> &A { + &self.target + } + + /// The URL to dial, carrying the connection credential in its query when an offered + /// version has no SETUP token. + pub fn url(&self) -> &Url { + &self.url + } + + /// The connection credential for [`Client::with_token`](crate::Client::with_token), if any. + pub fn token(&self) -> Option<&crate::setup::Token> { + self.token.as_ref() + } +} + +/// What [`Reconnect::poll`] asks of the caller. +/// +/// Exhaustive on purpose: a new action is one every driver must handle. +#[derive(Debug)] +pub enum Action { + /// Dial this attempt and report it with [`Reconnect::connected`] or + /// [`Reconnect::failed`], polling meanwhile. Building each session on the tokens' + /// handle, from [`Reconnect::with_tokens`], is the caller's. + Dial(Attempt), + /// Drop the dial in flight: it ran out its deadline, which already counted as its failure. + Abandon, + /// The live session got a GOAWAY and now drains as the [`predecessor`](Reconnect::predecessor) + /// while a replacement is dialed. + Migrating, + /// The predecessor closed or overstayed its handover. + Retired, + /// Nothing serves anymore: the live session ended with this error, or the predecessor + /// retired between sessions. A redial follows. + Offline(Option), + /// Why the policy is waiting to redial: a walk failed or a session ended too soon. + Retrying(E), + /// Poll again by this instant, or on a wake. + Wait(Option), + /// The policy gave up with an error, or ended cleanly. Every later poll waits forever. + Done(Result<(), E>), +} + +/// Client connect policy as a poll machine; see the [module](self) docs. +/// +/// `A` is the caller's address type and `E` its error type, which takes the +/// [`Error`] a session closes with and the [`Failure`]s the policy decides on. +pub struct Reconnect { + config: Config, + targets: Vec, + tokens: Option<(auth::Tokens, crate::Versions)>, + retry: Retry, + redirects: Redirects, + phase: Phase, + /// The failures of the walk under way. + walk: Walk, + /// An old session kept serving after a GOAWAY so its in-flight groups finish. + predecessor: Option, + queue: VecDeque>, +} + +enum Phase { + /// Check the retry window, then walk the targets. + Start, + /// Ask the caller to dial the target at `index`. + Next { index: usize }, + Dialing(Dialing), + Live(Box), + Backoff { + until: Instant, + /// After a restart's wait, the retry window counts from its end. + restart: bool, + }, + /// The policy ended; every later poll waits forever. + Done, +} + +struct Dialing { + index: usize, + deadline: Option<(Instant, Failure)>, + presented: Option, +} + +struct Live { + session: Session, + since: Instant, + index: usize, + /// Dropped once a restart is waited out: it is a latch, so it would keep firing. + goaway: Option, + presented: Option, +} + +struct Walk { + /// The latest failure a retry may repair. + transient: Option, + /// The latest failure of any kind. + last: Option, +} + +impl Default for Walk { + fn default() -> Self { + Self { + transient: None, + last: None, + } + } +} + +/// Why a live session stopped being the live one. +enum Ended { + Closed(Error), + /// The peer sent a GOAWAY; the session is still up and serving. + Goaway(goaway::Goaway), +} + +impl + From> Reconnect { + /// Walk `targets` in order under `config`, starting at `now`. + pub fn new(config: Config, targets: Vec, now: Instant) -> Result { + if targets.is_empty() { + return Err(ConfigError::Empty); + } + if config.backoff.healthy.is_zero() { + return Err(ConfigError::Healthy); + } + Ok(Self { + retry: Retry::new(Pacing::new(&config.backoff).initial, now), + config, + targets, + tokens: None, + redirects: Redirects::default(), + phase: Phase::Start, + walk: Walk::default(), + predecessor: None, + queue: VecDeque::new(), + }) + } + + /// Present `tokens` on every dial offering `versions`. + /// + /// A refused credential then earns another dial once the tokens change, and an ended + /// tokens handle ends the policy at its next walk. + pub fn with_tokens(mut self, tokens: auth::Tokens, versions: crate::Versions) -> Self { + self.tokens = Some((tokens, versions)); + self + } + + /// The live session, if one is serving and has not received a GOAWAY. + pub fn session(&self) -> Option<&Session> { + match &self.phase { + Phase::Live(live) => Some(&live.session), + _ => None, + } + } + + /// The session draining after a GOAWAY, and the deadline its handover ends by. + pub fn predecessor(&self) -> Option<(&Session, Instant)> { + self.predecessor + .as_ref() + .map(|predecessor| (&predecessor.session, predecessor.deadline)) + } + + /// Advance to `now`, returning what the caller must do next. + pub fn poll(&mut self, now: Instant, waiter: &kio::Waiter) -> Action { + loop { + if let Some(action) = self.queue.pop_front() { + return action; + } + + let live = matches!(self.phase, Phase::Live(_)); + if let Some(predecessor) = self.predecessor.as_mut() { + // The replacement serves, so a predecessor carrying nothing closes now. + if live { + predecessor.poll_idle(waiter); + } + if predecessor.poll(now, waiter) { + self.predecessor = None; + self.queue.push_back(Action::Retired); + // Nothing is serving once it retires between sessions, so a caller must + // stop handing out a session that is already closed. + if !live { + self.queue.push_back(Action::Offline(None)); + } + continue; + } + } + let handover = self.predecessor.as_ref().map(|predecessor| predecessor.deadline); + + match &mut self.phase { + Phase::Start => self.start(now), + Phase::Next { index } => { + let index = *index; + return self.dial(now, index); + } + Phase::Dialing(dialing) => match &dialing.deadline { + Some((at, failure)) if now >= *at => { + let failure = failure.clone(); + self.fail(now, failure.into(), Verdict::Retry); + return Action::Abandon; + } + deadline => return Action::Wait(earliest(deadline.as_ref().map(|(at, _)| *at), handover)), + }, + Phase::Live(live) => { + let mut ended = None; + // Checked before the close: a GOAWAY means the session is still up, and + // migrating from it is not the same as reconnecting after it died. + if let Some(Poll::Ready(Ok(msg))) = live.goaway.as_ref().map(|goaway| goaway.poll(waiter)) { + if self.config.restart == Restart::Wait && msg.uri().is_empty() { + tracing::info!("peer is restarting; serving until it closes the session"); + live.goaway = None; + } else { + ended = Some(Ended::Goaway(msg)); + } + } + if ended.is_none() + && let Poll::Ready(err) = live.session.poll_closed(waiter) + { + ended = Some(Ended::Closed(err)); + } + match ended { + Some(ended) => self.end(now, ended), + None => return Action::Wait(handover), + } + } + Phase::Backoff { until, restart } => { + if now < *until { + return Action::Wait(earliest(Some(*until), handover)); + } + let restart = *restart; + self.retry.delay = Pacing::new(&self.config.backoff).next(self.retry.delay); + if restart { + // After the wait, so the window counts only the time after the peer + // stops answering. + self.retry.start = now; + } + self.phase = Phase::Start; + } + Phase::Done => return Action::Wait(None), + } + } + } + + /// Report that the dial asked for connected `session`, at `now`. + /// + /// Ignored unless a dial is in flight, as after [`Action::Abandon`]. + pub fn connected(&mut self, now: Instant, session: Session) { + if !matches!(self.phase, Phase::Dialing(_)) { + return; + } + let Phase::Dialing(dialing) = std::mem::replace(&mut self.phase, Phase::Start) else { + unreachable!("checked above"); + }; + if let Some((tokens, _)) = &self.tokens { + tokens.connected(&session); + } + self.walk = Walk::default(); + self.phase = Phase::Live(Box::new(Live { + goaway: Some(session.draining()), + session, + since: now, + index: dialing.index, + presented: dialing.presented, + })); + } + + /// Report that the dial asked for failed with `err`, judged as `verdict`. + /// + /// Ignored unless a dial is in flight, as after [`Action::Abandon`]. + pub fn failed(&mut self, now: Instant, err: E, verdict: Verdict) { + if matches!(self.phase, Phase::Dialing(_)) { + self.fail(now, err, verdict); + } + } + + /// Start a walk, unless the retry window is spent or the tokens ended. + fn start(&mut self, now: Instant) { + let timeout = self.config.backoff.timeout; + if !timeout.is_zero() && now.saturating_duration_since(self.retry.start) >= timeout { + return self.done(Err(self.exhausted())); + } + // An ended auth handle refuses every later session, so no dial can recover. A fresh + // watch only reports an error once its handle has ended. + if let Some((tokens, _)) = &self.tokens + && let Poll::Ready(Err(err)) = tokens.handle().grant().poll_changed(&kio::Waiter::noop()) + { + return self.done(Err(err.into())); + } + self.phase = Phase::Next { index: 0 }; + } + + fn dial(&mut self, now: Instant, index: usize) -> Action { + let target = self.targets[index].clone(); + tracing::info!(peer = %Shown(target.url()), "connecting"); + + // Read before the dial starts its session's driver, so a request the session sends + // early waits on the held tokens' answers rather than being refused outside the + // connection credential's grant. + let presented = self + .tokens + .as_ref() + .map(|(tokens, versions)| tokens.dial(target.url(), versions)); + let (url, token) = match &presented { + Some(presented) => (presented.url.clone(), presented.token.clone()), + None => (target.url().clone(), None), + }; + + // The connect deadline covers the dial and the handshake, for every transport: it is + // the only bound some of them have. The fixed bound applies only with a fallback. + let connect = Some(self.config.timeout).filter(|timeout| !timeout.is_zero()); + let attempt = (index + 1 < self.targets.len()).then_some(ATTEMPT); + let deadline = match (attempt, connect) { + (Some(attempt), Some(connect)) if connect <= attempt => Some((now + connect, Failure::Timeout(connect))), + (Some(attempt), _) => Some((now + attempt, Failure::Attempt(Shown(target.url()).to_string()))), + (None, Some(connect)) => Some((now + connect, Failure::Timeout(connect))), + (None, None) => None, + }; + + self.phase = Phase::Dialing(Dialing { + index, + deadline, + presented, + }); + Action::Dial(Attempt { target, url, token }) + } + + /// The dial in flight failed: move on to the next target, or end the walk. + fn fail(&mut self, now: Instant, err: E, verdict: Verdict) { + let Phase::Dialing(dialing) = std::mem::replace(&mut self.phase, Phase::Start) else { + unreachable!("checked by the caller"); + }; + match verdict { + // The peer's answer, not this address's, so unless it invites another attempt it + // settles the whole walk. Carrying on would offer the same rejected credentials at + // the peer's other addresses, and a later transport failure would overwrite the + // real reason. + Verdict::Settled | Verdict::Refused | Verdict::Fatal => { + return self.walk_failed(now, err, verdict, dialing.presented.as_ref()); + } + Verdict::Retry => self.walk.transient = Some(err.clone()), + Verdict::Unusable => {} + } + tracing::debug!(peer = %Shown(self.targets[dialing.index].url()), %err, "address unreachable"); + self.walk.last = Some(err); + + if dialing.index + 1 < self.targets.len() { + self.phase = Phase::Next { + index: dialing.index + 1, + }; + return; + } + + // One unusable target must not turn another target's transient failure terminal. + let walk = std::mem::take(&mut self.walk); + match walk.transient { + Some(err) => self.walk_failed(now, err, Verdict::Retry, None), + None => self.walk_failed(now, walk.last.expect("a failed walk"), Verdict::Unusable, None), + } + } + + fn walk_failed(&mut self, now: Instant, err: E, verdict: Verdict, presented: Option<&auth::Dial>) { + self.walk = Walk::default(); + let terminal = match verdict { + Verdict::Fatal | Verdict::Unusable => true, + Verdict::Refused => !self.changed(presented), + Verdict::Retry | Verdict::Settled => false, + }; + if terminal || self.config.once { + return self.done(Err(err)); + } + self.queue.push_back(Action::Retrying(err.clone())); + self.retry.last = Some(err); + // No URL here: with several targets there isn't one to name, and each attempt + // already logged the address it tried. + self.back_off(now, self.config.backoff.timeout, false, "reconnecting after backoff"); + } + + /// Score how a live session ended and decide what follows. + fn end(&mut self, now: Instant, ended: Ended) { + let Phase::Live(live) = std::mem::replace(&mut self.phase, Phase::Start) else { + unreachable!("only a live session ends"); + }; + let tenure = now.saturating_duration_since(live.since); + let url = self.targets[live.index].url().clone(); + let pacing = Pacing::new(&self.config.backoff); + + // The connected target owns the policy, including when connecting once. A refused + // redirect is terminal: the peer is leaving and named somewhere we won't go, so + // redialing the old address or a fallback would ignore it. + let assigned = match &ended { + Ended::Goaway(msg) => match self.config.goaway.redirect.target(msg.uri(), &url, self.config.pinned) { + Ok(assigned) => assigned, + Err(failure) => return self.done(Err(failure.into())), + }, + Ended::Closed(_) => None, + }; + if assigned.is_some() && self.targets[live.index].fixed() { + return self.done(Err(Failure::Fixed.into())); + } + + // Connecting once leaves rather than migrating: there is no replacement to dial, and + // a GOAWAY naming no deadline never force-closes, so the peer stops accepting + // requests and waits for us. Ignoring it is how both sides end up waiting on each + // other forever. + if self.config.once && matches!(ended, Ended::Goaway(_)) { + return self.done(Ok(())); + } + + // Two peers that each hold a session past the healthy bar can still bounce us between + // them forever. Returning to an endpoint that already sent us elsewhere in this run of + // migrations skips the healthy-session reset, so the backoff escalates and the retry + // window still applies. A one-way rolling restart never revisits one, and a GOAWAY + // back to the same place is a restart, not a move, so it never counts. A session that + // outlived the capped delay already redials no faster than the backoff would allow, so + // it starts a new run: rolling restarts days apart never add up to a loop. + if matches!(ended, Ended::Closed(_)) || tenure >= pacing.settled() { + self.redirects.reset(); + } + // Only an accepted redirect is a move: an empty or ignored one reconnects in place. + let looped = assigned.as_ref().is_some_and(|target| { + let from = endpoint(&url); + let to = endpoint(target); + to != from && self.redirects.depart(from, &to) + }); + + // A session that stayed up past the healthy bar is healthy; one that ended sooner, or + // carried on a redirect loop, counts as a failed attempt. + let healthy = !looped && self.retry.settle(now, tenure, &pacing); + + let msg = match ended { + Ended::Goaway(msg) => msg, + Ended::Closed(err) => return self.closed(now, err, healthy, live.presented.as_ref(), &url), + }; + + // An accepted redirect is an assignment: keep dialing it from here on, and only it. + // The peer named exactly one place to go, which retires whatever other addresses got + // us to this session. + let url = match assigned { + Some(target) => { + self.targets = vec![A::from(target.clone())]; + target + } + None => url, + }; + + // Hand over gracefully however the backoff scores this session. The old one keeps + // serving until it closes or overstays, so its routes stay attached and live tracks + // splice onto the replacement at a group boundary. + tracing::info!(peer = %Shown(&url), "upstream GOAWAY; migrating"); + if let Some(old) = self.predecessor.take() { + old.retire(); + } + self.predecessor = Some(Predecessor { + deadline: now + self.config.goaway.handover(msg.timeout()), + session: live.session, + closing: false, + }); + self.queue.push_back(Action::Migrating); + + // No backoff: a handover off a healthy session is not a failure. + if healthy { + return; + } + + // Drained almost immediately, or looped back. The backoff still escalates, so the + // redials cannot hammer the peer, and the old session serves across the wait. + // + // An empty URI is "reconnect to me": the peer is restarting and drains each new + // session until it goes, which is no failure, since every one serves. A named URI is + // a redirect, scored as a failed attempt so two peers bouncing us between them + // eventually give up. + let restarting = msg.uri().is_empty(); + if !restarting { + let err: E = if looped { Failure::Loop } else { Failure::Redirected }.into(); + self.queue.push_back(Action::Retrying(err.clone())); + self.retry.last = Some(err); + } + // A restart's pacing is not capped by the window: a delay longer than the window + // would otherwise spend it all and end the loop without a redial. + let window = if restarting { + Duration::ZERO + } else { + self.config.backoff.timeout + }; + let reason = if looped { + "peer redirected back in a loop; retrying after backoff" + } else { + "peer drained the session immediately; retrying after backoff" + }; + self.back_off(now, window, restarting, reason); + } + + /// The live session closed with `err`. + fn closed(&mut self, now: Instant, err: Error, healthy: bool, presented: Option<&auth::Dial>, url: &Url) { + // Refusals and malformed protocol input are settled failures, however long the + // session lived, unless the tokens changed since this dial read them. + let terminal = match Verdict::from(&err) { + Verdict::Fatal => true, + Verdict::Refused => !self.changed(presented), + _ => false, + }; + let err = E::from(err); + // Connecting once, the session ending ends the policy with its close reason. + if terminal || self.config.once { + return self.done(Err(err)); + } + + self.queue.push_back(Action::Offline(Some(err.clone()))); + if healthy { + // The window was reset, so a one-off drop reconnects promptly. + tracing::warn!(peer = %Shown(url), "session closed, reconnecting"); + } else { + // Connected then dropped almost immediately, such as a server that accepts then + // resets. Keep the reason so the give-up reports a real cause, and back off so + // repeated flaps escalate instead of spinning. + tracing::warn!(peer = %Shown(url), %err, "session severed immediately, retrying"); + self.retry.last = Some(err); + } + self.back_off(now, self.config.backoff.timeout, false, "reconnecting after backoff"); + } + + /// Wait out the next jittered delay, capped by what remains of `window`. + fn back_off(&mut self, now: Instant, window: Duration, restart: bool, reason: &'static str) { + let Some(wait) = self.retry.wait(now, window) else { + return self.done(Err(self.exhausted())); + }; + tracing::warn!(?wait, "{reason}"); + self.phase = Phase::Backoff { + until: now + wait, + restart, + }; + } + + /// Whether the tokens changed since `presented` read them, so a refusal earns another dial. + fn changed(&self, presented: Option<&auth::Dial>) -> bool { + match (&self.tokens, presented) { + (Some((tokens, _)), Some(presented)) => tokens.changed(presented), + _ => false, + } + } + + fn exhausted(&self) -> E { + Failure::Exhausted { + after: self.config.backoff.timeout, + last: self.retry.last.as_ref().map(ToString::to_string), + } + .into() + } + + fn done(&mut self, result: Result<(), E>) { + self.predecessor = None; + self.phase = Phase::Done; + self.queue.push_back(Action::Done(result)); + } +} + +fn earliest(a: Option, b: Option) -> Option { + match (a, b) { + (Some(a), Some(b)) => Some(a.min(b)), + (a, b) => a.or(b), + } +} + +/// The retry pacing resolved from [`Backoff`]. +struct Pacing { + /// The first delay, already under the cap. + initial: Duration, + /// The delay cap, or `None` for uncapped. + max: Option, + /// At least 1: a multiplier of zero means no growth, not a delay that shrinks to nothing. + multiplier: u32, + /// The bar a session must clear to count as healthy. + healthy: Duration, +} + +impl Pacing { + fn new(backoff: &Backoff) -> Self { + let max = Some(backoff.max).filter(|max| !max.is_zero()); + Self { + initial: max.map_or(backoff.initial, |max| backoff.initial.min(max)), + max, + multiplier: backoff.multiplier.max(1), + healthy: backoff.healthy, + } + } + + /// The delay for the attempt after one that waited `delay`. + fn next(&self, delay: Duration) -> Duration { + let next = delay.saturating_mul(self.multiplier); + self.max.map_or(next, |max| next.min(max)) + } + + /// How long a session must last to start a new run of redirects. + /// + /// One that outlived the capped delay already redials no faster than the backoff would + /// allow. Uncapped, the healthy bar is the only rate the config names. + fn settled(&self) -> Duration { + self.max.unwrap_or(self.healthy) + } +} + +/// The backoff window across a run of failed attempts. +struct Retry { + /// The wait before the next attempt, before jitter. + delay: Duration, + /// When this run of failures began; the give-up timeout counts from here. + start: Instant, + /// The latest failure, so giving up reports a real cause. + last: Option, +} + +impl Retry { + fn new(initial: Duration, now: Instant) -> Self { + Self { + delay: initial, + start: now, + last: None, + } + } + + /// Score how long a session served: past the healthy bar starts a fresh retry window. + fn settle(&mut self, now: Instant, tenure: Duration, pacing: &Pacing) -> bool { + let healthy = tenure >= pacing.healthy; + if healthy { + *self = Self::new(pacing.initial, now); + } + healthy + } + + /// Jitter the next delay and cap it at what remains of `window`. `None` means the + /// window is spent; a zero window never is. + fn wait(&self, now: Instant, window: Duration) -> Option { + let wait = self.delay.mul_f64(0.5 + kio::clock::rng().random::() / 2.0); + if window.is_zero() { + return Some(wait); + } + Some(wait.min(window.checked_sub(now.saturating_duration_since(self.start))?)) + } +} + +/// An old session kept serving after a GOAWAY so its in-flight groups finish. +struct Predecessor { + session: Session, + /// When its handover ends. + deadline: Instant, + /// Whether [`Self::poll_idle`] started the graceful close. + closing: bool, +} + +impl Predecessor { + /// Close the old session now, whatever remains of its window. + fn retire(self) { + self.session.abort(Error::GoawayTimeout); + } + + /// Close the old session gracefully once it carries no subscription or fetch. + /// + /// Only while a replacement serves: until then everything new still lands here. The + /// handover window still bounds the close. + fn poll_idle(&mut self, waiter: &kio::Waiter) { + if self.closing || self.session.poll_idle(waiter).is_pending() { + return; + } + tracing::debug!("old session carries nothing after GOAWAY; closing"); + self.closing = true; + self.session.start_close(); + } + + /// Poll the drain; `true` once it is over. + fn poll(&mut self, now: Instant, waiter: &kio::Waiter) -> bool { + if self.session.poll_closed(waiter).is_ready() { + tracing::debug!("old session drained cleanly after GOAWAY"); + return true; + } + if now >= self.deadline { + tracing::warn!("old session did not drain in time; closing"); + self.session.abort(Error::GoawayTimeout); + return true; + } + false + } +} diff --git a/rs/moq-net/src/reconnect/redirect.rs b/rs/moq-net/src/reconnect/redirect.rs new file mode 100644 index 00000000000..90ba212326c --- /dev/null +++ b/rs/moq-net/src/reconnect/redirect.rs @@ -0,0 +1,515 @@ +//! Judging the URI a peer's GOAWAY names, and catching peers that bounce a connection between them. + +use url::Url; + +use super::Failure; + +/// What to do with the URI a peer names in its GOAWAY. +/// +/// The URI is dialed exactly as given, so it must carry whatever credentials the +/// new endpoint needs. Nothing from the current session is copied onto it. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +#[non_exhaustive] +pub enum Redirect { + /// Follow any URI that does not downgrade the scheme, letting the peer name the + /// host as well as the port. + /// + /// The host is dialed as written, so this hands the peer whatever this process + /// can reach: a name it controls can carry an A record for loopback, a private + /// range, or the metadata service, and nothing here resolves it to find out. A + /// *literal* loopback, private, link-local, or IPC target is still refused, but + /// that catches a misconfigured peer, not a hostile one. Use it only with an + /// upstream trusted with the local network and with the tokens every session presents. + Follow, + /// Follow only when the host matches the dialed URL, so a peer can move us + /// between ports or schemes but not to another host. The default. + #[default] + SameHost, + /// Ignore the URI and redial the current targets. + Ignore, +} + +impl Redirect { + /// The URL a GOAWAY assigns. `Ok(None)` keeps the current targets (the peer named no + /// URI, or the policy ignores a URI it could parse), `Ok(Some)` replaces them, and `Err` + /// is an explicit URI this policy refuses. A malformed URI is refused even under + /// [`Self::Ignore`]. + /// + /// `pinned` is a certificate pin, which can only verify the host it was configured + /// for, so it refuses a host change from an encrypted scheme even under [`Self::Follow`]. + pub(super) fn target(&self, uri: &str, current: &Url, pinned: bool) -> Result, Failure> { + if uri.is_empty() { + return Ok(None); + } + + // The URI can carry credentials, so the error names the reason, never the URI. + // Parse before `Ignore`: a malformed redirect is terminal even when the policy + // would otherwise stay on the current targets. + let target = uri + .parse::() + .map_err(|_| Failure::Refused("the GOAWAY URI is malformed"))?; + if matches!(self, Self::Ignore) { + return Ok(None); + } + + if scheme_tier(target.scheme()) < scheme_tier(current.scheme()) { + return Err(Failure::Refused("the GOAWAY redirect downgrades the scheme")); + } + + // Only as far as the URL itself says: a name is dialed, never resolved here, + // so this refuses a peer that names a local address outright and says + // nothing about one that hides the same address behind a hostname. That gap + // is why [`Self::SameHost`] is the default; see [`is_local`]. + if is_local(&target) && !is_local(current) { + return Err(Failure::Refused( + "the GOAWAY redirect widens reachability to a local address", + )); + } + + // Host only, not the full authority: the port is what a peer legitimately + // moves us across when it hands off to a sibling process on the same box. + let same_host = target.host_str() == current.host_str(); + if matches!(self, Self::SameHost) && !same_host { + return Err(Failure::Refused("the GOAWAY redirect leaves the current host")); + } + // Only an encrypted dial installs the verifier a pin configures; tcp, unix, and + // plaintext WebSocket never consult it. + if pinned && encrypted(current.scheme()) && !same_host { + return Err(Failure::Refused( + "the GOAWAY redirect leaves the host a certificate pin verifies", + )); + } + + Ok(Some(target)) + } +} + +/// Whether a dial over `scheme` is encrypted, verifying the peer's certificate. +fn encrypted(scheme: &str) -> bool { + scheme_tier(scheme) == 2 +} + +/// Rank a scheme so a peer-supplied redirect cannot silently drop encryption. +/// Unknown schemes rank lowest, so a forgotten classification is refused. +fn scheme_tier(scheme: &str) -> u8 { + match scheme { + "https" | "moqt" | "moql" | "wss" | "tls" => 2, + "tcp" | "ws" | "http" => 1, + // `unix` lands here deliberately: local IPC is not an upgrade over a + // network transport, it is a different reachability class (see `is_local`). + _ => 0, + } +} + +/// Whether a URL *says* it names something only reachable from this host or +/// network. +/// +/// A judgement about the URL, not about where a dial would land. Literals and +/// `localhost` are decided here; every other name is left alone, since the +/// address behind it is chosen by whoever runs its DNS and can change between +/// this call and the dial. So a `false` here means "not local on its face", never +/// "verified remote". +fn is_local(url: &Url) -> bool { + match url.host() { + // A non-special scheme (`moqt://127.0.0.1`, and every other scheme but + // `http`/`https`/`ws`/`wss`) parses its host as a domain even when it is an + // address literal, so the literal check can't be left to the URL parser. + Some(url::Host::Domain(host)) => match host.parse::() { + Ok(ip) => is_local_ip(ip), + Err(_) => host == "localhost" || host.ends_with(".localhost"), + }, + Some(url::Host::Ipv4(ip)) => is_local_ip(ip.into()), + Some(url::Host::Ipv6(ip)) => is_local_ip(ip.into()), + // No host at all, e.g. a `unix:` socket path. + None => true, + } +} + +/// Whether an address literal is one only this host or network can reach. +fn is_local_ip(ip: std::net::IpAddr) -> bool { + match ip { + std::net::IpAddr::V4(ip) => is_local_v4(ip), + // An IPv4-mapped address reaches the same host as the v4 it wraps, so judge + // it by that rather than by the v6 rules. + std::net::IpAddr::V6(ip) => match ip.to_ipv4_mapped() { + Some(v4) => is_local_v4(v4), + None => ip.is_loopback() || ip.is_unspecified() || ip.is_unique_local() || ip.is_unicast_link_local(), + }, + } +} + +fn is_local_v4(ip: std::net::Ipv4Addr) -> bool { + ip.is_loopback() || ip.is_private() || ip.is_link_local() || ip.is_unspecified() +} + +/// Where `url` dials, without its userinfo, query, or fragment, where a peer may mint fresh +/// credentials on every redirect. Two URLs that differ only there reach the same endpoint. +pub(super) fn endpoint(url: &Url) -> Url { + let mut bare = url.clone(); + // Only fails for a URL that cannot carry userinfo, which then has none to strip. + let _ = bare.set_username(""); + let _ = bare.set_password(None); + bare.set_query(None); + bare.set_fragment(None); + bare +} + +/// Formats a dial URL for logs and errors as scheme, host, and port only. +/// +/// A URL can carry a credential in its userinfo, query, or path (raw QUIC has no headers, +/// so a membership proof rides as a path segment). Dropping all three rather than redacting +/// known-secret spellings is deliberate: a denylist only covers the credentials that exist +/// today. +pub(super) struct Shown<'a>(pub &'a Url); + +impl std::fmt::Display for Shown<'_> { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}://", self.0.scheme())?; + let Some(host) = self.0.host_str() else { + // No host at all, e.g. a `unix:` socket, where the path is the address + // rather than a credential and is the only thing that identifies it. + return write!(f, "{}", self.0.path()); + }; + write!(f, "{host}")?; + match self.0.port() { + Some(port) => write!(f, ":{port}"), + None => Ok(()), + } + } +} + +/// A run of GOAWAY redirects, and whether it has revisited an endpoint. +/// +/// The peer picks every hop, so remembering them all would grow without bound. Brent's cycle +/// detection instead keeps one departed endpoint, replaced at doubling intervals, and catches a +/// cycle of any length within a few laps of it. Once caught, the run stays a loop until reset. +pub(super) struct Redirects { + mark: Option, + span: u64, + hops: u64, + looped: bool, +} + +impl Default for Redirects { + fn default() -> Self { + Self { + mark: None, + span: 1, + hops: 0, + looped: false, + } + } +} + +impl Redirects { + /// Record leaving `from` for a different endpoint `to`, and return whether this run loops. + pub(super) fn depart(&mut self, from: Url, to: &Url) -> bool { + if self.mark.as_ref() == Some(to) { + self.looped = true; + } + self.hops += 1; + if self.hops == self.span { + self.mark = Some(from); + self.span = self.span.saturating_mul(2); + self.hops = 0; + } + self.looped + } + + /// Start a new run. + pub(super) fn reset(&mut self) { + *self = Self::default(); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn url(value: &str) -> Url { + value.parse().expect("valid url") + } + + fn refused(res: Result, Failure>) -> bool { + matches!(res, Err(Failure::Refused(_))) + } + + /// A peer must not be able to redirect us somewhere we could not already + /// reach. IPv4-mapped and link-local forms are the easy ones to miss. + #[test] + fn local_targets_are_recognized() { + let local = [ + "https://127.0.0.1/", + "https://localhost/", + "https://[::1]/", + "https://10.0.0.1/", + "https://169.254.1.1/", + "https://[::ffff:127.0.0.1]/", + "https://[::ffff:10.0.0.1]/", + "https://[fe80::1]/", + "https://[fc00::1]/", + "https://0.0.0.0/", + ]; + for url in local { + assert!(is_local(&url.parse().unwrap()), "{url} should be local"); + } + + for url in ["https://example.com/", "https://8.8.8.8/", "https://[2606:4700::1]/"] { + assert!(!is_local(&url.parse().unwrap()), "{url} should not be local"); + } + + // A public endpoint may not redirect us inward, but a local one may stay local. + let public = url("https://relay.example/"); + let localhost = url("https://127.0.0.1:4443/"); + assert!(refused( + Redirect::Follow.target("https://[::ffff:127.0.0.1]/", &public, false) + )); + assert_eq!( + Redirect::Follow + .target("https://127.0.0.1:9999/", &localhost, false) + .unwrap() + .unwrap() + .port(), + Some(9999) + ); + } + + /// The scheme ranking is what stops a peer from quietly moving us off an + /// encrypted transport. An unknown scheme ranks lowest so a classification we + /// forgot to add is refused rather than trusted. + #[test] + fn scheme_tiers_rank_encrypted_above_plaintext() { + for scheme in ["https", "moqt", "moql", "wss", "tls"] { + assert_eq!(scheme_tier(scheme), 2, "{scheme} is encrypted"); + } + for scheme in ["tcp", "ws", "http"] { + assert_eq!(scheme_tier(scheme), 1, "{scheme} is plaintext"); + } + // `unix` is not an upgrade over a network transport, it is a different + // reachability class, so it must not outrank one. + for scheme in ["unix", "gopher", ""] { + assert_eq!(scheme_tier(scheme), 0, "{scheme} is unclassified"); + } + } + + /// A redirect may hold the scheme or improve it, never weaken it. A downgrade + /// is a refusal, not a reason to redial the address the peer is leaving. + #[test] + fn target_refuses_a_scheme_downgrade() { + let secure = url("https://relay.example/"); + let plain = url("http://relay.example/"); + + assert!(refused(Redirect::Follow.target("http://other.example/", &secure, false))); + assert!(refused(Redirect::Follow.target("unix:///tmp/moq.sock", &secure, false))); + + // Same tier and upgrades are followed. + let same = url("https://other.example/"); + assert_eq!( + Redirect::Follow + .target("https://other.example/", &secure, false) + .unwrap(), + Some(same.clone()) + ); + assert_eq!( + Redirect::Follow.target("https://other.example/", &plain, false).unwrap(), + Some(same) + ); + } + + /// Native schemes must classify address literals even when the URL parser returns a domain. + #[test] + fn a_non_special_scheme_classifies_its_literal_host() { + for url in [ + "moqt://127.0.0.1/", + "moql://10.0.0.1/", + "tcp://169.254.169.254/", + "moqt://[::1]/", + "moqt://[::ffff:127.0.0.1]/", + ] { + assert!(is_local(&url.parse().unwrap()), "{url} should be local"); + } + for url in ["moqt://8.8.8.8/", "moql://[2606:4700::1]/", "tcp://relay.example/"] { + assert!(!is_local(&url.parse().unwrap()), "{url} should not be local"); + } + + let public = url("moqt://relay.example/"); + assert!( + refused(Redirect::Follow.target("moqt://169.254.169.254/", &public, false)), + "a literal local target is refused whatever the scheme" + ); + } + + /// The reachability guard reads the URL, so it cannot see a local address + /// hiding behind a hostname: any name the peer controls can carry an A record + /// for loopback or a private range, and a second lookup at dial time may answer + /// differently anyway. The default policy is what closes that, by refusing to + /// let a peer name a host at all. + #[test] + fn a_peer_named_host_is_refused_by_default() { + let public = url("https://relay.example/"); + + // The bypass: a name is not local on its face, so the reachability guard + // passes it through whatever it resolves to. + let rebindable = url("https://rebind.attacker.example/"); + assert!(!is_local(&rebindable), "a hostname is never classified as local"); + assert_eq!( + Redirect::Follow.target(rebindable.as_str(), &public, false).unwrap(), + Some(rebindable.clone()), + "Follow is explicit trust: it dials the name the peer chose" + ); + + // So the default refuses the host change instead of trying to judge it. + assert_eq!(Redirect::default(), Redirect::SameHost); + assert!( + refused(Redirect::default().target(rebindable.as_str(), &public, false)), + "a peer-named host is refused without resolving it" + ); + assert_eq!( + super::super::Goaway::default().redirect, + Redirect::SameHost, + "and the default config carries that default" + ); + } + + /// Only an explicit URI can assign, and one the policy will not follow is an + /// error rather than a quiet fallback: the loop ends on it instead of redialing. + #[test] + fn only_an_accepted_redirect_replaces_the_targets() { + let current = url("https://relay.example/"); + + // No URI, or a policy that ignores it: keep the current targets. + assert_eq!(Redirect::Follow.target("", ¤t, false).unwrap(), None); + assert_eq!( + Redirect::Ignore + .target("https://relay.example:5443/", ¤t, false) + .unwrap(), + None + ); + + // An explicit URI the policy will not follow is refused. + for (policy, uri) in [ + (Redirect::SameHost, "https://other.example/"), + (Redirect::Follow, "not a url"), + (Redirect::Ignore, "not a url"), + (Redirect::Follow, "http://relay.example/"), + (Redirect::Follow, "https://127.0.0.1/"), + ] { + assert!(refused(policy.target(uri, ¤t, false)), "{policy:?}: {uri}"); + } + + // An explicit assignment remains an assignment even if its URL is unchanged. + assert_eq!( + Redirect::SameHost.target(current.as_str(), ¤t, false).unwrap(), + Some(current.clone()) + ); + let moved = "https://relay.example:5443/"; + assert_eq!( + Redirect::SameHost.target(moved, ¤t, false).unwrap(), + Some(url(moved)) + ); + } + + /// A fingerprint is a Rustls check, installed only on encrypted dials. Schemes that + /// never install that verifier must not inherit the pin. + #[test] + fn a_certificate_pin_only_covers_encrypted_schemes() { + for scheme in ["https", "wss", "moqt", "moql", "tls"] { + assert!(encrypted(scheme), "{scheme}"); + } + for scheme in ["http", "ws", "tcp", "unix"] { + assert!(!encrypted(scheme), "{scheme}"); + } + assert_eq!( + Redirect::Follow + .target("tcp://other.example:1/", &url("tcp://relay.example:1/"), true) + .unwrap(), + Some(url("tcp://other.example:1/")), + "a plaintext dial never consulted the pin" + ); + } + + /// A certificate pin verifies only the host it was configured for, so it + /// refuses a host change even when the policy would follow one. + #[test] + fn a_certificate_pin_holds_the_host() { + let current = url("https://relay.example/"); + assert!(refused(Redirect::Follow.target("https://other.example/", ¤t, true))); + let moved = "https://relay.example:5443/"; + assert_eq!(Redirect::Follow.target(moved, ¤t, true).unwrap(), Some(url(moved))); + } + + /// `SameHost` lets a peer move us between ports or schemes on the endpoint we + /// already chose, but not onto a different host. + #[test] + fn same_host_follows_a_port_not_a_host() { + let current = url("https://relay.example:4443/"); + + assert!( + refused(Redirect::SameHost.target("https://elsewhere.example/", ¤t, false)), + "another host is refused" + ); + + let moved = Redirect::SameHost + .target("https://relay.example:5443/", ¤t, false) + .unwrap() + .unwrap(); + assert_eq!(moved.port(), Some(5443), "a different port on the same host is followed"); + } + + #[test] + fn redirect_loop_detection_catches_any_cycle_in_constant_memory() { + for tail in 0..4 { + for cycle in 2..=8 { + // `tail` one-way moves, then `cycle` endpoints redirecting around a ring. + let at = |hop: usize| { + if hop < tail { + url(&format!("tcp://tail{hop}/")) + } else { + url(&format!("tcp://ring{}/", (hop - tail) % cycle)) + } + }; + let mut redirects = Redirects::default(); + let mut caught = None; + for hop in 0..4 * (tail + cycle) { + let looped = redirects.depart(at(hop), &at(hop + 1)); + match caught { + Some(_) => assert!(looped, "a caught loop stays caught"), + None if looped => caught = Some(hop), + None => {} + } + } + let caught = caught.unwrap_or_else(|| panic!("tail {tail}, cycle {cycle}: never caught")); + // Never before the run first returns to an endpoint it left. + assert!( + caught + 1 >= tail + cycle, + "tail {tail}, cycle {cycle}: caught at {caught}" + ); + } + } + + let mut redirects = Redirects::default(); + for hop in 0..1000 { + let from = url(&format!("tcp://a{hop}/")); + let to = url(&format!("tcp://a{}/", hop + 1)); + assert!(!redirects.depart(from, &to), "a one-way run never loops"); + } + } + + #[test] + fn an_endpoint_ignores_the_credentials() { + assert_eq!( + endpoint(&url("tcp://token-a:secret@relay:1/path?jwt=a#x")), + endpoint(&url("tcp://token-b@relay:1/path?jwt=b")) + ); + // The path is part of the request target, so it names a different endpoint. + assert_ne!(endpoint(&url("tcp://relay:1/a")), endpoint(&url("tcp://relay:1/b"))); + } + + #[test] + fn a_shown_url_drops_every_credential() { + let shown = Shown(&url("moqt://user:pass@relay:4443/.cluster/secret?jwt=token")).to_string(); + assert_eq!(shown, "moqt://relay:4443"); + assert_eq!(Shown(&url("unix:///run/moq.sock")).to_string(), "unix:///run/moq.sock"); + } +} + diff --git a/rs/moq-net/src/reconnect/tests.rs b/rs/moq-net/src/reconnect/tests.rs new file mode 100644 index 00000000000..c3185a9f836 --- /dev/null +++ b/rs/moq-net/src/reconnect/tests.rs @@ -0,0 +1,676 @@ +//! The policy driven over in-memory sessions on the simulated clock, so each timing is exact. + +use super::*; + +const INITIAL: Duration = Duration::from_millis(100); +const MAX: Duration = Duration::from_millis(400); + +#[derive(Clone, Debug, thiserror::Error)] +enum TestError { + #[error(transparent)] + Net(#[from] Error), + #[error(transparent)] + Policy(#[from] Failure), + #[error("dial failed")] + Dial, +} + +fn url(value: &str) -> Url { + value.parse().expect("valid url") +} + +/// What the peer at an address does with each dial. +#[derive(Clone)] +enum Peer { + /// Accept, then drop the session at once. + Close, + /// Accept, then send a GOAWAY naming this URI (empty for a restart) and keep serving. + Goaway(String), + /// Accept and keep serving. + Serve, + /// Fail the dial, judged as this. + Fail(Verdict), + /// Never answer. + Hang, +} + +/// A target that dials fixed socket addresses. +#[derive(Clone)] +struct Fixed(Url); + +impl From for Fixed { + fn from(url: Url) -> Self { + Self(url) + } +} + +impl Target for Fixed { + fn url(&self) -> &Url { + &self.0 + } + + fn fixed(&self) -> bool { + true + } +} + +fn version() -> crate::Version { + "moq-lite-07-wip".parse().unwrap() +} + +/// Run an in-memory MoQ handshake, returning the client and server sessions. +async fn handshake() -> (Session, Session) { + let (client, server) = crate::transport::memory::pair(crate::transport::memory::Config { + protocol: Some(version().alpn().to_string()), + ..Default::default() + }); + let dial = crate::Client::new().with_versions(version().into()); + let serve = crate::Server::new().with_versions(version().into()); + futures::join!( + async { + let (session, driver) = dial.connect(kio::clock::now(), client).await.unwrap(); + drop(kio::clock::sim::spawn(crate::time::run(driver))); + session + }, + async { + let (session, driver) = serve.accept(kio::clock::now(), server).await.unwrap(); + drop(kio::clock::sim::spawn(crate::time::run(driver))); + session + }, + ) +} + +/// The next action that is not a wait, sleeping on the simulated clock meanwhile. +async fn next + From>( + machine: &mut Reconnect, +) -> Action { + let mut sleep: Option = None; + kio::wait(|waiter| { + loop { + match machine.poll(kio::clock::now(), waiter) { + Action::Wait(Some(at)) => { + let timer = sleep.get_or_insert_with(|| kio::clock::sleep_until(at)); + if timer.deadline() != at { + timer.reset(at); + } + if waiter.poll_future(std::pin::Pin::new(timer)).is_pending() { + return Poll::Pending; + } + } + Action::Wait(None) => return Poll::Pending, + action => return Poll::Ready(action), + } + } + }) + .await +} + +/// A record of one policy run. +#[derive(Default)] +struct Run { + /// The URL and instant of each session that came up. + accepts: Vec<(Url, Instant)>, + /// The URL of every dial. + dials: Vec, + /// Every action other than a dial, a wait, or the end, by name. + events: Vec<&'static str>, + /// The server sides still serving. + servers: Vec, +} + +/// Drive `machine` against `peers` until it ends, or until `stop` says so. +async fn drive( + machine: &mut Reconnect, + peers: &[(Url, Peer)], + run: &mut Run, + mut stop: impl FnMut(&Run) -> bool, +) -> Option> { + while !stop(run) { + let attempt = match next(machine).await { + Action::Dial(attempt) => attempt, + Action::Done(res) => return Some(res), + Action::Abandon => { + run.events.push("abandon"); + continue; + } + Action::Migrating => { + run.events.push("migrating"); + continue; + } + Action::Retired => { + run.events.push("retired"); + continue; + } + Action::Offline(_) => { + run.events.push("offline"); + continue; + } + Action::Retrying(_) => { + run.events.push("retrying"); + continue; + } + Action::Wait(_) => unreachable!("next never returns a wait"), + }; + let url = attempt.target().url().clone(); + run.dials.push(url.clone()); + let (_, peer) = peers.iter().find(|(peer, _)| *peer == url).expect("an unknown peer"); + match peer { + Peer::Hang => {} + Peer::Fail(verdict) => machine.failed(kio::clock::now(), TestError::Dial, *verdict), + Peer::Close | Peer::Goaway(_) | Peer::Serve => { + let (client, server) = handshake().await; + run.accepts.push((url, kio::clock::now())); + match peer { + Peer::Close => drop(server), + Peer::Goaway(uri) => { + let goaway = match uri.is_empty() { + true => goaway::Goaway::new(), + false => goaway::Goaway::redirect(uri.clone()), + }; + server.drain().send(goaway).unwrap(); + run.servers.push(server); + } + _ => run.servers.push(server), + } + machine.connected(kio::clock::now(), client); + } + } + } + None +} + +fn config(initial: Duration, timeout: Duration) -> Config { + let mut config = Config::default(); + config.backoff.initial = initial; + config.backoff.max = MAX; + config.backoff.timeout = timeout; + config +} + +/// Check each gap between consecutive sessions against the jittered delay it should +/// have waited: between half of `delay` and all of it, starting at `initial` and +/// doubling up to `max`. +fn assert_paced(accepts: &[(Url, Instant)], initial: Duration, max: Duration) { + let mut delay = initial; + for (index, pair) in accepts.windows(2).enumerate() { + let gap = pair[1].1 - pair[0].1; + assert!( + gap >= delay / 2 && gap <= delay, + "retry {index} waited {gap:?}, outside the jittered {delay:?}" + ); + delay = (delay * 2).min(max); + } +} + +/// A peer that accepts and then closes every session at once never earns the healthy +/// reset: the delay escalates to the cap, and the timeout ends the policy. +#[kio::test] +async fn an_immediate_close_loop_escalates_until_the_timeout() { + const TIMEOUT: Duration = Duration::from_secs(2); + let peer = url("tcp://peer:1/"); + let started = kio::clock::now(); + let mut machine = Reconnect::new(config(INITIAL, TIMEOUT), vec![peer.clone()], started).unwrap(); + let mut run = Run::default(); + + let err = drive(&mut machine, &[(peer, Peer::Close)], &mut run, |_| false) + .await + .unwrap() + .expect_err("a flapping peer kept the connection"); + assert!( + matches!(err, TestError::Policy(Failure::Exhausted { last: Some(_), .. })), + "{err}" + ); + assert_eq!(kio::clock::now() - started, TIMEOUT, "the timeout did not end the policy"); + assert!(run.accepts.len() >= 6, "only {} sessions", run.accepts.len()); + assert_paced(&run.accepts, INITIAL, MAX); +} + +/// A peer restarting with an empty-URI GOAWAY on every session paces the redials like +/// failures, yet never spends the timeout: each drained session still served. +#[kio::test] +async fn an_immediate_restart_loop_escalates_without_giving_up() { + const TIMEOUT: Duration = Duration::from_millis(500); + let peer = url("tcp://peer:1/"); + let mut machine = Reconnect::new(config(INITIAL, TIMEOUT), vec![peer.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + + let ended = drive(&mut machine, &[(peer, Peer::Goaway(String::new()))], &mut run, |run| { + run.accepts.len() == 10 + }) + .await; + assert!(ended.is_none(), "a restarting peer ended the connection"); + assert!( + run.accepts[9].1 - run.accepts[0].1 > TIMEOUT * 2, + "the redials outran the backoff" + ); + assert!(!run.events.contains(&"retrying"), "a restart is no failure"); + assert_paced(&run.accepts, INITIAL, MAX); +} + +/// Two peers that redirect to each other the moment a session is up are failed attempts: +/// the delay escalates across both, and the timeout ends the policy. +#[kio::test] +async fn an_immediate_redirect_loop_escalates_until_the_timeout() { + const TIMEOUT: Duration = Duration::from_secs(2); + let a = url("tcp://peer:1/"); + let b = url("tcp://peer:2/"); + let peers = [ + (a.clone(), Peer::Goaway(b.to_string())), + (b.clone(), Peer::Goaway(a.to_string())), + ]; + let started = kio::clock::now(); + let mut machine = Reconnect::new(config(INITIAL, TIMEOUT), vec![a.clone()], started).unwrap(); + let mut run = Run::default(); + + let err = drive(&mut machine, &peers, &mut run, |_| false) + .await + .unwrap() + .expect_err("a redirect loop kept the connection"); + assert!(err.to_string().contains("redirect"), "{err}"); + assert_eq!(kio::clock::now() - started, TIMEOUT, "the timeout did not end the policy"); + assert!(run.accepts.len() >= 6, "only {} sessions", run.accepts.len()); + for (index, (peer, _)) in run.accepts.iter().enumerate() { + assert_eq!(*peer, if index % 2 == 0 { a.clone() } else { b.clone() }, "session {index}"); + } + assert_paced(&run.accepts, INITIAL, MAX); +} + +/// A zero `initial` turns retry waiting off: every redial follows the close at once. +#[kio::test] +async fn a_zero_initial_redials_at_once() { + let peer = url("tcp://peer:1/"); + let first = kio::clock::now(); + let mut machine = Reconnect::new( + config(Duration::ZERO, Duration::from_secs(1)), + vec![peer.clone()], + first, + ) + .unwrap(); + let mut run = Run::default(); + + drive(&mut machine, &[(peer, Peer::Close)], &mut run, |run| run.accepts.len() == 11).await; + for (_, at) in &run.accepts { + assert_eq!(*at, first, "a zero initial waited before a retry"); + } +} + +/// The first address that answers wins, however many dead ones precede it. +#[kio::test] +async fn the_walk_lands_on_the_first_address_that_answers() { + let targets = vec![url("tcp://a:1/"), url("tcp://b:1/"), url("tcp://c:1/")]; + let peers = [ + (targets[0].clone(), Peer::Fail(Verdict::Retry)), + (targets[1].clone(), Peer::Fail(Verdict::Unusable)), + (targets[2].clone(), Peer::Serve), + ]; + let started = kio::clock::now(); + let mut machine = Reconnect::new(Config::default(), targets.clone(), started).unwrap(); + let mut run = Run::default(); + + drive(&mut machine, &peers, &mut run, |run| !run.accepts.is_empty()).await; + assert_eq!(run.dials, targets, "every address in order"); + assert_eq!(run.accepts, [(targets[2].clone(), started)], "no wait between addresses"); + assert!(machine.session().is_some()); +} + +/// A black-holed address with a fallback gets the fixed bound; the last one runs to the +/// connect timeout. +#[kio::test] +async fn only_an_address_with_a_fallback_is_bounded() { + let targets = vec![url("tcp://a:1/"), url("tcp://b:1/")]; + let peers = [(targets[0].clone(), Peer::Hang), (targets[1].clone(), Peer::Hang)]; + let mut config = Config::default(); + config.backoff.timeout = Duration::from_secs(1); + let started = kio::clock::now(); + let mut machine = Reconnect::new(config, targets.clone(), started).unwrap(); + let mut run = Run::default(); + + let err = drive(&mut machine, &peers, &mut run, |_| false) + .await + .unwrap() + .expect_err("nothing answered"); + assert_eq!(run.events, ["abandon", "abandon"]); + assert_eq!(run.dials, targets, "a walk under way finishes past the retry window"); + assert_eq!(kio::clock::now() - started, ATTEMPT + config.timeout); + assert!( + matches!(&err, TestError::Policy(Failure::Timeout(timeout)) if *timeout == config.timeout), + "{err}" + ); +} + +/// A settled answer from one address ends the walk instead of being buried by another's +/// transport failure, and the policy retries after a backoff. +#[kio::test] +async fn a_settled_answer_stops_the_walk() { + let targets = vec![url("tcp://a:1/"), url("tcp://b:1/")]; + let peers = [ + (targets[0].clone(), Peer::Fail(Verdict::Settled)), + (targets[1].clone(), Peer::Serve), + ]; + let mut machine = Reconnect::new(config(INITIAL, Duration::from_secs(1)), targets.clone(), kio::clock::now()).unwrap(); + let mut run = Run::default(); + + drive(&mut machine, &peers, &mut run, |run| run.dials.len() == 2).await; + assert_eq!(run.dials, [targets[0].clone(), targets[0].clone()], "the walk stopped at the answer"); + assert_eq!(run.events, ["retrying"]); +} + +/// One unusable address must not turn another's transient failure terminal, but a walk +/// with nothing else ends on it. +#[kio::test] +async fn an_unusable_address_ends_only_a_walk_without_hope() { + let targets = vec![url("tcp://a:1/"), url("tcp://b:1/")]; + let peers = [ + (targets[0].clone(), Peer::Fail(Verdict::Retry)), + (targets[1].clone(), Peer::Fail(Verdict::Unusable)), + ]; + let mut machine = Reconnect::new(config(INITIAL, Duration::ZERO), targets.clone(), kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &peers, &mut run, |run| run.dials.len() == 4).await; + assert_eq!(run.events, ["retrying"], "an unlimited window keeps retrying"); + + let lone = url("tcp://c:1/"); + let started = kio::clock::now(); + let mut machine = Reconnect::new(config(INITIAL, Duration::ZERO), vec![lone.clone()], started).unwrap(); + let err = drive(&mut machine, &[(lone, Peer::Fail(Verdict::Unusable))], &mut Run::default(), |_| false) + .await + .unwrap() + .expect_err("an unusable address connected"); + assert!(matches!(err, TestError::Dial), "{err}"); + assert_eq!(kio::clock::now(), started, "ended without waiting"); +} + +/// A refused credential is terminal, unless the tokens changed since the dial read them. +#[kio::test] +async fn a_refusal_retries_only_new_tokens() { + let peer = url("tcp://peer:1/"); + let tokens = auth::Tokens::new(vec!["old".parse().unwrap()]); + let mut machine = Reconnect::new(config(INITIAL, Duration::ZERO), vec![peer.clone()], kio::clock::now()) + .unwrap() + .with_tokens(tokens.clone(), version().into()); + + let Action::Dial(_) = next(&mut machine).await else { + panic!("expected a dial") + }; + tokens.set(vec!["new".parse().unwrap()]); + machine.failed(kio::clock::now(), TestError::Dial, Verdict::Refused); + assert!(matches!(next(&mut machine).await, Action::Retrying(_))); + + let Action::Dial(_) = next(&mut machine).await else { + panic!("expected a redial") + }; + machine.failed(kio::clock::now(), TestError::Dial, Verdict::Refused); + assert!( + matches!(next(&mut machine).await, Action::Done(Err(TestError::Dial))), + "an unchanged credential is refused for good" + ); +} + +/// Ending the tokens' handle ends the policy at its next walk, rather than redialing +/// sessions the handle refuses. +#[kio::test] +async fn an_ended_auth_handle_ends_the_policy() { + let peer = url("tcp://peer:1/"); + let tokens = auth::Tokens::new(Vec::new()); + let mut machine = Reconnect::new(config(INITIAL, Duration::ZERO), vec![peer.clone()], kio::clock::now()) + .unwrap() + .with_tokens(tokens.clone(), version().into()); + let mut run = Run::default(); + + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| !run.accepts.is_empty()).await; + tokens.handle().abort(Error::App(9)); + run.servers.clear(); + let ended = drive(&mut machine, &[(peer, Peer::Serve)], &mut run, |_| false).await; + assert!( + matches!(ended, Some(Err(TestError::Net(Error::App(9))))), + "the policy kept redialing on an ended auth handle" + ); + assert_eq!(run.accepts.len(), 1); +} + +/// Connecting once, a GOAWAY ends the policy cleanly and a close ends it with its reason. +#[kio::test] +async fn connecting_once_ends_with_the_session() { + let peer = url("tcp://peer:1/"); + let config = Config { + once: true, + ..Default::default() + }; + + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let ended = drive(&mut machine, &[(peer.clone(), Peer::Goaway(String::new()))], &mut Run::default(), |_| false).await; + assert!(matches!(ended, Some(Ok(()))), "a GOAWAY is the cue to leave"); + + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let ended = drive(&mut machine, &[(peer.clone(), Peer::Close)], &mut Run::default(), |_| false).await; + assert!(matches!(ended, Some(Err(TestError::Net(_)))), "the close reason ends it"); + + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let ended = drive(&mut machine, &[(peer, Peer::Fail(Verdict::Retry))], &mut Run::default(), |_| false).await; + assert!(matches!(ended, Some(Err(TestError::Dial))), "a failed dial is not retried"); +} + +/// A target dialing fixed socket addresses cannot follow a redirect. +#[kio::test] +async fn a_fixed_target_refuses_a_redirect() { + let peer = url("moqt://relay.example:4443/"); + let moved = url("moqt://relay.example:5443/"); + let mut machine = Reconnect::new(Config::default(), vec![Fixed(peer.clone())], kio::clock::now()).unwrap(); + let ended = drive(&mut machine, &[(peer, Peer::Goaway(moved.to_string()))], &mut Run::default(), |_| false).await; + assert!(matches!(ended, Some(Err(TestError::Policy(Failure::Fixed))))); +} + +/// A redirect the policy refuses ends it rather than redialing the address the peer leaves. +#[kio::test] +async fn a_refused_redirect_is_terminal() { + let peer = url("moqt://relay.example:4443/"); + let mut machine = Reconnect::new(Config::default(), vec![peer.clone()], kio::clock::now()).unwrap(); + let ended = drive( + &mut machine, + &[(peer, Peer::Goaway("moqt://elsewhere.example/".into()))], + &mut Run::default(), + |_| false, + ) + .await; + assert!(matches!(ended, Some(Err(TestError::Policy(Failure::Refused(_)))))); +} + +/// A healthy session's GOAWAY migrates at once to the named address, and the drained +/// predecessor closes as soon as the replacement serves and it carries nothing. +#[kio::test] +async fn a_healthy_goaway_hands_over_to_the_redirect() { + let a = url("tcp://peer:1/"); + let b = url("tcp://peer:2/"); + let mut machine = Reconnect::new(Config::default(), vec![a.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &[(a.clone(), Peer::Serve)], &mut run, |run| !run.accepts.is_empty()).await; + + // Past the healthy bar, the GOAWAY is a handover, not a failure. + kio::clock::sleep(Duration::from_secs(2)).await; + run.servers[0].drain().send(goaway::Goaway::redirect(b.to_string())).unwrap(); + let peers = [(a.clone(), Peer::Serve), (b.clone(), Peer::Serve)]; + let migrated = kio::clock::now(); + drive(&mut machine, &peers, &mut run, |run| run.events.contains(&"retired")).await; + assert_eq!(run.dials, [a, b.clone()]); + assert_eq!(run.accepts[1], (b, migrated), "migrated without a backoff"); + assert_eq!(run.events, ["migrating", "retired"], "a replacement serves, so nothing went offline"); + assert!(machine.predecessor().is_none()); + assert!(machine.session().is_some()); +} + +/// Our handover window caps the peer's GOAWAY deadline, and a predecessor retiring while +/// no replacement serves leaves the connection offline. +#[kio::test] +async fn the_handover_takes_the_earlier_deadline() { + const HANDOVER: Duration = Duration::from_millis(500); + let peer = url("tcp://peer:1/"); + let mut config = Config::default(); + config.goaway.handover = HANDOVER; + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| !run.accepts.is_empty()).await; + + kio::clock::sleep(Duration::from_secs(2)).await; + run.servers[0] + .drain() + .send(goaway::Goaway::new().with_timeout(Duration::from_secs(3600))) + .unwrap(); + let drained = kio::clock::now(); + drive(&mut machine, &[(peer, Peer::Hang)], &mut run, |run| run.events.contains(&"offline")).await; + assert_eq!(run.events, ["migrating", "retired", "offline"]); + assert_eq!(kio::clock::now() - drained, HANDOVER, "a longer peer deadline must not extend our cap"); + + let goaway = Goaway::default(); + assert_eq!(goaway.handover(None), Duration::from_secs(10), "no deadline: ours"); + assert_eq!( + goaway.handover(Some(Duration::from_secs(3))), + Duration::from_secs(3), + "a shorter peer deadline wins: it force-closes then anyway" + ); +} + +/// Under [`Restart::Wait`] an empty-URI GOAWAY changes nothing while the peer keeps the +/// session: no migration and no redial. Its close then reconnects like any drop. +#[kio::test] +async fn a_waited_restart_redials_only_once_the_session_closes() { + let peer = url("tcp://peer:1/"); + let mut config = config(INITIAL, Duration::from_secs(1)); + config.restart = Restart::Wait; + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| !run.accepts.is_empty()).await; + + kio::clock::sleep(Duration::from_secs(2)).await; + run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); + kio::clock::sim::idle().await; + assert!( + matches!(machine.poll(kio::clock::now(), &kio::Waiter::noop()), Action::Wait(None)), + "the policy left the session" + ); + + run.servers.remove(0).abort(Error::GoawayTimeout); + drive(&mut machine, &[(peer, Peer::Serve)], &mut run, |run| run.accepts.len() == 2).await; + assert_eq!(run.events, ["offline"]); +} + +/// The policy must not log or report a credential riding in a dial URL. +/// +/// A LAN mesh's membership proof rides as a URL path segment, since raw QUIC has no +/// headers to put it in, and a dial URL logged verbatim would hand a replayable credential +/// to anyone who can read logs. This drives the real `tracing` events rather than the +/// formatter alone, so it also catches a log site that forgot to use it. +#[test] +fn dialing_never_logs_the_credential() { + const SECRET: &str = "b91d7fe20c4a"; + let targets = vec![ + url(&format!("moqt://user:{SECRET}@a.example/.cluster/{SECRET}?jwt={SECRET}")), + url(&format!("moqt://b.example/.cluster/{SECRET}")), + ]; + let started = Instant::now(); + let mut machine = Reconnect::::new(Config::default(), targets, started).unwrap(); + let waiter = kio::Waiter::noop(); + + let mut reported = Vec::new(); + let events = crate::test_tracing::capture(tracing::Level::DEBUG, || { + assert!(matches!(machine.poll(started, &waiter), Action::Dial(_))); + // The first address has a fallback, so it times out with an error naming it. + assert!(matches!(machine.poll(started + ATTEMPT, &waiter), Action::Abandon)); + assert!(matches!(machine.poll(started + ATTEMPT, &waiter), Action::Dial(_))); + machine.failed(started + ATTEMPT, TestError::Dial, Verdict::Retry); + loop { + match machine.poll(started + ATTEMPT, &waiter) { + Action::Retrying(err) => reported.push(err.to_string()), + Action::Wait(_) => break, + action => panic!("unexpected {action:?}"), + } + } + }); + + // The log lines did happen, so the assertion below isn't vacuously true. + assert!(events.iter().any(|event| event.contains("connecting")), "{events:?}"); + assert!(events.iter().any(|event| event.contains("a.example")), "{events:?}"); + for line in events.iter().chain(&reported) { + assert!(!line.contains(SECRET), "leaked the credential: {line}"); + } +} + +#[test] +fn invalid_configs_are_refused() { + assert!(matches!( + Reconnect::::new(Config::default(), Vec::new(), Instant::now()), + Err(ConfigError::Empty) + )); + let mut config = Config::default(); + config.backoff.healthy = Duration::ZERO; + assert!(matches!( + Reconnect::::new(config, vec![url("tcp://peer:1/")], Instant::now()), + Err(ConfigError::Healthy) + )); +} + +/// Each knob means one thing at zero: no wait, no cap, no growth. None of them touches +/// the healthy bar, which is what keeps an immediate close a failed attempt. +#[test] +fn zero_turns_each_pacing_knob_off() { + let defaults = Backoff::default(); + + let pacing = Pacing::new(&Backoff { + initial: Duration::ZERO, + ..defaults + }); + assert_eq!(pacing.initial, Duration::ZERO); + assert_eq!(pacing.next(pacing.initial), Duration::ZERO, "zero initial never waits"); + assert_eq!(pacing.healthy, defaults.healthy); + + let pacing = Pacing::new(&Backoff { + max: Duration::ZERO, + ..defaults + }); + assert_eq!(pacing.max, None); + let mut delay = pacing.initial; + for _ in 0..10 { + delay = pacing.next(delay); + } + assert_eq!(delay, defaults.initial * 1024, "zero max leaves the delay uncapped"); + assert_eq!(pacing.settled(), defaults.healthy); + + let pacing = Pacing::new(&Backoff { + multiplier: 0, + ..defaults + }); + assert_eq!(pacing.next(pacing.initial), pacing.initial, "zero multiplier never grows"); + + // A cap below the first delay caps it too. + let pacing = Pacing::new(&Backoff { + max: Duration::from_millis(300), + ..defaults + }); + assert_eq!(pacing.initial, Duration::from_millis(300)); +} + +/// A sane config is left alone: the delay doubles up to the cap and stays there. +#[test] +fn pacing_grows_to_the_cap() { + let pacing = Pacing::new(&Backoff { + initial: Duration::from_millis(100), + multiplier: 2, + max: Duration::from_millis(400), + ..Default::default() + }); + + assert_eq!(pacing.initial, Duration::from_millis(100)); + assert_eq!(pacing.settled(), Duration::from_millis(400)); + assert_eq!(pacing.next(Duration::from_millis(100)), Duration::from_millis(200)); + assert_eq!(pacing.next(Duration::from_millis(200)), Duration::from_millis(400)); + assert_eq!(pacing.next(Duration::from_millis(400)), Duration::from_millis(400)); +} + +/// Refusals and malformed input settle; everything else from MoQ is worth a retry. +#[test] +fn a_moq_error_is_judged_by_its_session_code() { + assert_eq!(Verdict::from(&Error::Unauthorized), Verdict::Refused); + assert_eq!(Verdict::from(&Error::Expired), Verdict::Refused); + assert_eq!(Verdict::from(&Error::ProtocolViolation), Verdict::Fatal); + assert_eq!(Verdict::from(&Error::Closed), Verdict::Retry); + assert_eq!(Verdict::from(&Error::Transport("reset".into())), Verdict::Retry); +} diff --git a/rs/moq-net/src/session.rs b/rs/moq-net/src/session.rs index 2ed971dfe3c..83b498ece83 100644 --- a/rs/moq-net/src/session.rs +++ b/rs/moq-net/src/session.rs @@ -299,11 +299,7 @@ impl Session { /// Both protocols withdraw this session's announcements and wait for their /// delivery. pub async fn close(self) -> Result<(), Error> { - if let Ok(mut close) = self.close.write() - && close.is_none() - { - *close = Some(Close::Drain); - } + self.start_close(); // The drain outlives this handle, so dropping it cannot cut the drain short. let closed = self.closed.clone(); drop(self); @@ -336,17 +332,28 @@ impl Session { /// completion, this resolves with [`Error::Cancel`]. A panic while polling the driver is /// a transport failure instead. pub async fn closed(&self) -> Error { - match self - .closed - .wait(|state| match &**state { - Some(ended) => Poll::Ready(ended.err.clone()), - None => Poll::Pending, - }) - .await - { - Ok(err) => err, + kio::wait(|waiter| self.poll_closed(waiter)).await + } + + /// Poll form of [`closed`](Self::closed). + pub(crate) fn poll_closed(&self, waiter: &kio::Waiter) -> Poll { + match self.closed.poll(waiter, |state| match &**state { + Some(ended) => Poll::Ready(ended.err.clone()), + None => Poll::Pending, + }) { + Poll::Ready(Ok(err)) => Poll::Ready(err), // The driver was dropped before it could observe the close. - Err(kio::Closed) => Error::Cancel, + Poll::Ready(Err(_)) => Poll::Ready(Error::Cancel), + Poll::Pending => Poll::Pending, + } + } + + /// Start the drain [`close`](Self::close) runs, without waiting for it or giving up this handle. + pub(crate) fn start_close(&self) { + if let Ok(mut close) = self.close.write() + && close.is_none() + { + *close = Some(Close::Drain); } } diff --git a/rs/moq-tokio/src/client.rs b/rs/moq-tokio/src/client.rs index 6a00b236d74..2931345bfa2 100644 --- a/rs/moq-tokio/src/client.rs +++ b/rs/moq-tokio/src/client.rs @@ -4,8 +4,7 @@ //! QUIC settings ([`crate::quic::Config`]) a binary shares with its accept half. The //! accept side is [`crate::server`]. -use crate::connection::Goaway; -use crate::{Addrs, Backoff, Connection, Error, Restart}; +use crate::{Addrs, Connection, Error, Restart}; #[cfg(all(feature = "websocket", feature = "quic"))] use std::future::Future; use url::Url; @@ -51,16 +50,8 @@ pub struct Client { connect: Option, /// The credentials each new connection starts presenting. pub(crate) tokens: Vec, - /// Deadline for one [`Client::connect`], from [`crate::connect::Config::timeout`]. Zero waits forever. - #[cfg(feature = "_transport")] - timeout: std::time::Duration, - pub(crate) reconnect: bool, - pub(crate) backoff: Backoff, - pub(crate) goaway: Goaway, - pub(crate) restart: Restart, - /// Whether the TLS config pins a certificate fingerprint, which only verifies - /// the configured host, so a GOAWAY may not redirect elsewhere. - pub(crate) pinned: bool, + /// The connect deadline, backoff, and GOAWAY handling each [`Connection`] runs under. + pub(crate) policy: moq_net::reconnect::Config, /// The resolved Happy Eyeballs timings, used by the `tcp://` and `tls://` /// dials here; the QUIC backend captures its own copy from the config. #[cfg(feature = "tcp")] @@ -91,10 +82,12 @@ pub struct Client { pub(crate) dialer: Option>, } -/// A test stand-in for [`Client::dial`]. +/// A test stand-in for [`Client::dial`], handed the address and the MoQ client a real dial +/// would handshake with. #[cfg(test)] -pub(crate) type Dialer = - dyn Fn(crate::connect::Addr) -> std::pin::Pin> + Send>> + Send + Sync; +pub(crate) type Dialer = dyn Fn(crate::connect::Addr, moq_net::Client) -> std::pin::Pin> + Send>> + + Send + + Sync; impl Client { /// Build a client from its config. @@ -146,7 +139,11 @@ impl Client { let tcp_tls = config.tls.clone(); #[cfg(feature = "websocket")] let tls_host_name = config.tls.host_name.clone(); - let timeout = resolved.timeout; + let mut policy = moq_net::reconnect::Config::default(); + policy.backoff = config.backoff.resolve(); + policy.goaway = config.goaway.resolve(); + policy.timeout = resolved.timeout; + policy.pinned = !config.tls.fingerprint.is_empty(); let mut moq = moq_net::Client::new() .with_versions(versions.clone()) @@ -160,12 +157,7 @@ impl Client { versions, connect: config.url, tokens: config.tokens, - timeout, - reconnect: true, - backoff: config.backoff, - goaway: config.goaway, - restart: Restart::default(), - pinned: !config.tls.fingerprint.is_empty(), + policy, #[cfg(feature = "tcp")] failover_delay, #[cfg(feature = "tcp")] @@ -247,13 +239,13 @@ impl Client { /// /// Defaults to true. Each clone can choose its own reconnect policy. pub fn with_reconnect(mut self, reconnect: bool) -> Self { - self.reconnect = reconnect; + self.policy.once = !reconnect; self } /// Choose what a GOAWAY naming no URI does. Defaults to [`Restart::Redial`]. pub fn with_restart(mut self, restart: Restart) -> Self { - self.restart = restart; + self.policy.restart = restart; self } @@ -339,11 +331,12 @@ impl Client { Ok::<_, Error>(connect_session(&self.moq, session).await?) }; - let session = match self.timeout.is_zero() { + let timeout = self.policy.timeout; + let session = match timeout.is_zero() { true => attempt.await?, - false => kio::clock::timeout(self.timeout, attempt) + false => kio::clock::timeout(timeout, attempt) .await - .map_err(|_| Error::ConnectTimeout(self.timeout))??, + .map_err(|_| Error::ConnectTimeout(timeout))??, }; tracing::info!(version = %session.version(), transport = %crate::Transport::WebRtc, "connected"); Ok(session) @@ -351,39 +344,31 @@ impl Client { .await } - /// What a dial to `url` presents of `tokens`, given the versions this client offers. - /// Dial the given URL and complete the MoQ handshake. + /// Dial one attempt of a [`Connection`]'s policy and complete the MoQ handshake. /// /// Errors if no transport feature is compiled in. #[cfg(not(any(feature = "quic", feature = "websocket", feature = "tcp", feature = "uds")))] pub(crate) async fn dial( &self, - _addr: crate::connect::Addr, - _presented: &moq_net::auth::Dial, + _attempt: moq_net::reconnect::Attempt, ) -> crate::Result { Err(Error::NoBackend( "no backend compiled; enable quic, websocket, tcp, or uds feature", )) } - /// Dial the given URL and complete the MoQ handshake, presenting what `presented` decided. + /// Dial one attempt of a [`Connection`]'s policy and complete the MoQ handshake, + /// presenting the URL and credential the attempt carries. /// /// The scheme picks the transport, and `https://` races QUIC against the /// WebSocket fallback so a blocked UDP path still connects. The session's /// protocol driver is spawned on the current tokio runtime, else the fallback; the session - /// closes once the last returned handle drops. + /// closes once the last returned handle drops. The policy bounds the attempt: dropping + /// it cancels whichever arm is still pending. #[cfg(any(feature = "quic", feature = "websocket", feature = "tcp", feature = "uds"))] - pub(crate) async fn dial( - &self, - addr: crate::connect::Addr, - presented: &moq_net::auth::Dial, - ) -> crate::Result { - let addr = addr.with_url(presented.url.clone()); - #[cfg(test)] - if let Some(dialer) = &self.dialer { - return dialer(addr).await; - } - let client = match &presented.token { + pub(crate) async fn dial(&self, attempt: moq_net::reconnect::Attempt) -> crate::Result { + let addr = attempt.target().with_url(attempt.url().clone()); + let client = match attempt.token() { Some(token) => { let mut client = self.clone(); client.moq = client.moq.with_token(token.clone()); @@ -391,22 +376,14 @@ impl Client { } None => std::borrow::Cow::Borrowed(self), }; + #[cfg(test)] + if let Some(dialer) = &self.dialer { + return dialer(addr, client.moq.clone()).await; + } // Each compiled backend adds state to this dispatch future. Keep it off the // caller's stack so all-feature builds remain safe on standard 2 MiB threads. - let attempt = Box::pin(client.connect_inner(addr)); - - // The deadline covers the dial AND the handshake, for every transport: it is the - // only bound some of them have. Dropping `attempt` on expiry cancels whichever - // arm was still pending. - let dialed = match self.timeout.is_zero() { - true => attempt.await?, - false => match kio::clock::timeout(self.timeout, attempt).await { - Ok(res) => res?, - Err(_) => return Err(Error::ConnectTimeout(self.timeout)), - }, - }; - + let dialed = Box::pin(client.connect_inner(addr)).await?; tracing::info!(version = %dialed.session.version(), transport = %dialed.transport, "connected"); Ok(dialed) } @@ -1347,20 +1324,21 @@ mod tests { ..Default::default() }; config.websocket.delay = Some((std::time::Duration::ZERO).into()); - let client = config.init(Default::default()).unwrap(); + // Connecting once: this is about one attempt's deadline, which a reconnecting + // connection would redial past. + let client = config.init(Default::default()).unwrap().with_reconnect(false); // Nothing is listening on UDP, so the QUIC arm fails and leaves the WebSocket // arm alone against the silent peer. let url: Url = format!("https://127.0.0.1:{}/", addr.port()).parse().unwrap(); - // `dial` rather than `connect`: this is about one attempt's deadline, and - // `connect` now hands back a reconnect loop that would redial past it. - let presented = moq_net::auth::Tokens::new(Vec::new()).dial(&url, &client.versions); - let mut attempt = Box::pin(client.dial(url.into(), &presented)); + let connection = client.connect(url); + let closed = connection.closed(); + tokio::pin!(closed); let _silent = tokio::select! { - res = &mut attempt => match res { + res = &mut closed => match res { Err(err) => panic!("connect failed before the silent peer accepted it: {err}"), - Ok(_) => panic!("connected to a peer that never spoke"), + Ok(()) => panic!("stopped before the silent peer accepted it"), }, res = listener.accept() => res.unwrap().0, }; @@ -1370,11 +1348,7 @@ mod tests { tokio::time::pause(); tokio::time::advance(timeout).await; - let err = match attempt.await { - Err(err) => err, - Ok(_) => panic!("connected to a peer that never spoke"), - }; - + let err = closed.await.expect_err("connected to a peer that never spoke"); assert!(matches!(err, Error::ConnectTimeout(_)), "unexpected error: {err}"); } } diff --git a/rs/moq-tokio/src/connect.rs b/rs/moq-tokio/src/connect.rs index 91b155f699f..b7ecd9a9715 100644 --- a/rs/moq-tokio/src/connect.rs +++ b/rs/moq-tokio/src/connect.rs @@ -102,6 +102,16 @@ impl From for Addr { } } +impl moq_net::reconnect::Target for Addr { + fn url(&self) -> &Url { + &self.url + } + + fn fixed(&self) -> bool { + self.resolved.is_some() + } +} + /// One or more addresses for the same peer, tried in order until one connects. /// /// Most callers have a single URL and never name this type: diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs index ec6d9d1f98d..b35618f25c2 100644 --- a/rs/moq-tokio/src/connection.rs +++ b/rs/moq-tokio/src/connection.rs @@ -1,4 +1,8 @@ //! The reconnect loop around a session, and the [`Monitor`] that observes it. +//! +//! The policy (the connect deadline, the address walk, backoff, and GOAWAY handling) is +//! [`moq_net::Reconnect`]'s; this module dials what it asks for, runs it on a task, and +//! publishes what it does. use std::task::{Poll, ready}; use std::time::Duration; @@ -6,69 +10,45 @@ use std::time::Duration; use moq_net::Version; use moq_net::bandwidth::{Consumer as BandwidthConsumer, Producer as BandwidthProducer}; use moq_net::kio; -use rand::RngExt; -use url::Url; +use moq_net::reconnect::{Action, Verdict}; use crate::abort::AbortOnDrop; use crate::connect::Endpoint; use crate::{Addrs, Client, Error}; -/// How long one address gets before [`Connection`] moves to the peer's next one. -const CONNECT_ATTEMPT: Duration = Duration::from_secs(5); +pub use moq_net::reconnect::Restart; -/// The deadline for the attempt at candidate `index` of `total`, or `None` to -/// let it run. +/// How a failed dial bears on the rest of the policy. /// -/// A black-holed address answers nothing rather than refusing, so without a -/// bound the QUIC idle timeout would decide how long the remaining candidates -/// wait. Only an attempt with a later candidate to fall back on is worth -/// bounding: cutting the last one short just turns a slow connect into a failed -/// one, on every reconnect cycle. -/// -/// The same goes for the retry window from [`Backoff::timeout`]: it decides -/// whether a new walk starts, not how long one may take, so a walk under way -/// tries every candidate and a handshake slower than the window still gets -/// [`crate::connect::Config::timeout`](crate::connect::Config::timeout). -fn attempt_timeout(index: usize, total: usize) -> Option { - (index + 1 < total).then_some(CONNECT_ATTEMPT) -} - -/// Whether `err` refused the credential a dial presented while the client has since been -/// given new tokens, which earns another dial instead of ending the loop. It never -/// retries a rejected credential: a refusal of unchanged tokens stays terminal. -fn refreshed(err: &Error, tokens: &moq_net::auth::Tokens, presented: Option<&moq_net::auth::Dial>) -> bool { - let refused = err.is_auth() - || matches!(err, Error::MoqNet(err) if moq_net::SessionError::from(err) == moq_net::SessionError::Unauthorized); - refused && presented.is_some_and(|presented| tokens.changed(presented)) -} - -/// Whether a failed candidate is the peer's answer for every candidate, ending the walk. -/// -/// An auth rejection arrives as [`Error::Connect`] with its status already consumed, so it -/// is checked alongside the statuses that do not invite another attempt. A terminal MoQ error -/// means the handshake reached the peer, which answered with malformed input. -fn settles_walk(err: &Error) -> bool { - err.is_auth() - || (matches!(err, Error::MoqNet(_)) && err.is_terminal()) +/// A status the peer actually sent is its answer, not this address's, so unless it invites +/// another attempt it settles the walk. An auth rejection arrives as [`Error::Connect`] with +/// its status already consumed, so it is checked alongside the statuses. +fn verdict(err: &Error) -> Verdict { + if let Error::MoqNet(err) = err { + return Verdict::from(err); + } + let terminal = err.is_terminal(); + if err.is_auth() && terminal { + return Verdict::Refused; + } + let settled = err.is_auth() || err .status() - .is_some_and(|status| !crate::error::status_retryable(status)) + .is_some_and(|status| !crate::error::status_retryable(status)); + match (settled, terminal) { + (true, true) => Verdict::Fatal, + (true, false) => Verdict::Settled, + (false, true) => Verdict::Unusable, + (false, false) => Verdict::Retry, + } } /// Exponential backoff configuration for reconnection attempts. /// /// Three knobs, each with one job: `initial` and `multiplier` set the wait before /// each retry, `max` caps it, and `healthy` decides which sessions count as failed -/// attempts. The delays carry jitter, so a fleet knocked offline together does not -/// reconnect in lockstep. The timeout bounds every failure; a settled rejection, -/// malformed protocol input, or invalid local configuration stops retrying. A GOAWAY -/// naming no new URI is no failure: a restarting peer sends one to each session until -/// it goes, and each of those serves, so it restarts the timeout instead. -/// -/// A GOAWAY off a healthy session migrates at once, unless it sends the connection back -/// to an endpoint that already redirected it elsewhere and the session lived less than -/// `max`. Two peers redirecting to each other thus wait out this backoff and count -/// against the timeout. +/// attempts. See [`moq_net::reconnect::Backoff`], which this resolves to, for how +/// the policy applies them. #[derive(Clone, Debug, usage_rs::Args, serde::Serialize, serde::Deserialize)] #[usage(unknown_flags = "error", args_override_self = false)] #[serde(default, deny_unknown_fields)] @@ -175,7 +155,7 @@ impl Default for Backoff { fn default() -> Self { Self { initial: None, - multiplier: DEFAULT_MULTIPLIER, + multiplier: moq_net::reconnect::Backoff::default().multiplier, max: None, healthy: None, timeout: None, @@ -190,29 +170,42 @@ impl Default for Backoff { impl Backoff { /// The configured initial retry delay, or its default; zero retries at once. pub fn initial(&self) -> Duration { - self.initial.map(Into::into).unwrap_or(DEFAULT_INITIAL) + self.resolve().initial } /// The configured retry-delay cap, or its default; zero leaves the delay uncapped. pub fn max(&self) -> Duration { - self.max.map(Into::into).unwrap_or(DEFAULT_MAX) + self.resolve().max } /// The retry deadline window, with zero meaning unlimited and unset using the default. pub fn timeout(&self) -> Duration { - self.timeout.map(Into::into).unwrap_or(DEFAULT_TIMEOUT) - } - - /// How long a session must last to count as healthy. - fn healthy(&self) -> Duration { - self.healthy.map(Into::into).unwrap_or(DEFAULT_HEALTHY) + self.resolve().timeout + } + + /// The policy these knobs configure, with each unset one at its default. + pub(crate) fn resolve(&self) -> moq_net::reconnect::Backoff { + let mut backoff = moq_net::reconnect::Backoff::default(); + backoff.multiplier = self.multiplier; + let knobs = [ + (&mut backoff.initial, self.initial), + (&mut backoff.max, self.max), + (&mut backoff.healthy, self.healthy), + (&mut backoff.timeout, self.timeout), + ]; + for (knob, set) in knobs { + if let Some(set) = set { + *knob = set.into(); + } + } + backoff } /// Refuse a zero healthy threshold, which would count every session as healthy and /// reset the timeout forever under a peer that drops each one on sight. #[cfg(feature = "_transport")] pub(crate) fn validate(&self) -> crate::Result<()> { - if self.healthy().is_zero() { + if self.resolve().healthy.is_zero() { return Err(Error::BackoffHealthy); } Ok(()) @@ -265,24 +258,15 @@ pub enum Status { Migrating, } -/// What to do with the URI a peer names in its GOAWAY. -/// -/// The URI is dialed exactly as given, so it must carry whatever credentials the -/// new endpoint needs. Nothing from the current session is copied onto it. +/// What to do with the URI a peer names in its GOAWAY, as `--connect-goaway-redirect` +/// spells it. See [`moq_net::reconnect::Redirect`] for what each one risks. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, usage_rs::ValueEnum, serde::Serialize, serde::Deserialize)] #[serde(rename_all = "kebab-case")] #[non_exhaustive] pub enum Redirect { /// Follow any URI that does not downgrade the scheme, letting the peer name the - /// host as well as the port. - /// - /// The host is dialed as written, so this hands the peer whatever this process - /// can reach: a name it controls can carry an A record for loopback, a private - /// range, or the metadata service, and nothing here resolves it to find out. A - /// *literal* loopback, private, link-local, or IPC target is still refused, but - /// that catches a misconfigured peer, not a hostile one. Use it only with an - /// upstream trusted with the local network and with the tokens on - /// [`Connection::auth`], which every session presents, on any host. + /// host as well as the port. Use it only with an upstream trusted with the local + /// network and with the tokens on [`Connection::auth`]. Follow, /// Follow only when the host matches the configured URL, so a peer can move us /// between ports or schemes but not to another host. The default. @@ -292,132 +276,16 @@ pub enum Redirect { Ignore, } -/// What to do when a peer's GOAWAY names no URI, meaning it is restarting where we reached it. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] -#[non_exhaustive] -pub enum Restart { - /// Redial the same addresses at once while the drained session keeps serving. Right for - /// an address that can lead to another node, such as a fleet name a fresh resolve moves - /// off the drained one. The default. - #[default] - Redial, - /// Keep serving on the drained session until the peer closes it, then reconnect as after - /// any drop. Right for an address that names one node, where a redial only reaches the - /// draining process again. The peer must close the session itself, as a relay does at - /// its drain deadline. - Wait, -} - -impl Redirect { - /// The URL a GOAWAY assigns. `Ok(None)` keeps the current address list (the - /// peer named no URI, or the policy ignores a URI it could parse), `Ok(Some)` - /// replaces it, and `Err` is an explicit URI this policy refuses. A malformed - /// URI is refused even under [`Self::Ignore`]. - /// - /// `pinned` is a certificate pin on the connection, which can only verify the - /// host it was configured for, so it refuses a host change even under - /// [`Self::Follow`]. - fn target(&self, uri: &str, current: &Url, pinned: bool) -> crate::Result> { - if uri.is_empty() { - return Ok(None); - } - - // The URI can carry credentials, so the error names the reason, never the URI. - // Parse before `Ignore`: a malformed redirect is terminal even when the policy - // would otherwise stay on the current address list. - let refuse = |reason: &str| Error::RefusedRedirect(reason.to_string()); - - let target = uri.parse::().map_err(|_| refuse("the GOAWAY URI is malformed"))?; - if matches!(self, Self::Ignore) { - return Ok(None); - } - - if scheme_tier(target.scheme()) < scheme_tier(current.scheme()) { - return Err(refuse("the GOAWAY redirect downgrades the scheme")); - } - - // Only as far as the URL itself says: a name is dialed, never resolved here, - // so this refuses a peer that names a local address outright and says - // nothing about one that hides the same address behind a hostname. That gap - // is why [`Self::SameHost`] is the default; see [`is_local`]. - if is_local(&target) && !is_local(current) { - return Err(refuse("the GOAWAY redirect widens reachability to a local address")); - } - - // Host only, not the full authority: the port is what a peer legitimately - // moves us across when it hands off to a sibling process on the same box. - let same_host = target.host_str() == current.host_str(); - if matches!(self, Self::SameHost) && !same_host { - return Err(refuse("the GOAWAY redirect leaves the current host")); - } - if pinned && !same_host { - return Err(refuse("the GOAWAY redirect leaves the host a certificate pin verifies")); +impl From for moq_net::reconnect::Redirect { + fn from(redirect: Redirect) -> Self { + match redirect { + Redirect::Follow => Self::Follow, + Redirect::SameHost => Self::SameHost, + Redirect::Ignore => Self::Ignore, } - - Ok(Some(target)) - } -} - -/// Whether this scheme's dial installs the Rustls verifier, so a configured -/// fingerprint actually checked the peer. Plain and non-Rustls transports do not. -fn fingerprint_pins(scheme: &str) -> bool { - matches!(scheme, "https" | "wss" | "moqt" | "moql" | "tls") -} - -/// Rank a scheme so a peer-supplied redirect cannot silently drop encryption. -/// Unknown schemes rank lowest, so a forgotten classification is refused. -fn scheme_tier(scheme: &str) -> u8 { - match scheme { - "https" | "moqt" | "moql" | "wss" | "tls" => 2, - "tcp" | "ws" | "http" => 1, - // `unix` lands here deliberately: local IPC is not an upgrade over a - // network transport, it is a different reachability class (see `is_local`). - _ => 0, - } -} - -/// Whether a URL *says* it names something only reachable from this host or -/// network. -/// -/// A judgement about the URL, not about where a dial would land. Literals and -/// `localhost` are decided here; every other name is left alone, since the -/// address behind it is chosen by whoever runs its DNS and can change between -/// this call and the dial. So a `false` here means "not local on its face", never -/// "verified remote". -fn is_local(url: &Url) -> bool { - match url.host() { - // A non-special scheme (`moqt://127.0.0.1`, and every other scheme this - // crate dials but `http`/`https`/`ws`/`wss`) parses its host as a domain - // even when it is an address literal, so the literal check can't be left to - // the URL parser. `crate::resolve::Candidates` handles the same thing. - Some(url::Host::Domain(host)) => match host.parse::() { - Ok(ip) => is_local_ip(ip), - Err(_) => host == "localhost" || host.ends_with(".localhost"), - }, - Some(url::Host::Ipv4(ip)) => is_local_ip(ip.into()), - Some(url::Host::Ipv6(ip)) => is_local_ip(ip.into()), - // No host at all, e.g. a `unix:` socket path. - None => true, - } -} - -/// Whether an address literal is one only this host or network can reach. -fn is_local_ip(ip: std::net::IpAddr) -> bool { - match ip { - std::net::IpAddr::V4(ip) => is_local_v4(ip), - // An IPv4-mapped address reaches the same host as the v4 it wraps, so judge - // it by that rather than by the v6 rules. - std::net::IpAddr::V6(ip) => match ip.to_ipv4_mapped() { - Some(v4) => is_local_v4(v4), - None => ip.is_loopback() || ip.is_unspecified() || ip.is_unique_local() || ip.is_unicast_link_local(), - }, } } -fn is_local_v4(ip: std::net::Ipv4Addr) -> bool { - ip.is_loopback() || ip.is_private() || ip.is_link_local() || ip.is_unspecified() -} - /// How a reconnect loop reacts to a peer's GOAWAY. #[derive(Clone, Debug, usage_rs::Args, serde::Serialize, serde::Deserialize)] #[usage(unknown_flags = "error", args_override_self = false)] @@ -483,57 +351,6 @@ impl Default for Goaway { } } -/// Defaults for the [`Backoff`] knobs, applied by its accessors when a field is unset. -const DEFAULT_INITIAL: Duration = Duration::from_secs(1); -const DEFAULT_MULTIPLIER: u32 = 2; -const DEFAULT_MAX: Duration = Duration::from_secs(5); -const DEFAULT_HEALTHY: Duration = Duration::from_secs(1); -const DEFAULT_TIMEOUT: Duration = Duration::from_secs(10); - -/// The retry pacing a connection actually runs on, resolved once from [`Backoff`]. -#[derive(Clone, Copy)] -struct Pacing { - /// The first delay, already under the cap. - initial: Duration, - /// The delay cap, or `None` for uncapped. - max: Option, - /// At least 1: a multiplier of zero means no growth, not a delay that shrinks to nothing. - multiplier: u32, - /// The bar a session must clear to count as healthy. Never zero. - healthy: Duration, -} - -impl Pacing { - fn new(backoff: &Backoff) -> Self { - let max = Some(backoff.max()).filter(|max| !max.is_zero()); - let initial = backoff.initial(); - Self { - initial: max.map_or(initial, |max| initial.min(max)), - max, - multiplier: backoff.multiplier.max(1), - healthy: backoff.healthy(), - } - } - - /// The delay for the attempt after one that waited `delay`. - fn next(&self, delay: Duration) -> Duration { - let next = delay.saturating_mul(self.multiplier); - self.max.map_or(next, |max| next.min(max)) - } - - /// How long a session must last to start a new run of redirects. - /// - /// One that outlived the capped delay already redials no faster than the backoff would - /// allow. Uncapped, the healthy bar is the only rate the config names. - fn settled(&self) -> Duration { - self.max.unwrap_or(self.healthy) - } -} - -/// Default handover window, and the ceiling a GOAWAY deadline is capped to when -/// the config names none. -const DEFAULT_HANDOVER: Duration = Duration::from_secs(10); - impl Goaway { pub(crate) fn deprecated(&self) -> crate::cli::Deprecated { let mut found = crate::cli::Deprecated::default(); @@ -554,26 +371,16 @@ impl Goaway { found } - fn resolve(&self) -> Resolved { - Resolved { - redirect: self.redirect, - handover: self.handover.map(Into::into).unwrap_or(DEFAULT_HANDOVER), + /// The policy these knobs configure, with each unset one at its default. + pub(crate) fn resolve(&self) -> moq_net::reconnect::Goaway { + let mut goaway = moq_net::reconnect::Goaway::default(); + goaway.redirect = self.redirect.into(); + if let Some(handover) = self.handover { + goaway.handover = handover.into(); } + goaway } } - -#[derive(Clone, Copy)] -struct Resolved { - redirect: Redirect, - handover: Duration, -} - -impl Resolved { - fn handover(self, timeout: Option) -> Duration { - std::cmp::min(self.handover, timeout.unwrap_or(Duration::MAX)) - } -} - /// Shared reconnect state, observed by consumers through a [`kio`] channel. /// /// The channel closing (all producers dropped) is the terminal signal; `error` @@ -606,7 +413,7 @@ struct State { /// The currently-connected session, or `None` while reconnecting. Read by /// [`Monitor`] to snapshot live connection stats. session: Option, - /// The loop's [`Draining`] predecessor and its handover deadline, so + /// The policy's draining predecessor and its handover deadline, so /// [`Connection::close`] drains it too without overstaying that window. predecessor: Option<(moq_net::Session, kio::clock::Instant)>, } @@ -673,12 +480,15 @@ impl Shared { } } - /// Nothing is live. Clears the session so a handle can't be handed a closed one, - /// and drops the estimates that belonged to it. - fn disconnected(&self) { - self.offline(None); + /// Mirror the policy's draining predecessor, for [`Connection::close`]. + fn predecessor(&self, predecessor: Option<(&moq_net::Session, kio::clock::Instant)>) { + if let Ok(mut state) = self.state.write() { + state.predecessor = predecessor.map(|(session, deadline)| (session.clone(), deadline)); + } } + /// Nothing is live. Clears the session so a handle can't be handed a closed one, + /// and drops the estimates that belonged to it. fn offline(&self, failure: Option<&Error>) { if let Ok(mut state) = self.state.write() { state.offline(); @@ -1011,7 +821,7 @@ impl Connection { let task_running = std::sync::Arc::downgrade(&running); let task = crate::runtime::spawn(async move { let shared = &running.shared; - let reconnect = client.reconnect; + let reconnect = !client.policy.once; // Isolated inside `shared`, so a panic still records a terminal error // instead of closing the channels as if the caller had stopped it. let res = crate::panic::isolate(Self::run(shared, client, addrs)) @@ -1117,328 +927,86 @@ impl Connection { Ok(session.and(predecessor)?) } + /// Drive moq-net's policy: dial each attempt it asks for and publish what it does. async fn run(shared: &Shared, client: Client, addrs: Addrs) -> crate::Result<()> { - let backoff = client.backoff.clone(); - let goaway = client.goaway.resolve(); - let pacing = Pacing::new(&backoff); - let timeout = backoff.timeout(); - let mut retry = Retry::new(pacing.initial); - // Sticky across migrations: a redirect is an assignment, not a detour, so a - // later drop redials wherever we were last sent. Scoped to this loop, so a - // fresh Connection starts from the configured addresses again. - let mut addrs = addrs; - // An old session kept alive after a GOAWAY so its in-flight groups finish. - let mut draining: Option = None; - // The current run of migrations, to tell a redirect loop from a healthy handover. - let mut redirects = Redirects::default(); + let mut policy = moq_net::Reconnect::new(client.policy, addrs.as_slice().to_vec(), kio::clock::now()) + .map_err(|_| Error::BackoffHealthy)? + .with_tokens(shared.tokens.clone(), client.versions.clone()); + // The dial in flight, and the address it was asked for. + let mut dial: Option<(url::Url, std::pin::Pin>>)> = None; + // Mirrors the live session's estimates while it serves. + let mut forward: Option = None; + let mut timer: Option = None; loop { - if !timeout.is_zero() && retry.start.elapsed() >= timeout { - return Err(timeout_error(timeout, retry.last_error.as_ref())); - } - // An ended auth handle refuses every later session, so no dial can recover. A fresh - // watch only reports an error once its handle has ended. - if let Poll::Ready(Err(err)) = shared.auth.grant().poll_changed(&kio::Waiter::noop()) { - return Err(err.into()); - } - - // What the last dial presented, so a refusal is judged against it. - let mut presented = None; - match Self::dial_any(shared, &client, &addrs, &mut draining, &mut presented).await { - Ok((addr, dialed)) => { - let url = addr.url().clone(); - tracing::info!(peer = %Endpoint(&url), transport = %dialed.transport, "connected"); - shared.connected(&dialed.session, dialed.transport); - shared.tokens.connected(&dialed.session); - let session = dialed.session; - - let connected = kio::clock::now(); - // Wait for the session to end, forwarding its bandwidth estimates into the - // persistent producers meanwhile so consumers track the live stats across the - // connection, and draining any predecessor left over from a migration. - let ended = run_session(shared, &session, &mut draining, client.restart).await; - // A refusal is terminal unless the tokens changed since this dial read them. - let refreshed = match &ended { - Ended::Closed(Err(err)) => refreshed(err, &shared.tokens, presented.as_ref()), - _ => false, - }; - - let tenure = connected.elapsed(); - - // The connected target owns the policy, including in one-shot mode. A - // refused redirect is terminal: the peer is leaving and named somewhere we - // won't go, so redialing the old address or a fallback would ignore it. - let assigned = match &ended { - // A fingerprint only checked the dial that installed the Rustls - // verifier. tcp, unix, and plaintext WebSocket never consult it. - Ended::Goaway(msg) => { - goaway - .redirect - .target(msg.uri(), &url, client.pinned && fingerprint_pins(url.scheme()))? - } - Ended::Closed(_) => None, - }; - if assigned.is_some() && addr.addresses().is_some() { - return Err(Error::PinnedRedirect); - } - - // One-shot mode leaves rather than migrating: there is no replacement to - // dial, and a GOAWAY naming no deadline never force-closes, so the peer - // stops accepting requests and waits for us. Ignoring it is how both - // sides end up waiting on each other forever. Checked before the - // migration branch below, which would otherwise loop back to redial. - if !client.reconnect && matches!(ended, Ended::Goaway(_)) { - return Ok(()); - } - - // Two peers that each hold a session past the healthy bar can still bounce us between - // them forever. Returning to an endpoint that already sent us elsewhere in this - // run of migrations skips the healthy-session reset, so the backoff escalates and - // the retry window still applies. A one-way rolling restart never revisits one, - // and a GOAWAY back to the same place is a restart, not a move, so it never - // counts. A session that outlived the capped delay already redials no faster - // than the backoff would allow, so it starts a new run: rolling restarts days - // apart never add up to a loop. - if matches!(ended, Ended::Closed(_)) || tenure >= pacing.settled() { - redirects.reset(); - } - // Only an accepted redirect is a move: an empty or ignored one reconnects in place. - let looped = assigned.as_ref().is_some_and(|target| { - let from = endpoint(&url); - let to = endpoint(target); - to != from && redirects.depart(from, &to) - }); - - // A session that stayed up past the healthy bar is healthy; one that - // ended sooner, or carried on a redirect loop, counts as a failed attempt. - let healthy = !looped && retry.settle(tenure, &pacing); - - if let Ended::Goaway(msg) = &ended { - // An accepted redirect is an assignment: keep dialing it from here on, and - // only it. The peer named exactly one place to go, which retires - // whatever other addresses got us to this session. - let url = if let Some(target) = assigned { - addrs = Addrs::new(target.clone()); - target - } else { - url - }; - - // Hand over gracefully however the backoff bookkeeping scores this - // session. The old one keeps serving until it closes or overstays, - // so its routes stay attached and live tracks splice onto the - // replacement at a group boundary. Tearing it down here instead - // would drop every group published until the replacement caught up. - tracing::info!(peer = %Endpoint(&url), "upstream GOAWAY; migrating"); - // Retire any predecessor first: overwriting would drop its deadline - // on the floor and leave it holding the connection open. - if let Some(mut old) = draining.take() { - old.retire(); - } - draining = Some(Draining::new(session, goaway.handover(msg.timeout()), &shared.state)); - // After the predecessor is published, so a close woken by this - // status finds it and honors its handover deadline. - shared.migrating(); - - // No backoff sleep: a handover off a healthy session is not a failure. - if healthy { - continue; - } - - // Drained almost immediately, or looped back. The backoff still escalates, so - // the redials cannot hammer the peer, and the old session serves across the - // sleep. - // - // An empty URI is "reconnect to me": the peer is restarting and drains - // each new session until it goes, which is no failure, since every one - // serves. A named URI is a redirect, scored as a failed attempt so two - // peers bouncing us between them eventually give up. - let restarting = msg.uri().is_empty(); - if !restarting { - let reason = if looped { - "redirect loop" - } else { - "peer redirected immediately" - }; - let err = Error::Reconnect(reason.to_string()); - shared.retrying(&err); - retry.last_error = Some(err); - } - // A restart's pacing is not capped by the window: a delay longer than the - // window would otherwise spend it all and end the loop without a redial. - let window = if restarting { Duration::ZERO } else { timeout }; - let Some(wait) = retry_wait(retry.delay, retry.start, window) else { - return Err(timeout_error(timeout, retry.last_error.as_ref())); - }; - if looped { - tracing::warn!(peer = %Endpoint(&url), ?wait, "peer redirected back in a loop; retrying after backoff"); - } else { - tracing::warn!(peer = %Endpoint(&url), ?wait, "peer drained the session immediately; retrying after backoff"); - } - // Keep the handover bounded across the sleep: nothing else polls the - // predecessor while the loop is between connections. - sleep_draining(wait, &mut draining, shared).await; - retry.delay = pacing.next(retry.delay); - if restarting { - // After the sleep, so the window counts only the time after the - // peer stops answering. - retry.start = kio::clock::now(); - } - continue; - } - - // Both returns below leave the offline bookkeeping to the terminal - // snapshot, so they never publish a `Disconnected` first. - // - // Refusals and malformed protocol input are settled failures, however long - // the session lived. A new connection cannot repair either one. - if let Ended::Closed(Err(err)) = &ended - && err.is_terminal() - && !refreshed + let action = kio::wait(|waiter| { + loop { + if let Some((_, attempt)) = dial.as_mut() + && let Poll::Ready(res) = waiter.poll_future(attempt.as_mut()) { - return Err(err.clone()); + let (url, _) = dial.take().expect("polled above"); + match res { + Ok(crate::client::Dialed { session, transport }) => { + tracing::info!(peer = %Endpoint(&url), %transport, "connected"); + shared.connected(&session, transport); + forward = Some(Forward::new(&session)); + policy.connected(kio::clock::now(), session); + } + Err(err) => { + let verdict = verdict(&err); + policy.failed(kio::clock::now(), err, verdict); + } + } } - - // One-shot mode: the session ending ends the connection. Its close - // reason is the terminal error, mirroring `moq_net::Session::closed`. - // A GOAWAY already returned above. - if !client.reconnect { - return match ended { - Ended::Closed(res) => res, - Ended::Goaway(_) => unreachable!("handled before the migration branch"), - }; + if let (Some(forward), Some(session)) = (forward.as_mut(), policy.session()) { + forward.poll(shared, session, waiter); } - // Publish the lost transport and why, in one lifecycle observation. - let failure = match &ended { - Ended::Closed(Err(err)) => Some(err), - _ => None, + let at = match policy.poll(kio::clock::now(), waiter) { + Action::Wait(Some(at)) => at, + Action::Wait(None) => return Poll::Pending, + action => return Poll::Ready(action), }; - shared.offline(failure); - - if healthy { - // The backoff window was reset, so a one-off drop reconnects promptly. - tracing::warn!(peer = %Endpoint(&url), "session closed, reconnecting"); - } else { - // Connected then dropped almost immediately (e.g. the server accepts then - // resets, or redirects us straight back out). Treat it as a failed - // connection: keep the reason so the give-up timeout reports a real cause, - // and fall through to the shared backoff sleep below so repeated flaps - // escalate instead of spinning the CPU. - let err = match ended { - Ended::Closed(Err(err)) => Some(err), - Ended::Closed(Ok(())) => None, - // Handled above: a GOAWAY never reaches here. - Ended::Goaway(_) => None, - }; - // NOTE: only UNAUTHORIZED is specified, and it is handled above. Any - // other MoQ-layer rejection (Request::reject after the transport is - // accepted) lands here as an untyped transport close, so it cannot be - // told apart from a network blip and is retried until the give-up - // timeout. Classifying the rest needs the transport to surface the - // close code; until then, one-shot mode (`reconnect = false`) is how - // a caller observes those rejections directly. - match err { - Some(err) => { - tracing::warn!(peer = %Endpoint(&url), %err, "session severed immediately, retrying"); - retry.last_error = Some(err); - } - None => tracing::warn!(peer = %Endpoint(&url), "session severed immediately, retrying"), - } + let sleep = timer.get_or_insert_with(|| kio::clock::sleep_until(at)); + if sleep.deadline() != at { + sleep.reset(at); } - } - Err(err) => { - if (err.is_terminal() && !refreshed(&err, &shared.tokens, presented.as_ref())) || !client.reconnect - { - return Err(err); + if waiter.poll_future(std::pin::Pin::new(sleep)).is_pending() { + return Poll::Pending; } - shared.retrying(&err); - retry.last_error = Some(err); } - } - - let Some(wait) = retry_wait(retry.delay, retry.start, timeout) else { - return Err(timeout_error(timeout, retry.last_error.as_ref())); - }; - // No URL here: with several candidates there isn't one to name, and each - // attempt already logged the address it tried. - tracing::warn!(?wait, "reconnecting after backoff"); - // Drain-aware: a GOAWAY off a healthy session continues straight to the - // replacement dial, so a predecessor can still be draining when that dial - // fails and lands here. A plain sleep would stop enforcing its handover - // until some later dial succeeded, holding an upstream that asked to drain - // open for the whole retry window, or forever with `--connect-backoff-timeout=0`. - sleep_draining(wait, &mut draining, shared).await; - retry.delay = pacing.next(retry.delay); - } - } + }) + .await; - /// Try each address in turn, returning the first session that connects and the - /// address that produced it. - /// - /// A peer discovered rather than configured can advertise several addresses, - /// only some of which route from here (its loopback, a container bridge, an - /// interface on another subnet). Nothing in the record says which, so every - /// attempt walks the whole list rather than pinning whichever sorted first. - /// - /// A predecessor left over from a migration keeps draining across the whole - /// walk, not just one attempt: its handover deadline is wall-clock, and a walk - /// past several black-holed addresses is exactly when it would overstay. - async fn dial_any( - shared: &Shared, - client: &Client, - addrs: &Addrs, - draining: &mut Option, - presented: &mut Option, - ) -> crate::Result<(crate::connect::Addr, crate::client::Dialed)> { - let candidates = addrs.as_slice(); - let mut last = None; - let mut transient = None; - - for (index, addr) in candidates.iter().enumerate() { - let url = addr.url(); - tracing::info!(peer = %Endpoint(url), "connecting"); - - // Held before the dial starts its session's driver, so a request the session - // sends early waits on their answers rather than being refused outside the - // connection credential's grant. - let presenting = presented.insert(shared.tokens.dial(url, &client.versions)); - let mut dial = std::pin::pin!(client.dial(addr.clone(), presenting)); - let dialed = kio::wait(|waiter| { - if poll_draining(draining, waiter) { - shared.disconnected(); + match action { + Action::Dial(attempt) => { + let url = attempt.target().url().clone(); + dial = Some((url, Box::pin(client.dial(attempt)))); } - waiter.poll_future(dial.as_mut()) - }); - - let dialed = match attempt_timeout(index, candidates.len()) { - None => dialed.await, - Some(limit) => match kio::clock::timeout(limit, dialed).await { - Ok(dialed) => dialed, - Err(_) => Err(Error::Reconnect(format!("timed out connecting to {}", Endpoint(url)))), - }, - }; - - match dialed { - Ok(dialed) => return Ok((addr.clone(), dialed)), - // A status the peer actually sent is its answer, not this address's, so - // unless it invites another attempt it settles the whole walk. Carrying - // on would offer the same rejected credentials at the peer's other - // addresses, and worse, a later transport failure would overwrite the - // real reason and leave the outer loop retrying until its budget ran out. - Err(err) if settles_walk(&err) => return Err(err), - Err(err) => { - tracing::debug!(peer = %Endpoint(url), %err, "address unreachable"); - if !err.is_terminal() { - transient = Some(err.clone()); - } - last = Some(err); + // The policy already counted the timeout; dropping the dial cancels it. + Action::Abandon => dial = None, + Action::Migrating => { + forward = None; + // After the predecessor is published, so a close woken by this status + // finds it and honors its handover deadline. + shared.predecessor(policy.predecessor()); + shared.migrating(); + } + Action::Retired => shared.predecessor(None), + Action::Offline(failure) => { + forward = None; + shared.offline(failure.as_ref()); + } + Action::Retrying(err) => shared.retrying(&err), + // The terminal snapshot does the offline bookkeeping, so a watcher never sees a + // `Disconnected` the loop will not redial from. + Action::Done(res) => { + shared.predecessor(None); + return res; } + Action::Wait(_) => unreachable!("handled while polling"), } } - - // One unusable candidate must not turn another candidate's transient failure terminal. - Err(transient.or(last).expect("Addrs is never empty")) } /// Poll until a session is established (the first connect, or the current one). @@ -1631,296 +1199,31 @@ impl Connection { } } -/// The reconnect loop's backoff window across a run of failed attempts. -struct Retry { - /// The wait before the next attempt, before jitter. - delay: Duration, - /// When this run of failures began; the give-up timeout counts from here. - start: kio::clock::Instant, - /// The latest failure, so giving up reports a real cause. - last_error: Option, -} - -impl Retry { - fn new(initial: Duration) -> Self { - Self { - delay: initial, - start: kio::clock::now(), - last_error: None, - } - } - - /// Score how long a session served: past the healthy bar starts a fresh retry window. - fn settle(&mut self, tenure: Duration, pacing: &Pacing) -> bool { - let healthy = tenure >= pacing.healthy; - if healthy { - *self = Self::new(pacing.initial); - } - healthy - } -} - -/// Build the terminal error for an exhausted retry window without discarding the last real cause. -fn timeout_error(timeout: Duration, last_error: Option<&Error>) -> Error { - let message = match last_error { - Some(err) => format!("reconnect timed out after {timeout:?}: {err}"), - None => format!("reconnect timed out after {timeout:?}"), - }; - Error::Reconnect(message) -} - -/// Jitter the next delay and cap it at the retry window. `None` means the window is exhausted. -fn retry_wait(delay: Duration, retry_start: kio::clock::Instant, timeout: Duration) -> Option { - let wait = delay.mul_f64(0.5 + kio::clock::rng().random::() / 2.0); - if timeout.is_zero() { - return Some(wait); - } - Some(wait.min(timeout.checked_sub(retry_start.elapsed())?)) -} - -/// Where `url` dials, without its userinfo, query, or fragment, where a peer may mint fresh -/// credentials on every redirect. Two URLs that differ only there reach the same endpoint. -fn endpoint(url: &Url) -> Url { - let mut bare = url.clone(); - // Only fails for a URL that cannot carry userinfo, which then has none to strip. - let _ = bare.set_username(""); - let _ = bare.set_password(None); - bare.set_query(None); - bare.set_fragment(None); - bare -} - -/// A run of GOAWAY redirects, and whether it has revisited an endpoint. -/// -/// The peer picks every hop, so remembering them all would grow without bound. Brent's cycle -/// detection instead keeps one departed endpoint, replaced at doubling intervals, and catches a -/// cycle of any length within a few laps of it. Once caught, the run stays a loop until reset. -struct Redirects { - mark: Option, - span: u64, - hops: u64, - looped: bool, -} - -impl Default for Redirects { - fn default() -> Self { - Self { - mark: None, - span: 1, - hops: 0, - looped: false, - } - } -} - -impl Redirects { - /// Record leaving `from` for a different endpoint `to`, and return whether this run loops. - fn depart(&mut self, from: Url, to: &Url) -> bool { - if self.mark.as_ref() == Some(to) { - self.looped = true; - } - self.hops += 1; - if self.hops == self.span { - self.mark = Some(from); - self.span = self.span.saturating_mul(2); - self.hops = 0; - } - self.looped - } - - /// Start a new run. - fn reset(&mut self) { - *self = Self::default(); - } -} - -/// Why a session stopped being the live one. -enum Ended { - /// The transport closed. - Closed(crate::Result<()>), - /// The peer sent a GOAWAY; the session is still up and serving. - Goaway(moq_net::goaway::Goaway), -} +/// A dial in flight. +type Dial<'a> = dyn Future> + Send + 'a; -/// An old session kept alive after a GOAWAY so its in-flight groups finish. -/// -/// Held by the reconnect loop rather than a detached task, so dropping the -/// [`Connection`] handle tears the old session down with everything else instead -/// of leaving an orphan holding the connection open. -struct Draining { - session: moq_net::Session, - /// Resolves once the session closes, on its own or through [`Self::poll_idle`]. - closed: std::pin::Pin + Send>>, - deadline: std::pin::Pin>, - /// Whether [`Self::poll_idle`] started the graceful close. - closing: bool, - /// Mirrors the session into [`State::predecessor`] for as long as this lives. - state: kio::Producer, +/// Mirrors one live session's send and receive estimates into the persistent producers, so +/// [`Connection`] consumers track the live estimates across the connection. +struct Forward { + send: Option, + recv: RecvBandwidth, } -impl Draining { - fn new(session: moq_net::Session, handover: Duration, state: &kio::Producer) -> Self { - let deadline = Box::pin(kio::clock::sleep(handover)); - if let Ok(mut state) = state.write() { - state.predecessor = Some((session.clone(), deadline.deadline())); - } - let closed = { - let session = session.clone(); - Box::pin(async move { - session.closed().await; - }) as std::pin::Pin + Send>> - }; - +impl Forward { + fn new(session: &moq_net::Session) -> Self { Self { - session, - closed, - deadline, - closing: false, - state: state.clone(), - } - } - - /// Close the old session now, whatever remains of its window. - fn retire(&mut self) { - self.session.abort(moq_net::Error::GoawayTimeout); - } - - /// Close the old session gracefully once it carries no subscription or fetch. - /// - /// Only while a replacement serves: until then everything new still lands here. The - /// handover window still bounds the close. - fn poll_idle(&mut self, waiter: &kio::Waiter) { - if self.closing || self.session.poll_idle(waiter).is_pending() { - return; + send: session.send_bandwidth(), + recv: RecvBandwidth::default(), } - tracing::debug!("old session carries nothing after GOAWAY; closing"); - self.closing = true; - let session = self.session.clone(); - self.closed = Box::pin(async move { - // A drain cut short by its own deadline still ends the session. - let _ = session.close().await; - }); } - /// Poll the drain; `true` once it is over and the handle should be released. - fn poll(&mut self, waiter: &kio::Waiter) -> bool { - if waiter.poll_future(self.closed.as_mut()).is_ready() { - tracing::debug!("old session drained cleanly after GOAWAY"); - return true; - } - if waiter.poll_future(self.deadline.as_mut()).is_ready() { - tracing::warn!("old session did not drain in time; closing"); - self.session.abort(moq_net::Error::GoawayTimeout); - return true; - } - false + /// A `poll_*` step: on return, `waiter` is registered for the next change. + fn poll(&mut self, shared: &Shared, session: &moq_net::Session, waiter: &kio::Waiter) { + poll_forward(&mut self.send, &shared.send_bw, waiter); + self.recv.poll(&shared.recv_bw, waiter, || session.recv_bandwidth()); } } -impl Drop for Draining { - fn drop(&mut self) { - if let Ok(mut state) = self.state.write() { - state.predecessor = None; - } - } -} - -/// Poll a draining predecessor, retiring it once it closes or overstays its -/// handover window. `true` on the pass that retires it. -/// -/// A `poll_*` step: on return the waiter is registered for the next change. The -/// caller decides what the retirement means, because that depends on whether a -/// replacement is already serving. -fn poll_draining(draining: &mut Option, waiter: &kio::Waiter) -> bool { - if let Some(old) = draining.as_mut() - && old.poll(waiter) - { - *draining = None; - return true; - } - false -} - -/// Sleep for `delay` while keeping a draining predecessor's deadline enforced. -/// -/// The drain is otherwise only polled from [`run_session`], which is reached only -/// after a successful connect, so a replacement that takes several backoff rounds -/// to reach would leave the old session holding its connection well past the -/// handover window. -async fn sleep_draining(delay: Duration, draining: &mut Option, shared: &Shared) { - let mut sleep = std::pin::pin!(kio::clock::sleep(delay)); - kio::wait(|waiter| { - // Nothing is serving once it retires: the predecessor closed or overstayed - // its handover, and the replacement is still a dial away. Leaving the state - // on `Migrating` would keep handing callers a session that is already closed, - // with its stats and version, until the next connect overwrote it. - if poll_draining(draining, waiter) { - shared.disconnected(); - } - waiter.poll_future(sleep.as_mut()) - }) - .await -} - -/// Wait for `session` to close or receive a GOAWAY, forwarding its send/recv bandwidth estimates -/// into the persistent producers meanwhile so [`Connection`] consumers track the live estimates -/// across the connection, and closing any predecessor left over from an earlier migration once -/// it carries nothing. -/// -/// One `poll_*` step drives it all: [`poll_forward`] mirrors each kio bandwidth estimate, the -/// GOAWAY consumer is a kio channel, and the transport's close future (the one non-kio source) is -/// polled through the waiter's own waker. -/// -/// Under [`Restart::Wait`] a GOAWAY naming no URI does not end the wait: the session serves on -/// until it closes. -async fn run_session( - shared: &Shared, - session: &moq_net::Session, - draining: &mut Option, - restart: Restart, -) -> Ended { - let mut send = session.send_bandwidth(); - let mut recv = RecvBandwidth::default(); - // Dropped once a restart is waited out: it is a latch, so it would keep firing. - let mut goaway = Some(session.draining()); - let closed = session.closed(); - tokio::pin!(closed); - - kio::wait(|waiter| { - poll_forward(&mut send, &shared.send_bw, waiter); - recv.poll(&shared.recv_bw, waiter, || session.recv_bandwidth()); - - // The replacement serves, so a predecessor carrying nothing closes now. One that - // closes or overstays its handover window retires, and the state stays as it is. - if let Some(old) = draining.as_mut() { - old.poll_idle(waiter); - } - poll_draining(draining, waiter); - - // Checked before the close arm: a GOAWAY means the session is still up, and - // migrating from it is not the same as reconnecting after it died. - // - // Polled in one-shot mode too, even though there is no replacement to dial. - // A GOAWAY naming no deadline never force-closes, and the peer stops - // accepting requests and waits for us to leave, so ignoring it is how both - // sides end up waiting on each other forever. The caller ends the connection - // instead: asked to leave, with no way to migrate, leaving is the answer. - if let Some(Poll::Ready(Ok(msg))) = goaway.as_ref().map(|goaway| goaway.poll(waiter)) { - if restart == Restart::Wait && msg.uri().is_empty() { - tracing::info!("peer is restarting; serving until it closes the session"); - goaway = None; - } else { - return Poll::Ready(Ended::Goaway(msg)); - } - } - - waiter - .poll_future(closed.as_mut()) - .map(|err| Ended::Closed(Err(err.into()))) - }) - .await -} - /// Own the live session's receive consumer only while an external consumer needs it. #[derive(Default)] struct RecvBandwidth { @@ -2039,14 +1342,9 @@ mod tests { assert_eq!(parsed.backoff.healthy, Some((Duration::from_secs(2)).into())); assert_eq!(parsed.backoff.timeout(), Duration::ZERO, "0 means retry forever"); - // With no flags, the parser defaults resolve to the typed defaults. + // With no flags, the parser defaults resolve to the policy's defaults. let parsed = Wrapper::parse_from(&[]).unwrap(); - assert_eq!(parsed.backoff.multiplier, DEFAULT_MULTIPLIER); - let pacing = Pacing::new(&parsed.backoff); - assert_eq!(pacing.initial, DEFAULT_INITIAL); - assert_eq!(pacing.max, Some(DEFAULT_MAX)); - assert_eq!(pacing.healthy, DEFAULT_HEALTHY); - assert_eq!(parsed.backoff.timeout(), DEFAULT_TIMEOUT); + assert_eq!(parsed.backoff.resolve(), moq_net::reconnect::Backoff::default()); // And a flag still lands where the merge can see it. let parsed = Wrapper::parse_from(&[ @@ -2054,7 +1352,7 @@ mod tests { std::ffi::OsStr::new("3s"), ]) .unwrap(); - assert_eq!(Pacing::new(&parsed.backoff).initial, Duration::from_secs(3)); + assert_eq!(parsed.backoff.resolve().initial, Duration::from_secs(3)); } /// GOAWAY typed defaults remain overrideable by a standing TOML layer. @@ -2071,9 +1369,7 @@ mod tests { // No flags passed: the typed defaults are present. let parsed = Wrapper::parse_from(&[]).unwrap(); assert_eq!(parsed.goaway.redirect, Redirect::SameHost); - let resolved = parsed.goaway.resolve(); - assert_eq!(resolved.redirect, Redirect::SameHost); - assert_eq!(resolved.handover(None), Duration::from_secs(10)); + assert_eq!(parsed.goaway.resolve(), moq_net::reconnect::Goaway::default()); // Flags passed: they land where the merge can see them. let parsed = Wrapper::parse_from(&[ @@ -2084,323 +1380,9 @@ mod tests { ]) .unwrap(); assert_eq!(parsed.goaway.redirect, Redirect::Ignore); - assert_eq!(parsed.goaway.resolve().handover(None), Duration::from_secs(3)); - } - - /// Our own window caps the peer's. A GOAWAY deadline shortens the handover but - /// never extends it, so an upstream naming an absurd one cannot make us hold a - /// dying session open for it. - #[test] - fn handover_takes_the_earlier_deadline() { - let config = Goaway { - handover: Some((Duration::from_secs(10)).into()), - ..Default::default() - }; - - let config = config.resolve(); - assert_eq!(config.handover(None), Duration::from_secs(10), "no deadline: ours"); - assert_eq!( - config.handover(Some(Duration::from_secs(3))), - Duration::from_secs(3), - "a shorter peer deadline wins: it force-closes then anyway" - ); - assert_eq!( - config.handover(Some(Duration::from_secs(3600))), - Duration::from_secs(10), - "a longer peer deadline must not extend our cap" - ); - - // The default is a cap too, not just a fallback for a silent peer. - let default = Goaway::default().resolve(); - assert_eq!(default.handover(Some(Duration::from_secs(3600))), DEFAULT_HANDOVER); - } - - /// A peer must not be able to redirect us somewhere we could not already - /// reach. IPv4-mapped and link-local forms are the easy ones to miss. - #[test] - fn local_targets_are_recognized() { - let local = [ - "https://127.0.0.1/", - "https://localhost/", - "https://[::1]/", - "https://10.0.0.1/", - "https://169.254.1.1/", - "https://[::ffff:127.0.0.1]/", - "https://[::ffff:10.0.0.1]/", - "https://[fe80::1]/", - "https://[fc00::1]/", - "https://0.0.0.0/", - ]; - for url in local { - assert!(is_local(&url.parse().unwrap()), "{url} should be local"); - } - - for url in ["https://example.com/", "https://8.8.8.8/", "https://[2606:4700::1]/"] { - assert!(!is_local(&url.parse().unwrap()), "{url} should not be local"); - } - - // A public endpoint may not redirect us inward, but a local one may stay local. - let public: Url = "https://relay.example/".parse().unwrap(); - let localhost: Url = "https://127.0.0.1:4443/".parse().unwrap(); - assert!(matches!( - Redirect::Follow.target("https://[::ffff:127.0.0.1]/", &public, false), - Err(Error::RefusedRedirect(_)) - )); - assert_eq!( - Redirect::Follow - .target("https://127.0.0.1:9999/", &localhost, false) - .unwrap() - .unwrap() - .port(), - Some(9999) - ); - } - - /// The scheme ranking is what stops a peer from quietly moving us off an - /// encrypted transport. An unknown scheme ranks lowest so a classification we - /// forgot to add is refused rather than trusted. - #[test] - fn scheme_tiers_rank_encrypted_above_plaintext() { - for scheme in ["https", "moqt", "moql", "wss", "tls"] { - assert_eq!(scheme_tier(scheme), 2, "{scheme} is encrypted"); - } - for scheme in ["tcp", "ws", "http"] { - assert_eq!(scheme_tier(scheme), 1, "{scheme} is plaintext"); - } - // `unix` is not an upgrade over a network transport, it is a different - // reachability class, so it must not outrank one. - for scheme in ["unix", "gopher", ""] { - assert_eq!(scheme_tier(scheme), 0, "{scheme} is unclassified"); - } - } - - /// A redirect may hold the scheme or improve it, never weaken it. A downgrade - /// is a refusal, not a reason to redial the address the peer is leaving. - #[test] - fn target_refuses_a_scheme_downgrade() { - let secure: Url = "https://relay.example/".parse().unwrap(); - let plain: Url = "http://relay.example/".parse().unwrap(); - - assert!(matches!( - Redirect::Follow.target("http://other.example/", &secure, false), - Err(Error::RefusedRedirect(_)) - )); - assert!(matches!( - Redirect::Follow.target("unix:///tmp/moq.sock", &secure, false), - Err(Error::RefusedRedirect(_)) - )); - - // Same tier and upgrades are followed. - let same: Url = "https://other.example/".parse().unwrap(); - assert_eq!( - Redirect::Follow - .target("https://other.example/", &secure, false) - .unwrap(), - Some(same.clone()) - ); - assert_eq!( - Redirect::Follow - .target("https://other.example/", &plain, false) - .unwrap(), - Some(same) - ); - } - - /// Native schemes must classify address literals even when the URL parser returns a domain. - #[test] - fn a_non_special_scheme_classifies_its_literal_host() { - for url in [ - "moqt://127.0.0.1/", - "moql://10.0.0.1/", - "tcp://169.254.169.254/", - "moqt://[::1]/", - "moqt://[::ffff:127.0.0.1]/", - ] { - assert!(is_local(&url.parse().unwrap()), "{url} should be local"); - } - for url in ["moqt://8.8.8.8/", "moql://[2606:4700::1]/", "tcp://relay.example/"] { - assert!(!is_local(&url.parse().unwrap()), "{url} should not be local"); - } - - let public: Url = "moqt://relay.example/".parse().unwrap(); - assert!( - matches!( - Redirect::Follow.target("moqt://169.254.169.254/", &public, false), - Err(Error::RefusedRedirect(_)) - ), - "a literal local target is refused whatever the scheme" - ); - } - - /// The reachability guard reads the URL, so it cannot see a local address - /// hiding behind a hostname: any name the peer controls can carry an A record - /// for loopback or a private range, and a second lookup at dial time may answer - /// differently anyway. The default policy is what closes that, by refusing to - /// let a peer name a host at all. - #[test] - fn a_peer_named_host_is_refused_by_default() { - let public: Url = "https://relay.example/".parse().unwrap(); - - // The bypass: a name is not local on its face, so the reachability guard - // passes it through whatever it resolves to. - let rebindable: Url = "https://rebind.attacker.example/".parse().unwrap(); - assert!(!is_local(&rebindable), "a hostname is never classified as local"); - assert_eq!( - Redirect::Follow.target(rebindable.as_str(), &public, false).unwrap(), - Some(rebindable.clone()), - "Follow is explicit trust: it dials the name the peer chose" - ); - - // So the default refuses the host change instead of trying to judge it. - assert_eq!(Redirect::default(), Redirect::SameHost); - assert!( - matches!( - Redirect::default().target(rebindable.as_str(), &public, false), - Err(Error::RefusedRedirect(_)) - ), - "a peer-named host is refused without resolving it" - ); - assert_eq!( - Goaway::default().resolve().redirect, - Redirect::SameHost, - "and the shipped config carries that default" - ); - } - - /// Only an explicit URI can assign, and one the policy will not follow is an - /// error rather than a quiet fallback: the loop ends on it instead of redialing. - #[test] - fn only_an_accepted_redirect_replaces_the_address_list() { - let current: Url = "https://relay.example/".parse().unwrap(); - - // No URI, or a policy that ignores it: keep the current address list. - assert_eq!(Redirect::Follow.target("", ¤t, false).unwrap(), None); - assert_eq!( - Redirect::Ignore - .target("https://relay.example:5443/", ¤t, false) - .unwrap(), - None - ); - - // An explicit URI the policy will not follow is refused. - for (policy, uri) in [ - (Redirect::SameHost, "https://other.example/"), - (Redirect::Follow, "not a url"), - (Redirect::Ignore, "not a url"), - (Redirect::Follow, "http://relay.example/"), - (Redirect::Follow, "https://127.0.0.1/"), - ] { - assert!( - matches!(policy.target(uri, ¤t, false), Err(Error::RefusedRedirect(_))), - "{policy:?}: {uri}" - ); - } - - // An explicit assignment remains an assignment even if its URL is unchanged. - assert_eq!( - Redirect::SameHost.target(current.as_str(), ¤t, false).unwrap(), - Some(current.clone()) - ); - let moved = "https://relay.example:5443/"; - assert_eq!( - Redirect::SameHost.target(moved, ¤t, false).unwrap(), - Some(moved.parse().unwrap()) - ); - } - - /// A fingerprint is a Rustls check. Schemes that never install that verifier - /// must not inherit the pin. - #[test] - fn a_fingerprint_pin_only_covers_rustls_schemes() { - for scheme in ["https", "wss", "moqt", "moql", "tls"] { - assert!(fingerprint_pins(scheme), "{scheme}"); - } - for scheme in ["http", "ws", "tcp", "unix"] { - assert!(!fingerprint_pins(scheme), "{scheme}"); - } - } - - /// A certificate pin verifies only the host it was configured for, so it - /// refuses a host change even when the policy would follow one. - #[test] - fn a_certificate_pin_holds_the_host() { - let current: Url = "https://relay.example/".parse().unwrap(); - assert!(matches!( - Redirect::Follow.target("https://other.example/", ¤t, true), - Err(Error::RefusedRedirect(_)) - )); - let moved = "https://relay.example:5443/"; - assert_eq!( - Redirect::Follow.target(moved, ¤t, true).unwrap(), - Some(moved.parse().unwrap()) - ); - } - - /// `SameHost` lets a peer move us between ports or schemes on the endpoint we - /// already chose, but not onto a different host. - #[test] - fn same_host_follows_a_port_not_a_host() { - let current: Url = "https://relay.example:4443/".parse().unwrap(); - - assert!( - matches!( - Redirect::SameHost.target("https://elsewhere.example/", ¤t, false), - Err(Error::RefusedRedirect(_)) - ), - "another host is refused" - ); - - let moved = Redirect::SameHost - .target("https://relay.example:5443/", ¤t, false) - .unwrap() - .unwrap(); - assert_eq!( - moved.port(), - Some(5443), - "a different port on the same host is followed" - ); - } - - /// Each knob means one thing at zero: no wait, no cap, no growth. None of them touches - /// the healthy bar, which is what keeps an immediate close a failed attempt. - #[test] - fn zero_turns_each_pacing_knob_off() { - let pacing = Pacing::new(&Backoff { - initial: Some((Duration::ZERO).into()), - ..Default::default() - }); - assert_eq!(pacing.initial, Duration::ZERO); - assert_eq!(pacing.next(pacing.initial), Duration::ZERO, "zero initial never waits"); - assert_eq!(pacing.healthy, DEFAULT_HEALTHY); - - let pacing = Pacing::new(&Backoff { - max: Some((Duration::ZERO).into()), - ..Default::default() - }); - assert_eq!(pacing.max, None); - let mut delay = pacing.initial; - for _ in 0..10 { - delay = pacing.next(delay); - } - assert_eq!(delay, DEFAULT_INITIAL * 1024, "zero max leaves the delay uncapped"); - assert_eq!(pacing.settled(), DEFAULT_HEALTHY); - - let pacing = Pacing::new(&Backoff { - multiplier: 0, - ..Default::default() - }); - assert_eq!( - pacing.next(pacing.initial), - pacing.initial, - "zero multiplier never grows" - ); - - // A cap below the first delay caps it too. - let pacing = Pacing::new(&Backoff { - max: Some((Duration::from_millis(300)).into()), - ..Default::default() - }); - assert_eq!(pacing.initial, Duration::from_millis(300)); + let resolved = parsed.goaway.resolve(); + assert_eq!(resolved.redirect, moq_net::reconnect::Redirect::Ignore); + assert_eq!(resolved.handover, Duration::from_secs(3)); } /// A zero healthy bar would count every session healthy, so it is refused at build. @@ -2415,23 +1397,6 @@ mod tests { assert!(matches!(err, Error::BackoffHealthy), "{err}"); } - /// A sane config is left alone: the delay doubles up to the cap and stays there. - #[test] - fn pacing_grows_to_the_cap() { - let pacing = Pacing::new(&Backoff { - initial: Some((Duration::from_millis(100)).into()), - multiplier: 2, - max: Some((Duration::from_millis(400)).into()), - ..Default::default() - }); - - assert_eq!(pacing.initial, Duration::from_millis(100)); - assert_eq!(pacing.settled(), Duration::from_millis(400)); - assert_eq!(pacing.next(Duration::from_millis(100)), Duration::from_millis(200)); - assert_eq!(pacing.next(Duration::from_millis(200)), Duration::from_millis(400)); - assert_eq!(pacing.next(Duration::from_millis(400)), Duration::from_millis(400)); - } - #[test] fn poll_forward_mirrors_until_the_source_closes() { let src = BandwidthProducer::new(); @@ -2765,45 +1730,38 @@ mod tests { /// that doesn't route from here ahead of the one that does. #[cfg(feature = "tcp")] #[tokio::test] - async fn dial_any_walks_past_the_dead_addresses() { - let client = client(); + async fn the_walk_lands_past_the_dead_addresses() { let (mut server, live) = live().await; - tokio::spawn(async move { while server.accept().await.is_ok() {} }); + let (accepted, mut sessions) = tokio::sync::mpsc::unbounded_channel(); + tokio::spawn(async move { + while let Ok(request) = server.accept().await { + let _ = accepted.send(()); + let _ = request.ok().await; + } + }); let dead_a = dead().await; let dead_b = dead().await; let addrs = Addrs::collect([dead_a, dead_b, live.clone()]).expect("not empty"); - let shared = shared(); - let mut draining = None; - - let (connected, _session) = kio::clock::timeout( - WALK_TIMEOUT, - Connection::dial_any(&shared, &client, &addrs, &mut draining, &mut None), - ) - .await - .expect("the walk must not hang") - .expect("the live address must connect"); - assert_eq!(connected.url(), &live, "the walk must land on the one that answers"); + let connection = kio::clock::timeout(WALK_TIMEOUT, client().with_reconnect(false).connect(addrs).established()) + .await + .expect("the walk must not hang") + .expect("the live address must connect"); + assert_eq!(connection.connects(), 1); + assert!(sessions.recv().await.is_some(), "the walk must land on the one that answers"); } /// With nothing reachable the walk reports a failure rather than hanging. #[cfg(feature = "tcp")] #[tokio::test] - async fn dial_any_reports_failure_when_nothing_answers() { - let client = client(); - + async fn the_walk_fails_when_nothing_answers() { let dead_a = dead().await; let dead_b = dead().await; let addrs = Addrs::collect([dead_a, dead_b]).expect("not empty"); - let shared = shared(); - let mut draining = None; - - let result = kio::clock::timeout( - WALK_TIMEOUT, - Connection::dial_any(&shared, &client, &addrs, &mut draining, &mut None), - ) - .await - .expect("the walk must not hang"); + let connection = client().with_reconnect(false).connect(addrs); + let result = kio::clock::timeout(WALK_TIMEOUT, connection.closed()) + .await + .expect("the walk must not hang"); assert!(result.is_err(), "every address hung up, so the walk must fail"); } @@ -2813,27 +1771,25 @@ mod tests { /// it in, and a dial URL logged verbatim would hand a replayable credential to /// anyone who can read logs. This drives the real `tracing` stack rather than /// [`Endpoint`]'s `Display`, so it also catches a log site that forgot to use - /// it. + /// it. moq-net's own log sites have a test of their own. #[cfg(feature = "tcp")] #[tracing_test::traced_test] #[tokio::test] async fn dialing_never_logs_the_credential() { const SECRET: &str = "b91d7fe20c4a"; - let client = client(); let mut target = dead().await; target.set_path(&format!("/.cluster/{SECRET}")); - let addrs = Addrs::new(target); - let shared = shared(); - let mut draining = None; - // `Session` isn't `Debug`, so unwrap the failure by hand. - let Err(err) = Connection::dial_any(&shared, &client, &addrs, &mut draining, &mut None).await else { - panic!("a dead address must fail"); - }; + let err = client() + .with_reconnect(false) + .connect(target) + .closed() + .await + .expect_err("a dead address must fail"); // The log line did happen, so the assertion below isn't vacuously true. - assert!(logs_contain("connecting"), "the dial never logged at all"); + assert!(logs_contain("connection closed"), "the loop never logged at all"); assert!(!logs_contain(SECRET), "a log line leaked the credential"); assert!(!format!("{err}").contains(SECRET), "the error leaked it: {err}"); } @@ -3164,23 +2120,21 @@ mod tests { /// failure from another candidate cannot replace it and keep an unlimited loop retrying. #[test] fn an_auth_rejection_stops_the_walk() { - assert!(settles_walk(&Error::Connect(crate::ConnectError::Unauthorized))); - assert!(settles_walk(&Error::Connect(crate::ConnectError::Forbidden))); - assert!(settles_walk(&Error::MoqNet(moq_net::Error::ProtocolViolation))); - assert!(!settles_walk(&Error::MoqNet(moq_net::Error::Closed))); - assert!(!settles_walk(&Error::Io(std::sync::Arc::new( - std::io::ErrorKind::ConnectionRefused.into() - )))); - } - - /// Only an attempt with somewhere to fall back on gets the fixed bound. - #[test] - fn only_a_candidate_with_a_fallback_is_bounded() { - assert_eq!(attempt_timeout(0, 1), None, "a lone address"); - assert_eq!(attempt_timeout(0, 2), Some(CONNECT_ATTEMPT), "one more to try"); - assert_eq!(attempt_timeout(1, 2), None, "the last of two"); - assert_eq!(attempt_timeout(1, 3), Some(CONNECT_ATTEMPT), "still one more"); - assert_eq!(attempt_timeout(2, 3), None, "the last of three"); + assert_eq!( + verdict(&Error::Connect(crate::ConnectError::Unauthorized)), + Verdict::Refused + ); + assert_eq!(verdict(&Error::Connect(crate::ConnectError::Forbidden)), Verdict::Refused); + assert_eq!(verdict(&Error::MoqNet(moq_net::Error::Unauthorized)), Verdict::Refused); + assert_eq!(verdict(&Error::MoqNet(moq_net::Error::ProtocolViolation)), Verdict::Fatal); + assert_eq!(verdict(&Error::MoqNet(moq_net::Error::Closed)), Verdict::Retry); + assert_eq!( + verdict(&Error::Io(std::sync::Arc::new( + std::io::ErrorKind::ConnectionRefused.into() + ))), + Verdict::Retry + ); + assert_eq!(verdict(&Error::NoBackend("none")), Verdict::Unusable); } /// Relays QUIC datagrams to `server`, holding them all until [`Forwarder::open`]. @@ -3506,132 +2460,13 @@ mod tests { .unwrap(); } - /// What a flapping peer does with each session the moment it is up. - #[cfg(feature = "tcp")] - #[derive(Clone)] - enum Flap { - /// Drop it, closing the session. - Close, - /// Send a GOAWAY naming this URI, empty for a restart, and keep serving. - Goaway(String), - } - - /// A client with the given pacing whose dials reach in-memory peers instead of a - /// transport, so the paused clock only moves once every task waits on a timer. - /// - /// The peer at each URL ends every session as its [`Flap`] says, and reports the - /// URL and instant each session came up on the returned channel. - #[cfg(feature = "tcp")] - fn flapping( - initial: Duration, - max: Duration, - timeout: Duration, - peers: Vec<(Url, Flap)>, - ) -> (Client, tokio::sync::mpsc::UnboundedReceiver<(Url, kio::clock::Instant)>) { - let mut config = crate::connect::Config::default(); - config.backoff.initial = Some(initial.into()); - config.backoff.max = Some(max.into()); - config.backoff.timeout = Some(timeout.into()); - let mut client = config.init(Default::default()).expect("build client"); - - let (accepted, accepts) = tokio::sync::mpsc::unbounded_channel(); - client.dialer = Some(std::sync::Arc::new(move |addr: crate::connect::Addr| { - let url = addr.url().clone(); - let (_, flap) = peers - .iter() - .find(|(peer, _)| *peer == url) - .expect("an unknown peer") - .clone(); - let accepted = accepted.clone(); - Box::pin(async move { - let session = flap_once(flap).await; - let _ = accepted.send((url, kio::clock::now())); - Ok(crate::client::Dialed { - session, - transport: crate::Transport::Tcp, - }) - }) - })); - (client, accepts) - } - - /// Run one in-memory MoQ handshake whose server side then ends the session as `flap` says. - #[cfg(feature = "tcp")] - async fn flap_once(flap: Flap) -> moq_net::Session { - let versions = moq_net::Versions::from("moq-lite-07-wip".parse::().unwrap()); - let config = || { - let mut config = qmux::Config::new(qmux::Version::QMux01); - config.protocol = qmux::Protocol::Negotiate(versions.alpns().iter().map(|alpn| alpn.to_string()).collect()); - config - }; - let stream = |io| qmux::transport::Stream::new(io); - let (client, server) = tokio::io::duplex(64 * 1024); - let (client, server) = tokio::try_join!( - qmux::Session::connect(stream(client), config()), - qmux::Session::accept(stream(server), config()) - ) - .unwrap(); - let (client, server) = (crate::spawn_qmux(client), crate::spawn_qmux(server)); - let now = moq_net::time::Instant::now(); - let client = async { - let (session, driver) = moq_net::Client::new() - .with_versions(versions.clone()) - .connect(now, client) - .await - .unwrap(); - tokio::spawn(moq_net::time::run(driver)); - session - }; - let server = async { - let request = moq_net::Server::new() - .with_versions(versions.clone()) - .accept_request(now, server) - .await - .unwrap(); - let (session, driver) = request.ok().await.unwrap(); - tokio::spawn(moq_net::time::run(driver)); - session - }; - let (client, server) = tokio::join!(client, server); - match flap { - Flap::Close => drop(server), - Flap::Goaway(uri) => { - let goaway = match uri.is_empty() { - true => moq_net::goaway::Goaway::new(), - false => moq_net::goaway::Goaway::redirect(uri), - }; - server.drain().send(goaway).unwrap(); - tokio::spawn(async move { - server.closed().await; - }); - } - } - client - } - - /// Check each gap between consecutive sessions against the jittered delay it should - /// have waited: between half of `delay` and all of it, starting at `initial` and - /// doubling up to `max`. - #[cfg(feature = "tcp")] - fn assert_paced(accepts: &[(Url, kio::clock::Instant)], initial: Duration, max: Duration) { - let mut delay = initial; - for (index, pair) in accepts.windows(2).enumerate() { - let gap = pair[1].1 - pair[0].1; - assert!( - gap >= delay / 2 && gap <= delay, - "retry {index} waited {gap:?}, outside the jittered {delay:?}" - ); - delay = (delay * 2).min(max); - } - } - /// A configured token is held before its dial starts the session, so a broadcast /// published before the dial, which only that token's grant covers, waits for its /// answer instead of closing the session against the connection credential's grant. #[cfg(feature = "tcp")] #[tokio::test(start_paused = true)] async fn a_publication_waits_for_a_configured_token() { - let shared = shared(); + let version: moq_net::Version = "moq-lite-07-wip".parse().unwrap(); let publish = crate::origin::spawn(); let broadcast = publish.create_broadcast("room").unwrap(); broadcast.announce(moq_net::origin::Route::default()).unwrap(); @@ -3639,15 +2474,15 @@ mod tests { let config = crate::connect::Config { tokens: vec!["connection".parse().unwrap(), "publish".parse().unwrap()], + version: vec![version], ..Default::default() }; let mut client = config.init(Default::default()).expect("build client"); - let auth = shared.auth.clone(); - let (publish_dial, server_dial) = (publish.clone(), server_origin.clone()); - client.dialer = Some(std::sync::Arc::new(move |_addr: crate::connect::Addr| { - let (auth, publish, server_origin) = (auth.clone(), publish_dial.clone(), server_dial.clone()); + let server_dial = server_origin.clone(); + client.dialer = Some(std::sync::Arc::new(move |_addr: crate::connect::Addr, moq: moq_net::Client| { + let server_origin = server_dial.clone(); Box::pin(async move { - let versions = moq_net::Versions::from("moq-lite-07-wip".parse::().unwrap()); + let versions = moq_net::Versions::from(version); let config = || { let mut config = qmux::Config::new(qmux::Version::QMux01); config.protocol = @@ -3662,10 +2497,6 @@ mod tests { .unwrap(); let (client, server) = (crate::spawn_qmux(client), crate::spawn_qmux(server)); let now = moq_net::time::Instant::now(); - let moq = moq_net::Client::new() - .with_versions(versions.clone()) - .with_auth(auth.clone()) - .with_publisher(&publish); let client = moq.connect(now, client); let server = async { let mut request = moq_net::Server::new() @@ -3708,9 +2539,10 @@ mod tests { }) })); - let addrs = Addrs::new("moqt://relay.example.com/".parse::().unwrap()); - shared.tokens.set(client.tokens.clone()); - let (_, dialed) = Connection::dial_any(&shared, &client, &addrs, &mut None, &mut None) + let connection = client + .with_publisher(&publish) + .connect("moqt://relay.example.com/".parse::().unwrap()) + .established() .await .expect("the dial failed"); @@ -3721,173 +2553,10 @@ mod tests { .expect("the server's origin closed"); assert!(matches!(arrived, moq_net::announce::Event::Start(update) if update.prefix.as_str() == "room")); assert!( - futures::FutureExt::now_or_never(dialed.session.closed()).is_none(), + connection.connected() && connection.connects() == 1, "the session closed before the configured token was answered" ); } - - #[cfg(feature = "tcp")] - const INITIAL: Duration = Duration::from_millis(100); - #[cfg(feature = "tcp")] - const MAX: Duration = Duration::from_millis(400); - - /// A peer that accepts and then closes every session at once never earns the healthy - /// reset: the delay escalates to the cap, and the timeout ends the loop. - #[cfg(feature = "tcp")] - #[tokio::test(start_paused = true)] - async fn an_immediate_close_loop_escalates_until_the_timeout() { - const TIMEOUT: Duration = Duration::from_secs(2); - let peer = url("tcp://peer:1/"); - let (client, mut accepted) = flapping(INITIAL, MAX, TIMEOUT, vec![(peer.clone(), Flap::Close)]); - - let started = kio::clock::now(); - let err = client - .connect(peer) - .closed() - .await - .expect_err("a flapping peer kept the connection"); - assert!(matches!(err, Error::Reconnect(_)), "{err}"); - assert_eq!(started.elapsed(), TIMEOUT, "the timeout did not end the loop"); - - let mut accepts = Vec::new(); - while let Ok(accept) = accepted.try_recv() { - accepts.push(accept); - } - assert!(accepts.len() >= 6, "only {} sessions", accepts.len()); - assert_paced(&accepts, INITIAL, MAX); - } - - /// A peer restarting with an empty-URI GOAWAY on every session paces the redials like - /// failures, yet never spends the timeout: each drained session still served. - #[cfg(feature = "tcp")] - #[tokio::test(start_paused = true)] - async fn an_immediate_restart_loop_escalates_without_giving_up() { - const TIMEOUT: Duration = Duration::from_millis(500); - let peer = url("tcp://peer:1/"); - let (client, mut accepted) = flapping(INITIAL, MAX, TIMEOUT, vec![(peer.clone(), Flap::Goaway(String::new()))]); - let connection = client.connect(peer); - - let mut accepts = Vec::new(); - for _ in 0..10 { - accepts.push(accepted.recv().await.expect("the dialer dropped")); - } - assert!( - accepts[9].1 - accepts[0].1 > TIMEOUT * 2, - "the redials outran the backoff" - ); - assert!( - futures::FutureExt::now_or_never(connection.closed()).is_none(), - "a restarting peer ended the connection" - ); - assert_paced(&accepts, INITIAL, MAX); - } - - /// Two peers that redirect to each other the moment a session is up are failed attempts: - /// the delay escalates across both, and the timeout ends the loop. - #[cfg(feature = "tcp")] - #[tokio::test(start_paused = true)] - async fn an_immediate_redirect_loop_escalates_until_the_timeout() { - const TIMEOUT: Duration = Duration::from_secs(2); - let a = url("tcp://peer:1/"); - let b = url("tcp://peer:2/"); - let peers = vec![ - (a.clone(), Flap::Goaway(b.to_string())), - (b.clone(), Flap::Goaway(a.to_string())), - ]; - let (client, mut accepted) = flapping(INITIAL, MAX, TIMEOUT, peers); - - let started = kio::clock::now(); - let err = client - .connect(a.clone()) - .closed() - .await - .expect_err("a redirect loop kept the connection"); - assert!( - matches!(&err, Error::Reconnect(reason) if reason.contains("redirect")), - "{err}" - ); - assert_eq!(started.elapsed(), TIMEOUT, "the timeout did not end the loop"); - - let mut accepts = Vec::new(); - while let Ok(accept) = accepted.try_recv() { - accepts.push(accept); - } - assert!(accepts.len() >= 6, "only {} sessions", accepts.len()); - for (index, (peer, _)) in accepts.iter().enumerate() { - assert_eq!( - *peer, - if index % 2 == 0 { a.clone() } else { b.clone() }, - "session {index}" - ); - } - assert_paced(&accepts, INITIAL, MAX); - } - - /// A zero `initial` turns retry waiting off: every redial follows the close at once. - #[cfg(feature = "tcp")] - #[tokio::test(start_paused = true)] - async fn a_zero_initial_redials_at_once() { - const TIMEOUT: Duration = Duration::from_secs(1); - let peer = url("tcp://peer:1/"); - let (client, mut accepted) = flapping(Duration::ZERO, MAX, TIMEOUT, vec![(peer.clone(), Flap::Close)]); - let connection = client.connect(peer); - - let first = accepted.recv().await.expect("the dialer dropped").1; - for _ in 0..10 { - let at = accepted.recv().await.expect("the dialer dropped").1; - assert_eq!(at, first, "a zero initial waited before a retry"); - } - drop(connection); - } - - #[test] - fn redirect_loop_detection_catches_any_cycle_in_constant_memory() { - for tail in 0..4 { - for cycle in 2..=8 { - // `tail` one-way moves, then `cycle` endpoints redirecting around a ring. - let at = |hop: usize| { - if hop < tail { - url(&format!("tcp://tail{hop}/")) - } else { - url(&format!("tcp://ring{}/", (hop - tail) % cycle)) - } - }; - let mut redirects = Redirects::default(); - let mut caught = None; - for hop in 0..4 * (tail + cycle) { - let looped = redirects.depart(at(hop), &at(hop + 1)); - match caught { - Some(_) => assert!(looped, "a caught loop stays caught"), - None if looped => caught = Some(hop), - None => {} - } - } - let caught = caught.unwrap_or_else(|| panic!("tail {tail}, cycle {cycle}: never caught")); - // Never before the run first returns to an endpoint it left. - assert!( - caught + 1 >= tail + cycle, - "tail {tail}, cycle {cycle}: caught at {caught}" - ); - } - } - - let mut redirects = Redirects::default(); - for hop in 0..1000 { - let from = url(&format!("tcp://a{hop}/")); - let to = url(&format!("tcp://a{}/", hop + 1)); - assert!(!redirects.depart(from, &to), "a one-way run never loops"); - } - } - - #[test] - fn an_endpoint_ignores_the_credentials() { - assert_eq!( - endpoint(&url("tcp://token-a:secret@relay:1/path?jwt=a#x")), - endpoint(&url("tcp://token-b@relay:1/path?jwt=b")) - ); - // The path is part of the request target, so it names a different endpoint. - assert_ne!(endpoint(&url("tcp://relay:1/a")), endpoint(&url("tcp://relay:1/b"))); - } } #[cfg(test)] @@ -3914,7 +2583,7 @@ mod duration_tests { Backoff, healthy, "--connect-backoff-healthy", - |c: &Backoff| c.healthy(), + |c: &Backoff| c.resolve().healthy, |d| d ); crate::cli::duration::test_config_duration!( diff --git a/rs/moq-tokio/src/error.rs b/rs/moq-tokio/src/error.rs index 7f98c3d6e6e..628d0155897 100644 --- a/rs/moq-tokio/src/error.rs +++ b/rs/moq-tokio/src/error.rs @@ -275,13 +275,8 @@ impl Error { #[cfg(feature = "websocket")] Self::TransportRace { quic, websocket } => quic.is_terminal() && websocket.is_terminal(), Self::MoqNet(err) => matches!( - moq_net::SessionError::from(err), - moq_net::SessionError::Unauthorized - | moq_net::SessionError::Expired - | moq_net::SessionError::MalformedAuthToken - | moq_net::SessionError::ProtocolViolation - | moq_net::SessionError::KeyValueFormatting - | moq_net::SessionError::InvalidRequestId + moq_net::reconnect::Verdict::from(err), + moq_net::reconnect::Verdict::Refused | moq_net::reconnect::Verdict::Fatal ), Self::Tls(err) => err.is_terminal(), Self::NoBackend(_) @@ -367,6 +362,20 @@ impl Error { } } +/// What a connection reports when moq-net's reconnect policy fails an attempt or gives up. +#[cfg(feature = "net")] +impl From for Error { + fn from(failure: moq_net::reconnect::Failure) -> Self { + use moq_net::reconnect::Failure; + match failure { + Failure::Timeout(timeout) => Self::ConnectTimeout(timeout), + Failure::Refused(reason) => Self::RefusedRedirect(reason.to_string()), + Failure::Fixed => Self::PinnedRedirect, + failure => Self::Reconnect(failure.to_string()), + } + } +} + // The wrapped sources aren't `Clone`, so `#[from]` can't store them behind `Arc` // directly. These hand-written conversions keep `?` ergonomic at the call sites. impl From for Error { From 75f51d1df484bb05eadca4b1d381d611dbdfbc19 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 12:35:35 -0800 Subject: [PATCH 03/10] fix(net): settle the policy tests and skip no-op predecessor writes Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/reconnect/tests.rs | 6 ++++-- rs/moq-tokio/src/connection.rs | 5 +++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/rs/moq-net/src/reconnect/tests.rs b/rs/moq-net/src/reconnect/tests.rs index c3185a9f836..82bc949efa0 100644 --- a/rs/moq-net/src/reconnect/tests.rs +++ b/rs/moq-net/src/reconnect/tests.rs @@ -329,11 +329,13 @@ async fn only_an_address_with_a_fallback_is_bounded() { .await .unwrap() .expect_err("nothing answered"); - assert_eq!(run.events, ["abandon", "abandon"]); + assert_eq!(run.events, ["abandon", "abandon", "retrying"]); assert_eq!(run.dials, targets, "a walk under way finishes past the retry window"); assert_eq!(kio::clock::now() - started, ATTEMPT + config.timeout); + // The give-up names the last address's connect timeout as its cause. + let last = Failure::Timeout(config.timeout).to_string(); assert!( - matches!(&err, TestError::Policy(Failure::Timeout(timeout)) if *timeout == config.timeout), + matches!(&err, TestError::Policy(Failure::Exhausted { last: Some(cause), .. }) if *cause == last), "{err}" ); } diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs index b35618f25c2..31aa625fbbd 100644 --- a/rs/moq-tokio/src/connection.rs +++ b/rs/moq-tokio/src/connection.rs @@ -482,6 +482,10 @@ impl Shared { /// Mirror the policy's draining predecessor, for [`Connection::close`]. fn predecessor(&self, predecessor: Option<(&moq_net::Session, kio::clock::Instant)>) { + // A write wakes every watcher, so skip one that changes nothing. + if predecessor.is_none() && self.state.read().predecessor.is_none() { + return; + } if let Ok(mut state) = self.state.write() { state.predecessor = predecessor.map(|(session, deadline)| (session.clone(), deadline)); } @@ -1310,6 +1314,7 @@ fn terminal(state: &State) -> Error { #[cfg(test)] mod tests { use super::*; + use url::Url; /// A millisecond track: the frames written here carry timestamps. fn timed() -> moq_net::track::Info { From 7f5492af5f3c288ba8b8ed58446a368fdef76bb6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 12:49:27 -0800 Subject: [PATCH 04/10] chore: format, and retarget the reconnect drill mutation at moq-net Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/reconnect.rs | 4 +- rs/moq-net/src/reconnect/redirect.rs | 36 +++- rs/moq-net/src/reconnect/tests.rs | 159 ++++++++++++++---- rs/moq-tokio/src/client.rs | 5 +- rs/moq-tokio/src/connection.rs | 144 +++++++++------- .../reconnect-stops-after-session-loss.patch | 28 +-- 6 files changed, 257 insertions(+), 119 deletions(-) diff --git a/rs/moq-net/src/reconnect.rs b/rs/moq-net/src/reconnect.rs index 08c05f25c49..587d112cadc 100644 --- a/rs/moq-net/src/reconnect.rs +++ b/rs/moq-net/src/reconnect.rs @@ -334,7 +334,9 @@ enum Phase { /// Check the retry window, then walk the targets. Start, /// Ask the caller to dial the target at `index`. - Next { index: usize }, + Next { + index: usize, + }, Dialing(Dialing), Live(Box), Backoff { diff --git a/rs/moq-net/src/reconnect/redirect.rs b/rs/moq-net/src/reconnect/redirect.rs index 90ba212326c..cf2b6325745 100644 --- a/rs/moq-net/src/reconnect/redirect.rs +++ b/rs/moq-net/src/reconnect/redirect.rs @@ -261,9 +261,11 @@ mod tests { // A public endpoint may not redirect us inward, but a local one may stay local. let public = url("https://relay.example/"); let localhost = url("https://127.0.0.1:4443/"); - assert!(refused( - Redirect::Follow.target("https://[::ffff:127.0.0.1]/", &public, false) - )); + assert!(refused(Redirect::Follow.target( + "https://[::ffff:127.0.0.1]/", + &public, + false + ))); assert_eq!( Redirect::Follow .target("https://127.0.0.1:9999/", &localhost, false) @@ -299,7 +301,11 @@ mod tests { let secure = url("https://relay.example/"); let plain = url("http://relay.example/"); - assert!(refused(Redirect::Follow.target("http://other.example/", &secure, false))); + assert!(refused(Redirect::Follow.target( + "http://other.example/", + &secure, + false + ))); assert!(refused(Redirect::Follow.target("unix:///tmp/moq.sock", &secure, false))); // Same tier and upgrades are followed. @@ -311,7 +317,9 @@ mod tests { Some(same.clone()) ); assert_eq!( - Redirect::Follow.target("https://other.example/", &plain, false).unwrap(), + Redirect::Follow + .target("https://other.example/", &plain, false) + .unwrap(), Some(same) ); } @@ -433,9 +441,16 @@ mod tests { #[test] fn a_certificate_pin_holds_the_host() { let current = url("https://relay.example/"); - assert!(refused(Redirect::Follow.target("https://other.example/", ¤t, true))); + assert!(refused(Redirect::Follow.target( + "https://other.example/", + ¤t, + true + ))); let moved = "https://relay.example:5443/"; - assert_eq!(Redirect::Follow.target(moved, ¤t, true).unwrap(), Some(url(moved))); + assert_eq!( + Redirect::Follow.target(moved, ¤t, true).unwrap(), + Some(url(moved)) + ); } /// `SameHost` lets a peer move us between ports or schemes on the endpoint we @@ -453,7 +468,11 @@ mod tests { .target("https://relay.example:5443/", ¤t, false) .unwrap() .unwrap(); - assert_eq!(moved.port(), Some(5443), "a different port on the same host is followed"); + assert_eq!( + moved.port(), + Some(5443), + "a different port on the same host is followed" + ); } #[test] @@ -512,4 +531,3 @@ mod tests { assert_eq!(Shown(&url("unix:///run/moq.sock")).to_string(), "unix:///run/moq.sock"); } } - diff --git a/rs/moq-net/src/reconnect/tests.rs b/rs/moq-net/src/reconnect/tests.rs index 82bc949efa0..bda6264c6e3 100644 --- a/rs/moq-net/src/reconnect/tests.rs +++ b/rs/moq-net/src/reconnect/tests.rs @@ -220,7 +220,11 @@ async fn an_immediate_close_loop_escalates_until_the_timeout() { matches!(err, TestError::Policy(Failure::Exhausted { last: Some(_), .. })), "{err}" ); - assert_eq!(kio::clock::now() - started, TIMEOUT, "the timeout did not end the policy"); + assert_eq!( + kio::clock::now() - started, + TIMEOUT, + "the timeout did not end the policy" + ); assert!(run.accepts.len() >= 6, "only {} sessions", run.accepts.len()); assert_paced(&run.accepts, INITIAL, MAX); } @@ -267,10 +271,18 @@ async fn an_immediate_redirect_loop_escalates_until_the_timeout() { .unwrap() .expect_err("a redirect loop kept the connection"); assert!(err.to_string().contains("redirect"), "{err}"); - assert_eq!(kio::clock::now() - started, TIMEOUT, "the timeout did not end the policy"); + assert_eq!( + kio::clock::now() - started, + TIMEOUT, + "the timeout did not end the policy" + ); assert!(run.accepts.len() >= 6, "only {} sessions", run.accepts.len()); for (index, (peer, _)) in run.accepts.iter().enumerate() { - assert_eq!(*peer, if index % 2 == 0 { a.clone() } else { b.clone() }, "session {index}"); + assert_eq!( + *peer, + if index % 2 == 0 { a.clone() } else { b.clone() }, + "session {index}" + ); } assert_paced(&run.accepts, INITIAL, MAX); } @@ -288,7 +300,10 @@ async fn a_zero_initial_redials_at_once() { .unwrap(); let mut run = Run::default(); - drive(&mut machine, &[(peer, Peer::Close)], &mut run, |run| run.accepts.len() == 11).await; + drive(&mut machine, &[(peer, Peer::Close)], &mut run, |run| { + run.accepts.len() == 11 + }) + .await; for (_, at) in &run.accepts { assert_eq!(*at, first, "a zero initial waited before a retry"); } @@ -309,7 +324,11 @@ async fn the_walk_lands_on_the_first_address_that_answers() { drive(&mut machine, &peers, &mut run, |run| !run.accepts.is_empty()).await; assert_eq!(run.dials, targets, "every address in order"); - assert_eq!(run.accepts, [(targets[2].clone(), started)], "no wait between addresses"); + assert_eq!( + run.accepts, + [(targets[2].clone(), started)], + "no wait between addresses" + ); assert!(machine.session().is_some()); } @@ -349,11 +368,20 @@ async fn a_settled_answer_stops_the_walk() { (targets[0].clone(), Peer::Fail(Verdict::Settled)), (targets[1].clone(), Peer::Serve), ]; - let mut machine = Reconnect::new(config(INITIAL, Duration::from_secs(1)), targets.clone(), kio::clock::now()).unwrap(); + let mut machine = Reconnect::new( + config(INITIAL, Duration::from_secs(1)), + targets.clone(), + kio::clock::now(), + ) + .unwrap(); let mut run = Run::default(); drive(&mut machine, &peers, &mut run, |run| run.dials.len() == 2).await; - assert_eq!(run.dials, [targets[0].clone(), targets[0].clone()], "the walk stopped at the answer"); + assert_eq!( + run.dials, + [targets[0].clone(), targets[0].clone()], + "the walk stopped at the answer" + ); assert_eq!(run.events, ["retrying"]); } @@ -374,10 +402,15 @@ async fn an_unusable_address_ends_only_a_walk_without_hope() { let lone = url("tcp://c:1/"); let started = kio::clock::now(); let mut machine = Reconnect::new(config(INITIAL, Duration::ZERO), vec![lone.clone()], started).unwrap(); - let err = drive(&mut machine, &[(lone, Peer::Fail(Verdict::Unusable))], &mut Run::default(), |_| false) - .await - .unwrap() - .expect_err("an unusable address connected"); + let err = drive( + &mut machine, + &[(lone, Peer::Fail(Verdict::Unusable))], + &mut Run::default(), + |_| false, + ) + .await + .unwrap() + .expect_err("an unusable address connected"); assert!(matches!(err, TestError::Dial), "{err}"); assert_eq!(kio::clock::now(), started, "ended without waiting"); } @@ -419,7 +452,10 @@ async fn an_ended_auth_handle_ends_the_policy() { .with_tokens(tokens.clone(), version().into()); let mut run = Run::default(); - drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| !run.accepts.is_empty()).await; + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; tokens.handle().abort(Error::App(9)); run.servers.clear(); let ended = drive(&mut machine, &[(peer, Peer::Serve)], &mut run, |_| false).await; @@ -440,16 +476,40 @@ async fn connecting_once_ends_with_the_session() { }; let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); - let ended = drive(&mut machine, &[(peer.clone(), Peer::Goaway(String::new()))], &mut Run::default(), |_| false).await; + let ended = drive( + &mut machine, + &[(peer.clone(), Peer::Goaway(String::new()))], + &mut Run::default(), + |_| false, + ) + .await; assert!(matches!(ended, Some(Ok(()))), "a GOAWAY is the cue to leave"); let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); - let ended = drive(&mut machine, &[(peer.clone(), Peer::Close)], &mut Run::default(), |_| false).await; - assert!(matches!(ended, Some(Err(TestError::Net(_)))), "the close reason ends it"); + let ended = drive( + &mut machine, + &[(peer.clone(), Peer::Close)], + &mut Run::default(), + |_| false, + ) + .await; + assert!( + matches!(ended, Some(Err(TestError::Net(_)))), + "the close reason ends it" + ); let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); - let ended = drive(&mut machine, &[(peer, Peer::Fail(Verdict::Retry))], &mut Run::default(), |_| false).await; - assert!(matches!(ended, Some(Err(TestError::Dial))), "a failed dial is not retried"); + let ended = drive( + &mut machine, + &[(peer, Peer::Fail(Verdict::Retry))], + &mut Run::default(), + |_| false, + ) + .await; + assert!( + matches!(ended, Some(Err(TestError::Dial))), + "a failed dial is not retried" + ); } /// A target dialing fixed socket addresses cannot follow a redirect. @@ -458,7 +518,13 @@ async fn a_fixed_target_refuses_a_redirect() { let peer = url("moqt://relay.example:4443/"); let moved = url("moqt://relay.example:5443/"); let mut machine = Reconnect::new(Config::default(), vec![Fixed(peer.clone())], kio::clock::now()).unwrap(); - let ended = drive(&mut machine, &[(peer, Peer::Goaway(moved.to_string()))], &mut Run::default(), |_| false).await; + let ended = drive( + &mut machine, + &[(peer, Peer::Goaway(moved.to_string()))], + &mut Run::default(), + |_| false, + ) + .await; assert!(matches!(ended, Some(Err(TestError::Policy(Failure::Fixed))))); } @@ -485,17 +551,27 @@ async fn a_healthy_goaway_hands_over_to_the_redirect() { let b = url("tcp://peer:2/"); let mut machine = Reconnect::new(Config::default(), vec![a.clone()], kio::clock::now()).unwrap(); let mut run = Run::default(); - drive(&mut machine, &[(a.clone(), Peer::Serve)], &mut run, |run| !run.accepts.is_empty()).await; + drive(&mut machine, &[(a.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; // Past the healthy bar, the GOAWAY is a handover, not a failure. kio::clock::sleep(Duration::from_secs(2)).await; - run.servers[0].drain().send(goaway::Goaway::redirect(b.to_string())).unwrap(); + run.servers[0] + .drain() + .send(goaway::Goaway::redirect(b.to_string())) + .unwrap(); let peers = [(a.clone(), Peer::Serve), (b.clone(), Peer::Serve)]; let migrated = kio::clock::now(); drive(&mut machine, &peers, &mut run, |run| run.events.contains(&"retired")).await; assert_eq!(run.dials, [a, b.clone()]); assert_eq!(run.accepts[1], (b, migrated), "migrated without a backoff"); - assert_eq!(run.events, ["migrating", "retired"], "a replacement serves, so nothing went offline"); + assert_eq!( + run.events, + ["migrating", "retired"], + "a replacement serves, so nothing went offline" + ); assert!(machine.predecessor().is_none()); assert!(machine.session().is_some()); } @@ -510,7 +586,10 @@ async fn the_handover_takes_the_earlier_deadline() { config.goaway.handover = HANDOVER; let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); let mut run = Run::default(); - drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| !run.accepts.is_empty()).await; + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; kio::clock::sleep(Duration::from_secs(2)).await; run.servers[0] @@ -518,9 +597,16 @@ async fn the_handover_takes_the_earlier_deadline() { .send(goaway::Goaway::new().with_timeout(Duration::from_secs(3600))) .unwrap(); let drained = kio::clock::now(); - drive(&mut machine, &[(peer, Peer::Hang)], &mut run, |run| run.events.contains(&"offline")).await; + drive(&mut machine, &[(peer, Peer::Hang)], &mut run, |run| { + run.events.contains(&"offline") + }) + .await; assert_eq!(run.events, ["migrating", "retired", "offline"]); - assert_eq!(kio::clock::now() - drained, HANDOVER, "a longer peer deadline must not extend our cap"); + assert_eq!( + kio::clock::now() - drained, + HANDOVER, + "a longer peer deadline must not extend our cap" + ); let goaway = Goaway::default(); assert_eq!(goaway.handover(None), Duration::from_secs(10), "no deadline: ours"); @@ -540,18 +626,27 @@ async fn a_waited_restart_redials_only_once_the_session_closes() { config.restart = Restart::Wait; let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); let mut run = Run::default(); - drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| !run.accepts.is_empty()).await; + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; kio::clock::sleep(Duration::from_secs(2)).await; run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); kio::clock::sim::idle().await; assert!( - matches!(machine.poll(kio::clock::now(), &kio::Waiter::noop()), Action::Wait(None)), + matches!( + machine.poll(kio::clock::now(), &kio::Waiter::noop()), + Action::Wait(None) + ), "the policy left the session" ); run.servers.remove(0).abort(Error::GoawayTimeout); - drive(&mut machine, &[(peer, Peer::Serve)], &mut run, |run| run.accepts.len() == 2).await; + drive(&mut machine, &[(peer, Peer::Serve)], &mut run, |run| { + run.accepts.len() == 2 + }) + .await; assert_eq!(run.events, ["offline"]); } @@ -565,7 +660,9 @@ async fn a_waited_restart_redials_only_once_the_session_closes() { fn dialing_never_logs_the_credential() { const SECRET: &str = "b91d7fe20c4a"; let targets = vec![ - url(&format!("moqt://user:{SECRET}@a.example/.cluster/{SECRET}?jwt={SECRET}")), + url(&format!( + "moqt://user:{SECRET}@a.example/.cluster/{SECRET}?jwt={SECRET}" + )), url(&format!("moqt://b.example/.cluster/{SECRET}")), ]; let started = Instant::now(); @@ -640,7 +737,11 @@ fn zero_turns_each_pacing_knob_off() { multiplier: 0, ..defaults }); - assert_eq!(pacing.next(pacing.initial), pacing.initial, "zero multiplier never grows"); + assert_eq!( + pacing.next(pacing.initial), + pacing.initial, + "zero multiplier never grows" + ); // A cap below the first delay caps it too. let pacing = Pacing::new(&Backoff { diff --git a/rs/moq-tokio/src/client.rs b/rs/moq-tokio/src/client.rs index 2931345bfa2..85f224f3f6d 100644 --- a/rs/moq-tokio/src/client.rs +++ b/rs/moq-tokio/src/client.rs @@ -366,7 +366,10 @@ impl Client { /// closes once the last returned handle drops. The policy bounds the attempt: dropping /// it cancels whichever arm is still pending. #[cfg(any(feature = "quic", feature = "websocket", feature = "tcp", feature = "uds"))] - pub(crate) async fn dial(&self, attempt: moq_net::reconnect::Attempt) -> crate::Result { + pub(crate) async fn dial( + &self, + attempt: moq_net::reconnect::Attempt, + ) -> crate::Result { let addr = attempt.target().with_url(attempt.url().clone()); let client = match attempt.token() { Some(token) => { diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs index 31aa625fbbd..baff36fb6d6 100644 --- a/rs/moq-tokio/src/connection.rs +++ b/rs/moq-tokio/src/connection.rs @@ -1748,12 +1748,18 @@ mod tests { let dead_a = dead().await; let dead_b = dead().await; let addrs = Addrs::collect([dead_a, dead_b, live.clone()]).expect("not empty"); - let connection = kio::clock::timeout(WALK_TIMEOUT, client().with_reconnect(false).connect(addrs).established()) - .await - .expect("the walk must not hang") - .expect("the live address must connect"); + let connection = kio::clock::timeout( + WALK_TIMEOUT, + client().with_reconnect(false).connect(addrs).established(), + ) + .await + .expect("the walk must not hang") + .expect("the live address must connect"); assert_eq!(connection.connects(), 1); - assert!(sessions.recv().await.is_some(), "the walk must land on the one that answers"); + assert!( + sessions.recv().await.is_some(), + "the walk must land on the one that answers" + ); } /// With nothing reachable the walk reports a failure rather than hanging. @@ -2129,9 +2135,15 @@ mod tests { verdict(&Error::Connect(crate::ConnectError::Unauthorized)), Verdict::Refused ); - assert_eq!(verdict(&Error::Connect(crate::ConnectError::Forbidden)), Verdict::Refused); + assert_eq!( + verdict(&Error::Connect(crate::ConnectError::Forbidden)), + Verdict::Refused + ); assert_eq!(verdict(&Error::MoqNet(moq_net::Error::Unauthorized)), Verdict::Refused); - assert_eq!(verdict(&Error::MoqNet(moq_net::Error::ProtocolViolation)), Verdict::Fatal); + assert_eq!( + verdict(&Error::MoqNet(moq_net::Error::ProtocolViolation)), + Verdict::Fatal + ); assert_eq!(verdict(&Error::MoqNet(moq_net::Error::Closed)), Verdict::Retry); assert_eq!( verdict(&Error::Io(std::sync::Arc::new( @@ -2484,65 +2496,67 @@ mod tests { }; let mut client = config.init(Default::default()).expect("build client"); let server_dial = server_origin.clone(); - client.dialer = Some(std::sync::Arc::new(move |_addr: crate::connect::Addr, moq: moq_net::Client| { - let server_origin = server_dial.clone(); - Box::pin(async move { - let versions = moq_net::Versions::from(version); - let config = || { - let mut config = qmux::Config::new(qmux::Version::QMux01); - config.protocol = - qmux::Protocol::Negotiate(versions.alpns().iter().map(|alpn| alpn.to_string()).collect()); - config - }; - let (client, server) = tokio::io::duplex(64 * 1024); - let (client, server) = tokio::try_join!( - qmux::Session::connect(qmux::transport::Stream::new(client), config()), - qmux::Session::accept(qmux::transport::Stream::new(server), config()) - ) - .unwrap(); - let (client, server) = (crate::spawn_qmux(client), crate::spawn_qmux(server)); - let now = moq_net::time::Instant::now(); - let client = moq.connect(now, client); - let server = async { - let mut request = moq_net::Server::new() - .with_versions(versions.clone()) - .with_subscriber(server_origin) - .accept_request(now, server) - .await - .unwrap(); - // The connection credential grants nothing; only the configured token - // covers the broadcast. - let mut requests = request.requests(); - let (session, driver) = request.ok().await.unwrap(); - tokio::spawn(moq_net::time::run(driver)); - tokio::spawn(async move { - let _session = session; - let mut issued = Vec::new(); - while let Some(request) = requests.next().await { - let grant = match request.token().is_empty() { - true => moq_net::auth::Grant::default(), - false => moq_net::auth::Grant::all(), - }; - issued.push(request.accept(grant)); - } - }); - }; - let client = async { - let (session, driver) = client.await.unwrap(); - tokio::spawn(moq_net::time::run(driver)); - session - }; - let (session, ()) = tokio::join!(client, server); - // Leave the session the time a multithreaded runtime may give it before the dial - // returns: on the paused clock, every exchange the session can make lands first, - // the connection credential's answer included. - kio::clock::sleep(Duration::from_millis(10)).await; - Ok(crate::client::Dialed { - session, - transport: crate::Transport::Tcp, + client.dialer = Some(std::sync::Arc::new( + move |_addr: crate::connect::Addr, moq: moq_net::Client| { + let server_origin = server_dial.clone(); + Box::pin(async move { + let versions = moq_net::Versions::from(version); + let config = || { + let mut config = qmux::Config::new(qmux::Version::QMux01); + config.protocol = + qmux::Protocol::Negotiate(versions.alpns().iter().map(|alpn| alpn.to_string()).collect()); + config + }; + let (client, server) = tokio::io::duplex(64 * 1024); + let (client, server) = tokio::try_join!( + qmux::Session::connect(qmux::transport::Stream::new(client), config()), + qmux::Session::accept(qmux::transport::Stream::new(server), config()) + ) + .unwrap(); + let (client, server) = (crate::spawn_qmux(client), crate::spawn_qmux(server)); + let now = moq_net::time::Instant::now(); + let client = moq.connect(now, client); + let server = async { + let mut request = moq_net::Server::new() + .with_versions(versions.clone()) + .with_subscriber(server_origin) + .accept_request(now, server) + .await + .unwrap(); + // The connection credential grants nothing; only the configured token + // covers the broadcast. + let mut requests = request.requests(); + let (session, driver) = request.ok().await.unwrap(); + tokio::spawn(moq_net::time::run(driver)); + tokio::spawn(async move { + let _session = session; + let mut issued = Vec::new(); + while let Some(request) = requests.next().await { + let grant = match request.token().is_empty() { + true => moq_net::auth::Grant::default(), + false => moq_net::auth::Grant::all(), + }; + issued.push(request.accept(grant)); + } + }); + }; + let client = async { + let (session, driver) = client.await.unwrap(); + tokio::spawn(moq_net::time::run(driver)); + session + }; + let (session, ()) = tokio::join!(client, server); + // Leave the session the time a multithreaded runtime may give it before the dial + // returns: on the paused clock, every exchange the session can make lands first, + // the connection credential's answer included. + kio::clock::sleep(Duration::from_millis(10)).await; + Ok(crate::client::Dialed { + session, + transport: crate::Transport::Tcp, + }) }) - }) - })); + }, + )); let connection = client .with_publisher(&publish) diff --git a/test/drill/mutations/reconnect-stops-after-session-loss.patch b/test/drill/mutations/reconnect-stops-after-session-loss.patch index a1f9d891bff..7e7c159fef9 100644 --- a/test/drill/mutations/reconnect-stops-after-session-loss.patch +++ b/test/drill/mutations/reconnect-stops-after-session-loss.patch @@ -2,19 +2,19 @@ # expect: subscriber: reconnect loop gave up # # Removes redialing after an established session is lost. The reconnect loop -# treats a healthy session's end the way one-shot mode does and stops, so a +# treats a healthy session's end the way connecting once does and stops, so a # relay restart becomes a permanent outage instead of a bounded gap the same # handles resume across. -diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs ---- a/rs/moq-tokio/src/connection.rs -+++ b/rs/moq-tokio/src/connection.rs -@@ -1289,7 +1289,8 @@ - // One-shot mode: the session ending ends the connection. Its close - // reason is the terminal error, mirroring `moq_net::Session::closed`. - // A GOAWAY already returned above. -- if !client.reconnect { -+ // MUTATION: losing an established session ends the loop instead of redialing. -+ if !client.reconnect || healthy { - return match ended { - Ended::Closed(res) => res, - Ended::Goaway(_) => unreachable!("handled before the migration branch"), +diff --git a/rs/moq-net/src/reconnect.rs b/rs/moq-net/src/reconnect.rs +--- a/rs/moq-net/src/reconnect.rs ++++ b/rs/moq-net/src/reconnect.rs +@@ -772,7 +772,8 @@ impl + From> Reco + }; + let err = E::from(err); + // Connecting once, the session ending ends the policy with its close reason. +- if terminal || self.config.once { ++ // MUTATION: losing an established session ends the policy instead of redialing. ++ if terminal || self.config.once || healthy { + return self.done(Err(err)); + } + From d4c8dcf2fbba64c1533e703d14489676db0eb535 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 13:34:03 -0800 Subject: [PATCH 05/10] fix(tokio): clear the mirrored predecessor when the loop ends Co-Authored-By: Claude Opus 5.5 --- rs/moq-tokio/src/connection.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs index d525609cc46..6fdc756ae25 100644 --- a/rs/moq-tokio/src/connection.rs +++ b/rs/moq-tokio/src/connection.rs @@ -531,6 +531,10 @@ impl Shared { return; } state.offline(); + // Including when the task was aborted mid-handover: nothing enforces the + // predecessor's deadline anymore, so a parked clone would hold its transport open. + // A graceful close took its own clone before aborting. + state.predecessor = None; state.ended = true; state.failure = None; state.error = err; From 1c259a9ded7e32849ab690a0638be415b3590727 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 09:25:14 -0800 Subject: [PATCH 06/10] fix(net): a connect cancels the offline a mid-dial retirement queued A predecessor retiring while its replacement dials queues Retired then Offline(None). A driver that reports the dial's success between the two then consumed a stale Offline and published the live session as gone. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/reconnect.rs | 3 +++ rs/moq-net/src/reconnect/tests.rs | 34 +++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/rs/moq-net/src/reconnect.rs b/rs/moq-net/src/reconnect.rs index 5317cf2ade0..97fd40aeb37 100644 --- a/rs/moq-net/src/reconnect.rs +++ b/rs/moq-net/src/reconnect.rs @@ -531,6 +531,9 @@ impl + From> Reco if let Some((tokens, _)) = &self.tokens { tokens.connected(&session); } + // A predecessor that retired mid-dial queued an offline this session now contradicts. + // Its `Retired` stays, so the caller still drops the predecessor. + self.queue.retain(|action| !matches!(action, Action::Offline(_))); self.walk = Walk::default(); self.phase = Phase::Live(Box::new(Live { goaway: Some(session.draining()), diff --git a/rs/moq-net/src/reconnect/tests.rs b/rs/moq-net/src/reconnect/tests.rs index bda6264c6e3..47f3b939477 100644 --- a/rs/moq-net/src/reconnect/tests.rs +++ b/rs/moq-net/src/reconnect/tests.rs @@ -617,6 +617,40 @@ async fn the_handover_takes_the_earlier_deadline() { ); } +/// A replacement that connects after the predecessor retired mid-dial serves, so the +/// offline the retirement queued no longer holds and is never reported. +#[kio::test] +async fn a_connect_cancels_the_offline_a_retirement_queued() { + const HANDOVER: Duration = Duration::from_millis(500); + let peer = url("tcp://peer:1/"); + let mut config = Config::default(); + config.goaway.handover = HANDOVER; + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; + + kio::clock::sleep(Duration::from_secs(2)).await; + run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); + assert!(matches!(next(&mut machine).await, Action::Migrating)); + assert!(matches!(next(&mut machine).await, Action::Dial(_))); + // The dial outlasts the handover, so the predecessor retires with nothing serving. + assert!(matches!(next(&mut machine).await, Action::Retired)); + + let (client, _server) = handshake().await; + machine.connected(kio::clock::now(), client); + assert!( + matches!( + machine.poll(kio::clock::now(), &kio::Waiter::noop()), + Action::Wait(None) + ), + "a stale offline followed the connect" + ); + assert!(machine.session().is_some()); +} + /// Under [`Restart::Wait`] an empty-URI GOAWAY changes nothing while the peer keeps the /// session: no migration and no redial. Its close then reconnects like any drop. #[kio::test] From 5ff394e75c74bed6e985cd6746d180655e23789e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 09:25:14 -0800 Subject: [PATCH 07/10] test(ffi): read no clock in the catalog warmup test The reader polls inside the FFI runtime, whose clock is the shared real one, while the paused test runtime stamped the groups on its own timeline. Once the process had run 500 ms of real time, the start's reach fallback fired and the reader started without waiting, failing the merge queue under load. Write every group before the first read instead, so the start decision reads no clock; moq-mux's start-reach tests cover the timed wait. Co-Authored-By: Claude Opus 5.5 --- rs/moq-ffi/src/media.rs | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/rs/moq-ffi/src/media.rs b/rs/moq-ffi/src/media.rs index 242e2e776b0..b7f8d2e6b88 100644 --- a/rs/moq-ffi/src/media.rs +++ b/rs/moq-ffi/src/media.rs @@ -1552,8 +1552,12 @@ mod read_tests { /// A catalog rendition's warmup survives conversion, widens the raw subscription past the /// caller's max delay, keeping the min delay as asked, and starts the reader that much further - /// back: with newest-first delivery it waits for a group covering the min delay plus the warmup. - #[tokio::test(start_paused = true)] + /// back. + /// + /// Reads no clock: the reader polls on the FFI runtime, whose clock is not a paused test + /// runtime's, so a wait for the reach would measure real time. moq-mux's start-reach tests + /// cover that wait on one clock. + #[tokio::test] async fn raw_media_catalog_warmup_reaches_the_subscription() { use crate::consumer::{MoqDelay, MoqSubscription, moq_subscription_defaults}; @@ -1620,15 +1624,10 @@ mod read_tests { .unwrap(); group.finish().unwrap(); }; - // Newest first: group 4 covers the min delay alone, but not the warmup too. + // Newest first, all before the first read. Group 4 covers the min delay alone, so + // without the warmup the start would skip group 3; only the warmup's reach keeps it. write(5, 5_000_000); write(4, 4_700_000); - assert!( - kio::clock::timeout(Duration::from_millis(50), reader.next()) - .await - .is_err(), - "video={video}: the start waits for a group covering the warmup" - ); write(3, 4_400_000); track.finish().unwrap(); let mut micros = Vec::new(); From 14eb297612bf1e1ae18a6ccea2c9c74b1474b571 Mon Sep 17 00:00:00 2001 From: "pullfrog[bot]" <226033991+pullfrog[bot]@users.noreply.github.com> Date: Thu, 5 Nov 2026 18:17:08 +0000 Subject: [PATCH 08/10] test(ffi): quarantine the requested datagram flake Check failed on test::requested_datagrams_cross_a_default_connection: the single datagram it appends after consumer.info() never arrived within the 10 s wait. It passes 30 of 30 focused runs on an idle host, and the plain datagram connection test is already quarantined in quest/m0/test-flakes-2/ffi-datagram-roundtrip.md for the same one-datagram timeout under CI load. Disable it and register it with that quest, per the flaky-test rule. The quest keeps the investigation and the re-enable. --- rs/moq-ffi/src/test.rs | 1 + sh/gh/test-inventory.allow | 1 + 2 files changed, 2 insertions(+) diff --git a/rs/moq-ffi/src/test.rs b/rs/moq-ffi/src/test.rs index 29f84a9d701..d3eaea515b9 100644 --- a/rs/moq-ffi/src/test.rs +++ b/rs/moq-ffi/src/test.rs @@ -6083,6 +6083,7 @@ async fn datagrams_cross_a_default_connection() { /// An on-demand datagram-only track reaches a reader on the default lite-06 path while a /// stream handler leaves the name unanswered: the datagram handler's answer serves it. #[tokio::test] +#[ignore = "flaky under CI load; tracked in quest/m0/test-flakes-2/ffi-datagram-roundtrip.md"] async fn requested_datagrams_cross_a_default_connection() { requested_datagrams_cross(vec![], "moq-lite-06").await; } diff --git a/sh/gh/test-inventory.allow b/sh/gh/test-inventory.allow index 6f44c275b3f..ff45519790d 100644 --- a/sh/gh/test-inventory.allow +++ b/sh/gh/test-inventory.allow @@ -3,6 +3,7 @@ # orphan missing here, and fails on an entry that has a caller. # Flaky under load; the quest makes it reliable, then removes its `#[ignore]`. +ignore moq-ffi::test::requested_datagrams_cross_a_default_connection /quest/m0/test-flakes-2/ffi-datagram-roundtrip.md ignore moq-gst::source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad /quest/m0/test-flakes-2/gst-reconnect-interleave.md # Need a GPU, a desktop, or a device that no CI runner has. From fc3a87f53105961eec7fff4345854fed25d374f3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 11:05:25 -0800 Subject: [PATCH 09/10] doc(moq-net): note Reconnect needs the async feature Co-Authored-By: Claude Opus 5.5 --- doc/lib/rs/moq-net.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/lib/rs/moq-net.md b/doc/lib/rs/moq-net.md index 84bfc86d665..a75522e0094 100644 --- a/doc/lib/rs/moq-net.md +++ b/doc/lib/rs/moq-net.md @@ -27,7 +27,7 @@ What you use it for, beyond what the concept page already describes: - **Demand says whether anyone is watching.** `demand()` on a track, group, or broadcast says whether a subscriber is attached. First-party capture does not wait for it: video encodes while enabled and audio keeps raw PCM to encode when someone listens, so a late subscription's `delay.max` or a FETCH finds the recent past. Demand gates bandwidth reservations, transcode rungs, and on-demand video where a publisher opts in, which saves the encode but leaves nothing from before the first subscriber. Holding a broadcast or track consumer is not demand: a track consumer requests nothing until it queries, subscribes, or fetches, and those count until dropped. A shared fetch stays up until its last reader leaves. Demand rolls up: a frame being read keeps its group wanted, a group a subscription or fetch handed out keeps its track wanted until read to its end, and any wanted track keeps its broadcast wanted. - **Use the model without a runtime.** By default `moq-net` is its `poll_*` APIs and origin driver with caller-supplied time. The `async` feature adds their async wrappers, `time::run`, and both wire sessions. - **You drive the session, or `moq-tokio` does.** `connect` and `accept` return a session plus a driver that never reads the clock itself. `moq_net::time::run` polls it on tokio. `moq-tokio` hides that. A custom transport implements `moq_net::transport::poll`, and `moq_net::transport::memory::pair` links two sessions in one process with no sockets, for tests or in-process links. -- **Reconnect policy comes with it.** `moq_net::Reconnect` decides which of a peer's addresses to dial, how long each dial and handshake may take, when to back off and give up, and how to follow a GOAWAY, without doing I/O: you dial each `Attempt` it asks for and report back. `moq-tokio`'s `Connection` runs it, so a client on another runtime gets the same policy rather than a copy. +- **Reconnect policy comes with it.** With the `async` feature, `moq_net::Reconnect` decides which of a peer's addresses to dial, how long each dial and handshake may take, when to back off and give up, and how to follow a GOAWAY, without doing I/O: you dial each `Attempt` it asks for and report back. `moq-tokio`'s `Connection` runs it, so a client on another runtime gets the same policy rather than a copy. - **A session bounds what its peer can hold.** Each request holds a stream, so the transport's stream credit bounds subscriptions and FETCHes. `session::Limits` (via `Client::with_limits` and `Server::with_limits`) bounds the announcements per session. The default suits a relay mesh, so lower it for untrusted peers. Past it, the session closes with `TOO_MANY_REQUESTS`. Peer-declared lengths are capped before they are buffered. - **Authorization is in band.** On moq-lite 07 (`moq-lite-07-wip`, opt-in) and on moq-transport draft-18+ with the [MoQ Auth extension](/draft/moq-auth), `session.auth()` presents tokens without reconnecting and watches the grant the peer sent back. `add(token)` resolves with the token and how the peer took it: `Answer::Granted` on its AUTH stream, or `Answer::Requests` on moq-transport without AUTH, where only a request that names it carries it. A handle passed to `Client::with_auth` outlives each session, so its tokens, and the routes naming them, carry across your own reconnects. A server verifies tokens itself by taking `handshake.requests()` before `ok()`. `authorize(&grant)` narrows or widens a live session on any version: what falls outside resets with `Unauthorized` and the session stays up. A client that publishes outside its grant closes the session, naming the path. - **A graceful close waits.** `session.close().await` withdraws announcements and drains for up to one second. `moq-lite-05` and newer wait for the subscriber to receive finished tails; older Lite versions and IETF wait only for transport ACKs. `abort` ends immediately. From 99aaaf591c40d22776def364c4cfc9bf0334670c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 01:06:23 -0800 Subject: [PATCH 10/10] quest: record the requested datagram test's quarantine #7904 ignores requested_datagrams_cross_a_default_connection after it missed its datagram in this PR's Check, the plain test's symptom. The datagram quest re-enables both. Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/ffi-datagram-roundtrip.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md b/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md index 8629ae1993e..525d707ba2b 100644 --- a/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md +++ b/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md @@ -39,8 +39,15 @@ Investigation 2026-11-05 on `main` at `419b799c75`: [moq-dev/moq#7951](https://github.com/moq-dev/moq/pull/7951) is real, but was not reproduced by these tests. Do not claim it caused this timeout. - `datagrams_cross_a_default_connection` is disabled pending diagnosis, per - the repository's flaky-test rule. Its requested siblings have not failed - in this investigation and remain enabled. No retry or longer wait was added. + the repository's flaky-test rule. No retry or longer wait was added. +- `requested_datagrams_cross_a_default_connection` is quarantined too, by + [moq-dev/moq#7904](https://github.com/moq-dev/moq/pull/7904): its single + datagram missed the 10 s wait in that PR's + [Check](https://github.com/moq-dev/moq/actions/runs/42076855917/job/127419261608) + (panic at `rs/moq-ffi/src/test.rs:6111`), the same symptom as the plain + test, then passed 100 of 100 stress runs on an idle host. Re-enable it with + the plain test once this quest's cause is fixed; its other requested + siblings stay enabled. ## Waiting