diff --git a/doc/lib/rs/moq-net.md b/doc/lib/rs/moq-net.md index 90c8bc15e34..39fa0046481 100644 --- a/doc/lib/rs/moq-net.md +++ b/doc/lib/rs/moq-net.md @@ -27,6 +27,7 @@ What you use it for, beyond what the concept page already describes: - **Demand says whether anyone is watching.** `demand()` on a track, group, or broadcast says whether a subscriber is attached. First-party capture does not wait for it: video encodes while enabled and audio keeps raw PCM to encode when someone listens, so a late subscription's `delay.max` or a FETCH finds the recent past. Demand gates bandwidth reservations, transcode rungs, and on-demand video where a publisher opts in, which saves the encode but leaves nothing from before the first subscriber. Holding a broadcast or track consumer is not demand: a track consumer requests nothing until it queries, subscribes, or fetches, and those count until dropped. A shared fetch stays up until its last reader leaves. Demand rolls up: a frame being read keeps its group wanted, a group a subscription or fetch handed out keeps its track wanted until read to its end, and any wanted track keeps its broadcast wanted. - **Use the model without a runtime.** By default `moq-net` is its `poll_*` APIs and origin driver with caller-supplied time. The `async` feature adds their async wrappers, `time::run`, and both wire sessions. - **You drive the session, or `moq-tokio` does.** `connect` and `accept` return a session plus a driver that never reads the clock itself. `moq_net::time::run` polls it on tokio. `moq-tokio` hides that. A custom transport implements `moq_net::transport::poll`, and `moq_net::transport::memory::pair` links two sessions in one process with no sockets, for tests or in-process links. +- **Reconnect policy comes with it.** With the `async` feature, `moq_net::Reconnect` decides which of a peer's addresses to dial, how long each dial and handshake may take, when to back off and give up, and how to follow a GOAWAY, without doing I/O: you dial each `Attempt` it asks for and report back. `moq-tokio`'s `Connection` runs it, so a client on another runtime gets the same policy rather than a copy. - **A session bounds what its peer can hold.** Each request holds a stream, so the transport's stream credit bounds subscriptions and FETCHes. `session::Limits` (via `Client::with_limits` and `Server::with_limits`) bounds the announcements per session. The default suits a relay mesh, so lower it for untrusted peers. Past it, the session closes with `TOO_MANY_REQUESTS`. Peer-declared lengths are capped before they are buffered. - **Authorization is in band.** On moq-lite 07 (`moq-lite-07-wip`, opt-in) and on moq-transport draft-18+ with the [MoQ Auth extension](/draft/moq-auth), `session.auth()` presents tokens without reconnecting and watches the grant the peer sent back. `add(token)` resolves with the token and how the peer took it: `Answer::Granted` on its AUTH stream, or `Answer::Requests` on moq-transport without AUTH, where only a request that names it carries it. A handle passed to `Client::with_auth` outlives each session, so its tokens, and the routes naming them, carry across your own reconnects. A server verifies tokens itself by taking `handshake.requests()` before `ok()`. `authorize(&grant)` narrows or widens a live session on any version: what falls outside resets with `Unauthorized` and the session stays up. A client that publishes outside its grant closes the session, naming the path. - **A graceful close waits.** `session.close().await` withdraws announcements and drains for up to one second. `moq-lite-05` and newer wait for the subscriber to receive finished tails; older Lite versions and IETF wait only for transport ACKs. `abort` ends immediately. diff --git a/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md b/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md index 8629ae1993e..525d707ba2b 100644 --- a/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md +++ b/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md @@ -39,8 +39,15 @@ Investigation 2026-11-05 on `main` at `419b799c75`: [moq-dev/moq#7951](https://github.com/moq-dev/moq/pull/7951) is real, but was not reproduced by these tests. Do not claim it caused this timeout. - `datagrams_cross_a_default_connection` is disabled pending diagnosis, per - the repository's flaky-test rule. Its requested siblings have not failed - in this investigation and remain enabled. No retry or longer wait was added. + the repository's flaky-test rule. No retry or longer wait was added. +- `requested_datagrams_cross_a_default_connection` is quarantined too, by + [moq-dev/moq#7904](https://github.com/moq-dev/moq/pull/7904): its single + datagram missed the 10 s wait in that PR's + [Check](https://github.com/moq-dev/moq/actions/runs/42076855917/job/127419261608) + (panic at `rs/moq-ffi/src/test.rs:6111`), the same symptom as the plain + test, then passed 100 of 100 stress runs on an idle host. Re-enable it with + the plain test once this quest's cause is fixed; its other requested + siblings stay enabled. ## Waiting diff --git a/quest/m1/session-policy/README.md b/quest/m1/session-policy/README.md index 05865ae6f61..2300de94688 100644 --- a/quest/m1/session-policy/README.md +++ b/quest/m1/session-policy/README.md @@ -66,6 +66,5 @@ Research (2026-11-04, `origin/main` at e6be13d5f8): ## Required - [Accept policy in moq-net](/quest/m1/session-policy/accept.md) - one accept deadline, admission bound and window widening for every transport -- [Connect policy in moq-net](/quest/m1/session-policy/connect.md) - the connect deadline and reconnect as a poll machine - [The io_uring relay path uses moq-net's policy](/quest/m1/session-policy/uring.md) - deletes `uring.rs`'s copy - [moq-tokio becomes a thin shell](/quest/m1/session-policy/tokio-handoff.md) - sockets, TLS, accept loops and spawning, plus the transport handoffs WebRTC needs diff --git a/quest/m1/session-policy/connect.md b/quest/m1/session-policy/connect.md deleted file mode 100644 index 716a18d992e..00000000000 --- a/quest/m1/session-policy/connect.md +++ /dev/null @@ -1,32 +0,0 @@ -# [L] Connect policy in moq-net - -## Goal - -moq-net owns client connect policy as poll machines: the connect deadline over -dial plus handshake, and reconnect with backoff, GOAWAY handling and the -candidate walk. moq-net does no I/O: the caller supplies each dial attempt and -the machine decides when to start, abandon, back off or move to the next -candidate. moq-tokio's `Client`/`Connection` drive these machines instead of -implementing the policy. - -## Plan - -Part of [Session policy lives in moq-net](/quest/m1/session-policy/README.md); -see it for the decisions and research. Decided 2026-11-04 (maintainer): -reconnect moves too, not just the deadline. - -- Today's policy: the connect deadline in moq-tokio's `client.rs` (default 30 s - in `connect.rs`), and `Backoff`, `Goaway`, `run`, `attempt_timeout`, - `settles_walk` and `refreshed` in `connection.rs`. All of it runs on - `kio::clock` and `select!`; only the dialing (DNS, Happy Eyeballs, sockets, - TLS) is tokio-specific and stays in moq-tokio. -- Keep the current behaviour and defaults; port the existing - `connection.rs` tests to the moq-net machines on kio's simulator, so the same - timings are asserted without tokio. -- This also gives a future io_uring client - ([uring-cluster-dial](/quest/m5/uring-cluster-dial.md)) the policy to reuse - rather than copy. - -Public API: breaking in moq-net's `Client` and new reconnect types; moq-tokio's -`Connection` API keeps its shape where possible, with MIGRATE notes otherwise. -Wire: none. diff --git a/quest/m1/session-policy/tokio-handoff.md b/quest/m1/session-policy/tokio-handoff.md index 3a9310bb17a..3adfce370bc 100644 --- a/quest/m1/session-policy/tokio-handoff.md +++ b/quest/m1/session-policy/tokio-handoff.md @@ -38,4 +38,3 @@ it re-exported, with MIGRATE notes. Wire: none. ## Required - [Accept policy in moq-net](/quest/m1/session-policy/accept.md) - the accept policy the handoff runs -- [Connect policy in moq-net](/quest/m1/session-policy/connect.md) - the connect policy the handoff runs diff --git a/quest/m2/recut-held.md b/quest/m2/recut-held.md index 59b0ee3575d..d3408490b32 100644 --- a/quest/m2/recut-held.md +++ b/quest/m2/recut-held.md @@ -32,6 +32,5 @@ padding probes in [lite07-finalize](/quest/m2/lite07-finalize.md). - [IETF datagram tracks](/quest/m1/datagram-tracks/ietf.md) - claim held datagram-tracks - [rind CPU error](/quest/m1/rind/cpu-error.md) - claim held rind -- [Connect policy](/quest/m1/session-policy/connect.md) - claim held session-policy - [Release on leave](/quest/m1/stream-credit/release-on-leave.md) - claim held stream-credit - [Probe padding](/quest/m2/probe/padding.md) - claim held probe diff --git a/quest/m5/uring-cluster-dial.md b/quest/m5/uring-cluster-dial.md index 8f4fc4b0d6f..36f99e2f6a6 100644 --- a/quest/m5/uring-cluster-dial.md +++ b/quest/m5/uring-cluster-dial.md @@ -13,8 +13,9 @@ lands only if the measurement shows a win. serves inbound traffic, so a ring relay forwards its cluster media through tokio. A port needs the ring client to grow what `moq-tokio` provides: DNS and address racing, reconnect, and fingerprint pinning (the raw ring -client takes one `SocketAddr`). Share that policy with `moq-tokio` rather -than duplicate it. +client takes one `SocketAddr`). The reconnect half already lives in +moq-net as `moq_net::Reconnect`, which a ring client drives with its own +dials; share the rest with `moq-tokio` rather than duplicate it. Measure first with a prototype dialing a fixed address from a worker: CPU per Gbps and latency on the relay-to-relay fanout shape, ring against diff --git a/rs/moq-ffi/src/media.rs b/rs/moq-ffi/src/media.rs index a32399c58c7..7753c9b1b3e 100644 --- a/rs/moq-ffi/src/media.rs +++ b/rs/moq-ffi/src/media.rs @@ -1548,8 +1548,12 @@ mod read_tests { /// A catalog rendition's warmup survives conversion, widens the raw subscription past the /// caller's max delay, keeping the min delay as asked, and starts the reader that much further - /// back: with newest-first delivery it waits for a group covering the min delay plus the warmup. - #[tokio::test(start_paused = true)] + /// back. + /// + /// Reads no clock: the reader polls on the FFI runtime, whose clock is not a paused test + /// runtime's, so a wait for the reach would measure real time. moq-mux's start-reach tests + /// cover that wait on one clock. + #[tokio::test] async fn raw_media_catalog_warmup_reaches_the_subscription() { use crate::consumer::{MoqDelay, MoqSubscription, moq_subscription_defaults}; @@ -1616,15 +1620,10 @@ mod read_tests { .unwrap(); group.finish().unwrap(); }; - // Newest first: group 4 covers the min delay alone, but not the warmup too. + // Newest first, all before the first read. Group 4 covers the min delay alone, so + // without the warmup the start would skip group 3; only the warmup's reach keeps it. write(5, 5_000_000); write(4, 4_700_000); - assert!( - kio::clock::timeout(Duration::from_millis(50), reader.next()) - .await - .is_err(), - "video={video}: the start waits for a group covering the warmup" - ); write(3, 4_400_000); track.finish().unwrap(); let mut micros = Vec::new(); diff --git a/rs/moq-ffi/src/test.rs b/rs/moq-ffi/src/test.rs index 7f5c2e452cd..f4f1fde0c14 100644 --- a/rs/moq-ffi/src/test.rs +++ b/rs/moq-ffi/src/test.rs @@ -6029,6 +6029,7 @@ async fn datagrams_cross_a_default_connection() { /// An on-demand datagram-only track reaches a reader on the default lite-06 path while a /// stream handler leaves the name unanswered: the datagram handler's answer serves it. #[tokio::test] +#[ignore = "flaky under CI load; tracked in quest/m0/test-flakes-2/ffi-datagram-roundtrip.md"] async fn requested_datagrams_cross_a_default_connection() { requested_datagrams_cross(vec![], "moq-lite-06").await; } diff --git a/rs/moq-net/src/lib.rs b/rs/moq-net/src/lib.rs index 1e04e26b1ad..fcffa24aca1 100644 --- a/rs/moq-net/src/lib.rs +++ b/rs/moq-net/src/lib.rs @@ -132,6 +132,8 @@ mod test_interop; mod util; mod version; +#[cfg(feature = "async")] +pub mod reconnect; #[cfg(feature = "async")] pub mod server; #[cfg(feature = "async")] @@ -150,6 +152,8 @@ pub use instance::{Instance, InvalidInstance}; pub use model::*; pub use path::{InvalidPattern, Path, Pattern, Patterns}; #[cfg(feature = "async")] +pub use reconnect::Reconnect; +#[cfg(feature = "async")] pub use server::Server; #[cfg(feature = "async")] pub use session::Session; diff --git a/rs/moq-net/src/reconnect.rs b/rs/moq-net/src/reconnect.rs new file mode 100644 index 00000000000..97fd40aeb37 --- /dev/null +++ b/rs/moq-net/src/reconnect.rs @@ -0,0 +1,962 @@ +//! Client connect policy: which of a peer's addresses to dial, how long each dial may +//! take, when to back off and when to give up, and how to follow a GOAWAY. +//! +//! [`Reconnect`] is a poll machine that does no I/O. The caller dials each [`Attempt`] +//! it asks for, reports the result with [`Reconnect::connected`] or +//! [`Reconnect::failed`], and otherwise polls it with the current time. The machine +//! watches each live session for its close and GOAWAY itself, keeps a drained +//! predecessor serving through its handover, and drops neither on the caller's behalf +//! until it reports [`Action::Done`]. + +mod redirect; +#[cfg(test)] +mod tests; + +pub use redirect::Redirect; + +use std::{collections::VecDeque, task::Poll, time::Duration}; + +use rand::RngExt; +use url::Url; + +use crate::{Error, Session, SessionError, auth, goaway, time::Instant}; +use redirect::{Redirects, Shown, endpoint}; + +/// How long one address gets before the walk moves to the peer's next one. +/// +/// A black-holed address answers nothing rather than refusing, so without a bound the +/// transport's idle timeout would decide how long the remaining addresses wait. Only an +/// attempt with a later address to fall back on is worth bounding: cutting the last one +/// short just turns a slow connect into a failed one, on every reconnect cycle. +const ATTEMPT: Duration = Duration::from_secs(5); + +/// When to redial after a failure, and when to give up. +/// +/// `initial` and `multiplier` set the wait before each retry, `max` caps it, and +/// `healthy` decides which sessions count as failed attempts. The delays carry jitter, +/// so a fleet knocked offline together does not reconnect in lockstep. `timeout` bounds +/// every failure; a settled rejection, malformed protocol input, or invalid local +/// configuration stops retrying. A GOAWAY naming no new URI is no failure: a restarting +/// peer sends one to each session until it goes, and each of those serves, so it +/// restarts the timeout instead. +/// +/// A GOAWAY off a healthy session migrates at once, unless it sends the connection back +/// to an endpoint that already redirected it elsewhere and the session lived less than +/// `max`. Two peers redirecting to each other thus wait out this backoff and count +/// against the timeout. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct Backoff { + /// Delay before the first redial. Later delays grow from it, so zero turns retry + /// waiting off entirely; an immediate close still counts as a failed attempt and + /// spends the timeout. Defaults to 1 s. + pub initial: Duration, + /// Growth of the delay after each failure. 0 or 1 keeps it constant. Defaults to 2. + pub multiplier: u32, + /// Cap on the delay, which caps `initial` too. Zero leaves it uncapped. Defaults to 5 s. + pub max: Duration, + /// How long a session must stay up to count as healthy. Must be nonzero. Defaults to 1 s. + /// + /// A healthy session's end resets the delay and the timeout. One that ends sooner is a + /// failed attempt, so a peer that accepts and then drops every session still escalates + /// the delay and exhausts the timeout. + pub healthy: Duration, + /// How long to keep retrying before giving up. Zero retries forever. Defaults to 10 s. + /// + /// No new walk of the peer's addresses starts once it has passed, but one under way + /// finishes: each address but the last gets up to 5 s, and the last runs to + /// [`Config::timeout`], so a handshake slower than this still lands. Giving up can + /// therefore take up to one whole walk longer. Resets after a healthy session outside a + /// redirect loop. + pub timeout: Duration, +} + +impl Default for Backoff { + fn default() -> Self { + Self { + initial: Duration::from_secs(1), + multiplier: 2, + max: Duration::from_secs(5), + healthy: Duration::from_secs(1), + timeout: Duration::from_secs(10), + } + } +} + +/// How a connection reacts to a peer's GOAWAY. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct Goaway { + /// What to do with the URI the GOAWAY names. Defaults to [`Redirect::SameHost`]. + pub redirect: Redirect, + /// The longest the old session keeps serving after a GOAWAY. Defaults to 10 s. + /// + /// It closes sooner, once its replacement is up and it carries no subscription or + /// fetch. A GOAWAY naming a shorter deadline wins, since the peer force-closes at its + /// own deadline regardless, but a longer one does not extend it. + pub handover: Duration, +} + +impl Default for Goaway { + fn default() -> Self { + Self { + redirect: Redirect::default(), + handover: Duration::from_secs(10), + } + } +} + +impl Goaway { + /// The handover window for a GOAWAY naming `deadline`: whichever is sooner. + fn handover(&self, deadline: Option) -> Duration { + deadline.map_or(self.handover, |deadline| deadline.min(self.handover)) + } +} + +/// What to do when a peer's GOAWAY names no URI, meaning it is restarting where we reached it. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +#[non_exhaustive] +pub enum Restart { + /// Redial the same addresses at once while the drained session keeps serving. Right for + /// an address that can lead to another node, such as a fleet name a fresh resolve moves + /// off the drained one. The default. + #[default] + Redial, + /// Keep serving on the drained session until the peer closes it, then reconnect as after + /// any drop. Right for an address that names one node, where a redial only reaches the + /// draining process again. The peer must close the session itself, as a relay does at + /// its drain deadline. + Wait, +} + +/// The whole connect policy for one peer. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct Config { + /// When to redial, and when to give up. + pub backoff: Backoff, + /// How to follow a peer's GOAWAY. + pub goaway: Goaway, + /// What a GOAWAY naming no URI does. + pub restart: Restart, + /// The deadline for one dial and its handshake. Zero waits forever. Defaults to 30 s. + pub timeout: Duration, + /// End when the first session ends instead of redialing. Defaults to false. + /// + /// A GOAWAY then ends it cleanly: with no replacement to dial, leaving is the answer. + pub once: bool, + /// Whether the dial pins a certificate fingerprint, which verifies only the host it + /// was configured for, so a GOAWAY may not move an encrypted dial to another host. + pub pinned: bool, +} + +impl Default for Config { + fn default() -> Self { + Self { + backoff: Backoff::default(), + goaway: Goaway::default(), + restart: Restart::default(), + timeout: Duration::from_secs(30), + once: false, + pinned: false, + } + } +} + +/// One of a peer's addresses, as the caller dials it. +pub trait Target: Clone + From { + /// The URL a dial reaches, which a GOAWAY redirect is judged against. + fn url(&self) -> &Url; + + /// Whether the dial skips resolving the URL for fixed socket addresses, which a + /// GOAWAY redirect cannot carry over. + fn fixed(&self) -> bool; +} + +impl Target for Url { + fn url(&self) -> &Url { + self + } + + fn fixed(&self) -> bool { + false + } +} + +/// What a failed dial means for the rest of the policy, judged by the caller. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub enum Verdict { + /// This address failed; try the next, then back off and retry. + Retry, + /// This address can never work, such as an unsupported scheme. Give up unless + /// another address fails transiently. + Unusable, + /// The peer's answer for every address, such as an HTTP 404: stop the walk, then + /// back off and retry. + Settled, + /// The peer refused the credential: give up unless the tokens changed since the dial. + Refused, + /// Nothing a retry can repair, such as malformed protocol input: give up. + Fatal, +} + +impl From<&Error> for Verdict { + /// Judge an error from the MoQ handshake or a session's close. + fn from(err: &Error) -> Self { + match SessionError::from(err) { + SessionError::Unauthorized | SessionError::Expired | SessionError::MalformedAuthToken => Self::Refused, + SessionError::ProtocolViolation | SessionError::KeyValueFormatting | SessionError::InvalidRequestId => { + Self::Fatal + } + _ => Self::Retry, + } + } +} + +/// A failure the policy itself decides on. +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] +#[non_exhaustive] +pub enum Failure { + /// An address with another to fall back on did not connect in time. + #[error("timed out connecting to {0}")] + Attempt(String), + /// A dial and its handshake outlasted [`Config::timeout`]. + #[error("connect timed out after {0:?}")] + Timeout(Duration), + /// The retry window ran out; carries the last failure's message. + #[error("reconnect timed out after {after:?}{}", last.as_ref().map(|last| format!(": {last}")).unwrap_or_default())] + Exhausted { + /// The retry window. + after: Duration, + /// Why the last attempt failed, if one did. + last: Option, + }, + /// Peers redirected the connection around in a loop. + #[error("redirect loop")] + Loop, + /// The peer redirected a session elsewhere before it was healthy. + #[error("peer redirected immediately")] + Redirected, + /// The redirect a GOAWAY named breaks the [`Redirect`] policy; carries why. + #[error("refused GOAWAY redirect: {0}")] + Refused(&'static str), + /// A GOAWAY named a redirect for a target dialing fixed socket addresses. + #[error("a GOAWAY redirected a target dialing fixed socket addresses")] + Fixed, +} + +/// Invalid input to [`Reconnect::new`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +#[non_exhaustive] +pub enum ConfigError { + /// There is no address to dial. + #[error("no target to dial")] + Empty, + /// [`Backoff::healthy`] is zero, which would count every session as healthy and reset + /// the timeout forever under a peer that drops each one on sight. + #[error("the healthy threshold must be nonzero")] + Healthy, +} + +/// One dial the caller runs for [`Reconnect`]. +#[derive(Clone, Debug)] +pub struct Attempt { + target: A, + url: Url, + token: Option, +} + +impl Attempt { + /// The address to dial. + pub fn target(&self) -> &A { + &self.target + } + + /// The URL to dial, carrying the connection credential in its query when an offered + /// version has no SETUP token. + pub fn url(&self) -> &Url { + &self.url + } + + /// The connection credential for [`Client::with_token`](crate::Client::with_token), if any. + pub fn token(&self) -> Option<&crate::setup::Token> { + self.token.as_ref() + } +} + +/// What [`Reconnect::poll`] asks of the caller. +/// +/// Exhaustive on purpose: a new action is one every driver must handle. +#[derive(Debug)] +pub enum Action { + /// Dial this attempt and report it with [`Reconnect::connected`] or + /// [`Reconnect::failed`], polling meanwhile. Building each session on the tokens' + /// handle, from [`Reconnect::with_tokens`], is the caller's. + Dial(Attempt), + /// Drop the dial in flight: it ran out its deadline, which already counted as its failure. + Abandon, + /// The live session got a GOAWAY and now drains as the [`predecessor`](Reconnect::predecessor) + /// while a replacement is dialed. + Migrating, + /// The predecessor closed or overstayed its handover. + Retired, + /// Nothing serves anymore: the live session ended with this error, or the predecessor + /// retired between sessions. A redial follows. + Offline(Option), + /// Why the policy is waiting to redial: a walk failed or a session ended too soon. + Retrying(E), + /// Poll again by this instant, or on a wake. + Wait(Option), + /// The policy gave up with an error, or ended cleanly. Every later poll waits forever. + Done(Result<(), E>), +} + +/// Client connect policy as a poll machine; see the [module](self) docs. +/// +/// `A` is the caller's address type and `E` its error type, which takes the +/// [`Error`] a session closes with and the [`Failure`]s the policy decides on. +pub struct Reconnect { + config: Config, + targets: Vec, + tokens: Option<(auth::Tokens, crate::Versions)>, + retry: Retry, + redirects: Redirects, + phase: Phase, + /// The failures of the walk under way. + walk: Walk, + /// An old session kept serving after a GOAWAY so its in-flight groups finish. + predecessor: Option, + queue: VecDeque>, +} + +enum Phase { + /// Check the retry window, then walk the targets. + Start, + /// Ask the caller to dial the target at `index`. + Next { + index: usize, + }, + Dialing(Dialing), + Live(Box), + Backoff { + until: Instant, + /// After a restart's wait, the retry window counts from its end. + restart: bool, + }, + /// The policy ended; every later poll waits forever. + Done, +} + +struct Dialing { + index: usize, + deadline: Option<(Instant, Failure)>, + presented: Option, +} + +struct Live { + session: Session, + since: Instant, + index: usize, + /// Dropped once a restart is waited out: it is a latch, so it would keep firing. + goaway: Option, + presented: Option, +} + +struct Walk { + /// The latest failure a retry may repair. + transient: Option, + /// The latest failure of any kind. + last: Option, +} + +impl Default for Walk { + fn default() -> Self { + Self { + transient: None, + last: None, + } + } +} + +/// Why a live session stopped being the live one. +enum Ended { + Closed(Error), + /// The peer sent a GOAWAY; the session is still up and serving. + Goaway(goaway::Goaway), +} + +impl + From> Reconnect { + /// Walk `targets` in order under `config`, starting at `now`. + pub fn new(config: Config, targets: Vec, now: Instant) -> Result { + if targets.is_empty() { + return Err(ConfigError::Empty); + } + if config.backoff.healthy.is_zero() { + return Err(ConfigError::Healthy); + } + Ok(Self { + retry: Retry::new(Pacing::new(&config.backoff).initial, now), + config, + targets, + tokens: None, + redirects: Redirects::default(), + phase: Phase::Start, + walk: Walk::default(), + predecessor: None, + queue: VecDeque::new(), + }) + } + + /// Present `tokens` on every dial offering `versions`. + /// + /// A refused credential then earns another dial once the tokens change, and an ended + /// tokens handle ends the policy at its next walk. + pub fn with_tokens(mut self, tokens: auth::Tokens, versions: crate::Versions) -> Self { + self.tokens = Some((tokens, versions)); + self + } + + /// The live session, if one is serving and has not received a GOAWAY. + pub fn session(&self) -> Option<&Session> { + match &self.phase { + Phase::Live(live) => Some(&live.session), + _ => None, + } + } + + /// The addresses the next walk dials, in order: the configured ones, or the one a + /// GOAWAY redirected to. + pub fn targets(&self) -> &[A] { + &self.targets + } + + /// The session draining after a GOAWAY, and the deadline its handover ends by. + pub fn predecessor(&self) -> Option<(&Session, Instant)> { + self.predecessor + .as_ref() + .map(|predecessor| (&predecessor.session, predecessor.deadline)) + } + + /// Advance to `now`, returning what the caller must do next. + pub fn poll(&mut self, now: Instant, waiter: &kio::Waiter) -> Action { + loop { + if let Some(action) = self.queue.pop_front() { + return action; + } + + let live = matches!(self.phase, Phase::Live(_)); + if let Some(predecessor) = self.predecessor.as_mut() { + // The replacement serves, so a predecessor carrying nothing closes now. + if live { + predecessor.poll_idle(waiter); + } + if predecessor.poll(now, waiter) { + self.predecessor = None; + self.queue.push_back(Action::Retired); + // Nothing is serving once it retires between sessions, so a caller must + // stop handing out a session that is already closed. + if !live { + self.queue.push_back(Action::Offline(None)); + } + continue; + } + } + let handover = self.predecessor.as_ref().map(|predecessor| predecessor.deadline); + + match &mut self.phase { + Phase::Start => self.start(now), + Phase::Next { index } => { + let index = *index; + return self.dial(now, index); + } + Phase::Dialing(dialing) => match &dialing.deadline { + Some((at, failure)) if now >= *at => { + let failure = failure.clone(); + self.fail(now, failure.into(), Verdict::Retry); + return Action::Abandon; + } + deadline => return Action::Wait(earliest(deadline.as_ref().map(|(at, _)| *at), handover)), + }, + Phase::Live(live) => { + let mut ended = None; + // Checked before the close: a GOAWAY means the session is still up, and + // migrating from it is not the same as reconnecting after it died. + if let Some(Poll::Ready(Ok(msg))) = live.goaway.as_ref().map(|goaway| goaway.poll(waiter)) { + if self.config.restart == Restart::Wait && msg.uri().is_empty() { + tracing::info!("peer is restarting; serving until it closes the session"); + live.goaway = None; + } else { + ended = Some(Ended::Goaway(msg)); + } + } + if ended.is_none() + && let Poll::Ready(err) = live.session.poll_closed(waiter) + { + ended = Some(Ended::Closed(err)); + } + match ended { + Some(ended) => self.end(now, ended), + None => return Action::Wait(handover), + } + } + Phase::Backoff { until, restart } => { + if now < *until { + return Action::Wait(earliest(Some(*until), handover)); + } + let restart = *restart; + self.retry.delay = Pacing::new(&self.config.backoff).next(self.retry.delay); + if restart { + // After the wait, so the window counts only the time after the peer + // stops answering. + self.retry.start = now; + } + self.phase = Phase::Start; + } + Phase::Done => return Action::Wait(None), + } + } + } + + /// Report that the dial asked for connected `session`, at `now`. + /// + /// Ignored unless a dial is in flight, as after [`Action::Abandon`]. + pub fn connected(&mut self, now: Instant, session: Session) { + if !matches!(self.phase, Phase::Dialing(_)) { + return; + } + let Phase::Dialing(dialing) = std::mem::replace(&mut self.phase, Phase::Start) else { + unreachable!("checked above"); + }; + if let Some((tokens, _)) = &self.tokens { + tokens.connected(&session); + } + // A predecessor that retired mid-dial queued an offline this session now contradicts. + // Its `Retired` stays, so the caller still drops the predecessor. + self.queue.retain(|action| !matches!(action, Action::Offline(_))); + self.walk = Walk::default(); + self.phase = Phase::Live(Box::new(Live { + goaway: Some(session.draining()), + session, + since: now, + index: dialing.index, + presented: dialing.presented, + })); + } + + /// Report that the dial asked for failed with `err`, judged as `verdict`. + /// + /// Ignored unless a dial is in flight, as after [`Action::Abandon`]. + pub fn failed(&mut self, now: Instant, err: E, verdict: Verdict) { + if matches!(self.phase, Phase::Dialing(_)) { + self.fail(now, err, verdict); + } + } + + /// Start a walk, unless the retry window is spent or the tokens ended. + fn start(&mut self, now: Instant) { + let timeout = self.config.backoff.timeout; + if !timeout.is_zero() && now.saturating_duration_since(self.retry.start) >= timeout { + return self.done(Err(self.exhausted())); + } + // An ended auth handle refuses every later session, so no dial can recover. A fresh + // watch only reports an error once its handle has ended. + if let Some((tokens, _)) = &self.tokens + && let Poll::Ready(Err(err)) = tokens.handle().grant().poll_changed(&kio::Waiter::noop()) + { + return self.done(Err(err.into())); + } + self.phase = Phase::Next { index: 0 }; + } + + fn dial(&mut self, now: Instant, index: usize) -> Action { + let target = self.targets[index].clone(); + tracing::info!(peer = %Shown(target.url()), "connecting"); + + // Read before the dial starts its session's driver, so a request the session sends + // early waits on the held tokens' answers rather than being refused outside the + // connection credential's grant. + let presented = self + .tokens + .as_ref() + .map(|(tokens, versions)| tokens.dial(target.url(), versions)); + let (url, token) = match &presented { + Some(presented) => (presented.url.clone(), presented.token.clone()), + None => (target.url().clone(), None), + }; + + // The connect deadline covers the dial and the handshake, for every transport: it is + // the only bound some of them have. The fixed bound applies only with a fallback. + let connect = Some(self.config.timeout).filter(|timeout| !timeout.is_zero()); + let attempt = (index + 1 < self.targets.len()).then_some(ATTEMPT); + let deadline = match (attempt, connect) { + (Some(attempt), Some(connect)) if connect <= attempt => Some((now + connect, Failure::Timeout(connect))), + (Some(attempt), _) => Some((now + attempt, Failure::Attempt(Shown(target.url()).to_string()))), + (None, Some(connect)) => Some((now + connect, Failure::Timeout(connect))), + (None, None) => None, + }; + + self.phase = Phase::Dialing(Dialing { + index, + deadline, + presented, + }); + Action::Dial(Attempt { target, url, token }) + } + + /// The dial in flight failed: move on to the next target, or end the walk. + fn fail(&mut self, now: Instant, err: E, verdict: Verdict) { + let Phase::Dialing(dialing) = std::mem::replace(&mut self.phase, Phase::Start) else { + unreachable!("checked by the caller"); + }; + match verdict { + // The peer's answer, not this address's, so unless it invites another attempt it + // settles the whole walk. Carrying on would offer the same rejected credentials at + // the peer's other addresses, and a later transport failure would overwrite the + // real reason. + Verdict::Settled | Verdict::Refused | Verdict::Fatal => { + return self.walk_failed(now, err, verdict, dialing.presented.as_ref()); + } + Verdict::Retry => self.walk.transient = Some(err.clone()), + Verdict::Unusable => {} + } + tracing::debug!(peer = %Shown(self.targets[dialing.index].url()), %err, "address unreachable"); + self.walk.last = Some(err); + + if dialing.index + 1 < self.targets.len() { + self.phase = Phase::Next { + index: dialing.index + 1, + }; + return; + } + + // One unusable target must not turn another target's transient failure terminal. + let walk = std::mem::take(&mut self.walk); + match walk.transient { + Some(err) => self.walk_failed(now, err, Verdict::Retry, None), + None => self.walk_failed(now, walk.last.expect("a failed walk"), Verdict::Unusable, None), + } + } + + fn walk_failed(&mut self, now: Instant, err: E, verdict: Verdict, presented: Option<&auth::Dial>) { + self.walk = Walk::default(); + let terminal = match verdict { + Verdict::Fatal | Verdict::Unusable => true, + Verdict::Refused => !self.changed(presented), + Verdict::Retry | Verdict::Settled => false, + }; + if terminal || self.config.once { + return self.done(Err(err)); + } + self.queue.push_back(Action::Retrying(err.clone())); + self.retry.last = Some(err); + // No URL here: with several targets there isn't one to name, and each attempt + // already logged the address it tried. + self.back_off(now, self.config.backoff.timeout, false, "reconnecting after backoff"); + } + + /// Score how a live session ended and decide what follows. + fn end(&mut self, now: Instant, ended: Ended) { + let Phase::Live(live) = std::mem::replace(&mut self.phase, Phase::Start) else { + unreachable!("only a live session ends"); + }; + let tenure = now.saturating_duration_since(live.since); + let url = self.targets[live.index].url().clone(); + let pacing = Pacing::new(&self.config.backoff); + + // The connected target owns the policy, including when connecting once. A refused + // redirect is terminal: the peer is leaving and named somewhere we won't go, so + // redialing the old address or a fallback would ignore it. + let assigned = match &ended { + Ended::Goaway(msg) => match self.config.goaway.redirect.target(msg.uri(), &url, self.config.pinned) { + Ok(assigned) => assigned, + Err(failure) => return self.done(Err(failure.into())), + }, + Ended::Closed(_) => None, + }; + if assigned.is_some() && self.targets[live.index].fixed() { + return self.done(Err(Failure::Fixed.into())); + } + + // Connecting once leaves rather than migrating: there is no replacement to dial, and + // a GOAWAY naming no deadline never force-closes, so the peer stops accepting + // requests and waits for us. Ignoring it is how both sides end up waiting on each + // other forever. + if self.config.once && matches!(ended, Ended::Goaway(_)) { + return self.done(Ok(())); + } + + // Two peers that each hold a session past the healthy bar can still bounce us between + // them forever. Returning to an endpoint that already sent us elsewhere in this run of + // migrations skips the healthy-session reset, so the backoff escalates and the retry + // window still applies. A one-way rolling restart never revisits one, and a GOAWAY + // back to the same place is a restart, not a move, so it never counts. A session that + // outlived the capped delay already redials no faster than the backoff would allow, so + // it starts a new run: rolling restarts days apart never add up to a loop. + if matches!(ended, Ended::Closed(_)) || tenure >= pacing.settled() { + self.redirects.reset(); + } + // Only an accepted redirect is a move: an empty or ignored one reconnects in place. + let looped = assigned.as_ref().is_some_and(|target| { + let from = endpoint(&url); + let to = endpoint(target); + to != from && self.redirects.depart(from, &to) + }); + + // A session that stayed up past the healthy bar is healthy; one that ended sooner, or + // carried on a redirect loop, counts as a failed attempt. + let healthy = !looped && self.retry.settle(now, tenure, &pacing); + + let msg = match ended { + Ended::Goaway(msg) => msg, + Ended::Closed(err) => return self.closed(now, err, healthy, live.presented.as_ref(), &url), + }; + + // An accepted redirect is an assignment: keep dialing it from here on, and only it. + // The peer named exactly one place to go, which retires whatever other addresses got + // us to this session. + let url = match assigned { + Some(target) => { + self.targets = vec![A::from(target.clone())]; + target + } + None => url, + }; + + // Hand over gracefully however the backoff scores this session. The old one keeps + // serving until it closes or overstays, so its routes stay attached and live tracks + // splice onto the replacement at a group boundary. + tracing::info!(peer = %Shown(&url), "upstream GOAWAY; migrating"); + if let Some(old) = self.predecessor.take() { + old.retire(); + } + self.predecessor = Some(Predecessor { + deadline: now + self.config.goaway.handover(msg.timeout()), + session: live.session, + closing: false, + }); + self.queue.push_back(Action::Migrating); + + // No backoff: a handover off a healthy session is not a failure. + if healthy { + return; + } + + // Drained almost immediately, or looped back. The backoff still escalates, so the + // redials cannot hammer the peer, and the old session serves across the wait. + // + // An empty URI is "reconnect to me": the peer is restarting and drains each new + // session until it goes, which is no failure, since every one serves. A named URI is + // a redirect, scored as a failed attempt so two peers bouncing us between them + // eventually give up. + let restarting = msg.uri().is_empty(); + if !restarting { + let err: E = if looped { Failure::Loop } else { Failure::Redirected }.into(); + self.queue.push_back(Action::Retrying(err.clone())); + self.retry.last = Some(err); + } + // A restart's pacing is not capped by the window: a delay longer than the window + // would otherwise spend it all and end the loop without a redial. + let window = if restarting { + Duration::ZERO + } else { + self.config.backoff.timeout + }; + let reason = if looped { + "peer redirected back in a loop; retrying after backoff" + } else { + "peer drained the session immediately; retrying after backoff" + }; + self.back_off(now, window, restarting, reason); + } + + /// The live session closed with `err`. + fn closed(&mut self, now: Instant, err: Error, healthy: bool, presented: Option<&auth::Dial>, url: &Url) { + // Refusals and malformed protocol input are settled failures, however long the + // session lived, unless the tokens changed since this dial read them. + let terminal = match Verdict::from(&err) { + Verdict::Fatal => true, + Verdict::Refused => !self.changed(presented), + _ => false, + }; + let err = E::from(err); + // Connecting once, the session ending ends the policy with its close reason. + if terminal || self.config.once { + return self.done(Err(err)); + } + + self.queue.push_back(Action::Offline(Some(err.clone()))); + if healthy { + // The window was reset, so a one-off drop reconnects promptly. + tracing::warn!(peer = %Shown(url), "session closed, reconnecting"); + } else { + // Connected then dropped almost immediately, such as a server that accepts then + // resets. Keep the reason so the give-up reports a real cause, and back off so + // repeated flaps escalate instead of spinning. + tracing::warn!(peer = %Shown(url), %err, "session severed immediately, retrying"); + self.retry.last = Some(err); + } + self.back_off(now, self.config.backoff.timeout, false, "reconnecting after backoff"); + } + + /// Wait out the next jittered delay, capped by what remains of `window`. + fn back_off(&mut self, now: Instant, window: Duration, restart: bool, reason: &'static str) { + let Some(wait) = self.retry.wait(now, window) else { + return self.done(Err(self.exhausted())); + }; + tracing::warn!(?wait, "{reason}"); + self.phase = Phase::Backoff { + until: now + wait, + restart, + }; + } + + /// Whether the tokens changed since `presented` read them, so a refusal earns another dial. + fn changed(&self, presented: Option<&auth::Dial>) -> bool { + match (&self.tokens, presented) { + (Some((tokens, _)), Some(presented)) => tokens.changed(presented), + _ => false, + } + } + + fn exhausted(&self) -> E { + Failure::Exhausted { + after: self.config.backoff.timeout, + last: self.retry.last.as_ref().map(ToString::to_string), + } + .into() + } + + fn done(&mut self, result: Result<(), E>) { + self.predecessor = None; + self.phase = Phase::Done; + self.queue.push_back(Action::Done(result)); + } +} + +fn earliest(a: Option, b: Option) -> Option { + match (a, b) { + (Some(a), Some(b)) => Some(a.min(b)), + (a, b) => a.or(b), + } +} + +/// The retry pacing resolved from [`Backoff`]. +struct Pacing { + /// The first delay, already under the cap. + initial: Duration, + /// The delay cap, or `None` for uncapped. + max: Option, + /// At least 1: a multiplier of zero means no growth, not a delay that shrinks to nothing. + multiplier: u32, + /// The bar a session must clear to count as healthy. + healthy: Duration, +} + +impl Pacing { + fn new(backoff: &Backoff) -> Self { + let max = Some(backoff.max).filter(|max| !max.is_zero()); + Self { + initial: max.map_or(backoff.initial, |max| backoff.initial.min(max)), + max, + multiplier: backoff.multiplier.max(1), + healthy: backoff.healthy, + } + } + + /// The delay for the attempt after one that waited `delay`. + fn next(&self, delay: Duration) -> Duration { + let next = delay.saturating_mul(self.multiplier); + self.max.map_or(next, |max| next.min(max)) + } + + /// How long a session must last to start a new run of redirects. + /// + /// One that outlived the capped delay already redials no faster than the backoff would + /// allow. Uncapped, the healthy bar is the only rate the config names. + fn settled(&self) -> Duration { + self.max.unwrap_or(self.healthy) + } +} + +/// The backoff window across a run of failed attempts. +struct Retry { + /// The wait before the next attempt, before jitter. + delay: Duration, + /// When this run of failures began; the give-up timeout counts from here. + start: Instant, + /// The latest failure, so giving up reports a real cause. + last: Option, +} + +impl Retry { + fn new(initial: Duration, now: Instant) -> Self { + Self { + delay: initial, + start: now, + last: None, + } + } + + /// Score how long a session served: past the healthy bar starts a fresh retry window. + fn settle(&mut self, now: Instant, tenure: Duration, pacing: &Pacing) -> bool { + let healthy = tenure >= pacing.healthy; + if healthy { + *self = Self::new(pacing.initial, now); + } + healthy + } + + /// Jitter the next delay and cap it at what remains of `window`. `None` means the + /// window is spent; a zero window never is. + fn wait(&self, now: Instant, window: Duration) -> Option { + let wait = self.delay.mul_f64(0.5 + kio::clock::rng().random::() / 2.0); + if window.is_zero() { + return Some(wait); + } + Some(wait.min(window.checked_sub(now.saturating_duration_since(self.start))?)) + } +} + +/// An old session kept serving after a GOAWAY so its in-flight groups finish. +struct Predecessor { + session: Session, + /// When its handover ends. + deadline: Instant, + /// Whether [`Self::poll_idle`] started the graceful close. + closing: bool, +} + +impl Predecessor { + /// Close the old session now, whatever remains of its window. + fn retire(self) { + self.session.abort(Error::GoawayTimeout); + } + + /// Close the old session gracefully once it carries no subscription or fetch. + /// + /// Only while a replacement serves: until then everything new still lands here. The + /// handover window still bounds the close. + fn poll_idle(&mut self, waiter: &kio::Waiter) { + if self.closing || self.session.poll_idle(waiter).is_pending() { + return; + } + tracing::debug!("old session carries nothing after GOAWAY; closing"); + self.closing = true; + self.session.start_close(); + } + + /// Poll the drain; `true` once it is over. + fn poll(&mut self, now: Instant, waiter: &kio::Waiter) -> bool { + if self.session.poll_closed(waiter).is_ready() { + tracing::debug!("old session drained cleanly after GOAWAY"); + return true; + } + if now >= self.deadline { + tracing::warn!("old session did not drain in time; closing"); + self.session.abort(Error::GoawayTimeout); + return true; + } + false + } +} diff --git a/rs/moq-net/src/reconnect/redirect.rs b/rs/moq-net/src/reconnect/redirect.rs new file mode 100644 index 00000000000..cf2b6325745 --- /dev/null +++ b/rs/moq-net/src/reconnect/redirect.rs @@ -0,0 +1,533 @@ +//! Judging the URI a peer's GOAWAY names, and catching peers that bounce a connection between them. + +use url::Url; + +use super::Failure; + +/// What to do with the URI a peer names in its GOAWAY. +/// +/// The URI is dialed exactly as given, so it must carry whatever credentials the +/// new endpoint needs. Nothing from the current session is copied onto it. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +#[non_exhaustive] +pub enum Redirect { + /// Follow any URI that does not downgrade the scheme, letting the peer name the + /// host as well as the port. + /// + /// The host is dialed as written, so this hands the peer whatever this process + /// can reach: a name it controls can carry an A record for loopback, a private + /// range, or the metadata service, and nothing here resolves it to find out. A + /// *literal* loopback, private, link-local, or IPC target is still refused, but + /// that catches a misconfigured peer, not a hostile one. Use it only with an + /// upstream trusted with the local network and with the tokens every session presents. + Follow, + /// Follow only when the host matches the dialed URL, so a peer can move us + /// between ports or schemes but not to another host. The default. + #[default] + SameHost, + /// Ignore the URI and redial the current targets. + Ignore, +} + +impl Redirect { + /// The URL a GOAWAY assigns. `Ok(None)` keeps the current targets (the peer named no + /// URI, or the policy ignores a URI it could parse), `Ok(Some)` replaces them, and `Err` + /// is an explicit URI this policy refuses. A malformed URI is refused even under + /// [`Self::Ignore`]. + /// + /// `pinned` is a certificate pin, which can only verify the host it was configured + /// for, so it refuses a host change from an encrypted scheme even under [`Self::Follow`]. + pub(super) fn target(&self, uri: &str, current: &Url, pinned: bool) -> Result, Failure> { + if uri.is_empty() { + return Ok(None); + } + + // The URI can carry credentials, so the error names the reason, never the URI. + // Parse before `Ignore`: a malformed redirect is terminal even when the policy + // would otherwise stay on the current targets. + let target = uri + .parse::() + .map_err(|_| Failure::Refused("the GOAWAY URI is malformed"))?; + if matches!(self, Self::Ignore) { + return Ok(None); + } + + if scheme_tier(target.scheme()) < scheme_tier(current.scheme()) { + return Err(Failure::Refused("the GOAWAY redirect downgrades the scheme")); + } + + // Only as far as the URL itself says: a name is dialed, never resolved here, + // so this refuses a peer that names a local address outright and says + // nothing about one that hides the same address behind a hostname. That gap + // is why [`Self::SameHost`] is the default; see [`is_local`]. + if is_local(&target) && !is_local(current) { + return Err(Failure::Refused( + "the GOAWAY redirect widens reachability to a local address", + )); + } + + // Host only, not the full authority: the port is what a peer legitimately + // moves us across when it hands off to a sibling process on the same box. + let same_host = target.host_str() == current.host_str(); + if matches!(self, Self::SameHost) && !same_host { + return Err(Failure::Refused("the GOAWAY redirect leaves the current host")); + } + // Only an encrypted dial installs the verifier a pin configures; tcp, unix, and + // plaintext WebSocket never consult it. + if pinned && encrypted(current.scheme()) && !same_host { + return Err(Failure::Refused( + "the GOAWAY redirect leaves the host a certificate pin verifies", + )); + } + + Ok(Some(target)) + } +} + +/// Whether a dial over `scheme` is encrypted, verifying the peer's certificate. +fn encrypted(scheme: &str) -> bool { + scheme_tier(scheme) == 2 +} + +/// Rank a scheme so a peer-supplied redirect cannot silently drop encryption. +/// Unknown schemes rank lowest, so a forgotten classification is refused. +fn scheme_tier(scheme: &str) -> u8 { + match scheme { + "https" | "moqt" | "moql" | "wss" | "tls" => 2, + "tcp" | "ws" | "http" => 1, + // `unix` lands here deliberately: local IPC is not an upgrade over a + // network transport, it is a different reachability class (see `is_local`). + _ => 0, + } +} + +/// Whether a URL *says* it names something only reachable from this host or +/// network. +/// +/// A judgement about the URL, not about where a dial would land. Literals and +/// `localhost` are decided here; every other name is left alone, since the +/// address behind it is chosen by whoever runs its DNS and can change between +/// this call and the dial. So a `false` here means "not local on its face", never +/// "verified remote". +fn is_local(url: &Url) -> bool { + match url.host() { + // A non-special scheme (`moqt://127.0.0.1`, and every other scheme but + // `http`/`https`/`ws`/`wss`) parses its host as a domain even when it is an + // address literal, so the literal check can't be left to the URL parser. + Some(url::Host::Domain(host)) => match host.parse::() { + Ok(ip) => is_local_ip(ip), + Err(_) => host == "localhost" || host.ends_with(".localhost"), + }, + Some(url::Host::Ipv4(ip)) => is_local_ip(ip.into()), + Some(url::Host::Ipv6(ip)) => is_local_ip(ip.into()), + // No host at all, e.g. a `unix:` socket path. + None => true, + } +} + +/// Whether an address literal is one only this host or network can reach. +fn is_local_ip(ip: std::net::IpAddr) -> bool { + match ip { + std::net::IpAddr::V4(ip) => is_local_v4(ip), + // An IPv4-mapped address reaches the same host as the v4 it wraps, so judge + // it by that rather than by the v6 rules. + std::net::IpAddr::V6(ip) => match ip.to_ipv4_mapped() { + Some(v4) => is_local_v4(v4), + None => ip.is_loopback() || ip.is_unspecified() || ip.is_unique_local() || ip.is_unicast_link_local(), + }, + } +} + +fn is_local_v4(ip: std::net::Ipv4Addr) -> bool { + ip.is_loopback() || ip.is_private() || ip.is_link_local() || ip.is_unspecified() +} + +/// Where `url` dials, without its userinfo, query, or fragment, where a peer may mint fresh +/// credentials on every redirect. Two URLs that differ only there reach the same endpoint. +pub(super) fn endpoint(url: &Url) -> Url { + let mut bare = url.clone(); + // Only fails for a URL that cannot carry userinfo, which then has none to strip. + let _ = bare.set_username(""); + let _ = bare.set_password(None); + bare.set_query(None); + bare.set_fragment(None); + bare +} + +/// Formats a dial URL for logs and errors as scheme, host, and port only. +/// +/// A URL can carry a credential in its userinfo, query, or path (raw QUIC has no headers, +/// so a membership proof rides as a path segment). Dropping all three rather than redacting +/// known-secret spellings is deliberate: a denylist only covers the credentials that exist +/// today. +pub(super) struct Shown<'a>(pub &'a Url); + +impl std::fmt::Display for Shown<'_> { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}://", self.0.scheme())?; + let Some(host) = self.0.host_str() else { + // No host at all, e.g. a `unix:` socket, where the path is the address + // rather than a credential and is the only thing that identifies it. + return write!(f, "{}", self.0.path()); + }; + write!(f, "{host}")?; + match self.0.port() { + Some(port) => write!(f, ":{port}"), + None => Ok(()), + } + } +} + +/// A run of GOAWAY redirects, and whether it has revisited an endpoint. +/// +/// The peer picks every hop, so remembering them all would grow without bound. Brent's cycle +/// detection instead keeps one departed endpoint, replaced at doubling intervals, and catches a +/// cycle of any length within a few laps of it. Once caught, the run stays a loop until reset. +pub(super) struct Redirects { + mark: Option, + span: u64, + hops: u64, + looped: bool, +} + +impl Default for Redirects { + fn default() -> Self { + Self { + mark: None, + span: 1, + hops: 0, + looped: false, + } + } +} + +impl Redirects { + /// Record leaving `from` for a different endpoint `to`, and return whether this run loops. + pub(super) fn depart(&mut self, from: Url, to: &Url) -> bool { + if self.mark.as_ref() == Some(to) { + self.looped = true; + } + self.hops += 1; + if self.hops == self.span { + self.mark = Some(from); + self.span = self.span.saturating_mul(2); + self.hops = 0; + } + self.looped + } + + /// Start a new run. + pub(super) fn reset(&mut self) { + *self = Self::default(); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn url(value: &str) -> Url { + value.parse().expect("valid url") + } + + fn refused(res: Result, Failure>) -> bool { + matches!(res, Err(Failure::Refused(_))) + } + + /// A peer must not be able to redirect us somewhere we could not already + /// reach. IPv4-mapped and link-local forms are the easy ones to miss. + #[test] + fn local_targets_are_recognized() { + let local = [ + "https://127.0.0.1/", + "https://localhost/", + "https://[::1]/", + "https://10.0.0.1/", + "https://169.254.1.1/", + "https://[::ffff:127.0.0.1]/", + "https://[::ffff:10.0.0.1]/", + "https://[fe80::1]/", + "https://[fc00::1]/", + "https://0.0.0.0/", + ]; + for url in local { + assert!(is_local(&url.parse().unwrap()), "{url} should be local"); + } + + for url in ["https://example.com/", "https://8.8.8.8/", "https://[2606:4700::1]/"] { + assert!(!is_local(&url.parse().unwrap()), "{url} should not be local"); + } + + // A public endpoint may not redirect us inward, but a local one may stay local. + let public = url("https://relay.example/"); + let localhost = url("https://127.0.0.1:4443/"); + assert!(refused(Redirect::Follow.target( + "https://[::ffff:127.0.0.1]/", + &public, + false + ))); + assert_eq!( + Redirect::Follow + .target("https://127.0.0.1:9999/", &localhost, false) + .unwrap() + .unwrap() + .port(), + Some(9999) + ); + } + + /// The scheme ranking is what stops a peer from quietly moving us off an + /// encrypted transport. An unknown scheme ranks lowest so a classification we + /// forgot to add is refused rather than trusted. + #[test] + fn scheme_tiers_rank_encrypted_above_plaintext() { + for scheme in ["https", "moqt", "moql", "wss", "tls"] { + assert_eq!(scheme_tier(scheme), 2, "{scheme} is encrypted"); + } + for scheme in ["tcp", "ws", "http"] { + assert_eq!(scheme_tier(scheme), 1, "{scheme} is plaintext"); + } + // `unix` is not an upgrade over a network transport, it is a different + // reachability class, so it must not outrank one. + for scheme in ["unix", "gopher", ""] { + assert_eq!(scheme_tier(scheme), 0, "{scheme} is unclassified"); + } + } + + /// A redirect may hold the scheme or improve it, never weaken it. A downgrade + /// is a refusal, not a reason to redial the address the peer is leaving. + #[test] + fn target_refuses_a_scheme_downgrade() { + let secure = url("https://relay.example/"); + let plain = url("http://relay.example/"); + + assert!(refused(Redirect::Follow.target( + "http://other.example/", + &secure, + false + ))); + assert!(refused(Redirect::Follow.target("unix:///tmp/moq.sock", &secure, false))); + + // Same tier and upgrades are followed. + let same = url("https://other.example/"); + assert_eq!( + Redirect::Follow + .target("https://other.example/", &secure, false) + .unwrap(), + Some(same.clone()) + ); + assert_eq!( + Redirect::Follow + .target("https://other.example/", &plain, false) + .unwrap(), + Some(same) + ); + } + + /// Native schemes must classify address literals even when the URL parser returns a domain. + #[test] + fn a_non_special_scheme_classifies_its_literal_host() { + for url in [ + "moqt://127.0.0.1/", + "moql://10.0.0.1/", + "tcp://169.254.169.254/", + "moqt://[::1]/", + "moqt://[::ffff:127.0.0.1]/", + ] { + assert!(is_local(&url.parse().unwrap()), "{url} should be local"); + } + for url in ["moqt://8.8.8.8/", "moql://[2606:4700::1]/", "tcp://relay.example/"] { + assert!(!is_local(&url.parse().unwrap()), "{url} should not be local"); + } + + let public = url("moqt://relay.example/"); + assert!( + refused(Redirect::Follow.target("moqt://169.254.169.254/", &public, false)), + "a literal local target is refused whatever the scheme" + ); + } + + /// The reachability guard reads the URL, so it cannot see a local address + /// hiding behind a hostname: any name the peer controls can carry an A record + /// for loopback or a private range, and a second lookup at dial time may answer + /// differently anyway. The default policy is what closes that, by refusing to + /// let a peer name a host at all. + #[test] + fn a_peer_named_host_is_refused_by_default() { + let public = url("https://relay.example/"); + + // The bypass: a name is not local on its face, so the reachability guard + // passes it through whatever it resolves to. + let rebindable = url("https://rebind.attacker.example/"); + assert!(!is_local(&rebindable), "a hostname is never classified as local"); + assert_eq!( + Redirect::Follow.target(rebindable.as_str(), &public, false).unwrap(), + Some(rebindable.clone()), + "Follow is explicit trust: it dials the name the peer chose" + ); + + // So the default refuses the host change instead of trying to judge it. + assert_eq!(Redirect::default(), Redirect::SameHost); + assert!( + refused(Redirect::default().target(rebindable.as_str(), &public, false)), + "a peer-named host is refused without resolving it" + ); + assert_eq!( + super::super::Goaway::default().redirect, + Redirect::SameHost, + "and the default config carries that default" + ); + } + + /// Only an explicit URI can assign, and one the policy will not follow is an + /// error rather than a quiet fallback: the loop ends on it instead of redialing. + #[test] + fn only_an_accepted_redirect_replaces_the_targets() { + let current = url("https://relay.example/"); + + // No URI, or a policy that ignores it: keep the current targets. + assert_eq!(Redirect::Follow.target("", ¤t, false).unwrap(), None); + assert_eq!( + Redirect::Ignore + .target("https://relay.example:5443/", ¤t, false) + .unwrap(), + None + ); + + // An explicit URI the policy will not follow is refused. + for (policy, uri) in [ + (Redirect::SameHost, "https://other.example/"), + (Redirect::Follow, "not a url"), + (Redirect::Ignore, "not a url"), + (Redirect::Follow, "http://relay.example/"), + (Redirect::Follow, "https://127.0.0.1/"), + ] { + assert!(refused(policy.target(uri, ¤t, false)), "{policy:?}: {uri}"); + } + + // An explicit assignment remains an assignment even if its URL is unchanged. + assert_eq!( + Redirect::SameHost.target(current.as_str(), ¤t, false).unwrap(), + Some(current.clone()) + ); + let moved = "https://relay.example:5443/"; + assert_eq!( + Redirect::SameHost.target(moved, ¤t, false).unwrap(), + Some(url(moved)) + ); + } + + /// A fingerprint is a Rustls check, installed only on encrypted dials. Schemes that + /// never install that verifier must not inherit the pin. + #[test] + fn a_certificate_pin_only_covers_encrypted_schemes() { + for scheme in ["https", "wss", "moqt", "moql", "tls"] { + assert!(encrypted(scheme), "{scheme}"); + } + for scheme in ["http", "ws", "tcp", "unix"] { + assert!(!encrypted(scheme), "{scheme}"); + } + assert_eq!( + Redirect::Follow + .target("tcp://other.example:1/", &url("tcp://relay.example:1/"), true) + .unwrap(), + Some(url("tcp://other.example:1/")), + "a plaintext dial never consulted the pin" + ); + } + + /// A certificate pin verifies only the host it was configured for, so it + /// refuses a host change even when the policy would follow one. + #[test] + fn a_certificate_pin_holds_the_host() { + let current = url("https://relay.example/"); + assert!(refused(Redirect::Follow.target( + "https://other.example/", + ¤t, + true + ))); + let moved = "https://relay.example:5443/"; + assert_eq!( + Redirect::Follow.target(moved, ¤t, true).unwrap(), + Some(url(moved)) + ); + } + + /// `SameHost` lets a peer move us between ports or schemes on the endpoint we + /// already chose, but not onto a different host. + #[test] + fn same_host_follows_a_port_not_a_host() { + let current = url("https://relay.example:4443/"); + + assert!( + refused(Redirect::SameHost.target("https://elsewhere.example/", ¤t, false)), + "another host is refused" + ); + + let moved = Redirect::SameHost + .target("https://relay.example:5443/", ¤t, false) + .unwrap() + .unwrap(); + assert_eq!( + moved.port(), + Some(5443), + "a different port on the same host is followed" + ); + } + + #[test] + fn redirect_loop_detection_catches_any_cycle_in_constant_memory() { + for tail in 0..4 { + for cycle in 2..=8 { + // `tail` one-way moves, then `cycle` endpoints redirecting around a ring. + let at = |hop: usize| { + if hop < tail { + url(&format!("tcp://tail{hop}/")) + } else { + url(&format!("tcp://ring{}/", (hop - tail) % cycle)) + } + }; + let mut redirects = Redirects::default(); + let mut caught = None; + for hop in 0..4 * (tail + cycle) { + let looped = redirects.depart(at(hop), &at(hop + 1)); + match caught { + Some(_) => assert!(looped, "a caught loop stays caught"), + None if looped => caught = Some(hop), + None => {} + } + } + let caught = caught.unwrap_or_else(|| panic!("tail {tail}, cycle {cycle}: never caught")); + // Never before the run first returns to an endpoint it left. + assert!( + caught + 1 >= tail + cycle, + "tail {tail}, cycle {cycle}: caught at {caught}" + ); + } + } + + let mut redirects = Redirects::default(); + for hop in 0..1000 { + let from = url(&format!("tcp://a{hop}/")); + let to = url(&format!("tcp://a{}/", hop + 1)); + assert!(!redirects.depart(from, &to), "a one-way run never loops"); + } + } + + #[test] + fn an_endpoint_ignores_the_credentials() { + assert_eq!( + endpoint(&url("tcp://token-a:secret@relay:1/path?jwt=a#x")), + endpoint(&url("tcp://token-b@relay:1/path?jwt=b")) + ); + // The path is part of the request target, so it names a different endpoint. + assert_ne!(endpoint(&url("tcp://relay:1/a")), endpoint(&url("tcp://relay:1/b"))); + } + + #[test] + fn a_shown_url_drops_every_credential() { + let shown = Shown(&url("moqt://user:pass@relay:4443/.cluster/secret?jwt=token")).to_string(); + assert_eq!(shown, "moqt://relay:4443"); + assert_eq!(Shown(&url("unix:///run/moq.sock")).to_string(), "unix:///run/moq.sock"); + } +} diff --git a/rs/moq-net/src/reconnect/tests.rs b/rs/moq-net/src/reconnect/tests.rs new file mode 100644 index 00000000000..47f3b939477 --- /dev/null +++ b/rs/moq-net/src/reconnect/tests.rs @@ -0,0 +1,813 @@ +//! The policy driven over in-memory sessions on the simulated clock, so each timing is exact. + +use super::*; + +const INITIAL: Duration = Duration::from_millis(100); +const MAX: Duration = Duration::from_millis(400); + +#[derive(Clone, Debug, thiserror::Error)] +enum TestError { + #[error(transparent)] + Net(#[from] Error), + #[error(transparent)] + Policy(#[from] Failure), + #[error("dial failed")] + Dial, +} + +fn url(value: &str) -> Url { + value.parse().expect("valid url") +} + +/// What the peer at an address does with each dial. +#[derive(Clone)] +enum Peer { + /// Accept, then drop the session at once. + Close, + /// Accept, then send a GOAWAY naming this URI (empty for a restart) and keep serving. + Goaway(String), + /// Accept and keep serving. + Serve, + /// Fail the dial, judged as this. + Fail(Verdict), + /// Never answer. + Hang, +} + +/// A target that dials fixed socket addresses. +#[derive(Clone)] +struct Fixed(Url); + +impl From for Fixed { + fn from(url: Url) -> Self { + Self(url) + } +} + +impl Target for Fixed { + fn url(&self) -> &Url { + &self.0 + } + + fn fixed(&self) -> bool { + true + } +} + +fn version() -> crate::Version { + "moq-lite-07-wip".parse().unwrap() +} + +/// Run an in-memory MoQ handshake, returning the client and server sessions. +async fn handshake() -> (Session, Session) { + let (client, server) = crate::transport::memory::pair(crate::transport::memory::Config { + protocol: Some(version().alpn().to_string()), + ..Default::default() + }); + let dial = crate::Client::new().with_versions(version().into()); + let serve = crate::Server::new().with_versions(version().into()); + futures::join!( + async { + let (session, driver) = dial.connect(kio::clock::now(), client).await.unwrap(); + drop(kio::clock::sim::spawn(crate::time::run(driver))); + session + }, + async { + let (session, driver) = serve.accept(kio::clock::now(), server).await.unwrap(); + drop(kio::clock::sim::spawn(crate::time::run(driver))); + session + }, + ) +} + +/// The next action that is not a wait, sleeping on the simulated clock meanwhile. +async fn next + From>( + machine: &mut Reconnect, +) -> Action { + let mut sleep: Option = None; + kio::wait(|waiter| { + loop { + match machine.poll(kio::clock::now(), waiter) { + Action::Wait(Some(at)) => { + let timer = sleep.get_or_insert_with(|| kio::clock::sleep_until(at)); + if timer.deadline() != at { + timer.reset(at); + } + if waiter.poll_future(std::pin::Pin::new(timer)).is_pending() { + return Poll::Pending; + } + } + Action::Wait(None) => return Poll::Pending, + action => return Poll::Ready(action), + } + } + }) + .await +} + +/// A record of one policy run. +#[derive(Default)] +struct Run { + /// The URL and instant of each session that came up. + accepts: Vec<(Url, Instant)>, + /// The URL of every dial. + dials: Vec, + /// Every action other than a dial, a wait, or the end, by name. + events: Vec<&'static str>, + /// The server sides still serving. + servers: Vec, +} + +/// Drive `machine` against `peers` until it ends, or until `stop` says so. +async fn drive( + machine: &mut Reconnect, + peers: &[(Url, Peer)], + run: &mut Run, + mut stop: impl FnMut(&Run) -> bool, +) -> Option> { + while !stop(run) { + let attempt = match next(machine).await { + Action::Dial(attempt) => attempt, + Action::Done(res) => return Some(res), + Action::Abandon => { + run.events.push("abandon"); + continue; + } + Action::Migrating => { + run.events.push("migrating"); + continue; + } + Action::Retired => { + run.events.push("retired"); + continue; + } + Action::Offline(_) => { + run.events.push("offline"); + continue; + } + Action::Retrying(_) => { + run.events.push("retrying"); + continue; + } + Action::Wait(_) => unreachable!("next never returns a wait"), + }; + let url = attempt.target().url().clone(); + run.dials.push(url.clone()); + let (_, peer) = peers.iter().find(|(peer, _)| *peer == url).expect("an unknown peer"); + match peer { + Peer::Hang => {} + Peer::Fail(verdict) => machine.failed(kio::clock::now(), TestError::Dial, *verdict), + Peer::Close | Peer::Goaway(_) | Peer::Serve => { + let (client, server) = handshake().await; + run.accepts.push((url, kio::clock::now())); + match peer { + Peer::Close => drop(server), + Peer::Goaway(uri) => { + let goaway = match uri.is_empty() { + true => goaway::Goaway::new(), + false => goaway::Goaway::redirect(uri.clone()), + }; + server.drain().send(goaway).unwrap(); + run.servers.push(server); + } + _ => run.servers.push(server), + } + machine.connected(kio::clock::now(), client); + } + } + } + None +} + +fn config(initial: Duration, timeout: Duration) -> Config { + let mut config = Config::default(); + config.backoff.initial = initial; + config.backoff.max = MAX; + config.backoff.timeout = timeout; + config +} + +/// Check each gap between consecutive sessions against the jittered delay it should +/// have waited: between half of `delay` and all of it, starting at `initial` and +/// doubling up to `max`. +fn assert_paced(accepts: &[(Url, Instant)], initial: Duration, max: Duration) { + let mut delay = initial; + for (index, pair) in accepts.windows(2).enumerate() { + let gap = pair[1].1 - pair[0].1; + assert!( + gap >= delay / 2 && gap <= delay, + "retry {index} waited {gap:?}, outside the jittered {delay:?}" + ); + delay = (delay * 2).min(max); + } +} + +/// A peer that accepts and then closes every session at once never earns the healthy +/// reset: the delay escalates to the cap, and the timeout ends the policy. +#[kio::test] +async fn an_immediate_close_loop_escalates_until_the_timeout() { + const TIMEOUT: Duration = Duration::from_secs(2); + let peer = url("tcp://peer:1/"); + let started = kio::clock::now(); + let mut machine = Reconnect::new(config(INITIAL, TIMEOUT), vec![peer.clone()], started).unwrap(); + let mut run = Run::default(); + + let err = drive(&mut machine, &[(peer, Peer::Close)], &mut run, |_| false) + .await + .unwrap() + .expect_err("a flapping peer kept the connection"); + assert!( + matches!(err, TestError::Policy(Failure::Exhausted { last: Some(_), .. })), + "{err}" + ); + assert_eq!( + kio::clock::now() - started, + TIMEOUT, + "the timeout did not end the policy" + ); + assert!(run.accepts.len() >= 6, "only {} sessions", run.accepts.len()); + assert_paced(&run.accepts, INITIAL, MAX); +} + +/// A peer restarting with an empty-URI GOAWAY on every session paces the redials like +/// failures, yet never spends the timeout: each drained session still served. +#[kio::test] +async fn an_immediate_restart_loop_escalates_without_giving_up() { + const TIMEOUT: Duration = Duration::from_millis(500); + let peer = url("tcp://peer:1/"); + let mut machine = Reconnect::new(config(INITIAL, TIMEOUT), vec![peer.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + + let ended = drive(&mut machine, &[(peer, Peer::Goaway(String::new()))], &mut run, |run| { + run.accepts.len() == 10 + }) + .await; + assert!(ended.is_none(), "a restarting peer ended the connection"); + assert!( + run.accepts[9].1 - run.accepts[0].1 > TIMEOUT * 2, + "the redials outran the backoff" + ); + assert!(!run.events.contains(&"retrying"), "a restart is no failure"); + assert_paced(&run.accepts, INITIAL, MAX); +} + +/// Two peers that redirect to each other the moment a session is up are failed attempts: +/// the delay escalates across both, and the timeout ends the policy. +#[kio::test] +async fn an_immediate_redirect_loop_escalates_until_the_timeout() { + const TIMEOUT: Duration = Duration::from_secs(2); + let a = url("tcp://peer:1/"); + let b = url("tcp://peer:2/"); + let peers = [ + (a.clone(), Peer::Goaway(b.to_string())), + (b.clone(), Peer::Goaway(a.to_string())), + ]; + let started = kio::clock::now(); + let mut machine = Reconnect::new(config(INITIAL, TIMEOUT), vec![a.clone()], started).unwrap(); + let mut run = Run::default(); + + let err = drive(&mut machine, &peers, &mut run, |_| false) + .await + .unwrap() + .expect_err("a redirect loop kept the connection"); + assert!(err.to_string().contains("redirect"), "{err}"); + assert_eq!( + kio::clock::now() - started, + TIMEOUT, + "the timeout did not end the policy" + ); + assert!(run.accepts.len() >= 6, "only {} sessions", run.accepts.len()); + for (index, (peer, _)) in run.accepts.iter().enumerate() { + assert_eq!( + *peer, + if index % 2 == 0 { a.clone() } else { b.clone() }, + "session {index}" + ); + } + assert_paced(&run.accepts, INITIAL, MAX); +} + +/// A zero `initial` turns retry waiting off: every redial follows the close at once. +#[kio::test] +async fn a_zero_initial_redials_at_once() { + let peer = url("tcp://peer:1/"); + let first = kio::clock::now(); + let mut machine = Reconnect::new( + config(Duration::ZERO, Duration::from_secs(1)), + vec![peer.clone()], + first, + ) + .unwrap(); + let mut run = Run::default(); + + drive(&mut machine, &[(peer, Peer::Close)], &mut run, |run| { + run.accepts.len() == 11 + }) + .await; + for (_, at) in &run.accepts { + assert_eq!(*at, first, "a zero initial waited before a retry"); + } +} + +/// The first address that answers wins, however many dead ones precede it. +#[kio::test] +async fn the_walk_lands_on_the_first_address_that_answers() { + let targets = vec![url("tcp://a:1/"), url("tcp://b:1/"), url("tcp://c:1/")]; + let peers = [ + (targets[0].clone(), Peer::Fail(Verdict::Retry)), + (targets[1].clone(), Peer::Fail(Verdict::Unusable)), + (targets[2].clone(), Peer::Serve), + ]; + let started = kio::clock::now(); + let mut machine = Reconnect::new(Config::default(), targets.clone(), started).unwrap(); + let mut run = Run::default(); + + drive(&mut machine, &peers, &mut run, |run| !run.accepts.is_empty()).await; + assert_eq!(run.dials, targets, "every address in order"); + assert_eq!( + run.accepts, + [(targets[2].clone(), started)], + "no wait between addresses" + ); + assert!(machine.session().is_some()); +} + +/// A black-holed address with a fallback gets the fixed bound; the last one runs to the +/// connect timeout. +#[kio::test] +async fn only_an_address_with_a_fallback_is_bounded() { + let targets = vec![url("tcp://a:1/"), url("tcp://b:1/")]; + let peers = [(targets[0].clone(), Peer::Hang), (targets[1].clone(), Peer::Hang)]; + let mut config = Config::default(); + config.backoff.timeout = Duration::from_secs(1); + let started = kio::clock::now(); + let mut machine = Reconnect::new(config, targets.clone(), started).unwrap(); + let mut run = Run::default(); + + let err = drive(&mut machine, &peers, &mut run, |_| false) + .await + .unwrap() + .expect_err("nothing answered"); + assert_eq!(run.events, ["abandon", "abandon", "retrying"]); + assert_eq!(run.dials, targets, "a walk under way finishes past the retry window"); + assert_eq!(kio::clock::now() - started, ATTEMPT + config.timeout); + // The give-up names the last address's connect timeout as its cause. + let last = Failure::Timeout(config.timeout).to_string(); + assert!( + matches!(&err, TestError::Policy(Failure::Exhausted { last: Some(cause), .. }) if *cause == last), + "{err}" + ); +} + +/// A settled answer from one address ends the walk instead of being buried by another's +/// transport failure, and the policy retries after a backoff. +#[kio::test] +async fn a_settled_answer_stops_the_walk() { + let targets = vec![url("tcp://a:1/"), url("tcp://b:1/")]; + let peers = [ + (targets[0].clone(), Peer::Fail(Verdict::Settled)), + (targets[1].clone(), Peer::Serve), + ]; + let mut machine = Reconnect::new( + config(INITIAL, Duration::from_secs(1)), + targets.clone(), + kio::clock::now(), + ) + .unwrap(); + let mut run = Run::default(); + + drive(&mut machine, &peers, &mut run, |run| run.dials.len() == 2).await; + assert_eq!( + run.dials, + [targets[0].clone(), targets[0].clone()], + "the walk stopped at the answer" + ); + assert_eq!(run.events, ["retrying"]); +} + +/// One unusable address must not turn another's transient failure terminal, but a walk +/// with nothing else ends on it. +#[kio::test] +async fn an_unusable_address_ends_only_a_walk_without_hope() { + let targets = vec![url("tcp://a:1/"), url("tcp://b:1/")]; + let peers = [ + (targets[0].clone(), Peer::Fail(Verdict::Retry)), + (targets[1].clone(), Peer::Fail(Verdict::Unusable)), + ]; + let mut machine = Reconnect::new(config(INITIAL, Duration::ZERO), targets.clone(), kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &peers, &mut run, |run| run.dials.len() == 4).await; + assert_eq!(run.events, ["retrying"], "an unlimited window keeps retrying"); + + let lone = url("tcp://c:1/"); + let started = kio::clock::now(); + let mut machine = Reconnect::new(config(INITIAL, Duration::ZERO), vec![lone.clone()], started).unwrap(); + let err = drive( + &mut machine, + &[(lone, Peer::Fail(Verdict::Unusable))], + &mut Run::default(), + |_| false, + ) + .await + .unwrap() + .expect_err("an unusable address connected"); + assert!(matches!(err, TestError::Dial), "{err}"); + assert_eq!(kio::clock::now(), started, "ended without waiting"); +} + +/// A refused credential is terminal, unless the tokens changed since the dial read them. +#[kio::test] +async fn a_refusal_retries_only_new_tokens() { + let peer = url("tcp://peer:1/"); + let tokens = auth::Tokens::new(vec!["old".parse().unwrap()]); + let mut machine = Reconnect::new(config(INITIAL, Duration::ZERO), vec![peer.clone()], kio::clock::now()) + .unwrap() + .with_tokens(tokens.clone(), version().into()); + + let Action::Dial(_) = next(&mut machine).await else { + panic!("expected a dial") + }; + tokens.set(vec!["new".parse().unwrap()]); + machine.failed(kio::clock::now(), TestError::Dial, Verdict::Refused); + assert!(matches!(next(&mut machine).await, Action::Retrying(_))); + + let Action::Dial(_) = next(&mut machine).await else { + panic!("expected a redial") + }; + machine.failed(kio::clock::now(), TestError::Dial, Verdict::Refused); + assert!( + matches!(next(&mut machine).await, Action::Done(Err(TestError::Dial))), + "an unchanged credential is refused for good" + ); +} + +/// Ending the tokens' handle ends the policy at its next walk, rather than redialing +/// sessions the handle refuses. +#[kio::test] +async fn an_ended_auth_handle_ends_the_policy() { + let peer = url("tcp://peer:1/"); + let tokens = auth::Tokens::new(Vec::new()); + let mut machine = Reconnect::new(config(INITIAL, Duration::ZERO), vec![peer.clone()], kio::clock::now()) + .unwrap() + .with_tokens(tokens.clone(), version().into()); + let mut run = Run::default(); + + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; + tokens.handle().abort(Error::App(9)); + run.servers.clear(); + let ended = drive(&mut machine, &[(peer, Peer::Serve)], &mut run, |_| false).await; + assert!( + matches!(ended, Some(Err(TestError::Net(Error::App(9))))), + "the policy kept redialing on an ended auth handle" + ); + assert_eq!(run.accepts.len(), 1); +} + +/// Connecting once, a GOAWAY ends the policy cleanly and a close ends it with its reason. +#[kio::test] +async fn connecting_once_ends_with_the_session() { + let peer = url("tcp://peer:1/"); + let config = Config { + once: true, + ..Default::default() + }; + + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let ended = drive( + &mut machine, + &[(peer.clone(), Peer::Goaway(String::new()))], + &mut Run::default(), + |_| false, + ) + .await; + assert!(matches!(ended, Some(Ok(()))), "a GOAWAY is the cue to leave"); + + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let ended = drive( + &mut machine, + &[(peer.clone(), Peer::Close)], + &mut Run::default(), + |_| false, + ) + .await; + assert!( + matches!(ended, Some(Err(TestError::Net(_)))), + "the close reason ends it" + ); + + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let ended = drive( + &mut machine, + &[(peer, Peer::Fail(Verdict::Retry))], + &mut Run::default(), + |_| false, + ) + .await; + assert!( + matches!(ended, Some(Err(TestError::Dial))), + "a failed dial is not retried" + ); +} + +/// A target dialing fixed socket addresses cannot follow a redirect. +#[kio::test] +async fn a_fixed_target_refuses_a_redirect() { + let peer = url("moqt://relay.example:4443/"); + let moved = url("moqt://relay.example:5443/"); + let mut machine = Reconnect::new(Config::default(), vec![Fixed(peer.clone())], kio::clock::now()).unwrap(); + let ended = drive( + &mut machine, + &[(peer, Peer::Goaway(moved.to_string()))], + &mut Run::default(), + |_| false, + ) + .await; + assert!(matches!(ended, Some(Err(TestError::Policy(Failure::Fixed))))); +} + +/// A redirect the policy refuses ends it rather than redialing the address the peer leaves. +#[kio::test] +async fn a_refused_redirect_is_terminal() { + let peer = url("moqt://relay.example:4443/"); + let mut machine = Reconnect::new(Config::default(), vec![peer.clone()], kio::clock::now()).unwrap(); + let ended = drive( + &mut machine, + &[(peer, Peer::Goaway("moqt://elsewhere.example/".into()))], + &mut Run::default(), + |_| false, + ) + .await; + assert!(matches!(ended, Some(Err(TestError::Policy(Failure::Refused(_)))))); +} + +/// A healthy session's GOAWAY migrates at once to the named address, and the drained +/// predecessor closes as soon as the replacement serves and it carries nothing. +#[kio::test] +async fn a_healthy_goaway_hands_over_to_the_redirect() { + let a = url("tcp://peer:1/"); + let b = url("tcp://peer:2/"); + let mut machine = Reconnect::new(Config::default(), vec![a.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &[(a.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; + + // Past the healthy bar, the GOAWAY is a handover, not a failure. + kio::clock::sleep(Duration::from_secs(2)).await; + run.servers[0] + .drain() + .send(goaway::Goaway::redirect(b.to_string())) + .unwrap(); + let peers = [(a.clone(), Peer::Serve), (b.clone(), Peer::Serve)]; + let migrated = kio::clock::now(); + drive(&mut machine, &peers, &mut run, |run| run.events.contains(&"retired")).await; + assert_eq!(run.dials, [a, b.clone()]); + assert_eq!(run.accepts[1], (b, migrated), "migrated without a backoff"); + assert_eq!( + run.events, + ["migrating", "retired"], + "a replacement serves, so nothing went offline" + ); + assert!(machine.predecessor().is_none()); + assert!(machine.session().is_some()); +} + +/// Our handover window caps the peer's GOAWAY deadline, and a predecessor retiring while +/// no replacement serves leaves the connection offline. +#[kio::test] +async fn the_handover_takes_the_earlier_deadline() { + const HANDOVER: Duration = Duration::from_millis(500); + let peer = url("tcp://peer:1/"); + let mut config = Config::default(); + config.goaway.handover = HANDOVER; + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; + + kio::clock::sleep(Duration::from_secs(2)).await; + run.servers[0] + .drain() + .send(goaway::Goaway::new().with_timeout(Duration::from_secs(3600))) + .unwrap(); + let drained = kio::clock::now(); + drive(&mut machine, &[(peer, Peer::Hang)], &mut run, |run| { + run.events.contains(&"offline") + }) + .await; + assert_eq!(run.events, ["migrating", "retired", "offline"]); + assert_eq!( + kio::clock::now() - drained, + HANDOVER, + "a longer peer deadline must not extend our cap" + ); + + let goaway = Goaway::default(); + assert_eq!(goaway.handover(None), Duration::from_secs(10), "no deadline: ours"); + assert_eq!( + goaway.handover(Some(Duration::from_secs(3))), + Duration::from_secs(3), + "a shorter peer deadline wins: it force-closes then anyway" + ); +} + +/// A replacement that connects after the predecessor retired mid-dial serves, so the +/// offline the retirement queued no longer holds and is never reported. +#[kio::test] +async fn a_connect_cancels_the_offline_a_retirement_queued() { + const HANDOVER: Duration = Duration::from_millis(500); + let peer = url("tcp://peer:1/"); + let mut config = Config::default(); + config.goaway.handover = HANDOVER; + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; + + kio::clock::sleep(Duration::from_secs(2)).await; + run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); + assert!(matches!(next(&mut machine).await, Action::Migrating)); + assert!(matches!(next(&mut machine).await, Action::Dial(_))); + // The dial outlasts the handover, so the predecessor retires with nothing serving. + assert!(matches!(next(&mut machine).await, Action::Retired)); + + let (client, _server) = handshake().await; + machine.connected(kio::clock::now(), client); + assert!( + matches!( + machine.poll(kio::clock::now(), &kio::Waiter::noop()), + Action::Wait(None) + ), + "a stale offline followed the connect" + ); + assert!(machine.session().is_some()); +} + +/// Under [`Restart::Wait`] an empty-URI GOAWAY changes nothing while the peer keeps the +/// session: no migration and no redial. Its close then reconnects like any drop. +#[kio::test] +async fn a_waited_restart_redials_only_once_the_session_closes() { + let peer = url("tcp://peer:1/"); + let mut config = config(INITIAL, Duration::from_secs(1)); + config.restart = Restart::Wait; + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; + + kio::clock::sleep(Duration::from_secs(2)).await; + run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); + kio::clock::sim::idle().await; + assert!( + matches!( + machine.poll(kio::clock::now(), &kio::Waiter::noop()), + Action::Wait(None) + ), + "the policy left the session" + ); + + run.servers.remove(0).abort(Error::GoawayTimeout); + drive(&mut machine, &[(peer, Peer::Serve)], &mut run, |run| { + run.accepts.len() == 2 + }) + .await; + assert_eq!(run.events, ["offline"]); +} + +/// The policy must not log or report a credential riding in a dial URL. +/// +/// A LAN mesh's membership proof rides as a URL path segment, since raw QUIC has no +/// headers to put it in, and a dial URL logged verbatim would hand a replayable credential +/// to anyone who can read logs. This drives the real `tracing` events rather than the +/// formatter alone, so it also catches a log site that forgot to use it. +#[test] +fn dialing_never_logs_the_credential() { + const SECRET: &str = "b91d7fe20c4a"; + let targets = vec![ + url(&format!( + "moqt://user:{SECRET}@a.example/.cluster/{SECRET}?jwt={SECRET}" + )), + url(&format!("moqt://b.example/.cluster/{SECRET}")), + ]; + let started = Instant::now(); + let mut machine = Reconnect::::new(Config::default(), targets, started).unwrap(); + let waiter = kio::Waiter::noop(); + + let mut reported = Vec::new(); + let events = crate::test_tracing::capture(tracing::Level::DEBUG, || { + assert!(matches!(machine.poll(started, &waiter), Action::Dial(_))); + // The first address has a fallback, so it times out with an error naming it. + assert!(matches!(machine.poll(started + ATTEMPT, &waiter), Action::Abandon)); + assert!(matches!(machine.poll(started + ATTEMPT, &waiter), Action::Dial(_))); + machine.failed(started + ATTEMPT, TestError::Dial, Verdict::Retry); + loop { + match machine.poll(started + ATTEMPT, &waiter) { + Action::Retrying(err) => reported.push(err.to_string()), + Action::Wait(_) => break, + action => panic!("unexpected {action:?}"), + } + } + }); + + // The log lines did happen, so the assertion below isn't vacuously true. + assert!(events.iter().any(|event| event.contains("connecting")), "{events:?}"); + assert!(events.iter().any(|event| event.contains("a.example")), "{events:?}"); + for line in events.iter().chain(&reported) { + assert!(!line.contains(SECRET), "leaked the credential: {line}"); + } +} + +#[test] +fn invalid_configs_are_refused() { + assert!(matches!( + Reconnect::::new(Config::default(), Vec::new(), Instant::now()), + Err(ConfigError::Empty) + )); + let mut config = Config::default(); + config.backoff.healthy = Duration::ZERO; + assert!(matches!( + Reconnect::::new(config, vec![url("tcp://peer:1/")], Instant::now()), + Err(ConfigError::Healthy) + )); +} + +/// Each knob means one thing at zero: no wait, no cap, no growth. None of them touches +/// the healthy bar, which is what keeps an immediate close a failed attempt. +#[test] +fn zero_turns_each_pacing_knob_off() { + let defaults = Backoff::default(); + + let pacing = Pacing::new(&Backoff { + initial: Duration::ZERO, + ..defaults + }); + assert_eq!(pacing.initial, Duration::ZERO); + assert_eq!(pacing.next(pacing.initial), Duration::ZERO, "zero initial never waits"); + assert_eq!(pacing.healthy, defaults.healthy); + + let pacing = Pacing::new(&Backoff { + max: Duration::ZERO, + ..defaults + }); + assert_eq!(pacing.max, None); + let mut delay = pacing.initial; + for _ in 0..10 { + delay = pacing.next(delay); + } + assert_eq!(delay, defaults.initial * 1024, "zero max leaves the delay uncapped"); + assert_eq!(pacing.settled(), defaults.healthy); + + let pacing = Pacing::new(&Backoff { + multiplier: 0, + ..defaults + }); + assert_eq!( + pacing.next(pacing.initial), + pacing.initial, + "zero multiplier never grows" + ); + + // A cap below the first delay caps it too. + let pacing = Pacing::new(&Backoff { + max: Duration::from_millis(300), + ..defaults + }); + assert_eq!(pacing.initial, Duration::from_millis(300)); +} + +/// A sane config is left alone: the delay doubles up to the cap and stays there. +#[test] +fn pacing_grows_to_the_cap() { + let pacing = Pacing::new(&Backoff { + initial: Duration::from_millis(100), + multiplier: 2, + max: Duration::from_millis(400), + ..Default::default() + }); + + assert_eq!(pacing.initial, Duration::from_millis(100)); + assert_eq!(pacing.settled(), Duration::from_millis(400)); + assert_eq!(pacing.next(Duration::from_millis(100)), Duration::from_millis(200)); + assert_eq!(pacing.next(Duration::from_millis(200)), Duration::from_millis(400)); + assert_eq!(pacing.next(Duration::from_millis(400)), Duration::from_millis(400)); +} + +/// Refusals and malformed input settle; everything else from MoQ is worth a retry. +#[test] +fn a_moq_error_is_judged_by_its_session_code() { + assert_eq!(Verdict::from(&Error::Unauthorized), Verdict::Refused); + assert_eq!(Verdict::from(&Error::Expired), Verdict::Refused); + assert_eq!(Verdict::from(&Error::ProtocolViolation), Verdict::Fatal); + assert_eq!(Verdict::from(&Error::Closed), Verdict::Retry); + assert_eq!(Verdict::from(&Error::Transport("reset".into())), Verdict::Retry); +} diff --git a/rs/moq-net/src/session.rs b/rs/moq-net/src/session.rs index d86c9f46f7d..339cee8cf6e 100644 --- a/rs/moq-net/src/session.rs +++ b/rs/moq-net/src/session.rs @@ -331,11 +331,7 @@ impl Session { /// Both protocols withdraw this session's announcements and wait for their /// delivery. pub async fn close(self) -> Result<(), Error> { - if let Ok(mut close) = self.close.write() - && close.is_none() - { - *close = Some(Close::Drain); - } + self.start_close(); // The drain outlives this handle, so dropping it cannot cut the drain short. let closed = self.closed.clone(); drop(self); @@ -368,8 +364,8 @@ impl Session { kio::wait(|waiter| self.poll_closed(waiter)).await } - /// See [`Self::closed`]. - fn poll_closed(&self, waiter: &kio::Waiter) -> Poll { + /// Poll form of [`closed`](Self::closed). + pub(crate) fn poll_closed(&self, waiter: &kio::Waiter) -> Poll { Poll::Ready( match ready!(poll_ended(&self.closed, waiter, |ended| ended.err.clone())) { Ok(err) => err, @@ -379,6 +375,15 @@ impl Session { ) } + /// Start the drain [`close`](Self::close) runs, without waiting for it or giving up this handle. + pub(crate) fn start_close(&self) { + if let Ok(mut close) = self.close.write() + && close.is_none() + { + *close = Some(Close::Drain); + } + } + /// Drain the peer gracefully: the handle for sending this session's single /// GOAWAY. /// diff --git a/rs/moq-tokio/src/client.rs b/rs/moq-tokio/src/client.rs index 63a3eeeeba4..b8925c4c73c 100644 --- a/rs/moq-tokio/src/client.rs +++ b/rs/moq-tokio/src/client.rs @@ -4,8 +4,7 @@ //! QUIC settings ([`crate::quic::Config`]) a binary shares with its accept half. The //! accept side is [`crate::server`]. -use crate::connection::Goaway; -use crate::{Addrs, Backoff, Connection, Error, Restart}; +use crate::{Addrs, Connection, Error, Restart}; #[cfg(all(feature = "websocket", feature = "quic"))] use std::future::Future; use url::Url; @@ -51,16 +50,8 @@ pub struct Client { connect: Option, /// The credentials each new connection starts presenting. pub(crate) tokens: Vec, - /// Deadline for one [`Client::connect`], from [`crate::connect::Config::timeout`]. Zero waits forever. - #[cfg(feature = "_transport")] - timeout: std::time::Duration, - pub(crate) reconnect: bool, - pub(crate) backoff: Backoff, - pub(crate) goaway: Goaway, - pub(crate) restart: Restart, - /// Whether the TLS config pins a certificate fingerprint, which only verifies - /// the configured host, so a GOAWAY may not redirect elsewhere. - pub(crate) pinned: bool, + /// The connect deadline, backoff, and GOAWAY handling each [`Connection`] runs under. + pub(crate) policy: moq_net::reconnect::Config, /// The resolved Happy Eyeballs timings, used by the `tcp://` and `tls://` /// dials here; the QUIC backend captures its own copy from the config. #[cfg(feature = "tcp")] @@ -91,10 +82,12 @@ pub struct Client { pub(crate) dialer: Option>, } -/// A test stand-in for [`Client::dial`]. +/// A test stand-in for [`Client::dial`], handed the address and the MoQ client a real dial +/// would handshake with. #[cfg(test)] -pub(crate) type Dialer = - dyn Fn(crate::connect::Addr) -> std::pin::Pin> + Send>> + Send + Sync; +pub(crate) type Dialer = dyn Fn(crate::connect::Addr, moq_net::Client) -> std::pin::Pin> + Send>> + + Send + + Sync; impl Client { /// Build a client from its config. @@ -146,7 +139,11 @@ impl Client { let tcp_tls = config.tls.clone(); #[cfg(feature = "websocket")] let tls_host_name = config.tls.host_name.clone(); - let timeout = resolved.timeout; + let mut policy = moq_net::reconnect::Config::default(); + policy.backoff = config.backoff.resolve(); + policy.goaway = config.goaway.resolve(); + policy.timeout = resolved.timeout; + policy.pinned = !config.tls.fingerprint.is_empty(); let mut moq = moq_net::Client::new() .with_versions(versions.clone()) @@ -160,12 +157,7 @@ impl Client { versions, connect: config.url, tokens: config.tokens, - timeout, - reconnect: true, - backoff: config.backoff, - goaway: config.goaway, - restart: Restart::default(), - pinned: !config.tls.fingerprint.is_empty(), + policy, #[cfg(feature = "tcp")] failover_delay, #[cfg(feature = "tcp")] @@ -247,13 +239,13 @@ impl Client { /// /// Defaults to true. Each clone can choose its own reconnect policy. pub fn with_reconnect(mut self, reconnect: bool) -> Self { - self.reconnect = reconnect; + self.policy.once = !reconnect; self } /// Choose what a GOAWAY naming no URI does. Defaults to [`Restart::Redial`]. pub fn with_restart(mut self, restart: Restart) -> Self { - self.restart = restart; + self.policy.restart = restart; self } @@ -353,11 +345,12 @@ impl Client { Ok::<_, Error>(connect_session(&self.moq, session).await?) }; - let session = match self.timeout.is_zero() { + let timeout = self.policy.timeout; + let session = match timeout.is_zero() { true => attempt.await?, - false => kio::clock::timeout(self.timeout, attempt) + false => kio::clock::timeout(timeout, attempt) .await - .map_err(|_| Error::ConnectTimeout(self.timeout))??, + .map_err(|_| Error::ConnectTimeout(timeout))??, }; tracing::info!(version = %session.version(), transport = %crate::Transport::WebRtc, "connected"); Ok(session) @@ -365,39 +358,34 @@ impl Client { .await } - /// What a dial to `url` presents of `tokens`, given the versions this client offers. - /// Dial the given URL and complete the MoQ handshake. + /// Dial one attempt of a [`Connection`]'s policy and complete the MoQ handshake. /// /// Errors if no transport feature is compiled in. #[cfg(not(any(feature = "quic", feature = "websocket", feature = "tcp", feature = "uds")))] pub(crate) async fn dial( &self, - _addr: crate::connect::Addr, - _presented: &moq_net::auth::Dial, + _attempt: moq_net::reconnect::Attempt, ) -> crate::Result { Err(Error::NoBackend( "no backend compiled; enable quic, websocket, tcp, or uds feature", )) } - /// Dial the given URL and complete the MoQ handshake, presenting what `presented` decided. + /// Dial one attempt of a [`Connection`]'s policy and complete the MoQ handshake, + /// presenting the URL and credential the attempt carries. /// /// The scheme picks the transport, and `https://` races QUIC against the /// WebSocket fallback so a blocked UDP path still connects. The session's /// protocol driver is spawned on the current tokio runtime, else the fallback; the session - /// closes once the last returned handle drops. + /// closes once the last returned handle drops. The policy bounds the attempt: dropping + /// it cancels whichever arm is still pending. #[cfg(any(feature = "quic", feature = "websocket", feature = "tcp", feature = "uds"))] pub(crate) async fn dial( &self, - addr: crate::connect::Addr, - presented: &moq_net::auth::Dial, + attempt: moq_net::reconnect::Attempt, ) -> crate::Result { - let addr = addr.with_url(presented.url.clone()); - #[cfg(test)] - if let Some(dialer) = &self.dialer { - return dialer(addr).await; - } - let client = match &presented.token { + let addr = attempt.target().with_url(attempt.url().clone()); + let client = match attempt.token() { Some(token) => { let mut client = self.clone(); client.moq = client.moq.with_token(token.clone()); @@ -405,22 +393,14 @@ impl Client { } None => std::borrow::Cow::Borrowed(self), }; + #[cfg(test)] + if let Some(dialer) = &self.dialer { + return dialer(addr, client.moq.clone()).await; + } // Each compiled backend adds state to this dispatch future. Keep it off the // caller's stack so all-feature builds remain safe on standard 2 MiB threads. - let attempt = Box::pin(client.connect_inner(addr)); - - // The deadline covers the dial AND the handshake, for every transport: it is the - // only bound some of them have. Dropping `attempt` on expiry cancels whichever - // arm was still pending. - let dialed = match self.timeout.is_zero() { - true => attempt.await?, - false => match kio::clock::timeout(self.timeout, attempt).await { - Ok(res) => res?, - Err(_) => return Err(Error::ConnectTimeout(self.timeout)), - }, - }; - + let dialed = Box::pin(client.connect_inner(addr)).await?; tracing::info!(version = %dialed.session.version(), transport = %dialed.transport, "connected"); Ok(dialed) } @@ -1361,20 +1341,21 @@ mod tests { ..Default::default() }; config.websocket.delay = Some((std::time::Duration::ZERO).into()); - let client = config.init(Default::default()).unwrap(); + // Connecting once: this is about one attempt's deadline, which a reconnecting + // connection would redial past. + let client = config.init(Default::default()).unwrap().with_reconnect(false); // Nothing is listening on UDP, so the QUIC arm fails and leaves the WebSocket // arm alone against the silent peer. let url: Url = format!("https://127.0.0.1:{}/", addr.port()).parse().unwrap(); - // `dial` rather than `connect`: this is about one attempt's deadline, and - // `connect` now hands back a reconnect loop that would redial past it. - let presented = moq_net::auth::Tokens::new(Vec::new()).dial(&url, &client.versions); - let mut attempt = Box::pin(client.dial(url.into(), &presented)); + let connection = client.connect(url); + let closed = connection.closed(); + tokio::pin!(closed); let _silent = tokio::select! { - res = &mut attempt => match res { + res = &mut closed => match res { Err(err) => panic!("connect failed before the silent peer accepted it: {err}"), - Ok(_) => panic!("connected to a peer that never spoke"), + Ok(()) => panic!("stopped before the silent peer accepted it"), }, res = listener.accept() => res.unwrap().0, }; @@ -1384,11 +1365,7 @@ mod tests { tokio::time::pause(); tokio::time::advance(timeout).await; - let err = match attempt.await { - Err(err) => err, - Ok(_) => panic!("connected to a peer that never spoke"), - }; - + let err = closed.await.expect_err("connected to a peer that never spoke"); assert!(matches!(err, Error::ConnectTimeout(_)), "unexpected error: {err}"); } } diff --git a/rs/moq-tokio/src/connect.rs b/rs/moq-tokio/src/connect.rs index 1a0404f08e3..e5e73673adc 100644 --- a/rs/moq-tokio/src/connect.rs +++ b/rs/moq-tokio/src/connect.rs @@ -102,6 +102,16 @@ impl From for Addr { } } +impl moq_net::reconnect::Target for Addr { + fn url(&self) -> &Url { + &self.url + } + + fn fixed(&self) -> bool { + self.resolved.is_some() + } +} + /// One or more addresses for the same peer, tried in order until one connects. /// /// Most callers have a single URL and never name this type: diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs index c63cd4e0f2c..898cca07e23 100644 --- a/rs/moq-tokio/src/connection.rs +++ b/rs/moq-tokio/src/connection.rs @@ -1,4 +1,8 @@ //! The reconnect loop around a session, and the [`Monitor`] that observes it. +//! +//! The policy (the connect deadline, the address walk, backoff, and GOAWAY handling) is +//! [`moq_net::Reconnect`]'s; this module dials what it asks for, runs it on a task, and +//! publishes what it does. use std::task::{Poll, ready}; use std::time::Duration; @@ -6,69 +10,45 @@ use std::time::Duration; use moq_net::Version; use moq_net::bandwidth::{Consumer as BandwidthConsumer, Producer as BandwidthProducer}; use moq_net::kio; -use rand::RngExt; -use url::Url; +use moq_net::reconnect::{Action, Verdict}; use crate::abort::AbortOnDrop; use crate::connect::Endpoint; use crate::{Addrs, Client, Error}; -/// How long one address gets before [`Connection`] moves to the peer's next one. -const CONNECT_ATTEMPT: Duration = Duration::from_secs(5); +pub use moq_net::reconnect::Restart; -/// The deadline for the attempt at candidate `index` of `total`, or `None` to -/// let it run. +/// How a failed dial bears on the rest of the policy. /// -/// A black-holed address answers nothing rather than refusing, so without a -/// bound the QUIC idle timeout would decide how long the remaining candidates -/// wait. Only an attempt with a later candidate to fall back on is worth -/// bounding: cutting the last one short just turns a slow connect into a failed -/// one, on every reconnect cycle. -/// -/// The same goes for the retry window from [`Backoff::timeout`]: it decides -/// whether a new walk starts, not how long one may take, so a walk under way -/// tries every candidate and a handshake slower than the window still gets -/// [`crate::connect::Config::timeout`](crate::connect::Config::timeout). -fn attempt_timeout(index: usize, total: usize) -> Option { - (index + 1 < total).then_some(CONNECT_ATTEMPT) -} - -/// Whether `err` refused the credential a dial presented while the client has since been -/// given new tokens, which earns another dial instead of ending the loop. It never -/// retries a rejected credential: a refusal of unchanged tokens stays terminal. -fn refreshed(err: &Error, tokens: &moq_net::auth::Tokens, presented: Option<&moq_net::auth::Dial>) -> bool { - let refused = err.is_auth() - || matches!(err, Error::MoqNet(err) if moq_net::SessionError::from(err) == moq_net::SessionError::Unauthorized); - refused && presented.is_some_and(|presented| tokens.changed(presented)) -} - -/// Whether a failed candidate is the peer's answer for every candidate, ending the walk. -/// -/// An auth rejection arrives as [`Error::Connect`] with its status already consumed, so it -/// is checked alongside the statuses that do not invite another attempt. A terminal MoQ error -/// means the handshake reached the peer, which answered with malformed input. -fn settles_walk(err: &Error) -> bool { - err.is_auth() - || (matches!(err, Error::MoqNet(_)) && err.is_terminal()) +/// A status the peer actually sent is its answer, not this address's, so unless it invites +/// another attempt it settles the walk. An auth rejection arrives as [`Error::Connect`] with +/// its status already consumed, so it is checked alongside the statuses. +fn verdict(err: &Error) -> Verdict { + if let Error::MoqNet(err) = err { + return Verdict::from(err); + } + let terminal = err.is_terminal(); + if err.is_auth() && terminal { + return Verdict::Refused; + } + let settled = err.is_auth() || err .status() - .is_some_and(|status| !crate::error::status_retryable(status)) + .is_some_and(|status| !crate::error::status_retryable(status)); + match (settled, terminal) { + (true, true) => Verdict::Fatal, + (true, false) => Verdict::Settled, + (false, true) => Verdict::Unusable, + (false, false) => Verdict::Retry, + } } /// Exponential backoff configuration for reconnection attempts. /// /// Three knobs, each with one job: `initial` and `multiplier` set the wait before /// each retry, `max` caps it, and `healthy` decides which sessions count as failed -/// attempts. The delays carry jitter, so a fleet knocked offline together does not -/// reconnect in lockstep. The timeout bounds every failure; a settled rejection, -/// malformed protocol input, or invalid local configuration stops retrying. A GOAWAY -/// naming no new URI is no failure: a restarting peer sends one to each session until -/// it goes, and each of those serves, so it restarts the timeout instead. -/// -/// A GOAWAY off a healthy session migrates at once, unless it sends the connection back -/// to an endpoint that already redirected it elsewhere and the session lived less than -/// `max`. Two peers redirecting to each other thus wait out this backoff and count -/// against the timeout. +/// attempts. See [`moq_net::reconnect::Backoff`], which this resolves to, for how +/// the policy applies them. #[derive(Clone, Debug, usage_rs::Args, serde::Serialize, serde::Deserialize)] #[usage(unknown_flags = "error", args_override_self = false)] #[serde(default, deny_unknown_fields)] @@ -175,7 +155,7 @@ impl Default for Backoff { fn default() -> Self { Self { initial: None, - multiplier: DEFAULT_MULTIPLIER, + multiplier: moq_net::reconnect::Backoff::default().multiplier, max: None, healthy: None, timeout: None, @@ -190,29 +170,42 @@ impl Default for Backoff { impl Backoff { /// The configured initial retry delay, or its default; zero retries at once. pub fn initial(&self) -> Duration { - self.initial.map(Into::into).unwrap_or(DEFAULT_INITIAL) + self.resolve().initial } /// The configured retry-delay cap, or its default; zero leaves the delay uncapped. pub fn max(&self) -> Duration { - self.max.map(Into::into).unwrap_or(DEFAULT_MAX) + self.resolve().max } /// The retry deadline window, with zero meaning unlimited and unset using the default. pub fn timeout(&self) -> Duration { - self.timeout.map(Into::into).unwrap_or(DEFAULT_TIMEOUT) - } - - /// How long a session must last to count as healthy. - fn healthy(&self) -> Duration { - self.healthy.map(Into::into).unwrap_or(DEFAULT_HEALTHY) + self.resolve().timeout + } + + /// The policy these knobs configure, with each unset one at its default. + pub(crate) fn resolve(&self) -> moq_net::reconnect::Backoff { + let mut backoff = moq_net::reconnect::Backoff::default(); + backoff.multiplier = self.multiplier; + let knobs = [ + (&mut backoff.initial, self.initial), + (&mut backoff.max, self.max), + (&mut backoff.healthy, self.healthy), + (&mut backoff.timeout, self.timeout), + ]; + for (knob, set) in knobs { + if let Some(set) = set { + *knob = set.into(); + } + } + backoff } /// Refuse a zero healthy threshold, which would count every session as healthy and /// reset the timeout forever under a peer that drops each one on sight. #[cfg(feature = "_transport")] pub(crate) fn validate(&self) -> crate::Result<()> { - if self.healthy().is_zero() { + if self.resolve().healthy.is_zero() { return Err(Error::BackoffHealthy); } Ok(()) @@ -265,24 +258,15 @@ pub enum Status { Migrating, } -/// What to do with the URI a peer names in its GOAWAY. -/// -/// The URI is dialed exactly as given, so it must carry whatever credentials the -/// new endpoint needs. Nothing from the current session is copied onto it. +/// What to do with the URI a peer names in its GOAWAY, as `--connect-goaway-redirect` +/// spells it. See [`moq_net::reconnect::Redirect`] for what each one risks. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, usage_rs::ValueEnum, serde::Serialize, serde::Deserialize)] #[serde(rename_all = "kebab-case")] #[non_exhaustive] pub enum Redirect { /// Follow any URI that does not downgrade the scheme, letting the peer name the - /// host as well as the port. - /// - /// The host is dialed as written, so this hands the peer whatever this process - /// can reach: a name it controls can carry an A record for loopback, a private - /// range, or the metadata service, and nothing here resolves it to find out. A - /// *literal* loopback, private, link-local, or IPC target is still refused, but - /// that catches a misconfigured peer, not a hostile one. Use it only with an - /// upstream trusted with the local network and with the tokens on - /// [`Connection::auth`], which every session presents, on any host. + /// host as well as the port. Use it only with an upstream trusted with the local + /// network and with the tokens on [`Connection::auth`]. Follow, /// Follow only when the host matches the configured URL, so a peer can move us /// between ports or schemes but not to another host. The default. @@ -292,132 +276,16 @@ pub enum Redirect { Ignore, } -/// What to do when a peer's GOAWAY names no URI, meaning it is restarting where we reached it. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] -#[non_exhaustive] -pub enum Restart { - /// Redial the same addresses at once while the drained session keeps serving. Right for - /// an address that can lead to another node, such as a fleet name a fresh resolve moves - /// off the drained one. The default. - #[default] - Redial, - /// Keep serving on the drained session until the peer closes it, then reconnect as after - /// any drop. Right for an address that names one node, where a redial only reaches the - /// draining process again. The peer must close the session itself, as a relay does at - /// its drain deadline. - Wait, -} - -impl Redirect { - /// The URL a GOAWAY assigns. `Ok(None)` keeps the current address list (the - /// peer named no URI, or the policy ignores a URI it could parse), `Ok(Some)` - /// replaces it, and `Err` is an explicit URI this policy refuses. A malformed - /// URI is refused even under [`Self::Ignore`]. - /// - /// `pinned` is a certificate pin on the connection, which can only verify the - /// host it was configured for, so it refuses a host change even under - /// [`Self::Follow`]. - fn target(&self, uri: &str, current: &Url, pinned: bool) -> crate::Result> { - if uri.is_empty() { - return Ok(None); - } - - // The URI can carry credentials, so the error names the reason, never the URI. - // Parse before `Ignore`: a malformed redirect is terminal even when the policy - // would otherwise stay on the current address list. - let refuse = |reason: &str| Error::RefusedRedirect(reason.to_string()); - - let target = uri.parse::().map_err(|_| refuse("the GOAWAY URI is malformed"))?; - if matches!(self, Self::Ignore) { - return Ok(None); - } - - if scheme_tier(target.scheme()) < scheme_tier(current.scheme()) { - return Err(refuse("the GOAWAY redirect downgrades the scheme")); - } - - // Only as far as the URL itself says: a name is dialed, never resolved here, - // so this refuses a peer that names a local address outright and says - // nothing about one that hides the same address behind a hostname. That gap - // is why [`Self::SameHost`] is the default; see [`is_local`]. - if is_local(&target) && !is_local(current) { - return Err(refuse("the GOAWAY redirect widens reachability to a local address")); - } - - // Host only, not the full authority: the port is what a peer legitimately - // moves us across when it hands off to a sibling process on the same box. - let same_host = target.host_str() == current.host_str(); - if matches!(self, Self::SameHost) && !same_host { - return Err(refuse("the GOAWAY redirect leaves the current host")); - } - if pinned && !same_host { - return Err(refuse("the GOAWAY redirect leaves the host a certificate pin verifies")); +impl From for moq_net::reconnect::Redirect { + fn from(redirect: Redirect) -> Self { + match redirect { + Redirect::Follow => Self::Follow, + Redirect::SameHost => Self::SameHost, + Redirect::Ignore => Self::Ignore, } - - Ok(Some(target)) - } -} - -/// Whether this scheme's dial installs the Rustls verifier, so a configured -/// fingerprint actually checked the peer. Plain and non-Rustls transports do not. -fn fingerprint_pins(scheme: &str) -> bool { - matches!(scheme, "https" | "wss" | "moqt" | "moql" | "tls") -} - -/// Rank a scheme so a peer-supplied redirect cannot silently drop encryption. -/// Unknown schemes rank lowest, so a forgotten classification is refused. -fn scheme_tier(scheme: &str) -> u8 { - match scheme { - "https" | "moqt" | "moql" | "wss" | "tls" => 2, - "tcp" | "ws" | "http" => 1, - // `unix` lands here deliberately: local IPC is not an upgrade over a - // network transport, it is a different reachability class (see `is_local`). - _ => 0, } } -/// Whether a URL *says* it names something only reachable from this host or -/// network. -/// -/// A judgement about the URL, not about where a dial would land. Literals and -/// `localhost` are decided here; every other name is left alone, since the -/// address behind it is chosen by whoever runs its DNS and can change between -/// this call and the dial. So a `false` here means "not local on its face", never -/// "verified remote". -fn is_local(url: &Url) -> bool { - match url.host() { - // A non-special scheme (`moqt://127.0.0.1`, and every other scheme this - // crate dials but `http`/`https`/`ws`/`wss`) parses its host as a domain - // even when it is an address literal, so the literal check can't be left to - // the URL parser. `crate::resolve::Candidates` handles the same thing. - Some(url::Host::Domain(host)) => match host.parse::() { - Ok(ip) => is_local_ip(ip), - Err(_) => host == "localhost" || host.ends_with(".localhost"), - }, - Some(url::Host::Ipv4(ip)) => is_local_ip(ip.into()), - Some(url::Host::Ipv6(ip)) => is_local_ip(ip.into()), - // No host at all, e.g. a `unix:` socket path. - None => true, - } -} - -/// Whether an address literal is one only this host or network can reach. -fn is_local_ip(ip: std::net::IpAddr) -> bool { - match ip { - std::net::IpAddr::V4(ip) => is_local_v4(ip), - // An IPv4-mapped address reaches the same host as the v4 it wraps, so judge - // it by that rather than by the v6 rules. - std::net::IpAddr::V6(ip) => match ip.to_ipv4_mapped() { - Some(v4) => is_local_v4(v4), - None => ip.is_loopback() || ip.is_unspecified() || ip.is_unique_local() || ip.is_unicast_link_local(), - }, - } -} - -fn is_local_v4(ip: std::net::Ipv4Addr) -> bool { - ip.is_loopback() || ip.is_private() || ip.is_link_local() || ip.is_unspecified() -} - /// How a reconnect loop reacts to a peer's GOAWAY. #[derive(Clone, Debug, usage_rs::Args, serde::Serialize, serde::Deserialize)] #[usage(unknown_flags = "error", args_override_self = false)] @@ -483,57 +351,6 @@ impl Default for Goaway { } } -/// Defaults for the [`Backoff`] knobs, applied by its accessors when a field is unset. -const DEFAULT_INITIAL: Duration = Duration::from_secs(1); -const DEFAULT_MULTIPLIER: u32 = 2; -const DEFAULT_MAX: Duration = Duration::from_secs(5); -const DEFAULT_HEALTHY: Duration = Duration::from_secs(1); -const DEFAULT_TIMEOUT: Duration = Duration::from_secs(10); - -/// The retry pacing a connection actually runs on, resolved once from [`Backoff`]. -#[derive(Clone, Copy)] -struct Pacing { - /// The first delay, already under the cap. - initial: Duration, - /// The delay cap, or `None` for uncapped. - max: Option, - /// At least 1: a multiplier of zero means no growth, not a delay that shrinks to nothing. - multiplier: u32, - /// The bar a session must clear to count as healthy. Never zero. - healthy: Duration, -} - -impl Pacing { - fn new(backoff: &Backoff) -> Self { - let max = Some(backoff.max()).filter(|max| !max.is_zero()); - let initial = backoff.initial(); - Self { - initial: max.map_or(initial, |max| initial.min(max)), - max, - multiplier: backoff.multiplier.max(1), - healthy: backoff.healthy(), - } - } - - /// The delay for the attempt after one that waited `delay`. - fn next(&self, delay: Duration) -> Duration { - let next = delay.saturating_mul(self.multiplier); - self.max.map_or(next, |max| next.min(max)) - } - - /// How long a session must last to start a new run of redirects. - /// - /// One that outlived the capped delay already redials no faster than the backoff would - /// allow. Uncapped, the healthy bar is the only rate the config names. - fn settled(&self) -> Duration { - self.max.unwrap_or(self.healthy) - } -} - -/// Default handover window, and the ceiling a GOAWAY deadline is capped to when -/// the config names none. -const DEFAULT_HANDOVER: Duration = Duration::from_secs(10); - impl Goaway { pub(crate) fn deprecated(&self) -> crate::cli::Deprecated { let mut found = crate::cli::Deprecated::default(); @@ -554,26 +371,16 @@ impl Goaway { found } - fn resolve(&self) -> Resolved { - Resolved { - redirect: self.redirect, - handover: self.handover.map(Into::into).unwrap_or(DEFAULT_HANDOVER), + /// The policy these knobs configure, with each unset one at its default. + pub(crate) fn resolve(&self) -> moq_net::reconnect::Goaway { + let mut goaway = moq_net::reconnect::Goaway::default(); + goaway.redirect = self.redirect.into(); + if let Some(handover) = self.handover { + goaway.handover = handover.into(); } + goaway } } - -#[derive(Clone, Copy)] -struct Resolved { - redirect: Redirect, - handover: Duration, -} - -impl Resolved { - fn handover(self, timeout: Option) -> Duration { - std::cmp::min(self.handover, timeout.unwrap_or(Duration::MAX)) - } -} - /// Shared reconnect state, observed by consumers through a [`kio`] channel. /// /// The channel closing (all producers dropped) is the terminal signal; `error` @@ -606,7 +413,7 @@ struct State { /// The currently-connected session, or `None` while reconnecting. Read by /// [`Monitor`] to snapshot live connection stats. session: Option, - /// The loop's [`Draining`] predecessor and its handover deadline, so + /// The policy's draining predecessor and its handover deadline, so /// [`Connection::close`] drains it too without overstaying that window. predecessor: Option<(moq_net::Session, kio::clock::Instant)>, } @@ -673,12 +480,19 @@ impl Shared { } } - /// Nothing is live. Clears the session so a handle can't be handed a closed one, - /// and drops the estimates that belonged to it. - fn disconnected(&self) { - self.offline(None); + /// Mirror the policy's draining predecessor, for [`Connection::close`]. + fn predecessor(&self, predecessor: Option<(&moq_net::Session, kio::clock::Instant)>) { + // A write wakes every watcher, so skip one that changes nothing. + if predecessor.is_none() && self.state.read().predecessor.is_none() { + return; + } + if let Ok(mut state) = self.state.write() { + state.predecessor = predecessor.map(|(session, deadline)| (session.clone(), deadline)); + } } + /// Nothing is live. Clears the session so a handle can't be handed a closed one, + /// and drops the estimates that belonged to it. fn offline(&self, failure: Option<&Error>) { if let Ok(mut state) = self.state.write() { state.offline(); @@ -717,6 +531,10 @@ impl Shared { return; } state.offline(); + // Including when the task was aborted mid-handover: nothing enforces the + // predecessor's deadline anymore, so a parked clone would hold its transport open. + // A graceful close took its own clone before aborting. + state.predecessor = None; state.ended = true; state.failure = None; state.error = err; @@ -1011,7 +829,7 @@ impl Connection { let task_running = std::sync::Arc::downgrade(&running); let task = crate::runtime::spawn(async move { let shared = &running.shared; - let reconnect = client.reconnect; + let reconnect = !client.policy.once; // Isolated inside `shared`, so a panic still records a terminal error // instead of closing the channels as if the caller had stopped it. let res = crate::panic::isolate(Self::run(shared, client, addrs)) @@ -1117,360 +935,134 @@ impl Connection { Ok(session.and(predecessor)?) } + /// Drive moq-net's policy: dial each attempt it asks for and publish what it does. async fn run(shared: &Shared, client: Client, addrs: Addrs) -> crate::Result<()> { #[cfg(feature = "quic")] let mut client = client; + let mut policy = moq_net::Reconnect::new(client.policy, addrs.as_slice().to_vec(), kio::clock::now()) + .map_err(|_| Error::BackoffHealthy)? + .with_tokens(shared.tokens.clone(), client.versions.clone()); + // The dial in flight, and the address it was asked for. It owns its client, so a + // fresh port can replace the loop's between walks. + let mut dial: Option<(url::Url, std::pin::Pin>)> = None; + // Mirrors the live session's estimates while it serves. + let mut forward: Option = None; + let mut timer: Option = None; + // Consecutive walks that failed since the last session, and whether a dial failed + // since then, which tells a failed walk from a session that ended too soon. #[cfg(feature = "quic")] - let mut failed_dials = 0; - let backoff = client.backoff.clone(); - let goaway = client.goaway.resolve(); - let pacing = Pacing::new(&backoff); - let timeout = backoff.timeout(); - let mut retry = Retry::new(pacing.initial); - // Sticky across migrations: a redirect is an assignment, not a detour, so a - // later drop redials wherever we were last sent. Scoped to this loop, so a - // fresh Connection starts from the configured addresses again. - let mut addrs = addrs; - // An old session kept alive after a GOAWAY so its in-flight groups finish. - let mut draining: Option = None; - // The current run of migrations, to tell a redirect loop from a healthy handover. - let mut redirects = Redirects::default(); + let (mut failed_walks, mut dial_failed) = (0, false); loop { - if !timeout.is_zero() && retry.start.elapsed() >= timeout { - return Err(timeout_error(timeout, retry.last_error.as_ref())); - } - // An ended auth handle refuses every later session, so no dial can recover. A fresh - // watch only reports an error once its handle has ended. - if let Poll::Ready(Err(err)) = shared.auth.grant().poll_changed(&kio::Waiter::noop()) { - return Err(err.into()); - } - - // What the last dial presented, so a refusal is judged against it. - let mut presented = None; - // Retry one transient failure on the same socket. Two failed walks are enough - // to try another flow. Only a retry window that allows a third walk gets here: - // a black-holed handshake outlasts the default window after the first walk. - #[cfg(feature = "quic")] - if failed_dials >= 2 { - match client.fresh_port() { - Ok(()) => failed_dials = 0, - // The old flow may have recovered, so dial it anyway. The count stays - // at the threshold, so the next walk retries the allocation. - Err(err) => tracing::warn!(%err, "no fresh UDP port; redialing from the old one"), - } - } - match Self::dial_any(shared, &client, &addrs, &mut draining, &mut presented).await { - Ok((addr, dialed)) => { - #[cfg(feature = "quic")] + let action = kio::wait(|waiter| { + loop { + if let Some((_, attempt)) = dial.as_mut() + && let Poll::Ready(res) = waiter.poll_future(attempt.as_mut()) { - failed_dials = 0; - } - let url = addr.url().clone(); - tracing::info!(peer = %Endpoint(&url), transport = %dialed.transport, "connected"); - shared.connected(&dialed.session, dialed.transport); - shared.tokens.connected(&dialed.session); - let session = dialed.session; - - let connected = kio::clock::now(); - // Wait for the session to end, forwarding its bandwidth estimates into the - // persistent producers meanwhile so consumers track the live stats across the - // connection, and draining any predecessor left over from a migration. - let ended = run_session(shared, &session, &mut draining, client.restart).await; - // A refusal is terminal unless the tokens changed since this dial read them. - let refreshed = match &ended { - Ended::Closed(Err(err)) => refreshed(err, &shared.tokens, presented.as_ref()), - _ => false, - }; - - let tenure = connected.elapsed(); - - // The connected target owns the policy, including in one-shot mode. A - // refused redirect is terminal: the peer is leaving and named somewhere we - // won't go, so redialing the old address or a fallback would ignore it. - let assigned = match &ended { - // A fingerprint only checked the dial that installed the Rustls - // verifier. tcp, unix, and plaintext WebSocket never consult it. - Ended::Goaway(msg) => { - goaway - .redirect - .target(msg.uri(), &url, client.pinned && fingerprint_pins(url.scheme()))? + let (url, _) = dial.take().expect("polled above"); + match res { + Ok(crate::client::Dialed { session, transport }) => { + tracing::info!(peer = %Endpoint(&url), %transport, "connected"); + #[cfg(feature = "quic")] + { + (failed_walks, dial_failed) = (0, false); + } + shared.connected(&session, transport); + forward = Some(Forward::new(&session)); + policy.connected(kio::clock::now(), session); + } + Err(err) => { + #[cfg(feature = "quic")] + { + dial_failed = true; + } + // An explicit shutdown cannot recover by retrying or opening another socket. + #[cfg(feature = "quic")] + let verdict = match quic_scheme(&url) && client.quic_closed() { + true => Verdict::Fatal, + false => verdict(&err), + }; + #[cfg(not(feature = "quic"))] + let verdict = verdict(&err); + policy.failed(kio::clock::now(), err, verdict); + } } - Ended::Closed(_) => None, - }; - if assigned.is_some() && addr.addresses().is_some() { - return Err(Error::PinnedRedirect); - } - - // One-shot mode leaves rather than migrating: there is no replacement to - // dial, and a GOAWAY naming no deadline never force-closes, so the peer - // stops accepting requests and waits for us. Ignoring it is how both - // sides end up waiting on each other forever. Checked before the - // migration branch below, which would otherwise loop back to redial. - if !client.reconnect && matches!(ended, Ended::Goaway(_)) { - return Ok(()); - } - - // Two peers that each hold a session past the healthy bar can still bounce us between - // them forever. Returning to an endpoint that already sent us elsewhere in this - // run of migrations skips the healthy-session reset, so the backoff escalates and - // the retry window still applies. A one-way rolling restart never revisits one, - // and a GOAWAY back to the same place is a restart, not a move, so it never - // counts. A session that outlived the capped delay already redials no faster - // than the backoff would allow, so it starts a new run: rolling restarts days - // apart never add up to a loop. - if matches!(ended, Ended::Closed(_)) || tenure >= pacing.settled() { - redirects.reset(); } - // Only an accepted redirect is a move: an empty or ignored one reconnects in place. - let looped = assigned.as_ref().is_some_and(|target| { - let from = endpoint(&url); - let to = endpoint(target); - to != from && redirects.depart(from, &to) - }); - - // A session that stayed up past the healthy bar is healthy; one that - // ended sooner, or carried on a redirect loop, counts as a failed attempt. - let healthy = !looped && retry.settle(tenure, &pacing); - - if let Ended::Goaway(msg) = &ended { - // An accepted redirect is an assignment: keep dialing it from here on, and - // only it. The peer named exactly one place to go, which retires - // whatever other addresses got us to this session. - let url = if let Some(target) = assigned { - addrs = Addrs::new(target.clone()); - target - } else { - url - }; - - // Hand over gracefully however the backoff bookkeeping scores this - // session. The old one keeps serving until it closes or overstays, - // so its routes stay attached and live tracks splice onto the - // replacement at a group boundary. Tearing it down here instead - // would drop every group published until the replacement caught up. - tracing::info!(peer = %Endpoint(&url), "upstream GOAWAY; migrating"); - // Retire any predecessor first: overwriting would drop its deadline - // on the floor and leave it holding the connection open. - if let Some(mut old) = draining.take() { - old.retire(); - } - draining = Some(Draining::new(session, goaway.handover(msg.timeout()), &shared.state)); - // After the predecessor is published, so a close woken by this - // status finds it and honors its handover deadline. - shared.migrating(); - - // No backoff sleep: a handover off a healthy session is not a failure. - if healthy { - continue; - } - - // Drained almost immediately, or looped back. The backoff still escalates, so - // the redials cannot hammer the peer, and the old session serves across the - // sleep. - // - // An empty URI is "reconnect to me": the peer is restarting and drains - // each new session until it goes, which is no failure, since every one - // serves. A named URI is a redirect, scored as a failed attempt so two - // peers bouncing us between them eventually give up. - let restarting = msg.uri().is_empty(); - if !restarting { - let reason = if looped { - "redirect loop" - } else { - "peer redirected immediately" - }; - let err = Error::Reconnect(reason.to_string()); - shared.retrying(&err); - retry.last_error = Some(err); - } - // A restart's pacing is not capped by the window: a delay longer than the - // window would otherwise spend it all and end the loop without a redial. - let window = if restarting { Duration::ZERO } else { timeout }; - let Some(wait) = retry_wait(retry.delay, retry.start, window) else { - return Err(timeout_error(timeout, retry.last_error.as_ref())); - }; - if looped { - tracing::warn!(peer = %Endpoint(&url), ?wait, "peer redirected back in a loop; retrying after backoff"); - } else { - tracing::warn!(peer = %Endpoint(&url), ?wait, "peer drained the session immediately; retrying after backoff"); - } - // Keep the handover bounded across the sleep: nothing else polls the - // predecessor while the loop is between connections. - sleep_draining(wait, &mut draining, shared).await; - retry.delay = pacing.next(retry.delay); - if restarting { - // After the sleep, so the window counts only the time after the - // peer stops answering. - retry.start = kio::clock::now(); - } - continue; + if let (Some(forward), Some(session)) = (forward.as_mut(), policy.session()) { + forward.poll(shared, session, waiter); } - // Both returns below leave the offline bookkeeping to the terminal - // snapshot, so they never publish a `Disconnected` first. - // - // Refusals and malformed protocol input are settled failures, however long - // the session lived. A new connection cannot repair either one. - if let Ended::Closed(Err(err)) = &ended - && err.is_terminal() - && !refreshed - { - return Err(err.clone()); + let at = match policy.poll(kio::clock::now(), waiter) { + Action::Wait(Some(at)) => at, + Action::Wait(None) => return Poll::Pending, + action => return Poll::Ready(action), + }; + let sleep = timer.get_or_insert_with(|| kio::clock::sleep_until(at)); + if sleep.deadline() != at { + sleep.reset(at); } - - // One-shot mode: the session ending ends the connection. Its close - // reason is the terminal error, mirroring `moq_net::Session::closed`. - // A GOAWAY already returned above. - if !client.reconnect { - return match ended { - Ended::Closed(res) => res, - Ended::Goaway(_) => unreachable!("handled before the migration branch"), - }; + if waiter.poll_future(std::pin::Pin::new(sleep)).is_pending() { + return Poll::Pending; } + } + }) + .await; - // Publish the lost transport and why, in one lifecycle observation. - let failure = match &ended { - Ended::Closed(Err(err)) => Some(err), - _ => None, - }; - shared.offline(failure); - - if healthy { - // The backoff window was reset, so a one-off drop reconnects promptly. - tracing::warn!(peer = %Endpoint(&url), "session closed, reconnecting"); - } else { - // Connected then dropped almost immediately (e.g. the server accepts then - // resets, or redirects us straight back out). Treat it as a failed - // connection: keep the reason so the give-up timeout reports a real cause, - // and fall through to the shared backoff sleep below so repeated flaps - // escalate instead of spinning the CPU. - let err = match ended { - Ended::Closed(Err(err)) => Some(err), - Ended::Closed(Ok(())) => None, - // Handled above: a GOAWAY never reaches here. - Ended::Goaway(_) => None, - }; - // NOTE: only UNAUTHORIZED is specified, and it is handled above. Any - // other MoQ-layer rejection (Request::reject after the transport is - // accepted) lands here as an untyped transport close, so it cannot be - // told apart from a network blip and is retried until the give-up - // timeout. Classifying the rest needs the transport to surface the - // close code; until then, one-shot mode (`reconnect = false`) is how - // a caller observes those rejections directly. - match err { - Some(err) => { - tracing::warn!(peer = %Endpoint(&url), %err, "session severed immediately, retrying"); - retry.last_error = Some(err); - } - None => tracing::warn!(peer = %Endpoint(&url), "session severed immediately, retrying"), - } - } + match action { + Action::Dial(attempt) => { + let url = attempt.target().url().clone(); + let client = client.clone(); + dial = Some((url, Box::pin(async move { client.dial(attempt).await }))); } - Err(err) => { - if (err.is_terminal() && !refreshed(&err, &shared.tokens, presented.as_ref())) || !client.reconnect - { - return Err(err); - } + // The policy already counted the timeout; dropping the dial cancels it. + Action::Abandon => { #[cfg(feature = "quic")] - if addrs - .as_slice() - .iter() - .any(|addr| matches!(addr.url().scheme(), "http" | "https" | "moqt" | "moql")) { - // Explicit shutdown cannot recover by retrying or opening another socket. - if client.quic_closed() { - return Err(err); - } - failed_dials += 1; + dial_failed = true; } - shared.retrying(&err); - retry.last_error = Some(err); + dial = None; } - } - - let Some(wait) = retry_wait(retry.delay, retry.start, timeout) else { - return Err(timeout_error(timeout, retry.last_error.as_ref())); - }; - // No URL here: with several candidates there isn't one to name, and each - // attempt already logged the address it tried. - tracing::warn!(?wait, "reconnecting after backoff"); - // Drain-aware: a GOAWAY off a healthy session continues straight to the - // replacement dial, so a predecessor can still be draining when that dial - // fails and lands here. A plain sleep would stop enforcing its handover - // until some later dial succeeded, holding an upstream that asked to drain - // open for the whole retry window, or forever with `--connect-backoff-timeout=0`. - sleep_draining(wait, &mut draining, shared).await; - retry.delay = pacing.next(retry.delay); - } - } - - /// Try each address in turn, returning the first session that connects and the - /// address that produced it. - /// - /// A peer discovered rather than configured can advertise several addresses, - /// only some of which route from here (its loopback, a container bridge, an - /// interface on another subnet). Nothing in the record says which, so every - /// attempt walks the whole list rather than pinning whichever sorted first. - /// - /// A predecessor left over from a migration keeps draining across the whole - /// walk, not just one attempt: its handover deadline is wall-clock, and a walk - /// past several black-holed addresses is exactly when it would overstay. - async fn dial_any( - shared: &Shared, - client: &Client, - addrs: &Addrs, - draining: &mut Option, - presented: &mut Option, - ) -> crate::Result<(crate::connect::Addr, crate::client::Dialed)> { - let candidates = addrs.as_slice(); - let mut last = None; - let mut transient = None; - - for (index, addr) in candidates.iter().enumerate() { - let url = addr.url(); - tracing::info!(peer = %Endpoint(url), "connecting"); - - // Held before the dial starts its session's driver, so a request the session - // sends early waits on their answers rather than being refused outside the - // connection credential's grant. - let presenting = presented.insert(shared.tokens.dial(url, &client.versions)); - let mut dial = std::pin::pin!(client.dial(addr.clone(), presenting)); - let dialed = kio::wait(|waiter| { - if poll_draining(draining, waiter) { - shared.disconnected(); + Action::Migrating => { + forward = None; + // After the predecessor is published, so a close woken by this status + // finds it and honors its handover deadline. + shared.predecessor(policy.predecessor()); + shared.migrating(); } - waiter.poll_future(dial.as_mut()) - }); - - let dialed = match attempt_timeout(index, candidates.len()) { - None => dialed.await, - Some(limit) => match kio::clock::timeout(limit, dialed).await { - Ok(dialed) => dialed, - Err(_) => Err(Error::Reconnect(format!("timed out connecting to {}", Endpoint(url)))), - }, - }; - - match dialed { - Ok(dialed) => return Ok((addr.clone(), dialed)), - // A status the peer actually sent is its answer, not this address's, so - // unless it invites another attempt it settles the whole walk. Carrying - // on would offer the same rejected credentials at the peer's other - // addresses, and worse, a later transport failure would overwrite the - // real reason and leave the outer loop retrying until its budget ran out. - Err(err) if settles_walk(&err) => return Err(err), - Err(err) => { - tracing::debug!(peer = %Endpoint(url), %err, "address unreachable"); - if !err.is_terminal() { - transient = Some(err.clone()); + Action::Retired => shared.predecessor(None), + Action::Offline(failure) => { + forward = None; + shared.offline(failure.as_ref()); + } + Action::Retrying(err) => { + shared.retrying(&err); + // A failed walk of a QUIC peer: retry one transient failure on the same + // socket, then try another flow before the next walk. Only a retry window + // that allows a third walk gets there: a black-holed handshake outlasts the + // default window after the first walk. + #[cfg(feature = "quic")] + if std::mem::take(&mut dial_failed) && policy.targets().iter().any(|addr| quic_scheme(addr.url())) { + failed_walks += 1; + if failed_walks >= 2 { + match client.fresh_port() { + Ok(()) => failed_walks = 0, + // The old flow may have recovered, so dial it anyway. The count + // stays at the threshold, so the next failed walk retries the + // allocation. + Err(err) => tracing::warn!(%err, "no fresh UDP port; redialing from the old one"), + } + } } - last = Some(err); } + // The terminal snapshot does the offline bookkeeping, so a watcher never sees a + // `Disconnected` the loop will not redial from. + Action::Done(res) => { + shared.predecessor(None); + return res; + } + Action::Wait(_) => unreachable!("handled while polling"), } } - - // One unusable candidate must not turn another candidate's transient failure terminal. - Err(transient.or(last).expect("Addrs is never empty")) } /// Poll until a session is established (the first connect, or the current one). @@ -1650,296 +1242,37 @@ impl Connection { } } -/// The reconnect loop's backoff window across a run of failed attempts. -struct Retry { - /// The wait before the next attempt, before jitter. - delay: Duration, - /// When this run of failures began; the give-up timeout counts from here. - start: kio::clock::Instant, - /// The latest failure, so giving up reports a real cause. - last_error: Option, -} - -impl Retry { - fn new(initial: Duration) -> Self { - Self { - delay: initial, - start: kio::clock::now(), - last_error: None, - } - } - - /// Score how long a session served: past the healthy bar starts a fresh retry window. - fn settle(&mut self, tenure: Duration, pacing: &Pacing) -> bool { - let healthy = tenure >= pacing.healthy; - if healthy { - *self = Self::new(pacing.initial); - } - healthy - } -} - -/// Build the terminal error for an exhausted retry window without discarding the last real cause. -fn timeout_error(timeout: Duration, last_error: Option<&Error>) -> Error { - let message = match last_error { - Some(err) => format!("reconnect timed out after {timeout:?}: {err}"), - None => format!("reconnect timed out after {timeout:?}"), - }; - Error::Reconnect(message) -} - -/// Jitter the next delay and cap it at the retry window. `None` means the window is exhausted. -fn retry_wait(delay: Duration, retry_start: kio::clock::Instant, timeout: Duration) -> Option { - let wait = delay.mul_f64(0.5 + kio::clock::rng().random::() / 2.0); - if timeout.is_zero() { - return Some(wait); - } - Some(wait.min(timeout.checked_sub(retry_start.elapsed())?)) -} - -/// Where `url` dials, without its userinfo, query, or fragment, where a peer may mint fresh -/// credentials on every redirect. Two URLs that differ only there reach the same endpoint. -fn endpoint(url: &Url) -> Url { - let mut bare = url.clone(); - // Only fails for a URL that cannot carry userinfo, which then has none to strip. - let _ = bare.set_username(""); - let _ = bare.set_password(None); - bare.set_query(None); - bare.set_fragment(None); - bare -} - -/// A run of GOAWAY redirects, and whether it has revisited an endpoint. -/// -/// The peer picks every hop, so remembering them all would grow without bound. Brent's cycle -/// detection instead keeps one departed endpoint, replaced at doubling intervals, and catches a -/// cycle of any length within a few laps of it. Once caught, the run stays a loop until reset. -struct Redirects { - mark: Option, - span: u64, - hops: u64, - looped: bool, -} - -impl Default for Redirects { - fn default() -> Self { - Self { - mark: None, - span: 1, - hops: 0, - looped: false, - } - } -} - -impl Redirects { - /// Record leaving `from` for a different endpoint `to`, and return whether this run loops. - fn depart(&mut self, from: Url, to: &Url) -> bool { - if self.mark.as_ref() == Some(to) { - self.looped = true; - } - self.hops += 1; - if self.hops == self.span { - self.mark = Some(from); - self.span = self.span.saturating_mul(2); - self.hops = 0; - } - self.looped - } +/// A dial in flight. +type Dial = dyn Future> + Send; - /// Start a new run. - fn reset(&mut self) { - *self = Self::default(); - } +/// Whether a dial to `url` runs over the QUIC endpoint, whose socket a fresh port replaces. +#[cfg(feature = "quic")] +fn quic_scheme(url: &url::Url) -> bool { + matches!(url.scheme(), "http" | "https" | "moqt" | "moql") } -/// Why a session stopped being the live one. -enum Ended { - /// The transport closed. - Closed(crate::Result<()>), - /// The peer sent a GOAWAY; the session is still up and serving. - Goaway(moq_net::goaway::Goaway), +/// Mirrors one live session's send and receive estimates into the persistent producers, so +/// [`Connection`] consumers track the live estimates across the connection. +struct Forward { + send: Option, + recv: RecvBandwidth, } -/// An old session kept alive after a GOAWAY so its in-flight groups finish. -/// -/// Held by the reconnect loop rather than a detached task, so dropping the -/// [`Connection`] handle tears the old session down with everything else instead -/// of leaving an orphan holding the connection open. -struct Draining { - session: moq_net::Session, - /// Resolves once the session closes, on its own or through [`Self::poll_idle`]. - closed: std::pin::Pin + Send>>, - deadline: std::pin::Pin>, - /// Whether [`Self::poll_idle`] started the graceful close. - closing: bool, - /// Mirrors the session into [`State::predecessor`] for as long as this lives. - state: kio::Producer, -} - -impl Draining { - fn new(session: moq_net::Session, handover: Duration, state: &kio::Producer) -> Self { - let deadline = Box::pin(kio::clock::sleep(handover)); - if let Ok(mut state) = state.write() { - state.predecessor = Some((session.clone(), deadline.deadline())); - } - let closed = { - let session = session.clone(); - Box::pin(async move { - session.closed().await; - }) as std::pin::Pin + Send>> - }; - +impl Forward { + fn new(session: &moq_net::Session) -> Self { Self { - session, - closed, - deadline, - closing: false, - state: state.clone(), - } - } - - /// Close the old session now, whatever remains of its window. - fn retire(&mut self) { - self.session.abort(moq_net::Error::GoawayTimeout); - } - - /// Close the old session gracefully once it carries no subscription or fetch. - /// - /// Only while a replacement serves: until then everything new still lands here. The - /// handover window still bounds the close. - fn poll_idle(&mut self, waiter: &kio::Waiter) { - if self.closing || self.session.poll_idle(waiter).is_pending() { - return; - } - tracing::debug!("old session carries nothing after GOAWAY; closing"); - self.closing = true; - let session = self.session.clone(); - self.closed = Box::pin(async move { - // A drain cut short by its own deadline still ends the session. - let _ = session.close().await; - }); - } - - /// Poll the drain; `true` once it is over and the handle should be released. - fn poll(&mut self, waiter: &kio::Waiter) -> bool { - if waiter.poll_future(self.closed.as_mut()).is_ready() { - tracing::debug!("old session drained cleanly after GOAWAY"); - return true; + send: session.send_bandwidth(), + recv: RecvBandwidth::default(), } - if waiter.poll_future(self.deadline.as_mut()).is_ready() { - tracing::warn!("old session did not drain in time; closing"); - self.session.abort(moq_net::Error::GoawayTimeout); - return true; - } - false } -} -impl Drop for Draining { - fn drop(&mut self) { - if let Ok(mut state) = self.state.write() { - state.predecessor = None; - } + /// A `poll_*` step: on return, `waiter` is registered for the next change. + fn poll(&mut self, shared: &Shared, session: &moq_net::Session, waiter: &kio::Waiter) { + poll_forward(&mut self.send, &shared.send_bw, waiter); + self.recv.poll(&shared.recv_bw, waiter, || session.recv_bandwidth()); } } -/// Poll a draining predecessor, retiring it once it closes or overstays its -/// handover window. `true` on the pass that retires it. -/// -/// A `poll_*` step: on return the waiter is registered for the next change. The -/// caller decides what the retirement means, because that depends on whether a -/// replacement is already serving. -fn poll_draining(draining: &mut Option, waiter: &kio::Waiter) -> bool { - if let Some(old) = draining.as_mut() - && old.poll(waiter) - { - *draining = None; - return true; - } - false -} - -/// Sleep for `delay` while keeping a draining predecessor's deadline enforced. -/// -/// The drain is otherwise only polled from [`run_session`], which is reached only -/// after a successful connect, so a replacement that takes several backoff rounds -/// to reach would leave the old session holding its connection well past the -/// handover window. -async fn sleep_draining(delay: Duration, draining: &mut Option, shared: &Shared) { - let mut sleep = std::pin::pin!(kio::clock::sleep(delay)); - kio::wait(|waiter| { - // Nothing is serving once it retires: the predecessor closed or overstayed - // its handover, and the replacement is still a dial away. Leaving the state - // on `Migrating` would keep handing callers a session that is already closed, - // with its stats and version, until the next connect overwrote it. - if poll_draining(draining, waiter) { - shared.disconnected(); - } - waiter.poll_future(sleep.as_mut()) - }) - .await -} - -/// Wait for `session` to close or receive a GOAWAY, forwarding its send/recv bandwidth estimates -/// into the persistent producers meanwhile so [`Connection`] consumers track the live estimates -/// across the connection, and closing any predecessor left over from an earlier migration once -/// it carries nothing. -/// -/// One `poll_*` step drives it all: [`poll_forward`] mirrors each kio bandwidth estimate, the -/// GOAWAY consumer is a kio channel, and the transport's close future (the one non-kio source) is -/// polled through the waiter's own waker. -/// -/// Under [`Restart::Wait`] a GOAWAY naming no URI does not end the wait: the session serves on -/// until it closes. -async fn run_session( - shared: &Shared, - session: &moq_net::Session, - draining: &mut Option, - restart: Restart, -) -> Ended { - let mut send = session.send_bandwidth(); - let mut recv = RecvBandwidth::default(); - // Dropped once a restart is waited out: it is a latch, so it would keep firing. - let mut goaway = Some(session.draining()); - let closed = session.closed(); - tokio::pin!(closed); - - kio::wait(|waiter| { - poll_forward(&mut send, &shared.send_bw, waiter); - recv.poll(&shared.recv_bw, waiter, || session.recv_bandwidth()); - - // The replacement serves, so a predecessor carrying nothing closes now. One that - // closes or overstays its handover window retires, and the state stays as it is. - if let Some(old) = draining.as_mut() { - old.poll_idle(waiter); - } - poll_draining(draining, waiter); - - // Checked before the close arm: a GOAWAY means the session is still up, and - // migrating from it is not the same as reconnecting after it died. - // - // Polled in one-shot mode too, even though there is no replacement to dial. - // A GOAWAY naming no deadline never force-closes, and the peer stops - // accepting requests and waits for us to leave, so ignoring it is how both - // sides end up waiting on each other forever. The caller ends the connection - // instead: asked to leave, with no way to migrate, leaving is the answer. - if let Some(Poll::Ready(Ok(msg))) = goaway.as_ref().map(|goaway| goaway.poll(waiter)) { - if restart == Restart::Wait && msg.uri().is_empty() { - tracing::info!("peer is restarting; serving until it closes the session"); - goaway = None; - } else { - return Poll::Ready(Ended::Goaway(msg)); - } - } - - waiter - .poll_future(closed.as_mut()) - .map(|err| Ended::Closed(Err(err.into()))) - }) - .await -} - /// Own the live session's receive consumer only while an external consumer needs it. #[derive(Default)] struct RecvBandwidth { @@ -2026,6 +1359,7 @@ fn terminal(state: &State) -> Error { #[cfg(test)] mod tests { use super::*; + use url::Url; /// A millisecond track: the frames written here carry timestamps. fn timed() -> moq_net::track::Info { @@ -2058,14 +1392,9 @@ mod tests { assert_eq!(parsed.backoff.healthy, Some((Duration::from_secs(2)).into())); assert_eq!(parsed.backoff.timeout(), Duration::ZERO, "0 means retry forever"); - // With no flags, the parser defaults resolve to the typed defaults. + // With no flags, the parser defaults resolve to the policy's defaults. let parsed = Wrapper::parse_from(&[]).unwrap(); - assert_eq!(parsed.backoff.multiplier, DEFAULT_MULTIPLIER); - let pacing = Pacing::new(&parsed.backoff); - assert_eq!(pacing.initial, DEFAULT_INITIAL); - assert_eq!(pacing.max, Some(DEFAULT_MAX)); - assert_eq!(pacing.healthy, DEFAULT_HEALTHY); - assert_eq!(parsed.backoff.timeout(), DEFAULT_TIMEOUT); + assert_eq!(parsed.backoff.resolve(), moq_net::reconnect::Backoff::default()); // And a flag still lands where the merge can see it. let parsed = Wrapper::parse_from(&[ @@ -2073,7 +1402,7 @@ mod tests { std::ffi::OsStr::new("3s"), ]) .unwrap(); - assert_eq!(Pacing::new(&parsed.backoff).initial, Duration::from_secs(3)); + assert_eq!(parsed.backoff.resolve().initial, Duration::from_secs(3)); } /// GOAWAY typed defaults remain overrideable by a standing TOML layer. @@ -2090,9 +1419,7 @@ mod tests { // No flags passed: the typed defaults are present. let parsed = Wrapper::parse_from(&[]).unwrap(); assert_eq!(parsed.goaway.redirect, Redirect::SameHost); - let resolved = parsed.goaway.resolve(); - assert_eq!(resolved.redirect, Redirect::SameHost); - assert_eq!(resolved.handover(None), Duration::from_secs(10)); + assert_eq!(parsed.goaway.resolve(), moq_net::reconnect::Goaway::default()); // Flags passed: they land where the merge can see them. let parsed = Wrapper::parse_from(&[ @@ -2103,323 +1430,9 @@ mod tests { ]) .unwrap(); assert_eq!(parsed.goaway.redirect, Redirect::Ignore); - assert_eq!(parsed.goaway.resolve().handover(None), Duration::from_secs(3)); - } - - /// Our own window caps the peer's. A GOAWAY deadline shortens the handover but - /// never extends it, so an upstream naming an absurd one cannot make us hold a - /// dying session open for it. - #[test] - fn handover_takes_the_earlier_deadline() { - let config = Goaway { - handover: Some((Duration::from_secs(10)).into()), - ..Default::default() - }; - - let config = config.resolve(); - assert_eq!(config.handover(None), Duration::from_secs(10), "no deadline: ours"); - assert_eq!( - config.handover(Some(Duration::from_secs(3))), - Duration::from_secs(3), - "a shorter peer deadline wins: it force-closes then anyway" - ); - assert_eq!( - config.handover(Some(Duration::from_secs(3600))), - Duration::from_secs(10), - "a longer peer deadline must not extend our cap" - ); - - // The default is a cap too, not just a fallback for a silent peer. - let default = Goaway::default().resolve(); - assert_eq!(default.handover(Some(Duration::from_secs(3600))), DEFAULT_HANDOVER); - } - - /// A peer must not be able to redirect us somewhere we could not already - /// reach. IPv4-mapped and link-local forms are the easy ones to miss. - #[test] - fn local_targets_are_recognized() { - let local = [ - "https://127.0.0.1/", - "https://localhost/", - "https://[::1]/", - "https://10.0.0.1/", - "https://169.254.1.1/", - "https://[::ffff:127.0.0.1]/", - "https://[::ffff:10.0.0.1]/", - "https://[fe80::1]/", - "https://[fc00::1]/", - "https://0.0.0.0/", - ]; - for url in local { - assert!(is_local(&url.parse().unwrap()), "{url} should be local"); - } - - for url in ["https://example.com/", "https://8.8.8.8/", "https://[2606:4700::1]/"] { - assert!(!is_local(&url.parse().unwrap()), "{url} should not be local"); - } - - // A public endpoint may not redirect us inward, but a local one may stay local. - let public: Url = "https://relay.example/".parse().unwrap(); - let localhost: Url = "https://127.0.0.1:4443/".parse().unwrap(); - assert!(matches!( - Redirect::Follow.target("https://[::ffff:127.0.0.1]/", &public, false), - Err(Error::RefusedRedirect(_)) - )); - assert_eq!( - Redirect::Follow - .target("https://127.0.0.1:9999/", &localhost, false) - .unwrap() - .unwrap() - .port(), - Some(9999) - ); - } - - /// The scheme ranking is what stops a peer from quietly moving us off an - /// encrypted transport. An unknown scheme ranks lowest so a classification we - /// forgot to add is refused rather than trusted. - #[test] - fn scheme_tiers_rank_encrypted_above_plaintext() { - for scheme in ["https", "moqt", "moql", "wss", "tls"] { - assert_eq!(scheme_tier(scheme), 2, "{scheme} is encrypted"); - } - for scheme in ["tcp", "ws", "http"] { - assert_eq!(scheme_tier(scheme), 1, "{scheme} is plaintext"); - } - // `unix` is not an upgrade over a network transport, it is a different - // reachability class, so it must not outrank one. - for scheme in ["unix", "gopher", ""] { - assert_eq!(scheme_tier(scheme), 0, "{scheme} is unclassified"); - } - } - - /// A redirect may hold the scheme or improve it, never weaken it. A downgrade - /// is a refusal, not a reason to redial the address the peer is leaving. - #[test] - fn target_refuses_a_scheme_downgrade() { - let secure: Url = "https://relay.example/".parse().unwrap(); - let plain: Url = "http://relay.example/".parse().unwrap(); - - assert!(matches!( - Redirect::Follow.target("http://other.example/", &secure, false), - Err(Error::RefusedRedirect(_)) - )); - assert!(matches!( - Redirect::Follow.target("unix:///tmp/moq.sock", &secure, false), - Err(Error::RefusedRedirect(_)) - )); - - // Same tier and upgrades are followed. - let same: Url = "https://other.example/".parse().unwrap(); - assert_eq!( - Redirect::Follow - .target("https://other.example/", &secure, false) - .unwrap(), - Some(same.clone()) - ); - assert_eq!( - Redirect::Follow - .target("https://other.example/", &plain, false) - .unwrap(), - Some(same) - ); - } - - /// Native schemes must classify address literals even when the URL parser returns a domain. - #[test] - fn a_non_special_scheme_classifies_its_literal_host() { - for url in [ - "moqt://127.0.0.1/", - "moql://10.0.0.1/", - "tcp://169.254.169.254/", - "moqt://[::1]/", - "moqt://[::ffff:127.0.0.1]/", - ] { - assert!(is_local(&url.parse().unwrap()), "{url} should be local"); - } - for url in ["moqt://8.8.8.8/", "moql://[2606:4700::1]/", "tcp://relay.example/"] { - assert!(!is_local(&url.parse().unwrap()), "{url} should not be local"); - } - - let public: Url = "moqt://relay.example/".parse().unwrap(); - assert!( - matches!( - Redirect::Follow.target("moqt://169.254.169.254/", &public, false), - Err(Error::RefusedRedirect(_)) - ), - "a literal local target is refused whatever the scheme" - ); - } - - /// The reachability guard reads the URL, so it cannot see a local address - /// hiding behind a hostname: any name the peer controls can carry an A record - /// for loopback or a private range, and a second lookup at dial time may answer - /// differently anyway. The default policy is what closes that, by refusing to - /// let a peer name a host at all. - #[test] - fn a_peer_named_host_is_refused_by_default() { - let public: Url = "https://relay.example/".parse().unwrap(); - - // The bypass: a name is not local on its face, so the reachability guard - // passes it through whatever it resolves to. - let rebindable: Url = "https://rebind.attacker.example/".parse().unwrap(); - assert!(!is_local(&rebindable), "a hostname is never classified as local"); - assert_eq!( - Redirect::Follow.target(rebindable.as_str(), &public, false).unwrap(), - Some(rebindable.clone()), - "Follow is explicit trust: it dials the name the peer chose" - ); - - // So the default refuses the host change instead of trying to judge it. - assert_eq!(Redirect::default(), Redirect::SameHost); - assert!( - matches!( - Redirect::default().target(rebindable.as_str(), &public, false), - Err(Error::RefusedRedirect(_)) - ), - "a peer-named host is refused without resolving it" - ); - assert_eq!( - Goaway::default().resolve().redirect, - Redirect::SameHost, - "and the shipped config carries that default" - ); - } - - /// Only an explicit URI can assign, and one the policy will not follow is an - /// error rather than a quiet fallback: the loop ends on it instead of redialing. - #[test] - fn only_an_accepted_redirect_replaces_the_address_list() { - let current: Url = "https://relay.example/".parse().unwrap(); - - // No URI, or a policy that ignores it: keep the current address list. - assert_eq!(Redirect::Follow.target("", ¤t, false).unwrap(), None); - assert_eq!( - Redirect::Ignore - .target("https://relay.example:5443/", ¤t, false) - .unwrap(), - None - ); - - // An explicit URI the policy will not follow is refused. - for (policy, uri) in [ - (Redirect::SameHost, "https://other.example/"), - (Redirect::Follow, "not a url"), - (Redirect::Ignore, "not a url"), - (Redirect::Follow, "http://relay.example/"), - (Redirect::Follow, "https://127.0.0.1/"), - ] { - assert!( - matches!(policy.target(uri, ¤t, false), Err(Error::RefusedRedirect(_))), - "{policy:?}: {uri}" - ); - } - - // An explicit assignment remains an assignment even if its URL is unchanged. - assert_eq!( - Redirect::SameHost.target(current.as_str(), ¤t, false).unwrap(), - Some(current.clone()) - ); - let moved = "https://relay.example:5443/"; - assert_eq!( - Redirect::SameHost.target(moved, ¤t, false).unwrap(), - Some(moved.parse().unwrap()) - ); - } - - /// A fingerprint is a Rustls check. Schemes that never install that verifier - /// must not inherit the pin. - #[test] - fn a_fingerprint_pin_only_covers_rustls_schemes() { - for scheme in ["https", "wss", "moqt", "moql", "tls"] { - assert!(fingerprint_pins(scheme), "{scheme}"); - } - for scheme in ["http", "ws", "tcp", "unix"] { - assert!(!fingerprint_pins(scheme), "{scheme}"); - } - } - - /// A certificate pin verifies only the host it was configured for, so it - /// refuses a host change even when the policy would follow one. - #[test] - fn a_certificate_pin_holds_the_host() { - let current: Url = "https://relay.example/".parse().unwrap(); - assert!(matches!( - Redirect::Follow.target("https://other.example/", ¤t, true), - Err(Error::RefusedRedirect(_)) - )); - let moved = "https://relay.example:5443/"; - assert_eq!( - Redirect::Follow.target(moved, ¤t, true).unwrap(), - Some(moved.parse().unwrap()) - ); - } - - /// `SameHost` lets a peer move us between ports or schemes on the endpoint we - /// already chose, but not onto a different host. - #[test] - fn same_host_follows_a_port_not_a_host() { - let current: Url = "https://relay.example:4443/".parse().unwrap(); - - assert!( - matches!( - Redirect::SameHost.target("https://elsewhere.example/", ¤t, false), - Err(Error::RefusedRedirect(_)) - ), - "another host is refused" - ); - - let moved = Redirect::SameHost - .target("https://relay.example:5443/", ¤t, false) - .unwrap() - .unwrap(); - assert_eq!( - moved.port(), - Some(5443), - "a different port on the same host is followed" - ); - } - - /// Each knob means one thing at zero: no wait, no cap, no growth. None of them touches - /// the healthy bar, which is what keeps an immediate close a failed attempt. - #[test] - fn zero_turns_each_pacing_knob_off() { - let pacing = Pacing::new(&Backoff { - initial: Some((Duration::ZERO).into()), - ..Default::default() - }); - assert_eq!(pacing.initial, Duration::ZERO); - assert_eq!(pacing.next(pacing.initial), Duration::ZERO, "zero initial never waits"); - assert_eq!(pacing.healthy, DEFAULT_HEALTHY); - - let pacing = Pacing::new(&Backoff { - max: Some((Duration::ZERO).into()), - ..Default::default() - }); - assert_eq!(pacing.max, None); - let mut delay = pacing.initial; - for _ in 0..10 { - delay = pacing.next(delay); - } - assert_eq!(delay, DEFAULT_INITIAL * 1024, "zero max leaves the delay uncapped"); - assert_eq!(pacing.settled(), DEFAULT_HEALTHY); - - let pacing = Pacing::new(&Backoff { - multiplier: 0, - ..Default::default() - }); - assert_eq!( - pacing.next(pacing.initial), - pacing.initial, - "zero multiplier never grows" - ); - - // A cap below the first delay caps it too. - let pacing = Pacing::new(&Backoff { - max: Some((Duration::from_millis(300)).into()), - ..Default::default() - }); - assert_eq!(pacing.initial, Duration::from_millis(300)); + let resolved = parsed.goaway.resolve(); + assert_eq!(resolved.redirect, moq_net::reconnect::Redirect::Ignore); + assert_eq!(resolved.handover, Duration::from_secs(3)); } /// A zero healthy bar would count every session healthy, so it is refused at build. @@ -2434,23 +1447,6 @@ mod tests { assert!(matches!(err, Error::BackoffHealthy), "{err}"); } - /// A sane config is left alone: the delay doubles up to the cap and stays there. - #[test] - fn pacing_grows_to_the_cap() { - let pacing = Pacing::new(&Backoff { - initial: Some((Duration::from_millis(100)).into()), - multiplier: 2, - max: Some((Duration::from_millis(400)).into()), - ..Default::default() - }); - - assert_eq!(pacing.initial, Duration::from_millis(100)); - assert_eq!(pacing.settled(), Duration::from_millis(400)); - assert_eq!(pacing.next(Duration::from_millis(100)), Duration::from_millis(200)); - assert_eq!(pacing.next(Duration::from_millis(200)), Duration::from_millis(400)); - assert_eq!(pacing.next(Duration::from_millis(400)), Duration::from_millis(400)); - } - #[test] fn poll_forward_mirrors_until_the_source_closes() { let src = BandwidthProducer::new(); @@ -2784,45 +1780,44 @@ mod tests { /// that doesn't route from here ahead of the one that does. #[cfg(feature = "tcp")] #[tokio::test] - async fn dial_any_walks_past_the_dead_addresses() { - let client = client(); + async fn the_walk_lands_past_the_dead_addresses() { let (mut server, live) = live().await; - tokio::spawn(async move { while server.accept().await.is_ok() {} }); + let (accepted, mut sessions) = tokio::sync::mpsc::unbounded_channel(); + tokio::spawn(async move { + while let Ok(request) = server.accept().await { + let _ = accepted.send(()); + let _ = request.ok().await; + } + }); let dead_a = dead().await; let dead_b = dead().await; let addrs = Addrs::collect([dead_a, dead_b, live.clone()]).expect("not empty"); - let shared = shared(); - let mut draining = None; - - let (connected, _session) = kio::clock::timeout( + let connection = kio::clock::timeout( WALK_TIMEOUT, - Connection::dial_any(&shared, &client, &addrs, &mut draining, &mut None), + client().with_reconnect(false).connect(addrs).established(), ) .await .expect("the walk must not hang") .expect("the live address must connect"); - assert_eq!(connected.url(), &live, "the walk must land on the one that answers"); + assert_eq!(connection.connects(), 1); + assert!( + sessions.recv().await.is_some(), + "the walk must land on the one that answers" + ); } /// With nothing reachable the walk reports a failure rather than hanging. #[cfg(feature = "tcp")] #[tokio::test] - async fn dial_any_reports_failure_when_nothing_answers() { - let client = client(); - + async fn the_walk_fails_when_nothing_answers() { let dead_a = dead().await; let dead_b = dead().await; let addrs = Addrs::collect([dead_a, dead_b]).expect("not empty"); - let shared = shared(); - let mut draining = None; - - let result = kio::clock::timeout( - WALK_TIMEOUT, - Connection::dial_any(&shared, &client, &addrs, &mut draining, &mut None), - ) - .await - .expect("the walk must not hang"); + let connection = client().with_reconnect(false).connect(addrs); + let result = kio::clock::timeout(WALK_TIMEOUT, connection.closed()) + .await + .expect("the walk must not hang"); assert!(result.is_err(), "every address hung up, so the walk must fail"); } @@ -2832,27 +1827,25 @@ mod tests { /// it in, and a dial URL logged verbatim would hand a replayable credential to /// anyone who can read logs. This drives the real `tracing` stack rather than /// [`Endpoint`]'s `Display`, so it also catches a log site that forgot to use - /// it. + /// it. moq-net's own log sites have a test of their own. #[cfg(feature = "tcp")] #[tracing_test::traced_test] #[tokio::test] async fn dialing_never_logs_the_credential() { const SECRET: &str = "b91d7fe20c4a"; - let client = client(); let mut target = dead().await; target.set_path(&format!("/.cluster/{SECRET}")); - let addrs = Addrs::new(target); - let shared = shared(); - let mut draining = None; - // `Session` isn't `Debug`, so unwrap the failure by hand. - let Err(err) = Connection::dial_any(&shared, &client, &addrs, &mut draining, &mut None).await else { - panic!("a dead address must fail"); - }; + let err = client() + .with_reconnect(false) + .connect(target) + .closed() + .await + .expect_err("a dead address must fail"); // The log line did happen, so the assertion below isn't vacuously true. - assert!(logs_contain("connecting"), "the dial never logged at all"); + assert!(logs_contain("connection closed"), "the loop never logged at all"); assert!(!logs_contain(SECRET), "a log line leaked the credential"); assert!(!format!("{err}").contains(SECRET), "the error leaked it: {err}"); } @@ -3183,23 +2176,27 @@ mod tests { /// failure from another candidate cannot replace it and keep an unlimited loop retrying. #[test] fn an_auth_rejection_stops_the_walk() { - assert!(settles_walk(&Error::Connect(crate::ConnectError::Unauthorized))); - assert!(settles_walk(&Error::Connect(crate::ConnectError::Forbidden))); - assert!(settles_walk(&Error::MoqNet(moq_net::Error::ProtocolViolation))); - assert!(!settles_walk(&Error::MoqNet(moq_net::Error::Closed))); - assert!(!settles_walk(&Error::Io(std::sync::Arc::new( - std::io::ErrorKind::ConnectionRefused.into() - )))); - } - - /// Only an attempt with somewhere to fall back on gets the fixed bound. - #[test] - fn only_a_candidate_with_a_fallback_is_bounded() { - assert_eq!(attempt_timeout(0, 1), None, "a lone address"); - assert_eq!(attempt_timeout(0, 2), Some(CONNECT_ATTEMPT), "one more to try"); - assert_eq!(attempt_timeout(1, 2), None, "the last of two"); - assert_eq!(attempt_timeout(1, 3), Some(CONNECT_ATTEMPT), "still one more"); - assert_eq!(attempt_timeout(2, 3), None, "the last of three"); + assert_eq!( + verdict(&Error::Connect(crate::ConnectError::Unauthorized)), + Verdict::Refused + ); + assert_eq!( + verdict(&Error::Connect(crate::ConnectError::Forbidden)), + Verdict::Refused + ); + assert_eq!(verdict(&Error::MoqNet(moq_net::Error::Unauthorized)), Verdict::Refused); + assert_eq!( + verdict(&Error::MoqNet(moq_net::Error::ProtocolViolation)), + Verdict::Fatal + ); + assert_eq!(verdict(&Error::MoqNet(moq_net::Error::Closed)), Verdict::Retry); + assert_eq!( + verdict(&Error::Io(std::sync::Arc::new( + std::io::ErrorKind::ConnectionRefused.into() + ))), + Verdict::Retry + ); + assert_eq!(verdict(&Error::NoBackend("none")), Verdict::Unusable); } /// Relays QUIC datagrams to `server`, holding them all until [`Forwarder::open`]. @@ -3525,210 +2522,90 @@ mod tests { .unwrap(); } - /// What a flapping peer does with each session the moment it is up. - #[cfg(feature = "tcp")] - #[derive(Clone)] - enum Flap { - /// Drop it, closing the session. - Close, - /// Send a GOAWAY naming this URI, empty for a restart, and keep serving. - Goaway(String), - } - - /// A client with the given pacing whose dials reach in-memory peers instead of a - /// transport, so the paused clock only moves once every task waits on a timer. - /// - /// The peer at each URL ends every session as its [`Flap`] says, and reports the - /// URL and instant each session came up on the returned channel. - #[cfg(feature = "tcp")] - fn flapping( - initial: Duration, - max: Duration, - timeout: Duration, - peers: Vec<(Url, Flap)>, - ) -> (Client, tokio::sync::mpsc::UnboundedReceiver<(Url, kio::clock::Instant)>) { - let mut config = crate::connect::Config::default(); - config.backoff.initial = Some(initial.into()); - config.backoff.max = Some(max.into()); - config.backoff.timeout = Some(timeout.into()); - let mut client = config.init(Default::default()).expect("build client"); - - let (accepted, accepts) = tokio::sync::mpsc::unbounded_channel(); - client.dialer = Some(std::sync::Arc::new(move |addr: crate::connect::Addr| { - let url = addr.url().clone(); - let (_, flap) = peers - .iter() - .find(|(peer, _)| *peer == url) - .expect("an unknown peer") - .clone(); - let accepted = accepted.clone(); - Box::pin(async move { - let session = flap_once(flap).await; - let _ = accepted.send((url, kio::clock::now())); - Ok(crate::client::Dialed { - session, - transport: crate::Transport::Tcp, - }) - }) - })); - (client, accepts) - } - - /// Run one in-memory MoQ handshake whose server side then ends the session as `flap` says. - #[cfg(feature = "tcp")] - async fn flap_once(flap: Flap) -> moq_net::Session { - let versions = moq_net::Versions::from("moq-lite-07-wip".parse::().unwrap()); - let config = || { - let mut config = qmux::Config::new(qmux::Version::QMux01); - config.protocol = qmux::Protocol::Negotiate(versions.alpns().iter().map(|alpn| alpn.to_string()).collect()); - config - }; - let stream = |io| qmux::transport::Stream::new(io); - let (client, server) = tokio::io::duplex(64 * 1024); - let (client, server) = tokio::try_join!( - qmux::Session::connect(stream(client), config()), - qmux::Session::accept(stream(server), config()) - ) - .unwrap(); - let (client, server) = (crate::spawn_qmux(client), crate::spawn_qmux(server)); - let now = moq_net::time::Instant::now(); - let client = async { - let (session, driver) = moq_net::Client::new() - .with_versions(versions.clone()) - .connect(now, client) - .await - .unwrap(); - tokio::spawn(moq_net::time::run(driver)); - session - }; - let server = async { - let request = moq_net::Server::new() - .with_versions(versions.clone()) - .accept_request(now, server) - .await - .unwrap(); - let (session, driver) = request.ok().await.unwrap(); - tokio::spawn(moq_net::time::run(driver)); - session - }; - let (client, server) = tokio::join!(client, server); - match flap { - Flap::Close => drop(server), - Flap::Goaway(uri) => { - let goaway = match uri.is_empty() { - true => moq_net::goaway::Goaway::new(), - false => moq_net::goaway::Goaway::redirect(uri), - }; - server.drain().send(goaway).unwrap(); - tokio::spawn(async move { - server.closed().await; - }); - } - } - client - } - - /// Check each gap between consecutive sessions against the jittered delay it should - /// have waited: between half of `delay` and all of it, starting at `initial` and - /// doubling up to `max`. - #[cfg(feature = "tcp")] - fn assert_paced(accepts: &[(Url, kio::clock::Instant)], initial: Duration, max: Duration) { - let mut delay = initial; - for (index, pair) in accepts.windows(2).enumerate() { - let gap = pair[1].1 - pair[0].1; - assert!( - gap >= delay / 2 && gap <= delay, - "retry {index} waited {gap:?}, outside the jittered {delay:?}" - ); - delay = (delay * 2).min(max); - } - } - /// A configured token is held before its dial starts the session, so a broadcast /// published before the dial, which only that token's grant covers, waits for its /// answer instead of closing the session against the connection credential's grant. #[cfg(feature = "tcp")] #[tokio::test(start_paused = true)] async fn a_publication_waits_for_a_configured_token() { - let shared = shared(); + let version: moq_net::Version = "moq-lite-07-wip".parse().unwrap(); let publish = crate::origin::spawn(); let _route = publish.route("room", moq_net::origin::RouteOptions::default()).unwrap(); let server_origin = crate::origin::spawn(); let config = crate::connect::Config { tokens: vec!["connection".parse().unwrap(), "publish".parse().unwrap()], + version: vec![version], ..Default::default() }; let mut client = config.init(Default::default()).expect("build client"); - let auth = shared.auth.clone(); - let (publish_dial, server_dial) = (publish.clone(), server_origin.clone()); - client.dialer = Some(std::sync::Arc::new(move |_addr: crate::connect::Addr| { - let (auth, publish, server_origin) = (auth.clone(), publish_dial.clone(), server_dial.clone()); - Box::pin(async move { - let versions = moq_net::Versions::from("moq-lite-07-wip".parse::().unwrap()); - let config = || { - let mut config = qmux::Config::new(qmux::Version::QMux01); - config.protocol = - qmux::Protocol::Negotiate(versions.alpns().iter().map(|alpn| alpn.to_string()).collect()); - config - }; - let (client, server) = tokio::io::duplex(64 * 1024); - let (client, server) = tokio::try_join!( - qmux::Session::connect(qmux::transport::Stream::new(client), config()), - qmux::Session::accept(qmux::transport::Stream::new(server), config()) - ) - .unwrap(); - let (client, server) = (crate::spawn_qmux(client), crate::spawn_qmux(server)); - let now = moq_net::time::Instant::now(); - let moq = moq_net::Client::new() - .with_versions(versions.clone()) - .with_auth(auth.clone()) - .with_publisher(&publish); - let client = moq.connect(now, client); - let server = async { - let mut request = moq_net::Server::new() - .with_versions(versions.clone()) - .with_subscriber(server_origin) - .accept_request(now, server) - .await - .unwrap(); - // The connection credential grants nothing; only the configured token - // covers the broadcast. - let mut requests = request.requests(); - let (session, driver) = request.ok().await.unwrap(); - tokio::spawn(moq_net::time::run(driver)); - tokio::spawn(async move { - let _session = session; - let mut issued = Vec::new(); - while let Some(request) = requests.next().await { - let grant = match request.token().is_empty() { - true => moq_net::auth::Grant::default(), - false => moq_net::auth::Grant::all(), - }; - issued.push(request.accept(grant)); - } - }); - }; - let client = async { - let (session, driver) = client.await.unwrap(); - tokio::spawn(moq_net::time::run(driver)); - session - }; - let (session, ()) = tokio::join!(client, server); - // Leave the session the time a multithreaded runtime may give it before the dial - // returns: on the paused clock, every exchange the session can make lands first, - // the connection credential's answer included. - kio::clock::sleep(Duration::from_millis(10)).await; - Ok(crate::client::Dialed { - session, - transport: crate::Transport::Tcp, + let server_dial = server_origin.clone(); + client.dialer = Some(std::sync::Arc::new( + move |_addr: crate::connect::Addr, moq: moq_net::Client| { + let server_origin = server_dial.clone(); + Box::pin(async move { + let versions = moq_net::Versions::from(version); + let config = || { + let mut config = qmux::Config::new(qmux::Version::QMux01); + config.protocol = + qmux::Protocol::Negotiate(versions.alpns().iter().map(|alpn| alpn.to_string()).collect()); + config + }; + let (client, server) = tokio::io::duplex(64 * 1024); + let (client, server) = tokio::try_join!( + qmux::Session::connect(qmux::transport::Stream::new(client), config()), + qmux::Session::accept(qmux::transport::Stream::new(server), config()) + ) + .unwrap(); + let (client, server) = (crate::spawn_qmux(client), crate::spawn_qmux(server)); + let now = moq_net::time::Instant::now(); + let client = moq.connect(now, client); + let server = async { + let mut request = moq_net::Server::new() + .with_versions(versions.clone()) + .with_subscriber(server_origin) + .accept_request(now, server) + .await + .unwrap(); + // The connection credential grants nothing; only the configured token + // covers the broadcast. + let mut requests = request.requests(); + let (session, driver) = request.ok().await.unwrap(); + tokio::spawn(moq_net::time::run(driver)); + tokio::spawn(async move { + let _session = session; + let mut issued = Vec::new(); + while let Some(request) = requests.next().await { + let grant = match request.token().is_empty() { + true => moq_net::auth::Grant::default(), + false => moq_net::auth::Grant::all(), + }; + issued.push(request.accept(grant)); + } + }); + }; + let client = async { + let (session, driver) = client.await.unwrap(); + tokio::spawn(moq_net::time::run(driver)); + session + }; + let (session, ()) = tokio::join!(client, server); + // Leave the session the time a multithreaded runtime may give it before the dial + // returns: on the paused clock, every exchange the session can make lands first, + // the connection credential's answer included. + kio::clock::sleep(Duration::from_millis(10)).await; + Ok(crate::client::Dialed { + session, + transport: crate::Transport::Tcp, + }) }) - }) - })); + }, + )); - let addrs = Addrs::new("moqt://relay.example.com/".parse::().unwrap()); - shared.tokens.set(client.tokens.clone()); - let (_, dialed) = Connection::dial_any(&shared, &client, &addrs, &mut None, &mut None) + let connection = client + .with_publisher(&publish) + .connect("moqt://relay.example.com/".parse::().unwrap()) + .established() .await .expect("the dial failed"); @@ -3739,173 +2616,10 @@ mod tests { .expect("the server's origin closed"); assert!(matches!(arrived, moq_net::announce::Event::Present(update) if update.prefix.as_str() == "room")); assert!( - futures::FutureExt::now_or_never(dialed.session.closed()).is_none(), + connection.connected() && connection.connects() == 1, "the session closed before the configured token was answered" ); } - - #[cfg(feature = "tcp")] - const INITIAL: Duration = Duration::from_millis(100); - #[cfg(feature = "tcp")] - const MAX: Duration = Duration::from_millis(400); - - /// A peer that accepts and then closes every session at once never earns the healthy - /// reset: the delay escalates to the cap, and the timeout ends the loop. - #[cfg(feature = "tcp")] - #[tokio::test(start_paused = true)] - async fn an_immediate_close_loop_escalates_until_the_timeout() { - const TIMEOUT: Duration = Duration::from_secs(2); - let peer = url("tcp://peer:1/"); - let (client, mut accepted) = flapping(INITIAL, MAX, TIMEOUT, vec![(peer.clone(), Flap::Close)]); - - let started = kio::clock::now(); - let err = client - .connect(peer) - .closed() - .await - .expect_err("a flapping peer kept the connection"); - assert!(matches!(err, Error::Reconnect(_)), "{err}"); - assert_eq!(started.elapsed(), TIMEOUT, "the timeout did not end the loop"); - - let mut accepts = Vec::new(); - while let Ok(accept) = accepted.try_recv() { - accepts.push(accept); - } - assert!(accepts.len() >= 6, "only {} sessions", accepts.len()); - assert_paced(&accepts, INITIAL, MAX); - } - - /// A peer restarting with an empty-URI GOAWAY on every session paces the redials like - /// failures, yet never spends the timeout: each drained session still served. - #[cfg(feature = "tcp")] - #[tokio::test(start_paused = true)] - async fn an_immediate_restart_loop_escalates_without_giving_up() { - const TIMEOUT: Duration = Duration::from_millis(500); - let peer = url("tcp://peer:1/"); - let (client, mut accepted) = flapping(INITIAL, MAX, TIMEOUT, vec![(peer.clone(), Flap::Goaway(String::new()))]); - let connection = client.connect(peer); - - let mut accepts = Vec::new(); - for _ in 0..10 { - accepts.push(accepted.recv().await.expect("the dialer dropped")); - } - assert!( - accepts[9].1 - accepts[0].1 > TIMEOUT * 2, - "the redials outran the backoff" - ); - assert!( - futures::FutureExt::now_or_never(connection.closed()).is_none(), - "a restarting peer ended the connection" - ); - assert_paced(&accepts, INITIAL, MAX); - } - - /// Two peers that redirect to each other the moment a session is up are failed attempts: - /// the delay escalates across both, and the timeout ends the loop. - #[cfg(feature = "tcp")] - #[tokio::test(start_paused = true)] - async fn an_immediate_redirect_loop_escalates_until_the_timeout() { - const TIMEOUT: Duration = Duration::from_secs(2); - let a = url("tcp://peer:1/"); - let b = url("tcp://peer:2/"); - let peers = vec![ - (a.clone(), Flap::Goaway(b.to_string())), - (b.clone(), Flap::Goaway(a.to_string())), - ]; - let (client, mut accepted) = flapping(INITIAL, MAX, TIMEOUT, peers); - - let started = kio::clock::now(); - let err = client - .connect(a.clone()) - .closed() - .await - .expect_err("a redirect loop kept the connection"); - assert!( - matches!(&err, Error::Reconnect(reason) if reason.contains("redirect")), - "{err}" - ); - assert_eq!(started.elapsed(), TIMEOUT, "the timeout did not end the loop"); - - let mut accepts = Vec::new(); - while let Ok(accept) = accepted.try_recv() { - accepts.push(accept); - } - assert!(accepts.len() >= 6, "only {} sessions", accepts.len()); - for (index, (peer, _)) in accepts.iter().enumerate() { - assert_eq!( - *peer, - if index % 2 == 0 { a.clone() } else { b.clone() }, - "session {index}" - ); - } - assert_paced(&accepts, INITIAL, MAX); - } - - /// A zero `initial` turns retry waiting off: every redial follows the close at once. - #[cfg(feature = "tcp")] - #[tokio::test(start_paused = true)] - async fn a_zero_initial_redials_at_once() { - const TIMEOUT: Duration = Duration::from_secs(1); - let peer = url("tcp://peer:1/"); - let (client, mut accepted) = flapping(Duration::ZERO, MAX, TIMEOUT, vec![(peer.clone(), Flap::Close)]); - let connection = client.connect(peer); - - let first = accepted.recv().await.expect("the dialer dropped").1; - for _ in 0..10 { - let at = accepted.recv().await.expect("the dialer dropped").1; - assert_eq!(at, first, "a zero initial waited before a retry"); - } - drop(connection); - } - - #[test] - fn redirect_loop_detection_catches_any_cycle_in_constant_memory() { - for tail in 0..4 { - for cycle in 2..=8 { - // `tail` one-way moves, then `cycle` endpoints redirecting around a ring. - let at = |hop: usize| { - if hop < tail { - url(&format!("tcp://tail{hop}/")) - } else { - url(&format!("tcp://ring{}/", (hop - tail) % cycle)) - } - }; - let mut redirects = Redirects::default(); - let mut caught = None; - for hop in 0..4 * (tail + cycle) { - let looped = redirects.depart(at(hop), &at(hop + 1)); - match caught { - Some(_) => assert!(looped, "a caught loop stays caught"), - None if looped => caught = Some(hop), - None => {} - } - } - let caught = caught.unwrap_or_else(|| panic!("tail {tail}, cycle {cycle}: never caught")); - // Never before the run first returns to an endpoint it left. - assert!( - caught + 1 >= tail + cycle, - "tail {tail}, cycle {cycle}: caught at {caught}" - ); - } - } - - let mut redirects = Redirects::default(); - for hop in 0..1000 { - let from = url(&format!("tcp://a{hop}/")); - let to = url(&format!("tcp://a{}/", hop + 1)); - assert!(!redirects.depart(from, &to), "a one-way run never loops"); - } - } - - #[test] - fn an_endpoint_ignores_the_credentials() { - assert_eq!( - endpoint(&url("tcp://token-a:secret@relay:1/path?jwt=a#x")), - endpoint(&url("tcp://token-b@relay:1/path?jwt=b")) - ); - // The path is part of the request target, so it names a different endpoint. - assert_ne!(endpoint(&url("tcp://relay:1/a")), endpoint(&url("tcp://relay:1/b"))); - } } #[cfg(test)] @@ -3932,7 +2646,7 @@ mod duration_tests { Backoff, healthy, "--connect-backoff-healthy", - |c: &Backoff| c.healthy(), + |c: &Backoff| c.resolve().healthy, |d| d ); crate::cli::duration::test_config_duration!( diff --git a/rs/moq-tokio/src/error.rs b/rs/moq-tokio/src/error.rs index 7f98c3d6e6e..628d0155897 100644 --- a/rs/moq-tokio/src/error.rs +++ b/rs/moq-tokio/src/error.rs @@ -275,13 +275,8 @@ impl Error { #[cfg(feature = "websocket")] Self::TransportRace { quic, websocket } => quic.is_terminal() && websocket.is_terminal(), Self::MoqNet(err) => matches!( - moq_net::SessionError::from(err), - moq_net::SessionError::Unauthorized - | moq_net::SessionError::Expired - | moq_net::SessionError::MalformedAuthToken - | moq_net::SessionError::ProtocolViolation - | moq_net::SessionError::KeyValueFormatting - | moq_net::SessionError::InvalidRequestId + moq_net::reconnect::Verdict::from(err), + moq_net::reconnect::Verdict::Refused | moq_net::reconnect::Verdict::Fatal ), Self::Tls(err) => err.is_terminal(), Self::NoBackend(_) @@ -367,6 +362,20 @@ impl Error { } } +/// What a connection reports when moq-net's reconnect policy fails an attempt or gives up. +#[cfg(feature = "net")] +impl From for Error { + fn from(failure: moq_net::reconnect::Failure) -> Self { + use moq_net::reconnect::Failure; + match failure { + Failure::Timeout(timeout) => Self::ConnectTimeout(timeout), + Failure::Refused(reason) => Self::RefusedRedirect(reason.to_string()), + Failure::Fixed => Self::PinnedRedirect, + failure => Self::Reconnect(failure.to_string()), + } + } +} + // The wrapped sources aren't `Clone`, so `#[from]` can't store them behind `Arc` // directly. These hand-written conversions keep `?` ergonomic at the call sites. impl From for Error { diff --git a/sh/gh/test-inventory.allow b/sh/gh/test-inventory.allow index f705ba6cc0e..90bd29be6e8 100644 --- a/sh/gh/test-inventory.allow +++ b/sh/gh/test-inventory.allow @@ -4,6 +4,7 @@ # Flaky under load; the quest makes it reliable, then removes its `#[ignore]`. ignore moq-ffi::test::datagrams_cross_a_default_connection /quest/m0/test-flakes-2/ffi-datagram-roundtrip.md +ignore moq-ffi::test::requested_datagrams_cross_a_default_connection /quest/m0/test-flakes-2/ffi-datagram-roundtrip.md # Need a GPU, a desktop, or a device that no CI runner has. ignore moq-carla-bridge::gpu_video_reaches_the_worker /quest/m5/carla-bridge.md diff --git a/test/drill/mutations/reconnect-stops-after-session-loss.patch b/test/drill/mutations/reconnect-stops-after-session-loss.patch index a1f9d891bff..7e7c159fef9 100644 --- a/test/drill/mutations/reconnect-stops-after-session-loss.patch +++ b/test/drill/mutations/reconnect-stops-after-session-loss.patch @@ -2,19 +2,19 @@ # expect: subscriber: reconnect loop gave up # # Removes redialing after an established session is lost. The reconnect loop -# treats a healthy session's end the way one-shot mode does and stops, so a +# treats a healthy session's end the way connecting once does and stops, so a # relay restart becomes a permanent outage instead of a bounded gap the same # handles resume across. -diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs ---- a/rs/moq-tokio/src/connection.rs -+++ b/rs/moq-tokio/src/connection.rs -@@ -1289,7 +1289,8 @@ - // One-shot mode: the session ending ends the connection. Its close - // reason is the terminal error, mirroring `moq_net::Session::closed`. - // A GOAWAY already returned above. -- if !client.reconnect { -+ // MUTATION: losing an established session ends the loop instead of redialing. -+ if !client.reconnect || healthy { - return match ended { - Ended::Closed(res) => res, - Ended::Goaway(_) => unreachable!("handled before the migration branch"), +diff --git a/rs/moq-net/src/reconnect.rs b/rs/moq-net/src/reconnect.rs +--- a/rs/moq-net/src/reconnect.rs ++++ b/rs/moq-net/src/reconnect.rs +@@ -772,7 +772,8 @@ impl + From> Reco + }; + let err = E::from(err); + // Connecting once, the session ending ends the policy with its close reason. +- if terminal || self.config.once { ++ // MUTATION: losing an established session ends the policy instead of redialing. ++ if terminal || self.config.once || healthy { + return self.done(Err(err)); + } +