diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index 1e34d888b7..c9615c6965 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -381,7 +381,11 @@ drain_timeout = "10s" # Top-level key, as --drain-timeout / MOQ_D The first SIGTERM or SIGINT starts a drain: every session is sent a GOAWAY asking it to reconnect elsewhere, including any that connect during the drain, -and is force-closed if it is still connected when the window ends. Accepted +and is force-closed if it is still connected when the window ends. The cluster +links the relay dialed are drained the same way and stop redialing, so a +neighbour hears the relay is draining and moves its routes before the link +closes. A neighbour keeps serving such a link rather than leave it, so a relay +that dialed a peer holds its drain to the end of the window. Accepted connections still handshaking or awaiting authorization hold the drain until they finish, fail, or reach that same deadline, where any still pending is refused or closed. Public HTTP/1.1 connections hold the drain from TCP accept @@ -391,7 +395,7 @@ ALPN because extended CONNECT is disabled. A later WebSocket upgrade on the same keep-alive connection joins the drain when its headers arrive. The internal listener is unaffected. -The relay exits as soon as every admitted connection has left, or immediately on +The relay exits as soon as every admitted connection and dialed link has left, or immediately on a second signal. When the window ends, it stops waiting for connections still pending and exits once every session it closed has reported its `end` to the auth server, each report bounded by that server's request timeout. Once it commits to exit, WebSocket @@ -403,6 +407,8 @@ closed when the window ends. An embedder can take over the signals; see An embedder's drain alert can page on the `forced` count returned by [`Observer::drained`](https://docs.rs/moq-relay/latest/moq_relay/shutdown/struct.Observer.html#method.drained). +It leaves out the dialed links the drain holds to its end, so a routine +restart reports none. Its `ending` count distinguishes sessions already closed before the deadline but still reporting their end. The live tally separates open sessions from that cleanup, and the summary's elapsed time stays the same for late observers. diff --git a/doc/bin/relay/http.md b/doc/bin/relay/http.md index 0f29bd583f..fb727412db 100644 --- a/doc/bin/relay/http.md +++ b/doc/bin/relay/http.md @@ -108,8 +108,8 @@ A few worth alerting on: out a release. - `moq_relay_stale_bytes_total` and friends: content dropped for drifting past a subscriber's latency budget. -- `moq_relay_draining_sessions`: sessions sent a GOAWAY during a - [shutdown drain](/bin/relay/config#shutdown) that have not left yet. +- `moq_relay_draining_sessions`: sessions and dialed cluster links sent a GOAWAY + during a [shutdown drain](/bin/relay/config#shutdown) that have not left yet. - `moq_relay_panics_total`: bugs that closed one session instead of the relay. Each logs an error with the message and the session's span. A panic in a long-lived task (the origin, the cache governor, the certificate reload, diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md index c3a17568a5..bf4f25145d 100644 --- a/quest/m2/multicast/README.md +++ b/quest/m2/multicast/README.md @@ -38,8 +38,7 @@ children, the two flaws it found, [a reader resuming on a detour learned after its session closed](/quest/m2/multicast/detour-resume.md) and [a route moving behind one next hop resending a group](/quest/m2/multicast/resend.md), whose fix changes the wire, -[the mesh replay](/quest/m2/multicast/mesh-replay.md), and -[draining dialed links](/quest/m2/multicast/drain-dialed-links.md), and on +and [the mesh replay](/quest/m2/multicast/mesh-replay.md), and on [route layer acceptance](/quest/m2/cluster-routes/acceptance.md) retiring the lite-06 mitigations. After the cut come warmth's wire ([cache-aware routing](/quest/m3/cache-routing/README.md)), diff --git a/quest/m2/multicast/drain-dialed-links.md b/quest/m2/multicast/drain-dialed-links.md deleted file mode 100644 index af7a29bc80..0000000000 --- a/quest/m2/multicast/drain-dialed-links.md +++ /dev/null @@ -1,42 +0,0 @@ -# [S] A draining relay drains the cluster links it dialed - -## Goal - -A relay told to shut down sends GOAWAY on the cluster sessions it dialed, as it -does on those it accepted, and keeps them up until its drain deadline or until -the peer leaves. A neighbour then hears DRAIN and moves its routes before the -relay goes, instead of hearing only DEAD once the link drops, as after a crash. - -## Plan - -Found while checking route counters -([moq-dev/moq#9126](https://github.com/moq-dev/moq/pull/9126)) on a local -root/leaf0/leaf1 cluster over moq-lite-07: leaf1 dials both of its peers, and -on SIGINT its log reads "drain ended with every session gone" within -microseconds. Its neighbours count a DEAD and no DRAIN. -`rs/moq-relay/src/shutdown.rs` drains only accepted sessions; the dial loop in -`rs/moq-relay/src/cluster.rs` closes its sessions with the process. - -Decided 2026-11-19 (coordinator, under the maintainer's pick that day to -merge on the coordinator's recommendation): ✅ kept, and under this epic, so -the lite-07 cut waits on it (rejected: m3, as the implementer recommended; -dropping it). A deployment that rolls one relay at a time restarts every -relay on purpose, and until this lands each of those restarts is a crash to -the neighbours the relay dialed. -[The mesh replay](/quest/m2/multicast/mesh-replay.md) measures a roll, and -without this it measures crashes. - -- Hand the shutdown observer to the cluster dial loop, so it sends GOAWAY on - its live session, stops redialing, and holds the drain open until the session - ends or the shared deadline passes. -- Test it on real relays, in a triangle where the drained relay dials both - others: each neighbour, hearing the GOAWAY itself, counts - `routes.sent.drain` toward the other, which counts `routes.received.drain`, - all before either counts a DEAD. No DRAIN comes from the draining relay - itself, which sends GOAWAY: a pair of relays alone shows none. - -Public API: none. Wire: none. - -## Related - -- [Real relays replay a mesh and its faults](/quest/m2/multicast/mesh-replay.md) - its roll drains each relay with GOAWAY diff --git a/quest/m2/multicast/mesh-replay.md b/quest/m2/multicast/mesh-replay.md index 9ab27713a1..164c45de88 100644 --- a/quest/m2/multicast/mesh-replay.md +++ b/quest/m2/multicast/mesh-replay.md @@ -78,4 +78,3 @@ Public API: none. Wire: none. - `rs/moq-net/tests/integration/route_audit/` ([moq-dev/moq#9111](https://github.com/moq-dev/moq/pull/9111)) - the same failures on mock sessions with mocked time - [Random fault schedules](/quest/m4/cluster-fault-schedules.md) - searches fault timings on the simulated clock - [A nightly soak](/quest/m4/relay-soak.md) - one real relay under churn -- [A draining relay drains the cluster links it dialed](/quest/m2/multicast/drain-dialed-links.md) - without it, a drained relay's dialed links end with DEAD and no DRAIN diff --git a/rs/moq-relay/src/cluster.rs b/rs/moq-relay/src/cluster.rs index ad27361f3c..a715f01bca 100644 --- a/rs/moq-relay/src/cluster.rs +++ b/rs/moq-relay/src/cluster.rs @@ -1061,6 +1061,12 @@ pub struct Options { /// meanwhile, and the first publish carries them. pub stats_hold: Option>, + /// The relay's drain, which every peer link this cluster dialed joins: once it + /// starts, each link stops redialing and sends its peer a GOAWAY, and the drain + /// holds it open until the peer leaves or the deadline passes. `None` closes + /// dialed links only on [`Cluster::close`]. + pub shutdown: Option, + /// How peers are reached in place of the network, for tests on `kio::clock::sim`: /// every peer dial becomes a [`Dial`](moq_tokio::sim::Dial) the test routes. /// `None` dials over sockets. @@ -1285,6 +1291,11 @@ impl Mesh { /// Apply one source's change to the dial set. fn apply(&mut self, command: DialCommand) { + // Once the relay drains, the dial set is frozen: replacing or releasing a dial + // would drop its link mid-drain, and a new one would only return at once. + if self.cluster.shutdown.deadline().is_some() { + return; + } let mut dial = dialer(&self.cluster, &self.token); match command { DialCommand::Api(desired) => self.dialed.reconcile_api(&desired, &mut dial), @@ -1353,6 +1364,8 @@ pub struct Cluster { dial: Option>, /// Every dial client, shared by all clones, for [`Self::close`]. dialers: Dialers, + /// The drain each dialed link joins; see [`Options::shutdown`]. + shutdown: crate::shutdown::Observer, /// Bound listen port and optional generated-certificate fingerprint. advertise: Option, /// Set by [`Self::start`] when LAN discovery is on, so inbound @@ -1466,6 +1479,7 @@ impl Cluster { peer, stats, stats_hold, + shutdown, #[cfg(feature = "test-support")] network, #[cfg(feature = "test-support")] @@ -1531,6 +1545,7 @@ impl Cluster { sim, dial: None, dialers: Dialers::new(), + shutdown: shutdown.unwrap_or_else(crate::shutdown::Observer::disabled), advertise: None, lan_auth: Arc::new(std::sync::OnceLock::new()), nodes, @@ -2162,8 +2177,13 @@ impl Cluster { dial: &crate::nodes::Dial, ) -> anyhow::Result<()> { let mut redial = Redial::default(); + let mut shutdown = self.shutdown.clone(); loop { + // A draining relay dials nothing new: its peers would only be drained again. + if shutdown.deadline().is_some() { + return Ok(()); + } let mut connected = false; let (result, presented) = match Self::dial_addrs(target, &token) { Ok((addrs, presented)) => ( @@ -2174,8 +2194,11 @@ impl Cluster { }; dial.disconnected(); - // An attempt only ends in an error: its own reconnects retry until they give up. - let Err(err) = result; + // An attempt ends in an error, its own reconnects having given up, or drained. + let err = match result { + Ok(()) => return Ok(()), + Err(err) => err, + }; tracing::warn!(%err, "cluster peer error; will retry"); // The session's own reconnects reuse the token it started with, so a // failure after the file rotated is most likely that token expiring: @@ -2190,7 +2213,11 @@ impl Cluster { } let wait = redial.next(connected, &mut kio::clock::rng()); - kio::clock::sleep(wait).await; + tokio::select! { + biased; + () = shutdown.started() => return Ok(()), + () = kio::clock::sleep(wait) => {} + } } } @@ -2219,14 +2246,15 @@ impl Cluster { } /// One session to `target` over `addrs`, which carry any credential. It runs until the - /// session gives up, and sets `connected` once the peer has accepted it. + /// session gives up, or until the relay's drain has drained it, and sets `connected` + /// once the peer has accepted it. async fn run_remote_once( &self, addrs: &moq_tokio::Addrs, target: &DialTarget, dial: &crate::nodes::Dial, connected: &mut bool, - ) -> anyhow::Result { + ) -> anyhow::Result<()> { // Each attempt is its own session, so it gets its own id. Matches the span an // accepted connection runs under, so both directions log the same way. let id = self.next_connection_id(); @@ -2244,7 +2272,7 @@ impl Cluster { target: &DialTarget, dial: &crate::nodes::Dial, connected: &mut bool, - ) -> anyhow::Result { + ) -> anyhow::Result<()> { // The peer URL may carry the cluster JWT in its query, so neither the log // line nor the node label below may show the raw URL. let first = addrs.as_slice().first().expect("Addrs is non-empty").url(); @@ -2289,6 +2317,9 @@ impl Cluster { if let Some(cost) = target.cost { client = client.with_cost(cost); } + // Counted by the relay's drain before the first dial, so a drain that starts + // while the session connects still waits for it. + let serving = self.shutdown.serve(); // The GOAWAY lifecycle lives in the reconnect loop: on an upstream GOAWAY it // dials the replacement while the old session keeps serving, so the old // routes stay attached and the origin hands live tracks over at a group @@ -2316,11 +2347,20 @@ impl Cluster { .connect(addrs.clone()); let monitor = reconnect.monitor(); let mut observation = monitor.state(); + let mut shutdown = self.shutdown.clone(); loop { dial.observe(id, &observation); *connected |= observation.connects > 0; - observation = monitor.state_changed(observation.revision).await?; + tokio::select! { + biased; + () = shutdown.started() => break, + changed = monitor.state_changed(observation.revision) => observation = changed?, + } } + // Drained like an accepted session, so the peer hears GOAWAY and moves its + // routes before this relay goes, rather than seeing the link drop as a crash. + serving.drain_link(reconnect).await; + Ok(()) } /// A peer client from the dial template, with the [`Options::peer`] QUIC @@ -3505,6 +3545,52 @@ mod tests { assert!(nodes.snapshot().nodes.is_empty(), "a removed peer takes its history"); } + /// Once the relay drains, no source change replaces, releases, or adds a dial, so + /// each link's own task finishes its drain to the deadline. + #[tokio::test] + async fn mesh_freezes_its_dials_once_the_relay_drains() { + let (trigger, shutdown) = crate::shutdown::Observer::new(Duration::from_secs(10)); + let mut options = Options::new(Config::default()); + options.shutdown = Some(shutdown); + let mut mesh = Mesh { + cluster: Cluster::spawn(options).expect("cluster"), + token: ClusterToken::default(), + dialed: DialMap::new(None), + sources: kio::Tasks::new(), + commands: DialCommands::default(), + }; + let api = |peers: &[&str]| { + DialCommand::Api( + peers + .iter() + .map(|peer| { + let target = DialTarget::parse(peer).unwrap(); + (target.key.clone(), target) + }) + .collect(), + ) + }; + mesh.apply(api(&["https://a.example/?cost=1", "https://b.example/"])); + let task = mesh.dialed.entries[&key("https://a.example/")].task; + + trigger.start(); + mesh.apply(api(&["https://a.example/?cost=2", "https://c.example/"])); + assert_eq!( + mesh.dialed.entries[&key("https://a.example/")].task, + task, + "a changed target must not replace a draining dial" + ); + assert!( + mesh.dialed.contains(&key("https://b.example/")), + "a removed peer must keep draining" + ); + assert!( + !mesh.dialed.contains(&key("https://c.example/")), + "a draining relay dials nothing new" + ); + assert_eq!(mesh.dialed.tasks.len(), 2); + } + /// A dial task's panic ends the mesh with it, rather than unwinding the task /// polling the mesh. #[tokio::test] diff --git a/rs/moq-relay/src/relay.rs b/rs/moq-relay/src/relay.rs index 0d85876137..1e2d0e616a 100644 --- a/rs/moq-relay/src/relay.rs +++ b/rs/moq-relay/src/relay.rs @@ -298,6 +298,7 @@ impl Relay { .with_stats(config.stats); cluster_options.client = config.limits.client; cluster_options.peer = config.limits.peer; + cluster_options.shutdown = Some(shutdown.clone()); // The node's stats broadcast has one instance at a time: the relay this one // replaces keeps it until it exits, and its last counters are read first. let alone = sockets.replacing().then(|| { diff --git a/rs/moq-relay/src/shutdown.rs b/rs/moq-relay/src/shutdown.rs index 41a53d7095..f85e385ecd 100644 --- a/rs/moq-relay/src/shutdown.rs +++ b/rs/moq-relay/src/shutdown.rs @@ -1,6 +1,6 @@ //! Graceful shutdown coordination: the first shutdown signal fires a broadcast //! that every accepted session observes, draining it with a GOAWAY before the -//! process exits. +//! process exits. The cluster links the relay dialed drain the same way. //! //! The drain has one deadline, fixed when it starts. A session accepted after //! that (a cached DNS resolve, a pool alias) is sent a GOAWAY at once, carrying @@ -120,7 +120,7 @@ impl Signals { #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] #[non_exhaustive] pub struct Tally { - /// Served sessions whose closure has not yet been observed by supervision. + /// Served sessions, and dialed cluster links, whose closure has not yet been observed. pub open: usize, /// Closed sessions still reporting their end. pub ending: usize, @@ -130,7 +130,8 @@ pub struct Tally { #[derive(Clone, Copy, Debug, PartialEq, Eq)] #[non_exhaustive] pub struct Drained { - /// Sessions still open at the deadline that were force-closed. + /// Sessions still open at the deadline that were force-closed. Cluster links the + /// relay dialed are held to the deadline by design, so they are never counted. pub forced: usize, /// Sessions closed strictly before the deadline and still reporting their end at it. pub ending: usize, @@ -273,6 +274,8 @@ impl Observer { /// /// One served after the deadline is closed at once and not counted: arrivals /// a listener still lets through must not hold a drain that is past it. + /// + /// A cluster link the relay dials is counted the same way, from before its first dial. pub fn serve(&self) -> Serving { let counted = self.deadline().is_none_or(|deadline| Instant::now() < deadline); if counted { @@ -342,7 +345,7 @@ impl Observer { elapsed: ended.saturating_duration_since(window.started), }; match drained.forced { - 0 => tracing::info!(elapsed = ?drained.elapsed, "drain ended with every session gone; exiting"), + 0 => tracing::info!(elapsed = ?drained.elapsed, "drain ended with no session force-closed; exiting"), forced => { tracing::warn!(elapsed = ?drained.elapsed, forced, "drain deadline force-closed sessions; exiting") } @@ -351,8 +354,9 @@ impl Observer { } /// Count a session as draining until the returned guard drops, and as forced - /// if it is still open at `deadline`. - fn draining(&self, deadline: Instant) -> Draining { + /// if it is still open at `deadline`. `None` counts a link the drain holds to + /// its end by design, which is never forced. + fn draining(&self, deadline: Option) -> Draining { self.state.send_modify(|state| state.draining += 1); Draining { state: self.state.clone(), @@ -385,7 +389,7 @@ impl Observer { let remaining = deadline.saturating_duration_since(now); // Dropped when this returns or is cancelled (a WebSocket driver ending // first), so the count holds whichever way the session goes. - let _draining = counted.then(|| self.draining(deadline)); + let _draining = counted.then(|| self.draining(Some(deadline))); // "Reconnect to me": the relay is restarting, not moving. An empty URI is // legal from either side and this runs once per session, so neither error @@ -403,6 +407,19 @@ impl Observer { session.abort(moq_net::Error::GoawayTimeout); session.closed().await; } + + /// Drain a cluster link this relay dialed as [`drain`](Self::drain) does an + /// accepted session: it stops redialing, sends a GOAWAY, and closes at the deadline. + async fn drain_link(&self, link: moq_tokio::Connection, counted: bool) { + let now = Instant::now(); + let deadline = self.deadline().unwrap_or(now.saturating_add(self.drain_timeout)); + // A neighbour keeps serving a link that sent it a GOAWAY, so reaching the + // deadline is how this link ends, not a session that failed to leave. + let _draining = counted.then(|| self.draining(None)); + if let Err(err) = link.drain(deadline.saturating_duration_since(now)).await { + tracing::debug!(%err, "dialed link did not leave before the drain deadline"); + } + } } /// A session the drain waits on, from [`Observer::serve`]. @@ -432,6 +449,12 @@ impl Serving { self.shutdown.drain(session, counted).await; } + /// Drain a cluster link this relay dialed; see [`Observer::drain_link`]. + pub(crate) async fn drain_link(&self, link: moq_tokio::Connection) { + let counted = !matches!(self.count, Count::Uncounted); + self.shutdown.drain_link(link, counted).await; + } + /// The session closed: count it as ending until this drops. Only the first call counts. pub(crate) fn closed(&mut self) { if !matches!(self.count, Count::Open) { @@ -470,14 +493,15 @@ impl Drop for Serving { /// A session sent a GOAWAY; see [`Observer::draining`]. struct Draining { state: watch::Sender, - deadline: Instant, + /// When the session counts as forced, or `None` for a link held to the end. + deadline: Option, } impl Drop for Draining { fn drop(&mut self) { // The force-close fires at `deadline`, so every session it closes is // counted. A peer leaving in that sliver was still there at the deadline too. - let forced = Instant::now() >= self.deadline; + let forced = self.deadline.is_some_and(|deadline| Instant::now() >= deadline); self.state.send_modify(|state| { state.draining -= 1; state.forced += usize::from(forced); @@ -799,7 +823,7 @@ mod tests { let window = Duration::from_secs(10); let (trigger, shutdown) = Observer::new(window); let mut established = shutdown.serve(); - let draining = shutdown.draining(kio::clock::now().saturating_add(window)); + let draining = shutdown.draining(Some(kio::clock::now().saturating_add(window))); trigger.start(); kio::clock::sleep(window / 2).await; established.closed(); @@ -846,9 +870,11 @@ mod tests { trigger.start(); let deadline = shutdown.deadline().expect("started"); - let left = shutdown.draining(deadline); - let straggler = shutdown.draining(deadline); - assert_eq!(shutdown.draining_count(), 2); + let left = shutdown.draining(Some(deadline)); + let straggler = shutdown.draining(Some(deadline)); + let mut link_serving = shutdown.serve(); + let link = shutdown.draining(None); + assert_eq!(shutdown.draining_count(), 3); kio::clock::sleep(window / 2).await; left_serving.closed(); @@ -856,12 +882,18 @@ mod tests { kio::clock::sleep(window / 2).await; drop(straggler); edge_serving.closed(); + drop(link); + link_serving.closed(); let state = *shutdown.state.borrow(); assert_eq!(state.draining, 0); - assert_eq!(state.forced, 1, "only the session open at the deadline is forced"); + assert_eq!( + state.forced, 1, + "only the session open at the deadline is forced, not the link held to it" + ); drop(left_serving); drop(edge_serving); + drop(link_serving); assert_eq!( shutdown.drained().await, Drained { diff --git a/rs/moq-relay/tests/integration/drain_dialed.rs b/rs/moq-relay/tests/integration/drain_dialed.rs new file mode 100644 index 0000000000..5ff592d1a9 --- /dev/null +++ b/rs/moq-relay/tests/integration/drain_dialed.rs @@ -0,0 +1,234 @@ +//! A draining relay drains the cluster links it dialed, as it does those it accepted: +//! each peer hears its GOAWAY and tells the others the relay is draining, well before +//! any link drops. +//! +//! Three real relays on loopback: C dials A, and L dials both. L drains, so A and C +//! each hear L's GOAWAY themselves, and the first to hear it sends the other a DRAIN +//! for it, while L, which sent the GOAWAY, sends none. Every link stays up until L's +//! drain deadline, so no relay counts a DEAD until then. + +use std::time::Duration; + +use moq_net::origin::{Cost, RouteKind}; +use moq_net::stats::{Registry, Routes, Withdrawal}; +use moq_relay::{Config, Relay, shutdown}; + +/// Only a hang reaches it. +const TIMEOUT: Duration = Duration::from_secs(10); + +/// How long L holds its drain open. Long enough that every DRAIN lands well inside it. +const DRAIN: Duration = Duration::from_secs(2); + +/// One relay running on the test runtime. +struct Node { + port: u16, + stats: Registry, + origin: moq_net::origin::Producer, + trigger: shutdown::Trigger, + shutdown: shutdown::Observer, + running: tokio::task::JoinHandle>, +} + +impl Node { + /// A relay `name` draining over `drain`, dialing each port in `peers` as a cluster + /// peer and admitting every session as one. + async fn start(name: &str, peers: &[u16], drain: Duration) -> Self { + let mut config = Config::default(); + config.listen.bind = Some("127.0.0.1:0".parse().unwrap()); + config.listen.tls.generate = vec!["localhost".into()]; + config.drain_timeout = Some(drain.into()); + // Counted, but published only once. + config.stats.enabled = true; + config.stats.node = Some(name.into()); + config.stats.interval = Some(Duration::from_secs(3600).into()); + config.connect.bind = Some("127.0.0.1:0".parse().unwrap()); + config.connect.tls.insecure = Some(true); + config.connect.websocket.enabled = Some(false); + // Only lite-07 carries the route layer, which a relay offers only when asked. + let lite07: moq_net::Version = "moq-lite-07-wip".parse().unwrap(); + config.listen.version = vec![lite07]; + config.connect.version = vec![lite07]; + config.cluster.connect = peers + .iter() + .map(|port| moq_relay::cluster::Peer::new(format!("https://127.0.0.1:{port}/"))) + .collect(); + let mut relay = Relay::load(config).await.expect("load relay").with_signals(false); + // A peer grant, as a deployment's auth gives its relays: only a peer exchanges routes. + let mut admissions = relay.admissions().expect("embedded auth"); + tokio::spawn(async move { + let mut leases = Vec::new(); + while let Some(admission) = admissions.next().await { + let mut grant = moq_auth::Grant::new(moq_auth::Pattern::all(), moq_auth::Pattern::all()); + grant.link = Some(moq_auth::Link::Peer); + let (lease, consumer) = moq_auth::lease::Producer::new(grant); + admission.grant(consumer); + leases.push(lease); + } + }); + let port = relay.addr().expect("bound UDP").port(); + let stats = relay.cluster().stats.clone(); + let origin = relay.cluster().origin.clone(); + let trigger = relay.shutdown_trigger().clone(); + let shutdown = relay.shutdown().clone(); + let ready = relay.ready(); + let running = tokio::spawn(relay.run()); + ready.wait().await.expect("relay ready"); + Self { + port, + stats, + origin, + trigger, + shutdown, + running, + } + } + + /// The route-layer counters of every session, summed over tiers. + fn routes(&self) -> Routes { + let mut routes = Routes::default(); + for (_, presence) in self.stats.snapshot().sessions() { + routes.add(presence.routes); + } + routes + } + + /// Sessions open now, accepted and dialed. + fn active(&self) -> u64 { + self.stats + .snapshot() + .sessions() + .iter() + .map(|(_, presence)| presence.active()) + .sum() + } + + /// Publish the broadcast `name` from this relay. + fn publish(&self, name: &str) -> moq_net::origin::Route { + self.origin + .route(name, moq_net::origin::RouteOptions::default()) + .expect("create broadcast") + } + + /// Resolve once this relay's route for `name` comes straight from its publisher: one + /// unpriced link, costing 1, where a route through another neighbour costs 2. The + /// relay forwards that route to its other neighbours as it takes it. + async fn direct(&self, name: &str) { + let mut routes = self.origin.consume().routes(); + kio::clock::timeout(TIMEOUT, async { + loop { + let event = routes.next().await.expect("origin open"); + let live = matches!(event.kind, RouteKind::Start | RouteKind::Update | RouteKind::Restart); + let route = &event.announce; + if live && route.prefix.as_str() == name && route.route.cost() == Cost::new(1) { + return; + } + } + }) + .await + .unwrap_or_else(|_| panic!("timed out taking a direct route to {name}")); + } + + /// Resolve once this relay has an active route for `name`. + async fn learns(&self, name: &str) { + kio::clock::timeout(TIMEOUT, self.origin.consume().follow(name, None).unwrap().available()) + .await + .unwrap_or_else(|_| panic!("timed out learning {name}")) + .expect("resolves"); + } + + /// Drain and wait for the relay to exit. + async fn stop(self) { + self.trigger.start(); + kio::clock::timeout(TIMEOUT, self.running) + .await + .expect("relay exits") + .expect("relay task") + .expect("relay run"); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_draining_relay_drains_the_links_it_dialed() { + kio::clock::host_time(); // real relays on real sockets + moq_tokio::crypto::install_default().unwrap(); + + // A and C first, until each learns the other's broadcast over the one link between + // them: their route exchange is then up, so each forwards L's routes to the other as + // it learns them, which is what a DRAIN for L is later sent along. + let a = Node::start("a", &[], Duration::ZERO).await; + let c = Node::start("c", &[a.port], Duration::ZERO).await; + let _a = a.publish("a"); + let _c = c.publish("c"); + a.learns("c").await; + c.learns("a").await; + + // Then L, dialing both, until each neighbour takes L's route straight from L and so + // has forwarded it to the other. + let l = Node::start("l", &[a.port, c.port], DRAIN).await; + let _l = l.publish("l"); + kio::clock::timeout(TIMEOUT, async { + while l.active() < 2 { + tokio::task::yield_now().await; + } + }) + .await + .expect("L dials both"); + a.direct("l").await; + c.direct("l").await; + + let started = kio::clock::now(); + l.trigger.start(); + + // Each neighbour hears the GOAWAY on the link L dialed and tells the other, unless the + // other told it first: a DRAIN never goes back to the neighbour it came from. Either + // way the link between them carries one, and each one sent arrives. + let told = || { + let (a, c) = (a.routes(), c.routes()); + let (a_sent, c_sent) = (a.sent.count(Withdrawal::Drain), c.sent.count(Withdrawal::Drain)); + a_sent + c_sent > 0 + && a_sent == c.received.count(Withdrawal::Drain) + && c_sent == a.received.count(Withdrawal::Drain) + }; + kio::clock::timeout(TIMEOUT, async { + while !told() { + tokio::task::yield_now().await; + } + }) + .await + .unwrap_or_else(|_| { + panic!( + "no DRAIN for L crossed between A and C: A {:?}, C {:?}", + a.routes(), + c.routes() + ) + }); + + // ...before any link dropped: L holds them open to its deadline. + assert!(started.elapsed() < DRAIN, "the DRAINs took {:?}", started.elapsed()); + for (name, node) in [("A", &a), ("C", &c)] { + let routes = node.routes(); + assert_eq!( + routes.sent.count(Withdrawal::Dead), + 0, + "{name} sent a DEAD before L left" + ); + assert_eq!( + routes.received.count(Withdrawal::Dead), + 0, + "{name} received a DEAD before L left" + ); + } + assert!(!l.running.is_finished(), "L exited before its drain deadline"); + + // The draining relay, which sent the GOAWAY, names itself in no DRAIN. + let sent = l.routes().sent.count(Withdrawal::Drain); + let observer = l.shutdown.clone(); + l.stop().await; + // The links it held to the deadline by design are no session it failed to drain. + assert_eq!(observer.drained().await.forced, 0, "the held links counted as forced"); + assert_eq!(sent, 0, "the draining relay sent a DRAIN"); + assert!(started.elapsed() >= DRAIN, "L left before its drain deadline"); + + c.stop().await; + a.stop().await; +} diff --git a/rs/moq-relay/tests/integration/main.rs b/rs/moq-relay/tests/integration/main.rs index c79c629d03..f140aa0588 100644 --- a/rs/moq-relay/tests/integration/main.rs +++ b/rs/moq-relay/tests/integration/main.rs @@ -22,6 +22,7 @@ mod cluster_unknown; mod connection; #[cfg(feature = "test-support")] mod drain; +mod drain_dialed; mod embed; #[cfg(feature = "test-support")] mod goaway_cluster; diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs index 6e476b5b75..269baca83b 100644 --- a/rs/moq-tokio/src/connection.rs +++ b/rs/moq-tokio/src/connection.rs @@ -17,6 +17,22 @@ use crate::{Addrs, Client, Error}; pub use moq_net::reconnect::Restart; +/// Close a predecessor still finishing a handover, aborting it at its deadline. +async fn close_predecessor(predecessor: Option<(moq_net::Session, kio::clock::Instant)>) -> Result<(), moq_net::Error> { + let Some((session, deadline)) = predecessor else { + return Ok(()); + }; + let abort = session.clone(); + let mut close = std::pin::pin!(session.close()); + tokio::select! { + res = &mut close => res, + _ = kio::clock::sleep_until(deadline) => { + abort.abort(moq_net::Error::GoawayTimeout); + close.await + } + } +} + /// How a failed dial to `url` bears on the address walk. /// /// An address no backend of `client` can dial never works. A status the peer actually sent @@ -1077,44 +1093,92 @@ impl Connection { // Abort stops redials, but the sessions still need their token grants until // the drain completes. Dropping this owner also ends auth if close is cancelled. let running = self.task.running.upgrade(); - // Refuse redials and take the sessions under one lock: see [`CloseGuard`]. - let (session, predecessor) = { - let mut closed = self.task.closed.lock().unwrap(); - *closed = Some(moq_net::Error::Cancel); - let state = self.state.read(); - let session = state.lifecycle.live().map(|live| live.session.clone()); - let predecessor = state.lifecycle.predecessor(); - ( - session, - predecessor.map(|predecessor| (predecessor.session.clone(), predecessor.deadline)), - ) - }; - self.task.handle.abort(); + let (session, predecessor) = self.stop(); let session = async move { match session { Some(session) => session.close().await, None => Ok(()), } }; - let predecessor = async move { - let Some((session, deadline)) = predecessor else { + let (session, predecessor) = + crate::runtime::scope(async { tokio::join!(session, close_predecessor(predecessor)) }).await; + drop(running); + Ok(session.and(predecessor)?) + } + + /// Stop the loop for every clone and drain the live session with an empty-URI + /// GOAWAY ("reconnect to me"), waiting for the peer to close it. A peer relay + /// does not close it on its own: it ranks the session's routes below every other + /// and keeps serving it until this side closes it. + /// + /// At `timeout` the session is closed with [`moq_net::Error::GoawayTimeout`] on + /// this side's own timer, so a peer stalling the GOAWAY's write cannot hold it + /// open. A predecessor still finishing a handover closes as in + /// [`close`](Self::close), aborted by `timeout` if its handover deadline is later, + /// unless no replacement has connected yet and it is the session that serves. + /// Tokens stay presented until both end. Returns `Ok` + /// when nothing was live or the peer left before `timeout`, whatever its close + /// code, and [`moq_net::Error::GoawayTimeout`] when `timeout` closed the session. + pub async fn drain(self, timeout: Duration) -> crate::Result<()> { + let running = self.task.running.upgrade(); + let deadline = kio::clock::now().saturating_add(timeout); + let (session, predecessor) = self.stop(); + // A predecessor that still serves, its replacement not yet connected, drains as the + // live session rather than closing on its handover deadline. A distinct one closes + // by its handover deadline, but no later than this drain's. + let predecessor = predecessor + .filter(|(predecessor, _)| session.as_ref() != Some(predecessor)) + .map(|(predecessor, handover)| (predecessor, handover.min(deadline))); + let session = async move { + let Some(session) = session else { return Ok(()); }; - let abort = session.clone(); - let mut close = std::pin::pin!(session.close()); + // A client never names a redirect, and the loop that could send another is + // stopped, so neither error is reachable; the deadline closes it regardless. + if let Err(err) = session.goaway(moq_net::goaway::Goaway::new().with_timeout(Some(timeout))) { + tracing::warn!(%err, "failed to drain session"); + } tokio::select! { - res = &mut close => res, - _ = kio::clock::sleep_until(deadline) => { - abort.abort(moq_net::Error::GoawayTimeout); - close.await + biased; + _ = session.closed() => Ok(()), + () = kio::clock::sleep_until(deadline) => { + session.abort(moq_net::Error::GoawayTimeout); + session.closed().await; + Err(moq_net::Error::GoawayTimeout) } } }; - let (session, predecessor) = crate::runtime::scope(async { tokio::join!(session, predecessor) }).await; + let (session, predecessor) = + crate::runtime::scope(async { tokio::join!(session, close_predecessor(predecessor)) }).await; drop(running); Ok(session.and(predecessor)?) } + /// Refuse redials for every clone and stop the loop, returning the live session + /// and any predecessor still finishing a handover, with its deadline. Until its + /// replacement connects, the predecessor is the live session itself. + fn stop( + &self, + ) -> ( + Option, + Option<(moq_net::Session, kio::clock::Instant)>, + ) { + // Refuse redials and take the sessions under one lock: see [`CloseGuard`]. + let taken = { + let mut closed = self.task.closed.lock().unwrap(); + *closed = Some(moq_net::Error::Cancel); + let state = self.state.read(); + let session = state.lifecycle.live().map(|live| live.session.clone()); + let predecessor = state + .lifecycle + .predecessor() + .map(|predecessor| (predecessor.session.clone(), predecessor.deadline)); + (session, predecessor) + }; + self.task.handle.abort(); + taken + } + /// Drive moq-net's policy: dial each attempt it asks for and publish what it does. async fn run(shared: &Shared, client: Client, addrs: Addrs) -> crate::Result<()> { let mut client = client; @@ -2958,6 +3022,64 @@ mod tests { assert_eq!(g1.start, 1, "the drained session stopped delivering"); } + /// A drain while a GOAWAY handover still waits for its replacement sends this side's + /// GOAWAY on the session that serves, which is also the handover's predecessor, and + /// holds it to the timeout instead of closing it as the predecessor. + #[tokio::test] + async fn a_drain_mid_handover_holds_the_serving_session() { + kio::clock::host_time(); // real TCP sockets + const WAIT: Duration = Duration::from_secs(10); + const DRAIN: Duration = Duration::from_secs(3); + + let origin = crate::origin::spawn(); + let (addr, mut accepted, _task) = serve(origin).await; + + // Unique to this test: the table is process-wide. + const HOST: &str = "drain-mid-handover.test"; + crate::resolve::hosts::point(HOST, [addr]); + + let mut config = crate::connect::Config::default(); + config.goaway.handover = Some((WAIT * 3).into()); + config.backoff.initial = Some((FAST).into()); + config.backoff.healthy = Some((FAST).into()); + let connection = config + .init(Default::default()) + .unwrap() + .connect(url(&format!("tcp://{HOST}:1/"))); + let monitor = connection.monitor(); + let session = kio::clock::timeout(WAIT, accepted.recv()).await.unwrap().unwrap(); + outlast_fast().await; + + // The replacement can never connect, so the drained session keeps serving. + let dead = dead().await; + crate::resolve::hosts::point(HOST, dead.socket_addrs(|| None).unwrap()); + session.goaway(moq_net::goaway::Goaway::new()).unwrap(); + let mut state = monitor.state(); + while !matches!(state.phase, Phase::Migrating { .. }) { + state = kio::clock::timeout(WAIT, monitor.state_changed(state.revision)) + .await + .unwrap() + .unwrap(); + } + + let started = kio::clock::now(); + let drained = tokio::spawn(connection.drain(DRAIN)); + let draining = session.draining(); + kio::clock::timeout(WAIT, kio::wait(|waiter| draining.poll(waiter))) + .await + .expect("the peer never heard the GOAWAY") + .expect("the session closed without one"); + assert!( + kio::clock::timeout(DRAIN / 2, session.closed()).await.is_err(), + "the serving session closed as its own predecessor" + ); + assert!( + kio::clock::timeout(WAIT, drained).await.unwrap().unwrap().is_err(), + "the drain timeout did not close the session" + ); + assert!(started.elapsed() >= DRAIN, "closed after {:?}", started.elapsed()); + } + /// Under [`Restart::Handoff`] an empty-URI GOAWAY that drains a session before it is /// healthy changes nothing while the peer keeps the session: no migration and no redial. /// Its close then reconnects like any drop. diff --git a/test/drill/mutations/peer-link-never-redialed.patch b/test/drill/mutations/peer-link-never-redialed.patch index 51f60b6bdd..c8a9603a50 100644 --- a/test/drill/mutations/peer-link-never-redialed.patch +++ b/test/drill/mutations/peer-link-never-redialed.patch @@ -6,10 +6,10 @@ # the route for good: the groups caught in the flap and every burst after it # never arrive. A steady link never drops, so only the flap drill depends on it. diff --git a/rs/moq-relay/src/cluster.rs b/rs/moq-relay/src/cluster.rs -index 0ba85ab451..fe18bdbfc0 100644 +index 24082409c0..4371686fd6 100644 --- a/rs/moq-relay/src/cluster.rs +++ b/rs/moq-relay/src/cluster.rs -@@ -2304,6 +2304,11 @@ impl Cluster { +@@ -2346,6 +2346,11 @@ impl Cluster { loop { dial.observe(id, &observation); *connected |= observation.connects > 0; @@ -18,6 +18,6 @@ index 0ba85ab451..fe18bdbfc0 100644 + drop(reconnect); + return std::future::pending().await; + } - observation = monitor.state_changed(observation.revision).await?; - } - } + tokio::select! { + biased; + () = shutdown.started() => break,