diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index c9615c6965..5ee900db6a 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -384,8 +384,12 @@ asking it to reconnect elsewhere, including any that connect during the drain, and is force-closed if it is still connected when the window ends. The cluster links the relay dialed are drained the same way and stop redialing, so a neighbour hears the relay is draining and moves its routes before the link -closes. A neighbour keeps serving such a link rather than leave it, so a relay -that dialed a peer holds its drain to the end of the window. Accepted +closes. A neighbour keeps serving such a link rather than leave it, so the +relay closes the link itself once its accepted connections have left and +nothing reads through it, or at the end of the window, where a link still open +counts as force-closed. Until then the link keeps announcing the broadcasts of +publishers still connected to the relay, so neighbours don't lose them while +those publishers move. Accepted connections still handshaking or awaiting authorization hold the drain until they finish, fail, or reach that same deadline, where any still pending is refused or closed. Public HTTP/1.1 connections hold the drain from TCP accept diff --git a/quest/m0/nightly/drain-exit.md b/quest/m0/nightly/drain-exit.md deleted file mode 100644 index 1a742fb5ba..0000000000 --- a/quest/m0/nightly/drain-exit.md +++ /dev/null @@ -1,28 +0,0 @@ -# [M] A draining relay exits once its last session leaves - -## Goal - -A relay told to drain exits soon after its last session leaves, so -`Tests (test drain)` and the 6-relay mesh replay's rolls pass again. - -## Plan - -Found by the 2026-11-19 triage of Nightly run -[44562804083](https://github.com/moq-dev/moq/actions/runs/44562804083); the drain test passed on 2026-11-18. - -``` -error: relay A was still running 10s after its last session left -``` - -The 6-relay mesh replay hung the same night on `roll f to new: hung x1` and -`roll b to old: hung x1`, which a relay that never exits would explain. -Likely cause: [moq-dev/moq#9158](https://github.com/moq-dev/moq/pull/9158) -(landed 2026-11-19 10:52 UTC), which holds dialed cluster links until the peer -closes or the deadline passes. Confirm by reverting it locally first. The fix -keeps what #9158 wanted for a live relay while letting a draining one close -its dialed links once nothing needs them. - -Caused by: [moq-dev/moq#9158](https://github.com/moq-dev/moq/pull/9158), if -confirmed. - -Public API: none expected. Wire: none. diff --git a/quest/m0/nightly/mesh-replay-violations.md b/quest/m0/nightly/mesh-replay-violations.md index 73da9a9acc..66a3e0107f 100644 --- a/quest/m0/nightly/mesh-replay-violations.md +++ b/quest/m0/nightly/mesh-replay-violations.md @@ -16,8 +16,12 @@ already listed as known and point at existing quests. These have no entry: - ring of 4: `end a's broadcasts: shown x64` - ring of 6: `reopen link a-b: unfed (stuck) x74 ... ended: internal error` -The 6-relay rolls that hung (`roll f to new`, `roll b to old`) belong to -[drain-exit](/quest/m0/nightly/drain-exit.md); re-check them once it lands. +The 6-relay rolls that hung (`roll f to new`, `roll b to old`) are still +unexplained: [moq-dev/moq#9213](https://github.com/moq-dev/moq/pull/9213) fixed +the drain that waited out its whole window, and a local ring6 run with 32 +broadcasts on it passed with no hang, but `roll f to new` hung on the old +release build, which that drain change never reached. If a Nightly hangs a roll +again, reproduce it with the relay logs the run keeps. Reproduce each with the replay's seed and fix in moq-net, one cause per commit; if one turns out to be a known quest's case, add the entry instead. diff --git a/rs/moq-net/src/session.rs b/rs/moq-net/src/session.rs index d7cdf506f0..b791580a5e 100644 --- a/rs/moq-net/src/session.rs +++ b/rs/moq-net/src/session.rs @@ -459,7 +459,9 @@ impl Session { /// Ready at once while it carries none. A track request still being set up, waiting /// on authorization or the track's info, counts too. Announcements and the other /// control streams don't, so a session can be idle while it still serves discovery. - pub(crate) fn poll_idle(&self, waiter: &kio::Waiter) -> Poll<()> { + /// A session that sent a GOAWAY can then close without cutting any reader off. + #[cfg_attr(feature = "async", kio::wrapper)] + pub fn poll_idle(&self, waiter: &kio::Waiter) -> Poll<()> { self.work.poll_idle(waiter) } diff --git a/rs/moq-relay/src/cluster.rs b/rs/moq-relay/src/cluster.rs index a715f01bca..0691aee23b 100644 --- a/rs/moq-relay/src/cluster.rs +++ b/rs/moq-relay/src/cluster.rs @@ -2319,7 +2319,7 @@ impl Cluster { } // Counted by the relay's drain before the first dial, so a drain that starts // while the session connects still waits for it. - let serving = self.shutdown.serve(); + let serving = self.shutdown.serve_link(); // The GOAWAY lifecycle lives in the reconnect loop: on an upstream GOAWAY it // dials the replacement while the old session keeps serving, so the old // routes stay attached and the origin hands live tracks over at a group @@ -3546,7 +3546,7 @@ mod tests { } /// Once the relay drains, no source change replaces, releases, or adds a dial, so - /// each link's own task finishes its drain to the deadline. + /// each link's own task finishes its drain. #[tokio::test] async fn mesh_freezes_its_dials_once_the_relay_drains() { let (trigger, shutdown) = crate::shutdown::Observer::new(Duration::from_secs(10)); diff --git a/rs/moq-relay/src/shutdown.rs b/rs/moq-relay/src/shutdown.rs index f85e385ecd..e6e5059f85 100644 --- a/rs/moq-relay/src/shutdown.rs +++ b/rs/moq-relay/src/shutdown.rs @@ -130,8 +130,8 @@ pub struct Tally { #[derive(Clone, Copy, Debug, PartialEq, Eq)] #[non_exhaustive] pub struct Drained { - /// Sessions still open at the deadline that were force-closed. Cluster links the - /// relay dialed are held to the deadline by design, so they are never counted. + /// Sessions still open at the deadline that were force-closed, including cluster + /// links the relay dialed that a reader still read through. pub forced: usize, /// Sessions closed strictly before the deadline and still reporting their end at it. pub ending: usize, @@ -148,6 +148,8 @@ struct State { forced: usize, /// Sessions already closed before the deadline but still reporting at it. ending: usize, + /// Dialed cluster links in `tally.open`, so the rest are accepted sessions. + links: usize, /// The most recent counted session finished reporting. last_left: Option, } @@ -277,9 +279,21 @@ impl Observer { /// /// A cluster link the relay dials is counted the same way, from before its first dial. pub fn serve(&self) -> Serving { + self.serve_as(false) + } + + /// Count a cluster link this relay dials, as [`serve`](Self::serve) does a session. + pub(crate) fn serve_link(&self) -> Serving { + self.serve_as(true) + } + + fn serve_as(&self, link: bool) -> Serving { let counted = self.deadline().is_none_or(|deadline| Instant::now() < deadline); if counted { - self.state.send_modify(|state| state.tally.open += 1); + self.state.send_modify(|state| { + state.tally.open += 1; + state.links += usize::from(link); + }); } Serving { _active: self.admission.enter(), @@ -287,10 +301,18 @@ impl Observer { true => Count::Open, false => Count::Uncounted, }, + link, shutdown: self.clone(), } } + /// Resolve once every accepted session has closed, leaving only dialed links open. + async fn accepted_left(&self) { + let mut state = self.state.subscribe(); + // The sender lives in `self`, so this cannot close. + let _ = state.wait_for(|state| state.tally.open == state.links).await; + } + /// The sessions counted so far. pub fn tally(&self) -> Tally { self.state.borrow().tally @@ -345,7 +367,7 @@ impl Observer { elapsed: ended.saturating_duration_since(window.started), }; match drained.forced { - 0 => tracing::info!(elapsed = ?drained.elapsed, "drain ended with no session force-closed; exiting"), + 0 => tracing::info!(elapsed = ?drained.elapsed, "drain ended with every session gone; exiting"), forced => { tracing::warn!(elapsed = ?drained.elapsed, forced, "drain deadline force-closed sessions; exiting") } @@ -354,9 +376,8 @@ impl Observer { } /// Count a session as draining until the returned guard drops, and as forced - /// if it is still open at `deadline`. `None` counts a link the drain holds to - /// its end by design, which is never forced. - fn draining(&self, deadline: Option) -> Draining { + /// if it is still open at `deadline`. + fn draining(&self, deadline: Instant) -> Draining { self.state.send_modify(|state| state.draining += 1); Draining { state: self.state.clone(), @@ -389,7 +410,7 @@ impl Observer { let remaining = deadline.saturating_duration_since(now); // Dropped when this returns or is cancelled (a WebSocket driver ending // first), so the count holds whichever way the session goes. - let _draining = counted.then(|| self.draining(Some(deadline))); + let _draining = counted.then(|| self.draining(deadline)); // "Reconnect to me": the relay is restarting, not moving. An empty URI is // legal from either side and this runs once per session, so neither error @@ -409,14 +430,28 @@ impl Observer { } /// Drain a cluster link this relay dialed as [`drain`](Self::drain) does an - /// accepted session: it stops redialing, sends a GOAWAY, and closes at the deadline. + /// accepted session: it stops redialing and sends a GOAWAY. A neighbour never + /// leaves such a link, so it closes once every accepted session has left and + /// nothing reads through it, or at the deadline. Until then it keeps announcing + /// the broadcasts of publishers still moving off this relay, so a neighbour whose + /// only route is this link never sees them withdrawn mid-handover. The GOAWAY + /// meanwhile moves the neighbour's readers to other routes. async fn drain_link(&self, link: moq_tokio::Connection, counted: bool) { let now = Instant::now(); let deadline = self.deadline().unwrap_or(now.saturating_add(self.drain_timeout)); - // A neighbour keeps serving a link that sent it a GOAWAY, so reaching the - // deadline is how this link ends, not a session that failed to leave. - let _draining = counted.then(|| self.draining(None)); - if let Err(err) = link.drain(deadline.saturating_duration_since(now)).await { + let _draining = counted.then(|| self.draining(deadline)); + let hold = link.hold(); + let drain = link.drain(deadline.saturating_duration_since(now)); + tokio::pin!(drain); + let res = tokio::select! { + biased; + res = &mut drain => res, + () = self.accepted_left() => { + drop(hold); + drain.await + } + }; + if let Err(err) = res { tracing::debug!(%err, "dialed link did not leave before the drain deadline"); } } @@ -429,6 +464,8 @@ pub struct Serving { // admission prevents while it is held before the SETUP reply. _active: Option, count: Count, + /// A cluster link this relay dialed, counted in [`State::links`] while open. + link: bool, pub(crate) shutdown: Observer, } @@ -461,9 +498,11 @@ impl Serving { return; } self.count = Count::Ending(Instant::now()); + let link = self.link; self.shutdown.state.send_modify(|state| { state.tally.open -= 1; state.tally.ending += 1; + state.links -= usize::from(link); }); } } @@ -482,7 +521,10 @@ impl Drop for Serving { self.shutdown.state.send_modify(|state| { match closed { Some(_) => state.tally.ending -= 1, - None => state.tally.open -= 1, + None => { + state.tally.open -= 1; + state.links -= usize::from(self.link); + } } state.ending += usize::from(ending); state.last_left = Some(state.last_left.map_or(now, |last| last.max(now))); @@ -493,15 +535,15 @@ impl Drop for Serving { /// A session sent a GOAWAY; see [`Observer::draining`]. struct Draining { state: watch::Sender, - /// When the session counts as forced, or `None` for a link held to the end. - deadline: Option, + /// When the session counts as forced. + deadline: Instant, } impl Drop for Draining { fn drop(&mut self) { // The force-close fires at `deadline`, so every session it closes is // counted. A peer leaving in that sliver was still there at the deadline too. - let forced = self.deadline.is_some_and(|deadline| Instant::now() >= deadline); + let forced = Instant::now() >= self.deadline; self.state.send_modify(|state| { state.draining -= 1; state.forced += usize::from(forced); @@ -823,7 +865,7 @@ mod tests { let window = Duration::from_secs(10); let (trigger, shutdown) = Observer::new(window); let mut established = shutdown.serve(); - let draining = shutdown.draining(Some(kio::clock::now().saturating_add(window))); + let draining = shutdown.draining(kio::clock::now().saturating_add(window)); trigger.start(); kio::clock::sleep(window / 2).await; established.closed(); @@ -870,11 +912,9 @@ mod tests { trigger.start(); let deadline = shutdown.deadline().expect("started"); - let left = shutdown.draining(Some(deadline)); - let straggler = shutdown.draining(Some(deadline)); - let mut link_serving = shutdown.serve(); - let link = shutdown.draining(None); - assert_eq!(shutdown.draining_count(), 3); + let left = shutdown.draining(deadline); + let straggler = shutdown.draining(deadline); + assert_eq!(shutdown.draining_count(), 2); kio::clock::sleep(window / 2).await; left_serving.closed(); @@ -882,18 +922,12 @@ mod tests { kio::clock::sleep(window / 2).await; drop(straggler); edge_serving.closed(); - drop(link); - link_serving.closed(); let state = *shutdown.state.borrow(); assert_eq!(state.draining, 0); - assert_eq!( - state.forced, 1, - "only the session open at the deadline is forced, not the link held to it" - ); + assert_eq!(state.forced, 1, "only the session open at the deadline is forced"); drop(left_serving); drop(edge_serving); - drop(link_serving); assert_eq!( shutdown.drained().await, Drained { diff --git a/rs/moq-relay/tests/integration/drain_dialed.rs b/rs/moq-relay/tests/integration/drain_dialed.rs index 5ff592d1a9..615b4c3557 100644 --- a/rs/moq-relay/tests/integration/drain_dialed.rs +++ b/rs/moq-relay/tests/integration/drain_dialed.rs @@ -1,11 +1,12 @@ //! A draining relay drains the cluster links it dialed, as it does those it accepted: -//! each peer hears its GOAWAY and tells the others the relay is draining, well before -//! any link drops. +//! each peer hears its GOAWAY and tells the others the relay is draining, and the +//! relay closes each link once its accepted sessions have left and nothing reads +//! through it. //! //! Three real relays on loopback: C dials A, and L dials both. L drains, so A and C //! each hear L's GOAWAY themselves, and the first to hear it sends the other a DRAIN -//! for it, while L, which sent the GOAWAY, sends none. Every link stays up until L's -//! drain deadline, so no relay counts a DEAD until then. +//! for it, while L, which sent the GOAWAY, sends none. A reader at A reads L's +//! broadcast, so a link stays up while it reads, and L exits once it stops. use std::time::Duration; @@ -16,8 +17,10 @@ use moq_relay::{Config, Relay, shutdown}; /// Only a hang reaches it. const TIMEOUT: Duration = Duration::from_secs(10); -/// How long L holds its drain open. Long enough that every DRAIN lands well inside it. -const DRAIN: Duration = Duration::from_secs(2); +/// A millisecond track: the frames written here carry timestamps. +fn timed() -> moq_net::track::Info { + moq_net::track::Info::default().with_timescale(moq_net::Timescale::MILLI) +} /// One relay running on the test runtime. struct Node { @@ -147,87 +150,234 @@ impl Node { } } -#[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn a_draining_relay_drains_the_links_it_dialed() { - kio::clock::host_time(); // real relays on real sockets - moq_tokio::crypto::install_default().unwrap(); - - // A and C first, until each learns the other's broadcast over the one link between - // them: their route exchange is then up, so each forwards L's routes to the other as - // it learns them, which is what a DRAIN for L is later sent along. - let a = Node::start("a", &[], Duration::ZERO).await; - let c = Node::start("c", &[a.port], Duration::ZERO).await; - let _a = a.publish("a"); - let _c = c.publish("c"); - a.learns("c").await; - c.learns("a").await; - - // Then L, dialing both, until each neighbour takes L's route straight from L and so - // has forwarded it to the other. - let l = Node::start("l", &[a.port, c.port], DRAIN).await; - let _l = l.publish("l"); - kio::clock::timeout(TIMEOUT, async { - while l.active() < 2 { - tokio::task::yield_now().await; - } - }) - .await - .expect("L dials both"); - a.direct("l").await; - c.direct("l").await; +/// A and C linked, then L dialing both and draining over `drain`, with a reader at A +/// reading L's broadcast `l` through L's links. +struct Mesh { + a: Node, + c: Node, + l: Node, + _broadcasts: [moq_net::origin::Route; 3], + track: moq_net::track::Producer, + reader: moq_net::track::Subscriber, +} - let started = kio::clock::now(); - l.trigger.start(); +impl Mesh { + async fn start(drain: Duration) -> Self { + kio::clock::host_time(); // real relays on real sockets + moq_tokio::crypto::install_default().unwrap(); + + // A and C first, until each learns the other's broadcast over the one link between + // them: their route exchange is then up, so each forwards L's routes to the other as + // it learns them, which is what a DRAIN for L is later sent along. + let a = Node::start("a", &[], Duration::ZERO).await; + let c = Node::start("c", &[a.port], Duration::ZERO).await; + let broadcast_a = a.publish("a"); + let broadcast_c = c.publish("c"); + a.learns("c").await; + c.learns("a").await; + + // Then L, dialing both, until each neighbour takes L's route straight from L and so + // has forwarded it to the other. + let l = Node::start("l", &[a.port, c.port], drain).await; + let broadcast_l = l.publish("l"); + let track = broadcast_l.broadcast().create_track("video", timed()).unwrap(); + kio::clock::timeout(TIMEOUT, async { + while l.active() < 2 { + tokio::task::yield_now().await; + } + }) + .await + .expect("L dials both"); + a.direct("l").await; + c.direct("l").await; + + let remote = kio::clock::timeout(TIMEOUT, a.origin.consume().follow("l", None).unwrap().available()) + .await + .expect("A learns l") + .expect("resolves") + .expect("announced"); + let reader = remote.track("video").unwrap().subscribe(None).await.unwrap(); + let mut mesh = Self { + a, + c, + l, + _broadcasts: [broadcast_a, broadcast_c, broadcast_l], + track, + reader, + }; + mesh.reads().await; + mesh + } - // Each neighbour hears the GOAWAY on the link L dialed and tells the other, unless the - // other told it first: a DRAIN never goes back to the neighbour it came from. Either - // way the link between them carries one, and each one sent arrives. - let told = || { - let (a, c) = (a.routes(), c.routes()); - let (a_sent, c_sent) = (a.sent.count(Withdrawal::Drain), c.sent.count(Withdrawal::Drain)); - a_sent + c_sent > 0 - && a_sent == c.received.count(Withdrawal::Drain) - && c_sent == a.received.count(Withdrawal::Drain) - }; - kio::clock::timeout(TIMEOUT, async { - while !told() { - tokio::task::yield_now().await; + /// Write a group at L and wait for the reader at A to receive it. + async fn reads(&mut self) { + let sequence = self.track.largest().map_or(0, |largest| largest.group + 1); + let mut group = self + .track + .append_group(moq_net::Timestamp::from_millis(sequence).unwrap()) + .unwrap(); + group + .write_frame(moq_net::Timestamp::from_millis(sequence).unwrap(), b"l".as_ref()) + .unwrap(); + group.finish().unwrap(); + loop { + let group = kio::clock::timeout(TIMEOUT, self.reader.recv_group()) + .await + .expect("the reader at A starved") + .expect("the read failed") + .expect("the track ended"); + if group.start >= sequence { + return; + } } - }) - .await - .unwrap_or_else(|_| { - panic!( - "no DRAIN for L crossed between A and C: A {:?}, C {:?}", - a.routes(), - c.routes() - ) - }); - - // ...before any link dropped: L holds them open to its deadline. - assert!(started.elapsed() < DRAIN, "the DRAINs took {:?}", started.elapsed()); - for (name, node) in [("A", &a), ("C", &c)] { - let routes = node.routes(); - assert_eq!( - routes.sent.count(Withdrawal::Dead), - 0, - "{name} sent a DEAD before L left" - ); - assert_eq!( - routes.received.count(Withdrawal::Dead), - 0, - "{name} received a DEAD before L left" - ); } - assert!(!l.running.is_finished(), "L exited before its drain deadline"); + + /// Each neighbour hears the GOAWAY on the link L dialed and tells the other, unless the + /// other told it first: a DRAIN never goes back to the neighbour it came from. Either + /// way the link between them carries one, and each one sent arrives. + async fn told(&self) { + let told = || { + let (a, c) = (self.a.routes(), self.c.routes()); + let (a_sent, c_sent) = (a.sent.count(Withdrawal::Drain), c.sent.count(Withdrawal::Drain)); + a_sent + c_sent > 0 + && a_sent == c.received.count(Withdrawal::Drain) + && c_sent == a.received.count(Withdrawal::Drain) + }; + kio::clock::timeout(TIMEOUT, async { + while !told() { + tokio::task::yield_now().await; + } + }) + .await + .unwrap_or_else(|_| { + panic!( + "no DRAIN for L crossed between A and C: A {:?}, C {:?}", + self.a.routes(), + self.c.routes() + ) + }); + } +} + +/// L holds a link while the reader at A reads through it, and exits once it stops, +/// long before its deadline, with nothing force-closed. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_draining_relay_closes_its_dialed_links_once_nothing_reads_them() { + // Far past every wait below, so only links closing on their own let L exit in time. + const DRAIN: Duration = Duration::from_secs(600); + let mut mesh = Mesh::start(DRAIN).await; + + let started = kio::clock::now(); + mesh.l.trigger.start(); + mesh.told().await; + + // The reader still reads, so L still serves it. + mesh.reads().await; + assert!(!mesh.l.running.is_finished(), "L exited under a reader"); // The draining relay, which sent the GOAWAY, names itself in no DRAIN. - let sent = l.routes().sent.count(Withdrawal::Drain); + assert_eq!( + mesh.l.routes().sent.count(Withdrawal::Drain), + 0, + "the draining relay sent a DRAIN" + ); + + let Mesh { a, c, l, reader, .. } = mesh; + drop(reader); let observer = l.shutdown.clone(); l.stop().await; - // The links it held to the deadline by design are no session it failed to drain. - assert_eq!(observer.drained().await.forced, 0, "the held links counted as forced"); - assert_eq!(sent, 0, "the draining relay sent a DRAIN"); + assert!(started.elapsed() < DRAIN, "L held its links to the deadline"); + assert_eq!( + observer.drained().await.forced, + 0, + "a link closed on its own counted as forced" + ); + + c.stop().await; + a.stop().await; +} + +/// A link a reader still reads through at the deadline is closed by it, and counted as +/// forced like any session that never left. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_dialed_link_still_read_at_the_deadline_is_forced() { + const DRAIN: Duration = Duration::from_secs(2); + let mesh = Mesh::start(DRAIN).await; + + let started = kio::clock::now(); + let observer = mesh.l.shutdown.clone(); + let Mesh { a, c, l, reader, .. } = mesh; + l.stop().await; assert!(started.elapsed() >= DRAIN, "L left before its drain deadline"); + assert!( + observer.drained().await.forced > 0, + "the read link was not counted as forced" + ); + + drop(reader); + c.stop().await; + a.stop().await; +} + +/// A publisher's session to the relay at `port`, publishing the broadcast `name`. +fn publisher(port: u16, name: &str) -> (moq_tokio::Connection, moq_net::origin::Route) { + let mut config = moq_tokio::connect::Config::default(); + config.bind = Some("127.0.0.1:0".parse().unwrap()); + config.tls.insecure = Some(true); + config.websocket.enabled = Some(false); + config.version = vec!["moq-lite-07-wip".parse().unwrap()]; + let origin = moq_tokio::origin::spawn(); + let broadcast = origin.route(name, moq_net::origin::RouteOptions::default()).unwrap(); + let url: url::Url = format!("https://127.0.0.1:{port}/").parse().unwrap(); + let connection = config + .init(Default::default()) + .unwrap() + .with_publisher(origin.consume()) + // Redials the draining relay once and keeps serving there, as a publisher + // whose replacement relay has yet to come up. + .with_restart(moq_tokio::Restart::Handoff) + .connect(url); + (connection, broadcast) +} + +/// A link nothing reads through still announces the broadcast of a publisher that is yet +/// to leave the draining relay, so a neighbour whose only route is that link keeps it, +/// and the relay exits once the publisher leaves. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_dialed_link_announces_until_the_relays_publishers_leave() { + const DRAIN: Duration = Duration::from_secs(600); + let Mesh { a, c, l, reader, .. } = Mesh::start(DRAIN).await; + drop(reader); + + let (publisher, _broadcast) = publisher(l.port, "p"); + a.learns("p").await; + let mut routes = a.origin.consume().routes(); + + let started = kio::clock::now(); + l.trigger.start(); + // Long past an idle link's close, were the publisher not held to. + let withdrawn = kio::clock::timeout(Duration::from_secs(2), async { + loop { + let event = routes.next().await.expect("origin open"); + let gone = matches!(event.kind, RouteKind::End | RouteKind::Offline); + if gone && event.announce.prefix.as_str() == "p" { + return; + } + } + }) + .await; + assert!(withdrawn.is_err(), "A lost p while its publisher was still on L"); + assert!(!l.running.is_finished(), "L exited under a publisher"); + + publisher.close().await.expect("publisher closes"); + let observer = l.shutdown.clone(); + l.stop().await; + assert!(started.elapsed() < DRAIN, "L held its links to the deadline"); + assert_eq!( + observer.drained().await.forced, + 0, + "a link closed on its own counted as forced" + ); c.stop().await; a.stop().await; diff --git a/rs/moq-tokio/src/connection.rs b/rs/moq-tokio/src/connection.rs index 36b97db871..7462e8f4fd 100644 --- a/rs/moq-tokio/src/connection.rs +++ b/rs/moq-tokio/src/connection.rs @@ -19,9 +19,14 @@ pub use moq_net::reconnect::Restart; /// Close a predecessor still finishing a handover, aborting it at its deadline. async fn close_predecessor(predecessor: Option<(moq_net::Session, kio::clock::Instant)>) -> Result<(), moq_net::Error> { - let Some((session, deadline)) = predecessor else { - return Ok(()); - }; + match predecessor { + Some((session, deadline)) => close_by(session, deadline).await, + None => Ok(()), + } +} + +/// Close `session` after draining its queued data, aborting it at `deadline`. +async fn close_by(session: moq_net::Session, deadline: kio::clock::Instant) -> Result<(), moq_net::Error> { let abort = session.clone(); let mut close = std::pin::pin!(session.close()); tokio::select! { @@ -933,6 +938,43 @@ struct Task { /// A graceful close retains the loop's resources while its sessions drain. running: std::sync::Weak, closed: CloseGuard, + /// How many [`Hold`]s keep a drained session open while it is idle. + holds: kio::Producer, +} + +/// Ready once no [`Hold`] is left and `idle` is ready, checked under the hold count's +/// lock so a [`Hold`] taken meanwhile waits for the decision instead of slipping past it. +fn poll_released( + holds: &kio::Consumer, + waiter: &kio::Waiter, + idle: impl FnOnce(&kio::Waiter) -> Poll<()>, +) -> Poll<()> { + let mut idle = Some(idle); + match holds.poll(waiter, |count| match **count { + 0 => idle.take().expect("polled once")(waiter), + _ => Poll::Pending, + }) { + Poll::Ready(Ok(())) => Poll::Ready(()), + // Every Hold keeps the count open, so a closed one has none left. + Poll::Ready(Err(_)) => idle.map_or(Poll::Pending, |idle| idle(waiter)), + Poll::Pending => Poll::Pending, + } +} + +/// Keeps a [`Connection::drain`] from closing its idle session until this drops. +/// +/// From [`Connection::hold`]. The drain still closes the session at its timeout. +#[must_use = "the drain may close the idle session once this drops"] +pub struct Hold { + holds: kio::Producer, +} + +impl Drop for Hold { + fn drop(&mut self) { + if let Ok(mut holds) = self.holds.write() { + *holds -= 1; + } + } } /// Resources owned by the loop, retained by a graceful close until its sessions drain. @@ -1013,6 +1055,7 @@ impl Connection { handle: task, running: task_running, closed, + holds: kio::Producer::default(), }), state, auth, @@ -1076,22 +1119,37 @@ impl Connection { Ok(session.and(predecessor)?) } + /// Keep a [`drain`](Self::drain) from closing the session for carrying nothing + /// until the returned guard drops, such as while the session still announces + /// broadcasts whose publishers have yet to move elsewhere. + pub fn hold(&self) -> Hold { + let holds = self.task.holds.clone(); + if let Ok(mut count) = holds.write() { + *count += 1; + } + Hold { holds } + } + /// Stop the loop for every clone and drain the live session with an empty-URI - /// GOAWAY ("reconnect to me"), waiting for the peer to close it. A peer relay - /// does not close it on its own: it ranks the session's routes below every other - /// and keeps serving it until this side closes it. + /// GOAWAY ("reconnect to me"), until the peer closes it or it carries no + /// subscription or fetch ([`moq_net::Session::idle`]) and no [`Hold`] is left, + /// when this side closes it as [`close`](Self::close) does. A peer relay never + /// closes it on its own: it ranks the session's routes below every other and + /// keeps serving it, so its readers move to another route and the session goes idle. /// /// At `timeout` the session is closed with [`moq_net::Error::GoawayTimeout`] on - /// this side's own timer, so a peer stalling the GOAWAY's write cannot hold it - /// open. A predecessor still finishing a handover closes as in - /// [`close`](Self::close), aborted by `timeout` if its handover deadline is later, - /// unless no replacement has connected yet and it is the session that serves. - /// Tokens stay presented until both end. Returns `Ok` + /// this side's own timer, so a peer stalling the GOAWAY's write, or still reading + /// through the session, cannot hold it open. A predecessor still finishing a + /// handover closes as in [`close`](Self::close), aborted by `timeout` if its + /// handover deadline is later, unless no replacement has connected yet and it is + /// the session that serves. Tokens stay presented until both end. Returns `Ok` /// when nothing was live or the peer left before `timeout`, whatever its close - /// code, and [`moq_net::Error::GoawayTimeout`] when `timeout` closed the session. + /// code, the close's result when the session went idle first, and + /// [`moq_net::Error::GoawayTimeout`] when `timeout` closed the session. pub async fn drain(self, timeout: Duration) -> crate::Result<()> { let running = self.task.running.upgrade(); let deadline = kio::clock::now().saturating_add(timeout); + let holds = self.task.holds.consume(); let (session, predecessor) = self.stop(); // A predecessor that still serves, its replacement not yet connected, drains as the // live session rather than closing on its handover deadline. A distinct one closes @@ -1116,6 +1174,10 @@ impl Connection { session.closed().await; Err(moq_net::Error::GoawayTimeout) } + () = kio::wait(|waiter| poll_released(&holds, waiter, |waiter| session.poll_idle(waiter))) => { + tracing::debug!("drained session carries nothing; closing"); + close_by(session, deadline).await + } } }; let (session, predecessor) = @@ -2989,9 +3051,33 @@ mod tests { assert_eq!(g1.start, 1, "the drained session stopped delivering"); } + /// A subscription to a track `origin` publishes, read through `subscriber`, so the + /// session carrying it is not idle while the returned subscriber lives. + async fn reading( + origin: &moq_net::origin::Producer, + subscriber: &moq_net::origin::Producer, + name: &str, + ) -> ( + moq_net::origin::Route, + moq_net::track::Producer, + moq_net::track::Subscriber, + ) { + const WAIT: Duration = Duration::from_secs(10); + let broadcast = origin.route(name, moq_net::origin::RouteOptions::default()).unwrap(); + let track = broadcast.broadcast().create_track("video", timed()).unwrap(); + let remote = kio::clock::timeout(WAIT, subscriber.consume().follow(name, None).unwrap().available()) + .await + .unwrap() + .unwrap() + .unwrap(); + let sub = remote.track("video").unwrap().subscribe(None).await.unwrap(); + (broadcast, track, sub) + } + /// A drain while a GOAWAY handover still waits for its replacement sends this side's /// GOAWAY on the session that serves, which is also the handover's predecessor, and - /// holds it to the timeout instead of closing it as the predecessor. + /// holds it while it carries a subscription, to the timeout, instead of closing it + /// as the predecessor. #[tokio::test] async fn a_drain_mid_handover_holds_the_serving_session() { kio::clock::host_time(); // real TCP sockets @@ -2999,12 +3085,13 @@ mod tests { const DRAIN: Duration = Duration::from_secs(3); let origin = crate::origin::spawn(); - let (addr, mut accepted, _task) = serve(origin).await; + let (addr, mut accepted, _task) = serve(origin.clone()).await; // Unique to this test: the table is process-wide. const HOST: &str = "drain-mid-handover.test"; crate::resolve::hosts::point(HOST, [addr]); + let subscriber = crate::origin::spawn(); let mut config = crate::connect::Config::default(); config.goaway.handover = Some((WAIT * 3).into()); config.backoff.initial = Some((FAST).into()); @@ -3012,9 +3099,11 @@ mod tests { let connection = config .init(Default::default()) .unwrap() + .with_subscriber(subscriber.clone()) .connect(url(&format!("tcp://{HOST}:1/"))); let monitor = connection.monitor(); let session = kio::clock::timeout(WAIT, accepted.recv()).await.unwrap().unwrap(); + let _reading = reading(&origin, &subscriber, "cam").await; outlast_fast().await; // The replacement can never connect, so the drained session keeps serving. @@ -3047,6 +3136,171 @@ mod tests { assert!(started.elapsed() >= DRAIN, "closed after {:?}", started.elapsed()); } + /// A drained session that a peer relay keeps serving closes once nothing reads through + /// it, well before the drain's timeout, and the drain reports that clean close. + #[tokio::test] + async fn a_drain_closes_the_session_once_it_carries_nothing() { + kio::clock::host_time(); // real TCP sockets + const WAIT: Duration = Duration::from_secs(10); + // Far past every wait below, so only the idle close can end the drain in time. + const DRAIN: Duration = Duration::from_secs(600); + + let origin = crate::origin::spawn(); + let (addr, mut accepted, _task) = serve(origin.clone()).await; + let subscriber = crate::origin::spawn(); + let connection = crate::connect::Config::default() + .init(Default::default()) + .unwrap() + .with_subscriber(subscriber.clone()) + .connect(url(&format!("tcp://{addr}/"))); + let session = kio::clock::timeout(WAIT, accepted.recv()).await.unwrap().unwrap(); + let reading = reading(&origin, &subscriber, "cam").await; + + let drained = tokio::spawn(connection.drain(DRAIN)); + let draining = session.draining(); + kio::clock::timeout(WAIT, kio::wait(|waiter| draining.poll(waiter))) + .await + .expect("the peer never heard the GOAWAY") + .expect("the session closed without one"); + assert!( + kio::clock::timeout(Duration::from_millis(500), session.closed()) + .await + .is_err(), + "closed while a reader still read through it" + ); + + drop(reading); + kio::clock::timeout(WAIT, drained) + .await + .expect("the idle session was held open") + .unwrap() + .expect("the idle close failed"); + kio::clock::timeout(WAIT, session.closed()) + .await + .expect("the peer still holds the session"); + } + + /// A [`Hold`] keeps a drained session open while it carries nothing, and the drain + /// closes it once the hold drops. + #[tokio::test] + async fn a_hold_keeps_an_idle_drained_session_open() { + kio::clock::host_time(); // real TCP sockets + const WAIT: Duration = Duration::from_secs(10); + const DRAIN: Duration = Duration::from_secs(600); + + let origin = crate::origin::spawn(); + let (addr, mut accepted, _task) = serve(origin).await; + let connection = crate::connect::Config::default() + .init(Default::default()) + .unwrap() + .connect(url(&format!("tcp://{addr}/"))); + let session = kio::clock::timeout(WAIT, accepted.recv()).await.unwrap().unwrap(); + let connection = kio::clock::timeout(WAIT, connection.established()) + .await + .unwrap() + .unwrap(); + + let hold = connection.hold(); + let drained = tokio::spawn(connection.drain(DRAIN)); + let draining = session.draining(); + kio::clock::timeout(WAIT, kio::wait(|waiter| draining.poll(waiter))) + .await + .expect("the peer never heard the GOAWAY") + .expect("the session closed without one"); + assert!( + kio::clock::timeout(Duration::from_millis(500), session.closed()) + .await + .is_err(), + "closed while held" + ); + + drop(hold); + kio::clock::timeout(WAIT, drained) + .await + .expect("the released session was held open") + .unwrap() + .expect("the idle close failed"); + } + + /// A [`Hold`] taken while a drain decides the session is idle waits for that decision, + /// so it never slips in between the hold count and the idle check. + #[test] + fn a_hold_taken_mid_decision_waits_for_it() { + let holds = kio::Producer::::default(); + let consumer = holds.consume(); + let order = std::sync::Mutex::new(Vec::new()); + let (trying, started) = std::sync::mpsc::channel(); + std::thread::scope(|scope| { + let decided = poll_released(&consumer, &kio::Waiter::noop(), |_| { + scope.spawn(|| { + trying.send(()).unwrap(); + *holds.write().ok().unwrap() += 1; + order.lock().unwrap().push("held"); + }); + started.recv().unwrap(); + // Give the hold every chance to land; it never can while the decision holds the count. + for _ in 0..10_000 { + if !order.lock().unwrap().is_empty() { + break; + } + std::thread::yield_now(); + } + order.lock().unwrap().push("decided"); + Poll::Ready(()) + }); + assert!(decided.is_ready()); + }); + assert_eq!( + *order.lock().unwrap(), + ["decided", "held"], + "a hold landed mid-decision" + ); + } + + /// A [`Hold`] taken once a drain is waiting on a session's last reader still holds the + /// session open after that reader leaves. + #[tokio::test] + async fn a_hold_taken_mid_drain_still_holds() { + kio::clock::host_time(); // real TCP sockets + const WAIT: Duration = Duration::from_secs(10); + const DRAIN: Duration = Duration::from_secs(600); + + let origin = crate::origin::spawn(); + let (addr, mut accepted, _task) = serve(origin.clone()).await; + let subscriber = crate::origin::spawn(); + let connection = crate::connect::Config::default() + .init(Default::default()) + .unwrap() + .with_subscriber(subscriber.clone()) + .connect(url(&format!("tcp://{addr}/"))); + let session = kio::clock::timeout(WAIT, accepted.recv()).await.unwrap().unwrap(); + let reading = reading(&origin, &subscriber, "cam").await; + + let clone = connection.clone(); + let drained = tokio::spawn(connection.drain(DRAIN)); + let draining = session.draining(); + kio::clock::timeout(WAIT, kio::wait(|waiter| draining.poll(waiter))) + .await + .expect("the peer never heard the GOAWAY") + .expect("the session closed without one"); + + let hold = clone.hold(); + drop(reading); + assert!( + kio::clock::timeout(Duration::from_millis(500), session.closed()) + .await + .is_err(), + "closed while held" + ); + + drop(hold); + kio::clock::timeout(WAIT, drained) + .await + .expect("the released session was held open") + .unwrap() + .expect("the idle close failed"); + } + /// Under [`Restart::Handoff`] an empty-URI GOAWAY that drains a session before it is /// healthy changes nothing while the peer keeps the session: no migration and no redial. /// Its close then reconnects like any drop. diff --git a/rs/moq-tokio/src/lib.rs b/rs/moq-tokio/src/lib.rs index 77595c50e1..55ff1c21ed 100644 --- a/rs/moq-tokio/src/lib.rs +++ b/rs/moq-tokio/src/lib.rs @@ -83,7 +83,7 @@ fn spawn_qmux( // new `pub` item in these modules doesn't silently join it. pub use client::Client; pub use connect::{Addrs, ConnectError}; -pub use connection::{Backoff, Connection, Redirect, Restart}; +pub use connection::{Backoff, Connection, Hold, Redirect, Restart}; pub use error::{Error, Result}; pub use log::{Log, RedactedUrl}; pub use server::{Listener, Server};