Skip to content
Merged
8 changes: 6 additions & 2 deletions doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -384,8 +384,12 @@ asking it to reconnect elsewhere, including any that connect during the drain,
and is force-closed if it is still connected when the window ends. The cluster
links the relay dialed are drained the same way and stop redialing, so a
neighbour hears the relay is draining and moves its routes before the link
closes. A neighbour keeps serving such a link rather than leave it, so a relay
that dialed a peer holds its drain to the end of the window. Accepted
closes. A neighbour keeps serving such a link rather than leave it, so the
relay closes the link itself once its accepted connections have left and
nothing reads through it, or at the end of the window, where a link still open
counts as force-closed. Until then the link keeps announcing the broadcasts of
publishers still connected to the relay, so neighbours don't lose them while
those publishers move. Accepted
connections still handshaking or awaiting authorization hold the drain until
they finish, fail, or reach that same deadline, where any still pending is
refused or closed. Public HTTP/1.1 connections hold the drain from TCP accept
Expand Down
28 changes: 0 additions & 28 deletions quest/m0/nightly/drain-exit.md

This file was deleted.

8 changes: 6 additions & 2 deletions quest/m0/nightly/mesh-replay-violations.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,12 @@ already listed as known and point at existing quests. These have no entry:
- ring of 4: `end a's broadcasts: shown x64`
- ring of 6: `reopen link a-b: unfed (stuck) x74 ... ended: internal error`

The 6-relay rolls that hung (`roll f to new`, `roll b to old`) belong to
[drain-exit](/quest/m0/nightly/drain-exit.md); re-check them once it lands.
The 6-relay rolls that hung (`roll f to new`, `roll b to old`) are still
unexplained: [moq-dev/moq#9213](https://github.com/moq-dev/moq/pull/9213) fixed
the drain that waited out its whole window, and a local ring6 run with 32
broadcasts on it passed with no hang, but `roll f to new` hung on the old
release build, which that drain change never reached. If a Nightly hangs a roll
again, reproduce it with the relay logs the run keeps.
Reproduce each with the replay's seed and fix in moq-net, one cause per
commit; if one turns out to be a known quest's case, add the entry instead.

Expand Down
4 changes: 3 additions & 1 deletion rs/moq-net/src/session.rs
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,9 @@ impl Session {
/// Ready at once while it carries none. A track request still being set up, waiting
/// on authorization or the track's info, counts too. Announcements and the other
/// control streams don't, so a session can be idle while it still serves discovery.
pub(crate) fn poll_idle(&self, waiter: &kio::Waiter) -> Poll<()> {
/// A session that sent a GOAWAY can then close without cutting any reader off.
#[cfg_attr(feature = "async", kio::wrapper)]
pub fn poll_idle(&self, waiter: &kio::Waiter) -> Poll<()> {
self.work.poll_idle(waiter)
}

Expand Down
4 changes: 2 additions & 2 deletions rs/moq-relay/src/cluster.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2319,7 +2319,7 @@ impl Cluster {
}
// Counted by the relay's drain before the first dial, so a drain that starts
// while the session connects still waits for it.
let serving = self.shutdown.serve();
let serving = self.shutdown.serve_link();
// The GOAWAY lifecycle lives in the reconnect loop: on an upstream GOAWAY it
// dials the replacement while the old session keeps serving, so the old
// routes stay attached and the origin hands live tracks over at a group
Expand Down Expand Up @@ -3546,7 +3546,7 @@ mod tests {
}

/// Once the relay drains, no source change replaces, releases, or adds a dial, so
/// each link's own task finishes its drain to the deadline.
/// each link's own task finishes its drain.
#[tokio::test]
async fn mesh_freezes_its_dials_once_the_relay_drains() {
let (trigger, shutdown) = crate::shutdown::Observer::new(Duration::from_secs(10));
Expand Down
94 changes: 64 additions & 30 deletions rs/moq-relay/src/shutdown.rs
Original file line number Diff line number Diff line change
Expand Up @@ -130,8 +130,8 @@ pub struct Tally {
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
#[non_exhaustive]
pub struct Drained {
/// Sessions still open at the deadline that were force-closed. Cluster links the
/// relay dialed are held to the deadline by design, so they are never counted.
/// Sessions still open at the deadline that were force-closed, including cluster
/// links the relay dialed that a reader still read through.
pub forced: usize,
/// Sessions closed strictly before the deadline and still reporting their end at it.
pub ending: usize,
Expand All @@ -148,6 +148,8 @@ struct State {
forced: usize,
/// Sessions already closed before the deadline but still reporting at it.
ending: usize,
/// Dialed cluster links in `tally.open`, so the rest are accepted sessions.
links: usize,
/// The most recent counted session finished reporting.
last_left: Option<Instant>,
}
Expand Down Expand Up @@ -277,20 +279,40 @@ impl Observer {
///
/// A cluster link the relay dials is counted the same way, from before its first dial.
pub fn serve(&self) -> Serving {
self.serve_as(false)
}

/// Count a cluster link this relay dials, as [`serve`](Self::serve) does a session.
pub(crate) fn serve_link(&self) -> Serving {
self.serve_as(true)
}

fn serve_as(&self, link: bool) -> Serving {
let counted = self.deadline().is_none_or(|deadline| Instant::now() < deadline);
if counted {
self.state.send_modify(|state| state.tally.open += 1);
self.state.send_modify(|state| {
state.tally.open += 1;
state.links += usize::from(link);
});
}
Serving {
_active: self.admission.enter(),
count: match counted {
true => Count::Open,
false => Count::Uncounted,
},
link,
shutdown: self.clone(),
}
}

/// Resolve once every accepted session has closed, leaving only dialed links open.
async fn accepted_left(&self) {
let mut state = self.state.subscribe();
// The sender lives in `self`, so this cannot close.
let _ = state.wait_for(|state| state.tally.open == state.links).await;
}

/// The sessions counted so far.
pub fn tally(&self) -> Tally {
self.state.borrow().tally
Expand Down Expand Up @@ -345,7 +367,7 @@ impl Observer {
elapsed: ended.saturating_duration_since(window.started),
};
match drained.forced {
0 => tracing::info!(elapsed = ?drained.elapsed, "drain ended with no session force-closed; exiting"),
0 => tracing::info!(elapsed = ?drained.elapsed, "drain ended with every session gone; exiting"),
forced => {
tracing::warn!(elapsed = ?drained.elapsed, forced, "drain deadline force-closed sessions; exiting")
}
Expand All @@ -354,9 +376,8 @@ impl Observer {
}

/// Count a session as draining until the returned guard drops, and as forced
/// if it is still open at `deadline`. `None` counts a link the drain holds to
/// its end by design, which is never forced.
fn draining(&self, deadline: Option<Instant>) -> Draining {
/// if it is still open at `deadline`.
fn draining(&self, deadline: Instant) -> Draining {
self.state.send_modify(|state| state.draining += 1);
Draining {
state: self.state.clone(),
Expand Down Expand Up @@ -389,7 +410,7 @@ impl Observer {
let remaining = deadline.saturating_duration_since(now);
// Dropped when this returns or is cancelled (a WebSocket driver ending
// first), so the count holds whichever way the session goes.
let _draining = counted.then(|| self.draining(Some(deadline)));
let _draining = counted.then(|| self.draining(deadline));

// "Reconnect to me": the relay is restarting, not moving. An empty URI is
// legal from either side and this runs once per session, so neither error
Expand All @@ -409,14 +430,28 @@ impl Observer {
}

/// Drain a cluster link this relay dialed as [`drain`](Self::drain) does an
/// accepted session: it stops redialing, sends a GOAWAY, and closes at the deadline.
/// accepted session: it stops redialing and sends a GOAWAY. A neighbour never
/// leaves such a link, so it closes once every accepted session has left and
/// nothing reads through it, or at the deadline. Until then it keeps announcing
/// the broadcasts of publishers still moving off this relay, so a neighbour whose
/// only route is this link never sees them withdrawn mid-handover. The GOAWAY
/// meanwhile moves the neighbour's readers to other routes.
async fn drain_link(&self, link: moq_tokio::Connection, counted: bool) {
let now = Instant::now();
let deadline = self.deadline().unwrap_or(now.saturating_add(self.drain_timeout));
// A neighbour keeps serving a link that sent it a GOAWAY, so reaching the
// deadline is how this link ends, not a session that failed to leave.
let _draining = counted.then(|| self.draining(None));
if let Err(err) = link.drain(deadline.saturating_duration_since(now)).await {
let _draining = counted.then(|| self.draining(deadline));
let hold = link.hold();
let drain = link.drain(deadline.saturating_duration_since(now));
tokio::pin!(drain);
let res = tokio::select! {
biased;
res = &mut drain => res,
() = self.accepted_left() => {
drop(hold);
drain.await
}
};
if let Err(err) = res {
tracing::debug!(%err, "dialed link did not leave before the drain deadline");
}
}
Expand All @@ -429,6 +464,8 @@ pub struct Serving {
// admission prevents while it is held before the SETUP reply.
_active: Option<moq_net::server::Admitted>,
count: Count,
/// A cluster link this relay dialed, counted in [`State::links`] while open.
link: bool,
pub(crate) shutdown: Observer,
}

Expand Down Expand Up @@ -461,9 +498,11 @@ impl Serving {
return;
}
self.count = Count::Ending(Instant::now());
let link = self.link;
self.shutdown.state.send_modify(|state| {
state.tally.open -= 1;
state.tally.ending += 1;
state.links -= usize::from(link);
});
}
}
Expand All @@ -482,7 +521,10 @@ impl Drop for Serving {
self.shutdown.state.send_modify(|state| {
match closed {
Some(_) => state.tally.ending -= 1,
None => state.tally.open -= 1,
None => {
state.tally.open -= 1;
state.links -= usize::from(self.link);
}
}
state.ending += usize::from(ending);
state.last_left = Some(state.last_left.map_or(now, |last| last.max(now)));
Expand All @@ -493,15 +535,15 @@ impl Drop for Serving {
/// A session sent a GOAWAY; see [`Observer::draining`].
struct Draining {
state: watch::Sender<State>,
/// When the session counts as forced, or `None` for a link held to the end.
deadline: Option<Instant>,
/// When the session counts as forced.
deadline: Instant,
}

impl Drop for Draining {
fn drop(&mut self) {
// The force-close fires at `deadline`, so every session it closes is
// counted. A peer leaving in that sliver was still there at the deadline too.
let forced = self.deadline.is_some_and(|deadline| Instant::now() >= deadline);
let forced = Instant::now() >= self.deadline;
self.state.send_modify(|state| {
state.draining -= 1;
state.forced += usize::from(forced);
Expand Down Expand Up @@ -823,7 +865,7 @@ mod tests {
let window = Duration::from_secs(10);
let (trigger, shutdown) = Observer::new(window);
let mut established = shutdown.serve();
let draining = shutdown.draining(Some(kio::clock::now().saturating_add(window)));
let draining = shutdown.draining(kio::clock::now().saturating_add(window));
trigger.start();
kio::clock::sleep(window / 2).await;
established.closed();
Expand Down Expand Up @@ -870,30 +912,22 @@ mod tests {
trigger.start();
let deadline = shutdown.deadline().expect("started");

let left = shutdown.draining(Some(deadline));
let straggler = shutdown.draining(Some(deadline));
let mut link_serving = shutdown.serve();
let link = shutdown.draining(None);
assert_eq!(shutdown.draining_count(), 3);
let left = shutdown.draining(deadline);
let straggler = shutdown.draining(deadline);
assert_eq!(shutdown.draining_count(), 2);

kio::clock::sleep(window / 2).await;
left_serving.closed();
drop(left);
kio::clock::sleep(window / 2).await;
drop(straggler);
edge_serving.closed();
drop(link);
link_serving.closed();

let state = *shutdown.state.borrow();
assert_eq!(state.draining, 0);
assert_eq!(
state.forced, 1,
"only the session open at the deadline is forced, not the link held to it"
);
assert_eq!(state.forced, 1, "only the session open at the deadline is forced");
drop(left_serving);
drop(edge_serving);
drop(link_serving);
assert_eq!(
shutdown.drained().await,
Drained {
Expand Down
Loading