Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -381,7 +381,11 @@ drain_timeout = "10s" # Top-level key, as --drain-timeout / MOQ_D

The first SIGTERM or SIGINT starts a drain: every session is sent a GOAWAY
asking it to reconnect elsewhere, including any that connect during the drain,
and is force-closed if it is still connected when the window ends. Accepted
and is force-closed if it is still connected when the window ends. The cluster
links the relay dialed are drained the same way and stop redialing, so a
neighbour hears the relay is draining and moves its routes before the link
closes. A neighbour keeps serving such a link rather than leave it, so a relay
that dialed a peer holds its drain to the end of the window. Accepted
connections still handshaking or awaiting authorization hold the drain until
they finish, fail, or reach that same deadline, where any still pending is
refused or closed. Public HTTP/1.1 connections hold the drain from TCP accept
Expand All @@ -391,7 +395,7 @@ ALPN because extended CONNECT is disabled. A later WebSocket upgrade on the same
keep-alive connection joins the drain when its headers arrive. The internal
listener is unaffected.

The relay exits as soon as every admitted connection has left, or immediately on
The relay exits as soon as every admitted connection and dialed link has left, or immediately on
a second signal. When the window ends, it stops waiting for connections still
pending and exits once every session it closed has reported its `end` to the
auth server, each report bounded by that server's request timeout. Once it commits to exit, WebSocket
Expand All @@ -403,6 +407,8 @@ closed when the window ends. An embedder can take over the signals; see

An embedder's drain alert can page on the `forced` count returned by
[`Observer::drained`](https://docs.rs/moq-relay/latest/moq_relay/shutdown/struct.Observer.html#method.drained).
It leaves out the dialed links the drain holds to its end, so a routine
restart reports none.
Its `ending` count distinguishes sessions already closed before the deadline
but still reporting their end. The live tally separates open sessions from
that cleanup, and the summary's elapsed time stays the same for late observers.
Expand Down
4 changes: 2 additions & 2 deletions doc/bin/relay/http.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,8 @@ A few worth alerting on:
out a release.
- `moq_relay_stale_bytes_total` and friends: content dropped for drifting past
a subscriber's latency budget.
- `moq_relay_draining_sessions`: sessions sent a GOAWAY during a
[shutdown drain](/bin/relay/config#shutdown) that have not left yet.
- `moq_relay_draining_sessions`: sessions and dialed cluster links sent a GOAWAY
during a [shutdown drain](/bin/relay/config#shutdown) that have not left yet.
- `moq_relay_panics_total`: bugs that closed one session instead of the
relay. Each logs an error with the message and the session's span. A panic
in a long-lived task (the origin, the cache governor, the certificate reload,
Expand Down
3 changes: 1 addition & 2 deletions quest/m2/multicast/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,7 @@ children, the two flaws it found,
[a reader resuming on a detour learned after its session closed](/quest/m2/multicast/detour-resume.md)
and [a route moving behind one next hop resending a group](/quest/m2/multicast/resend.md),
whose fix changes the wire,
[the mesh replay](/quest/m2/multicast/mesh-replay.md), and
[draining dialed links](/quest/m2/multicast/drain-dialed-links.md), and on
and [the mesh replay](/quest/m2/multicast/mesh-replay.md), and on
[route layer acceptance](/quest/m2/cluster-routes/acceptance.md) retiring
the lite-06 mitigations. After the cut come warmth's wire
([cache-aware routing](/quest/m3/cache-routing/README.md)),
Expand Down
42 changes: 0 additions & 42 deletions quest/m2/multicast/drain-dialed-links.md

This file was deleted.

1 change: 0 additions & 1 deletion quest/m2/multicast/mesh-replay.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,4 +78,3 @@ Public API: none. Wire: none.
- `rs/moq-net/tests/integration/route_audit/` ([moq-dev/moq#9111](https://github.com/moq-dev/moq/pull/9111)) - the same failures on mock sessions with mocked time
- [Random fault schedules](/quest/m4/cluster-fault-schedules.md) - searches fault timings on the simulated clock
- [A nightly soak](/quest/m4/relay-soak.md) - one real relay under churn
- [A draining relay drains the cluster links it dialed](/quest/m2/multicast/drain-dialed-links.md) - without it, a drained relay's dialed links end with DEAD and no DRAIN
100 changes: 93 additions & 7 deletions rs/moq-relay/src/cluster.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1061,6 +1061,12 @@ pub struct Options {
/// meanwhile, and the first publish carries them.
pub stats_hold: Option<tokio::sync::oneshot::Receiver<()>>,

/// The relay's drain, which every peer link this cluster dialed joins: once it
/// starts, each link stops redialing and sends its peer a GOAWAY, and the drain
/// holds it open until the peer leaves or the deadline passes. `None` closes
/// dialed links only on [`Cluster::close`].
pub shutdown: Option<crate::shutdown::Observer>,

/// How peers are reached in place of the network, for tests on `kio::clock::sim`:
/// every peer dial becomes a [`Dial`](moq_tokio::sim::Dial) the test routes.
/// `None` dials over sockets.
Expand Down Expand Up @@ -1285,6 +1291,11 @@ impl Mesh {

/// Apply one source's change to the dial set.
fn apply(&mut self, command: DialCommand) {
// Once the relay drains, the dial set is frozen: replacing or releasing a dial
// would drop its link mid-drain, and a new one would only return at once.
if self.cluster.shutdown.deadline().is_some() {
return;
}
let mut dial = dialer(&self.cluster, &self.token);
match command {
DialCommand::Api(desired) => self.dialed.reconcile_api(&desired, &mut dial),
Expand Down Expand Up @@ -1353,6 +1364,8 @@ pub struct Cluster {
dial: Option<Arc<moq_tokio::Client>>,
/// Every dial client, shared by all clones, for [`Self::close`].
dialers: Dialers,
/// The drain each dialed link joins; see [`Options::shutdown`].
shutdown: crate::shutdown::Observer,
/// Bound listen port and optional generated-certificate fingerprint.
advertise: Option<LanAdvertise>,
/// Set by [`Self::start`] when LAN discovery is on, so inbound
Expand Down Expand Up @@ -1466,6 +1479,7 @@ impl Cluster {
peer,
stats,
stats_hold,
shutdown,
#[cfg(feature = "test-support")]
network,
#[cfg(feature = "test-support")]
Expand Down Expand Up @@ -1531,6 +1545,7 @@ impl Cluster {
sim,
dial: None,
dialers: Dialers::new(),
shutdown: shutdown.unwrap_or_else(crate::shutdown::Observer::disabled),
advertise: None,
lan_auth: Arc::new(std::sync::OnceLock::new()),
nodes,
Expand Down Expand Up @@ -2162,8 +2177,13 @@ impl Cluster {
dial: &crate::nodes::Dial,
) -> anyhow::Result<()> {
let mut redial = Redial::default();
let mut shutdown = self.shutdown.clone();

loop {
// A draining relay dials nothing new: its peers would only be drained again.
if shutdown.deadline().is_some() {
return Ok(());
}
let mut connected = false;
let (result, presented) = match Self::dial_addrs(target, &token) {
Ok((addrs, presented)) => (
Expand All @@ -2174,8 +2194,11 @@ impl Cluster {
};
dial.disconnected();

// An attempt only ends in an error: its own reconnects retry until they give up.
let Err(err) = result;
// An attempt ends in an error, its own reconnects having given up, or drained.
let err = match result {
Ok(()) => return Ok(()),
Err(err) => err,
};
tracing::warn!(%err, "cluster peer error; will retry");
// The session's own reconnects reuse the token it started with, so a
// failure after the file rotated is most likely that token expiring:
Expand All @@ -2190,7 +2213,11 @@ impl Cluster {
}

let wait = redial.next(connected, &mut kio::clock::rng());
kio::clock::sleep(wait).await;
tokio::select! {
biased;
() = shutdown.started() => return Ok(()),
() = kio::clock::sleep(wait) => {}
}
}
}

Expand Down Expand Up @@ -2219,14 +2246,15 @@ impl Cluster {
}

/// One session to `target` over `addrs`, which carry any credential. It runs until the
/// session gives up, and sets `connected` once the peer has accepted it.
/// session gives up, or until the relay's drain has drained it, and sets `connected`
/// once the peer has accepted it.
async fn run_remote_once(
&self,
addrs: &moq_tokio::Addrs,
target: &DialTarget,
dial: &crate::nodes::Dial,
connected: &mut bool,
) -> anyhow::Result<std::convert::Infallible> {
) -> anyhow::Result<()> {
// Each attempt is its own session, so it gets its own id. Matches the span an
// accepted connection runs under, so both directions log the same way.
let id = self.next_connection_id();
Expand All @@ -2244,7 +2272,7 @@ impl Cluster {
target: &DialTarget,
dial: &crate::nodes::Dial,
connected: &mut bool,
) -> anyhow::Result<std::convert::Infallible> {
) -> anyhow::Result<()> {
// The peer URL may carry the cluster JWT in its query, so neither the log
// line nor the node label below may show the raw URL.
let first = addrs.as_slice().first().expect("Addrs is non-empty").url();
Expand Down Expand Up @@ -2289,6 +2317,9 @@ impl Cluster {
if let Some(cost) = target.cost {
client = client.with_cost(cost);
}
// Counted by the relay's drain before the first dial, so a drain that starts
// while the session connects still waits for it.
let serving = self.shutdown.serve();
// The GOAWAY lifecycle lives in the reconnect loop: on an upstream GOAWAY it
// dials the replacement while the old session keeps serving, so the old
// routes stay attached and the origin hands live tracks over at a group
Expand Down Expand Up @@ -2316,11 +2347,20 @@ impl Cluster {
.connect(addrs.clone());
let monitor = reconnect.monitor();
let mut observation = monitor.state();
let mut shutdown = self.shutdown.clone();
loop {
dial.observe(id, &observation);
*connected |= observation.connects > 0;
observation = monitor.state_changed(observation.revision).await?;
tokio::select! {
biased;
() = shutdown.started() => break,
changed = monitor.state_changed(observation.revision) => observation = changed?,
}
}
// Drained like an accepted session, so the peer hears GOAWAY and moves its
// routes before this relay goes, rather than seeing the link drop as a crash.
serving.drain_link(reconnect).await;
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
Ok(())
}

/// A peer client from the dial template, with the [`Options::peer`] QUIC
Expand Down Expand Up @@ -3505,6 +3545,52 @@ mod tests {
assert!(nodes.snapshot().nodes.is_empty(), "a removed peer takes its history");
}

/// Once the relay drains, no source change replaces, releases, or adds a dial, so
/// each link's own task finishes its drain to the deadline.
#[tokio::test]
async fn mesh_freezes_its_dials_once_the_relay_drains() {
let (trigger, shutdown) = crate::shutdown::Observer::new(Duration::from_secs(10));
let mut options = Options::new(Config::default());
options.shutdown = Some(shutdown);
let mut mesh = Mesh {
cluster: Cluster::spawn(options).expect("cluster"),
token: ClusterToken::default(),
dialed: DialMap::new(None),
sources: kio::Tasks::new(),
commands: DialCommands::default(),
};
let api = |peers: &[&str]| {
DialCommand::Api(
peers
.iter()
.map(|peer| {
let target = DialTarget::parse(peer).unwrap();
(target.key.clone(), target)
})
.collect(),
)
};
mesh.apply(api(&["https://a.example/?cost=1", "https://b.example/"]));
let task = mesh.dialed.entries[&key("https://a.example/")].task;

trigger.start();
mesh.apply(api(&["https://a.example/?cost=2", "https://c.example/"]));
assert_eq!(
mesh.dialed.entries[&key("https://a.example/")].task,
task,
"a changed target must not replace a draining dial"
);
assert!(
mesh.dialed.contains(&key("https://b.example/")),
"a removed peer must keep draining"
);
assert!(
!mesh.dialed.contains(&key("https://c.example/")),
"a draining relay dials nothing new"
);
assert_eq!(mesh.dialed.tasks.len(), 2);
}

/// A dial task's panic ends the mesh with it, rather than unwinding the task
/// polling the mesh.
#[tokio::test]
Expand Down
1 change: 1 addition & 0 deletions rs/moq-relay/src/relay.rs
Original file line number Diff line number Diff line change
Expand Up @@ -298,6 +298,7 @@ impl Relay {
.with_stats(config.stats);
cluster_options.client = config.limits.client;
cluster_options.peer = config.limits.peer;
cluster_options.shutdown = Some(shutdown.clone());
// The node's stats broadcast has one instance at a time: the relay this one
// replaces keeps it until it exits, and its last counters are read first.
let alone = sockets.replacing().then(|| {
Expand Down
Loading