Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions doc/.vitepress/drafts.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -128,20 +128,20 @@ describe("tables render as tables", () => {

const rendered = html(page as DraftPage);

// The Bidirectional Streams table: one header, six body rows, each
// The Bidirectional Streams table: one header, seven body rows, each
// separated by a kramdown rule in the source. Cells carry the source's
// alignment, so match around the style attribute.
const header = /<th[^>]*>Stream<\/th>/.exec(rendered);
expect(header).not.toBeNull();

for (const stream of ["Announce", "Subscribe", "Fetch", "Probe", "Goaway", "Track"]) {
for (const stream of ["Announce", "Subscribe", "Fetch", "Probe", "Goaway", "Track", "Topology"]) {
expect(rendered).toMatch(new RegExp(`<td[^>]*>${stream}</td>`));
}

// One header row plus six body rows, in a single table.
// One header row plus seven body rows, in a single table.
const start = rendered.lastIndexOf("<table", header?.index);
const rows = rendered.slice(start).split("</table>")[0];
expect(rows.match(/<tr>/g) ?? []).toHaveLength(7);
expect(rows.match(/<tr>/g) ?? []).toHaveLength(8);
});
});

Expand Down
18 changes: 18 additions & 0 deletions doc/bin/relay/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,24 @@ connect_api = "https://api.example.com/cluster/peers"
node = "https://us-west.example.com/"
```

## Relay graph

Relays also learn the graph of links between them, apart from any route. Each
relay reports whether each of its links is up and what it costs, and cluster
sessions flood those reports, so every relay knows every link and its shortest
path (by cost, then hop count) to every other relay without an announcement.
A link counts once both ends report it up; a report is batched for 50 ms.

Only sessions on `moq-lite-07-wip` carry the graph, so cluster dials offer it
first and the listener accepts it by default. An accepted session joins only
when its grant names a cluster peer (`peer: true`, or an accepted LAN peer).
A peer on an older version, or on moq-transport, still carries broadcasts but
is no link in the graph.

A restarted relay reports under a new incarnation (its start time), which
replaces whatever its last run reported. With a random Hop ID a restart is a
new node instead, so pin `cluster.id` for a stable graph.

## Identity

Each relay has a Hop ID: the value it adds to a route's hop list for loop
Expand Down
6 changes: 6 additions & 0 deletions doc/bin/relay/http.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,3 +101,9 @@ logs use). A route is priced twice: `cost` as the cluster stands, which reads 0
through a relay already carrying the broadcast, and `cold_cost` with those
discounts removed, which is what tells two warm relays apart. It is best-effort
correlation, not authenticated identity.

`topology` is the relay graph learned over [cluster links](/bin/relay/cluster#relay-graph):
every relay that reported its links (`id`, the `incarnation` its reports come
from, and each link's `peer` and `cost`, `null` once down), with this relay's
shortest `distance` to it (`cost`, then `hops`; `null` while unreachable) and
the `next` neighbour on that path. Ids are strings, like the Hop ID above.
95 changes: 95 additions & 0 deletions drafts/draft-lcurley-moq-lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,8 @@ There's a 1-byte STREAM_TYPE at the beginning of each stream.
| ------- | ------------- | ----------- |
| 0x6 | Track | Subscriber |
| ------- | ------------- | ----------- |
| 0x7 | Topology | Client |
| ------- | ------------- | ----------- |

### Announce
A subscriber can open an Announce Stream to discover routes matching a prefix.
Expand Down Expand Up @@ -514,6 +516,30 @@ The peer MUST NOT open new streams on the current session after receiving a GOAW
The sender closes the stream (FIN) when it is ready to terminate the session.
The peer SHOULD close all streams and the session after migrating or when it no longer needs the session.

### Topology {#topology}
Relays in one cluster flood the liveness of their links on a Topology Stream (0x7), so each learns the relay graph once, apart from routes, and computes its shortest path to every other relay.
Only a session between two relays of one cluster carries one.
The client opens it; a server that does not treat the client as a cluster peer resets it, and the session continues without being a link.
An endpoint MUST close the session with a PROTOCOL_VIOLATION if it receives a second Topology Stream.

A relay's id is the non-zero Hop ID it declares in SETUP (see [Hop Parameter](#hop-parameter)).
Its incarnation MUST grow each time it restarts under the same id; the start time in milliseconds suffices.

A relay reports each of its links with a sequence number, starting at 1 and growing with every change to that link, and the link's cost while it is up (see [Cost Parameter](#cost-parameter)).
One report is newer than another from the same relay when its incarnation is greater, or equal with a greater sequence.
A receiver keeps only the newest report per link, and one from a greater incarnation replaces every link held for that relay.
A relay never sends a peer the peer's own reports, and never applies a report carrying its own id.
If such a report's incarnation is greater than the relay's own, a previous run's clock was ahead, and the relay SHOULD move its incarnation past it and report every link again.

Each side sends one TOPOLOGY_DIGEST, then any number of TOPOLOGY_REPORT messages.
On receiving the peer's digest, a relay reports the link up with a new sequence, then sends every report it holds that the digest lacks, leaving out the peer's own.
Reporting the link first keeps the relay's report of the link's last failure from reaching the peer.
From then on it sends every report it newly applies from another link, and every change to its own links, including this link going down when the session ends.
A relay SHOULD hold what it sends on each stream for a short time (50 ms is RECOMMENDED), sending only the newest report per link.

A link is usable once both ends report it up.
A path's distance is the sum of its link costs, each as reported by the relay the link leaves, then its number of links, so every link lengthens a path even at cost 0.

# Delivery
The most important concept in moq-lite is how to deliver a subscription.
QUIC can only improve the user experience if data is delivered out-of-order during congestion.
Expand Down Expand Up @@ -1280,6 +1306,74 @@ A peer that reconnects to a provided URI SHOULD keep using that URI for subseque
A relay that receives a GOAWAY SHOULD treat the announcements that arrived on that session as the most expensive routes available, so a subscription it can serve from another session moves at the next Group boundary rather than when the draining session finally closes.
The routes stay usable: a broadcast reachable only over the draining session MUST keep being served until the session ends, which is what makes the sender's deadline a handover window rather than a cutoff.

## TOPOLOGY_DIGEST {#topology-digest}
The first message on each side of a [Topology Stream](#topology): each relay the sender holds reports from, and the sequence of each link reported.

~~~
TOPOLOGY_DIGEST Message {
Message Length (i)
Node ID (i)
Reporter Count (i)
Digest Reporter (..) ...
}

Digest Reporter {
Node ID (i)
Incarnation (i)
Link Count (i)
Digest Link (..) ...
}

Digest Link {
Peer ID (i)
Sequence (i)
}
~~~

**Node ID**:
In the message, the sender's id, which MUST equal the Hop ID it declared in SETUP.
In a Digest Reporter, the reporting relay's id.

**Peer ID**:
The id of the relay at the far end of the link.

A receiver MUST close the session with a PROTOCOL_VIOLATION if any id is 0, a Node ID repeats among the reporters, a Peer ID repeats within one reporter, or a Peer ID equals its reporter's Node ID.

## TOPOLOGY_REPORT {#topology-report}
Every later message on each side of a [Topology Stream](#topology): a batch of link reports.

~~~
TOPOLOGY_REPORT Message {
Message Length (i)
Reporter Count (i)
Report Reporter (..) ...
}

Report Reporter {
Node ID (i)
Incarnation (i)
Link Count (i)
Report Link (..) ...
}

Report Link {
Peer ID (i)
Sequence (i)
Up (i)
Cost (i)
}
~~~

**Up**:
1 while the link is up, 0 once it is down.
Any other value is a PROTOCOL_VIOLATION.

**Cost**:
The link's cost while it is up, in the units of the [Cost Parameter](#cost-parameter).
Ignored when `Up` is 0.

The id rules of [TOPOLOGY_DIGEST](#topology-digest) apply.

## GROUP
The GROUP message contains information about a Group, as well as a reference to the subscription being served.

Expand Down Expand Up @@ -1346,6 +1440,7 @@ The `Message Length` describes the payload size on the wire.
- Removed SUBSCRIBE_DROP and its type 0x2; a group without a Group Stream is not counted.
- The Subscribe Stream FIN now follows once every counted Group Stream has finished or been reset.
- Added announce compression: ANNOUNCE_START gains `Path Base` and `Path Keep` to copy the head of a live advertisement's suffix, and ANNOUNCE_START and ANNOUNCE_UPDATE gain `Hop Base` and `Hop Keep` to copy the tail of a live advertisement's Hop ID list.
- Added the Topology Stream (0x7), TOPOLOGY_DIGEST, and TOPOLOGY_REPORT: relays in one cluster flood per-link liveness and cost, scoped to each relay's incarnation, and compute shortest paths from it.

## moq-lite-06

Expand Down
2 changes: 2 additions & 0 deletions js/net/src/lite/stream.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ export const StreamId = {
Probe: 4,
Goaway: 5,
Track: 6,
/** Relay-to-relay only (lite-07+): no JS endpoint is a cluster link, so one is reset as unknown. */
Topology: 7,
ClientCompat: 0x20,
ServerCompat: 0x21,
} as const;
Expand Down
9 changes: 9 additions & 0 deletions rs/moq-cli/src/args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,15 @@ impl MoqSide {
config.tls.generate = vec!["moq-cluster-lan".to_string()];
}
}
// A cluster peer dials in on the version carrying the relay graph, which the
// default set leaves out; clients never offer it unless they opt in.
if config.version.is_empty() && (self.lan() || self.cluster_dials()) {
config.version = hang::moq_net::Versions::topology()
.iter()
.chain(hang::moq_net::Versions::all().iter())
.copied()
.collect();
}
config
}

Expand Down
16 changes: 16 additions & 0 deletions rs/moq-net/src/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ pub struct Client {
setup_path: Option<String>,
cost: Option<u64>,
peer_hop: Option<crate::Hop>,
topology: Option<crate::topology::Database>,
}

impl Client {
Expand Down Expand Up @@ -125,6 +126,17 @@ impl Client {
self
}

/// Make this session a cluster link that floods the relay graph in `topology`.
///
/// Only moq-lite-07 and newer carry the Topology stream; on any other version
/// the session runs as usual and takes no part. The accepting side must
/// attach its own database too ([`Server::with_topology`](crate::Server::with_topology)),
/// or the session is not a link.
pub fn with_topology(mut self, topology: crate::topology::Database) -> Self {
self.topology = Some(topology);
self
}

/// The origin pair a session attaches, tagged and filtered.
///
/// Reads through the publish (egress) consumer and writes through the
Expand Down Expand Up @@ -189,6 +201,8 @@ impl Client {
version,
our_setup,
peer_setup: None,
topology: self.topology.clone(),
client: true,
})?;

Ok(Session::new(
Expand Down Expand Up @@ -376,6 +390,8 @@ impl Client {
// (pre-lite-05), which have no Setup Stream.
our_setup: lite::Setup::default(),
peer_setup: None,
topology: None,
client: true,
})?;

(
Expand Down
1 change: 1 addition & 0 deletions rs/moq-net/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ pub mod server;
pub mod session;
pub mod stats;
pub mod time;
pub mod topology;
pub mod transport;

pub use client::*;
Expand Down
2 changes: 2 additions & 0 deletions rs/moq-net/src/lite/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ mod subscribe;
mod subscriber;
#[cfg(test)]
pub(crate) mod test_transport;
mod topology;
mod track;
mod version;

Expand All @@ -46,6 +47,7 @@ pub use setup::*;
pub use stream::*;
pub use subscribe::*;
use subscriber::*;
pub(crate) use topology::*;
#[allow(unused_imports)]
pub use track::*;
pub use version::Version;
56 changes: 56 additions & 0 deletions rs/moq-net/src/lite/publisher.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,11 @@ pub(super) struct PublisherConfig<S: crate::transport::poll::Session> {
/// The origin (hop) id assigned to the peer, used whenever the peer doesn't
/// declare one itself. See `Client::with_peer_hop`.
pub peer_hop: Option<Hop>,
/// The relay graph to flood when the peer opens a Topology stream, set only on
/// an accepted cluster link.
pub topology: Option<crate::topology::Database>,
/// What we charge for this link, when we priced it ourselves.
pub cost: Option<u64>,
}

/// Context shared by every control-stream child.
Expand All @@ -65,6 +70,10 @@ struct Shared<S: crate::transport::poll::Session> {
goaway: crate::goaway::Protocol,
// Control streams still serving the peer data, which a draining close waits for.
owed: AtomicUsize,
topology: Option<crate::topology::Database>,
cost: Option<u64>,
// Set once the peer opens its Topology stream; a second is a protocol violation.
topology_opened: std::sync::atomic::AtomicBool,
}

/// Largest millisecond duration every implementation can carry losslessly.
Expand Down Expand Up @@ -165,6 +174,9 @@ impl<S: crate::transport::poll::Session> Publisher<S> {
version: config.version,
goaway: config.goaway,
owed: AtomicUsize::new(0),
topology: config.topology,
cost: config.cost,
topology_opened: Default::default(),
}),
runtime: config.runtime,
accept,
Expand Down Expand Up @@ -251,6 +263,8 @@ enum ControlState<S: crate::transport::poll::Session> {
Goaway {
stream: Stream<S, Version>,
},
/// Flooding the relay graph with a cluster peer that dialed us.
Topology(Box<lite::Topology<S>>),
Done,
}

Expand Down Expand Up @@ -281,6 +295,31 @@ impl<S: crate::transport::poll::Session> kio::Task for Control<S> {
}

impl<S: crate::transport::poll::Session> Control<S> {
/// Serve a Topology stream the peer opened. Refused unless this session is a
/// cluster link on a version that carries one, and only once per session.
fn topology(&self, stream: Stream<S, Version>) -> Result<lite::Topology<S>, Error> {
let shared = &self.shared;
let Some(database) = shared.topology.clone().filter(|_| shared.version.has_topology()) else {
tracing::warn!("peer opened a topology stream, but this session is not a cluster link");
return Err(Error::UnexpectedStream);
};
if shared.topology_opened.swap(true, Ordering::Relaxed) {
shared
.session
.clone()
.close(SessionError::ProtocolViolation.to_code(), "duplicate topology stream");
return Err(Error::ProtocolViolation);
}
Ok(lite::Topology::accept(
database,
stream,
shared.version,
self.runtime.clone(),
shared.peer_setup.clone(),
shared.cost,
))
}

fn poll_serve(&mut self, waiter: &kio::Waiter) -> Poll<Result<(), Error>> {
loop {
match &mut self.state {
Expand All @@ -306,6 +345,7 @@ impl<S: crate::transport::poll::Session> Control<S> {
ControlState::Probe(ProbeServe::new(self.shared.clone(), self.runtime.clone(), stream))
}
lite::ControlType::Goaway => ControlState::Goaway { stream },
lite::ControlType::Topology => ControlState::Topology(Box::new(self.topology(stream)?)),
lite::ControlType::Session => return Poll::Ready(Err(Error::UnexpectedStream)),
};
if self.state.owes() {
Expand All @@ -317,6 +357,18 @@ impl<S: crate::transport::poll::Session> Control<S> {
ControlState::Fetch(serve) => return serve.poll(waiter),
ControlState::TrackInfo(serve) => return serve.poll(waiter),
ControlState::Probe(serve) => return serve.poll(waiter),
ControlState::Topology(topology) => {
let res = ready!(topology.poll(waiter));
// Control errors are only logged, but a peer breaking the topology
// protocol ends the session, as it does on the dialing side.
if let Err(err) = &res {
self.shared
.session
.clone()
.close(SessionError::from(err).to_code(), &err.to_string());
}
return Poll::Ready(res);
}
ControlState::Goaway { stream } => {
// A decode error propagates to the caller, which logs and continues: a
// malformed GOAWAY must not tear down the session it is trying to drain.
Expand Down Expand Up @@ -3542,6 +3594,8 @@ mod tests {
peer_setup,
goaway,
peer_hop: Some(assigned),
topology: None,
cost: None,
});

let serving = kio::wait(|waiter| publisher.shared.poll_serving_origin(waiter)).await;
Expand Down Expand Up @@ -3703,6 +3757,8 @@ mod tests {
peer_setup: crate::lite::PeerSetup::default(),
goaway,
peer_hop: None,
topology: None,
cost: None,
});
ProbeServe::new(publisher.shared.clone(), publisher.runtime.clone(), stream)
}
Expand Down
Loading
Loading