Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions quest/m1/cluster-routing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,9 @@ vector the 2026-09-30 cache-tiers audit kept:

Kept from 2026-09-30: core links are configured and may skip PoPs; Warm and
Cold collapse to one cost ([One route cost](/quest/m1/route-cost.md));
`--hop` removal and One route cost land on their own. Decided 2026-10-03: a
concrete path is a source's identity, whoever serves it
([Selection](/quest/m1/cluster-routing/selection.md)).
`--hop` removal and One route cost land on their own. Decided 2026-10-06: a
path plus its [publisher epoch](/doc/concept/moq-lite.md#publisher-epochs) is
a source's identity, whoever serves it.
Anything specific to moq.pro's deployment is planned in moq.pro.

### Simulator findings
Expand Down Expand Up @@ -131,7 +131,6 @@ Once every child has landed:

## Required

- [Selection](/quest/m1/cluster-routing/selection.md) - a broadcast under overlapping prefixes routes to one origin deterministically, and same-path origins are one source
- [Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md) - an endpoint holds sessions to several CDNs, uses its preferred one, and fails over to the next
- [Simulate the split](/quest/m1/cluster-routing/sim.md) - moq.pro's simulator compares the route layer with path vector before the wire is written
- [Routes and announces](/quest/m1/cluster-routing/routes.md) - ROUTE per origin node and path-less ANNOUNCE on one stream, loop-free by Babel feasibility
Expand Down
17 changes: 6 additions & 11 deletions quest/m1/cluster-routing/multi-cdn.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,14 @@ Decided 2026-10-01 (moq-dev/moq#4694):
wins, equal by default), separate from the additive link cost, so metrics
from different operators are never added together or compared; within one
preference the metric decides as before. Babel permits any choice among
feasible routes, so loop freedom is unchanged. The ranking lives in
[Selection](/quest/m1/cluster-routing/selection.md); this quest adds the
preference input and the endpoint side.
feasible routes, so loop freedom is unchanged. The ranking is `route_order`
in `rs/moq-net/src/model/origin.rs`; this quest adds the preference input
and the endpoint side.
- A session that speaks no ROUTE (moq-transport, older lite) is a link whose
announces resolve to a session-scoped origin at that link's cost.
- Identity: two CDNs never share node ids, but any same-path route resumes,
so failover across them needs only the same path. The publisher side is
the redundant pair of [Selection](/quest/m1/cluster-routing/selection.md),
from one process.
- Identity: two CDNs never share node ids, but any route with the same path
and epoch resumes, so failover across them needs only that. The publisher
side is a redundant pair under one explicit epoch, from one process.
- API: the Rust client config and `js/net` accept several upstream URLs, each
with its preference (and cost), feeding one shared origin. Mirror the shape
in the bindings that expose connect config. Pick the smallest shape that
Expand All @@ -42,7 +41,3 @@ connect docs.

Public API: per-upstream preference in client config, in Rust, JS, and the
bindings. Wire: none.

## Required

- [Selection](/quest/m1/cluster-routing/selection.md) - the ranking this adds the preference input to
5 changes: 5 additions & 0 deletions quest/m1/cluster-routing/routes.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,11 @@ Decided 2026-10-01 (see the [line's decisions](/quest/m1/cluster-routing/README.
rendezvous hash among equal next hops). When the next hop changes, adopt
the new neighbour's stored set and send downstream only the difference.
No per-announce seqno.
- Origin selection keeps `route_order`'s shape and swaps its inputs: the
candidates become the origin nodes announcing the path, ranked by longest
prefix, then the link's preference
([Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md)), then the
route metric to the node, then the path-keyed rendezvous hash.
- Down-only bit: set on a route learned on an upstream link, kept across
other links, and a route carrying it is never sent on an upstream link
(extending the `upstream` link mark in `doc/bin/relay/cluster.md`).
Expand Down
61 changes: 0 additions & 61 deletions quest/m1/cluster-routing/selection.md

This file was deleted.

4 changes: 2 additions & 2 deletions quest/m3/p2p/cost-scopes.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ Decided 2026-10-01 (moq-dev/moq#4694):
ANNOUNCE names its origin node, and the roster maps that node to a peer the
application may dial, so an app learns whom to dial from the announce.
- Switching between the peer and the relay resumes the subscription, since
every route announcing a path is one source, per
[Selection](/quest/m1/cluster-routing/selection.md).
every route announcing a path under one
[publisher epoch](/doc/concept/moq-lite.md#publisher-epochs) is one source.

Deliverables: the `Peers` and `moq-cli` cost knobs with their defaults, the
rule written beside route selection in the routing concept page the cluster
Expand Down
12 changes: 11 additions & 1 deletion rs/moq-net/src/model/origin.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5381,7 +5381,8 @@ mod tests {

/// Equal-cost advertisers of one prefix share its paths: a set of requested
/// paths spreads across the pool, and one path always resolves to the same
/// advertiser, whatever order the routes arrived in.
/// advertiser, whatever order the routes arrived in. The split is a rendezvous
/// hash, so losing an advertiser moves only the paths it won.
#[test]
fn equal_cost_pool_spreads_paths() {
const WORKERS: [u64; 4] = [10, 11, 12, 13];
Expand Down Expand Up @@ -5420,6 +5421,15 @@ mod tests {
let share = forward.iter().filter(|hop| **hop == origin(worker)).count();
assert!(share >= PATHS / 16, "worker {worker} took {share} of {PATHS} paths");
}

// A mod-N style hash would reshuffle most paths here.
let lost = origin(WORKERS[1]);
let survivors = winners(WORKERS.into_iter().filter(|id| origin(*id) != lost));
for (i, (before, after)) in forward.iter().zip(&survivors).enumerate() {
if *before != lost {
assert_eq!(after, before, "pool/job-{i} moved off a surviving worker");
}
}
}

#[test]
Expand Down
157 changes: 151 additions & 6 deletions rs/moq-net/tests/route_change.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
//!
//! Older wire versions cannot carry the epoch, so `R` cannot tell the two routes
//! serve the same bytes: its subscription stays on its route and ends with it.
//!
//! Two publishers sharing one explicit epoch, a redundant pair, are one source the
//! same way.

mod support;

Expand Down Expand Up @@ -40,6 +43,11 @@ fn payload(group: u64, frame: u64) -> Vec<u8> {
format!("{group}.{frame}").into_bytes()
}

/// [`payload`], prefixed with the publisher that wrote it.
fn tagged(name: &str, group: u64, frame: u64) -> Vec<u8> {
format!("{name}:{group}.{frame}").into_bytes()
}

/// Let every task settle. Time is paused, so this returns once the runtime is idle.
async fn settle() {
moq_net_sim::sleep(Duration::from_millis(500)).await;
Expand Down Expand Up @@ -152,8 +160,11 @@ impl Topology {
}
}

/// A frame as the reader saw it: its group, then its timestamp and payload.
type Delivery = (u64, moq_net::Result<(Timestamp, Vec<u8>)>);

/// Read every group in full, reporting each frame as it arrives.
fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver<(u64, moq_net::Result<Vec<u8>>)> {
fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver<Delivery> {
let (tx, rx) = mpsc::unbounded();
moq_net_sim::spawn(async move {
loop {
Expand All @@ -168,7 +179,8 @@ fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver<(u64, moq_net::Re
loop {
match group.read_frame().await {
Ok(Some(frame)) => {
if tx.unbounded_send((group.sequence, Ok(frame.payload.to_vec()))).is_err() {
let frame = (frame.timestamp, frame.payload.to_vec());
if tx.unbounded_send((group.sequence, Ok(frame))).is_err() {
return;
}
}
Expand All @@ -185,12 +197,19 @@ fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver<(u64, moq_net::Re
}

/// Wait for the next frame the reader reports, failing on an error or a hang.
async fn next(rx: &mut mpsc::UnboundedReceiver<(u64, moq_net::Result<Vec<u8>>)>) -> (u64, Vec<u8>) {
async fn next(rx: &mut mpsc::UnboundedReceiver<Delivery>) -> (u64, Vec<u8>) {
let (group, _, payload) = next_timed(rx).await;
(group, payload)
}

/// [`next`], keeping the frame's timestamp.
async fn next_timed(rx: &mut mpsc::UnboundedReceiver<Delivery>) -> (u64, Timestamp, Vec<u8>) {
let (group, frame) = moq_net_sim::timeout(Duration::from_secs(10), rx.next())
.await
.expect("reader hung")
.expect("reader ended");
(group, frame.unwrap_or_else(|err| panic!("group {group} failed: {err}")))
let (timestamp, payload) = frame.unwrap_or_else(|err| panic!("group {group} failed: {err}"));
(group, timestamp, payload)
}

async fn route_change(version: &str, trigger: Trigger, position: Position) {
Expand Down Expand Up @@ -313,7 +332,7 @@ async fn route_dies_without_an_epoch(version: &str, trigger: Trigger) {
.expect("reader hung")
{
None | Some((_, Err(_))) => {}
Some((group, Ok(frame))) => panic!(
Some((group, Ok((_, frame)))) => panic!(
"{version} {trigger:?}: resumed through another route at {group}:{}",
String::from_utf8_lossy(&frame)
),
Expand Down Expand Up @@ -417,7 +436,7 @@ pinned_tests! {
/// partway through group 1: `A` has handed on (1, 0) and (1, 1) while `P` has already
/// moved on to group 2. `B` (warmed by another reader on `W`) holds everything, so `R`
/// resumes there from (1, 2).
async fn lagging_route_dies(version: Version) -> mpsc::UnboundedReceiver<(u64, moq_net::Result<Vec<u8>>)> {
async fn lagging_route_dies(version: Version) -> mpsc::UnboundedReceiver<Delivery> {
async fn lagged(version: Version, from: &origin::Producer, to: &origin::Producer, lag: Duration) -> MockPair {
let mut options = MockConnectOptions::new(version);
options.server_publish = Some(from.consume());
Expand Down Expand Up @@ -516,3 +535,129 @@ macro_rules! route_flap_tests {
route_flap_tests! {
flaps_lite_07: "moq-lite-07-wip",
}

/// How the incumbent of a redundant pair goes away.
#[derive(Clone, Copy, Debug)]
enum Loss {
/// Its session to `R` dies abruptly.
Unreachable,
/// It ends the broadcast, while its session to `R` stays up.
Ends,
}

/// One publisher of a redundant pair, writing its own copy of `live`.
struct Replica {
/// Tags its payloads, so the reader shows which replica served each frame.
name: &'static str,
origin: origin::Producer,
broadcast: broadcast::Producer,
track: track::Producer,
group: Option<moq_net::group::Producer>,
link: Option<MockPair>,
}

/// A redundant pair: `P1` and `P2` publish `live` under one explicit epoch and write
/// the same frames, tagged with their name. `R` reads through `P1`, priced below `P2`
/// so it starts there, until `P1` goes mid-group. `R` resumes on `P2` from the first frame its reader
/// lacks: every frame once, in order, and no timestamp ever rewinds.
async fn redundant_pair_fails_over(version: &str, loss: Loss) {
let version: Version = version.parse().unwrap();
let epoch = moq_net::Epoch::mint();
let subscriber = produce_origin(4);
let mut replicas = Vec::new();
for (name, hop, cost) in [("P1", 1, 1), ("P2", 2, 5)] {
let publisher = produce_origin(hop);
let broadcast = publisher.create_broadcast("live").unwrap();
let track = broadcast.create_track("video", None).unwrap();
broadcast
.announce(origin::Route::default().with_epoch(epoch.clone()).with_cost(cost))
.unwrap();
let link = link(version, &publisher, &subscriber).await;
replicas.push(Replica {
name,
origin: publisher,
broadcast,
track,
group: None,
link: Some(link),
});
}
settle().await;

let consumer = subscriber.consume();
let remote = consumer.request_broadcast("live").await.unwrap();
let prefs = track::Subscription::default().with_max_age(Duration::from_secs(60));
let mut rx = read(remote.track("video").unwrap().subscribe(prefs).await.unwrap());

let mut expected = Vec::new();
let mut seen = Vec::new();
let mut kept = None;
for sequence in 0..3 {
for replica in &mut replicas {
replica.group = Some(replica.track.append_group().unwrap());
}
for frame in 0..FRAMES {
if sequence == 1 && frame == FRAMES / 2 {
let Replica {
origin,
broadcast,
link,
..
} = replicas.remove(0);
let link = link.unwrap();
match loss {
Loss::Unreachable => {
link.server.abort(Error::Cancel);
link.client.abort(Error::Cancel);
}
// Keep the session and its origin up: only the broadcast, its track, and its open group go.
Loss::Ends => kept = Some((link, origin)),
}
drop(broadcast);
settle().await;
}
let timestamp = Timestamp::from_micros(1_000_000 + sequence * 100_000 + frame * 1_000).unwrap();
for replica in &mut replicas {
let group = replica.group.as_mut().unwrap();
group
.write_frame(timestamp, tagged(replica.name, sequence, frame))
.unwrap();
}
// The cheapest replica still up serves it.
expected.push((sequence, tagged(replicas[0].name, sequence, frame)));
seen.push(next_timed(&mut rx).await);
}
for replica in &mut replicas {
replica.group.take().unwrap().finish().unwrap();
}
}

let frames: Vec<_> = seen.iter().map(|(group, _, frame)| (*group, frame.clone())).collect();
assert_eq!(frames, expected, "{version} {loss:?}");
for pair in seen.windows(2) {
assert!(
pair[0].1 < pair[1].1,
"{version} {loss:?}: timestamp rewound at {pair:?}"
);
}
settle().await;
assert!(rx.try_recv().is_err(), "{version} {loss:?}: trailing delivery");
drop(kept);
}

#[moq_net_sim::test]
async fn redundant_pair_fails_over_when_unreachable() {
moq_net_sim::timeout(
TEST_TIMEOUT,
redundant_pair_fails_over("moq-lite-07-wip", Loss::Unreachable),
)
.await
.expect("timed out");
}

#[moq_net_sim::test]
async fn redundant_pair_fails_over_when_the_incumbent_ends() {
moq_net_sim::timeout(TEST_TIMEOUT, redundant_pair_fails_over("moq-lite-07-wip", Loss::Ends))
.await
.expect("timed out");
}
Loading