From 5830ee8106c9c6723142c4bb217a1ccde799b8cf Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 6 Oct 2026 23:17:07 -0700 Subject: [PATCH 1/4] quest(cluster-routing/selection): claim Co-Authored-By: Claude Opus 5.5 From ed633873025e126ed646c9c6c2304cbf1710c5ac Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 6 Oct 2026 23:28:45 -0700 Subject: [PATCH 2/4] quest(cluster-routing/selection): pin HRW stability and redundant-pair failover The ranking, path-keyed rendezvous hash, epoch identity, and upstream spread already landed (#4403, #4741, #4816, #4942). What was left were tests: the equal-cost pool test now also proves losing an advertiser moves only its paths, and route_change covers a redundant pair (two publishers under one explicit epoch) failing over mid-group, whether the incumbent becomes unreachable or ends, with no timestamp rewind. Deletes the finished quest and moves its ranking guidance into Routes and announces. Co-Authored-By: Claude Opus 5.5 --- quest/m1/cluster-routing/README.md | 7 +- quest/m1/cluster-routing/multi-cdn.md | 17 ++-- quest/m1/cluster-routing/routes.md | 5 + quest/m1/cluster-routing/selection.md | 61 ----------- quest/m3/p2p/cost-scopes.md | 4 +- rs/moq-net/src/model/origin.rs | 12 ++- rs/moq-net/tests/route_change.rs | 141 ++++++++++++++++++++++++-- 7 files changed, 162 insertions(+), 85 deletions(-) delete mode 100644 quest/m1/cluster-routing/selection.md diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index 6c69ebf2921..e2764f8ca2f 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -86,9 +86,9 @@ vector the 2026-09-30 cache-tiers audit kept: Kept from 2026-09-30: core links are configured and may skip PoPs; Warm and Cold collapse to one cost ([One route cost](/quest/m1/route-cost.md)); -`--hop` removal and One route cost land on their own. Decided 2026-10-03: a -concrete path is a source's identity, whoever serves it -([Selection](/quest/m1/cluster-routing/selection.md)). +`--hop` removal and One route cost land on their own. Decided 2026-10-06: a +path plus its [publisher epoch](/doc/concept/moq-lite.md#publisher-epochs) is +a source's identity, whoever serves it. Anything specific to moq.pro's deployment is planned in moq.pro. ### Simulator findings @@ -131,7 +131,6 @@ Once every child has landed: ## Required -- [Selection](/quest/m1/cluster-routing/selection.md) - a broadcast under overlapping prefixes routes to one origin deterministically, and same-path origins are one source - [Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md) - an endpoint holds sessions to several CDNs, uses its preferred one, and fails over to the next - [Simulate the split](/quest/m1/cluster-routing/sim.md) - moq.pro's simulator compares the route layer with path vector before the wire is written - [Routes and announces](/quest/m1/cluster-routing/routes.md) - ROUTE per origin node and path-less ANNOUNCE on one stream, loop-free by Babel feasibility diff --git a/quest/m1/cluster-routing/multi-cdn.md b/quest/m1/cluster-routing/multi-cdn.md index 498f96a843e..c4134ed48bc 100644 --- a/quest/m1/cluster-routing/multi-cdn.md +++ b/quest/m1/cluster-routing/multi-cdn.md @@ -18,15 +18,14 @@ Decided 2026-10-01 (moq-dev/moq#4694): wins, equal by default), separate from the additive link cost, so metrics from different operators are never added together or compared; within one preference the metric decides as before. Babel permits any choice among - feasible routes, so loop freedom is unchanged. The ranking lives in - [Selection](/quest/m1/cluster-routing/selection.md); this quest adds the - preference input and the endpoint side. + feasible routes, so loop freedom is unchanged. The ranking is `route_order` + in `rs/moq-net/src/model/origin.rs`; this quest adds the preference input + and the endpoint side. - A session that speaks no ROUTE (moq-transport, older lite) is a link whose announces resolve to a session-scoped origin at that link's cost. -- Identity: two CDNs never share node ids, but any same-path route resumes, - so failover across them needs only the same path. The publisher side is - the redundant pair of [Selection](/quest/m1/cluster-routing/selection.md), - from one process. +- Identity: two CDNs never share node ids, but any route with the same path + and epoch resumes, so failover across them needs only that. The publisher + side is a redundant pair under one explicit epoch, from one process. - API: the Rust client config and `js/net` accept several upstream URLs, each with its preference (and cost), feeding one shared origin. Mirror the shape in the bindings that expose connect config. Pick the smallest shape that @@ -42,7 +41,3 @@ connect docs. Public API: per-upstream preference in client config, in Rust, JS, and the bindings. Wire: none. - -## Required - -- [Selection](/quest/m1/cluster-routing/selection.md) - the ranking this adds the preference input to diff --git a/quest/m1/cluster-routing/routes.md b/quest/m1/cluster-routing/routes.md index c59473872f6..3210f62b05d 100644 --- a/quest/m1/cluster-routing/routes.md +++ b/quest/m1/cluster-routing/routes.md @@ -66,6 +66,11 @@ Decided 2026-10-01 (see the [line's decisions](/quest/m1/cluster-routing/README. rendezvous hash among equal next hops). When the next hop changes, adopt the new neighbour's stored set and send downstream only the difference. No per-announce seqno. +- Origin selection keeps `route_order`'s shape and swaps its inputs: the + candidates become the origin nodes announcing the path, ranked by longest + prefix, then the link's preference + ([Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md)), then the + route metric to the node, then the path-keyed rendezvous hash. - Down-only bit: set on a route learned on an upstream link, kept across other links, and a route carrying it is never sent on an upstream link (extending the `upstream` link mark in `doc/bin/relay/cluster.md`). diff --git a/quest/m1/cluster-routing/selection.md b/quest/m1/cluster-routing/selection.md deleted file mode 100644 index 9636711a4af..00000000000 --- a/quest/m1/cluster-routing/selection.md +++ /dev/null @@ -1,61 +0,0 @@ -# [M] Deterministic origin selection - -## Goal - -A broadcast under overlapping announcements routes to one origin chosen -deterministically, and a relay spreads paths over equally ranked next hops -(an edge over its region's cores) so every relay sends a given path the same -way. -Origins that announce the same concrete path under the same epoch are one -source, so a subscriber resumes on another when the incumbent ends or becomes -unreachable. - -## Plan - -Most of this landed with Wildcard (#4403), #4741, and publisher epochs -(#4942): `route_order` ranks the longest covering prefix first, then the -newest epoch, keys its hash on the requested path so equal-cost advertisers of -one prefix split paths between them, and a path plus epoch is one source with -mid-group resume. Terminal refusal landed too. What remains (cut 2026-10-06, -after #4403 merged): the HRW split test, the same-path failover test, and -spreading paths over upstream links. - -Decided: - -- Origin choice is per broadcast, not per announced prefix. -- A path plus its [publisher epoch](/doc/concept/moq-lite.md#publisher-epochs) is - a source's identity (decided 2026-10-06, replacing the 2026-10-03 - path-only rule): origins announcing the same path under the same epoch - are interchangeable, which is how a redundant pair works under one - explicit epoch, and the reply names no origin. Each publisher instance, - claim workers included, mints its own epoch, so instances are never - spliced: the newest epoch wins and the older one's subscriptions end. A - route without an epoch names no instance and is never resumed elsewhere. - Failover within an epoch resumes from the first frame the subscriber - lacks, mid-group. -- Every hop re-selects (decided 2026-10-01): SUBSCRIBE names no origin, since - a pin breaks subscription aggregation. -- A refusal is terminal, and an origin sheds load by withdrawing or - re-pricing its route instead. - -Candidate mechanics: - -- Upstream-link spread: at an edge, the rendezvous hash (HRW) of the - requested path and the route's origin spreads paths over the region's - cores and makes every edge pick the same core for a path; failover moves - only the paths the lost core won. -- Once [Routes and announces](/quest/m1/cluster-routing/routes.md) lands, the - candidates are the origin nodes announcing the path, ranked by longest - prefix, then the link's preference - ([Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md)), then route - metric to the node, then HRW. Write the ranking so that change swaps its - inputs, not its shape. - -Wire: none expected; if one is needed it goes in lite-07 (the current wip -version, where the route layer also lands) with the draft. Tests cover an HRW -split across an equal-cost pool and a same-path pair failing over mid-group -with no timestamp rewind. - -## Related - -- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - the per-instance identity failover resumes within diff --git a/quest/m3/p2p/cost-scopes.md b/quest/m3/p2p/cost-scopes.md index 26765a8967e..66263ad53ed 100644 --- a/quest/m3/p2p/cost-scopes.md +++ b/quest/m3/p2p/cost-scopes.md @@ -28,8 +28,8 @@ Decided 2026-10-01 (moq-dev/moq#4694): ANNOUNCE names its origin node, and the roster maps that node to a peer the application may dial, so an app learns whom to dial from the announce. - Switching between the peer and the relay resumes the subscription, since - every route announcing a path is one source, per - [Selection](/quest/m1/cluster-routing/selection.md). + every route announcing a path under one + [publisher epoch](/doc/concept/moq-lite.md#publisher-epochs) is one source. Deliverables: the `Peers` and `moq-cli` cost knobs with their defaults, the rule written beside route selection in the routing concept page the cluster diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index f296af2738d..eb9b5f88dea 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -5675,7 +5675,8 @@ mod tests { /// Equal-cost advertisers of one prefix share its paths: a set of requested /// paths spreads across the pool, and one path always resolves to the same - /// advertiser, whatever order the routes arrived in. + /// advertiser, whatever order the routes arrived in. The split is a rendezvous + /// hash, so losing an advertiser moves only the paths it won. #[test] fn equal_cost_pool_spreads_paths() { const WORKERS: [u64; 4] = [10, 11, 12, 13]; @@ -5714,6 +5715,15 @@ mod tests { let share = forward.iter().filter(|hop| **hop == origin(worker)).count(); assert!(share >= PATHS / 16, "worker {worker} took {share} of {PATHS} paths"); } + + // A mod-N style hash would reshuffle most paths here. + let lost = origin(WORKERS[1]); + let survivors = winners(WORKERS.into_iter().filter(|id| origin(*id) != lost)); + for (i, (before, after)) in forward.iter().zip(&survivors).enumerate() { + if *before != lost { + assert_eq!(after, before, "pool/job-{i} moved off a surviving worker"); + } + } } #[test] diff --git a/rs/moq-net/tests/route_change.rs b/rs/moq-net/tests/route_change.rs index f52ecdf8500..8fa5f7f7ba7 100644 --- a/rs/moq-net/tests/route_change.rs +++ b/rs/moq-net/tests/route_change.rs @@ -8,6 +8,9 @@ //! //! Older wire versions cannot carry the epoch, so `R` cannot tell the two routes //! serve the same bytes: its subscription stays on its route and ends with it. +//! +//! Two publishers sharing one explicit epoch, a redundant pair, are one source the +//! same way. mod support; @@ -152,8 +155,11 @@ impl Topology { } } +/// A frame as the reader saw it: its group, then its timestamp and payload. +type Delivery = (u64, moq_net::Result<(Timestamp, Vec)>); + /// Read every group in full, reporting each frame as it arrives. -fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver<(u64, moq_net::Result>)> { +fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver { let (tx, rx) = mpsc::unbounded(); moq_net_sim::spawn(async move { loop { @@ -168,7 +174,8 @@ fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver<(u64, moq_net::Re loop { match group.read_frame().await { Ok(Some(frame)) => { - if tx.unbounded_send((group.sequence, Ok(frame.payload.to_vec()))).is_err() { + let frame = (frame.timestamp, frame.payload.to_vec()); + if tx.unbounded_send((group.sequence, Ok(frame))).is_err() { return; } } @@ -185,12 +192,19 @@ fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver<(u64, moq_net::Re } /// Wait for the next frame the reader reports, failing on an error or a hang. -async fn next(rx: &mut mpsc::UnboundedReceiver<(u64, moq_net::Result>)>) -> (u64, Vec) { +async fn next(rx: &mut mpsc::UnboundedReceiver) -> (u64, Vec) { + let (group, _, payload) = next_timed(rx).await; + (group, payload) +} + +/// [`next`], keeping the frame's timestamp. +async fn next_timed(rx: &mut mpsc::UnboundedReceiver) -> (u64, Timestamp, Vec) { let (group, frame) = moq_net_sim::timeout(Duration::from_secs(10), rx.next()) .await .expect("reader hung") .expect("reader ended"); - (group, frame.unwrap_or_else(|err| panic!("group {group} failed: {err}"))) + let (timestamp, payload) = frame.unwrap_or_else(|err| panic!("group {group} failed: {err}")); + (group, timestamp, payload) } async fn route_change(version: &str, trigger: Trigger, position: Position) { @@ -313,7 +327,7 @@ async fn route_dies_without_an_epoch(version: &str, trigger: Trigger) { .expect("reader hung") { None | Some((_, Err(_))) => {} - Some((group, Ok(frame))) => panic!( + Some((group, Ok((_, frame)))) => panic!( "{version} {trigger:?}: resumed through another route at {group}:{}", String::from_utf8_lossy(&frame) ), @@ -417,7 +431,7 @@ pinned_tests! { /// partway through group 1: `A` has handed on (1, 0) and (1, 1) while `P` has already /// moved on to group 2. `B` (warmed by another reader on `W`) holds everything, so `R` /// resumes there from (1, 2). -async fn lagging_route_dies(version: Version) -> mpsc::UnboundedReceiver<(u64, moq_net::Result>)> { +async fn lagging_route_dies(version: Version) -> mpsc::UnboundedReceiver { async fn lagged(version: Version, from: &origin::Producer, to: &origin::Producer, lag: Duration) -> MockPair { let mut options = MockConnectOptions::new(version); options.server_publish = Some(from.consume()); @@ -516,3 +530,118 @@ macro_rules! route_flap_tests { route_flap_tests! { flaps_lite_07: "moq-lite-07-wip", } + +/// How the incumbent of a redundant pair goes away. +#[derive(Clone, Copy, Debug)] +enum Loss { + /// Its session to `R` dies abruptly. + Unreachable, + /// It ends the broadcast, while its session to `R` stays up. + Ends, +} + +/// One publisher of a redundant pair, writing its own copy of `live`. +struct Replica { + _origin: origin::Producer, + _broadcast: broadcast::Producer, + track: track::Producer, + group: Option, + link: Option, +} + +/// A redundant pair: `P1` and `P2` publish `live` under one explicit epoch and write +/// the same frames. `R` reads through `P1`, priced below `P2` so it starts there, +/// until `P1` goes mid-group. `R` resumes on `P2` from the first frame its reader +/// lacks: every frame once, in order, and no timestamp ever rewinds. +async fn redundant_pair_fails_over(version: &str, loss: Loss) { + let version: Version = version.parse().unwrap(); + let epoch = moq_net::Epoch::mint(); + let subscriber = produce_origin(4); + let mut replicas = Vec::new(); + for (hop, cost) in [(1, 1), (2, 5)] { + let publisher = produce_origin(hop); + let broadcast = publisher.create_broadcast("live").unwrap(); + let track = broadcast.create_track("video", None).unwrap(); + broadcast + .announce(origin::Route::default().with_epoch(epoch.clone()).with_cost(cost)) + .unwrap(); + let link = link(version, &publisher, &subscriber).await; + replicas.push(Replica { + _origin: publisher, + _broadcast: broadcast, + track, + group: None, + link: Some(link), + }); + } + settle().await; + + let consumer = subscriber.consume(); + let remote = consumer.request_broadcast("live").await.unwrap(); + let prefs = track::Subscription::default().with_max_age(Duration::from_secs(60)); + let mut rx = read(remote.track("video").unwrap().subscribe(prefs).await.unwrap()); + + let mut expected = Vec::new(); + let mut seen = Vec::new(); + let mut kept = None; + for sequence in 0..3 { + for replica in &mut replicas { + replica.group = Some(replica.track.append_group().unwrap()); + } + for frame in 0..FRAMES { + if sequence == 1 && frame == FRAMES / 2 { + let mut incumbent = replicas.remove(0); + let link = incumbent.link.take().unwrap(); + match loss { + Loss::Unreachable => { + link.server.abort(Error::Cancel); + link.client.abort(Error::Cancel); + } + // Keep the session up: only the broadcast goes. + Loss::Ends => kept = Some(link), + } + drop(incumbent); + settle().await; + } + let timestamp = Timestamp::from_micros(1_000_000 + sequence * 100_000 + frame * 1_000).unwrap(); + for replica in &mut replicas { + let group = replica.group.as_mut().unwrap(); + group.write_frame(timestamp, payload(sequence, frame)).unwrap(); + } + expected.push((sequence, payload(sequence, frame))); + seen.push(next_timed(&mut rx).await); + } + for replica in &mut replicas { + replica.group.take().unwrap().finish().unwrap(); + } + } + + let frames: Vec<_> = seen.iter().map(|(group, _, frame)| (*group, frame.clone())).collect(); + assert_eq!(frames, expected, "{version} {loss:?}"); + for pair in seen.windows(2) { + assert!( + pair[0].1 < pair[1].1, + "{version} {loss:?}: timestamp rewound at {pair:?}" + ); + } + settle().await; + assert!(rx.try_recv().is_err(), "{version} {loss:?}: trailing delivery"); + drop(kept); +} + +#[moq_net_sim::test] +async fn redundant_pair_fails_over_when_unreachable() { + moq_net_sim::timeout( + TEST_TIMEOUT, + redundant_pair_fails_over("moq-lite-07-wip", Loss::Unreachable), + ) + .await + .expect("timed out"); +} + +#[moq_net_sim::test] +async fn redundant_pair_fails_over_when_the_incumbent_ends() { + moq_net_sim::timeout(TEST_TIMEOUT, redundant_pair_fails_over("moq-lite-07-wip", Loss::Ends)) + .await + .expect("timed out"); +} From 9bde4f54581138745e3abf332c2e6e3712107476 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 7 Oct 2026 10:02:45 -0700 Subject: [PATCH 3/4] test(net): pin which replica serves each frame in the redundant-pair failover Each replica tags its payloads, so the test fails if P2 served from the start. The Ends case keeps P1's origin up with its session, so only the broadcast goes. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/tests/route_change.rs | 44 ++++++++++++++++++++++---------- 1 file changed, 30 insertions(+), 14 deletions(-) diff --git a/rs/moq-net/tests/route_change.rs b/rs/moq-net/tests/route_change.rs index 8fa5f7f7ba7..9815cc75e4d 100644 --- a/rs/moq-net/tests/route_change.rs +++ b/rs/moq-net/tests/route_change.rs @@ -43,6 +43,11 @@ fn payload(group: u64, frame: u64) -> Vec { format!("{group}.{frame}").into_bytes() } +/// [`payload`], prefixed with the publisher that wrote it. +fn tagged(name: &str, group: u64, frame: u64) -> Vec { + format!("{name}:{group}.{frame}").into_bytes() +} + /// Let every task settle. Time is paused, so this returns once the runtime is idle. async fn settle() { moq_net_sim::sleep(Duration::from_millis(500)).await; @@ -542,23 +547,25 @@ enum Loss { /// One publisher of a redundant pair, writing its own copy of `live`. struct Replica { - _origin: origin::Producer, - _broadcast: broadcast::Producer, + /// Tags its payloads, so the reader shows which replica served each frame. + name: &'static str, + origin: origin::Producer, + broadcast: broadcast::Producer, track: track::Producer, group: Option, link: Option, } /// A redundant pair: `P1` and `P2` publish `live` under one explicit epoch and write -/// the same frames. `R` reads through `P1`, priced below `P2` so it starts there, -/// until `P1` goes mid-group. `R` resumes on `P2` from the first frame its reader +/// the same frames, tagged with their name. `R` reads through `P1`, priced below `P2` +/// so it starts there, until `P1` goes mid-group. `R` resumes on `P2` from the first frame its reader /// lacks: every frame once, in order, and no timestamp ever rewinds. async fn redundant_pair_fails_over(version: &str, loss: Loss) { let version: Version = version.parse().unwrap(); let epoch = moq_net::Epoch::mint(); let subscriber = produce_origin(4); let mut replicas = Vec::new(); - for (hop, cost) in [(1, 1), (2, 5)] { + for (name, hop, cost) in [("P1", 1, 1), ("P2", 2, 5)] { let publisher = produce_origin(hop); let broadcast = publisher.create_broadcast("live").unwrap(); let track = broadcast.create_track("video", None).unwrap(); @@ -567,8 +574,9 @@ async fn redundant_pair_fails_over(version: &str, loss: Loss) { .unwrap(); let link = link(version, &publisher, &subscriber).await; replicas.push(Replica { - _origin: publisher, - _broadcast: broadcast, + name, + origin: publisher, + broadcast, track, group: None, link: Some(link), @@ -590,25 +598,33 @@ async fn redundant_pair_fails_over(version: &str, loss: Loss) { } for frame in 0..FRAMES { if sequence == 1 && frame == FRAMES / 2 { - let mut incumbent = replicas.remove(0); - let link = incumbent.link.take().unwrap(); + let Replica { + origin, + broadcast, + link, + .. + } = replicas.remove(0); + let link = link.unwrap(); match loss { Loss::Unreachable => { link.server.abort(Error::Cancel); link.client.abort(Error::Cancel); } - // Keep the session up: only the broadcast goes. - Loss::Ends => kept = Some(link), + // Keep the session and its origin up: only the broadcast goes. + Loss::Ends => kept = Some((link, origin)), } - drop(incumbent); + drop(broadcast); settle().await; } let timestamp = Timestamp::from_micros(1_000_000 + sequence * 100_000 + frame * 1_000).unwrap(); for replica in &mut replicas { let group = replica.group.as_mut().unwrap(); - group.write_frame(timestamp, payload(sequence, frame)).unwrap(); + group + .write_frame(timestamp, tagged(replica.name, sequence, frame)) + .unwrap(); } - expected.push((sequence, payload(sequence, frame))); + // The cheapest replica still up serves it. + expected.push((sequence, tagged(replicas[0].name, sequence, frame))); seen.push(next_timed(&mut rx).await); } for replica in &mut replicas { From a9cc312cde66b3f58819970596d7121e6dd9370f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 7 Oct 2026 10:26:09 -0700 Subject: [PATCH 4/4] test(net): say what the Ends loss drops Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/tests/route_change.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rs/moq-net/tests/route_change.rs b/rs/moq-net/tests/route_change.rs index 9815cc75e4d..94d844c2010 100644 --- a/rs/moq-net/tests/route_change.rs +++ b/rs/moq-net/tests/route_change.rs @@ -610,7 +610,7 @@ async fn redundant_pair_fails_over(version: &str, loss: Loss) { link.server.abort(Error::Cancel); link.client.abort(Error::Cancel); } - // Keep the session and its origin up: only the broadcast goes. + // Keep the session and its origin up: only the broadcast, its track, and its open group go. Loss::Ends => kept = Some((link, origin)), } drop(broadcast);