diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index 6c69ebf2921..e2764f8ca2f 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -86,9 +86,9 @@ vector the 2026-09-30 cache-tiers audit kept: Kept from 2026-09-30: core links are configured and may skip PoPs; Warm and Cold collapse to one cost ([One route cost](/quest/m1/route-cost.md)); -`--hop` removal and One route cost land on their own. Decided 2026-10-03: a -concrete path is a source's identity, whoever serves it -([Selection](/quest/m1/cluster-routing/selection.md)). +`--hop` removal and One route cost land on their own. Decided 2026-10-06: a +path plus its [publisher epoch](/doc/concept/moq-lite.md#publisher-epochs) is +a source's identity, whoever serves it. Anything specific to moq.pro's deployment is planned in moq.pro. ### Simulator findings @@ -131,7 +131,6 @@ Once every child has landed: ## Required -- [Selection](/quest/m1/cluster-routing/selection.md) - a broadcast under overlapping prefixes routes to one origin deterministically, and same-path origins are one source - [Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md) - an endpoint holds sessions to several CDNs, uses its preferred one, and fails over to the next - [Simulate the split](/quest/m1/cluster-routing/sim.md) - moq.pro's simulator compares the route layer with path vector before the wire is written - [Routes and announces](/quest/m1/cluster-routing/routes.md) - ROUTE per origin node and path-less ANNOUNCE on one stream, loop-free by Babel feasibility diff --git a/quest/m1/cluster-routing/multi-cdn.md b/quest/m1/cluster-routing/multi-cdn.md index 498f96a843e..c4134ed48bc 100644 --- a/quest/m1/cluster-routing/multi-cdn.md +++ b/quest/m1/cluster-routing/multi-cdn.md @@ -18,15 +18,14 @@ Decided 2026-10-01 (moq-dev/moq#4694): wins, equal by default), separate from the additive link cost, so metrics from different operators are never added together or compared; within one preference the metric decides as before. Babel permits any choice among - feasible routes, so loop freedom is unchanged. The ranking lives in - [Selection](/quest/m1/cluster-routing/selection.md); this quest adds the - preference input and the endpoint side. + feasible routes, so loop freedom is unchanged. The ranking is `route_order` + in `rs/moq-net/src/model/origin.rs`; this quest adds the preference input + and the endpoint side. - A session that speaks no ROUTE (moq-transport, older lite) is a link whose announces resolve to a session-scoped origin at that link's cost. -- Identity: two CDNs never share node ids, but any same-path route resumes, - so failover across them needs only the same path. The publisher side is - the redundant pair of [Selection](/quest/m1/cluster-routing/selection.md), - from one process. +- Identity: two CDNs never share node ids, but any route with the same path + and epoch resumes, so failover across them needs only that. The publisher + side is a redundant pair under one explicit epoch, from one process. - API: the Rust client config and `js/net` accept several upstream URLs, each with its preference (and cost), feeding one shared origin. Mirror the shape in the bindings that expose connect config. Pick the smallest shape that @@ -42,7 +41,3 @@ connect docs. Public API: per-upstream preference in client config, in Rust, JS, and the bindings. Wire: none. - -## Required - -- [Selection](/quest/m1/cluster-routing/selection.md) - the ranking this adds the preference input to diff --git a/quest/m1/cluster-routing/routes.md b/quest/m1/cluster-routing/routes.md index c59473872f6..3210f62b05d 100644 --- a/quest/m1/cluster-routing/routes.md +++ b/quest/m1/cluster-routing/routes.md @@ -66,6 +66,11 @@ Decided 2026-10-01 (see the [line's decisions](/quest/m1/cluster-routing/README. rendezvous hash among equal next hops). When the next hop changes, adopt the new neighbour's stored set and send downstream only the difference. No per-announce seqno. +- Origin selection keeps `route_order`'s shape and swaps its inputs: the + candidates become the origin nodes announcing the path, ranked by longest + prefix, then the link's preference + ([Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md)), then the + route metric to the node, then the path-keyed rendezvous hash. - Down-only bit: set on a route learned on an upstream link, kept across other links, and a route carrying it is never sent on an upstream link (extending the `upstream` link mark in `doc/bin/relay/cluster.md`). diff --git a/quest/m1/cluster-routing/selection.md b/quest/m1/cluster-routing/selection.md deleted file mode 100644 index 9636711a4af..00000000000 --- a/quest/m1/cluster-routing/selection.md +++ /dev/null @@ -1,61 +0,0 @@ -# [M] Deterministic origin selection - -## Goal - -A broadcast under overlapping announcements routes to one origin chosen -deterministically, and a relay spreads paths over equally ranked next hops -(an edge over its region's cores) so every relay sends a given path the same -way. -Origins that announce the same concrete path under the same epoch are one -source, so a subscriber resumes on another when the incumbent ends or becomes -unreachable. - -## Plan - -Most of this landed with Wildcard (#4403), #4741, and publisher epochs -(#4942): `route_order` ranks the longest covering prefix first, then the -newest epoch, keys its hash on the requested path so equal-cost advertisers of -one prefix split paths between them, and a path plus epoch is one source with -mid-group resume. Terminal refusal landed too. What remains (cut 2026-10-06, -after #4403 merged): the HRW split test, the same-path failover test, and -spreading paths over upstream links. - -Decided: - -- Origin choice is per broadcast, not per announced prefix. -- A path plus its [publisher epoch](/doc/concept/moq-lite.md#publisher-epochs) is - a source's identity (decided 2026-10-06, replacing the 2026-10-03 - path-only rule): origins announcing the same path under the same epoch - are interchangeable, which is how a redundant pair works under one - explicit epoch, and the reply names no origin. Each publisher instance, - claim workers included, mints its own epoch, so instances are never - spliced: the newest epoch wins and the older one's subscriptions end. A - route without an epoch names no instance and is never resumed elsewhere. - Failover within an epoch resumes from the first frame the subscriber - lacks, mid-group. -- Every hop re-selects (decided 2026-10-01): SUBSCRIBE names no origin, since - a pin breaks subscription aggregation. -- A refusal is terminal, and an origin sheds load by withdrawing or - re-pricing its route instead. - -Candidate mechanics: - -- Upstream-link spread: at an edge, the rendezvous hash (HRW) of the - requested path and the route's origin spreads paths over the region's - cores and makes every edge pick the same core for a path; failover moves - only the paths the lost core won. -- Once [Routes and announces](/quest/m1/cluster-routing/routes.md) lands, the - candidates are the origin nodes announcing the path, ranked by longest - prefix, then the link's preference - ([Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md)), then route - metric to the node, then HRW. Write the ranking so that change swaps its - inputs, not its shape. - -Wire: none expected; if one is needed it goes in lite-07 (the current wip -version, where the route layer also lands) with the draft. Tests cover an HRW -split across an equal-cost pool and a same-path pair failing over mid-group -with no timestamp rewind. - -## Related - -- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - the per-instance identity failover resumes within diff --git a/quest/m3/p2p/cost-scopes.md b/quest/m3/p2p/cost-scopes.md index 26765a8967e..66263ad53ed 100644 --- a/quest/m3/p2p/cost-scopes.md +++ b/quest/m3/p2p/cost-scopes.md @@ -28,8 +28,8 @@ Decided 2026-10-01 (moq-dev/moq#4694): ANNOUNCE names its origin node, and the roster maps that node to a peer the application may dial, so an app learns whom to dial from the announce. - Switching between the peer and the relay resumes the subscription, since - every route announcing a path is one source, per - [Selection](/quest/m1/cluster-routing/selection.md). + every route announcing a path under one + [publisher epoch](/doc/concept/moq-lite.md#publisher-epochs) is one source. Deliverables: the `Peers` and `moq-cli` cost knobs with their defaults, the rule written beside route selection in the routing concept page the cluster diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index dc720f2a6e4..6e25404648e 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -5381,7 +5381,8 @@ mod tests { /// Equal-cost advertisers of one prefix share its paths: a set of requested /// paths spreads across the pool, and one path always resolves to the same - /// advertiser, whatever order the routes arrived in. + /// advertiser, whatever order the routes arrived in. The split is a rendezvous + /// hash, so losing an advertiser moves only the paths it won. #[test] fn equal_cost_pool_spreads_paths() { const WORKERS: [u64; 4] = [10, 11, 12, 13]; @@ -5420,6 +5421,15 @@ mod tests { let share = forward.iter().filter(|hop| **hop == origin(worker)).count(); assert!(share >= PATHS / 16, "worker {worker} took {share} of {PATHS} paths"); } + + // A mod-N style hash would reshuffle most paths here. + let lost = origin(WORKERS[1]); + let survivors = winners(WORKERS.into_iter().filter(|id| origin(*id) != lost)); + for (i, (before, after)) in forward.iter().zip(&survivors).enumerate() { + if *before != lost { + assert_eq!(after, before, "pool/job-{i} moved off a surviving worker"); + } + } } #[test] diff --git a/rs/moq-net/tests/route_change.rs b/rs/moq-net/tests/route_change.rs index f52ecdf8500..94d844c2010 100644 --- a/rs/moq-net/tests/route_change.rs +++ b/rs/moq-net/tests/route_change.rs @@ -8,6 +8,9 @@ //! //! Older wire versions cannot carry the epoch, so `R` cannot tell the two routes //! serve the same bytes: its subscription stays on its route and ends with it. +//! +//! Two publishers sharing one explicit epoch, a redundant pair, are one source the +//! same way. mod support; @@ -40,6 +43,11 @@ fn payload(group: u64, frame: u64) -> Vec { format!("{group}.{frame}").into_bytes() } +/// [`payload`], prefixed with the publisher that wrote it. +fn tagged(name: &str, group: u64, frame: u64) -> Vec { + format!("{name}:{group}.{frame}").into_bytes() +} + /// Let every task settle. Time is paused, so this returns once the runtime is idle. async fn settle() { moq_net_sim::sleep(Duration::from_millis(500)).await; @@ -152,8 +160,11 @@ impl Topology { } } +/// A frame as the reader saw it: its group, then its timestamp and payload. +type Delivery = (u64, moq_net::Result<(Timestamp, Vec)>); + /// Read every group in full, reporting each frame as it arrives. -fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver<(u64, moq_net::Result>)> { +fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver { let (tx, rx) = mpsc::unbounded(); moq_net_sim::spawn(async move { loop { @@ -168,7 +179,8 @@ fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver<(u64, moq_net::Re loop { match group.read_frame().await { Ok(Some(frame)) => { - if tx.unbounded_send((group.sequence, Ok(frame.payload.to_vec()))).is_err() { + let frame = (frame.timestamp, frame.payload.to_vec()); + if tx.unbounded_send((group.sequence, Ok(frame))).is_err() { return; } } @@ -185,12 +197,19 @@ fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver<(u64, moq_net::Re } /// Wait for the next frame the reader reports, failing on an error or a hang. -async fn next(rx: &mut mpsc::UnboundedReceiver<(u64, moq_net::Result>)>) -> (u64, Vec) { +async fn next(rx: &mut mpsc::UnboundedReceiver) -> (u64, Vec) { + let (group, _, payload) = next_timed(rx).await; + (group, payload) +} + +/// [`next`], keeping the frame's timestamp. +async fn next_timed(rx: &mut mpsc::UnboundedReceiver) -> (u64, Timestamp, Vec) { let (group, frame) = moq_net_sim::timeout(Duration::from_secs(10), rx.next()) .await .expect("reader hung") .expect("reader ended"); - (group, frame.unwrap_or_else(|err| panic!("group {group} failed: {err}"))) + let (timestamp, payload) = frame.unwrap_or_else(|err| panic!("group {group} failed: {err}")); + (group, timestamp, payload) } async fn route_change(version: &str, trigger: Trigger, position: Position) { @@ -313,7 +332,7 @@ async fn route_dies_without_an_epoch(version: &str, trigger: Trigger) { .expect("reader hung") { None | Some((_, Err(_))) => {} - Some((group, Ok(frame))) => panic!( + Some((group, Ok((_, frame)))) => panic!( "{version} {trigger:?}: resumed through another route at {group}:{}", String::from_utf8_lossy(&frame) ), @@ -417,7 +436,7 @@ pinned_tests! { /// partway through group 1: `A` has handed on (1, 0) and (1, 1) while `P` has already /// moved on to group 2. `B` (warmed by another reader on `W`) holds everything, so `R` /// resumes there from (1, 2). -async fn lagging_route_dies(version: Version) -> mpsc::UnboundedReceiver<(u64, moq_net::Result>)> { +async fn lagging_route_dies(version: Version) -> mpsc::UnboundedReceiver { async fn lagged(version: Version, from: &origin::Producer, to: &origin::Producer, lag: Duration) -> MockPair { let mut options = MockConnectOptions::new(version); options.server_publish = Some(from.consume()); @@ -516,3 +535,129 @@ macro_rules! route_flap_tests { route_flap_tests! { flaps_lite_07: "moq-lite-07-wip", } + +/// How the incumbent of a redundant pair goes away. +#[derive(Clone, Copy, Debug)] +enum Loss { + /// Its session to `R` dies abruptly. + Unreachable, + /// It ends the broadcast, while its session to `R` stays up. + Ends, +} + +/// One publisher of a redundant pair, writing its own copy of `live`. +struct Replica { + /// Tags its payloads, so the reader shows which replica served each frame. + name: &'static str, + origin: origin::Producer, + broadcast: broadcast::Producer, + track: track::Producer, + group: Option, + link: Option, +} + +/// A redundant pair: `P1` and `P2` publish `live` under one explicit epoch and write +/// the same frames, tagged with their name. `R` reads through `P1`, priced below `P2` +/// so it starts there, until `P1` goes mid-group. `R` resumes on `P2` from the first frame its reader +/// lacks: every frame once, in order, and no timestamp ever rewinds. +async fn redundant_pair_fails_over(version: &str, loss: Loss) { + let version: Version = version.parse().unwrap(); + let epoch = moq_net::Epoch::mint(); + let subscriber = produce_origin(4); + let mut replicas = Vec::new(); + for (name, hop, cost) in [("P1", 1, 1), ("P2", 2, 5)] { + let publisher = produce_origin(hop); + let broadcast = publisher.create_broadcast("live").unwrap(); + let track = broadcast.create_track("video", None).unwrap(); + broadcast + .announce(origin::Route::default().with_epoch(epoch.clone()).with_cost(cost)) + .unwrap(); + let link = link(version, &publisher, &subscriber).await; + replicas.push(Replica { + name, + origin: publisher, + broadcast, + track, + group: None, + link: Some(link), + }); + } + settle().await; + + let consumer = subscriber.consume(); + let remote = consumer.request_broadcast("live").await.unwrap(); + let prefs = track::Subscription::default().with_max_age(Duration::from_secs(60)); + let mut rx = read(remote.track("video").unwrap().subscribe(prefs).await.unwrap()); + + let mut expected = Vec::new(); + let mut seen = Vec::new(); + let mut kept = None; + for sequence in 0..3 { + for replica in &mut replicas { + replica.group = Some(replica.track.append_group().unwrap()); + } + for frame in 0..FRAMES { + if sequence == 1 && frame == FRAMES / 2 { + let Replica { + origin, + broadcast, + link, + .. + } = replicas.remove(0); + let link = link.unwrap(); + match loss { + Loss::Unreachable => { + link.server.abort(Error::Cancel); + link.client.abort(Error::Cancel); + } + // Keep the session and its origin up: only the broadcast, its track, and its open group go. + Loss::Ends => kept = Some((link, origin)), + } + drop(broadcast); + settle().await; + } + let timestamp = Timestamp::from_micros(1_000_000 + sequence * 100_000 + frame * 1_000).unwrap(); + for replica in &mut replicas { + let group = replica.group.as_mut().unwrap(); + group + .write_frame(timestamp, tagged(replica.name, sequence, frame)) + .unwrap(); + } + // The cheapest replica still up serves it. + expected.push((sequence, tagged(replicas[0].name, sequence, frame))); + seen.push(next_timed(&mut rx).await); + } + for replica in &mut replicas { + replica.group.take().unwrap().finish().unwrap(); + } + } + + let frames: Vec<_> = seen.iter().map(|(group, _, frame)| (*group, frame.clone())).collect(); + assert_eq!(frames, expected, "{version} {loss:?}"); + for pair in seen.windows(2) { + assert!( + pair[0].1 < pair[1].1, + "{version} {loss:?}: timestamp rewound at {pair:?}" + ); + } + settle().await; + assert!(rx.try_recv().is_err(), "{version} {loss:?}: trailing delivery"); + drop(kept); +} + +#[moq_net_sim::test] +async fn redundant_pair_fails_over_when_unreachable() { + moq_net_sim::timeout( + TEST_TIMEOUT, + redundant_pair_fails_over("moq-lite-07-wip", Loss::Unreachable), + ) + .await + .expect("timed out"); +} + +#[moq_net_sim::test] +async fn redundant_pair_fails_over_when_the_incumbent_ends() { + moq_net_sim::timeout(TEST_TIMEOUT, redundant_pair_fails_over("moq-lite-07-wip", Loss::Ends)) + .await + .expect("timed out"); +}