From 7c53c18541af1c5313895737de0ddd3f7b35db37 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 09:46:29 -0800 Subject: [PATCH 01/12] chore(quest): claim quest/m2/cache-routing-proof.md Quest-Claim: ec6bec3ae9667597 From e3c54099f301a1fa903e34385a1557c5713f3c2e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 13:08:42 -0800 Subject: [PATCH 02/12] test(net): prove cache-aware routing over the route layer Model per-broadcast demand, parents, and per-neighbor warm routes over the real Router, with fixtures and a churn sweep. Loop-freedom holds at every step; convergence needs the attribution and marginal pricing amendments the epic's Plan now proposes, and 4 of 3000 scenarios still never settle (behind #[ignore]). Router::routes becomes the crate-private Router::eligible. The warm message is placed in an additive SETUP extension. Co-Authored-By: Claude Opus 5.5 --- quest/m2/README.md | 1 - quest/m2/cache-routing-proof.md | 75 -- quest/m2/lite07-finalize.md | 1 - quest/m3/cache-routing/README.md | 62 +- quest/m3/cache-routing/select.md | 25 +- rs/moq-net/src/route/mod.rs | 28 +- rs/moq-net/src/route/node.rs | 6 +- rs/moq-net/src/route/tests/cache.rs | 1116 +++++++++++++++++++++++++++ rs/moq-net/src/route/tests/mod.rs | 36 +- 9 files changed, 1254 insertions(+), 96 deletions(-) delete mode 100644 quest/m2/cache-routing-proof.md create mode 100644 rs/moq-net/src/route/tests/cache.rs diff --git a/quest/m2/README.md b/quest/m2/README.md index 763c3a74b6e..30a4f3b1f1c 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -198,7 +198,6 @@ quests that require them. - [Promote the routes epic out of cluster routing](/quest/m2/promote-cluster-routes.md) - no epic holds another, once the open PRs under it land - [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - the wire names every abandoned group and the drop signal is public, so readers count losses instead of stalling - [Stream credit by priority](/quest/m1/stream-credit/README.md) - every stream opens through one priority entry point, and a higher-priority group takes over or resets a lower one when credit runs out (held in m1 by an open claim) -- [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - warm routes and the eligibility rule hold in route-layer sweeps, and the warm message's wire placement is chosen before the lite-07 cut - [Route latency](/quest/m2/route-latency.md) - the maintainer decides whether a lite-07 ROUTE carries an informational latency sum, and it lands before the cut if so - [Finalize moq-lite-07](/quest/m2/lite07-finalize.md) - when the maintainer cuts it, lite-07 negotiates as `moq-lite-07` and the next release ships it - [Unbounded groups](/quest/m2/unbounded-groups/README.md) - groups and frames lose their size caps; a group keeps a 32 MB tail, and readers behind it replay through paced flights or refresh diff --git a/quest/m2/cache-routing-proof.md b/quest/m2/cache-routing-proof.md deleted file mode 100644 index e95de051d54..00000000000 --- a/quest/m2/cache-routing-proof.md +++ /dev/null @@ -1,75 +0,0 @@ -# [L] Prove cache-aware routing - -## Goal - -Sweeps over the route layer show that the [epic](/quest/m3/cache-routing/README.md)'s -warm routes and eligibility rule keep per-broadcast forwarding loop-free at -every step, under stale, reordered, and in-flight messages. Once demand and -topology stop changing, every demanded broadcast is delivered, parent choices -stop changing, and a relay carries a broadcast only when it has a local -reader, feeds two or more downstream sessions, or is its one consumer's -cheapest cold route, as the epic's Goal states. The warm message's shape and wire placement are chosen -and recorded in the epic's Plan. - -## Plan - -Decided 2026-11-09 (maintainer, /quest-plan from Kixel's audit of its fleet -plan): ✅ pull the proof into m2, ahead of the lite-07 cut, so a warm -message that ships in lite-07 is the proven one (rejected: a design-only -quest in m2 that sizes the message with no proof; leaving this in m3 and -taking an additive SETUP extension if it lands after the cut). This reverses -the 2026-11-08 choice of m3 for the proof alone; -[the implementation](/quest/m3/cache-routing/select.md) stays in m3. If the -proof places the message in lite-07, this quest also lands its draft section -and its codec in Rust and JS, parsed and ignored until m3, so the cut carries -it; a SETUP extension needs neither before the cut, and -[Finalize moq-lite-07](/quest/m2/lite07-finalize.md) then drops this quest -from its list. Either placement is now this quest's to record, with no -second trip to the maintainer for choosing lite-07. - -Build on the sans-IO `Router` in `rs/moq-net/src/route/`, which each -origin's `route::Node` drives, and the harness behind its handoff sweeps, -not the reference model in -`rs/moq-net/tests/integration/routing_contracts/`, which -[Route layer acceptance](/quest/m2/cluster-routing/routes/acceptance.md) -deletes. The router exposes one selected route per origin; the proof needs -each origin's eligible neighbors as well, so add that as a crate-private -query. Add per-broadcast demand and forwarding on top of the router state: -readers joining and leaving, each relay's parent per broadcast, warm routes -per neighbor, and the existing linger. - -Fixtures, each a deterministic counterexample the rules must survive: - -- A chain of three relays plus a skip link, priced 3 and 5: cold skip, warm - join, and the return to the skip when the skipped relay's reader leaves. -- A relay whose only consumer is N withdraws its warm route toward N. N - bypasses it over a cheaper skip link, and stays when the relay is N's - cheapest cold route. -- Two relays at equal cold metric, both warm: only the lower hash keeps its - upstream, the other joins it, and they never adopt each other. -- A ring of relays each holding a stale, lower report of a neighbor's cold - metric, the hazard behind the pre-#3225 hold: no loop and no broadcast left - without a source. -- A warm withdrawal in flight while N's request is on its way: the former - carrier forwards the request along its own route, since it answers the same - warm or cold, and forwarding stays loop-free. -- An origin seqno advance, a route loss, and a relay that drains (GOAWAY - makes a neighbor least preferred) while broadcasts are being pulled. -- A warm route for another instance of the same path is never joined. -- k LAN relays with their own uplinks: one uplink copy per broadcast in a - full mesh; record the copies a sparse ring keeps, which the epic accepts. - -Sweeps churn random meshes: readers come and go, sessions drop and return, -and messages are held and delivered out of step. Check at every step that the -per-broadcast forwarding graph has no cycle, and at quiescence the Goal's -steady state. Report warm messages per demand change and confirm an idle -broadcast sends none. - -Wire: design the warm message, which names at least the path, the origin's -Route ID, and the instance, and say which stream carries it and how a peer -that never sends one behaves. Size it against lite-07 and an additive SETUP -extension and record the choice and its reason in the epic's Plan. - -## Related - -- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the feasibility rules and handoff sweeps this extends diff --git a/quest/m2/lite07-finalize.md b/quest/m2/lite07-finalize.md index f284423a969..658b9cf68fe 100644 --- a/quest/m2/lite07-finalize.md +++ b/quest/m2/lite07-finalize.md @@ -126,7 +126,6 @@ whose lite-07 encoding [lite-07 by start](/quest/m2/time-addressing/lite07.md) c - [Takeover](/quest/m1/stream-credit/takeover.md) - lite-07 lets a group stream take over for another group - [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - may add SUBSCRIBE_DROP for a stopped group; leaves this list if it takes the no-wire path - [Delete the old start bookkeeping](/quest/m2/subscribe-ranges/start-bookkeeping.md) - may change START per update on lite-07 -- [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - may add the warm-route message to lite-07; leaves this list if the proof places it in a SETUP extension - [Route latency](/quest/m2/route-latency.md) - may add a latency sum to ROUTE; leaves this list if the maintainer takes the no-wire path ## Related diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index 00f80c27ec1..5ea7b2a28ff 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -74,6 +74,67 @@ Decided 2026-11-08 (maintainer, /quest-plan): accepting a SETUP extension). The implementation stays in m3. This replaces item 6's rule that choosing lite-07 went back to the maintainer. +Proof results 2026-11-10 ([moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)), +modeled in `rs/moq-net/src/route/tests/cache.rs` over the real `Router`: + +8. **Item 4 holds.** No sweep step in 3000 churned scenarios formed a + per-broadcast cycle, and the fixtures fail when either the feasibility + check or the hash order is dropped. The router answers it with the + crate-private `Router::eligible`, which marks a route tied at the + feasibility distance so the caller applies the hash. The argument needs a + feasibility distance that never rises while a neighbor holds the advert; + the overtaking echo that + [moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447) found breaks + that, which [the implementation](/quest/m3/cache-routing/select.md) must + not ship before it is fixed. +9. **Proposed amendment to item 2: each pull names whom it serves.** "Demand + besides N's" can't be told from one hop: if B's only demand is a relay + pulling on N's behalf, B counts warm toward N, N joins B, B's demand + vanishes, and the cycle repeats every linger. A SUBSCRIBE carries + `Serves`: alone (a local reader, or two or more roots below it), a single + root relay's Node ID, or unknown (a peer that never negotiated, which + warms nothing). N counts B warm only when B serves alone or a root other + than N and N's own root. A narrowed warm route stands until the copy's + linger runs out, as the withdrawal already does. +10. **Proposed amendment to item 3: price a dependent copy at its cost.** + Link-only pricing has no steady state when two relays each keep the + other warm (sweep scenario 2062 flips forever). A warm route carries its + sender's own price for its pull; a receiver the copy depends on (its + serves names only the receiver's root) pays link plus that cost, an + independent one pays the link alone, and a relay moves only for a + strictly lower `(draining, price)`. With exact information that is a + potential game on the forest's total link cost. Under the sweep's churn, + 2996 of 3000 scenarios settle; 4 (the first is 478) never do, as two + carriers keep a narrowed route standing while two relays swap between + them. `sweep_cache_routing_long` reproduces them behind `#[ignore]`. + The Goal's "cheapest cold route" becomes "cheapest given the copies the + others hold" under this rule. +11. **Item 6 placement: an additive SETUP extension, Warm Routes**, decided + by the proof. Only m3 relays use it, so lite-07 would carry Rust and JS + codecs that js/net never runs, and it would freeze a shape items 9 and + 10 still revise; the bytes are the same either way, and it stays clear of + [Route latency](/quest/m2/route-latency.md)'s ROUTE change. The + parameter's value is empty. `WARM_START` (a new announce-stream type, + not 0x5, which ANNOUNCE_TAKEOVER held) carries the Route ID it travels + under, the path suffix, the Instance and Minted encoded as in + ANNOUNCE_START, then the serves and cost, and assigns a Warm ID; `WARM_END` carries the + Warm ID. It follows the ROUTE_START it names on the announce stream the + carrier answers, and a ROUTE_END retires the warm routes under it. + SUBSCRIBE and SUBSCRIBE_UPDATE gain a trailing Serves field when the + extension is negotiated. The hash is the Resolution section's FNV-1a + rendezvous hash of the path and the neighbor's Node ID. A peer that + never negotiates sees only cold routes, and its pulls serve unknown. + About 4.5 warm messages per reader change across the sweep's churn, + which also flaps links and advances seqnos; an idle broadcast sends + none. +12. **Costs the sweeps measured.** A full LAN mesh of 6 relays pulls one + uplink copy per broadcast; a ring of 6 keeps 13 copies for 8 broadcasts, + one per hash local minimum. An origin seqno advance that reaches a child + before its warm parent makes the parent's older route ineligible, so the + child moves off and back (2 moves in the chain fixture); origins advance + on every announce stream open and close, so fewer advances cut this + churn. + Prior art: warm/cold route prices ([moq-dev/moq#2925](https://github.com/moq-dev/moq/pull/2925)) were deleted by prefix routes ([moq-dev/moq#3225](https://github.com/moq-dev/moq/pull/3225)), @@ -98,7 +159,6 @@ This README's own work, once its children land: ## Required -- [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - warm routes and the eligibility rule hold in route-layer sweeps, and the wire placement is chosen - [Warm routes and per-broadcast parent selection](/quest/m3/cache-routing/select.md) - relays advertise warm routes and pull each broadcast through the cheapest eligible neighbor ## Related diff --git a/quest/m3/cache-routing/select.md b/quest/m3/cache-routing/select.md index 68db1467b78..1f141e27b86 100644 --- a/quest/m3/cache-routing/select.md +++ b/quest/m3/cache-routing/select.md @@ -12,10 +12,18 @@ and how many moves warm routes caused. ## Plan -- Implement what [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - settles and pass its fixtures: the warm message where it placed it, warm - state per neighbor derived from each front's demand, and eligibility over - the router's feasibility distances. +- Implement the rules the epic's Plan records as proven, which + `rs/moq-net/src/route/tests/cache.rs` models over the real router: the + Warm Routes SETUP extension (warm routes on the announce stream, and the + relay each pull serves on SUBSCRIBE), warm state per neighbor derived from + each front's demand, and eligibility through `Router::eligible` with the + rendezvous hash ordering tied routes. Port that file's fixtures and sweep + to the implementation, or drive it from them. +- First make `sweep_cache_routing_long` settle: 4 of its 3000 scenarios + never do under the epic's item 10, and its `#[ignore]` comes off once they + do. Land after the overtaking-echo fix that + [moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447) found, which + item 8's argument needs. - Extend today's selection rather than adding a second selector. `route_order` in `rs/moq-net/src/model/origin.rs` already hashes equal routes per path, and a front already moves to a better route of its own @@ -25,7 +33,7 @@ and how many moves warm routes caused. - Bound the warm state a peer can make a relay hold by design, as announcements are bounded. - Only relay sessions that negotiated warm routes exchange them. A browser - never parents, so `js/net` needs at most to skip the message. + never parents and never negotiates them, so `js/net` needs nothing. - Export upstream copies held, warm routes sent and received, and moves by cause through the relay's existing metrics. - `rs/moq-net/AGENTS.md` says a relay behaves the same warm or cold. That @@ -34,11 +42,8 @@ and how many moves warm routes caused. rewording to the maintainer. Public API: report any relay configuration added for warm routes. Wire: the -warm message as the proof placed it, with its draft section in the same PR. - -## Required - -- [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - the fixtures and wire placement this implements +Warm Routes extension as the epic's Plan designs it, with its draft section in +the same PR. ## Related diff --git a/rs/moq-net/src/route/mod.rs b/rs/moq-net/src/route/mod.rs index 6a74dacea35..c1061414b6b 100644 --- a/rs/moq-net/src/route/mod.rs +++ b/rs/moq-net/src/route/mod.rs @@ -201,6 +201,18 @@ pub(crate) struct Selected { pub down_only: bool, } +/// A usable route a request for one of an identity's broadcasts may go through. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct Eligible { + pub next: N, + /// This node's metric through it. + pub metric: Metric, + /// Advertised at exactly this node's feasibility distance: a request goes through + /// it only when the neighbor's rendezvous hash for the broadcast is below this + /// node's, so the hash orders the tie and no two tied nodes pull through each other. + pub tied: bool, +} + /// How a node's reader sees one announced claim. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) struct Shown { @@ -1032,15 +1044,21 @@ impl Router { best.map(|(_, selected)| selected) } - /// Every usable feasible route to `id`, with this node's metric through each. Any - /// of them forwards without a loop, if it [carries](Self::carries) the claim. - pub fn routes(&self, id: Id) -> Vec<(N, Metric)> { + /// Every usable route to `id` a request may go through without a loop, if it + /// [carries](Self::carries) the claim: each feasible one, and each tied with this + /// node's feasibility distance. + pub fn eligible(&self, id: Id) -> Vec> { if self.origins.contains_key(&id) { return Vec::new(); } + let fd = self.fd.get(&id); (self.rx.get(&id).into_iter().flatten()) - .filter(|(_, entry)| !entry.held && self.feasible(id, entry.advert)) - .filter_map(|(&next, entry)| Some((next, add(entry.advert.metric, self.neighbors[&next].link.cost)?))) + .filter(|(_, entry)| !entry.held) + .filter_map(|(&next, entry)| { + let metric = add(entry.advert.metric, self.neighbors[&next].link.cost)?; + let tied = fd.is_some_and(|&fd| fd == (entry.advert.seqno, entry.advert.metric)); + (tied || self.feasible(id, entry.advert)).then_some(Eligible { next, metric, tied }) + }) .collect() } diff --git a/rs/moq-net/src/route/node.rs b/rs/moq-net/src/route/node.rs index c17fc75ddbc..f3b524e0c43 100644 --- a/rs/moq-net/src/route/node.rs +++ b/rs/moq-net/src/route/node.rs @@ -678,8 +678,10 @@ impl State { /// there is none. Only `claims`, whose shown state changed, are served anew, unless /// the routes changed too. fn home(&mut self, id: Id, claims: BTreeSet, ended: &BTreeMap<(Id, Path), bool>) { - let routes: Routes = (self.router.routes(id).into_iter()) - .map(|(key, metric)| (key, metric, self.router.name(key, id))) + // A tied route needs a broadcast's hash, which the claim ranking here lacks. + let routes: Routes = (self.router.eligible(id).into_iter()) + .filter(|route| !route.tied) + .map(|route| (route.next, route.metric, self.router.name(route.next, id))) .collect(); let last = self.routes.get(&id).map_or(&[][..], Vec::as_slice); let claims = match last == routes.as_slice() { diff --git a/rs/moq-net/src/route/tests/cache.rs b/rs/moq-net/src/route/tests/cache.rs new file mode 100644 index 00000000000..60ee688ac9c --- /dev/null +++ b/rs/moq-net/src/route/tests/cache.rs @@ -0,0 +1,1116 @@ +//! Cache-aware routing over the route layer, as `quest/m3/cache-routing/README.md` +//! decides it. +//! +//! [`Mesh`] adds per-broadcast demand and forwarding to [`Sim`]: readers join and +//! leave, and each relay pulls a broadcast it has demand for through one neighbor, +//! its parent. A relay picks the cheapest neighbor its eligibility rule allows +//! ([`Router::eligible`], a tied route only toward the lower rendezvous hash), where +//! a warm neighbor costs the link alone and a cold one the link plus its cold +//! metric; ties go to warm, then to the lower hash. A copy with no demand left +//! lingers. +//! +//! Every pull names whom it serves ([`Serves`]): a pull whose whole demand traces +//! back to one relay names that relay, its root. A carrier tells each neighbor whose +//! cold metric is at least its own that it is warm, naming whom its copy serves, and +//! the neighbor counts it warm unless that is the neighbor's own root or itself: +//! demand a neighbor causes, through any number of relays, never warms a carrier +//! toward it. A carrier narrows what it tells only once the linger runs out. +//! +//! Each relay recomputes its choices in the same step as its router changes, as the +//! router's own node does, and every step checks that each broadcast's parents form +//! no cycle. Warm routes ride the stream carrying the sender's routes, so a +//! ROUTE_END retires them; subscriptions ride streams of their own, unordered +//! against both. Lingers run out only when a fixture or sweep says time passed. + +use super::*; + +/// A path its origin serves under one instance: two broadcasts sharing a path +/// and origin are two instances of it. +#[derive(Clone, Copy, Debug)] +struct Broadcast { + path: &'static str, + origin: Node, +} + +/// Whom a pull or a copy serves. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(super) enum Serves { + /// Demand of its own: a reader, or pulls tracing back to two or more relays. + #[default] + Alone, + /// Demand that traces back to this one relay alone. + Relay(Node), + /// Demand some of which came from a relay that names no one. + Unknown, +} + +/// A warm route: whom the sender's copy serves, and what keeping it costs a +/// receiver it serves alone. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) struct Warm { + pub serves: Serves, + /// The sender's price for its own pull, the cost of the copy to the demand it serves. + pub cost: Metric, +} + +/// One node's state for one broadcast. +#[derive(Clone, Debug, Default)] +struct Carry { + reader: bool, + /// Neighbors whose subscription arrived and was not cancelled, and whom each serves. + downstream: BTreeMap, + /// The neighbor this node pulls the broadcast through. + parent: Option, + /// Whom this node's pull serves, as it last told its parent. + serves: Serves, + /// Demand is gone and the copy lingers. + lingering: bool, + /// What this node advertises warm, or `None` when cold. + advert: Option, + /// The advert is narrower than it was, and the old one stands until the linger + /// runs out. + fading: bool, + /// What each neighbor was last told. + told: BTreeMap, + /// Each neighbor's warm route that arrived. + warm: BTreeMap, +} + +/// What the cache layer sent and moved. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(super) struct CacheStats { + /// Warm route starts, updates, and ends. + pub warm: usize, + /// Subscriptions, updates, and cancels. + pub subscribes: usize, + /// A relay moved a copy from one parent to another. + pub moves: usize, +} + +pub(super) struct Mesh { + pub sim: Sim, + broadcasts: Vec, + carry: Vec>, + /// Subscriptions, each broadcast's on a stream of its own. + subs: BTreeMap<(Node, Node, usize), VecDeque>, + /// `(node, neighbor)`: the neighbor sent `node` GOAWAY. + draining: BTreeSet<(Node, Node)>, + /// Relays that never negotiated warm routes: each pulls through its selected + /// route, sends no warm route, and its pulls name no one. + legacy: BTreeSet, + pub stats: CacheStats, + /// Warm messages per broadcast. + warm: Vec, +} + +/// The rendezvous hash that orders a tie, by node index standing in for the Node ID. +fn hash(path: &str, node: Node) -> u64 { + fnv(path, &[node as u64]) +} + +/// A change to one subscription. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Sub { + /// Pull, or update whom the pull serves. + Pull(Serves), + Cancel, +} + +/// A neighbor a node may pull a broadcast through. +#[derive(Clone, Copy, Debug)] +struct Candidate { + next: Node, + /// The node's metric through it. + cold: Metric, + /// The link to it alone, what pulling through it costs when it is warm. + link: Metric, + draining: bool, +} + +impl Mesh { + pub fn new(nodes: usize) -> Self { + Self { + sim: Sim::new(nodes), + broadcasts: Vec::new(), + carry: vec![Vec::new(); nodes], + subs: BTreeMap::new(), + draining: BTreeSet::new(), + legacy: BTreeSet::new(), + stats: CacheStats::default(), + warm: Vec::new(), + } + } + + fn nodes(&self) -> usize { + self.carry.len() + } + + /// A publisher at `node`, a leaf that originates `id` with a claim covering + /// every broadcast path. Called before its links, which a leaf configures. + pub fn origin(&mut self, node: Node, id: Id) { + self.sim.leaf(node); + self.sim.originate(node, id, 0); + self.sim.announce(node, "live/", 0, None); + self.step(); + } + + /// `node` never negotiates warm routes. + pub fn legacy(&mut self, node: Node) { + self.legacy.insert(node); + } + + /// A broadcast at `path` served by the origin at `origin`. + pub fn broadcast(&mut self, path: &'static str, origin: Node) -> usize { + self.broadcasts.push(Broadcast { path, origin }); + for carry in &mut self.carry { + carry.push(Carry::default()); + } + self.warm.push(0); + self.broadcasts.len() - 1 + } + + pub fn join(&mut self, a: Node, b: Node, cost: Metric) { + self.sim.join(a, b, cost); + self.step(); + } + + /// A reader at `node` starts or stops reading `broadcast`. + pub fn read(&mut self, node: Node, broadcast: usize, reading: bool) { + self.carry[node][broadcast].reader = reading; + self.step(); + } + + pub fn reading(&self, node: Node, broadcast: usize) -> bool { + self.carry[node][broadcast].reader + } + + pub fn connect(&mut self, a: Node, b: Node) { + self.sim.connect(a, b); + self.step(); + } + + /// The session closes: everything on it goes, subscriptions and warm routes too. + pub fn disconnect(&mut self, a: Node, b: Node) { + self.sim.disconnect(a, b); + for (x, y) in [(a, b), (b, a)] { + self.subs.retain(|&(from, to, _), _| (from, to) != (x, y)); + self.draining.remove(&(x, y)); + for carry in &mut self.carry[x] { + carry.downstream.remove(&y); + carry.told.remove(&y); + carry.warm.remove(&y); + } + } + self.step(); + } + + /// `node` sends GOAWAY on every session: its neighbors rank it last. + pub fn goaway(&mut self, node: Node) { + let peers: Vec = self.sim.peers(node).collect(); + for peer in peers { + self.draining.insert((peer, node)); + } + self.sim.goaway(node); + self.step(); + } + + /// The origin at `node` advances its seqno. + pub fn bump(&mut self, node: Node) { + self.sim.bump(node); + self.step(); + } + + pub fn hold(&mut self, from: Node, to: Node) { + self.sim.hold(from, to); + } + + pub fn release(&mut self, from: Node, to: Node) { + self.sim.release(from, to); + } + + /// The linger at `node` for `broadcast` runs out: a copy with no demand goes, and + /// a narrowed advert takes effect. + pub fn tick(&mut self, node: Node, broadcast: usize) { + let carry = &mut self.carry[node][broadcast]; + let parent = match carry.lingering { + true => { + carry.lingering = false; + carry.parent.take() + } + false => None, + }; + if carry.fading { + carry.fading = false; + self.carry[node][broadcast].advert = self.desired(node, broadcast); + } + if let Some(parent) = parent { + self.subscribe(node, parent, broadcast, Sub::Cancel); + } + self.step(); + } + + /// Deliver the next message on `from`'s route stream to `to`. + pub fn deliver(&mut self, from: Node, to: Node) -> bool { + let delivered = self.sim.deliver(from, to); + self.step(); + delivered + } + + pub fn deliver_ack(&mut self, from: Node, to: Node) -> bool { + let delivered = self.sim.deliver_ack(from, to); + self.step(); + delivered + } + + /// Deliver the next subscription change for `broadcast` from `from` to `to`. + pub fn deliver_sub(&mut self, from: Node, to: Node, broadcast: usize) -> bool { + let Some(sub) = self.subs.get_mut(&(from, to, broadcast)).and_then(VecDeque::pop_front) else { + return false; + }; + self.subs.retain(|_, queue| !queue.is_empty()); + let downstream = &mut self.carry[to][broadcast].downstream; + match sub { + Sub::Pull(serves) => downstream.insert(from, serves), + Sub::Cancel => downstream.remove(&from), + }; + self.step(); + true + } + + /// Every queue with something to deliver: routes, echoes, then subscriptions. + fn busy(&self) -> Vec { + (self.sim.busy().into_iter().map(|(from, to)| Queue::Route(from, to))) + .chain(self.sim.busy_acks().into_iter().map(|(from, to)| Queue::Ack(from, to))) + .chain(self.subs.keys().map(|&(from, to, b)| Queue::Sub(from, to, b))) + .collect() + } + + fn deliver_one(&mut self, queue: Queue) { + match queue { + Queue::Route(from, to) => self.deliver(from, to), + Queue::Ack(from, to) => self.deliver_ack(from, to), + Queue::Sub(from, to, b) => self.deliver_sub(from, to, b), + }; + } + + /// Deliver round-robin until every unheld queue is empty, letting no linger run + /// out. False if it never settles. + pub fn run(&mut self) -> bool { + for _ in 0..10_000 { + let ready: Vec = (self.busy().into_iter()) + .filter(|queue| !self.sim.held.contains(&queue.pair())) + .collect(); + if ready.is_empty() { + return true; + } + for queue in ready { + self.deliver_one(queue); + } + } + false + } + + /// Run, then let every linger run out, until nothing changes. + pub fn settle(&mut self) -> bool { + for _ in 0..100 { + if !self.run() { + return false; + } + let lingering: Vec<(Node, usize)> = (0..self.nodes()) + .flat_map(|node| (0..self.broadcasts.len()).map(move |b| (node, b))) + .filter(|&(node, b)| { + let carry = &self.carry[node][b]; + carry.lingering || carry.fading + }) + .collect(); + if lingering.is_empty() { + return true; + } + for (node, b) in lingering { + self.tick(node, b); + } + } + false + } + + pub fn parent(&self, node: Node, broadcast: usize) -> Option { + self.carry[node][broadcast].parent + } + + /// Whether `node` pulls `broadcast`. + pub fn carries(&self, node: Node, broadcast: usize) -> bool { + self.parent(node, broadcast).is_some() + } + + /// Whether `node` counts `from` warm for `broadcast`: its copy stays whether or not + /// `node` pulls through it. + pub fn warm_at(&self, node: Node, from: Node, broadcast: usize) -> bool { + let carry = &self.carry[node][broadcast]; + carry.warm.get(&from).is_some_and(|warm| match warm.serves { + Serves::Alone => true, + Serves::Relay(root) => match carry.serves { + Serves::Unknown => false, + Serves::Alone => root != node, + Serves::Relay(own) => root != node && root != own, + }, + Serves::Unknown => false, + }) + } + + /// What pulling `broadcast` through `c` costs `node`: the link alone through a warm + /// neighbor, the link plus the copy's cost through one whose copy serves only + /// `node`'s demand, and the cold metric through any other. + fn price(&self, node: Node, broadcast: usize, c: &Candidate) -> Metric { + match self.carry[node][broadcast].warm.get(&c.next) { + _ if self.warm_at(node, c.next, broadcast) => c.link, + Some(warm) if warm.serves != Serves::Unknown => c.link + warm.cost, + _ => c.cold, + } + } + + /// The relays pulling `broadcast` straight from its origin. + pub fn copies(&self, broadcast: usize) -> usize { + let origin = self.broadcasts[broadcast].origin; + (0..self.nodes()) + .filter(|&node| self.parent(node, broadcast) == Some(origin)) + .count() + } + + /// Warm messages sent for `broadcast`. + pub fn warm_sent(&self, broadcast: usize) -> usize { + self.warm[broadcast] + } + + fn id(&self, broadcast: usize) -> Id { + self.sim.proven(self.broadcasts[broadcast].origin) + } + + fn subscribe(&mut self, from: Node, to: Node, broadcast: usize, sub: Sub) { + self.subs.entry((from, to, broadcast)).or_default().push_back(sub); + self.stats.subscribes += 1; + } + + /// Take the deliveries, recompute every node's choices, and check for a loop. + fn step(&mut self) { + for note in std::mem::take(&mut self.sim.notes) { + match note { + Note::Warm { + from, + to, + broadcast, + warm, + } => { + let map = &mut self.carry[to][broadcast].warm; + match warm { + Some(warm) => map.insert(from, warm), + None => map.remove(&from), + }; + } + Note::Ended { from, to, id } => { + for b in 0..self.broadcasts.len() { + if self.id(b) == id { + self.carry[to][b].warm.remove(&from); + } + } + } + } + } + for node in 0..self.nodes() { + for b in 0..self.broadcasts.len() { + self.choose(node, b); + self.tell(node, b); + } + } + self.check_loops(); + } + + /// The neighbors `node` may pull `broadcast` through. + fn candidates(&self, node: Node, broadcast: usize) -> Vec { + let path = self.broadcasts[broadcast].path; + (self.sim.nodes[node].eligible(self.id(broadcast)).into_iter()) + .filter(|route| !route.tied || hash(path, route.next) < hash(path, node)) + .map(|route| Candidate { + next: route.next, + cold: route.metric, + link: self.sim.links[node][route.next].expect("a configured link").cost, + draining: self.draining.contains(&(node, route.next)), + }) + .collect() + } + + /// The neighbor `node` pulls `broadcast` through: its current parent unless another + /// candidate is strictly cheaper, otherwise the cheapest, ties going to warm and + /// then to the lower hash. + fn best(&self, node: Node, broadcast: usize) -> Option { + if self.legacy.contains(&node) { + return self.sim.nodes[node] + .selected(self.id(broadcast)) + .and_then(|route| route.next); + } + let path = self.broadcasts[broadcast].path; + let ranked: Vec<_> = (self.candidates(node, broadcast).into_iter()) + .map(|c| { + let warm = self.warm_at(node, c.next, broadcast); + ( + (c.draining, self.price(node, broadcast, &c)), + !warm, + hash(path, c.next), + c.next, + ) + }) + .collect(); + let best = ranked.iter().min()?; + let parent = self.carry[node][broadcast].parent; + match ranked.iter().find(|rank| Some(rank.3) == parent) { + Some(current) if current.0 <= best.0 => parent, + _ => Some(best.3), + } + } + + /// What `node` pays for its own pull of `broadcast`. + fn cost(&self, node: Node, broadcast: usize) -> Option { + let parent = self.carry[node][broadcast].parent?; + let c = (self.candidates(node, broadcast).into_iter()).find(|c| c.next == parent)?; + Some(self.price(node, broadcast, &c)) + } + + /// Whom `node`'s demand for `broadcast` serves: itself for a reader, the one root + /// its pulls trace back to, or no one in particular across two or more roots. + fn served(&self, node: Node, broadcast: usize) -> Serves { + let carry = &self.carry[node][broadcast]; + if self.legacy.contains(&node) { + return Serves::Unknown; + } + if carry.reader { + return Serves::Alone; + } + let mut roots = BTreeSet::new(); + let mut unknown = false; + for (&from, &serves) in &carry.downstream { + match serves { + Serves::Alone => roots.insert(from), + Serves::Relay(root) => roots.insert(root), + Serves::Unknown => { + unknown = true; + false + } + }; + } + match (roots.len(), unknown) { + (2.., _) => Serves::Alone, + (1, false) => Serves::Relay(*roots.first().expect("one root")), + _ => Serves::Unknown, + } + } + + /// What `node` would advertise for `broadcast` now: warm while it pulls for demand + /// it can name. + fn desired(&self, node: Node, broadcast: usize) -> Option { + let carry = &self.carry[node][broadcast]; + let demand = carry.reader || !carry.downstream.is_empty(); + if node == self.broadcasts[broadcast].origin || self.legacy.contains(&node) || !demand { + return None; + } + let serves = self.served(node, broadcast); + let cost = self.cost(node, broadcast)?; + (serves != Serves::Unknown).then_some(Warm { serves, cost }) + } + + /// Recompute `node`'s parent for `broadcast`, and move or update the subscription. + fn choose(&mut self, node: Node, broadcast: usize) { + if node == self.broadcasts[broadcast].origin { + return; + } + let carry = &mut self.carry[node][broadcast]; + let demand = carry.reader || !carry.downstream.is_empty(); + if demand { + carry.lingering = false; + } else if carry.parent.is_some() { + carry.lingering = true; + } + let want = demand || carry.lingering; + // A lingering copy serves whom it last served. + let serves = match demand { + true => self.served(node, broadcast), + false => self.carry[node][broadcast].serves, + }; + let parent = match want { + true => self.best(node, broadcast), + false => None, + }; + let carry = &mut self.carry[node][broadcast]; + let (old, told) = (carry.parent, carry.serves); + carry.parent = parent; + carry.serves = serves; + if parent != old + && let Some(old) = old + && self.sim.connected(node, old) + { + self.subscribe(node, old, broadcast, Sub::Cancel); + } + if let Some(parent) = parent + && (Some(parent) != old || serves != told) + { + self.subscribe(node, parent, broadcast, Sub::Pull(serves)); + } + if old.is_some() && parent.is_some() && parent != old { + self.stats.moves += 1; + } + } + + /// Update `node`'s advert for `broadcast`, widening at once and narrowing once the + /// linger runs out, and tell each neighbor that may use it. + fn tell(&mut self, node: Node, broadcast: usize) { + let desired = self.desired(node, broadcast); + let carry = &mut self.carry[node][broadcast]; + // Narrower: some neighbor that counted the copy warm would no longer. + let narrower = match (carry.advert, desired) { + (Some(old), Some(new)) => old.serves != new.serves && new.serves != Serves::Alone, + (Some(_), None) => true, + (None, _) => false, + }; + carry.fading = narrower; + if !narrower { + carry.advert = desired; + } + let advert = carry.advert; + let id = self.id(broadcast); + let peers: Vec = (self.sim.peers(node)) + .filter(|&peer| !self.sim.leaf[peer] && !self.legacy.contains(&peer)) + .collect(); + for to in peers { + let router = &self.sim.nodes[node]; + // A warm route names the route it travels under, which a ROUTE_END retires. + let routed = router.sent.contains_key(&(id, to)); + let own = router.selected(id).map(|route| route.metric); + // A neighbor with no usable route to the origin has an infinite cold metric. + let behind = (router.rx.get(&id).and_then(|routes| routes.get(&to))) + .is_none_or(|entry| entry.held || own.is_some_and(|own| entry.advert.metric >= own)); + let carry = &mut self.carry[node][broadcast]; + if !routed { + carry.told.remove(&to); + } + let want = advert.filter(|_| routed && own.is_some() && behind); + if want == carry.told.get(&to).copied() { + continue; + } + match want { + Some(warm) => carry.told.insert(to, warm), + None => carry.told.remove(&to), + }; + let msg = Msg::Warm { broadcast, warm: want }; + self.sim.queues.entry((node, to)).or_default().push_back(msg); + self.stats.warm += 1; + self.warm[broadcast] += 1; + } + } + + /// Each broadcast's parents form no cycle. + fn check_loops(&self) { + for b in 0..self.broadcasts.len() { + for node in 0..self.nodes() { + let mut at = node; + for hop in 0.. { + let Some(next) = self.parent(at, b) else { + break; + }; + assert!( + hop <= self.nodes(), + "broadcast {b} loops from node {node}: {:?}", + (0..self.nodes()).map(|n| self.parent(n, b)).collect::>() + ); + at = next; + } + } + } + } + + /// The steady state once demand and topology stop changing: every reader is + /// delivered, subscriptions and warm routes match what their senders hold, nothing + /// would change, every carrier has demand, and no relay could pull more cheaply + /// given the copies the others hold. + pub fn steady(&mut self, label: &str) { + for (b, broadcast) in self.broadcasts.iter().enumerate() { + for node in 0..self.nodes() { + let carry = &self.carry[node][b]; + if carry.reader { + let mut at = node; + while let Some(next) = self.parent(at, b) { + at = next; + } + assert_eq!(at, broadcast.origin, "{label}: broadcast {b} reaches node {node}"); + } + let feeds: BTreeMap = (0..self.nodes()) + .filter(|&n| self.parent(n, b) == Some(node)) + .map(|n| (n, self.carry[n][b].serves)) + .collect(); + assert_eq!(carry.downstream, feeds, "{label}: broadcast {b} at node {node}"); + assert!( + !carry.fading && !carry.lingering, + "{label}: broadcast {b} at node {node}" + ); + for to in 0..self.nodes() { + assert_eq!( + carry.told.get(&to), + self.carry[to][b].warm.get(&node), + "{label}: broadcast {b} warm from {node} at {to}" + ); + } + if node == broadcast.origin || carry.parent.is_none() { + continue; + } + assert!( + carry.reader || !feeds.is_empty(), + "{label}: node {node} carries broadcast {b} for no one" + ); + } + for node in 0..self.nodes() { + if node == broadcast.origin || self.legacy.contains(&node) { + continue; + } + let Some(cost) = self.cost(node, b) else { + continue; + }; + let parent = self.parent(node, b); + let draining = parent.is_some_and(|parent| self.draining.contains(&(node, parent))); + for c in self.candidates(node, b) { + assert!( + (c.draining, self.price(node, b, &c)) >= (draining, cost), + "{label}: node {node} would move broadcast {b} from {parent:?} to {}", + c.next + ); + } + } + } + let stats = self.stats; + let queued: usize = self.sim.queues.values().map(VecDeque::len).sum(); + self.step(); + assert_eq!(self.stats, stats, "{label}: a choice changed at quiescence"); + assert_eq!(queued, self.sim.queues.values().map(VecDeque::len).sum::()); + } +} + +/// A queue the cache layer delivers from. +#[derive(Clone, Copy, Debug)] +enum Queue { + Route(Node, Node), + Ack(Node, Node), + Sub(Node, Node, usize), +} + +impl Queue { + fn pair(self) -> (Node, Node) { + match self { + Self::Route(from, to) | Self::Ack(from, to) | Self::Sub(from, to, _) => (from, to), + } + } +} + +/// An origin behind relay `a`, then relays `a`, `b`, `c` in a chain priced 3, with +/// a skip from `a` to `c` priced `skip`, and one broadcast. +fn chain(skip: Metric) -> (Mesh, usize) { + let (o, a, b, c) = (0, 1, 2, 3); + let mut mesh = Mesh::new(4); + mesh.origin(o, 7); + mesh.join(o, a, 1); + mesh.join(a, b, 3); + mesh.join(b, c, 3); + mesh.join(a, c, skip); + let x = mesh.broadcast("live/x", o); + assert!(mesh.settle()); + (mesh, x) +} + +/// A cold broadcast takes the skip. A reader on the skipped relay makes it pull +/// through `a` and warm toward `c`, which joins it. When that reader leaves, its +/// warm route goes after the linger and `c` returns to the skip. +#[test] +fn a_skipped_relay_draws_its_neighbor_while_it_has_a_reader() { + let (a, b, c) = (1, 2, 3); + let (mut mesh, x) = chain(5); + mesh.read(c, x, true); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, x), Some(a), "a cold broadcast takes the skip"); + assert!(!mesh.carries(b, x)); + mesh.steady("cold"); + + mesh.read(b, x, true); + assert!(mesh.settle()); + assert_eq!(mesh.parent(b, x), Some(a)); + assert_eq!(mesh.parent(c, x), Some(b), "c joins b's copy"); + mesh.steady("warm"); + + mesh.read(b, x, false); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, x), Some(a), "back to the skip"); + assert!(!mesh.carries(b, x)); + mesh.steady("cold again"); +} + +/// A relay whose only consumer is `c` withdraws its warm route toward `c`. With +/// a cheaper skip, `c` bypasses it; when the relay is `c`'s cheapest cold route, +/// `c` stays and the relay keeps carrying for it alone. +#[test] +fn a_relay_feeding_one_consumer_stays_only_on_its_cheapest_cold_route() { + let (a, b, c) = (1, 2, 3); + for (skip, stays) in [(5, false), (9, true)] { + let (mut mesh, x) = chain(skip); + mesh.read(b, x, true); + mesh.read(c, x, true); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, x), Some(b), "skip {skip}"); + mesh.read(b, x, false); + assert!(mesh.settle()); + assert!(!mesh.warm_at(c, b, x), "skip {skip}: b withdrew"); + assert_eq!(mesh.parent(c, x), Some(if stays { b } else { a }), "skip {skip}"); + assert_eq!(mesh.carries(b, x), stays, "skip {skip}"); + mesh.steady(&format!("skip {skip}")); + } +} + +/// `r2`'s cheapest cold route runs through `r1` to `r3`, so `r3`'s only demand is +/// `r1`'s pull on `r2`'s behalf. Counted as demand besides `r2`'s, it would warm `r3` +/// toward `r2`: `r2` would join `r3` over its direct link, `r1`'s pull and with it +/// `r3`'s warmth would go, and `r2` would return to `r1`, once per linger forever. +/// A pull names the relay it serves, so `r3` stays cold toward `r2`. +#[test] +fn a_pull_on_a_neighbors_behalf_never_warms_its_parent_toward_it() { + let (o, r1, r2, r3) = (0, 1, 2, 3); + let mut mesh = Mesh::new(4); + mesh.origin(o, 7); + mesh.join(o, r3, 1); + mesh.join(r3, r1, 1); + mesh.join(r1, r2, 1); + mesh.join(r2, r3, 3); + let x = mesh.broadcast("live/x", o); + mesh.read(r2, x, true); + assert!(mesh.settle()); + assert_eq!(mesh.parent(r2, x), Some(r1)); + assert_eq!(mesh.parent(r1, x), Some(r3)); + assert!(!mesh.warm_at(r2, r3, x)); + mesh.steady("behalf"); + + // A reader of its own makes r1's pull stand alone and r3 warm toward r2, but r1, + // warm too, is the cheaper join. + mesh.read(r1, x, true); + assert!(mesh.settle()); + assert!(mesh.warm_at(r2, r3, x)); + assert_eq!(mesh.parent(r2, x), Some(r1)); + mesh.steady("own reader"); +} + +const PATHS: [&str; 8] = [ + "live/0", "live/1", "live/2", "live/3", "live/4", "live/5", "live/6", "live/7", +]; + +/// Two relays at equal cold metric, both warm toward each other: per broadcast, +/// only the lower hash keeps its uplink and the other joins it, so the pair pulls +/// one copy, and the owner differs across broadcasts. +#[test] +fn equal_relays_pull_one_copy_owned_by_the_lower_hash() { + let (o, x, y) = (0, 1, 2); + let mut mesh = Mesh::new(3); + mesh.origin(o, 7); + mesh.join(o, x, 3); + mesh.join(o, y, 3); + mesh.join(x, y, 1); + let broadcasts: Vec = PATHS.iter().map(|path| mesh.broadcast(path, o)).collect(); + for &b in &broadcasts { + mesh.read(x, b, true); + mesh.read(y, b, true); + } + assert!(mesh.settle()); + let mut owners = BTreeSet::new(); + for (&b, path) in broadcasts.iter().zip(PATHS) { + let (low, high) = match hash(path, x) < hash(path, y) { + true => (x, y), + false => (y, x), + }; + assert_eq!(mesh.parent(low, b), Some(o), "{path}"); + assert_eq!(mesh.parent(high, b), Some(low), "{path}"); + assert_eq!(mesh.copies(b), 1, "{path}"); + owners.insert(low); + } + assert_eq!(owners.len(), 2, "the hash spreads the owner"); + mesh.steady("pair"); +} + +/// A ring of relays, each holding its neighbors' reports from while every relay +/// had its own link to the origin, loses all but one of those links. Each report +/// is now lower than the truth, the hazard that let a whole ring let go at once +/// under the old rank. Delivered in any order, no broadcast loops, and once quiet +/// every reader is fed through the one link left. +#[test] +fn a_ring_of_stale_lower_reports_forms_no_loop() { + const RING: usize = 5; + let o = 0; + let relays: Vec = (1..=RING).collect(); + for seed in 0..20u64 { + let mut mesh = Mesh::new(RING + 1); + mesh.origin(o, 7); + for (i, &relay) in relays.iter().enumerate() { + mesh.join(o, relay, 1); + mesh.join(relay, relays[(i + 1) % RING], 1); + } + let x = mesh.broadcast("live/x", o); + for &relay in &relays { + mesh.read(relay, x, true); + } + assert!(mesh.settle()); + for (i, &relay) in relays.iter().enumerate() { + let next = relays[(i + 1) % RING]; + mesh.hold(relay, next); + mesh.hold(next, relay); + } + for &relay in &relays[1..] { + mesh.disconnect(o, relay); + } + let mut rng = XorShift(0x5717_e000 + seed); + loop { + let busy = mesh.busy(); + if busy.is_empty() { + break; + } + mesh.deliver_one(busy[rng.next(busy.len() as u64) as usize]); + } + assert!(mesh.settle()); + mesh.steady(&format!("seed {seed}")); + assert_eq!(mesh.copies(x), 1, "seed {seed}"); + assert_eq!(mesh.parent(relays[0], x), Some(o), "seed {seed}"); + } +} + +/// `b` withdraws its warm route toward `c` while `c`'s subscription to it is on +/// its way. The subscription reaches `b` with no copy left, and `b` pulls it +/// along its own route, as for any request; `c` then moves back to the skip. +#[test] +fn a_request_crossing_a_warm_withdrawal_is_forwarded_without_a_loop() { + let (a, b, c) = (1, 2, 3); + let (mut mesh, x) = chain(5); + mesh.read(b, x, true); + assert!(mesh.settle()); + assert!(mesh.warm_at(c, b, x)); + mesh.hold(c, b); + mesh.hold(b, c); + mesh.read(c, x, true); + assert!(mesh.run()); + assert_eq!(mesh.parent(c, x), Some(b), "c's subscription waits"); + + mesh.read(b, x, false); + mesh.tick(b, x); + assert!(mesh.run()); + assert!(!mesh.carries(b, x), "b let its copy go"); + assert!(mesh.warm_at(c, b, x), "its withdrawal waits"); + + assert!(mesh.deliver_sub(c, b, x)); + assert_eq!(mesh.parent(b, x), Some(a), "b forwards the request along its own route"); + mesh.release(c, b); + mesh.release(b, c); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, x), Some(a)); + assert!(!mesh.carries(b, x)); + mesh.steady("crossed"); +} + +/// A warm route for another instance of the same path is never joined: `c`'s +/// reader of the new instance takes the skip although `b` is warm for the old one. +#[test] +fn a_warm_route_for_another_instance_is_never_joined() { + let (o, a, b, c) = (0, 1, 2, 3); + let (mut mesh, old) = chain(5); + let new = mesh.broadcast("live/x", o); + mesh.read(b, old, true); + mesh.read(c, new, true); + assert!(mesh.settle()); + assert!(mesh.warm_at(c, b, old)); + assert_eq!(mesh.parent(c, new), Some(a)); + assert!(!mesh.carries(b, new)); + mesh.steady("instances"); +} + +/// `k` relays on a LAN, each with its own uplink and a reader of every broadcast. +/// In a full mesh each broadcast crosses one uplink. In a ring a relay joins only a +/// neighbor with a lower hash, so each relay whose neighbors both hash higher keeps +/// an uplink copy, which the epic accepts. +#[test] +fn lan_relays_share_uplink_copies() { + const K: usize = 6; + let o = 0; + let relays: Vec = (1..=K).collect(); + for full in [true, false] { + let mut mesh = Mesh::new(K + 1); + mesh.origin(o, 7); + for i in 0..K { + mesh.join(o, relays[i], 3); + } + for i in 0..K { + for j in i + 1..K { + if full || j == i + 1 || (i == 0 && j == K - 1) { + mesh.join(relays[i], relays[j], 1); + } + } + } + let broadcasts: Vec = PATHS.iter().map(|path| mesh.broadcast(path, o)).collect(); + for &b in &broadcasts { + for &relay in &relays { + mesh.read(relay, b, true); + } + } + assert!(mesh.settle()); + mesh.steady(if full { "full mesh" } else { "ring" }); + let mut copies = 0; + for (&b, path) in broadcasts.iter().zip(PATHS) { + let minima = (0..K) + .filter(|&i| { + let h = hash(path, relays[i]); + h < hash(path, relays[(i + 1) % K]) && h < hash(path, relays[(i + K - 1) % K]) + }) + .count(); + assert_eq!(mesh.copies(b), if full { 1 } else { minima }, "{path}"); + copies += mesh.copies(b); + } + // Recorded: one copy per broadcast in the full mesh, 13 for 8 in the ring. + assert_eq!(copies, if full { 8 } else { 13 }); + } +} + +/// While a broadcast is pulled: its origin advances its seqno, a relay loses its +/// route, and a relay drains. The seqno reaches `c` through the skip before `b` +/// passes it on, and `b`'s older route is then ineligible, so `c` moves off `b` +/// and back: what staying loop-free across seqnos costs a joined copy. +#[test] +fn a_seqno_advance_a_lost_route_and_a_drain_keep_pulls_loop_free() { + let (o, a, b, c) = (0, 1, 2, 3); + let (mut mesh, x) = chain(5); + mesh.read(b, x, true); + mesh.read(c, x, true); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, x), Some(b)); + + let moves = mesh.stats.moves; + mesh.bump(o); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, x), Some(b)); + assert_eq!(mesh.stats.moves - moves, 2, "c left b while b's route was older"); + mesh.steady("seqno"); + + mesh.disconnect(a, b); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, x), Some(a)); + assert_eq!(mesh.parent(b, x), Some(c), "b reaches the origin only through c"); + mesh.steady("route loss"); + + mesh.connect(a, b); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, x), Some(b)); + mesh.goaway(b); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, x), Some(a), "c leaves the draining relay"); + mesh.steady("drain"); +} + +/// Random relay meshes with two origins under churn: readers come and go, +/// sessions drop and return, messages are held and delivered out of step, lingers +/// run out, origins advance their seqno, and relays drain. No broadcast's parents +/// ever form a cycle. Once churn stops the mesh settles on the steady state, and a +/// broadcast nobody reads has sent no warm route. +#[test] +fn sweep_cache_routing() { + sweep(100); +} + +/// The sweep at length. Loop-free throughout, but 4 of 3000 scenarios (the first is +/// 478) never settle: two carriers each keep a narrowed advert standing while two +/// relays swap between them. Run it with `--ignored` while fixing convergence. +#[test] +#[ignore = "4 of 3000 scenarios never settle"] +fn sweep_cache_routing_long() { + sweep(3000); +} + +fn sweep(scenarios: usize) { + const RELAYS: usize = 6; + let mut rng = XorShift(0xcac4_e5ee); + for scenario in 0..scenarios { + let label = format!("scenario {scenario}"); + let origins = [RELAYS, RELAYS + 1]; + let mut mesh = Mesh::new(RELAYS + 2); + mesh.origin(origins[0], 100); + mesh.origin(origins[1], 200); + if rng.chance(30) { + mesh.legacy(rng.next(RELAYS as u64) as usize); + } + let mut links: Vec<(Node, Node)> = (0..RELAYS).map(|relay| (relay, (relay + 1) % RELAYS)).collect(); + for _ in 0..2 { + let (a, b) = (rng.next(RELAYS as u64) as usize, rng.next(RELAYS as u64) as usize); + if a != b && !links.contains(&(a, b)) && !links.contains(&(b, a)) { + links.push((a, b)); + } + } + for origin in origins { + let a = rng.next(RELAYS as u64) as usize; + links.push((origin, a)); + links.push((origin, (a + 1 + rng.next(RELAYS as u64 - 1) as usize) % RELAYS)); + } + for &(a, b) in &links { + mesh.join(a, b, 1 + rng.next(5)); + } + let read = [ + mesh.broadcast("live/a", origins[0]), + mesh.broadcast("live/b", origins[0]), + mesh.broadcast("live/a", origins[1]), + // Another instance of the first. + mesh.broadcast("live/a", origins[0]), + ]; + let idle = mesh.broadcast("live/idle", origins[0]); + assert!(mesh.settle(), "{label}"); + + for _ in 0..60 { + match rng.next(12) { + 0..=2 => { + let (relay, b) = (rng.next(RELAYS as u64) as usize, read[rng.next(4) as usize]); + let reading = mesh.reading(relay, b); + mesh.read(relay, b, !reading); + } + 3 => { + let (a, b) = links[rng.next(links.len() as u64) as usize]; + match mesh.sim.connected(a, b) { + true => mesh.disconnect(a, b), + false => mesh.connect(a, b), + } + } + 4 => { + let (a, b) = links[rng.next(links.len() as u64) as usize]; + let (from, to) = if rng.chance(50) { (a, b) } else { (b, a) }; + match rng.chance(70) { + true => mesh.hold(from, to), + false => mesh.release(from, to), + } + } + 5 => mesh.tick(rng.next(RELAYS as u64) as usize, read[rng.next(4) as usize]), + 6 if rng.chance(50) => mesh.bump(origins[rng.next(2) as usize]), + 6 if rng.chance(20) => mesh.goaway(rng.next(RELAYS as u64) as usize), + _ => { + for _ in 0..rng.next(8) { + let busy = mesh.busy(); + if busy.is_empty() { + break; + } + mesh.deliver_one(busy[rng.next(busy.len() as u64) as usize]); + } + } + } + } + + for &(a, b) in &links { + mesh.release(a, b); + mesh.release(b, a); + if !mesh.sim.connected(a, b) { + mesh.connect(a, b); + } + } + assert!(mesh.settle(), "{label}"); + mesh.steady(&label); + assert_eq!(mesh.warm_sent(idle), 0, "{label}"); + } +} diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index f3060404833..fd046623919 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -5,6 +5,7 @@ //! are written with a trailing `/` where the contract means a wildcard; every //! lite-07 claim already covers the paths below it, so the slash is dropped. +mod cache; mod handoff; mod routing; @@ -104,6 +105,25 @@ enum Msg { id: Id, seqno: Seqno, }, + /// A warm route for one of [`cache`]'s broadcasts, on the stream carrying the + /// sender's routes. + Warm { + broadcast: usize, + warm: Option, + }, +} + +/// A delivery the cache layer reacts to. +#[derive(Clone, Copy, Debug)] +enum Note { + Warm { + from: Node, + to: Node, + broadcast: usize, + warm: Option, + }, + /// A ROUTE_END, which retires the warm routes under it. + Ended { from: Node, to: Node, id: Id }, } #[derive(Clone, Debug, PartialEq, Eq)] @@ -157,6 +177,8 @@ pub(super) struct Sim { requested: BTreeSet<(Node, Node, Id, Seqno)>, pub stats: Stats, pub events: Vec, + /// Deliveries the cache layer has yet to take. + notes: Vec, } impl Sim { @@ -177,6 +199,7 @@ impl Sim { requested: BTreeSet::new(), stats: Stats::default(), events: Vec::new(), + notes: Vec::new(), } } @@ -389,7 +412,12 @@ impl Sim { }; let router = &mut self.nodes[to]; match msg { - Msg::Route { id, reach } => router.route(from, id, reach), + Msg::Route { id, reach } => { + router.route(from, id, reach); + if reach.is_none() { + self.notes.push(Note::Ended { from, to, id }); + } + } Msg::Batch { id, changes, @@ -397,6 +425,12 @@ impl Sim { interest, } => router.batch(from, id, changes, floor, interest), Msg::Request { id, seqno } => router.request(from, id, seqno), + Msg::Warm { broadcast, warm } => self.notes.push(Note::Warm { + from, + to, + broadcast, + warm, + }), } self.stats.delivered += 1; self.flush(); From 6341038f2ded10fe90884233ba3f1dfe45476701 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 13:20:33 -0800 Subject: [PATCH 03/12] test(net): list the long cache sweep in the test inventory Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/route/tests/cache.rs | 4 ++-- sh/gh/test-inventory.allow | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/rs/moq-net/src/route/tests/cache.rs b/rs/moq-net/src/route/tests/cache.rs index 60ee688ac9c..b877847100b 100644 --- a/rs/moq-net/src/route/tests/cache.rs +++ b/rs/moq-net/src/route/tests/cache.rs @@ -940,8 +940,8 @@ fn lan_relays_share_uplink_copies() { for full in [true, false] { let mut mesh = Mesh::new(K + 1); mesh.origin(o, 7); - for i in 0..K { - mesh.join(o, relays[i], 3); + for &relay in &relays { + mesh.join(o, relay, 3); } for i in 0..K { for j in i + 1..K { diff --git a/sh/gh/test-inventory.allow b/sh/gh/test-inventory.allow index c8debe20bb3..7a96a3e8209 100644 --- a/sh/gh/test-inventory.allow +++ b/sh/gh/test-inventory.allow @@ -41,3 +41,6 @@ ignore moq-video::render::export::dx12::tests::media_foundation_encodes_the_expo ignore moq-net::test_interop::lite_tracks_interop /quest/m2/cluster-routing/routes/js.md ignore moq-tokio::datagram_tracks::datagram_tracks_javascript /quest/m2/cluster-routing/routes/js.md recipe test::datagram-tracks /quest/m2/cluster-routing/routes/js.md + +# Known to fail until the quest makes every scenario settle. +ignore moq-net::route::tests::cache::sweep_cache_routing_long /quest/m3/cache-routing/select.md From 520f68a847b3e1695baa8549cef3d63f660c3823 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Tue, 10 Nov 2026 16:39:53 -0800 Subject: [PATCH 04/12] feat(net)!: carry warm routes and Serves in moq-lite-07 The maintainer placed the cache-aware routing wire in lite-07 itself rather than a SETUP extension. The draft gains a Warm Routes section, WARM_START (0xC), WARM_UPDATE (0xD), and WARM_END (0xE) on the announce stream, and a trailing Serves on SUBSCRIBE and SUBSCRIBE_UPDATE. Rust decodes the warm routes and Serves and ignores them, saying Unknown; js/net says Alone, since it forwards nothing. The epic records the three decisions, and select.md implements against the landed wire. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 151 ++++++++++++++++++++++- js/net/src/lite/subscribe.test.ts | 5 +- js/net/src/lite/subscribe.ts | 26 ++++ js/net/src/lite/version.ts | 17 +++ quest/m3/cache-routing/README.md | 100 ++++++++------- quest/m3/cache-routing/select.md | 20 +-- rs/moq-net/src/lite/announce.rs | 133 +++++++++++++++++++- rs/moq-net/src/lite/instance.rs | 10 ++ rs/moq-net/src/lite/subscribe.rs | 91 +++++++++++++- rs/moq-net/src/lite/subscriber.rs | 5 +- rs/moq-net/src/lite/subscriber/routed.rs | 3 + rs/moq-net/src/lite/version.rs | 10 ++ 12 files changed, 505 insertions(+), 66 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index f026f67e54a..cd18694f846 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -415,6 +415,9 @@ The publisher replies with a single ANNOUNCE_OK message followed by the matching - ANNOUNCE_UPDATE: a claim's cost changed. - ANNOUNCE_RESTART: a claim now serves another Instance, or a higher Epoch of the same one. - ANNOUNCE_FLOOR: a batch of a route's claims is complete. +- WARM_START: the publisher keeps a copy of a broadcast whether or not the subscriber pulls through it (see [Warm Routes](#warm-routes)). +- WARM_UPDATE: a warm route's Serves or Warm Cost changed. +- WARM_END: the publisher withdrew a warm route. - SYNCED: the initial set is complete. In the other direction, the subscriber sends ROUTE_REQUEST and echoes every ROUTE_END (see [Routing](#routing)). @@ -425,10 +428,10 @@ The initial set is relative to this hop: a relay answers with what it holds with The publisher MAY coalesce changes to a claim it has not yet sent, and sends no more claims before SYNCED than the initial set holds. A retraction of a claim or route already sent MAY precede SYNCED. -Each ROUTE_START implicitly assigns the next Route ID on the stream, and each ANNOUNCE_START the next Announce ID: two counters starting at 0 that increment by 1 per assignment. -Neither id appears on the wire where it is assigned; both endpoints derive it from the message order on the (reliable, ordered) stream, and other messages reference it instead of repeating the origin node or the claim's prefix. +Each ROUTE_START implicitly assigns the next Route ID on the stream, each ANNOUNCE_START the next Announce ID, and each WARM_START the next Warm ID: three counters starting at 0 that increment by 1 per assignment. +No id appears on the wire where it is assigned; both endpoints derive it from the message order on the (reliable, ordered) stream, and other messages reference it instead of repeating the origin node, the claim's prefix, or the warm route's path. Ids are never reused within a stream. -A ROUTE_END retires its Route ID and the Announce ID of every claim under the route; an ANNOUNCE_END retires one Announce ID. +A ROUTE_END retires its Route ID, the Announce ID of every claim under the route, and the Warm ID of every warm route under it; an ANNOUNCE_END retires one Announce ID, and a WARM_END one Warm ID. A stream has at most one live route per origin node, and each route at most one live claim per prefix. Another origin claiming the same prefix is another claim, under its own route. @@ -519,7 +522,7 @@ A node drops a feasibility distance once it neither selects nor advertises the r A SUBSCRIBE, FETCH, TRACK, TRACKS, or DATAGRAMS request names the origin node that serves it. A subscriber that names 0 lets the publisher select one (see [Resolution](#resolution)), and a relay forwarding a request names the origin node it selected or was given. -A relay forwards a request naming an origin node only on its selected route to that node, and resets it with UNROUTABLE when it has no usable route to it, so the request never reaches another origin. +A relay forwards a request naming an origin node only on its selected route to that node, or a SUBSCRIBE through the neighbor [Warm Routes](#warm-routes) chooses for its broadcast, and resets it with UNROUTABLE when it has no usable route to it, so the request never reaches another origin. Claims with the same Instance serve the same Broadcast, so a relay MAY move a live subscription between them, including onto a higher Epoch, continuing from the first frame the subscriber lacks, and a route change (a reconnect, a cheaper path, a draining session) is invisible to the subscriber. A relay MUST NOT move a subscription to a claim with a different Instance: it stays on its Instance's claims and ends with the last of them. @@ -550,6 +553,41 @@ An advertiser that will not serve a resolved request resets the request stream w Every refusal is terminal and propagates without re-resolution, so probing unserved paths costs one round trip per path. An advertiser signals capacity through its claim alone, withdrawing or re-pricing it before it runs out. +#### Warm Routes {#warm-routes} +A relay MAY pull each broadcast through a neighbor of its choosing rather than along its selected route to the origin node, so a broadcast crosses fewer relays and neighboring relays share one copy. +A warm route tells a neighbor that the sender carries a broadcast and keeps it whether or not that neighbor pulls through it. +An endpoint that does not choose per broadcast ignores every WARM_START, WARM_UPDATE, and WARM_END, and sends a `Serves` of Unknown unless it serves only its own readers. + +Every SUBSCRIBE names whom it serves in `Serves`: + +- Alone: demand of the subscriber's own, either a local reader or pulls that trace back to two or more relays. +- Relay: demand that traces back to one relay alone, its root, named by its Node ID. A downstream pull that serves Alone traces back to the relay that sent it, and one that serves Relay to its root. +- Unknown: the subscriber does not say, or some of its demand came from a pull that serves Unknown and the rest traces back to fewer than two relays. + +A copy kept after its demand is gone serves whom it last served. + +A relay that pulls a broadcast for demand MAY send a warm route for it, under its route to the broadcast's origin node, on an Announce Stream it answers whose request covers the path. +It sends one only while its `Serves` is not Unknown and its selected route to the origin node is usable, and only to a neighbor that has no usable route to that origin node or whose last route to it has a Metric at least the relay's own. +A warm route travels one hop: a receiver never forwards it. +It carries the sender's `Serves` and its `Warm Cost`, the price it pays for its own pull, as defined below. +The sender sends a WARM_UPDATE when either changes, and a WARM_END once it no longer pulls the broadcast. +A change that narrows whom the copy serves, from Alone to a Relay, from one Relay to another, or to no warm route at all, waits until the sender has kept the copy without the old demand for as long as it keeps any copy whose demand is gone, so a neighbor that just joined it does not move straight back. + +A receiver counts a neighbor warm for a broadcast when the receiver's own pull does not serve Unknown, and the neighbor's warm route serves Alone or serves a Relay that is neither the receiver nor the root of the receiver's own pull. +Demand a receiver causes, through any number of relays, then never warms a neighbor toward it. +The receiver prices a pull through each neighbor as: + +- the link cost alone, through a neighbor it counts warm, whose copy is paid for already; +- the link cost plus the neighbor's `Warm Cost`, through a neighbor with a warm route it does not count warm, whose copy depends on the receiver's demand; +- otherwise its metric through that neighbor. + +A relay MUST pull a broadcast only through a neighbor eligible for it: one whose last route to the request's origin node is feasible (see [Routing](#routing)), or was advertised with exactly the Seqno and Metric of the relay's feasibility distance while the neighbor's Rendezvous Hash for the broadcast's path is lower than the relay's own. +The Rendezvous Hash is as defined in [Resolution](#resolution), with the neighbor's or the relay's own Node ID (see [Node ID Parameter](#node-id-parameter)). +A feasibility distance only falls within a Seqno, so the Seqno rises, or the Metric and then the hash fall, along every pull, and the pulls of a broadcast never form a loop, even over stale routes. + +Among the eligible neighbors, a relay SHOULD prefer a session that has not received a GOAWAY, then the lowest price, then a neighbor it counts warm, then the lowest Rendezvous Hash. +It SHOULD move an existing pull only to a neighbor strictly better by GOAWAY and then price, so two relays pricing each other's copies settle instead of swapping. + ### Subscribe A subscriber opens Subscribe Streams to request a Track. @@ -1133,7 +1171,7 @@ An Announce Stream ending before SYNCED is a PROTOCOL_VIOLATION; a Tracks Stream ## ROUTE_START {#route-start} A publisher sends a ROUTE_START message when it reaches an origin node (see [Routing](#routing)). -Each ROUTE_START implicitly assigns the next Route ID on the stream, later referenced by ROUTE_UPDATE, ROUTE_HOLD, ROUTE_END, ANNOUNCE_FLOOR, and ANNOUNCE_START (see [Announce](#announce)). +Each ROUTE_START implicitly assigns the next Route ID on the stream, later referenced by ROUTE_UPDATE, ROUTE_HOLD, ROUTE_END, ANNOUNCE_FLOOR, ANNOUNCE_START, and WARM_START (see [Announce](#announce)). ~~~ ROUTE_START Message { @@ -1421,6 +1459,92 @@ The ordinal implicitly assigned by a prior ROUTE_START on this stream. The origin's Revision at which the floor certifies the stream's claims for the route, plus one, or 0 when it certifies nothing. +## WARM_START {#warm-start} +A publisher sends a WARM_START message to tell the subscriber that it carries a broadcast and keeps it whether or not the subscriber pulls through it (see [Warm Routes](#warm-routes)). +Each WARM_START implicitly assigns the next Warm ID on the stream, later referenced by WARM_UPDATE and WARM_END (see [Announce](#announce)). + +~~~ +WARM_START Message { + Type (i) = 0xC + Message Length (i) + Route ID (i), + Broadcast Path Suffix (s), + Instance (b), + Minted (8), + Serves (..), + Warm Cost (i), +} +~~~ + +**Type**: +Set to 0xC to indicate a WARM_START message. + +**Route ID**: +The route to the broadcast's origin node: the ordinal implicitly assigned by a prior ROUTE_START on this stream. +A warm route never changes its Route ID. + +**Broadcast Path Suffix**: +Combined with the requested prefix to form the broadcast's path. + +**Instance** and **Minted**: +The Instance of the copy the publisher carries, as defined for [ANNOUNCE_START](#announce-start). + +**Serves**: +Whom the publisher's own pull of the broadcast serves, as defined for [SUBSCRIBE](#subscribe). +Unknown is a PROTOCOL_VIOLATION here. + +**Warm Cost**: +The publisher's price for its own pull of the broadcast (see [Warm Routes](#warm-routes)), in the units of the route's Metric. + +A stream has at most one live warm route per route and path. +A subscriber that uses warm routes MUST close the session with a PROTOCOL_VIOLATION on a WARM_START naming a Route ID that was never assigned or already retired, or a path its route already has a live warm route for. + + +## WARM_UPDATE {#warm-update} +A publisher sends a WARM_UPDATE message when a warm route's `Serves` or `Warm Cost` changes, referencing its Warm ID. +The id stays live. + +~~~ +WARM_UPDATE Message { + Type (i) = 0xD + Message Length (i) + Warm ID (i), + Serves (..), + Warm Cost (i), +} +~~~ + +**Type**: +Set to 0xD to indicate a WARM_UPDATE message. + +**Warm ID**: +The ordinal implicitly assigned by a prior WARM_START on this stream. + +**Serves** and **Warm Cost**: +As defined for [WARM_START](#warm-start), replacing the warm route's. + + +## WARM_END {#warm-end} +A publisher sends a WARM_END message when it withdraws a warm route, referencing its Warm ID. +The id is retired and MUST NOT be referenced again. +A ROUTE_END retires the warm routes under its route without a WARM_END. + +~~~ +WARM_END Message { + Type (i) = 0xE + Message Length (i) + Warm ID (i) +} +~~~ + +**Type**: +Set to 0xE to indicate a WARM_END message. + +**Warm ID**: +As defined for [WARM_UPDATE](#warm-update). +A subscriber that uses warm routes MUST close the session with a PROTOCOL_VIOLATION on a WARM_UPDATE or WARM_END naming a Warm ID that was never assigned or already retired. + + ## SUBSCRIBE SUBSCRIBE is sent by a subscriber to start a subscription. @@ -1440,6 +1564,7 @@ SUBSCRIBE Message { Range Count (i) Range (..) ... Update (i) + Serves (..) } Range { @@ -1454,6 +1579,11 @@ Anchor { [Groups Back (i)] [Timestamp (i)] } + +Serves { + Serves Type (i) + [Node ID (i)] +} ~~~ **Subscribe ID**: @@ -1512,6 +1642,15 @@ A Range resolves once: an update that leaves it live does not resolve it again. The number of the subscription's terms this message sets: 0 on SUBSCRIBE, and on each SUBSCRIBE_UPDATE a larger value than the one before it. A [SUBSCRIBE_DROP](#subscribe-drop) echoes the number in effect when its groups were given up, so the subscriber can tell a stale verdict on its current terms from one an update has since replaced. +**Serves Type**: +Whom the subscription serves (see [Warm Routes](#warm-routes)), and which field follows: + +- Unknown (0x0): no fields. +- Alone (0x1): no fields. +- Relay (0x2): the `Node ID` of the one relay its demand traces back to. 0 is a PROTOCOL_VIOLATION. + +Any other value is a PROTOCOL_VIOLATION. + A `Start` is a floor, not a request; `Subscriber Max Age` is the only thing that asks for data. The publisher SHOULD start at the oldest group at or above the floor that [Expiration](#expiration) has not expired, and MUST NOT deliver an older one. A `Subscriber Max Age` of 0 therefore starts at the latest group, since every older group is already stale. @@ -1547,6 +1686,7 @@ SUBSCRIBE_UPDATE Message { Range (..) ... Update (i) Wanted Group (i) + Serves (..) } ~~~ @@ -2241,6 +2381,7 @@ The `Message Length` describes the payload size on the wire. - The Cost parameter must be at least 1. - Replaced the shortest-path, Hop ID, Instance Hash, and Chain Hash tie-breaks with the local priority of the route's session, the Announce Cost plus the route's metric, and a Rendezvous Hash of the requested path and the origin's Node ID. - Removed the standby-ordering deployment requirements on Route Cost seeds and bounds. +- Added warm routes: WARM_START (0xC), WARM_UPDATE (0xD), and WARM_END (0xE) on the Announce Stream, and `Serves` on SUBSCRIBE and SUBSCRIBE_UPDATE. A relay MAY pull each broadcast through a neighbor that already carries it, priced by whom that neighbor's copy serves, and only through a neighbor whose route is feasible or tied at its feasibility distance with a lower Rendezvous Hash, so the pulls never loop. - Added the temporary Lite-06 Chain Setup Parameter (0x9) and a trailing Lite-06 Chain on ANNOUNCE_START, ANNOUNCE_UPDATE, and ANNOUNCE_RESTART: the moq-lite-06 Hop ID list of a claim learned over moq-lite-06, which a node drops when it names that node. ### Tracks {#changelog-07-tracks} diff --git a/js/net/src/lite/subscribe.test.ts b/js/net/src/lite/subscribe.test.ts index 57cb6e1a841..8cf45ace599 100644 --- a/js/net/src/lite/subscribe.test.ts +++ b/js/net/src/lite/subscribe.test.ts @@ -200,10 +200,11 @@ test("lite-07 carries the span as its only range, exclusive at the end", async ( test("a lite-07 list decodes as the span covering it", async () => { // Priority, order, max delay, and an unstated min delay as SubscribeUpdate writes them, - // then no Range ID to retire, the ranges it adds, the update number, and an absent wanted group. + // then no Range ID to retire, the ranges it adds, the update number, an absent wanted group, and + // a pull that serves Alone. const head = (await encodeMessage(Version.DRAFT_07, new SubscribeUpdate({ priority: 0, ranges: [] }))).slice(1, 5); const update = async (ranges: number[]) => { - const body = [...head, 0, ...ranges, 0, 0]; + const body = [...head, 0, ...ranges, 0, 0, 1]; const got = await SubscribeUpdate.decode( new Reader(undefined, new Uint8Array([body.length, ...body]), Version.DRAFT_07), Version.DRAFT_07, diff --git a/js/net/src/lite/subscribe.ts b/js/net/src/lite/subscribe.ts index acc41d97b5b..1986acbd04c 100644 --- a/js/net/src/lite/subscribe.ts +++ b/js/net/src/lite/subscribe.ts @@ -28,6 +28,7 @@ import { hasRanges, hasRefresh, hasSkip, + hasWarm, resolvesStart, Version, } from "./version.ts"; @@ -42,6 +43,27 @@ async function encodeUpdate(w: Writer, version: Version, update: bigint) { if (hasSkip(version)) await w.u62(update); } +// `Serves Type`: whom a pull serves. A relay names the one relay its demand traces back to. +const SERVES_UNKNOWN = 0n; +const SERVES_ALONE = 1n; +const SERVES_RELAY = 2n; + +/** Encode whom a pull serves: Alone, since this endpoint forwards nothing and every pull is its own. */ +async function encodeServes(w: Writer, version: Version) { + if (hasWarm(version)) await w.u62(SERVES_ALONE); +} + +/** Decode whom a pull serves, and ignore it: only a relay choosing a parent per broadcast prices by it. */ +async function decodeServes(r: Reader, version: Version) { + if (!hasWarm(version)) return; + const serves = await r.u62(); + if (serves === SERVES_RELAY) { + if ((await r.u62()) === 0n) throw new ProtocolViolation("serves names relay 0"); + } else if (serves !== SERVES_UNKNOWN && serves !== SERVES_ALONE) { + throw new ProtocolViolation(`unknown serves type ${serves}`); + } +} + /** * Encode the `Group Start` field shared by SUBSCRIBE and SUBSCRIBE_UPDATE. * @@ -731,6 +753,7 @@ export class SubscribeUpdate { } await encodeUpdate(w, version, this.update); if (hasRefresh(version)) await w.u62(this.wantedGroup === undefined ? 0n : this.wantedGroup + 1n); + await encodeServes(w, version); break; } } @@ -757,6 +780,7 @@ export class SubscribeUpdate { } const update = await decodeUpdate(r, version); const wanted = hasRefresh(version) ? await r.u62() : 0n; + await decodeServes(r, version); return new SubscribeUpdate({ wantedGroup: wanted === 0n ? undefined : wanted - 1n, priority, @@ -889,6 +913,7 @@ export class Subscribe { break; } await encodeUpdate(w, version, this.update); + await encodeServes(w, version); } static async #decode(r: Reader, version: Version): Promise { @@ -908,6 +933,7 @@ export class Subscribe { const minDelay = await decodeMinDelay(r, version); const span = await decodeSpan(r, version); const update = await decodeUpdate(r, version); + await decodeServes(r, version); return new Subscribe({ id, broadcast, diff --git a/js/net/src/lite/version.ts b/js/net/src/lite/version.ts index e19f9fd3a29..601b098d73f 100644 --- a/js/net/src/lite/version.ts +++ b/js/net/src/lite/version.ts @@ -488,6 +488,23 @@ export function hasAnnounceCompression(version: Version): boolean { } } +/** Whether the announce stream carries warm routes (WARM_START, WARM_UPDATE, and WARM_END), and + * SUBSCRIBE and SUBSCRIBE_UPDATE say whom each pull serves. Added in lite-07. */ +export function hasWarm(version: Version): boolean { + // Explicitly list older versions so future versions keep the lite-07+ behavior. + switch (version) { + case Version.DRAFT_01: + case Version.DRAFT_02: + case Version.DRAFT_03: + case Version.DRAFT_04: + case Version.DRAFT_05: + case Version.DRAFT_06: + return false; + default: + return true; + } +} + /** Whether TRACK_INFO carries the refresh ratio and SUBSCRIBE_UPDATE a wanted group. Added in lite-07. * Older peers see a track that never refreshes. */ export function hasRefresh(version: Version): boolean { diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index 5ea7b2a28ff..89fe86a9fb1 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -3,8 +3,8 @@ ## Goal A relay stays on a broadcast's path only while it has a local reader, feeds -two or more downstream sessions, or is its one consumer's cheapest cold -route. Otherwise its child bypasses it, so a broadcast with one viewer +two or more downstream sessions, or is its one consumer's cheapest route +given the copies the others hold. Otherwise its child bypasses it, so a broadcast with one viewer crosses fewer relays. A relay joins a neighbor that already carries a broadcast instead of pulling a second copy, so a LAN of relays with their own uplinks pulls one copy over one uplink. Routing @@ -72,7 +72,8 @@ Decided 2026-11-08 (maintainer, /quest-plan): message's draft section and codec there if it places it in lite-07 (rejected: a design-only sizing quest in m2; leaving the proof in m3 and accepting a SETUP extension). The implementation stays in m3. This - replaces item 6's rule that choosing lite-07 went back to the maintainer. + replaces item 6's rule that choosing lite-07 went back to the maintainer; + item 11 records the placement the maintainer chose. Proof results 2026-11-10 ([moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)), modeled in `rs/moq-net/src/route/tests/cache.rs` over the real `Router`: @@ -87,46 +88,61 @@ modeled in `rs/moq-net/src/route/tests/cache.rs` over the real `Router`: [moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447) found breaks that, which [the implementation](/quest/m3/cache-routing/select.md) must not ship before it is fixed. -9. **Proposed amendment to item 2: each pull names whom it serves.** "Demand - besides N's" can't be told from one hop: if B's only demand is a relay - pulling on N's behalf, B counts warm toward N, N joins B, B's demand - vanishes, and the cycle repeats every linger. A SUBSCRIBE carries - `Serves`: alone (a local reader, or two or more roots below it), a single - root relay's Node ID, or unknown (a peer that never negotiated, which - warms nothing). N counts B warm only when B serves alone or a root other - than N and N's own root. A narrowed warm route stands until the copy's - linger runs out, as the withdrawal already does. -10. **Proposed amendment to item 3: price a dependent copy at its cost.** - Link-only pricing has no steady state when two relays each keep the - other warm (sweep scenario 2062 flips forever). A warm route carries its - sender's own price for its pull; a receiver the copy depends on (its - serves names only the receiver's root) pays link plus that cost, an - independent one pays the link alone, and a relay moves only for a - strictly lower `(draining, price)`. With exact information that is a - potential game on the forest's total link cost. Under the sweep's churn, - 2996 of 3000 scenarios settle; 4 (the first is 478) never do, as two - carriers keep a narrowed route standing while two relays swap between - them. `sweep_cache_routing_long` reproduces them behind `#[ignore]`. - The Goal's "cheapest cold route" becomes "cheapest given the copies the - others hold" under this rule. -11. **Item 6 placement: an additive SETUP extension, Warm Routes**, decided - by the proof. Only m3 relays use it, so lite-07 would carry Rust and JS - codecs that js/net never runs, and it would freeze a shape items 9 and - 10 still revise; the bytes are the same either way, and it stays clear of - [Route latency](/quest/m2/route-latency.md)'s ROUTE change. The - parameter's value is empty. `WARM_START` (a new announce-stream type, - not 0x5, which ANNOUNCE_TAKEOVER held) carries the Route ID it travels - under, the path suffix, the Instance and Minted encoded as in - ANNOUNCE_START, then the serves and cost, and assigns a Warm ID; `WARM_END` carries the - Warm ID. It follows the ROUTE_START it names on the announce stream the +Decided 2026-11-10 (maintainer, /quest-iterate on +[moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)), from the +proof's proposals: + +9. ✅ **Each pull names whom it serves**, amending item 2: a trailing + `Serves` field on SUBSCRIBE and SUBSCRIBE_UPDATE. (rejected: no field) + "Demand besides N's" can't be told from one hop: if B's only demand is a + relay pulling on N's behalf, B counts warm toward N, N joins B, B's demand + vanishes, and the cycle repeats every linger. `Serves` is alone (a local + reader, or two or more roots below it), a single root relay's Node ID, or + unknown (an endpoint that doesn't say, which warms nothing). N counts B + warm only when B serves alone or a root other than N and N's own root. A + narrowed warm route stands until the copy's linger runs out, as the + withdrawal already does. +10. ✅ **Price a dependent copy at its cost, and move only on strict + improvement**, amending item 3; the Goal's "cheapest cold route" became + "cheapest route given the copies the others hold". (rejected: dropping + mutual warmth so only a relay with a local reader advertises, which + converges trivially but loses the LAN sharing that feeding two sessions + gives; link-only pricing) Link-only pricing has no steady state when two + relays each keep the other warm (sweep scenario 2062 flips forever). A + warm route carries its sender's own price for its pull; a receiver the + copy depends on (its serves names only the receiver's root) pays link + plus that cost, an independent one pays the link alone, and a relay + moves only for a strictly lower `(draining, price)`. With exact + information that is a potential game on the forest's total link cost. + Under the sweep's churn, 2996 of 3000 scenarios settle; 4 (the first is + 478) never do, as two carriers keep a narrowed route standing while two + relays swap between them. `sweep_cache_routing_long` reproduces them + behind `#[ignore]`, and [the implementation](/quest/m3/cache-routing/select.md) + settles them first. +11. ✅ **The wire is in moq-lite-07 itself**, overturning the proof's choice + of a SETUP extension. (rejected: an additive Warm Routes SETUP + extension) Lite-07 is unpublished, so its shape can still change until + the cut, and a later change goes to the next version. The draft's + Warm Routes section and the Rust codec landed with the proof: + `WARM_START` (0xC) carries the Route ID it travels under, the path + suffix, the Instance and Minted encoded as in ANNOUNCE_START, then the + serves and cost, and assigns a Warm ID; `WARM_UPDATE` (0xD) replaces a + warm route's serves and cost by Warm ID, so a price change costs one + message rather than an end and a start; `WARM_END` (0xE) withdraws it. + A warm route follows the ROUTE_START it names on the announce stream the carrier answers, and a ROUTE_END retires the warm routes under it. - SUBSCRIBE and SUBSCRIBE_UPDATE gain a trailing Serves field when the - extension is negotiated. The hash is the Resolution section's FNV-1a - rendezvous hash of the path and the neighbor's Node ID. A peer that - never negotiates sees only cold routes, and its pulls serve unknown. - About 4.5 warm messages per reader change across the sweep's churn, - which also flaps links and advances seqnos; an idle broadcast sends - none. + SUBSCRIBE and SUBSCRIBE_UPDATE end with `Serves`: a type of unknown + (0x0), alone (0x1), or relay (0x2) followed by its Node ID. The hash is + the Resolution section's FNV-1a rendezvous hash of the path and the + neighbor's Node ID. Until the implementation, Rust reads warm routes and + serves and ignores them, and says unknown; `js/net` says alone, since it + forwards nothing, and skips the warm types as unknown, which is how an + endpoint that does not choose per broadcast ignores them. About 4.5 warm + messages per reader change across the sweep's churn, which also flaps + links and advances seqnos; an idle broadcast sends none. + +Measured by the proof: + 12. **Costs the sweeps measured.** A full LAN mesh of 6 relays pulls one uplink copy per broadcast; a ring of 6 keeps 13 copies for 8 broadcasts, one per hash local minimum. An origin seqno advance that reaches a child diff --git a/quest/m3/cache-routing/select.md b/quest/m3/cache-routing/select.md index 1f141e27b86..a1c6d09b53a 100644 --- a/quest/m3/cache-routing/select.md +++ b/quest/m3/cache-routing/select.md @@ -13,9 +13,10 @@ and how many moves warm routes caused. ## Plan - Implement the rules the epic's Plan records as proven, which - `rs/moq-net/src/route/tests/cache.rs` models over the real router: the - Warm Routes SETUP extension (warm routes on the announce stream, and the - relay each pull serves on SUBSCRIBE), warm state per neighbor derived from + `rs/moq-net/src/route/tests/cache.rs` models over the real router: lite-07's + warm routes (WARM_START, WARM_UPDATE, and WARM_END on the announce stream, + and the `Serves` each SUBSCRIBE and SUBSCRIBE_UPDATE carries, whose codec + `rs/moq-net/src/lite/` already reads and ignores), warm state per neighbor derived from each front's demand, and eligibility through `Router::eligible` with the rendezvous hash ordering tied routes. Port that file's fixtures and sweep to the implementation, or drive it from them. @@ -32,8 +33,10 @@ and how many moves warm routes caused. only to a copy of its own instance. - Bound the warm state a peer can make a relay hold by design, as announcements are bounded. -- Only relay sessions that negotiated warm routes exchange them. A browser - never parents and never negotiates them, so `js/net` needs nothing. +- Each pull states whom it serves, where Rust says unknown today: alone for + a local reader or a client session, which forwards nothing. `js/net` + already says alone and skips the warm types as unknown, so it needs + nothing. - Export upstream copies held, warm routes sent and received, and moves by cause through the relay's existing metrics. - `rs/moq-net/AGENTS.md` says a relay behaves the same warm or cold. That @@ -41,9 +44,10 @@ and how many moves warm routes caused. request goes to. If the rule's wording contradicts warm routes, bring a rewording to the maintainer. -Public API: report any relay configuration added for warm routes. Wire: the -Warm Routes extension as the epic's Plan designs it, with its draft section in -the same PR. +Public API: report any relay configuration added for warm routes. Wire: +lite-07 already carries warm routes as the epic's item 11 records; a change +settling the long sweep needs updates the draft's Warm Routes section in the +same PR, in lite-07 while it is unpublished and in the next version after. ## Related diff --git a/rs/moq-net/src/lite/announce.rs b/rs/moq-net/src/lite/announce.rs index d23df0e92e9..43838cdd20f 100644 --- a/rs/moq-net/src/lite/announce.rs +++ b/rs/moq-net/src/lite/announce.rs @@ -8,7 +8,7 @@ use crate::{ }; use super::{ - Message, Version, + Message, Serves, Version, message::{MAX_MESSAGE_SIZE, decode_size}, }; @@ -28,6 +28,10 @@ const ROUTE_END: u64 = 8; const ANNOUNCE_FLOOR: u64 = 9; const ROUTE_REQUEST: u64 = 10; const ROUTE_HOLD: u64 = 11; +// lite-07: warm routes, one hop toward the subscriber. +const WARM_START: u64 = 12; +const WARM_UPDATE: u64 = 13; +const WARM_END: u64 = 14; /// Whether the negotiated version can update a live advertisement's metadata. On lite-05 an /// update travels as a duplicate ANNOUNCE (a second `active` for an already-announced path); on @@ -112,6 +116,35 @@ pub enum AnnounceBroadcast { /// with this id, certifying them across the whole requested prefix at the origin's /// revision, or certifying nothing (`None`). Floor { id: u64, revision: Option }, + /// WARM_START (lite-07+): under the route with Route ID `route`, the sender keeps its + /// copy of the broadcast at `suffix` whether or not the receiver pulls through it, and + /// assigns it the next Warm ID. `serves` is never [`Serves::Unknown`]. + WarmStart { + route: u64, + suffix: Path, + instance: Instance, + serves: Serves, + cost: u64, + }, + /// WARM_UPDATE (lite-07+): the warm route with this id serves `serves` at `cost` now. + WarmUpdate { id: u64, serves: Serves, cost: u64 }, + /// WARM_END (lite-07+): the sender withdrew the warm route with this id. + WarmEnd { id: u64 }, +} + +/// Encode whom a warm route's copy serves, which it always names. +fn encode_warm_serves(w: &mut Encoder<'_>, serves: Serves) -> Result<(), EncodeError> { + match serves { + Serves::Unknown => Err(EncodeError::InvalidState), + serves => serves.encode(w), + } +} + +fn decode_warm_serves(r: &mut Decoder<'_>) -> Result { + match Serves::decode(r)? { + Serves::Unknown => Err(DecodeError::InvalidValue), + serves => Ok(serves), + } } /// Encode a seqno, metric, and down-only bit. No origin advertises the last seqno, @@ -307,11 +340,17 @@ impl Encode for AnnounceBroadcast { Self::RouteHold { .. } if version.has_routes() => ROUTE_HOLD, Self::RouteEnd { .. } if version.has_routes() => ROUTE_END, Self::Floor { .. } if version.has_routes() => ANNOUNCE_FLOOR, + Self::WarmStart { .. } if version.has_warm() => WARM_START, + Self::WarmUpdate { .. } if version.has_warm() => WARM_UPDATE, + Self::WarmEnd { .. } if version.has_warm() => WARM_END, Self::RouteStart { .. } | Self::RouteUpdate { .. } | Self::RouteHold { .. } | Self::RouteEnd { .. } - | Self::Floor { .. } => return Err(EncodeError::Version), + | Self::Floor { .. } + | Self::WarmStart { .. } + | Self::WarmUpdate { .. } + | Self::WarmEnd { .. } => return Err(EncodeError::Version), // The pre-lite-06 path-form retraction has no place on lite-06. Self::Ended { .. } => return Err(EncodeError::Version), // Decode-only: an unknown type is never sent. @@ -369,6 +408,25 @@ impl Encode for AnnounceBroadcast { } Self::RouteHold { id } | Self::RouteEnd { id } => w.varint(*id)?, Self::Floor { id, revision } => encode_floor(w, *id, *revision)?, + Self::WarmStart { + route, + suffix, + instance, + serves, + cost, + } => { + w.varint(*route)?; + suffix.encode(w, version)?; + super::instance::encode_copy(w, instance)?; + encode_warm_serves(w, *serves)?; + w.varint(*cost)?; + } + Self::WarmUpdate { id, serves, cost } => { + w.varint(*id)?; + encode_warm_serves(w, *serves)?; + w.varint(*cost)?; + } + Self::WarmEnd { id } => w.varint(*id)?, Self::Ended { .. } | Self::Skipped => unreachable!("refused above"), } if w.since(&prefix) > MAX_MESSAGE_SIZE { @@ -412,7 +470,10 @@ impl Encode for AnnounceBroadcast { | Self::RouteUpdate { .. } | Self::RouteHold { .. } | Self::RouteEnd { .. } - | Self::Floor { .. } => { + | Self::Floor { .. } + | Self::WarmStart { .. } + | Self::WarmUpdate { .. } + | Self::WarmEnd { .. } => { return Err(EncodeError::Version); } } @@ -493,6 +554,19 @@ impl<'a> Decode<'a, Version> for AnnounceBroadcast { ROUTE_HOLD if version.has_routes() => Self::RouteHold { id: body.varint()? }, ROUTE_END if version.has_routes() => Self::RouteEnd { id: body.varint()? }, ANNOUNCE_FLOOR if version.has_routes() => decode_floor(&mut body)?, + WARM_START if version.has_warm() => Self::WarmStart { + route: body.varint()?, + suffix: Path::decode(&mut body, version)?, + instance: super::instance::decode_copy(&mut body)?, + serves: decode_warm_serves(&mut body)?, + cost: body.varint()?, + }, + WARM_UPDATE if version.has_warm() => Self::WarmUpdate { + id: body.varint()?, + serves: decode_warm_serves(&mut body)?, + cost: body.varint()?, + }, + WARM_END if version.has_warm() => Self::WarmEnd { id: body.varint()? }, // Unknown types are skipped by length so an earlier Lite06 build // negotiating the same ALPN does not kill the announce stream. _ => { @@ -1143,6 +1217,59 @@ mod tests { ); } + /// The draft's warm route layouts, byte for byte: WARM_START is the Route ID, the path + /// suffix, the instance and its minted flag, whom the copy serves, and the cost; + /// WARM_UPDATE the Warm ID, serves, and cost; WARM_END the Warm ID. A warm route always + /// names whom it serves, and lite-06 skips the types as unknown. + #[test] + fn lite07_warm_route_layout() { + let encode = |msg: &AnnounceBroadcast, version: Version| { + let mut buf = Vec::new(); + msg.encode(&mut Encoder::new(&mut buf, version.into()), version) + .map(|()| buf) + }; + let start = AnnounceBroadcast::WarmStart { + route: 3, + suffix: Path::from_static("c"), + instance: "a".parse().unwrap(), + serves: Serves::Relay(9), + cost: 4, + }; + let update = AnnounceBroadcast::WarmUpdate { + id: 7, + serves: Serves::Alone, + cost: 4, + }; + let end = AnnounceBroadcast::WarmEnd { id: 7 }; + let layouts: [(&AnnounceBroadcast, &[u8]); 3] = [ + (&start, &[12, 9, 3, 1, b'c', 1, b'a', 0, 2, 9, 4]), + (&update, &[13, 3, 7, 1, 4]), + (&end, &[14, 1, 7]), + ]; + for (msg, wire) in layouts { + assert_eq!(encode(msg, Version::Lite07).unwrap(), wire); + assert_eq!(&broadcast_round_trip(msg, Version::Lite07), msg); + assert!(matches!(encode(msg, Version::Lite06), Err(EncodeError::Version))); + let got = + crate::coding::decode_buf(&mut &wire[..], Version::Lite06, |r, v| AnnounceBroadcast::decode(r, v)); + assert_eq!(got.unwrap(), AnnounceBroadcast::Skipped); + } + + let unknown = AnnounceBroadcast::WarmUpdate { + id: 7, + serves: Serves::Unknown, + cost: 4, + }; + assert!(matches!( + encode(&unknown, Version::Lite07), + Err(EncodeError::InvalidState) + )); + let decode = |wire: &[u8]| { + crate::coding::decode_buf(&mut &wire[..], Version::Lite07, |r, v| AnnounceBroadcast::decode(r, v)) + }; + assert!(decode(&[13, 3, 7, 0, 4]).is_err()); + } + /// Lite-07 carries a lite-06 chain after the cost, only when it is not empty, so an /// explicitly empty one is refused. #[test] diff --git a/rs/moq-net/src/lite/instance.rs b/rs/moq-net/src/lite/instance.rs index 584fab5a763..60687d09000 100644 --- a/rs/moq-net/src/lite/instance.rs +++ b/rs/moq-net/src/lite/instance.rs @@ -80,6 +80,16 @@ pub(super) fn decode_route(r: &mut Decoder<'_>, version: Version) -> Result<(Opt Ok((Some(instance), r.varint()?)) } +/// Encode the instance of a copy a warm route names, which is never none. +pub(super) fn encode_copy(w: &mut Encoder<'_>, instance: &Instance) -> Result<(), EncodeError> { + encode(w, instance) +} + +/// Decode the instance of a copy a warm route names; see [`encode_copy`]. +pub(super) fn decode_copy(r: &mut Decoder<'_>) -> Result { + decode(r.bytes()?, r.bool()?) +} + fn encode(w: &mut Encoder<'_>, instance: &Instance) -> Result<(), EncodeError> { w.bytes(instance.as_str().as_bytes())?; w.bool(instance.is_minted()); diff --git a/rs/moq-net/src/lite/subscribe.rs b/rs/moq-net/src/lite/subscribe.rs index 5c4c5f5857d..3b1d83307ff 100644 --- a/rs/moq-net/src/lite/subscribe.rs +++ b/rs/moq-net/src/lite/subscribe.rs @@ -120,6 +120,62 @@ impl Range { } } +/// Whom a pull serves, which a relay choosing a parent per broadcast prices warm routes by. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Serves { + /// The subscriber does not say. + Unknown, + /// Demand of the subscriber's own: a local reader, or pulls tracing back to two or more + /// relays. + Alone, + /// Demand tracing back to the relay with this Node ID alone. + Relay(u64), +} + +impl Serves { + pub(super) fn encode(self, w: &mut Encoder<'_>) -> Result<(), EncodeError> { + match self { + Self::Unknown => w.varint(0), + Self::Alone => w.varint(1), + Self::Relay(0) => Err(EncodeError::InvalidState), + Self::Relay(node) => { + w.varint(2)?; + w.varint(node) + } + } + } + + pub(super) fn decode(r: &mut Decoder<'_>) -> Result { + Ok(match r.varint()? { + 0 => Self::Unknown, + 1 => Self::Alone, + 2 => match r.varint()? { + 0 => return Err(DecodeError::InvalidValue), + node => Self::Relay(node), + }, + _ => return Err(DecodeError::InvalidValue), + }) + } +} + +/// Encode whom a pull serves. This endpoint chooses no parent per broadcast, so it does not +/// say. +fn encode_serves(w: &mut Encoder<'_>, version: Version) -> Result<(), EncodeError> { + match version.has_warm() { + true => Serves::Unknown.encode(w), + false => Ok(()), + } +} + +/// Decode whom a pull serves, and ignore it: only a relay choosing a parent per broadcast +/// prices by it, which this endpoint does not do. +fn decode_serves(r: &mut Decoder<'_>, version: Version) -> Result<(), DecodeError> { + if version.has_warm() { + Serves::decode(r)?; + } + Ok(()) +} + impl Version { /// Whether this version's SUBSCRIBE carries the subscriber's max delay preference. /// @@ -153,6 +209,7 @@ impl<'a> Message<'a> for Subscribe<'a> { } }; let update = decode_update(r, version)?; + decode_serves(r, version)?; Ok(Self { id, @@ -188,6 +245,7 @@ impl<'a> Message<'a> for Subscribe<'a> { } } encode_update(w, version, self.update)?; + encode_serves(w, version)?; Ok(()) } @@ -762,6 +820,7 @@ impl<'a> Message<'a> for SubscribeUpdate { true => r.varint()?.checked_sub(1), false => None, }; + decode_serves(r, version)?; Ok(Self { wanted_group, @@ -802,7 +861,7 @@ impl<'a> Message<'a> for SubscribeUpdate { if version.has_refresh() { w.varint_opt(self.wanted_group)?; } - Ok(()) + encode_serves(w, version) } } @@ -1886,9 +1945,10 @@ mod test { msg.ranges = vec![Range::OLDEST]; let mut buf = encode(&msg, version).unwrap(); - // The list is last but for the update number: a count of one, then two anchors of type 0. - let at = buf.len() - 4; - assert_eq!(&buf[at..], [1, 0, 0, 0]); + // The list is last but for the update number and whom the pull serves: a count of one, + // then two anchors of type 0. + let at = buf.len() - 5; + assert_eq!(&buf[at..], [1, 0, 0, 0, 0]); buf[at] = 0; buf.truncate(at + 1); assert!(matches!( @@ -1897,7 +1957,7 @@ mod test { )); let mut buf = encode(&msg, version).unwrap(); - let last = buf.len() - 2; + let last = buf.len() - 3; buf[last] = 4; assert!(matches!( crate::coding::decode_from(&mut buf.as_slice(), version, Subscribe::decode_msg), @@ -1951,6 +2011,27 @@ mod test { ); } + /// Lite-07 ends SUBSCRIBE with whom the pull serves: this endpoint says Unknown, and reads + /// and ignores any well-formed value. An unknown type or a relay named 0 is malformed, and + /// older versions carry nothing. + #[test] + fn serves_trails_subscribe_only_on_lite07() { + let msg = subscribe_sample(); + let lite07 = encode(&msg, Version::Lite07).unwrap(); + assert_eq!(lite07.last(), Some(&0)); + let head = &lite07[..lite07.len() - 1]; + let decode = |serves: &[u8]| { + let buf = [head, serves].concat(); + crate::coding::decode_from(&mut buf.as_slice(), Version::Lite07, Subscribe::decode_msg).map(|_| ()) + }; + for serves in [&[0][..], &[1], &[2, 9]] { + decode(serves).unwrap(); + } + for serves in [&[3][..], &[2, 0], &[]] { + assert!(decode(serves).is_err(), "{serves:?}"); + } + } + /// Lite-07 carries `Wanted Group` on SUBSCRIBE_UPDATE; older versions drop it. #[test] fn wanted_group_roundtrips_only_on_lite07() { diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index ab10fbfd8a2..8c15ba8b425 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -655,7 +655,10 @@ impl Subscriber { | lite::AnnounceBroadcast::RouteUpdate { .. } | lite::AnnounceBroadcast::RouteHold { .. } | lite::AnnounceBroadcast::RouteEnd { .. } - | lite::AnnounceBroadcast::Floor { .. } => return Err(Error::ProtocolViolation), + | lite::AnnounceBroadcast::Floor { .. } + | lite::AnnounceBroadcast::WarmStart { .. } + | lite::AnnounceBroadcast::WarmUpdate { .. } + | lite::AnnounceBroadcast::WarmEnd { .. } => return Err(Error::ProtocolViolation), } Ok(()) } diff --git a/rs/moq-net/src/lite/subscriber/routed.rs b/rs/moq-net/src/lite/subscriber/routed.rs index 41a55264a6a..579ad641541 100644 --- a/rs/moq-net/src/lite/subscriber/routed.rs +++ b/rs/moq-net/src/lite/subscriber/routed.rs @@ -252,6 +252,9 @@ impl Routed { } // Lite-07 ends a claim by its Announce ID only. Msg::Ended { .. } => return Err(Error::ProtocolViolation), + // Only a relay choosing a parent per broadcast uses warm routes, which this + // endpoint does not do. + Msg::WarmStart { .. } | Msg::WarmUpdate { .. } | Msg::WarmEnd { .. } => {} Msg::Skipped | Msg::Synced => {} } Ok(()) diff --git a/rs/moq-net/src/lite/version.rs b/rs/moq-net/src/lite/version.rs index 471f194f7a5..accb1bb2c45 100644 --- a/rs/moq-net/src/lite/version.rs +++ b/rs/moq-net/src/lite/version.rs @@ -377,6 +377,16 @@ impl Version { ) } + /// Whether the announce stream carries warm routes (WARM_START, WARM_UPDATE, and + /// WARM_END), and SUBSCRIBE and SUBSCRIBE_UPDATE say whom each pull serves. Added in + /// lite-07. + pub(crate) fn has_warm(self) -> bool { + !matches!( + self, + Self::Lite01 | Self::Lite02 | Self::Lite03 | Self::Lite04 | Self::Lite05 | Self::Lite06 + ) + } + /// Whether TRACK_INFO carries the refresh ratio and SUBSCRIBE_UPDATE a wanted group, /// so a join at any hop reaches the publisher. Added in lite-07; older peers see a /// track that never refreshes. From 42922ba2cf199d238c6ec597f3481714b7d5610d Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Tue, 10 Nov 2026 16:42:47 -0800 Subject: [PATCH 05/12] quest: plan the origin seqno churn that warm children pay Co-Authored-By: Claude Opus 5.5 --- quest/m3/cache-routing/README.md | 2 +- quest/m3/cache-routing/seqno-churn.md | 38 +++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 1 deletion(-) create mode 100644 quest/m3/cache-routing/seqno-churn.md diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index 89fe86a9fb1..354b767daae 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -149,7 +149,7 @@ Measured by the proof: before its warm parent makes the parent's older route ineligible, so the child moves off and back (2 moves in the chain fixture); origins advance on every announce stream open and close, so fewer advances cut this - churn. + churn, which [seqno churn](/quest/m3/cache-routing/seqno-churn.md) takes up. Prior art: warm/cold route prices ([moq-dev/moq#2925](https://github.com/moq-dev/moq/pull/2925)) were deleted diff --git a/quest/m3/cache-routing/seqno-churn.md b/quest/m3/cache-routing/seqno-churn.md new file mode 100644 index 00000000000..b98711ced09 --- /dev/null +++ b/quest/m3/cache-routing/seqno-churn.md @@ -0,0 +1,38 @@ +# [M] Origins advance their seqno only when a neighbor needs it + +## Goal + +An origin's seqno advances only when some node needs a newer route to it, +so a warm child stops moving off its warm parent and back each time the +origin opens or closes an announce stream. Recovery after an origin +reconnects or restarts under the same Node ID stays as fast as it is today. + +## Plan + +Found by the cache-routing proof +([moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457), the epic's +item 12): an origin advances its seqno on every announce stream open and +close, and an advance that reaches a child before its warm parent makes the +parent's older route ineligible, so the child moves off and back (2 moves in +the chain fixture of `rs/moq-net/src/route/tests/cache.rs`). + +The advance on every session change exists so feasibility distances held +elsewhere accept a restart reusing an identity and an origin that moved to a +farther attachment (`restart_reusing_identity_is_refused`, +`reconnect_without_restart_needs_a_seqno_advance`, and +[Routes and announces](/quest/m2/cluster-routing/routes/README.md)). +Recommendation: advance only on a ROUTE_REQUEST, as Babel does, plus once +per restart, and measure what the extra round trip costs those two tests' +recoveries. Update the draft's Routing section, which says an origin +advances whenever its own set of sessions changes, in the same PR. + +## Waiting + +- Maintainer: whether one ROUTE_REQUEST round trip on a reconnect to a + farther attachment is an acceptable price for fewer warm moves. An agent + can measure both but not weigh them. Recommendation: yes, since the + request path already exists and an idle origin then sends no advances. + +## Related + +- [Warm routes and per-broadcast parent selection](/quest/m3/cache-routing/select.md) - the warm children whose moves this cuts From f4c5a9180f94fd4b38a1840072de2153818c07c8 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Tue, 10 Nov 2026 17:21:32 -0800 Subject: [PATCH 06/12] test(interop): quarantine the go -> cpp cell and plan its fix One loaded `just test interop --all` failed it and a rerun passed: the cpp subscriber closed its session for publishing the broadcast it consumed, outside its subscribe-only grant. Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/README.md | 1 + quest/m0/test-flakes-2/interop-go-cpp.md | 32 ++++++++++++++++++++++++ test/interop/interop.sh | 6 +++++ 3 files changed, 39 insertions(+) create mode 100644 quest/m0/test-flakes-2/interop-go-cpp.md diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index a77e44674b1..f7dba37c8c6 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -272,3 +272,4 @@ Public API: none. Wire: none. - [js/common's worklet dev-server test holds up under load](/quest/m0/test-flakes-2/worklet-dev-server.md) - the dev-server test passes on a loaded host, and its `test.skip` is removed - [Interop's finish cases hold up under load](/quest/m0/test-flakes-2/interop-finish.md) - every publisher's finish cells pass under load, and the scenario's quarantine is removed +- [Interop's go -> cpp cell holds up under load](/quest/m0/test-flakes-2/interop-go-cpp.md) - the cell passes on a loaded host, and its quarantine is removed diff --git a/quest/m0/test-flakes-2/interop-go-cpp.md b/quest/m0/test-flakes-2/interop-go-cpp.md new file mode 100644 index 00000000000..83bf7ded45d --- /dev/null +++ b/quest/m0/test-flakes-2/interop-go-cpp.md @@ -0,0 +1,32 @@ +# [S] Interop's go -> cpp cell holds up under load + +## Goal + +`just test interop --all` passes the `go -> cpp` media cell on a loaded +host, and its quarantine in `test/interop/interop.sh`'s `run_round` is +removed. + +## Plan + +Planned 2026-11-10 under the repository's flaky-test rule, while iterating +[moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457). One local +`just test interop --all` failed `go -> cpp` after 18 of 30 seconds and +passed on rerun; every other `go ->` cell and `cpp -> go` passed. The matrix +negotiates its default version there, not `moq-lite-07-wip`, so that PR's +lite-07 wire changes are not in its path. + +The cpp subscriber's log, after its grant of `subscribe=["**/"]` +and `publish=[]`, ends with `moq_net::lite::publisher` logging +`publishing outside our grant; closing the session` for the broadcast it +was subscribing to, then `container consumer: unroutable`. A subscribe-only +client found the broadcast it consumes among its own publications: suspect +the cpp client (or `moq-ffi` beneath it) sharing one origin between what it +publishes and what it consumes, so a consumed broadcast is sometimes +announced back to the relay. Reproduce with +`just test interop --all --publishers go --subscribers cpp` in a loop under +load, find why the consumed broadcast reaches the publish side, and fix it +there. + +## Related + +- [Interop's finish cases hold up under load](/quest/m0/test-flakes-2/interop-finish.md) - another interop quarantine seen under the same load diff --git a/test/interop/interop.sh b/test/interop/interop.sh index 5f56bbe24e3..73a70611c4f 100755 --- a/test/interop/interop.sh +++ b/test/interop/interop.sh @@ -1610,6 +1610,12 @@ run_round() { overall=1 continue fi + # Quarantined: quest/m0/test-flakes-2/interop-go-cpp.md. + if [[ "$pub" == go && "$sub" == cpp ]]; then + echo " SKIP $pub -> $sub (quarantined)" + record skip case="$case" publisher="$publisher" subscriber="$sub" detail="quarantined: quest/m0/test-flakes-2/interop-go-cpp.md" + continue + fi harness_spawn "$pub-$sub" "$LOGS/$pub-$sub.log" run_cell "$pub" "$sub" "$broadcast" "$sub_url" "$tone" "$codecs" pids+=("$HARNESS_PID") names+=("$sub") From 196c218007dfa77a9477de8d4fcb023619610d23 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Tue, 10 Nov 2026 17:57:40 -0800 Subject: [PATCH 07/12] test(doc): quarantine the doc API's failing-render test and plan its fix It renders TypeDoc inside bun's 5 second test timeout, the only case that does, and timed out once under a loaded `just check`. Co-Authored-By: Claude Opus 5.5 --- doc/.vitepress/api/index.test.ts | 3 ++- quest/m0/test-flakes-2/README.md | 1 + .../test-flakes-2/doc-api-typedoc-timeout.md | 23 +++++++++++++++++++ 3 files changed, 26 insertions(+), 1 deletion(-) create mode 100644 quest/m0/test-flakes-2/doc-api-typedoc-timeout.md diff --git a/doc/.vitepress/api/index.test.ts b/doc/.vitepress/api/index.test.ts index aa009c9a3fe..00a59fb96b2 100644 --- a/doc/.vitepress/api/index.test.ts +++ b/doc/.vitepress/api/index.test.ts @@ -151,7 +151,8 @@ export declare class Thing {} }); }); -test("an unexported declaration that a public one names fails the run", async () => { +// Quarantined: quest/m0/test-flakes-2/doc-api-typedoc-timeout.md. +test.skip("an unexported declaration that a public one names fails the run", async () => { const files = { "src/index.ts": `export { read } from "./access"; /** A public value. */ diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index f7dba37c8c6..3091067347f 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -273,3 +273,4 @@ Public API: none. Wire: none. - [js/common's worklet dev-server test holds up under load](/quest/m0/test-flakes-2/worklet-dev-server.md) - the dev-server test passes on a loaded host, and its `test.skip` is removed - [Interop's finish cases hold up under load](/quest/m0/test-flakes-2/interop-finish.md) - every publisher's finish cells pass under load, and the scenario's quarantine is removed - [Interop's go -> cpp cell holds up under load](/quest/m0/test-flakes-2/interop-go-cpp.md) - the cell passes on a loaded host, and its quarantine is removed +- [The doc API's failing-render test holds up under load](/quest/m0/test-flakes-2/doc-api-typedoc-timeout.md) - the test passes on a loaded host, and its `test.skip` is removed diff --git a/quest/m0/test-flakes-2/doc-api-typedoc-timeout.md b/quest/m0/test-flakes-2/doc-api-typedoc-timeout.md new file mode 100644 index 00000000000..d497e64815a --- /dev/null +++ b/quest/m0/test-flakes-2/doc-api-typedoc-timeout.md @@ -0,0 +1,23 @@ +# [S] The doc API's failing-render test holds up under load + +## Goal + +`doc/.vitepress/api/index.test.ts`'s "an unexported declaration that a +public one names fails the run" passes on a loaded host, and its +`test.skip` is removed. + +## Plan + +Planned 2026-11-10 under the repository's flaky-test rule, while iterating +[moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457): one loaded +`just check` failed it as `this test timed out after 5000ms`, and the run +before passed. [moq-dev/moq#8426](https://github.com/moq-dev/moq/pull/8426) +saw the same timeout. It is the only case that renders inside its test body: +the other cases render at module load, which bun does not time out, so each +run of TypeDoc there races bun's default 5 second test timeout. Render it at +module load as the others do (capturing the rejection), or otherwise take +TypeDoc's run out of the timed body, rather than raising the timeout. + +## Related + +- [Interop's go -> cpp cell holds up under load](/quest/m0/test-flakes-2/interop-go-cpp.md) - quarantined from the same PR From c12d988a119c7716a0b60dc2c45cd87698a1c5c7 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Tue, 10 Nov 2026 18:02:47 -0800 Subject: [PATCH 08/12] fix(drafts): an Alone warm route counts warm whatever the receiver's pull serves, as the model proves Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index cd18694f846..c5d6707b897 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -573,7 +573,7 @@ It carries the sender's `Serves` and its `Warm Cost`, the price it pays for its The sender sends a WARM_UPDATE when either changes, and a WARM_END once it no longer pulls the broadcast. A change that narrows whom the copy serves, from Alone to a Relay, from one Relay to another, or to no warm route at all, waits until the sender has kept the copy without the old demand for as long as it keeps any copy whose demand is gone, so a neighbor that just joined it does not move straight back. -A receiver counts a neighbor warm for a broadcast when the receiver's own pull does not serve Unknown, and the neighbor's warm route serves Alone or serves a Relay that is neither the receiver nor the root of the receiver's own pull. +A receiver counts a neighbor warm for a broadcast when the neighbor's warm route serves Alone, or serves a Relay that is neither the receiver nor the root of the receiver's own pull while that pull does not serve Unknown. Demand a receiver causes, through any number of relays, then never warms a neighbor toward it. The receiver prices a pull through each neighbor as: From 20801d16c3c57e7176de414f27ddc8bc9a02864d Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Tue, 10 Nov 2026 19:23:31 -0800 Subject: [PATCH 09/12] quest: record the other interop lanes that failed once under load Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/interop-go-cpp.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/quest/m0/test-flakes-2/interop-go-cpp.md b/quest/m0/test-flakes-2/interop-go-cpp.md index 83bf7ded45d..dc3cb48b17a 100644 --- a/quest/m0/test-flakes-2/interop-go-cpp.md +++ b/quest/m0/test-flakes-2/interop-go-cpp.md @@ -27,6 +27,16 @@ announced back to the relay. Reproduce with load, find why the consumed broadcast reaches the publish side, and fix it there. +A later full run on the same branch, with this cell quarantined, failed +three other lanes once each, all passing when rerun; like +[the finish cases](/quest/m0/test-flakes-2/interop-finish.md)' extra lanes, +they are recorded here, not quarantined: `rust-swapped -> js` timed out +after decoding only the opus rendition into the swapped tone, and +`rust-swapped -> js-native-node` ended with `Error: unroutable`, both on +`moq-lite-06`; the Python/Rust sockets lane on `moq-lite-07-wip` also ended +with only `Error: unroutable`, and passed in three narrowed reruns. Check +whether they share this cell's cause before quarantining them. + ## Related - [Interop's finish cases hold up under load](/quest/m0/test-flakes-2/interop-finish.md) - another interop quarantine seen under the same load From 492ef3dab7f675bfd766642643fca37e67cd4a7b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 11:53:42 -0800 Subject: [PATCH 10/12] style: format the merged route tests Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/route/mod.rs | 7 ++++++- rs/moq-net/src/route/tests/rtt.rs | 10 +++++++++- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/rs/moq-net/src/route/mod.rs b/rs/moq-net/src/route/mod.rs index c190187a463..71a8a573fac 100644 --- a/rs/moq-net/src/route/mod.rs +++ b/rs/moq-net/src/route/mod.rs @@ -1129,7 +1129,12 @@ impl Router { } let metric = self.metric(next, entry)?; let rtt = through_rtt(&entry.advert, &self.neighbors[&next]); - Some(Eligible { next, metric, rtt, tied }) + Some(Eligible { + next, + metric, + rtt, + tied, + }) }) .collect() } diff --git a/rs/moq-net/src/route/tests/rtt.rs b/rs/moq-net/src/route/tests/rtt.rs index 07960939b89..5c64d861b1d 100644 --- a/rs/moq-net/src/route/tests/rtt.rs +++ b/rs/moq-net/src/route/tests/rtt.rs @@ -45,7 +45,15 @@ fn each_relay_adds_its_link_to_the_sum() { assert_eq!(advertised(&sim, a, b, 7), ms(10)); assert_eq!(advertised(&sim, b, c, 7), ms(30)); let routes = sim.nodes[c].eligible(7); - assert_eq!(routes, [Eligible { next: b, metric: 3, rtt: ms(60), tied: false }]); + assert_eq!( + routes, + [Eligible { + next: b, + metric: 3, + rtt: ms(60), + tied: false + }] + ); } /// A shift on one link sends one ROUTE_UPDATE on each stream carrying the route from From 8b16ca88dccc90a814680a138d22ce8c952b93e4 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Wed, 11 Nov 2026 23:17:45 -0800 Subject: [PATCH 11/12] test(interop): quarantine the rust finish cells, and let INTEROP_QUARANTINED run quarantined lanes Also hash the cache-routing model with the draft's Rendezvous Hash basis, re-recording the ring's copies and UNSETTLED, and drop the seqno-churn follow-up now that routes drop the seqno. Co-Authored-By: Claude Opus 5.5 --- .../test-flakes-2/interop-finish-fallback.md | 34 +++++++++++++++++++ test/interop/interop.sh | 8 +++++ 2 files changed, 42 insertions(+) create mode 100644 quest/m0/test-flakes-2/interop-finish-fallback.md diff --git a/quest/m0/test-flakes-2/interop-finish-fallback.md b/quest/m0/test-flakes-2/interop-finish-fallback.md new file mode 100644 index 00000000000..35582b09183 --- /dev/null +++ b/quest/m0/test-flakes-2/interop-finish-fallback.md @@ -0,0 +1,34 @@ +# [S] Interop's finish readers keep their session across a transport fallback + +## Goal + +`just test interop --all --finish` passes the `finish rust -> *` cells on a +loaded host, and their quarantine in `test/interop/interop.sh`'s +`run_finishes` is removed. + +## Plan + +Found 2026-11-12 under the flaky-test rule, while iterating +[moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457), on a host at a +load average near 24. Every `finish rust -> *` cell failed together, and +`just test interop --all --finish` passed all of them on rerun. The Rust +publisher never printed `finish acknowledged` because three readers did not +end clean, so the harness never sent the ack: + +- The gst reader connected over the WebSocket fallback, since QUIC was not + reachable in time, then logged `preferred transport reachable; leaving the + fallback` and moved to WebTransport mid-test. +- The cpp reader also used the fallback and ended with `health EOF or + session continuity`. +- The c reader reconnected (`connects=2`) and ended with `finish did not + drain naturally (frames=0 expected=10)`. + +Suspected: a client that leaves the WebSocket fallback for QUIC opens a new +session, which the finish case reads as lost continuity. Decide whether the +move should keep the reader's subscriptions whole (a behavior fix in +`moq_tokio::Connection`) or whether the finish case must pin its transport, +reproduce under load with `INTEROP_QUARANTINED=1` and the logs kept, and fix +the cause, never with a longer limit. Other publishers' finish cells share +the readers, so check whether they fail the same way. + +Public API: none expected. Wire: none. diff --git a/test/interop/interop.sh b/test/interop/interop.sh index a2bc1bf15f8..bb3e3bc1c27 100755 --- a/test/interop/interop.sh +++ b/test/interop/interop.sh @@ -1824,6 +1824,14 @@ run_finishes() { return fi for pub in "${PUB_LIST[@]}"; do + # Quarantined: quest/m0/test-flakes-2/interop-finish-fallback.md. + if [[ "$QUARANTINED" -eq 0 && "$pub" == rust ]]; then + for sub in "${SUB_LIST[@]}"; do + echo " SKIP finish $pub -> $sub (quarantined)" + record skip case=finish publisher="$pub" subscriber="$sub" detail="quarantined: quest/m0/test-flakes-2/interop-finish-fallback.md" + done + continue + fi name="finish-$pub" # The fixture and its .health sibling share a parent, so one prefix grant covers both: # moq-transport requests name prefixes alone. From a6a0933d27f9f93667b688b1b30eee7cf5d14bd3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 13:40:22 -0800 Subject: [PATCH 12/12] test(interop): a rerun line carries INTEROP_QUARANTINED A failing INTEROP_QUARANTINED=1 run printed a rerun command that ran with the quarantine back on. Co-Authored-By: Claude Opus 5.5 --- test/interop/interop.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/interop/interop.sh b/test/interop/interop.sh index 4890100ff83..0534dcd6a1e 100755 --- a/test/interop/interop.sh +++ b/test/interop/interop.sh @@ -50,7 +50,7 @@ source "$INTEROP_DIR/../lib/harness.sh" # Captured before the parse below consumes it, so the rerun command carries every # flag and every environment override this run was actually given. -RERUN="$(harness_env INTEROP_TIMEOUT INTEROP_FPS INTEROP_SIZE INTEROP_PORT INTEROP_PROFILE RELAY_BIN CLI_BIN INTEROP_SUB_MOQ INTEROP_VERSION INTEROP_COMPAT_CLIENT INTEROP_DECODE INTEROP_PUB_LIVE INTEROP_FETCH_DATA INTEROP_FETCH_TRACK)just test interop$(harness_argv "$@")" +RERUN="$(harness_env INTEROP_TIMEOUT INTEROP_FPS INTEROP_SIZE INTEROP_PORT INTEROP_PROFILE INTEROP_QUARANTINED RELAY_BIN CLI_BIN INTEROP_SUB_MOQ INTEROP_VERSION INTEROP_COMPAT_CLIENT INTEROP_DECODE INTEROP_PUB_LIVE INTEROP_FETCH_DATA INTEROP_FETCH_TRACK)just test interop$(harness_argv "$@")" PUBLISHERS="rust" SUBSCRIBERS="rust"