diff --git a/quest/m1/route-seqno-bumps.md b/quest/m1/route-seqno-bumps.md deleted file mode 100644 index cc040af565c..00000000000 --- a/quest/m1/route-seqno-bumps.md +++ /dev/null @@ -1,55 +0,0 @@ -# [S] A node advances its route seqno only when a route link changes - -## Goal - -A lite-07 node raises its own route Seqno unprompted only when a session it -exchanges routes over opens or closes, and otherwise only to answer a -ROUTE_REQUEST. A client connecting to or leaving a relay no longer makes the -relay send a ROUTE_UPDATE for its own identity on every link that carries it. -A test counts the ROUTE messages a mesh sends per client session and holds -them at zero. - -## Plan - -Decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing), asked when -an origin should advance its seqno unprompted: ✅ only when a route link -changes (rejected: only to answer a ROUTE_REQUEST, which makes every path -change that lengthens a route wait a round trip to the origin; keeping -today's per-stream bump until a sweep measures it). Milestone: ✅ m1 -(rejected: m2 in the routes epic; m0), since relays run lite-07 in production -now and this changes no wire. - -Why it exists: relays that remember a short distance to a node refuse its -new, longer path until the node issues a higher seqno. Bumping when its links -change saves the ROUTE_REQUEST round trip. - -What is wrong today: `register` and `Drop for Stream` in -`rs/moq-net/src/route/node.rs` call `router.bump()` for every announce stream -of every lite-07 session, and `Routed::limit` in -`rs/moq-net/src/lite/subscriber/routed.rs` registers again on a permit -change. A relay originates the claims of its lite-06 and moq-transport -publishers under its own identity, so each client session bumps that identity -at least four times. This was read from the code, not measured: reproduce it -first with a count of ROUTE messages per client connect and disconnect on a -three-relay mesh. - -Open for the implementer, with a recommendation: what counts as a route -link. Recommended: a session whose peer forwards, known because it has sent a -route to a node other than itself, plus every session of a node that does not -forward (all of a publisher's links are route links). A session that only -ever carries claims, or only the peer's own route, never bumps. Confirm the -choice against `reconnect_without_restart_needs_a_seqno_advance` and -`restart_reusing_identity_is_refused` in `route/tests/`, which the bump -exists for, and say in the PR if a narrower rule (bump on close only, since a -new link lengthens no path) also keeps them passing. - -Mirror any change to `bump`'s callers in `RouteHandoff.tla` as the -`verify-route` skill says, and state the ROUTE budget per session event so -the mesh sweep (`rs/moq-net/tests/integration/mesh_sweep.rs`) can assert it. - -Public API: none. Wire: none. - -## Related - -- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the contract's seqno rule this narrows -- [Each side says everything about routing on one stream](/quest/m2/cluster-routing/routes/route-stream.md) - registers one neighbour per session, which changes where the bump is called diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index 52cf1df74f5..cf6f979600e 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -35,7 +35,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 18. ✅ Asked who is an origin node: **every publisher stays an origin node.** Its identity is for signing, for one node across several links (redundant publishers), and because browsers will forward ([browser mesh](/quest/m5/browser-mesh.md)). Rejected: a relay originating, under its own Node ID, the claims of any peer that sends no Node ID or holds a scoped publish grant, as it already does for lite-06 and moq-transport publishers. That would make route state follow relays rather than publishers and remove qualified Node IDs, the Route Key, and the echo, but a publisher would have no identity of its own. Accepted cost: routes, holds, and seqno requests scale with publishers, so a node must still forget dead identities, and the echo that makes forgetting safe has to be sound. 19. ✅ Asked whether to keep them: **ROUTE_HOLD stays, and the newest floor wins per identity.** Rejected: dropping holds, where each stall takes every claim behind the node offline and re-sends them all; and claims that follow only the selected next hop, which re-pulls a node's whole claim table on each next-hop change and showed revivals and false ENDs in the sweeps below. -20. ✅ Asked when an origin advances its seqno unprompted: **only when a route link changes**, which [the seqno bumps](/quest/m1/route-seqno-bumps.md) implements. Today every announce stream open or close bumps it. Rejected: only to answer a ROUTE_REQUEST; keeping the per-stream bump. +20. ✅ Asked when an origin advances its seqno unprompted: **only when a route link changes**, implemented as a bump when a route link closes. Rejected: only to answer a ROUTE_REQUEST; keeping the per-stream bump. 21. ✅ Asked how one side's routing messages travel so none overtakes another: **one unidirectional Announce Stream per side**, carrying its requests, its answers, and its echoes in order ([the route stream](/quest/m2/cluster-routing/routes/route-stream.md)). It replaces "There MAY be multiple Announce Streams". Rejected: one bidirectional Announce Stream per direction with the echo moved to the answering half; one bidirectional stream per session. 22. ✅ **A side's interest is a set of prefixes edited in place.** The live set never nests, and one edit may swap prefixes for a wider or narrower one without re-sending what both cover ([the interest set](/quest/m2/cluster-routing/routes/interest-set.md)). A token-scope change is such an edit, with one prefix per region of the scope. This supersedes items 16 and 17, and the handoff contract's "An interest change opens a new stream" below, once the route stream lands. 23. ✅ Asked what a floor covers with several prefixes on one stream: **one prefix, named by its id.** This amends item 10: the floor is still a marker and never a region list. Rejected: a floor over the whole set. @@ -137,11 +137,13 @@ the bullets below are the contract it carries. stay unverified. Switching between paths resumes only across a matching explicit epoch. Keep administrative identity and broadcast epoch separate. The origin - advances its seqno whenever its own set of sessions changes: feasibility - distances held elsewhere otherwise refuse a restart that reuses an - identity, and starve the old neighbors of an origin that moved to a - farther attachment (`restart_reusing_identity_is_refused`, - `reconnect_without_restart_needs_a_seqno_advance`). A random identity alone + advances its seqno unprompted when a route link closes: a session whose + peer forwarded it a route, or any session carrying the route of a node + that forwards nothing. Feasibility distances held elsewhere otherwise + starve the old neighbors of an origin that moved to a farther attachment + until a ROUTE_REQUEST reaches it + (`reconnect_without_restart_needs_a_seqno_advance`). A new link lengthens + no path, and a client session that forwards nothing changes none. A random identity alone does not keep a dead incarnation away. What bounds it is ordered sessions, state scoped to the session that delivered it, never advertising an unreachable route, keeping the feasibility distance until every diff --git a/quest/m2/cluster-routing/routes/route-stream.md b/quest/m2/cluster-routing/routes/route-stream.md index 4f557833dfa..3f6b900cffa 100644 --- a/quest/m2/cluster-routing/routes/route-stream.md +++ b/quest/m2/cluster-routing/routes/route-stream.md @@ -133,6 +133,5 @@ Public API: none expected. Wire: lite-07-wip only. ## Related - [js/net route layer](/quest/m2/cluster-routing/routes/js.md) - the JS side of the same wire -- [A node advances its route seqno only when a route link changes](/quest/m1/route-seqno-bumps.md) - changes the same registration path - [Announce interest streams share one route table](/quest/m4/announce-interest-memory.md) - lite-07 then holds one interest stream per side; the bound still matters for older versions - [A regrant re-checks only the routes it can change](/quest/m4/regrant-route-recheck.md) - its per-stream product loses a factor on lite-07 diff --git a/rs/moq-net/src/lite/subscriber/routed.rs b/rs/moq-net/src/lite/subscriber/routed.rs index 224dd6d83d6..cbaa658c125 100644 --- a/rs/moq-net/src/lite/subscriber/routed.rs +++ b/rs/moq-net/src/lite/subscriber/routed.rs @@ -373,6 +373,7 @@ impl Routed { if self.drained { stream.goaway(); } + self.stream.replaced_by(&stream); drop(std::mem::replace(&mut self.stream, stream)); self.previous = previous; self.outputs(subscriber); diff --git a/rs/moq-net/src/route/node.rs b/rs/moq-net/src/route/node.rs index e0076100f02..0aa3005b988 100644 --- a/rs/moq-net/src/route/node.rs +++ b/rs/moq-net/src/route/node.rs @@ -15,7 +15,7 @@ //! only once another serves it, so the origin always holds a route for it. use std::{ - collections::{BTreeMap, BTreeSet, HashMap, VecDeque}, + collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque}, sync::{Arc, Mutex, MutexGuard}, task::{Poll, ready}, }; @@ -356,6 +356,14 @@ struct State { /// with the name each stream's peer gives it: while they stay the same, only a /// claim whose shown state changed is served differently. routes: HashMap, + /// The streams whose close changes a path to this node, which then advances its + /// seqno so a node remembering a shorter path accepts the longer one without a + /// ROUTE_REQUEST: one whose peer forwarded it a route, or, on a node that does + /// not forward, one this node's route goes out on. A new stream lengthens no path. + links: HashSet, + /// Answer streams of a node that does not forward, which join `links` once this + /// node's route goes out on them. + origins: HashSet, } /// An identity's usable feasible routes: each stream, its metric and RTT, and the @@ -418,6 +426,8 @@ impl Node { queues: HashMap::new(), homes: BTreeMap::new(), routes: HashMap::new(), + links: HashSet::new(), + origins: HashSet::new(), })), } } @@ -465,17 +475,21 @@ impl Node { /// Register a stream this node asked the peer for under `prefix`. pub fn receive(&self, link: Link, prefix: Path) -> Result { - self.register(link, |router, key| router.asks(key, prefix)) + self.register(link, false, |router, key| router.asks(key, prefix)) } /// Register a stream answering the peer's request for `prefix`. pub fn answer(&self, link: Link, prefix: Path) -> Result { - self.register(link, |router, key| router.peer_interest(key, prefix)) + // A node that forwards nothing knows no better than that every stream carrying + // its route is a path to it; one whose prefix matches no claim carries none yet. + let origin = !link.proxy; + self.register(link, origin, |router, key| router.peer_interest(key, prefix)) } fn register( &self, link: Link, + origin: bool, open: impl FnOnce(&mut Router, Key) -> Result<(), Error>, ) -> Result { let mut state = self.lock(); @@ -484,13 +498,16 @@ impl Node { state.router.connect(key, link)?; let queue = kio::Shared::default(); state.queues.insert(key, queue.clone()); + if origin { + state.origins.insert(key); + } if let Err(err) = open(&mut state.router, key) { state.router.disconnect(key); state.queues.remove(&key); + state.origins.remove(&key); + state.links.remove(&key); return Err(err); } - // The node's own set of sessions changed. - let _ = state.router.bump(); state.flush(); Ok(Stream { node: self.clone(), @@ -565,7 +582,25 @@ impl Stream { /// ROUTE_START, ROUTE_UPDATE, ROUTE_HOLD (`Held`), or ROUTE_END (`None`) for /// the identity the peer names `named`. pub fn route(&self, named: Id, reach: Option) { - self.with(|router, key| router.route(key, named, reach)); + let mut state = self.node.lock(); + // Only the peer's own route has metric 0: any other means the peer forwards. + if matches!(&reach, Some(Reach::Finite(advert)) if advert.metric > 0) { + state.links.insert(self.key); + } + state.router.route(self.key, named, reach); + state.flush(); + } + + /// `next`, of the same session, takes this stream's place: it inherits whether + /// this one was a route link, so closing this one changes no path. + pub fn replaced_by(&self, next: &Stream) { + let mut state = self.node.lock(); + if state.links.remove(&self.key) { + state.links.insert(next.key); + } + if state.origins.remove(&self.key) { + state.origins.insert(next.key); + } } /// A batch of claim changes for the route the peer names `named`, closed by @@ -625,7 +660,10 @@ impl Drop for Stream { for home in state.homes.values_mut() { home.keys.remove(&self.key); } - let _ = state.router.bump(); + state.origins.remove(&self.key); + if state.links.remove(&self.key) { + let _ = state.router.bump(); + } state.flush(); } } @@ -643,7 +681,12 @@ impl State { let mut acked = Vec::new(); for out in self.router.drain() { let (key, output) = match out { - Out::Route { to, id, reach } => (to, Output::Route { id, reach }), + Out::Route { to, id, reach } => { + if reach.is_some() && Some(id) == self.router.declared() && self.origins.contains(&to) { + self.links.insert(to); + } + (to, Output::Route { id, reach }) + } Out::Batch { to, id, changes, floor } => (to, Output::Batch { id, changes, floor }), Out::Request { to, id, seqno } => { if let Some(queue) = self.queues.get(&to) { @@ -1310,6 +1353,98 @@ mod tests { assert_eq!(read.len(), 2, "the route, then its claim"); } + /// The seqnos of the finite ROUTE messages for `id` a stream would send. + fn routes(stream: &Stream, id: Id) -> Vec { + (outputs(stream).into_iter()) + .filter_map(|output| match output { + Output::Route { id: to, reach } if to == id => match reach { + Some(Reach::Finite(advert)) => Some(advert.seqno), + reach => panic!("unexpected reach {reach:?}"), + }, + _ => None, + }) + .collect() + } + + fn finite(seqno: Seqno, metric: Metric) -> Option { + Some(Reach::Finite(Advert { + seqno, + metric, + down_only: false, + rtt: None, + hops: [].into(), + })) + } + + /// A relay's own seqno advances only when a session over which another node + /// learned a path to it closes: a client that forwards nothing costs no ROUTE. + #[test] + fn a_relay_bumps_only_when_a_route_link_closes() { + let relay = Node::new(2, 2); + relay.own(p("live/r"), Some(claim(&Instance::random(), 0))); + let forwarding = Link::default(); + // Two peer relays, each forwarding a route of a third node. + let to_a = relay.answer(forwarding.clone(), Path::empty()).unwrap(); + let from_a = relay.receive(Link::default(), Path::empty()).unwrap(); + let to_b = relay.answer(forwarding.clone(), Path::empty()).unwrap(); + let from_b = relay.receive(Link::default(), Path::empty()).unwrap(); + from_a.route(5, finite(0, 1)); + from_b.route(6, finite(0, 1)); + assert_eq!(routes(&to_b, 2), [0]); + outputs(&to_a); + + // A client sending only its own route connects and leaves. + let to_client = relay.answer(forwarding, Path::empty()).unwrap(); + let from_client = relay.receive(Link::default(), Path::empty()).unwrap(); + from_client.route(9, finite(0, 0)); + drop((to_client, from_client)); + assert!(routes(&to_a, 2).is_empty()); + assert!(routes(&to_b, 2).is_empty()); + + // A permit change refiling a peer's stream is no link change either. + // The refile replays no forwarded route, yet the session stays a route link. + let refiled = relay.receive(Link::default(), Path::empty()).unwrap(); + from_a.replaced_by(&refiled); + drop(from_a); + assert!(routes(&to_b, 2).is_empty()); + + // Losing a forwarding peer does change a path. + drop((to_a, refiled)); + assert_eq!(routes(&to_b, 2), [1]); + drop(from_b); + } + + /// A node that forwards nothing advances its seqno whenever a session carrying + /// its route closes, so moving to a farther relay needs no ROUTE_REQUEST. + #[test] + fn a_publisher_bumps_when_any_session_closes() { + let publisher = Node::new(9, 9); + publisher.own(p("live/p"), Some(claim(&Instance::random(), 0))); + let origin = Link { + proxy: false, + ..Link::default() + }; + let first = publisher.answer(origin.clone(), Path::empty()).unwrap(); + let second = publisher.answer(origin, Path::empty()).unwrap(); + assert_eq!(routes(&second, 9), [0]); + drop(first); + assert_eq!(routes(&second, 9), [1]); + + // A session whose prefix matches no claim carries no route, so its close moves no path. + let unmatched = publisher + .answer( + Link { + proxy: false, + ..Link::default() + }, + p("other"), + ) + .unwrap(); + assert!(outputs(&unmatched).is_empty()); + drop(unmatched); + assert!(routes(&second, 9).is_empty()); + } + #[test] fn a_route_lost_with_its_session_leaves_the_others_serving() { let instance = Instance::random();