Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 0 additions & 55 deletions quest/m1/route-seqno-bumps.md

This file was deleted.

14 changes: 8 additions & 6 deletions quest/m2/cluster-routing/routes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout

18. ✅ Asked who is an origin node: **every publisher stays an origin node.** Its identity is for signing, for one node across several links (redundant publishers), and because browsers will forward ([browser mesh](/quest/m5/browser-mesh.md)). Rejected: a relay originating, under its own Node ID, the claims of any peer that sends no Node ID or holds a scoped publish grant, as it already does for lite-06 and moq-transport publishers. That would make route state follow relays rather than publishers and remove qualified Node IDs, the Route Key, and the echo, but a publisher would have no identity of its own. Accepted cost: routes, holds, and seqno requests scale with publishers, so a node must still forget dead identities, and the echo that makes forgetting safe has to be sound.
19. ✅ Asked whether to keep them: **ROUTE_HOLD stays, and the newest floor wins per identity.** Rejected: dropping holds, where each stall takes every claim behind the node offline and re-sends them all; and claims that follow only the selected next hop, which re-pulls a node's whole claim table on each next-hop change and showed revivals and false ENDs in the sweeps below.
20. ✅ Asked when an origin advances its seqno unprompted: **only when a route link changes**, which [the seqno bumps](/quest/m1/route-seqno-bumps.md) implements. Today every announce stream open or close bumps it. Rejected: only to answer a ROUTE_REQUEST; keeping the per-stream bump.
20. ✅ Asked when an origin advances its seqno unprompted: **only when a route link changes**, implemented as a bump when a route link closes. Rejected: only to answer a ROUTE_REQUEST; keeping the per-stream bump.
21. ✅ Asked how one side's routing messages travel so none overtakes another: **one unidirectional Announce Stream per side**, carrying its requests, its answers, and its echoes in order ([the route stream](/quest/m2/cluster-routing/routes/route-stream.md)). It replaces "There MAY be multiple Announce Streams". Rejected: one bidirectional Announce Stream per direction with the echo moved to the answering half; one bidirectional stream per session.
22. ✅ **A side's interest is a set of prefixes edited in place.** The live set never nests, and one edit may swap prefixes for a wider or narrower one without re-sending what both cover ([the interest set](/quest/m2/cluster-routing/routes/interest-set.md)). A token-scope change is such an edit, with one prefix per region of the scope. This supersedes items 16 and 17, and the handoff contract's "An interest change opens a new stream" below, once the route stream lands.
23. ✅ Asked what a floor covers with several prefixes on one stream: **one prefix, named by its id.** This amends item 10: the floor is still a marker and never a region list. Rejected: a floor over the whole set.
Expand Down Expand Up @@ -137,11 +137,13 @@ the bullets below are the contract it carries.
stay unverified.
Switching between paths resumes only across a matching explicit epoch.
Keep administrative identity and broadcast epoch separate. The origin
advances its seqno whenever its own set of sessions changes: feasibility
distances held elsewhere otherwise refuse a restart that reuses an
identity, and starve the old neighbors of an origin that moved to a
farther attachment (`restart_reusing_identity_is_refused`,
`reconnect_without_restart_needs_a_seqno_advance`). A random identity alone
advances its seqno unprompted when a route link closes: a session whose
peer forwarded it a route, or any session carrying the route of a node
that forwards nothing. Feasibility distances held elsewhere otherwise
starve the old neighbors of an origin that moved to a farther attachment
until a ROUTE_REQUEST reaches it
(`reconnect_without_restart_needs_a_seqno_advance`). A new link lengthens
no path, and a client session that forwards nothing changes none. A random identity alone
does not keep a dead incarnation away. What bounds it is ordered sessions,
state scoped to the session that delivered it, never advertising an
unreachable route, keeping the feasibility distance until every
Expand Down
1 change: 0 additions & 1 deletion quest/m2/cluster-routing/routes/route-stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,5 @@ Public API: none expected. Wire: lite-07-wip only.
## Related

- [js/net route layer](/quest/m2/cluster-routing/routes/js.md) - the JS side of the same wire
- [A node advances its route seqno only when a route link changes](/quest/m1/route-seqno-bumps.md) - changes the same registration path
- [Announce interest streams share one route table](/quest/m4/announce-interest-memory.md) - lite-07 then holds one interest stream per side; the bound still matters for older versions
- [A regrant re-checks only the routes it can change](/quest/m4/regrant-route-recheck.md) - its per-stream product loses a factor on lite-07
1 change: 1 addition & 0 deletions rs/moq-net/src/lite/subscriber/routed.rs
Original file line number Diff line number Diff line change
Expand Up @@ -373,6 +373,7 @@ impl Routed {
if self.drained {
stream.goaway();
}
self.stream.replaced_by(&stream);
drop(std::mem::replace(&mut self.stream, stream));
self.previous = previous;
self.outputs(subscriber);
Expand Down
151 changes: 143 additions & 8 deletions rs/moq-net/src/route/node.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
//! only once another serves it, so the origin always holds a route for it.

use std::{
collections::{BTreeMap, BTreeSet, HashMap, VecDeque},
collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque},
sync::{Arc, Mutex, MutexGuard},
task::{Poll, ready},
};
Expand Down Expand Up @@ -356,6 +356,14 @@ struct State {
/// with the name each stream's peer gives it: while they stay the same, only a
/// claim whose shown state changed is served differently.
routes: HashMap<Id, Routes>,
/// The streams whose close changes a path to this node, which then advances its
/// seqno so a node remembering a shorter path accepts the longer one without a
/// ROUTE_REQUEST: one whose peer forwarded it a route, or, on a node that does
/// not forward, one this node's route goes out on. A new stream lengthens no path.
links: HashSet<Key>,
/// Answer streams of a node that does not forward, which join `links` once this
/// node's route goes out on them.
origins: HashSet<Key>,
}

/// An identity's usable feasible routes: each stream, its metric and RTT, and the
Expand Down Expand Up @@ -418,6 +426,8 @@ impl Node {
queues: HashMap::new(),
homes: BTreeMap::new(),
routes: HashMap::new(),
links: HashSet::new(),
origins: HashSet::new(),
})),
}
}
Expand Down Expand Up @@ -465,17 +475,21 @@ impl Node {

/// Register a stream this node asked the peer for under `prefix`.
pub fn receive(&self, link: Link, prefix: Path) -> Result<Stream, Error> {
self.register(link, |router, key| router.asks(key, prefix))
self.register(link, false, |router, key| router.asks(key, prefix))
}

/// Register a stream answering the peer's request for `prefix`.
pub fn answer(&self, link: Link, prefix: Path) -> Result<Stream, Error> {
self.register(link, |router, key| router.peer_interest(key, prefix))
// A node that forwards nothing knows no better than that every stream carrying
// its route is a path to it; one whose prefix matches no claim carries none yet.
let origin = !link.proxy;
self.register(link, origin, |router, key| router.peer_interest(key, prefix))
}

fn register(
&self,
link: Link,
origin: bool,
open: impl FnOnce(&mut Router<Key, Claim>, Key) -> Result<(), Error>,
) -> Result<Stream, Error> {
let mut state = self.lock();
Expand All @@ -484,13 +498,16 @@ impl Node {
state.router.connect(key, link)?;
let queue = kio::Shared::default();
state.queues.insert(key, queue.clone());
if origin {
state.origins.insert(key);
}
if let Err(err) = open(&mut state.router, key) {
state.router.disconnect(key);
state.queues.remove(&key);
state.origins.remove(&key);
state.links.remove(&key);
return Err(err);
}
// The node's own set of sessions changed.
let _ = state.router.bump();
state.flush();
Ok(Stream {
node: self.clone(),
Expand Down Expand Up @@ -565,7 +582,25 @@ impl Stream {
/// ROUTE_START, ROUTE_UPDATE, ROUTE_HOLD (`Held`), or ROUTE_END (`None`) for
/// the identity the peer names `named`.
pub fn route(&self, named: Id, reach: Option<Reach>) {
self.with(|router, key| router.route(key, named, reach));
let mut state = self.node.lock();
// Only the peer's own route has metric 0: any other means the peer forwards.
if matches!(&reach, Some(Reach::Finite(advert)) if advert.metric > 0) {
state.links.insert(self.key);
}
state.router.route(self.key, named, reach);
state.flush();
}

/// `next`, of the same session, takes this stream's place: it inherits whether
/// this one was a route link, so closing this one changes no path.
pub fn replaced_by(&self, next: &Stream) {
let mut state = self.node.lock();
if state.links.remove(&self.key) {
state.links.insert(next.key);
}
if state.origins.remove(&self.key) {
state.origins.insert(next.key);
}
}

/// A batch of claim changes for the route the peer names `named`, closed by
Expand Down Expand Up @@ -625,7 +660,10 @@ impl Drop for Stream {
for home in state.homes.values_mut() {
home.keys.remove(&self.key);
}
let _ = state.router.bump();
state.origins.remove(&self.key);
if state.links.remove(&self.key) {
let _ = state.router.bump();
}
state.flush();
}
}
Expand All @@ -643,7 +681,12 @@ impl State {
let mut acked = Vec::new();
for out in self.router.drain() {
let (key, output) = match out {
Out::Route { to, id, reach } => (to, Output::Route { id, reach }),
Out::Route { to, id, reach } => {
if reach.is_some() && Some(id) == self.router.declared() && self.origins.contains(&to) {
self.links.insert(to);
}
(to, Output::Route { id, reach })
}
Out::Batch { to, id, changes, floor } => (to, Output::Batch { id, changes, floor }),
Out::Request { to, id, seqno } => {
if let Some(queue) = self.queues.get(&to) {
Expand Down Expand Up @@ -1310,6 +1353,98 @@ mod tests {
assert_eq!(read.len(), 2, "the route, then its claim");
}

/// The seqnos of the finite ROUTE messages for `id` a stream would send.
fn routes(stream: &Stream, id: Id) -> Vec<Seqno> {
(outputs(stream).into_iter())
.filter_map(|output| match output {
Output::Route { id: to, reach } if to == id => match reach {
Some(Reach::Finite(advert)) => Some(advert.seqno),
reach => panic!("unexpected reach {reach:?}"),
},
_ => None,
})
.collect()
}

fn finite(seqno: Seqno, metric: Metric) -> Option<Reach> {
Some(Reach::Finite(Advert {
seqno,
metric,
down_only: false,
rtt: None,
hops: [].into(),
}))
}

/// A relay's own seqno advances only when a session over which another node
/// learned a path to it closes: a client that forwards nothing costs no ROUTE.
#[test]
fn a_relay_bumps_only_when_a_route_link_closes() {
let relay = Node::new(2, 2);
relay.own(p("live/r"), Some(claim(&Instance::random(), 0)));
let forwarding = Link::default();
// Two peer relays, each forwarding a route of a third node.
let to_a = relay.answer(forwarding.clone(), Path::empty()).unwrap();
let from_a = relay.receive(Link::default(), Path::empty()).unwrap();
let to_b = relay.answer(forwarding.clone(), Path::empty()).unwrap();
let from_b = relay.receive(Link::default(), Path::empty()).unwrap();
from_a.route(5, finite(0, 1));
from_b.route(6, finite(0, 1));
assert_eq!(routes(&to_b, 2), [0]);
outputs(&to_a);

// A client sending only its own route connects and leaves.
let to_client = relay.answer(forwarding, Path::empty()).unwrap();
let from_client = relay.receive(Link::default(), Path::empty()).unwrap();
from_client.route(9, finite(0, 0));
drop((to_client, from_client));
assert!(routes(&to_a, 2).is_empty());
assert!(routes(&to_b, 2).is_empty());

// A permit change refiling a peer's stream is no link change either.
// The refile replays no forwarded route, yet the session stays a route link.
let refiled = relay.receive(Link::default(), Path::empty()).unwrap();
from_a.replaced_by(&refiled);
drop(from_a);
assert!(routes(&to_b, 2).is_empty());

// Losing a forwarding peer does change a path.
drop((to_a, refiled));
assert_eq!(routes(&to_b, 2), [1]);
drop(from_b);
}

/// A node that forwards nothing advances its seqno whenever a session carrying
/// its route closes, so moving to a farther relay needs no ROUTE_REQUEST.
#[test]
fn a_publisher_bumps_when_any_session_closes() {
let publisher = Node::new(9, 9);
publisher.own(p("live/p"), Some(claim(&Instance::random(), 0)));
let origin = Link {
proxy: false,
..Link::default()
};
let first = publisher.answer(origin.clone(), Path::empty()).unwrap();
let second = publisher.answer(origin, Path::empty()).unwrap();
assert_eq!(routes(&second, 9), [0]);
drop(first);
assert_eq!(routes(&second, 9), [1]);

// A session whose prefix matches no claim carries no route, so its close moves no path.
let unmatched = publisher
.answer(
Link {
proxy: false,
..Link::default()
},
p("other"),
)
.unwrap();
assert!(outputs(&unmatched).is_empty());
drop(unmatched);
assert!(routes(&second, 9).is_empty());
}

#[test]
fn a_route_lost_with_its_session_leaves_the_others_serving() {
let instance = Instance::random();
Expand Down