Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion quest/m2/multicast/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ waits on this epic's children, the other flaw,
whose fix changes the wire,
and two flaws the mesh replay found,
[feeding an old relay's subscription mid-roll](/quest/m2/multicast/roll-echo-stall.md)
and [a reader starving after a lite-07 link reopens](/quest/m2/multicast/reopen-starve.md);
and [a lite-07 front that stops delivering after it switches copies](/quest/m2/multicast/splice-stall.md);
and on
[route layer acceptance](/quest/m2/cluster-routes/acceptance.md) retiring
the lite-06 mitigations. It does not wait on
Expand Down
60 changes: 0 additions & 60 deletions quest/m2/multicast/reopen-starve.md

This file was deleted.

62 changes: 62 additions & 0 deletions quest/m2/multicast/splice-stall.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# [M] A front keeps delivering after it switches copies on lite-07

## Goal

On lite-07, a relay's front that switches to a new copy after a link reopens
or a relay crashes keeps delivering every group to its readers, as the
release (lite-06) does. Today it can deliver for a few seconds, then hand
out one group every 15-25s for a minute or more, with its route unchanged.

No reader starves in
`just test mesh-replay --mesh test/mesh-replay/ring4.toml --script test/mesh-replay/reopen.toml --broadcasts 32`
or in the nightly ring meshes, once `[[known]]` no longer names this quest in
`test/mesh-replay/script.toml`.

## Plan

Found by [the mesh replay](/test/mesh-replay/README.md) while fixing
[reopen-starve](https://github.com/moq-dev/moq/pull/9179), whose fix leaves no
request refused and never asked again. With that fix, 9 of 13 `reopen.toml`
runs on ring4 at 32 broadcasts passed (2 of 3 with
[#9154](https://github.com/moq-dev/moq/pull/9154)), and the others starved 3
to 205 reader paths. Before it, 17 of 24 passed, and failures starved 1 to 32
paths, some of them stranded requests.

What an instrumented failing run shows, after every link of `a` reopened:

- `a`'s front on `b/0` switched to the direct route to `b` as `b`'s copy of
the claims caught up, spliced the new copy, and fed its reader for 13s.
From then on its route never changed and no request was refused, yet
`b`'s groups reached `a`'s reader once every 15-25s.
- Over the same minute, `d`'s reader stalled the same way on all of `c`'s
broadcasts, through a front on the `c`-`d` link that the fault never
touched and that never switched.
- No shaper dropped or overflowed a packet, and every reader's status kept
time.

Pre-fix runs showed the same signature on a handful of paths: a front
spliced a copy that delivered, then its reader got nothing for 57-75s.

Likely cause, untested: [roll-echo-stall](https://github.com/moq-dev/moq/pull/9174)'s
first fix. A shared lite-07 copy never places a time start that a wider
range already covers, so the subscription it feeds waits. A spliced copy
resumes from a time. Its head could not be tested here: it no longer builds
on main, and its merge is being redone. Once it lands, rerun `reopen.toml`
several times. If no reader starves, delete this quest and its `[[known]]`
entry. Otherwise, instrument the stalled subscription on both ends of the
link: which groups the publisher serves, and what the subscriber's window
waits on.

The lite-07 cut waits on it, as decided for reopen-starve on 2026-11-19
(under the maintainer's delegation for lite-07 routing calls): it is the
same starved reader after a link flap or a crash, which a deployment sees
daily.

Not a wire change.

## Related

- [reopen-starve](https://github.com/moq-dev/moq/pull/9179) - a request
refused before the neighbour's claims returned, and never asked again
- [roll-echo-stall](/quest/m2/multicast/roll-echo-stall.md) - a time start a
shared copy never places
51 changes: 51 additions & 0 deletions rs/moq-net/src/route/node/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -579,3 +579,54 @@ fn a_waiting_request_wakes_when_its_claim_moves() {
assert!(flag.woken(), "the claim went");
assert_eq!(node.poll_forward(2, None, &hall, &waiter), Poll::Ready(None));
}

/// A neighbour that routes to a node before its claims arrive serves none of them, and is
/// handed each once its own copy holds it, though nothing changed of the claim here.
#[test]
fn a_neighbour_serves_a_claim_once_its_copy_holds_it() {
let (node, _clock) = node();
let room = cast("room");
let far = id(9);
let route = |via: u64, cost| Msg::Route {
origin: far,
offer: Some(Offer {
path: [
Hop { node: far, session: 0 },
Hop {
node: id(via),
session: 1,
},
]
.into(),
draining: false,
priority: i8::MAX,
cost,
marked: false,
}),
};
let claims = Msg::Claims {
origin: far,
changes: vec![(room.clone(), live(0))],
floor: 1,
};
let lagging = node.connect(1, port(Role::Routing(id(7)))).unwrap();
lagging.recv(Msg::Edited).unwrap();
let ahead = node.connect(2, port(Role::Routing(id(8)))).unwrap();
ahead.recv(Msg::Edited).unwrap();
lagging.recv(route(7, 1)).unwrap();
lagging.flush();
ahead.recv(route(8, 2)).unwrap();
ahead.recv(claims.clone()).unwrap();
ahead.flush();
assert!(serves(&lagging).is_empty(), "its copy lacks the claim");
assert_eq!(confirm(&ahead), 1);
let waiter = kio::Waiter::noop();
assert_eq!(
node.poll_forward(3, Some(far), &room, &waiter),
Poll::Ready(Some(Via::Route { node: far }))
);

lagging.recv(claims).unwrap();
lagging.flush();
assert_eq!(confirm(&lagging), 1);
}
132 changes: 127 additions & 5 deletions rs/moq-net/src/route/relay.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,10 @@
//! claim stay; a run it is first told unavailable, which nothing can serve, it takes only
//! once told it available. A request
//! follows the claim at the most specific path covering it that the reader shows, at its
//! claimant too, and fails closed while that one is unavailable or lost. Among the claims
//! claimant too, and fails closed while that one is unavailable or lost. It leaves toward
//! another node only through a neighbour whose own copy of that node's claims holds the
//! claim: a route can arrive before the claims behind it, and a neighbour refuses a request
//! for a claim it has not heard. Among the claims
//! of one Instance at one path, only the one shown closest to its own path, at the newest
//! epoch among those, may serve, reachable or not, as `model::origin` ranks them; a
//! route's rank chooses among those alone. A route whose cost a link would carry past the
Expand Down Expand Up @@ -1894,6 +1897,10 @@ impl<N: Key, I: Ord + Clone + fmt::Debug, A: Item> Router<N, I, A> {
};
}
heard.floor = floor;
// What a request may follow through the sender moved with its copy, even when this
// node's view of `origin` did not.
self.dirty
.extend(changes.iter().map(|(broadcast, _)| broadcast.clone()));
let known = self.known.entry(origin).or_default();
// The newest floor wins, so an older copy never revives what a newer one withdrew.
if floor <= known.floor && !mutated!(self, OldFloor) {
Expand Down Expand Up @@ -2280,10 +2287,48 @@ impl<N: Key, I: Ord + Clone + fmt::Debug, A: Item> Router<N, I, A> {
let best = (self.claims_of(&claim)).map(|(_, claim)| precedence(&claim.item)).min();
(!own.unavailable && best == Some(precedence(&own.item))).then_some(Next::Local(source))
}
false => self.best.get(&origin).map(|best| Next::Via(best.via)),
// The best route whose neighbour holds the claim: one that has yet to hear it
// would refuse the request.
false => {
let known = self.known.get(&origin);
let claim = (ancestors(path).rev())
.map(|above| (above, instance.clone()))
.find(|claim| known.is_some_and(|known| known.items.contains_key(claim)));
match claim {
Some(claim) => self.lead(origin, &claim).map(Next::Via),
None => self.best.get(&origin).map(|best| Next::Via(best.via)),
}
}
}
}

/// Whether `via`'s copy of `origin`'s claims holds `broadcast` as this node knows it:
/// available, and of the same precedence. A neighbour routes to `origin` before its claims
/// reach it, and refuses a request for a claim it has not heard, or heard only unavailable
/// or in an older revision.
fn holds(&self, origin: NodeId, via: N, broadcast: &Broadcast<I>) -> bool {
let Some(known) = (self.known.get(&origin)).and_then(|known| known.items.get(broadcast)) else {
return false;
};
(self.heard.get(&origin))
.and_then(|offers| offers.get(&via))
.and_then(|heard| heard.items.get(broadcast))
.is_some_and(|heard| !heard.unavailable && precedence(&heard.item) == precedence(&known.item))
}

/// The neighbour a request for `origin`'s claim of `broadcast` leaves by: the route to
/// `origin` when its neighbour holds the claim, else the best usable route whose does.
fn lead(&self, origin: NodeId, broadcast: &Broadcast<I>) -> Option<N> {
let best = self.best.get(&origin)?;
if self.holds(origin, best.via, broadcast) {
return Some(best.via);
}
(self.usable(origin).into_iter())
.filter(|route| self.holds(origin, route.via, broadcast))
.min_by_key(|route| order(route, Some(best.via)))
.map(|route| route.via)
}

/// Every claim of `broadcast` this node knows, with its claimant, reachable or not.
fn claims_of<'a>(
&'a self,
Expand Down Expand Up @@ -2333,14 +2378,18 @@ impl<N: Key, I: Ord + Clone + fmt::Debug, A: Item> Router<N, I, A> {
}
continue;
}
// Every usable route to its claimant serves it, for the same reason.
let best = self.best.get(&origin).map(|best| best.via);
// Every usable route to its claimant whose neighbour holds the claim serves it,
// for the same reason.
let lead = self.lead(origin, broadcast);
for route in self.usable(origin) {
if !self.holds(origin, route.via, broadcast) {
continue;
}
ways.push(Way {
origin,
next: Next::Via(route.via),
claim: claim.over(&route.offer),
closest: closest == Some(origin) && best == Some(route.via),
closest: closest == Some(origin) && lead == Some(route.via),
});
}
}
Expand Down Expand Up @@ -3152,6 +3201,72 @@ mod tests {
assert_eq!(router.step(me, &under), Some(Next::Local(2)));
}

/// A neighbour routes to a node before that node's claims reach it, and refuses a request
/// for one it has not heard. A request follows a claim only through a neighbour whose own
/// copy holds it, and through the cheaper one once that copy catches up.
#[test]
fn a_request_follows_only_a_neighbour_that_holds_the_claim() {
let mut router = Router::<usize, u64, Run>::new(node(1), debounce(HOP));
open(&mut router, 2, Role::Routing(node(2)));
open(&mut router, 3, Role::Routing(node(3)));
let broadcast = (p("live"), 7);
let claims = |floor| Msg::Claims {
origin: node(5),
changes: vec![(broadcast.clone(), claim(run(1), 0))],
floor,
};
// Both route to node 5, the cheaper through 2, whose copy of its claims is empty.
router.recv(2, route(5, 2, 1)).unwrap();
router.recv(3, route(5, 3, 2)).unwrap();
router.recv(3, claims(1)).unwrap();
assert_eq!(router.best(node(5)).map(|(via, _)| via), Some(2));
let ways = |router: &Router<usize, u64, Run>| {
(router.ways(&broadcast).into_iter())
.map(|way| (way.next, way.closest))
.collect::<Vec<_>>()
};
assert_eq!(ways(&router), [(Next::Via(3), true)]);
assert_eq!(router.step(node(5), &broadcast), Some(Next::Via(3)));
router.recv(2, claims(1)).unwrap();
assert_eq!(ways(&router), [(Next::Via(2), true), (Next::Via(3), false)]);
assert_eq!(router.step(node(5), &broadcast), Some(Next::Via(2)));
}

/// A neighbour whose copy holds the claim only unavailable refuses a request for it too,
/// though this node knows it available from another. Its copy catching up makes it a way.
#[test]
fn a_request_skips_a_neighbour_that_holds_the_claim_unavailable() {
let mut router = Router::<usize, u64, Run>::new(node(1), debounce(HOP));
open(&mut router, 2, Role::Routing(node(2)));
open(&mut router, 3, Role::Routing(node(3)));
let broadcast = (p("live"), 7);
let claims = |unavailable, floor| Msg::Claims {
origin: node(5),
changes: vec![(
broadcast.clone(),
match claim(run(1), 0) {
Change::Live(claim) => Change::Live(Claim { unavailable, ..claim }),
other => other,
},
)],
floor,
};
router.recv(2, route(5, 2, 1)).unwrap();
router.recv(3, route(5, 3, 2)).unwrap();
router.recv(2, claims(true, 1)).unwrap();
router.recv(3, claims(false, 2)).unwrap();
let ways = |router: &Router<usize, u64, Run>| {
(router.ways(&broadcast).into_iter())
.map(|way| (way.next, way.closest))
.collect::<Vec<_>>()
};
assert_eq!(ways(&router), [(Next::Via(3), true)]);
assert_eq!(router.step(node(5), &broadcast), Some(Next::Via(3)));
router.recv(2, claims(false, 3)).unwrap();
assert_eq!(ways(&router), [(Next::Via(2), true), (Next::Via(3), false)]);
assert_eq!(router.step(node(5), &broadcast), Some(Next::Via(2)));
}

/// Among claims of one Instance at one path, one exactly there outranks a broader one
/// shown there, and among those a newer epoch outranks an older one however much
/// cheaper the older is, or whether a route reaches the newer: among a node's
Expand Down Expand Up @@ -5462,7 +5577,14 @@ mod tests {
assert_eq!(router.pick(&room), Some(node(4)), "both marked: the lower Node ID");
assert_eq!(closest(&router), vec![(node(4), Next::Via(2), true)]);

// A route that starts brings the claims as they stand.
router.recv(3, route(4, 3, 1)).unwrap();
let claims = Msg::Claims {
origin: node(4),
changes: vec![(room.clone(), claim(run(0), 0))],
floor: 1,
};
router.recv(3, claims).unwrap();
router.drain();
assert_eq!(router.changes().routes, BTreeSet::from([node(4)]), "an unmarked route");
assert_eq!(router.pick(&room), Some(node(4)), "the unmarked claim is closest");
Expand Down
6 changes: 6 additions & 0 deletions rs/moq-net/src/route/tests/multicast.rs
Original file line number Diff line number Diff line change
Expand Up @@ -914,6 +914,11 @@ const UNAVAILABLE: [(&str, &str); 5] = [
/// And where a claim moves to another node, the router forwards the new node's route
/// before the old node's withdrawal at every hop, so some readers never see it offline
/// and never start it again: fewer of each, by as many.
///
/// And a request follows a claim only through a neighbour whose copy of the claimant's
/// claims holds it, where the prototype follows the route alone. While links flap, a
/// neighbour's route can arrive before its copy of the claims, and a request through it
/// would come back: fewer loops.
#[test]
fn the_router_gives_the_prototypes_answers() {
let mut out = Vec::new();
Expand All @@ -934,6 +939,7 @@ fn the_router_gives_the_prototypes_answers() {
"settle_rounds" => unavailable && b.rounds == a.rounds + 1,
"starts" => unavailable && b.sent.starts < a.sent.starts,
"offline" => unavailable && b.sent.offline < a.sent.offline,
"loops" if b.loops < a.loops => true,
"loops" => unavailable && shape.name == "three_meshes_ring" && b.loops == a.loops + 1,
_ => false,
};
Expand Down
Loading