diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md index 7ccc76cea1..b5b453fe53 100644 --- a/quest/m2/multicast/README.md +++ b/quest/m2/multicast/README.md @@ -42,7 +42,7 @@ waits on this epic's children, the other flaw, whose fix changes the wire, and two flaws the mesh replay found, [feeding an old relay's subscription mid-roll](/quest/m2/multicast/roll-echo-stall.md) -and [a reader starving after a lite-07 link reopens](/quest/m2/multicast/reopen-starve.md); +and [a lite-07 front that stops delivering after it switches copies](/quest/m2/multicast/splice-stall.md); and on [route layer acceptance](/quest/m2/cluster-routes/acceptance.md) retiring the lite-06 mitigations. It does not wait on diff --git a/quest/m2/multicast/reopen-starve.md b/quest/m2/multicast/reopen-starve.md deleted file mode 100644 index cf943f5484..0000000000 --- a/quest/m2/multicast/reopen-starve.md +++ /dev/null @@ -1,60 +0,0 @@ -# [M] A reader reads again after a lite-07 link reopens - -## Goal - -On lite-07, a reader behind a link that reopened or a relay that crashed reads -again once a route to its broadcast returns, as the release (lite-06) does -within seconds. Today a reader two hops away can show the broadcast live and -read nothing for 75s or more, until some later fault moves its route. - -No reader starves in -`just test mesh-replay --mesh test/mesh-replay/ring4.toml --script test/mesh-replay/reopen.toml --broadcasts 32` -or in the nightly ring meshes, once `[[known]]` no longer names this quest in -`test/mesh-replay/script.toml`. - -## Plan - -Found by [the mesh replay](/test/mesh-replay/README.md), at 32 broadcasts per -relay: - -- ring4 with every relay on the new build and no roll (`reopen.toml`): reader - `c` starved on all 32 of `a`'s broadcasts for 75s after `a`'s links reopened, - and a second run starved readers in both reopens and the kill. The same faults - with every relay on the release, under the same load, fed every reader again - within 4.1s. -- ring6 nightly, after the forward roll: a cut of `a`-`b` starved 186 reader - paths, the kill of `b` 269, and one path stayed starved through four later - events. -- At 4 broadcasts it is rare: 1 path after a cut and 1 after a kill in one of - three ring6 runs, and none in ten cuts of a ring6 that started on the new - build. - -No shaper dropped or overflowed a packet, and each reader's once-a-second -status kept time while its paths starved, so it is not the host's load. - -What the relay logs show: the reader's relay subscribed through a neighbour -whose own route had not returned yet, the neighbour refused it as unroutable, -and the relay never asked again, though the neighbour's route came back -seconds later. The reader's follow stays online with no group, so nothing -above it retries either. - -Likely cause: a track refused as unroutable is `stranded` on its source and -waits "for another source, or for the table to move" (`rs/moq-net/src/model/front.rs`, -`Track::stranded`, and `unstrand`, which asks again only when the serving -source changes). When the neighbour's route returns behind the same source, -this relay's table does not move, so the track is never asked for again. A fix -re-asks a stranded track when its source's route to the broadcast is shown -again, without asking a source that still refuses in a loop. Rerun the -`verify-failover` model, since this changes the front lifecycle it covers. - -Decided 2026-11-19 (under the maintainer's delegation for lite-07 routing -calls while away): ✅ the lite-07 cut waits on it (rejected: cutting without -it, or deferring it to after the cut). lite-06 settles the same faults in 4s, -so it is a lite-07 regression on a link flap or a crash, which a deployment -sees daily. - -Not a wire change. - -## Related - -- [roll-echo-stall](/quest/m2/multicast/roll-echo-stall.md) - the mid-roll starvation, through a lite-06 neighbour diff --git a/quest/m2/multicast/splice-stall.md b/quest/m2/multicast/splice-stall.md new file mode 100644 index 0000000000..252f67f24c --- /dev/null +++ b/quest/m2/multicast/splice-stall.md @@ -0,0 +1,62 @@ +# [M] A front keeps delivering after it switches copies on lite-07 + +## Goal + +On lite-07, a relay's front that switches to a new copy after a link reopens +or a relay crashes keeps delivering every group to its readers, as the +release (lite-06) does. Today it can deliver for a few seconds, then hand +out one group every 15-25s for a minute or more, with its route unchanged. + +No reader starves in +`just test mesh-replay --mesh test/mesh-replay/ring4.toml --script test/mesh-replay/reopen.toml --broadcasts 32` +or in the nightly ring meshes, once `[[known]]` no longer names this quest in +`test/mesh-replay/script.toml`. + +## Plan + +Found by [the mesh replay](/test/mesh-replay/README.md) while fixing +[reopen-starve](https://github.com/moq-dev/moq/pull/9179), whose fix leaves no +request refused and never asked again. With that fix, 9 of 13 `reopen.toml` +runs on ring4 at 32 broadcasts passed (2 of 3 with +[#9154](https://github.com/moq-dev/moq/pull/9154)), and the others starved 3 +to 205 reader paths. Before it, 17 of 24 passed, and failures starved 1 to 32 +paths, some of them stranded requests. + +What an instrumented failing run shows, after every link of `a` reopened: + +- `a`'s front on `b/0` switched to the direct route to `b` as `b`'s copy of + the claims caught up, spliced the new copy, and fed its reader for 13s. + From then on its route never changed and no request was refused, yet + `b`'s groups reached `a`'s reader once every 15-25s. +- Over the same minute, `d`'s reader stalled the same way on all of `c`'s + broadcasts, through a front on the `c`-`d` link that the fault never + touched and that never switched. +- No shaper dropped or overflowed a packet, and every reader's status kept + time. + +Pre-fix runs showed the same signature on a handful of paths: a front +spliced a copy that delivered, then its reader got nothing for 57-75s. + +Likely cause, untested: [roll-echo-stall](https://github.com/moq-dev/moq/pull/9174)'s +first fix. A shared lite-07 copy never places a time start that a wider +range already covers, so the subscription it feeds waits. A spliced copy +resumes from a time. Its head could not be tested here: it no longer builds +on main, and its merge is being redone. Once it lands, rerun `reopen.toml` +several times. If no reader starves, delete this quest and its `[[known]]` +entry. Otherwise, instrument the stalled subscription on both ends of the +link: which groups the publisher serves, and what the subscriber's window +waits on. + +The lite-07 cut waits on it, as decided for reopen-starve on 2026-11-19 +(under the maintainer's delegation for lite-07 routing calls): it is the +same starved reader after a link flap or a crash, which a deployment sees +daily. + +Not a wire change. + +## Related + +- [reopen-starve](https://github.com/moq-dev/moq/pull/9179) - a request + refused before the neighbour's claims returned, and never asked again +- [roll-echo-stall](/quest/m2/multicast/roll-echo-stall.md) - a time start a + shared copy never places diff --git a/rs/moq-net/src/route/node/tests.rs b/rs/moq-net/src/route/node/tests.rs index bf09c543c4..cf30ec9850 100644 --- a/rs/moq-net/src/route/node/tests.rs +++ b/rs/moq-net/src/route/node/tests.rs @@ -579,3 +579,54 @@ fn a_waiting_request_wakes_when_its_claim_moves() { assert!(flag.woken(), "the claim went"); assert_eq!(node.poll_forward(2, None, &hall, &waiter), Poll::Ready(None)); } + +/// A neighbour that routes to a node before its claims arrive serves none of them, and is +/// handed each once its own copy holds it, though nothing changed of the claim here. +#[test] +fn a_neighbour_serves_a_claim_once_its_copy_holds_it() { + let (node, _clock) = node(); + let room = cast("room"); + let far = id(9); + let route = |via: u64, cost| Msg::Route { + origin: far, + offer: Some(Offer { + path: [ + Hop { node: far, session: 0 }, + Hop { + node: id(via), + session: 1, + }, + ] + .into(), + draining: false, + priority: i8::MAX, + cost, + marked: false, + }), + }; + let claims = Msg::Claims { + origin: far, + changes: vec![(room.clone(), live(0))], + floor: 1, + }; + let lagging = node.connect(1, port(Role::Routing(id(7)))).unwrap(); + lagging.recv(Msg::Edited).unwrap(); + let ahead = node.connect(2, port(Role::Routing(id(8)))).unwrap(); + ahead.recv(Msg::Edited).unwrap(); + lagging.recv(route(7, 1)).unwrap(); + lagging.flush(); + ahead.recv(route(8, 2)).unwrap(); + ahead.recv(claims.clone()).unwrap(); + ahead.flush(); + assert!(serves(&lagging).is_empty(), "its copy lacks the claim"); + assert_eq!(confirm(&ahead), 1); + let waiter = kio::Waiter::noop(); + assert_eq!( + node.poll_forward(3, Some(far), &room, &waiter), + Poll::Ready(Some(Via::Route { node: far })) + ); + + lagging.recv(claims).unwrap(); + lagging.flush(); + assert_eq!(confirm(&lagging), 1); +} diff --git a/rs/moq-net/src/route/relay.rs b/rs/moq-net/src/route/relay.rs index db87ffc285..a40b9a96b7 100644 --- a/rs/moq-net/src/route/relay.rs +++ b/rs/moq-net/src/route/relay.rs @@ -52,7 +52,10 @@ //! claim stay; a run it is first told unavailable, which nothing can serve, it takes only //! once told it available. A request //! follows the claim at the most specific path covering it that the reader shows, at its -//! claimant too, and fails closed while that one is unavailable or lost. Among the claims +//! claimant too, and fails closed while that one is unavailable or lost. It leaves toward +//! another node only through a neighbour whose own copy of that node's claims holds the +//! claim: a route can arrive before the claims behind it, and a neighbour refuses a request +//! for a claim it has not heard. Among the claims //! of one Instance at one path, only the one shown closest to its own path, at the newest //! epoch among those, may serve, reachable or not, as `model::origin` ranks them; a //! route's rank chooses among those alone. A route whose cost a link would carry past the @@ -1894,6 +1897,10 @@ impl Router { }; } heard.floor = floor; + // What a request may follow through the sender moved with its copy, even when this + // node's view of `origin` did not. + self.dirty + .extend(changes.iter().map(|(broadcast, _)| broadcast.clone())); let known = self.known.entry(origin).or_default(); // The newest floor wins, so an older copy never revives what a newer one withdrew. if floor <= known.floor && !mutated!(self, OldFloor) { @@ -2280,10 +2287,48 @@ impl Router { let best = (self.claims_of(&claim)).map(|(_, claim)| precedence(&claim.item)).min(); (!own.unavailable && best == Some(precedence(&own.item))).then_some(Next::Local(source)) } - false => self.best.get(&origin).map(|best| Next::Via(best.via)), + // The best route whose neighbour holds the claim: one that has yet to hear it + // would refuse the request. + false => { + let known = self.known.get(&origin); + let claim = (ancestors(path).rev()) + .map(|above| (above, instance.clone())) + .find(|claim| known.is_some_and(|known| known.items.contains_key(claim))); + match claim { + Some(claim) => self.lead(origin, &claim).map(Next::Via), + None => self.best.get(&origin).map(|best| Next::Via(best.via)), + } + } } } + /// Whether `via`'s copy of `origin`'s claims holds `broadcast` as this node knows it: + /// available, and of the same precedence. A neighbour routes to `origin` before its claims + /// reach it, and refuses a request for a claim it has not heard, or heard only unavailable + /// or in an older revision. + fn holds(&self, origin: NodeId, via: N, broadcast: &Broadcast) -> bool { + let Some(known) = (self.known.get(&origin)).and_then(|known| known.items.get(broadcast)) else { + return false; + }; + (self.heard.get(&origin)) + .and_then(|offers| offers.get(&via)) + .and_then(|heard| heard.items.get(broadcast)) + .is_some_and(|heard| !heard.unavailable && precedence(&heard.item) == precedence(&known.item)) + } + + /// The neighbour a request for `origin`'s claim of `broadcast` leaves by: the route to + /// `origin` when its neighbour holds the claim, else the best usable route whose does. + fn lead(&self, origin: NodeId, broadcast: &Broadcast) -> Option { + let best = self.best.get(&origin)?; + if self.holds(origin, best.via, broadcast) { + return Some(best.via); + } + (self.usable(origin).into_iter()) + .filter(|route| self.holds(origin, route.via, broadcast)) + .min_by_key(|route| order(route, Some(best.via))) + .map(|route| route.via) + } + /// Every claim of `broadcast` this node knows, with its claimant, reachable or not. fn claims_of<'a>( &'a self, @@ -2333,14 +2378,18 @@ impl Router { } continue; } - // Every usable route to its claimant serves it, for the same reason. - let best = self.best.get(&origin).map(|best| best.via); + // Every usable route to its claimant whose neighbour holds the claim serves it, + // for the same reason. + let lead = self.lead(origin, broadcast); for route in self.usable(origin) { + if !self.holds(origin, route.via, broadcast) { + continue; + } ways.push(Way { origin, next: Next::Via(route.via), claim: claim.over(&route.offer), - closest: closest == Some(origin) && best == Some(route.via), + closest: closest == Some(origin) && lead == Some(route.via), }); } } @@ -3152,6 +3201,72 @@ mod tests { assert_eq!(router.step(me, &under), Some(Next::Local(2))); } + /// A neighbour routes to a node before that node's claims reach it, and refuses a request + /// for one it has not heard. A request follows a claim only through a neighbour whose own + /// copy holds it, and through the cheaper one once that copy catches up. + #[test] + fn a_request_follows_only_a_neighbour_that_holds_the_claim() { + let mut router = Router::::new(node(1), debounce(HOP)); + open(&mut router, 2, Role::Routing(node(2))); + open(&mut router, 3, Role::Routing(node(3))); + let broadcast = (p("live"), 7); + let claims = |floor| Msg::Claims { + origin: node(5), + changes: vec![(broadcast.clone(), claim(run(1), 0))], + floor, + }; + // Both route to node 5, the cheaper through 2, whose copy of its claims is empty. + router.recv(2, route(5, 2, 1)).unwrap(); + router.recv(3, route(5, 3, 2)).unwrap(); + router.recv(3, claims(1)).unwrap(); + assert_eq!(router.best(node(5)).map(|(via, _)| via), Some(2)); + let ways = |router: &Router| { + (router.ways(&broadcast).into_iter()) + .map(|way| (way.next, way.closest)) + .collect::>() + }; + assert_eq!(ways(&router), [(Next::Via(3), true)]); + assert_eq!(router.step(node(5), &broadcast), Some(Next::Via(3))); + router.recv(2, claims(1)).unwrap(); + assert_eq!(ways(&router), [(Next::Via(2), true), (Next::Via(3), false)]); + assert_eq!(router.step(node(5), &broadcast), Some(Next::Via(2))); + } + + /// A neighbour whose copy holds the claim only unavailable refuses a request for it too, + /// though this node knows it available from another. Its copy catching up makes it a way. + #[test] + fn a_request_skips_a_neighbour_that_holds_the_claim_unavailable() { + let mut router = Router::::new(node(1), debounce(HOP)); + open(&mut router, 2, Role::Routing(node(2))); + open(&mut router, 3, Role::Routing(node(3))); + let broadcast = (p("live"), 7); + let claims = |unavailable, floor| Msg::Claims { + origin: node(5), + changes: vec![( + broadcast.clone(), + match claim(run(1), 0) { + Change::Live(claim) => Change::Live(Claim { unavailable, ..claim }), + other => other, + }, + )], + floor, + }; + router.recv(2, route(5, 2, 1)).unwrap(); + router.recv(3, route(5, 3, 2)).unwrap(); + router.recv(2, claims(true, 1)).unwrap(); + router.recv(3, claims(false, 2)).unwrap(); + let ways = |router: &Router| { + (router.ways(&broadcast).into_iter()) + .map(|way| (way.next, way.closest)) + .collect::>() + }; + assert_eq!(ways(&router), [(Next::Via(3), true)]); + assert_eq!(router.step(node(5), &broadcast), Some(Next::Via(3))); + router.recv(2, claims(false, 3)).unwrap(); + assert_eq!(ways(&router), [(Next::Via(2), true), (Next::Via(3), false)]); + assert_eq!(router.step(node(5), &broadcast), Some(Next::Via(2))); + } + /// Among claims of one Instance at one path, one exactly there outranks a broader one /// shown there, and among those a newer epoch outranks an older one however much /// cheaper the older is, or whether a route reaches the newer: among a node's @@ -5462,7 +5577,14 @@ mod tests { assert_eq!(router.pick(&room), Some(node(4)), "both marked: the lower Node ID"); assert_eq!(closest(&router), vec![(node(4), Next::Via(2), true)]); + // A route that starts brings the claims as they stand. router.recv(3, route(4, 3, 1)).unwrap(); + let claims = Msg::Claims { + origin: node(4), + changes: vec![(room.clone(), claim(run(0), 0))], + floor: 1, + }; + router.recv(3, claims).unwrap(); router.drain(); assert_eq!(router.changes().routes, BTreeSet::from([node(4)]), "an unmarked route"); assert_eq!(router.pick(&room), Some(node(4)), "the unmarked claim is closest"); diff --git a/rs/moq-net/src/route/tests/multicast.rs b/rs/moq-net/src/route/tests/multicast.rs index d714bc3555..84eb5deee3 100644 --- a/rs/moq-net/src/route/tests/multicast.rs +++ b/rs/moq-net/src/route/tests/multicast.rs @@ -914,6 +914,11 @@ const UNAVAILABLE: [(&str, &str); 5] = [ /// And where a claim moves to another node, the router forwards the new node's route /// before the old node's withdrawal at every hop, so some readers never see it offline /// and never start it again: fewer of each, by as many. +/// +/// And a request follows a claim only through a neighbour whose copy of the claimant's +/// claims holds it, where the prototype follows the route alone. While links flap, a +/// neighbour's route can arrive before its copy of the claims, and a request through it +/// would come back: fewer loops. #[test] fn the_router_gives_the_prototypes_answers() { let mut out = Vec::new(); @@ -934,6 +939,7 @@ fn the_router_gives_the_prototypes_answers() { "settle_rounds" => unavailable && b.rounds == a.rounds + 1, "starts" => unavailable && b.sent.starts < a.sent.starts, "offline" => unavailable && b.sent.offline < a.sent.offline, + "loops" if b.loops < a.loops => true, "loops" => unavailable && shape.name == "three_meshes_ring" && b.loops == a.loops + 1, _ => false, }; diff --git a/test/mesh-replay/script.toml b/test/mesh-replay/script.toml index c1cb3e938a..701e274ae8 100644 --- a/test/mesh-replay/script.toml +++ b/test/mesh-replay/script.toml @@ -54,10 +54,10 @@ kind = "starved" event = "roll" quest = "/quest/m2/multicast/roll-echo-stall.md" -# On lite-07 alone, a reader behind a reopened link or a crashed relay starves when its relay -# asked a neighbour whose own route had not returned yet. After the roll entry, so a roll's +# On lite-07 alone, a front that switched copies after a reopened link or a crashed relay can +# deliver a group only every 15-25s on an unchanged route. After the roll entry, so a roll's # starved reader stays with roll-echo-stall. `reopen.toml` reproduces it and lists nothing known. [[known]] check = "unfed" kind = "starved" -quest = "/quest/m2/multicast/reopen-starve.md" +quest = "/quest/m2/multicast/splice-stall.md"