diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index fd344f0455..673c08a48b 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -526,7 +526,8 @@ A receiver MUST close the session with a PROTOCOL_VIOLATION on a ROUTE whose Pat A SUBSCRIBE, FETCH, TRACK, TRACKS, or DATAGRAMS request names the routing node that announces the claim serving it. A subscriber that names none lets the publisher select one (see [Resolution](#resolution)), and a relay forwarding a request names the node it selected or was given. -A relay forwards a request naming a routing node only along its selected route to that node, or to the endpoint whose claim it announces when the request names the relay itself, and resets it with UNROUTABLE when it has neither. +A relay forwards a request naming a routing node only along a route to that node through a neighbour that sent it the claim, its selected route when that neighbour did, along its selected route to that node when it holds no claim of that node covering the path, or to the endpoint whose claim it announces when the request names the relay itself, and resets it with UNROUTABLE when it has none of these. +A route can arrive before the claims behind it, so while a relay routes to the named node and holds its claim available and eligible (see [Resolution](#resolution)) but no neighbour with a route there has sent it yet, it holds the request until one does or the claim stops being available and eligible, rather than reset a request for a claim it still shows available. Claims with the same Instance serve the same Broadcast, so a relay MAY move a live subscription between them, including onto a higher Epoch, continuing from the first frame the subscriber lacks, and a route change (a reconnect, a cheaper path, a draining session) is invisible to the subscriber. A relay MUST NOT move a subscription to a claim with a different Instance: it stays on its Instance's claims and ends with the last of them. diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md index 13b121627e..fe9429f30f 100644 --- a/quest/m2/multicast/README.md +++ b/quest/m2/multicast/README.md @@ -1,4 +1,4 @@ -# Cache-aware multicast routing +# [S] Cache-aware multicast routing ## Goal @@ -37,10 +37,11 @@ test covered on real sessions. It found two flaws, both now fixed: [moq-dev/moq#9154](https://github.com/moq-dev/moq/pull/9154) lets a front ride through a lost claim's debounce, so a reader resumes on a detour learned after its session closed, and a route moving behind one next hop no longer resends a -group, since a lite-07 range may start partway through one. The lite-07 cut -still waits on this epic's children, a flaw the mesh replay found, -[a lite-07 reader its relay refuses as unroutable and never re-asks](/quest/m2/multicast/splice-stall.md). -It does not wait on +group, since a lite-07 range may start partway through one. The mesh replay +then found a reader its relay refused as unroutable while still showing the +broadcast, which never asked again: a lite-07 hop now holds such a request +(maintainer, 2026-11-20), and a session closes its sources before it fails +their copies. The lite-07 cut does not wait on [warmth's port](/quest/m3/cache-routing/select.md): its wire joins lite-07 only if the port has landed and passed the mesh replay when the cut is made, and otherwise moves to the first wip version after it (maintainer, @@ -344,6 +345,19 @@ Lessons from the history that bind every child: - Nothing ships unexercised: a loop check that never fired cost 32 bounces per announce in production. +## Waiting + +- Maintainer: whether this epic is done now that its last child, + [splice-stall](https://github.com/moq-dev/moq/pull/9231), landed. Its Goal + reads met: the route layer shipped as lite-07's, the route algebra proves + it, and the bake-off measured it. Recommended: yes. Delete this README, + first moving the requirements and settled items that m3 quests cite by + number (cache-routing, route-health) into those quests, and remove every + link to it, including its entry in the Required list of + [the lite-07 cut](/quest/m2/lite07-finalize.md), which lifts that wait. An + agent cannot take this step: it decides whether the cut still waits on + routing. + ## Related - [Cache-aware routing](/quest/m3/cache-routing/README.md) - implements the warmth pick diff --git a/quest/m2/multicast/splice-stall.md b/quest/m2/multicast/splice-stall.md deleted file mode 100644 index f09ac73960..0000000000 --- a/quest/m2/multicast/splice-stall.md +++ /dev/null @@ -1,87 +0,0 @@ -# [M] A reader refused as unroutable reads again on lite-07 - -## Goal - -On lite-07, a reader whose relay still shows its broadcast keeps reading after -a link reopens or a relay crashes, as the release (lite-06) does. Today, about -one run in twenty, a relay answers a reader's live subscription with -Unroutable while it keeps announcing the broadcast unchanged. The reader's -track strands, waiting for a route change that never comes, and starves for -the rest of the run without an error. - -No reader starves in -`just test mesh-replay --mesh test/mesh-replay/ring4.toml --script test/mesh-replay/reopen.toml --broadcasts 32` -or in the nightly ring meshes, once `[[known]]` no longer names this quest in -`test/mesh-replay/script.toml`. - -## Plan - -Found by [the mesh replay](/test/mesh-replay/README.md). On main after -[roll-echo-stall](https://github.com/moq-dev/moq/pull/9174) and -[reopen-starve](https://github.com/moq-dev/moq/pull/9179), 19 of 20 -`reopen.toml` runs (ring4, 32 broadcasts, debug build) passed. In the failing -run, `b`'s reader followed `d/0` and `d/3` through `a`. When the `a`-`b` -session idle-timed out, relay `b` reset the reader's live subscriptions for -exactly those two paths with Unroutable, and never withdrew or updated `d/0`. -The reader's front strands a track refused as unroutable until its table moves -(`model/front.rs` `refuse`, `Track::stranded`, `unstrand`), so it never asked -again. The other 30 paths switched to the copy through `c`. That run had info -logs only. - -Decided 2026-11-20 by the maintainer: - -- ✅ The hop holds (rejected: the reader re-asks on a timer, which adds a - backstop timer and can loop on a refusal that stands). A hop never answers a - live subscription with Unroutable while it still shows that broadcast - available. Its front either waits for a route, as it already does for an - orphaned claim (`Front::waiting`), or changes its announcement so the reader - asks again. -- ✅ The lite-07 cut waits on this fix. - -What [moq-dev/moq#9219](https://github.com/moq-dev/moq/pull/9219) did: - -- Closed one way a hop broke that rule. A front reads whether a source is - closing from its broadcast, and takes a failure from a source still open as - the track's verdict. A lite session could fail the copies it carries, or a - query waiting on TRACK_INFO, before it closed the broadcasts they came from: - a front that saw a copy die that way asked the dying source again and ended - the track with its answer, which a relay passes on as Unroutable. The lite - subscriber now closes a source before failing its tracks with the session's - end (`TrackServe::close_if_ending`, `SubscriberDriver::abort`), so the front - holds the track for the next route. It reads the session's state, not the - error, since a stream's own reset can carry the same error. The IETF - subscriber is unchanged. `/verify-failover` models the window: a dead relay's - route still shows until it leaves the table (`Shows`, `Withdraw`). -- Added a debug line for every front step (`moq_net::model`), so a failing run - shows how a front ended a track. - -That race was not confirmed as this failure. On an unloaded host, 59 base runs -passed, as did 21 release runs of the fix that landed (and 24 debug runs of an -earlier, error-based fix it replaced), and no reader got Unroutable during the -single link reopen in any of them. Holding the drain open 2s after the transport closed -did not make the base fail either. Traced relays showed the source closing -before its copies end, the order the fix now guarantees. Debug runs on the CI host at load 70 or more starve -readers by load alone: sessions to the detours idle-time out too. - -Next steps: capture one failure with -`RUST_LOG=info,moq_net::model=debug`, on a host under real but not saturating -load (relays below a core each), and record per-relay CPU with it. Rerun -`reopen.toml` on a debug build until a reader gets Unroutable during the -single link reopen; that alone marks the bug, starving or not. In the failing -relay's log, find the front step that ended that path. Then write the test at -the lowest layer that shows it, fix it as decided, rerun `/verify-failover`, -and rerun the replay at least 20 times before dropping the `[[known]]` entries -in `test/mesh-replay/script.toml`. - -Pauses are not part of this quest. A debug build pauses about as long on -lite-06 as on lite-07 under the same load, and a release build the same on -both versions. - -Not a wire change. - -## Related - -- [reopen-starve](https://github.com/moq-dev/moq/pull/9179) - a request - refused before the neighbour's claims returned, and never asked again -- [roll-echo-stall](https://github.com/moq-dev/moq/pull/9174) - a time start a - shared copy never places diff --git a/rs/moq-net/src/route/node.rs b/rs/moq-net/src/route/node.rs index ff6560c2a5..3f5d9a0b0d 100644 --- a/rs/moq-net/src/route/node.rs +++ b/rs/moq-net/src/route/node.rs @@ -422,7 +422,8 @@ impl Node { /// Where a request for `broadcast` goes that the peer of the session `tag` names sent /// naming `named`, or no node: here, along a claim the origin serves, or `None` when /// nothing reaches it or the way would lead back to the peer. Waits while the session - /// it leaves on has yet to serve the claim. + /// it leaves on has yet to serve the claim, and while the node shows the claim available + /// but no neighbour's copy holds it yet. pub fn poll_forward( &self, tag: u64, @@ -1003,7 +1004,12 @@ impl State { return Some(None); }; let Some(next) = self.router.step(origin, broadcast) else { - return Some(None); + // Shown available with no way yet: hold the request rather than answer it + // unroutable, which nothing this node shows would ever make its sender retry. + return match self.router.awaits(origin, broadcast) { + true => None, + false => Some(None), + }; }; let (Next::Via(session) | Next::Local(session)) = next; let (path, instance) = broadcast; diff --git a/rs/moq-net/src/route/node/tests.rs b/rs/moq-net/src/route/node/tests.rs index 3bbc99c38d..3481241a38 100644 --- a/rs/moq-net/src/route/node/tests.rs +++ b/rs/moq-net/src/route/node/tests.rs @@ -710,3 +710,176 @@ fn a_neighbour_serves_a_claim_once_its_copy_holds_it() { lagging.flush(); assert_eq!(confirm(&lagging), 1); } + +/// The neighbour whose copy holds a claim leaves while another still routes to its node +/// with a copy that lacks it. The node keeps showing the claim available, so a request for +/// it waits for that copy rather than fail: nothing the node shows would change once the +/// copy arrives, so a request refused meanwhile would never be asked again. +#[test] +fn a_request_for_a_claim_shown_available_waits_for_a_copy_that_holds_it() { + let (node, _clock) = node(); + let room = cast("room"); + let far = id(9); + let route = |via: u64, cost| Msg::Route { + origin: far, + offer: Some(Offer { + path: [ + Hop { node: far, session: 0 }, + Hop { + node: id(via), + session: 1, + }, + ] + .into(), + draining: false, + priority: i8::MAX, + cost, + marked: false, + }), + }; + let claims = Msg::Claims { + origin: far, + changes: vec![(room.clone(), live(0))], + floor: 1, + }; + let viewer = viewer(&node, 5); + let lagging = node.connect(1, port(Role::Routing(id(7)))).unwrap(); + lagging.recv(Msg::Edited).unwrap(); + let ahead = node.connect(2, port(Role::Routing(id(8)))).unwrap(); + ahead.recv(Msg::Edited).unwrap(); + lagging.recv(route(7, 1)).unwrap(); + lagging.flush(); + ahead.recv(route(8, 2)).unwrap(); + ahead.recv(claims.clone()).unwrap(); + ahead.flush(); + assert_eq!(confirm(&ahead), 1); + assert_eq!(told(&viewer), [("room".to_string(), true)]); + + drop(ahead); + assert!(told(&viewer).is_empty(), "the claim still shows available"); + assert!(serves(&lagging).is_empty(), "its copy lacks the claim"); + let (flag, waiter) = flag(); + assert!(node.poll_forward(5, None, &room, &waiter).is_pending()); + assert!(node.poll_forward(5, Some(far), &room, &waiter).is_pending()); + + lagging.recv(claims).unwrap(); + lagging.flush(); + assert_eq!(confirm(&lagging), 1); + assert!(flag.woken(), "a way appeared"); + assert_eq!( + node.poll_forward(5, None, &room, &waiter), + Poll::Ready(Some(Via::Route { node: far })) + ); +} + +/// A request waiting for a way to a claim fails once the node no longer routes to the +/// claim's node, which shows the claim unavailable. +#[test] +fn a_request_waiting_for_a_way_fails_once_its_route_goes() { + let (node, _clock) = node(); + let room = cast("room"); + let far = id(9); + let route = |via: u64, cost| Msg::Route { + origin: far, + offer: Some(Offer { + path: [ + Hop { node: far, session: 0 }, + Hop { + node: id(via), + session: 1, + }, + ] + .into(), + draining: false, + priority: i8::MAX, + cost, + marked: false, + }), + }; + let viewer = viewer(&node, 5); + let lagging = node.connect(1, port(Role::Routing(id(7)))).unwrap(); + lagging.recv(Msg::Edited).unwrap(); + let ahead = node.connect(2, port(Role::Routing(id(8)))).unwrap(); + ahead.recv(Msg::Edited).unwrap(); + lagging.recv(route(7, 1)).unwrap(); + lagging.flush(); + ahead.recv(route(8, 2)).unwrap(); + ahead + .recv(Msg::Claims { + origin: far, + changes: vec![(room.clone(), live(0))], + floor: 1, + }) + .unwrap(); + ahead.flush(); + confirm(&ahead); + drop(ahead); + told(&viewer); + let (flag, waiter) = flag(); + assert!(node.poll_forward(5, None, &room, &waiter).is_pending()); + + lagging + .recv(Msg::Route { + origin: far, + offer: None, + }) + .unwrap(); + lagging.flush(); + assert!(flag.woken(), "the route went"); + assert_eq!(node.poll_forward(5, None, &room, &waiter), Poll::Ready(None)); + assert_eq!(node.poll_forward(5, Some(far), &room, &waiter), Poll::Ready(None)); +} + +/// A request waiting for a way to a claim fails once a claim of better precedence outranks +/// it, as the node then shows the broadcast by that claim, which its sender asks anew. +#[test] +fn a_request_waiting_for_a_way_fails_once_its_claim_is_outranked() { + let (node, _clock) = node(); + let room = cast("room"); + let far = id(9); + let route = |via: u64, cost| Msg::Route { + origin: far, + offer: Some(Offer { + path: [ + Hop { node: far, session: 0 }, + Hop { + node: id(via), + session: 1, + }, + ] + .into(), + draining: false, + priority: i8::MAX, + cost, + marked: false, + }), + }; + let viewer = viewer(&node, 5); + let publisher = publisher(&node, 6); + let lagging = node.connect(1, port(Role::Routing(id(7)))).unwrap(); + lagging.recv(Msg::Edited).unwrap(); + let ahead = node.connect(2, port(Role::Routing(id(8)))).unwrap(); + ahead.recv(Msg::Edited).unwrap(); + lagging.recv(route(7, 1)).unwrap(); + lagging.flush(); + ahead.recv(route(8, 2)).unwrap(); + ahead + .recv(Msg::Claims { + origin: far, + changes: vec![(room.clone(), live(0))], + floor: 1, + }) + .unwrap(); + ahead.flush(); + confirm(&ahead); + drop(ahead); + told(&viewer); + let (flag, waiter) = flag(); + assert!(node.poll_forward(5, Some(far), &room, &waiter).is_pending()); + + announce(&publisher, &room, live(1)); + confirm(&publisher); + assert!(flag.woken(), "a newer claim arrived"); + assert_eq!(told(&viewer), [("room".to_string(), true)], "shown by the newer claim"); + assert_eq!(node.poll_forward(5, Some(far), &room, &waiter), Poll::Ready(None)); +} diff --git a/rs/moq-net/src/route/relay.rs b/rs/moq-net/src/route/relay.rs index fb7752d8a4..b9f23abcd2 100644 --- a/rs/moq-net/src/route/relay.rs +++ b/rs/moq-net/src/route/relay.rs @@ -59,7 +59,9 @@ //! claimant too, and fails closed while that one is unavailable or lost. It leaves toward //! another node only through a neighbour whose own copy of that node's claims holds the //! claim: a route can arrive before the claims behind it, and a neighbour refuses a request -//! for a claim it has not heard. Among the claims +//! for a claim it has not heard. While none does, and the node still routes to that node and +//! knows the claim available, the request waits: the node shows the claim available, so it +//! never answers it unroutable. Among the claims //! of one Instance at one path, only the one shown closest to its own path, at the newest //! epoch among those, may serve, reachable or not, as `model::origin` ranks them; a //! route's rank chooses among those alone. A route whose cost a link would carry past the @@ -2538,6 +2540,30 @@ impl Router { } } + /// Whether a request for `broadcast` naming `origin`, which [`Self::step`] has nowhere + /// to send, waits for a way rather than fails: this node routes to `origin` and knows its + /// claim available and of the best precedence, so it shows the claim, but no neighbour's + /// copy holds it yet. A route arrives before the claims behind it, and a copy that held + /// the claim can leave with its session while this node's own copy stays, so nothing it + /// shows changes when a way appears: a request it refused now would never be asked again. + pub fn awaits(&self, origin: NodeId, (path, instance): &Broadcast) -> bool { + if self.is_self(origin) || !self.best.contains_key(&origin) { + return false; + } + let Some(known) = self.known.get(&origin) else { + return false; + }; + (ancestors(path).rev()) + .map(|above| (above, instance.clone())) + .find(|claim| known.items.contains_key(claim)) + .is_some_and(|claim| { + // Only a claim of the best precedence there is shown, as in `closest`. + let best = (self.claims_of(&claim)).map(|(_, claim)| precedence(&claim.item)).min(); + (self.claims_of(&claim)) + .any(|(by, claim)| by == origin && !claim.unavailable && best == Some(precedence(&claim.item))) + }) + } + /// Whether `via`'s copy of `origin`'s claims holds `broadcast` as this node knows it: /// available, and of the same precedence. A neighbour routes to `origin` before its claims /// reach it, and refuses a request for a claim it has not heard, or heard only unavailable diff --git a/test/mesh-replay/script.toml b/test/mesh-replay/script.toml index 4919b0359b..b2fde35d2e 100644 --- a/test/mesh-replay/script.toml +++ b/test/mesh-replay/script.toml @@ -44,24 +44,3 @@ check = "unfed" kind = "stuck" event = "roll" quest = "/quest/m3/release/cut.md" - -# On lite-07 alone, a relay can answer a reader's live subscription Unroutable after a reopened -# link or a crashed relay while still showing the broadcast, and the reader never asks again. -# Not on a roll, so a roll's starved reader still fails. `reopen.toml` lists nothing known. -[[known]] -check = "unfed" -kind = "starved" -event = "reopen-link" -quest = "/quest/m2/multicast/splice-stall.md" - -[[known]] -check = "unfed" -kind = "starved" -event = "reopen-relay" -quest = "/quest/m2/multicast/splice-stall.md" - -[[known]] -check = "unfed" -kind = "starved" -event = "kill" -quest = "/quest/m2/multicast/splice-stall.md"