diff --git a/quest/m2/README.md b/quest/m2/README.md index 7fb904262da..44f31144ac8 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -9,6 +9,11 @@ complete it before cutting main into release. ## Plan +Decided 2026-11-15 (maintainer, /quest-plan on the whole route layer): ✅ the +cut waits on [cache-aware multicast](/quest/m2/multicast/README.md), which +replaces lite-07's route layer before it freezes (rejected: lite-08). The +routes chain stays the critical path, now through that epic. + Decided 2026-11-05 (maintainer, milestone re-cut): m2 holds the breaking half of what was "everything before the cut" (2026-10-20). Breaking covers every published surface, as the root [questline](/quest/README.md) defines: the @@ -222,10 +227,11 @@ maintainer asked that what remains be unblocked or punted, and picked these on [moq-dev/moq#8768](https://github.com/moq-dev/moq/pull/8768)). m2 keeps only what would need a breaking release had it landed right after the cut: -- ✅ Left m2: [the request loop](/quest/m2/cluster-routes/request-loop.md) +- ✅ Left m2: [the request loop](/quest/m2/multicast/requests.md) to m3's cluster-routing epic (a fixture and a fix, no wire), until it - returned to m2's cluster-routes epic on 2026-11-15 (maintainer, on - [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)); + returned to m2 on 2026-11-15 (maintainer, on + [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)) and became + the request quest of [cache-aware multicast](/quest/m2/multicast/README.md); [media consumers](/quest/m3/time-addressing-consumers.md) to m3 (its API breaks ride the model; dropping hang's markers is tolerated by readers); [moq-transport time addressing](/quest/m4/ietf-time-addressing.md) to m4 diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index f8dc4710b0a..f8b0b7ea5a3 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -82,6 +82,8 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 30. ✅ Decided 2026-11-12 (maintainer, /quest-audit before the cut), after a survey of BGP, EIGRP, and their successors against item 27: most of the path rule already matches that prior art (AS_PATH loop detection, root-cause withdrawal, graceful restart, End-of-RIB, outbound route filtering, and item 29's Only-To-Customer flag). Three gaps are taken, all in [the switch](https://github.com/moq-dev/moq/pull/8731) and [acceptance](/quest/m2/cluster-routes/acceptance.md): **a dead-hop message on a scoped session applies only to the routes heard on that session and casts no doubt**, since in BGP a neighbour withdraws only what it sent, and a scoped peer (a federated relay, later a browser) could otherwise name a core hop it only read in a path and drop healthy routes outside its scope; **holds are bounded in time**, a neighbour's missing answer counting as nothing after a deadline, and a session whose announce stream has stopped draining is closed, which avoids EIGRP's stuck-in-active and BGP's zombie routes (RFC 9687), and is safe because a hold protects presence, not loop freedom; and **three tests before any design change**: neighbours noticing a dead relay up to the 3 s keep-alive apart (to size the doubt timer, measured in hops of delay today), a three-relay priority dispute wheel (RFC 4264's wedgies, since priority ranks above cost), and a primary that fails and returns. No wire bytes change; the first defines what the dead-hop message means. Rejected: declaring the link relation in SETUP and refusing leaked routes on receipt (RFC 9234), since the accepting side already knows the relation from its grant and it adds wire for a misconfiguration; MRAI and route-flap damping, which exist because BGP delays announcements; link state; per-hop signatures. +31. Reopened 2026-11-15 (maintainer, /quest-plan on the whole route layer): ✅ **this rule is the baseline, and [cache-aware multicast](/quest/m2/multicast/README.md) replaces it on lite-07-wip before the cut.** An audit of what is built found requirements the path rule does not meet: its exhaustive checking stops at 3 relays and rests on a timer that outlasts every stale offer, route state follows publishers, nothing prefers a warm copy, and a stalled route is never left. Superseded there: item 18 (every publisher an origin node), item 19 (ROUTE_HOLD), item 26's route per scope, item 27's doubt and remembered dead hops, the Route Key and qualified Node IDs of the contract below, and item 14. Standing: the path with its self-naming rule and the withdrawal that names a dead hop (item 27), revisions and floors, one announce stream per side with its interest set (items 21 to 24), the sticky down-only flag (item 29), and drain, then priority, then cost. This epic's children still land: they are what the [bake-off](/quest/m2/multicast/bakeoff.md) measures against. Rejected: repairing this rule in place; freezing lite-07 with it and redesigning in lite-08. + The route layer's core landed in [moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676): a crate-private, sans-IO `Router` in `rs/moq-net/src/route/`, generic over the diff --git a/quest/m2/cluster-routes/acceptance.md b/quest/m2/cluster-routes/acceptance.md index dbaf6f6cae1..003f627db02 100644 --- a/quest/m2/cluster-routes/acceptance.md +++ b/quest/m2/cluster-routes/acceptance.md @@ -27,6 +27,12 @@ mount continuity (`mount.rs`), and operation-count regressions (`route/tests/cost.rs`, `route/node/tests.rs`). They found the flaws the quests under Required fix; the tests that pin each flaw say so in their docs. +Reordered 2026-11-15 (maintainer, /quest-plan on +[cache-aware multicast](/quest/m2/multicast/README.md)): the retirement below +waits for [the switch](/quest/m2/multicast/switch.md), which rewrites these +fixtures' assertions for the new rule and deletes the Required quests it +makes moot. + What remains: - Retire the quarantine, the dedup, and the hold once the Required quests @@ -48,6 +54,7 @@ What remains: ## Required +- [The switch](/quest/m2/multicast/switch.md) - the rule whose fixtures replace the mitigations, once the new route layer is on the wire - [Held after close](/quest/m2/cluster-routes/held-after-close.md) - a relay whose own session closed shows its held claims unavailable, not offline - [Doubt holds](/quest/m2/cluster-routes/doubt-holds.md) - a relay keeps holding while doubt alone keeps it off its routes - [Route fan-out](/quest/m2/cluster-routes/route-fanout.md) - a route change costs only the streams and broadcasts it touches diff --git a/quest/m2/cluster-routes/doubt-holds.md b/quest/m2/cluster-routes/doubt-holds.md index 2898d468152..bb494ab31df 100644 --- a/quest/m2/cluster-routes/doubt-holds.md +++ b/quest/m2/cluster-routes/doubt-holds.md @@ -8,6 +8,13 @@ go while its neighbours answer that they have nothing. ## Plan +Reordered 2026-11-15 (maintainer, /quest-plan on [cache-aware +multicast](/quest/m2/multicast/README.md)): this waits on the +[bake-off](/quest/m2/multicast/bakeoff.md). The new rule has neither holds nor +doubt, so nothing would be left for this to fix. If the new rule goes ahead, +[the switch](/quest/m2/multicast/switch.md) deletes this quest with the +fixtures that pin it; if it does not, this starts as written. + Kept 2026-11-15 (maintainer, /quest-complete on [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ keep it in m2 (rejected: drop; moving it to m3). @@ -34,6 +41,10 @@ neighbour that holds only because of doubt should answer with ROUTE_HOLD, not nothing. Check the rule in `RouteHandoff.tla` with the `verify-route` skill before changing `rs/moq-net/src/route/path.rs`. +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - decides whether the hold and the doubt this fixes remain + ## Related - [Routes and announces](/quest/m2/cluster-routes/README.md) - Plan item 27 and the hold rule diff --git a/quest/m2/cluster-routes/doubt-keepalive.md b/quest/m2/cluster-routes/doubt-keepalive.md index 859b9212d17..be6a172a659 100644 --- a/quest/m2/cluster-routes/doubt-keepalive.md +++ b/quest/m2/cluster-routes/doubt-keepalive.md @@ -7,6 +7,20 @@ apart its neighbours notice the death within the keep-alive. ## Plan +Reordered 2026-11-15 (maintainer, /quest-plan on [cache-aware +multicast](/quest/m2/multicast/README.md)): this waits on the +[bake-off](/quest/m2/multicast/bakeoff.md). The maintainer deferred how the +doubt covers a silent death on 2026-11-15, "not yet sure about the doubt +design", and decided the same day that evidence replaces the doubt. If the new +rule goes ahead, [the switch](/quest/m2/multicast/switch.md) deletes this +quest with the fixtures that pin it; if it does not, this starts as written. + +If the doubt stays, the decision returns to the maintainer: (a) the doubt +lasts at least the configured idle timeout, so it covers the spread (the +recommendation: no wire change, and a relay already knows its idle timeout); +(b) relay keep-alives shorten below the doubt, which costs traffic on every +relay link; (c) accept the window and keep the test pinning it. + Kept 2026-11-15 (maintainer, /quest-complete on [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ keep it in m2 (rejected: drop; moving it to m3). @@ -21,16 +35,9 @@ dead `X` at 1 s, holds again at 1.5 s, and moves onto `R3`'s route through at 3 s. Readers see each move as available for a moment, and a request in that window goes to a dead relay. A 1 s doubt does not cover a 3 s spread. -## Waiting +## Required -- Maintainer: how the doubt covers a silent death. Options: (a) the doubt - lasts at least the configured idle timeout, so it covers the spread (the - recommendation: no wire change, and a relay already knows its idle - timeout); (b) relay keep-alives shorten below the doubt, which costs - traffic on every relay link; (c) accept the window and keep the test - pinning it. Check with `/quest-grill` or the PR that picks it up. The - maintainer deferred this decision on 2026-11-15, not yet sure about the - doubt design. +- [The bake-off](/quest/m2/multicast/bakeoff.md) - decides whether the doubt this sizes remains ## Related diff --git a/quest/m2/cluster-routes/held-after-close.md b/quest/m2/cluster-routes/held-after-close.md index 829a4336926..561e0d9edf0 100644 --- a/quest/m2/cluster-routes/held-after-close.md +++ b/quest/m2/cluster-routes/held-after-close.md @@ -9,6 +9,14 @@ route recovers. ## Plan +Reordered 2026-11-15 (maintainer, /quest-plan on [cache-aware +multicast](/quest/m2/multicast/README.md)): this waits on the +[bake-off](/quest/m2/multicast/bakeoff.md). The new rule has no hold: a reader +debounces unavailable to offline locally, which answers the same symptom. If +the new rule goes ahead, [the switch](/quest/m2/multicast/switch.md) deletes +this quest with the fixtures that pin it; if it does not, this starts as +written. + Kept 2026-11-15 (maintainer, /quest-complete on [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ keep it in m2 (rejected: drop; moving it to m3). @@ -32,6 +40,10 @@ move it, unavailable, onto a stream that still hears the held route, or show it through the node itself, before the closing stream withdraws it. Turn the three pinned fixtures into the contract's held-then-available sequence. +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - decides whether the hold this shows remains + ## Related - [Routes and announces](/quest/m2/cluster-routes/README.md) - the hold rule and "Presence and availability are separate" diff --git a/quest/m2/cluster-routes/request-loop.md b/quest/m2/cluster-routes/request-loop.md deleted file mode 100644 index abf8f591f7d..00000000000 --- a/quest/m2/cluster-routes/request-loop.md +++ /dev/null @@ -1,64 +0,0 @@ -# [M] A request loop heals once routes settle - -## Goal - -A subscription that loops between relays while routes settle reaches the -publisher once the next hops stop looping, instead of hanging silently or -ending with `Unroutable` while its relay shows the broadcast available. -Under the path rule, selected next hops can loop for a moment -([epic](/quest/m2/cluster-routes/README.md) Plan item 27). A request -forwarded around that loop meets a relay already requesting the same track -upstream, and aggregation joins it to that request. Nothing errors, so a -pending request that never follows its route's change waits forever with -no data and no failure. - -## Plan - -Decided 2026-11-15 (maintainer, on -[moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ move it -back to m2, under the cluster-routes epic (rejected: keep it in m3). Relays -that show a broadcast available while ending readers' subscriptions should -block the milestone. - -Moved from m2 to m3 2026-11-14 (the m2 triage before the cut, under -/quest-afk, for the maintainer's review): ✅ m3, under the cluster-routing -epic (rejected: m2). A fixture and, if it hangs, a fix to where a relay's -pending request goes, with no wire field (the hop count was rejected below), -so it lands after the cut as a bug fix in shipped code. - -Decided 2026-11-13 (maintainer, /quest-plan on the BGP failure modes): - -- ✅ A fixture on real relays, with no TLA+ (rejected: adding pending - requests and aggregation to `RouteHandoff.tla`). The model's - `RequestsLoopFree` already checks next hops; this checks that the - relay's request state follows them. -- ✅ If the fixture hangs, a relay's pending upstream request follows its - route when the selected next hop changes, even before the request is - answered, so the loop clears with the routes (rejected: a hop count on - every request that fails one at the limit, which adds a field to SUBSCRIBE, - FETCH, and the other requests and turns a loop that would heal into an - error the subscriber sees). Epic Plan item 27 already rejected keeping a - request from going back through the relay that sent it. - -Found again by [acceptance](/quest/m2/cluster-routes/acceptance.md) once -[moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) merged: -`route_failover::live_mesh_link_failure_keeps_presence` cuts all but one of a -publishing relay's links on a 34-relay mesh, and 3 readers on relays that -stayed available end with `Unroutable` instead of hanging, all for one -publisher. Each such relay moved onto a neighbour's direct route that the -neighbour's own close made stale; the replaced doubt rule held them and -ended none. The test pins those readers: flip it once a pending request -follows its route and they read on. - -The fixture runs real lite-07 relays with mocked time. Form the loop the -churn sweeps found (two unrelated failures at once, so two relays briefly -route a publisher through each other), subscribe at both while it lasts, -then let the routes settle. Both subscribers receive the next group, and no -upstream request is left pending. Cover a request answered before the loop -and one still unanswered when it forms. Check which relay state carries the -pending request (the front in `rs/moq-net/src/model/front.rs`, and how -`route/node.rs` `home`s it) before choosing where the fix goes. - -## Related - -- [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) - its doubt rule lets a publisher leaving three or more relays at once loop a request until the news arrives diff --git a/quest/m2/cluster-routes/route-fanout.md b/quest/m2/cluster-routes/route-fanout.md index 09a134c7092..3b474bce219 100644 --- a/quest/m2/cluster-routes/route-fanout.md +++ b/quest/m2/cluster-routes/route-fanout.md @@ -8,6 +8,15 @@ remembered for sessions nothing was learned on. ## Plan +Reordered 2026-11-15 (maintainer, /quest-plan on [cache-aware +multicast](/quest/m2/multicast/README.md)): this waits on the +[bake-off](/quest/m2/multicast/bakeoff.md). Its first and third findings are +in `path::Router`'s indexes and its doubts and dead hops, which the new router +replaces; the second, per-broadcast rewrites in `route/node.rs`, the +[router](/quest/m2/multicast/router.md) takes as a requirement. If the new +rule goes ahead, [the switch](/quest/m2/multicast/switch.md) deletes this +quest with the fixtures that pin it; if it does not, this starts as written. + Kept 2026-11-15 (maintainer, /quest-complete on [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ keep it in m2 (rejected: drop; moving it to m3). @@ -43,6 +52,10 @@ once fixed: Build on [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734), which also edits `path.rs`. +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - decides whether the path router these findings are in remains + ## Related - [Route wakes](/quest/m4/route-wakes.md) - origin-side wakes per route change diff --git a/quest/m2/lite07-finalize.md b/quest/m2/lite07-finalize.md index 8f1c89affa5..858eac6c026 100644 --- a/quest/m2/lite07-finalize.md +++ b/quest/m2/lite07-finalize.md @@ -129,7 +129,7 @@ maintainer picked these on [moq-dev/moq#8768](https://github.com/moq-dev/moq/pull/8768)), applying the maintainer's rule that m2 keeps only what would need a breaking release had it landed right after the cut: -- ✅ [The request loop](/quest/m2/cluster-routes/request-loop.md) moved +- ✅ [The request loop](/quest/m2/multicast/requests.md) moved to m3 and no longer gates this: a fixture and a fix with no wire change. It returned to m2 on 2026-11-15 and still does not gate this. - ✅ [Route layer acceptance](/quest/m2/cluster-routes/acceptance.md) stays @@ -155,8 +155,18 @@ would need a breaking release had it landed right after the cut: both publishers already name a Group Stream the subscriber stopped in SUBSCRIBE_DROP, which the draft now states, even after a reliable reset. +Decided 2026-11-15 (maintainer, /quest-plan on the whole route layer): ✅ +lite-07 carries the redesigned route layer of +[cache-aware multicast](/quest/m2/multicast/README.md), and the cut waits +for it (rejected: freezing lite-07 with the path rule and redesigning in +lite-08; freezing lite-07 with no route layer). That redesign removes the +Route Key Parameter, so the 2026-11-10 wait on its symbolic model and the +route key lane's early signal above end when +[the switch](/quest/m2/multicast/switch.md) lands. + ## Required +- [Cache-aware multicast](/quest/m2/multicast/README.md) - lite-07 carries the redesigned route layer: routes to forwarding nodes, no holds or doubt, claims crossing trust boundaries by a chain, and no Route Key - [Routes and announces](/quest/m2/cluster-routes/README.md) - lite-07 carries the route layer: ROUTE per origin node and path-less ANNOUNCE, with the native hop list gone (only the temporary lite-06 compatibility chain remains) - [Lite-07 by start](/quest/m2/time-addressing/lite07.md) - lite-07 carries group starts, previous-start links, GROUP_END, and time bounds instead of sequence numbers - [@moq/net by start](/quest/m2/time-addressing/js.md) - JS speaks the same lite-07 wire diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md new file mode 100644 index 00000000000..a9c81669e2e --- /dev/null +++ b/quest/m2/multicast/README.md @@ -0,0 +1,203 @@ +# Cache-aware multicast routing + +## Goal + +moq-net gets one routing design, proven and measured before it replaces +lite-07's route layer. A subscription joins the nearest node already carrying +the same Instance, so no link carries a group twice, and moves between routes +of equal Instance without the reader noticing. State and messages grow with +forwarding nodes and demanded broadcasts, not publishers. There is no leader, +global view, or single point of failure, on CDN tiers and on a self-forming +mesh alike. + +Non-goals: choosing, dialing, and pricing links from measured quality, which +is Kixel's; browsers as transit; stitching different Instances together; +moq-transport routing parity beyond each node using its nearest publisher; +and a moq-transport node between two of our relays. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan), after an audit of the lite-07 +route layer as built, the history of about thirty routing attempts since +2023, and a survey of the literature. +[Routes and announces](/quest/m2/cluster-routes/README.md) holds the rule +this replaces and its measurements. + +The maintainer's requirements, which every child is judged against: + +1. A subscription resumes only onto equal identity. +2. A publisher may hold two connections (a GOAWAY handover, redundant + ingest), and routes switch between them seamlessly. Both serve one + Instance, so redundant encoders are synchronized. +3. Some connections do not proxy: an edge passes its clients to its cores + and its cores' routes to its clients, never one core's routes to another, + and a leaf, such as a relay on a camera, advertises only what its host + serves. +4. The specification stays simple, reusing messages and concepts. +5. Traffic grows as little as possible with the cluster. +6. No single point of failure. +7. It is proven to work. +8. Routing prefers warm nodes: a route has a cold cost and a warm cost, and + routes coalesce onto a node that already has demand for a broadcast, so + no link carries the same data twice. A CDN join stops at no pop without + demand, and drones with an uplink each settle on one that reaches the CDN. +9. Messages scale with the cluster: streams of messages, never one blob. +10. One stream carries what must stay ordered. +11. moq-transport is supported to a degree: identity does not carry, but two + publishers announcing one name on opposite sides of the network work. A + moq-transport node is always an endpoint. +12. No global view. With competing claims each node uses its closest, and + nothing elects a leader. +13. A secondary route may carry metadata only, so a stalled primary is found + without waiting on QUIC keep-alives, where a node opts in. +14. No approach is assumed: earlier attempts are evidence, not constraints. + +What the audit found of today's layer: 6 and 12 hold, and 1 holds with no +model covering it; 4, 8, and 13 do not hold (16 message types on the announce +stream and about 57 concepts against lite-06's 5 messages, no cache term in +any ranking, and a payload stall on live sessions that is never recovered); +the rest hold in part. Exhaustive checking stops at 3 relays and one session +close, safety only, and rests on a timer that outlasts every stale offer, +which a 1 s doubt does not when a dead relay's neighbours notice up to 3 s +apart. Route state and churn follow publishers. + +Settled: + +1. ✅ **Replace, not repair.** Today's ROUTE, DEAD, hold, and doubt wire is a + candidate like any other (rejected: keeping the path router as the base + and adding cache awareness and the metadata secondary on top, which leaves + requirement 4 unmet and state following publishers; moq also owning link + formation and pricing for the mesh). +2. ✅ **lite-07 carries it, and the cut waits** (rejected: lite-08, which + publishes a wire we expect to replace and makes two route layers + interoperate for a release; lite-07 without a route layer, a large strip + that leaves lite-07 clusters with lite-06's hunting). The route stream and + the acceptance fixtures still land: they are the baseline the + [bake-off](/quest/m2/multicast/bakeoff.md) measures against, and one + announce stream per side is wanted by any design. The quests that only + polish holds, doubt, or the path router's indexes wait on the bake-off. +3. ✅ **A routing node is a node that forwards and is trusted to; claims, + not routes, cross a trust boundary.** Every CDN relay and every mesh + device routes under its own Node ID. An endpoint that never forwards (an + app, an encoder, a browser) is local to the relay it connects to, which + announces its broadcasts under the relay's Node ID. Meshes sharing a fleet + credential are one routing domain. Across a scoped grant the receiving + node announces the claim again under its own Node ID with a chain of the + nodes that did so, and refuses a claim whose chain names it, which is the + Lite-06 Chain made general. The maintainer first answered "we need every + publisher so meshes can communicate with other meshes"; the chain covers + that across any number of scoped hops, where a qualified identity crosses + one. Redundant ingest is two relays claiming one Instance. This removes + the Route Key Parameter with its proof, qualified Node IDs, and routes per + prefix, and reverses Plan items 18 and 26 of + [Routes and announces](/quest/m2/cluster-routes/README.md) (rejected: + every publisher an origin node, as today; measuring both, which doubles + the prototype). +4. ✅ **Evidence, not timers.** The route layer is a plain path vector with + the dead-hop withdrawal as an accelerator: no ROUTE_HOLD, no doubt, and no + dead hop remembered past the routes that name it. Routing state is a + hint. A request is confirmed only once its upstream is, data is identified + by Instance, track, group, and frame, and a stale or stalled route is left + on evidence ([requests](/quest/m2/multicast/requests.md), + [route health](/quest/m3/route-health/README.md)). What a hold protected, + a claim flickering while routes settle, becomes a reader's local debounce + from unavailable to offline, never on the wire. This reverses items 19 and + 27's doubt (rejected: keeping holds and doubt with the doubt tied to the + idle timeout; a doubt lifted by an epoch the surviving node floods). +5. ✅ **Three ways to find a warm copy are measured before one is picked** + ([warmth](/quest/m2/multicast/warmth.md)): joins that stop at the first + node carrying the Instance and nothing more; that plus a neighbour saying + on each SUBSCRIBE answer whether it carries the Instance and at what cost; + and carrying nodes advertising it. One constraint binds all three: a warm + cost is strictly positive and summed from the source along the path the + data takes. A warm discount lets two warm relays prefer each other, which + happened four times here. +6. ✅ **One mark per link.** A route learned over a marked link is never sent + over another marked link, as `upstream` works today. An edge marks its + core links and a leaf marks every relay link (rejected: a strict mark that + sends only what the node originates, which ends a LAN peer publishing + through a neighbour whose uplink survived; both switches). +7. ✅ **The proof is mechanized, with TLA+ and sweeps beside it** + ([proof](/quest/m2/multicast/proof.md); rejected: a written proof with + property tests, the kind of argument that passed review here and failed + for the echo, the deferred echo, and belief by distance; TLA+ and sweeps + alone, which say nothing about an arbitrary topology). +8. Carried from the route-health epic deleted 2026-11-09 + ([moq-dev/moq#8423](https://github.com/moq-dev/moq/pull/8423)), whose + decisions the maintainer made 2026-10-13 and 2026-11-02 and nothing + reversed: metadata-only delivery is a mode of SUBSCRIBE, never its own + message; a stall is found by comparing the two routes, never by silence; + and a demoted route never takes a track back while the working one + delivers. + +Decided 2026-11-15 (maintainer, answering the requirements the plan's review +flagged as costly, on +[moq-dev/moq#8873](https://github.com/moq-dev/moq/pull/8873)): + +9. ✅ **Warm means demand, and two cases define requirement 8.** A node is + warm for an Instance while something downstream of it asks for it, not + because groups sit in its cache: "downstream demand instead of a yes/no + cache state". Pop skipping: a join stops at no pop without demand. Drones + that "self-elect who is the one who reaches out to the CDN": with an + uplink each, "the warm cost to a drone beats the cold cost to the CDN" + and one uplink carries the broadcast (rejected: narrowing 8 to joins that + stop at a carrier already on the path, which delivers neither case). + [Warmth](/quest/m2/multicast/warmth.md) measures both. +10. ✅ **A moq-transport session is an endpoint, never transit.** Its wire + has no path, chain, or Instance, so a moq-transport relay announcing + back a name it learned from us cannot be told from a nearer publisher. + That is unsupported: it must cost bounded work and touch no other + broadcast ([requests](/quest/m2/multicast/requests.md)), and nothing + detects it (rejected: ignoring a claim for a name already announced to + that session, which also hides a real second publisher behind it). +11. ✅ **Redundant ingest is one Instance from synchronized encoders.** Two + connections claiming one Instance serve the same frames under the same + group and frame numbers, and a publisher that restarts hard mints a new + Instance, as the draft already requires. Nothing checks it: a pair that + breaks it hurts its own broadcast. Encoders that are not synchronized + use two Instances, and a move between them is a restart the reader sees; + our importers are that case, since same-epoch importers were dropped + 2026-10-19. moq-transport never gets redundant ingest: it has no way to + flag an encoder restart, so every claim over it is a minted Instance. +12. ✅ **The metadata secondary is opt in** + ([route health](/quest/m3/route-health/README.md)): a mesh on links that + stall wants it, and the CDN's own tiers run without one (rejected: a + standby wherever a second route exists, which about doubles a track's + subscription state across the cluster). + +The shape those decisions give, for the children to confirm or improve; each +departure is a decision to list in its PR: + +- **Identity.** A path and an Instance name content, and a request names the + Instance. A relay holds one upstream subscription per Instance and track, + whichever origin node its downstream requests named, so a join stops at the + first node already carrying it. +- **Reachability.** One route per forwarding node: its path, drain, priority, + and cost. A node never uses a route whose path names it. Rank is not + draining, then the lowest priority on the path, then the summed cost, each + link adding at least 1, so extending a route never improves it. +- **Claims.** A prefix, Instance, Epoch, and cost, announced by a forwarding + node for itself and its local endpoints, ordered by the node's revision and + certified by floors as today, which measured no revival and no false END. +- **Boundaries.** The chain of item 3. +- **Requests.** A request names the origin node its first hop chose, so + relays that rank differently never loop it, and fails closed. + +Lessons from the history that bind every child: + +- A fact carries its own order: a revision, a path, a withdrawal that names + its cause. Holds, quarantines, hold-downs, and split horizon never stopped + a stale copy reviving a withdrawn fact. +- Every input to a rank travels in the offer and never improves when a route + is extended. An input that depends on demand needs its own argument. +- Each side says everything that must stay ordered on one stream. +- A failure costs messages per session or per forwarding node, never per + broadcast or per publisher behind it. +- Nothing ships unexercised: a loop check that never fired cost 32 bounces + per announce in production. + +## Related + +- [Routes and announces](/quest/m2/cluster-routes/README.md) - the rule this replaces, its contract, and its measurements +- [Cache-aware routing](/quest/m3/cache-routing/README.md) - implements the warmth pick +- [Route health](/quest/m3/route-health/README.md) - the metadata-only secondary diff --git a/quest/m2/multicast/accept.md b/quest/m2/multicast/accept.md new file mode 100644 index 00000000000..95c220c747e --- /dev/null +++ b/quest/m2/multicast/accept.md @@ -0,0 +1,48 @@ +# [L] The cases no test covers run on real sessions + +## Goal + +On real lite-07 sessions with mocked time, the switched route layer passes +the cases the 2026-11-15 audit found neither a model nor a test for, and the +PR reports the measured interruption for each kind of failure: + +- two publishers claiming one path from opposite sides of a mesh, where + each relay uses its nearest and none loops; +- one publisher ingesting on two relays under one Instance, losing either, + with readers across the mesh reading on, and a pair under two Instances, + where the move is a restart and nothing is spliced; +- a publisher handed over by GOAWAY to another relay, with no group missed + or repeated; +- moq-transport and lite-06 publishers through the route layer, including + two announcing one namespace on different relays, each a minted Instance + that no reader is moved between mid-stream; +- three meshes joined by scoped links, in a chain and in a ring, where no + claim circles and a claim ended at its source ends everywhere; +- an edge that marks its core links and a leaf that marks every link, each + passing only what requirement 3 allows; +- a relay that dies silently while its neighbours notice seconds apart; +- failures at the same instant on a mesh no exhaustive check reaches; +- and no group crossing one link twice for one subscription. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): the redesign is accepted on +the implementation before lite-07 freezes, as the maintainer decided for the +path rule on 2026-11-14. + +[Route layer acceptance](/quest/m2/cluster-routes/acceptance.md) owns the +fixtures that exist, which the [switch](/quest/m2/multicast/switch.md) +rewrites for the new rule; this adds the cases above beside them in +`rs/moq-net/tests/integration/`. Impair links with bufferbloat, not random +loss. Where a case needs `js/net`, run it in `just test interop --all`. + +A case that fails is a flaw to fix on lite-07-wip, in the quest that owns +the code, before the cut. + +Public API: none. Wire: none expected. + +## Required + +- [The switch](/quest/m2/multicast/switch.md) - the sessions these cases run on +- [Requests](/quest/m2/multicast/requests.md) - a request fails closed and follows its Instance +- [The proof](/quest/m2/multicast/proof.md) - what these cases need not show diff --git a/quest/m2/multicast/bakeoff.md b/quest/m2/multicast/bakeoff.md new file mode 100644 index 00000000000..c6350036dd9 --- /dev/null +++ b/quest/m2/multicast/bakeoff.md @@ -0,0 +1,90 @@ +# [XL] The new route rule is measured against today's + +## Goal + +A test-only prototype of the [epic](/quest/m2/multicast/README.md)'s route +rule runs beside `route::path::Router` on the same meshes and failure +schedules, and the PR reports for both what each requirement costs: messages +and retained state as forwarding nodes, publishers, broadcasts, and links +grow; the time to a settled route and the time a reader goes unfed after each +kind of failure; and every loop, stale route, false END, and revival seen. +The maintainer picks from those numbers whether the rule goes ahead as +decided and with what changes, and the PR amends the +[router](/quest/m2/multicast/router.md), +[switch](/quest/m2/multicast/switch.md), and +[proof](/quest/m2/multicast/proof.md) quests with what it learned. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): the rule's direction is settled +in the epic (items 3, 4, and 6); this measures it before anything is built on +it, as [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533) measured +paths against Babel. + +Build on `rs/moq-net/src/route/tests/`: `Sim` drives a router over a mesh +with messages held and delivered out of step, `compare.rs` ran two routers +over one schedule, and `search.rs` walks every delivery order on a small +mesh. The prototype stays under `tests/`; nothing ships from this PR. + +The rule to prototype: + +- A route per forwarding node, ranked not draining, then the lowest priority + on its path, then cost. A node never uses a route whose path names it. +- A closed session is named once per neighbour, and every node drops every + route through that hop. No moved list, no hold, no doubt, and a dead hop + is forgotten once no heard route names it. +- An endpoint's claims belong to the relay it connects to. A claim crossing + a scoped grant is announced again with the chain of the epic's item 3. +- One mark per link, as `upstream` is today. +- A request is confirmed only once its upstream is, and one fed around a + loop reports no success. Count what that costs readers: this is the + accepted price of having no timer. + +Shapes, each at more than one size: + +- CDN tiers: edges on two cores per region with the cores meshed, up to at + least 32 relays and 123 links at diameter 4. +- Meshes: a ring with log N chords, a sparse random mesh, a clique, and a + mesh with leaves that mark every link. +- Two meshes joined by one scoped link and by two, and three meshes in a + chain and in a ring, where the chain must stop a claim circling. +- One broadcast per publisher, the case where routes per publisher helped + least. + +Failures: a clean close; a silent death its neighbours notice up to the idle +timeout apart; a relay with many neighbours dying; a link that flaps faster +than any timer; a publisher on two relays losing one, then both; a publisher +moving; a GOAWAY drain; a partition that heals; and unrelated failures at the +same instant. + +Today's numbers, to beat or to explain: + +| Case | Today | +| --- | --- | +| A relay with several neighbours dies, 24 relays | 124 route messages, no loop | +| A link closes in a clique, one publisher behind it, then eight | 44, then 198 messages | +| A detour, settled | 12 hops of delay, 6.2 avoiding the far end | +| A publisher moves to another relay, clique of 24 | 1,100 ms dark | +| Churn, 120,000 steps | 32 steps with a route loop | +| A silent death noticed over 3 s | three relays each move onto a dead route twice | +| A link comes up on a 32-relay full mesh | 592 route messages | +| One route change beside 1,000 unrelated streams | 1.09 ms, against 9.8 µs beside 10 | + +The known risk is the reason doubt exists. Paths with no dead-hop message +hunted for 65,219 messages when a publisher left a clique of 12, and with the +dead hop but no doubt, selected routes looped when a relay with several +neighbours died. The epic accepts a loop among selected routes while +requests fail closed; show what it costs in messages and unfed readers. If +that is worse than today, try evidence before any timer: for one, rank every +route through the far end of a closed session as draining until that node's +own word about the session arrives, which reuses the drain and needs no +clock. A timer in the rule goes back to the maintainer. + +Keep the gate fast: the fixtures that pin a property run in the merge gate, +and the long sweeps run nightly from `test/route-sweep.sh`. + +Public API: none. Wire: none. + +## Related + +- [Routes and announces](/quest/m2/cluster-routes/README.md) - where today's numbers were measured diff --git a/quest/m2/multicast/proof.md b/quest/m2/multicast/proof.md new file mode 100644 index 00000000000..0343dc184ac --- /dev/null +++ b/quest/m2/multicast/proof.md @@ -0,0 +1,62 @@ +# [M] The route rule is proven to converge on any topology + +## Goal + +The route rule's algebra is instantiated against the published Agda library +for path-vector convergence, with every axiom discharged: how a node ranks +routes, how a route changes crossing a link, the mark's export filter, the +rule that a path naming its receiver is invalid, and the chain across a trust +boundary. The theorem then applies on any topology: from any starting state, +under loss, reordering, and duplication, the nodes reach one stable state. A +Rust property test asserts that the router's own rank and extend functions +meet the same axioms, a nightly lane checks the Agda, and +`rs/moq-net/verify/README.md` says what is proven and what is not. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): ✅ mechanized, checked nightly +and not in the merge gate, with TLA+ and sweeps beside it (rejected: a +written proof with property tests; TLA+ and sweeps alone). Today's checking +is exhaustive only at 2 to 3 relays with one failure, and the maintainer +ended the unbounded TLA+ attempt on 2026-11-13. + +The theorem is Daggitt and Griffin's, "Formally Verified Convergence of +Policy-Rich DBF Routing Protocols" ([arXiv 2106.01184](https://arxiv.org/abs/2106.01184)), +Theorem 5: an increasing path algebra always converges. Its library is +[MatthewDaggitt/agda-routing](https://github.com/MatthewDaggitt/agda-routing); +pin it and Agda through Nix. + +What to show, each a lemma about one message handler: + +- Selection is a total order with a fixed tie-break. +- No change across a session makes a route preferred to what it was: adding + the link's cost, taking the lower priority, keeping a drain, appending the + hop. A filter maps a route to invalid. +- A route whose path names the receiver is invalid, and a path grows by + exactly the hop it crossed. +- What a node advertises is a function of the latest it heard from each + neighbour. The dead-hop withdrawal is extra: show it removes only routes + the iteration would remove, so it only arrives sooner. +- The chain is the same algebra at the grain of trust boundaries. + +State plainly what the theorem does not give, and where each is covered: + +- No loop at every instant: [requests](/quest/m2/multicast/requests.md) fail + closed instead. +- A cost that depends on who subscribed: + [warmth](/quest/m2/multicast/warmth.md) argues it separately. +- Liveness under churn that never stops. +- That the tree is loop-free once routes settle: one paragraph, since next + hops toward one origin node at a fixed point cannot cycle and a request + names its origin node. + +TLA+ stays for the session, subscription, and failover state machines, where +orderings matter. `RouteHandoff.tla` describes the rule being replaced and +goes with it in the [switch](/quest/m2/multicast/switch.md). Report how long +the Agda check takes. + +Public API: none. Wire: none. + +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - fixes the rank and the transformations to prove diff --git a/quest/m2/multicast/requests.md b/quest/m2/multicast/requests.md new file mode 100644 index 00000000000..dd28f3db54d --- /dev/null +++ b/quest/m2/multicast/requests.md @@ -0,0 +1,88 @@ +# [L] A request treats routing as a hint + +## Goal + +No subscription, fetch, or track request depends on routing state being +fresh. A relay confirms a request only once its upstream has, and serves +every request for one Instance and track from one upstream subscription, +whichever origin node the requests named. A request whose route ends, is +refused as unroutable, or loops moves to another route of its Instance +instead of ending the track, hanging, or ending with `Unroutable` while its +relay shows the broadcast available. No group crosses one link twice for one +subscription, and the old and new copies of a move overlap for a bounded +time. Tests on real lite-07 relays with mocked time pin each. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): this is the data half of +"evidence, not timers" ([epic](/quest/m2/multicast/README.md) item 4). With +no hold and no doubt, selected routes may loop or go stale while news is in +flight, and that is safe only if a request fails closed and follows its +Instance. It started as the request loop quest, which this absorbs. + +Decided 2026-11-15 (maintainer, on +[moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ m2 +(rejected: m3). Relays that show a broadcast available while ending readers' +subscriptions should block the milestone. + +Decided 2026-11-13 (maintainer, /quest-plan on the BGP failure modes): + +- ✅ A fixture on real relays, with no TLA+ for pending requests and + aggregation (rejected: adding them to the route model). +- ✅ A relay's pending upstream request follows its route when the selected + next hop changes, even before the request is answered, so a loop clears + with the routes (rejected: a hop count on every request that fails one at + the limit, which adds a field to SUBSCRIBE, FETCH, and the other requests + and turns a loop that would heal into an error the subscriber sees). + +Why a request cannot find its own loop: once relays aggregate requests, one +fed around a loop meets a relay already asking upstream and joins it, so +neither a nonce nor a carried path ever returns to its sender. Nothing +errors and nobody is fed. What holds instead: a pending request confirms +nothing downstream, and every relay re-asks when its selected route changes. + +Hazards to reproduce first, from the 2026-11-15 audit and the acceptance +fixtures: + +- `route_failover::live_mesh_link_failure_keeps_presence` cuts all but one + of a publishing relay's links on a 34-relay mesh, and 3 readers on relays + that stayed available end with `Unroutable`. Flip it once they read on. +- An upstream that loses its own route resets the subscription `Unroutable` + and sends the route news on another stream. If the reset is handled first, + the front aborts a track that has delivered and never splices it again + (`rs/moq-net/src/model/front.rs` `track_ended`), while + `rs/moq-net/src/model/resume.rs` treats an unroutable copy as waiting for + another route. Found by reading; reproduce it two hops below a live + upstream that loses its route, and for a fresh TRACK naming a route that + just ended. +- During a move the old and new upstream subscriptions overlap with no time + bound if the old route keeps winning, and the newest group is asked for + again. +- Two route entries for one claim through one session send two SUBSCRIBEs + that differ only in depth. +- A moq-transport peer that announces back a name it learned from this + relay looks like a nearer publisher, and a request sent to it comes + straight back. Unsupported + ([epic](/quest/m2/multicast/README.md) item 10): pin that the pair of + requests stays pending without retrying in a loop, and that other + broadcasts on both sessions read on. + +The loop fixture: form the loop the churn sweeps found (two unrelated +failures at once, so two relays briefly route a publisher through each +other), subscribe at both while it lasts, then let the routes settle. Both +subscribers receive the next group and no upstream request is left pending. +Cover a request answered before the loop and one still unanswered when it +forms. + +Fix in the per-reader path, which stays: the producer-replacement redesign +that would have deleted it was dropped 2026-11-15 (maintainer, +/quest-complete). Run the `verify-failover` and `verify-resume` skills on +what they cover. + +Public API: none. Wire: none expected. If a refusal must tell "this route is +gone" from "no such track", that is a lite-07 wire change to report, which is +why this sits in m2. + +## Related + +- [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) - its doubt rule lets a publisher leaving three or more relays at once loop a request until the news arrives diff --git a/quest/m2/multicast/router.md b/quest/m2/multicast/router.md new file mode 100644 index 00000000000..4edfba9453b --- /dev/null +++ b/quest/m2/multicast/router.md @@ -0,0 +1,51 @@ +# [XL] The new route rule runs as a sans-IO router + +## Goal + +`rs/moq-net/src/route/` holds a production router for the +[epic](/quest/m2/multicast/README.md)'s rule beside `path::Router`, sans-IO +and driven only by tests until the [switch](/quest/m2/multicast/switch.md). +The [bake-off](/quest/m2/multicast/bakeoff.md)'s fixtures and sweeps run on +it, the exhaustive delivery-order search checks it, and a benchmark swept +over forwarding nodes, publishers, broadcasts, and sessions shows that an +event costs work in what it changed. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): the router first with no wire +change, then the switch, the order +[Routes and announces](/quest/m2/cluster-routes/README.md) Plan item 28 used +for the path rule. + +- Build the rule as the bake-off's pick leaves it: routes to forwarding nodes + only, the dead-hop withdrawal with no moved list, no hold, no doubt, no + timer, claims ordered by revision and certified by floors, the chain + across a scoped grant, and the one mark. +- Keep what is independent of the rule: the interest set edited in place + ([moq-dev/moq#8728](https://github.com/moq-dev/moq/pull/8728)), and + `Reader` and `View`. +- The reader debounces unavailable to offline locally, with mocked time, for + as long as the bake-off measured routes take to settle. It replaces what a + hold protected and is never on the wire. +- A relay stays on its next hop among routes of equal rank, and never sends + a route back over the session it learned it on. Both were quests of their + own until 2026-11-15: a change of hops alone was 3.5% to 11% of route + messages, one link coming up on a 32-relay full mesh cost 592 route + messages against 251, and advertising a route back to its source doubled + route traffic between every pair of relays. +- Per-event work follows what changed, never every answering stream or every + claim behind a route. [Route fan-out](/quest/m2/cluster-routes/route-fanout.md) + pins where the path router fails this. +- Bound what a peer can make the router hold by its own requests, not by a + constant. +- The exhaustive search over every delivery order + ([moq-dev/moq#8729](https://github.com/moq-dev/moq/pull/8729)) moves to + this router, and each property it checks is shown to fail on a mutation + that breaks it. The maintainer chose on 2026-11-14 to keep that search over + the compiled code and retire the TLA+ route model. + +Public API: none. Wire: none. + +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - the pick, and the fixtures this promotes diff --git a/quest/m2/multicast/switch.md b/quest/m2/multicast/switch.md new file mode 100644 index 00000000000..21ab958223a --- /dev/null +++ b/quest/m2/multicast/switch.md @@ -0,0 +1,79 @@ +# [XL] lite-07 sessions run the new route rule + +## Goal + +Every lite-07 session in Rust and `js/net` drives the +[new router](/quest/m2/multicast/router.md) on one announce stream per side, +the draft specifies it, and the old rule is gone: ROUTE_HOLD, the doubt, +DEAD's moved list, the Route Key Parameter with its proof and symbolic model, +qualified Node IDs, routes per prefix with their Route Suffix and Route +Depth, `path::Router`, and `RouteHandoff.tla`. The draft's routing text is +shorter than today's and the announce stream carries fewer message types. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): lite-07 carries the new rule +and the cut waits for it ([epic](/quest/m2/multicast/README.md) item 2). +lite-07 is `-wip`, so relays pinned to `main` roll together, as each earlier +lite-07 wire change did. + +Recommended wire, for the implementer to confirm or improve; each departure +is a decision to list in the PR: + +- A route is started, updated, and ended, and a closed session is named in + one fixed-size message. Nothing answers a hold, and no message lists moved + routes: the updates follow in order on the same stream. +- A claim carries its chain as a field of its own, replacing the temporary + Lite-06 Chain and its Setup Parameter: the nodes that announced it again + across a scoped grant or an older version. +- A request keeps its Instance and the origin node its first hop chose, and + loses Route Depth. +- An endpoint that never forwards declares no Node ID. A peer holding a + scoped publish grant sends claims and no routes, and the relay announces + them under its own Node ID. Peers that trust each other with the whole + tree exchange routes. +- The link mark stays as `upstream`. moq-net's per-link `proxy` switch, which + the relay hard-wires on, goes if nothing else needs it; that is a public + API change to report. + +Delete with the old rule: + +- `rs/moq-net/src/route/proof.rs`, `js/net/src/lite/route_key.ts`, + `rs/moq-net/verify/route-key/`, the `verify-route-key` skill, and the + interop route key lane (`test/interop/clients/js/route-key.ts`). +- `rs/moq-net/verify/route/` and the `verify-route` skill, or rewrite the + skill for the exhaustive search. +- [Firefox tabs prove their route key](/quest/m5/routes-browser-proof.md), + and the release cut's reason to wait on + [the Playwright bump](/quest/m1/playwright-pin.md). +- [Doubt holds](/quest/m2/cluster-routes/doubt-holds.md), + [doubt and keep-alive](/quest/m2/cluster-routes/doubt-keepalive.md), + [held after close](/quest/m2/cluster-routes/held-after-close.md), and + whatever of [route fan-out](/quest/m2/cluster-routes/route-fanout.md) the + new router answers, with the fixtures that pin them. + +Rewrite the assertions of the acceptance fixtures +(`tests/integration/route_acceptance.rs`, `route_failover.rs`, +`route_cluster.rs`) for the new rule, keeping their topologies. + +Quests written against the old wire, to retarget or delete in this PR: +[outbound federation](/quest/m3/cluster-routing/federation.md), +[the routing guide](/quest/m3/routing-guide.md), +[route wakes](/quest/m4/route-wakes.md), +[handoff routes](/quest/m4/relay-handoff-routes.md), +[regrant recheck](/quest/m4/regrant-route-recheck.md), +[announce interest memory](/quest/m4/announce-interest-memory.md), +[the mesh condition](/quest/m5/lite07-mesh.md), and +[route memory](/quest/m9/perf/cluster-route-memory.md). + +Update the draft's Announce, Floors, Routing, Dead Hops, and Resolution +sections, the message sections, the Setup Parameters, and the lite-07 +changelog, and `doc/bin/relay/cluster.md`. Run `just check`, +`just test interop --all`, and `just drafts check`. + +Public API: the per-link `proxy` switch, if removed. Wire: lite-07-wip only. + +## Required + +- [The router](/quest/m2/multicast/router.md) - the rule these sessions drive +- [The route stream](/quest/m2/cluster-routes/route-stream.md) - one announce stream per side, which this builds on diff --git a/quest/m2/multicast/warmth.md b/quest/m2/multicast/warmth.md new file mode 100644 index 00000000000..4302b7a7338 --- /dev/null +++ b/quest/m2/multicast/warmth.md @@ -0,0 +1,104 @@ +# [L] Three ways to find a warm copy are measured + +## Goal + +The [bake-off](/quest/m2/multicast/bakeoff.md)'s harness runs three ways for +a relay to learn that a neighbour already has demand for a broadcast, over +the prototype route rule, and the PR reports for each: whether a CDN join +skips every pop without demand and whether drones with an uplink each settle +on one; the copies each link carries and the total link cost of each +broadcast's tree, against plain shortest-path joins and against the cheapest +tree; control messages per reader change; the time to a first frame; and +unfed readers and pull cycles under churn. The maintainer picks one, and the +PR rewrites [the implementation quest](/quest/m3/cache-routing/select.md) +for it. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): ✅ measure all three, then +pick (rejected: picking one unmeasured). The warm wire was stripped from +lite-07 on 2026-11-14 because its benefit over plain path routing was never +measured. + +Decided 2026-11-15 (maintainer, on +[moq-dev/moq#8873](https://github.com/moq-dev/moq/pull/8873); +[epic](/quest/m2/multicast/README.md) item 9): + +- ✅ A node is warm for an Instance while it has demand for it: something + downstream asked and its own upstream subscription is confirmed. Groups + left in a cache do not make it warm (rejected: a yes/no cache state). + Whether the signal is that fact alone or says how much demand stands + behind it is for the measurement. +- ✅ Two cases must work, so the pick is the cheapest mechanism that + delivers both: + - **Pop skipping.** Pops joined by links that pass over other pops. A join + for a broadcast no pop in between has demand for takes the long link and + costs those pops nothing. Once one of them has demand, a farther pop + joins through it when that is cheaper than the long link, and goes back + when the demand ends. + - **One uplink per group of drones.** Drones meshed with each other, each + with a costly uplink to the CDN. The first with demand reaches the CDN, + and the rest join it over the mesh, because a warm drone costs less than + a cold uplink. Two that reach out at once end on one uplink, and when + the drone holding it loses it, another takes over. + +The three: + +1. **On the path only.** A join stops at the first relay with demand for the + Instance, and equal-cost next hops are chosen by the rendezvous hash, so + siblings facing the same choice converge on one. No warm signal. A join + never learns of demand off its own path, so this delivers neither case: + it is the baseline for cost, not a candidate. +2. **Ask on subscribe.** Each SUBSCRIBE answer, payload or metadata-only, + says whether the neighbour has demand for the Instance and its summed + cost from the source, and the relay takes the cheapest. A + [metadata-only subscription](/quest/m3/route-health/meta.md) doubles as + the query, so nothing is sent for a broadcast nobody asks about. Sweep how + many neighbours a relay asks. +3. **Advertise.** A relay with demand announces it on the announce stream. + +The constraint the maintainer set for all three: a warm link costs its warm +price and a cold link its cold price, both at least 1, summed from the source +along the path the data takes, so a route never ranks better for being +extended. The stripped rule broke it: a carrier offered its copy at no cost +and a hash ranked carriers, and pulls formed a cycle in 3.3% of churn steps, +most leaving a reader unfed. + +What that constraint gives and what it leaves: with each relay's cost summed +along the path its data takes, two relays can never each prefer the other, +since one would need `b + link < a` and the other `a + link < b`. For the +same reason two drones that reached out at once, each at the cost of its own +warm uplink, never merge. Merging them needs a rule both compute alike, such +as the lower rendezvous hash of Instance and Node ID keeping its uplink, and +a move that completes only once the new copy delivers. Show that rule adds +no cycle, or report that simultaneous starts stay on two uplinks and what +that costs. + +Joining the cheapest point of the tree that exists is the greedy online +Steiner tree, which stays within a logarithmic factor of the cheapest tree +when readers only join (Imase and Waxman, 1991). Leaves and moves are what +the sweep adds to that. + +A cost that depends on who subscribed is outside the +[proof](/quest/m2/multicast/proof.md)'s theorem. Argue it separately and +sweep it: taking a route only makes it cheaper for others, and leaving one +only worsens an offer nobody uses. A cycle that outlasts the messages in +flight fails the mechanism; count the transient ones with the readers they +leave unfed. + +Shapes where the three differ: the two cases above; two edges that each +prefer a different core; a ring with a far source; and one viewer per +broadcast, where the first must cost no message at all. Earlier sweeps to +compare with: a LAN mesh of 6 pulled one uplink copy per broadcast, a ring +of 6 kept 12 copies for 8 broadcasts, and warm routes cost 3.1 to 4.5 +messages per reader change. + +The cut waits on the pick, not on the implementation. An additive field +lands in whichever wip is current with its implementation, as decided +2026-11-14; the PR says whether the pick changes bytes lite-07 would freeze. + +Public API: none. Wire: none. + +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - the harness and the route rule the three run over diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index 08e865e68b5..63405116fcc 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -2,263 +2,67 @@ ## Goal -A relay stays on a broadcast's path only while it has a local reader, feeds -two or more downstream sessions, or is its one consumer's cheapest route -given the copies the others hold. Otherwise its child bypasses it, so a broadcast with one viewer -crosses fewer relays. A relay joins a neighbor -that already carries a broadcast instead of pulling a second copy, so a LAN -of relays with their own uplinks pulls one copy over one uplink. Routing -stays loop-free under stale state, converges once demand and topology stop -changing, and sends no per-broadcast message for an idle broadcast. +A relay pulls each broadcast from the cheapest copy: a neighbour that +already has demand for its Instance when that costs less than a fresh copy +from the source. A join stops at no pop without demand, drones with their +own uplinks pull one copy over one uplink, edges that prefer different cores +share one copy between regions, and no link carries a group twice. Routing +converges once demand and topology stop changing, and a broadcast nobody +reads costs no message of its own. Non-goals: steering clients to the relay that holds a broadcast; decisions -from measured load or RTT (link quality, planned downstream in Kixel); -browsers serving copies they receive; a relay on neither consumer's cheapest -route becoming a shared stop; one copy across a multi-hop sparse overlay -rather than among direct neighbors; and creating links, since routing -chooses only among the sessions an operator configured. +from measured load or RTT; browsers serving copies they receive; and creating +links, since routing chooses only among the sessions an operator configured. ## Plan -Decided 2026-11-08 (maintainer, /quest-plan): - -1. ✅ **Node routes are the cold cost, and optional per-broadcast warm routes - are the warm cost.** The per-origin ROUTE metric of - [Cluster routing](/quest/m3/cluster-routing/README.md) stays the cold - cost, static and one per origin. A relay carrying a broadcast may - advertise a warm route for it. This revises Cluster routing's non-goal of - cache-dependent route prices. (rejected: per-broadcast forwarding with a - cache-only subscribe and no advertisement; keeping routing cache-agnostic) -2. ✅ **A warm route is per neighbor and travels one hop.** Reopened by - item 13; replaced by item 14, where a warm route propagates and a relay - offers each neighbor the best copy whose Hops name neither. It went only to - direct neighbors and was never forwarded. B was warm toward N only while B - had demand besides N's (a local reader or another downstream session), - and withdraws once that demand has been gone for the copy's existing - linger, with no new timer. B sends it only to neighbors whose cold metric - for the origin is at least its own, since no other neighbor may use it - (item 4). Without the per-neighbor rule, a relay whose reader left stays - warm because it still feeds N, and N never skips it again. (rejected: - broadcast-wide warmth, as the pre-#3225 design had; forwarding warmth k - hops, which floods every warm transition) -3. ✅ **One cost per link, priced marginally.** Replaced by item 14: a copy - ranks as a route does, drain then priority then cost, and its carrier offers - it at no cost, so joining a copy costs only the links it crosses. Pulling through a warm - neighbor costs the link alone, since that neighbor's cold metric is - already paid; a cold neighbor costs the link plus its cold metric. A tie - goes to the warm route, then to item 4's hash. A topology that wants both - behaviors prices a link that skips one relay at more than one neighbor - link and less than two: a cold broadcast then takes the skip, and once - the skipped relay pulls for its own reader, its neighbor joins it. With - uniform prices both cases tie and fall to the hash. (rejected: a warm and - a cold price per link; a warm price the carrier seeds) -4. **Eligibility, which the proof must hold.** Reopened by item 13. A relay - may pull broadcast P - through neighbor N only when N's advertised route to the request's pinned - origin has a newer seqno, or an equal seqno with - `(cold metric, hash(P, N))` below `(feasibility distance, hash(P, self))`. - A feasibility distance never exceeds what its node advertised at that - seqno, so it strictly decreases along every forwarding edge even when an - advertisement is stale: no loop, no rank, no hold timer. The pre-#3225 - rank was a cold cost each relay reported about itself, so a stale report - could let a ring of relays all let go at once, which its 500 ms hold only - narrowed. Route advertisements stay one selection per origin. -5. ✅ **Proof first:** model and sweeps before the implementation. - (rejected: implementing with targeted fixtures only) -6. ✅ **The proof picks the wire placement.** The warm message is designed - fresh rather than reviving ANNOUNCE_TAKEOVER, which - [moq-dev/moq#8063](https://github.com/moq-dev/moq/pull/8063) deleted in - favor of instance continuity. The proof sizes it and - picks lite-07 or an additive SETUP extension. (rejected: - reusing ANNOUNCE_TAKEOVER's slot) -7. ✅ m3, after the m2 route layer it ranks on. (rejected: m2; m4) Amended - 2026-11-09 (maintainer, /quest-plan from Kixel's audit of its fleet plan): - ✅ the proof alone moves to m2, ahead of the lite-07 cut, and lands the - message's draft section and codec there if it places it in lite-07 - (rejected: a design-only sizing quest in m2; leaving the proof in m3 and - accepting a SETUP extension). The implementation stays in m3. This - replaces item 6's rule that choosing lite-07 went back to the maintainer; - item 11 records the placement the maintainer chose. - -Proof results 2026-11-10 ([moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)), -modeled in `rs/moq-net/src/route/tests/cache.rs` over the real `Router`: - -8. **Item 4 holds.** No sweep step in 3000 churned scenarios formed a - per-broadcast cycle, and the fixtures fail when either the feasibility - check or the hash order is dropped. The router answers it with the - crate-private `Router::eligible`, which marks a route tied at the - feasibility distance so the caller applies the hash. The argument needs a - feasibility distance that never rises while a neighbor holds the advert; - the overtaking echo that - [moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447) found breaks - that, which [the implementation](/quest/m3/cache-routing/select.md) must - not ship before it is fixed. -Decided 2026-11-10 (maintainer, /quest-iterate on -[moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)), from the -proof's proposals: - -9. ✅ **Each pull names whom it serves**, amending item 2; replaced by item 14, - whose SUBSCRIBE says only whether it joins a copy, and lite-07 no longer - carries `Serves`: a trailing `Serves` field on SUBSCRIBE and SUBSCRIBE_UPDATE. (rejected: no field) - "Demand besides N's" can't be told from one hop: if B's only demand is a - relay pulling on N's behalf, B counts warm toward N, N joins B, B's demand - vanishes, and the cycle repeats every linger. `Serves` is alone (a local - reader, or two or more roots below it), a single root relay's Node ID, or - unknown (an endpoint that doesn't say, which warms nothing). N counts B - warm only when B serves alone or a root other than N and N's own root. A - narrowed warm route stands until the copy's linger runs out, as the - withdrawal already does. -10. ✅ **Price a dependent copy at its cost, and move only on strict - improvement**, amending item 3; replaced by item 14, whose offers do not - depend on who pulls what, and whose sweep settles every scenario with no - `UNSETTLED` list; the Goal's "cheapest cold route" became - "cheapest route given the copies the others hold". (rejected: dropping - mutual warmth so only a relay with a local reader advertises, which - converges trivially but loses the LAN sharing that feeding two sessions - gives; link-only pricing) Link-only pricing has no steady state when two - relays each keep the other warm (sweep scenario 2062 flips forever). A - warm route carries its sender's own price for its pull; a receiver the - copy depends on (its serves names only the receiver's root) pays link - plus that cost, an independent one pays the link alone, and a relay - moves only for a strictly lower `(draining, price)`. With exact - information that is a potential game on the forest's total link cost. - Under the sweep's churn, with the draft's Rendezvous Hash, 2999 of 3000 - scenarios settle; scenario 2251 never does, as two carriers keep a - narrowed route standing while two relays swap between them. The sweep names them in - `UNSETTLED` and checks each still never settles; item 14's rule settles - them. -11. ✅ **The wire is in moq-lite-07 itself** (its warm fields and `Serves` - replaced by item 14's), overturning the proof's choice - of a SETUP extension. (rejected: an additive Warm Routes SETUP - extension) Lite-07 is unpublished, so its shape can still change until - the cut, and a later change goes to the next version. The draft's - Warm Routes section and the Rust codec landed with the proof: - `WARM_START` (0xC) carries the Route ID it travels under, the path - suffix, the Instance and Minted encoded as in ANNOUNCE_START, then the - serves and cost, and assigns a Warm ID; `WARM_UPDATE` (0xD) replaces a - warm route's serves and cost by Warm ID, so a price change costs one - message rather than an end and a start; `WARM_END` (0xE) withdraws it. - A warm route follows the ROUTE_START it names on the announce stream the - carrier answers, and a ROUTE_END retires the warm routes under it. - SUBSCRIBE and SUBSCRIBE_UPDATE end with `Serves`: a type of unknown - (0x0), alone (0x1), or relay (0x2) followed by its Node ID. The hash is - the Resolution section's FNV-1a rendezvous hash of the path and the - neighbor's Node ID. Until the implementation, Rust reads warm routes and - serves and ignores them, and says unknown; `js/net` says alone, since it - forwards nothing, and skips the warm types as unknown, which is how an - endpoint that does not choose per broadcast ignores them. About 4.5 warm - messages per reader change across the sweep's churn, which also flaps - links and advances seqnos; an idle broadcast sends none. - -Measured by the proof: - -12. **Costs the sweeps measured.** A full LAN mesh of 6 relays pulls one - uplink copy per broadcast; a ring of 6 keeps 12 copies for 8 broadcasts, - one per hash local minimum. An origin seqno advance that reaches a child - before its warm parent makes the parent's older route ineligible, so the - child moves off and back (2 moves in the chain fixture); origins advance - on every announce stream open and close. Item 13's paths drop the seqno - ([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27), - and this churn with it. - -13. Reopened 2026-11-11 (maintainer, /quest-plan on the route redesign): - items 2 and 4. In the maintainer's words: "I think we need warm routes - propagating, that decision was made based on this flawed setup." ✅ **A - warm route is per broadcast and propagates**, for as long as it is - cheaper than the route its publisher's broadcasts share - ([epic](/quest/m2/cluster-routes/README.md) Plan item 26). A - broadcast no relay carries rides the shared route and costs no message of - its own. Item 4 rests safety on Babel's distance, which cannot let a relay - join a copy at a neighbour that is no closer to the publisher: the join - the Goal wants, and why item 2 stopped at one hop. Items 1 and 3 stand. - Decided 2026-11-11 (maintainer, on - [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533)): what keeps a warm - route free of loops and stale copies is its path, as for every route - (routes epic Plan item 27), and the proof then holds that rule. The - prototype in `rs/moq-net/src/route/tests/copies.rs` showed three things - the proof starts from. A warm route carries the path its copy took and - names the relay that carries it, and a relay refuses one whose path names - it, so it joins a copy at a neighbour farther from the publisher than - itself. Two relays that start carrying in the same instant trade copies - for good unless they are ranked, so a relay already pulling moves onto - another's copy only if that carrier ranks below it. And when a carrier's - uplink closes, it and a relay pulling from it point at each other until - the news crosses their link. - [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) proves items - 2 and 4 as first decided. - -14. **The proven rule** ([moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732)), - proven in `rs/moq-net/src/route/tests/cache.rs` over `route::path::Router` - ([moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)), in three - layers that each settle before the next depends on them: - - **Routes** are item 27's, which copies never change. - - **Offers.** A relay with a local reader is its copy's carrier and offers - every neighbor its copy, its own Hop ID the only hop, at no cost. A relay - with no reader offers each neighbor the best of its route and the copies - offered to it whose Hops name neither relay, when that best is a copy. - Copies rank as routes do, drain then priority then cost, so an offer - depends only on readers, routes, and other offers, and settles as a - monotone path vector does. - - **Pulls.** A carrier pulls through its best source among its route and - the copies of carriers ranked below it by the Rendezvous Hash. A relay - with no reader pulls through its best source whose Hops name no - neighbor it serves, so long as that is, for each such neighbor, the copy - it offered it or a lower-ranked carrier's; otherwise it follows its - route. SUBSCRIBE's `Warm` says whether a pull joins the copy offered or - follows the route, and a relay pulled along its route follows its own. - - Each join gets a copy whose carrier, then whose rank, is strictly better - than what it gives the relays joined to it, so once the offers and - subscriptions in flight are delivered the pulls are free of loops, as the - maintainer accepted on 2026-11-13. Unranked, two carriers join each other - and neither is fed. This replaces the rules of - [moq-dev/moq#8615](https://github.com/moq-dev/moq/pull/8615) (monotone - copy costs, carrier rank on every move, a relay bound by the carriers - behind it), which failed a few scenarios per seed because what a relay - offered depended on who pulled through it. - - Measured by `sweep_cache_routing_long` (3000 scenarios each with and - without held links, 6 relays, 2 publishers, priorities and drains), under - 12 seeds (72,000 scenarios): every scenario settles on the offers computed - centrally, an idle broadcast sends no warm message, and about 3.1 warm - messages go out per reader change. A pull lingering with no demand can hold - a copy its parent promised another neighbour, so the pulls around it may - move until the linger (30 s) runs out; settling counts from there. During - churn a broadcast's pulls form a cycle after 3.3% of steps, in about half - the scenarios, each gone within 142 deliveries. The sweep runs nightly - from `test/route-sweep.sh`. - - The wire: WARM_START and WARM_UPDATE carry a copy's Warm Cost, a Flags - byte whose low bit marks a draining copy, its priority, and its Hops, - carrier first and the sender last. SUBSCRIBE and SUBSCRIBE_UPDATE end with - a `Warm` byte. Rust's codec reads and writes both, and `js/net` the `Warm` - byte, which both send as 0 (follow the route) and ignore when read, as - `js/net` skips the warm messages; [the implementation](/quest/m3/cache-routing/select.md) - acts on them. - -Prior art: warm/cold route prices -([moq-dev/moq#2925](https://github.com/moq-dev/moq/pull/2925)) were deleted -by prefix routes ([moq-dev/moq#3225](https://github.com/moq-dev/moq/pull/3225)), -and the pop-skipping questline (warm advertise, rank, peer reconfigure) was -pruned in [moq-dev/moq#4393](https://github.com/moq-dev/moq/pull/4393). -Related designs: PIM-SM's switch from a shared tree to a shortest-path tree -([RFC 7761](https://www.rfc-editor.org/rfc/rfc7761.html)), NHRP shortcuts -([RFC 2332](https://www.rfc-editor.org/rfc/rfc2332.html)), and Babel's -feasible successors ([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966.html)). - -This README's own work, once its children land: +Reopened 2026-11-15 (maintainer, /quest-plan on +[cache-aware multicast](/quest/m2/multicast/README.md), requirement 8: +"there is a cold cost for a route and a warm cost", and routes coalesce onto +a node that says it carries a broadcast). What stands and what goes: + +- ✅ Joins stop at the first relay carrying the Instance, which the route + layer gives with no message + ([requests](/quest/m2/multicast/requests.md)). +- ✅ How a relay learns of a warm neighbour off its path is picked from + measurements ([warmth](/quest/m2/multicast/warmth.md)), which + [the implementation](/quest/m3/cache-routing/select.md) then builds. +- ✅ A warm cost is strictly positive and summed from the source along the + path the data takes. This replaces item 14's rule below, under which a + carrier offered its copy at no cost and a hash ranked carriers. +- ✅ Warm means demand, not cache contents, and pop skipping and one uplink + per group of drones are the two cases that must work (maintainer, + 2026-11-15, [epic](/quest/m2/multicast/README.md) item 9). +- The non-goal of "one copy across a multi-hop sparse overlay rather than + among direct neighbors" is dropped: a mesh is the product this serves. + +The earlier Plan, items 1 to 14 (2026-11-08 to 2026-11-14), is in +`git show bf04cecfe1:quest/m3/cache-routing/README.md`. What it measured: + +- Cache-aware cost was built four times before it (2026-07 to 2026-08), and + each time two warm relays adopted each other. +- Item 4's rule rested on Babel's feasibility distance, which cannot let a + relay join a copy at a neighbour no closer to the publisher. +- Item 14's rule settled every one of 72,000 sweep scenarios and sent about + 3.1 warm messages per reader change, but its pulls formed a cycle in 3.3% + of churn steps, about 9 in 10 of which left a reader unfed, and its gain + over plain path routing was never measured + ([moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732)). The + maintainer stripped its wire from lite-07 on 2026-11-14 + ([warm routes on the wire](/quest/m2/warm-routes-wire.md)). + +This README's own work, once its child lands: - End-to-end tests over real sessions, time mocked. A chain of three relays - plus a skip link, priced 3 and 5: a cold broadcast takes the skip, a reader - on the skipped relay makes the far relay join it, and that reader leaving - sends it back to the skip. Two LAN relays with their own upstream links and - readers on both pull one copy over one uplink, with the owner spread across - broadcasts by the hash. -- Docs: warm routes and how to price skip links in `doc/bin/relay/cluster.md`, - and a section on the routing concept page once the + plus a skip link: a cold broadcast takes the skip, a reader on the skipped + relay makes the far relay join it, and that reader leaving sends it back. + Two LAN relays with their own uplinks and readers on both pull one copy + over one uplink, whether the readers arrive in turn or at once, and the + other uplink takes over when that one is lost. +- Docs: warm and cold costs in `doc/bin/relay/cluster.md`, and a section on + the routing concept page once the [routing guide](/quest/m3/routing-guide.md) has added it. ## Related -- [Cluster routing](/quest/m3/cluster-routing/README.md) - the node routes and feasibility distances this ranks on -- [Relays split horizon per session](/quest/m3/route-split-horizon.md) - per-session suppression of node routes, beside warm routes' per-neighbor rule +- [Cluster routing](/quest/m3/cluster-routing/README.md) - the node routes this ranks on diff --git a/quest/m3/cache-routing/select.md b/quest/m3/cache-routing/select.md index 0700156904c..05685a6cf14 100644 --- a/quest/m3/cache-routing/select.md +++ b/quest/m3/cache-routing/select.md @@ -1,60 +1,56 @@ -# [XL] Warm routes and per-broadcast parent selection +# [XL] Relays pull each broadcast from the cheapest copy ## Goal -Relays send warm routes as the -[epic](/quest/m3/cache-routing/README.md) decides. Each relay pulls every -demanded broadcast through its cheapest eligible neighbor, and moves an -existing copy when a cheaper eligible one appears or its warm parent -withdraws. A peer that never sends warm routes sees only cold routes and -behaves as it does today. Relays export how many upstream copies they hold -and how many moves warm routes caused. +Relays learn of warm copies the way +[warmth](/quest/m2/multicast/warmth.md) picked, and each pulls every demanded +broadcast from the cheapest copy of its Instance, moving an existing pull +only when the route it uses gets worse or ends, or another is strictly +cheaper in summed cost. A peer that sends no warm signal is treated as cold +and behaves as it does today. Relays export how many upstream copies they +hold and how many moves a warm copy caused. ## Plan -- Implement the rule the [epic](/quest/m3/cache-routing/README.md)'s item 14 - records ([moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732)) by - moving `decide` and `offer_from` out of `rs/moq-net/src/route/tests/cache.rs` - into production code that the fixtures and sweep then drive, the way `Sim` - drives `path::Router`: today they test the rule against itself and pass - without touching the session. Add a check of the Goal's efficiency to - `steady()`, copies held against a cold-routing baseline, since a rule that - never offered a copy passes the sweep now. The rule covers the - warm messages and SUBSCRIBE's `Warm` byte (which return to the wire with - it, below), warm state derived from each front's demand, and the choice of - what to pull. The long sweep already runs nightly from - `test/route-sweep.sh`. +Rewritten 2026-11-15 (maintainer, /quest-plan on +[cache-aware multicast](/quest/m2/multicast/README.md)): the rule this was +to build, item 14 of the [epic](/quest/m3/cache-routing/README.md)'s earlier +Plan, is replaced. The warmth quest's PR fills in the mechanism here once +the maintainer has picked it. + +What holds whichever is picked: + +- A warm link costs its warm price and a cold one its cold price, both at + least 1, summed from the source along the path the data takes. - Extend today's selection rather than adding a second selector. `route_order` in `rs/moq-net/src/model/origin.rs` already hashes equal routes per path, and a front already moves to a better route of its own - instance and resplices (`rs/moq-net/src/model/front.rs`). -- A warm route names the origin and instance it carries, and a front moves - only to a copy of its own instance. -- Bound the warm state a peer can make a relay hold by design, as - announcements are bounded. -- Each pull sets SUBSCRIBE's `Warm` byte, which #8732 put in place of the - `Serves` field; `js/net`, which forwards nothing, sends 0 (follow the - route) and skips the warm types as unknown. -- Export upstream copies held, warm routes sent and received, and moves by - cause through the relay's existing metrics. + Instance and resplices (`rs/moq-net/src/model/front.rs`). A front moves + only to a copy of its own Instance. +- The rule runs in production code that the fixtures and sweeps drive. The + stripped rule lived only in a test simulator that checked it against + itself, so a rule that never offered a copy passed. +- Check the Goal's efficiency: copies held against plain path routing. +- Bound the warm state a peer can make a relay hold by design. +- Export upstream copies held and moves by cause through the relay's + existing metrics. - `rs/moq-net/AGENTS.md` says a relay behaves the same warm or cold. That - covers answers to requests; warm routes change only which neighbor a - request goes to. If the rule's wording contradicts warm routes, bring a - rewording to the maintainer. + covers answers to requests; this changes which neighbour a request goes + to. Bring a rewording to the maintainer if the two contradict. + +Decided 2026-11-14 (maintainer, pick on +[moq-dev/moq#8764](https://github.com/moq-dev/moq/pull/8764)): ✅ lite-07 +freezes without a warm wire, so whatever wire the pick needs lands in the +then-current wip version with this. + +Public API: report any relay configuration added, such as a warm cost per +link. Wire: whatever the pick needs, in the then-current wip version. -Decided 2026-11-14 (maintainer, pick on [moq-dev/moq#8764](https://github.com/moq-dev/moq/pull/8764)): -✅ lite-07 freezes without the warm wire -([warm routes on the wire](/quest/m2/warm-routes-wire.md)), so this quest -re-adds WARM_START, WARM_UPDATE, WARM_END, and SUBSCRIBE's `Warm` byte in the -then-current wip version, and first measures copies and relays crossed -against plain path routing. +## Required -Public API: report any relay configuration added for warm routes. Wire: the -warm messages and the `Warm` byte return in the then-current wip version, -with the draft's Warm Routes section. +- [Warmth](/quest/m2/multicast/warmth.md) - picks the mechanism this builds +- [The switch](/quest/m2/multicast/switch.md) - the route layer it ranks on ## Related -- [Routes and announces](/quest/m2/cluster-routes/README.md) - the route layer driving lite-07 sessions since [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023); its switch to path routes deletes the feasibility distances this rule once read -- [Warm routes on the wire](/quest/m2/warm-routes-wire.md) - may strip the warm wire from lite-07 or move this epic downstream before this starts -- [Transient cycles](/quest/m3/cache-routing/transient-cycles.md) - lowers the churn-time pull cycles the rule allows +- [Route health](/quest/m3/route-health/README.md) - its metadata-only subscription is the query if the pick is to ask on subscribe diff --git a/quest/m3/cache-routing/transient-cycles.md b/quest/m3/cache-routing/transient-cycles.md deleted file mode 100644 index d165450bed5..00000000000 --- a/quest/m3/cache-routing/transient-cycles.md +++ /dev/null @@ -1,41 +0,0 @@ -# [M] Fewer pull cycles while warm routes churn - -## Goal - -The cache-aware rule's transient pull cycles during churn happen less often -than the 3.3% of churn steps [moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732) -measured, or clear faster than its 142 deliveries, without slowing a join the -maintainer has not agreed to slow, and the sweep reports the rate so a change -that raises it shows. - -## Plan - -Decided 2026-11-14 (maintainer, /quest-complete on #8732): ✅ merge, and plan -a quest to lower the rate (rejected: accept the rate as is, hold the PR, -delete). Planned 2026-11-14 by the -[feature review](/quest/m3/release/feature-review.md), which found the pick -had no quest yet. - -What #8732 measured: about 9 in 10 cycle steps leave some reader unfed until -the crossing message lands; #8615's earlier sweep saw 0.39% and 53 -deliveries; at seed 3, dropping per-link priorities (2.8 to 3.0%) or letting -carriers exclude the neighbours they serve (2.9 to 3.1%) explained neither the -gap. Start by finding which of the sweep's churn or the per-neighbour offers -causes it, then propose fixes with their join-latency cost. - -A second baseline to account for: [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) -(the path rule for a flapping publisher) re-pinned -`route::tests::cache::a_ring_of_stale_offers_settles_on_the_link_left`'s cap -on cycle steps from 40 to 204 across 20 seeds, 4 to 22 per seed, each cycle -cleared within 20 deliveries; the rule it replaced gave 2 per seed, each -cleared within 2. Explain that rise along with #8732's rate, and measure any -fix against both the sweep and this fixture. - -If [warm routes on the wire](/quest/m2/warm-routes-wire.md) moves the epic -downstream, this moves with it. - -Decided 2026-11-14 (maintainer, pick on [moq-dev/moq#8764](https://github.com/moq-dev/moq/pull/8764)): ✅ a fix adds no join latency; one that only clears cycles faster comes first (rejected: a carrier waiting for a crossing message before it joins). - -Public API: none. Wire: none expected; a rule change that needs one updates -the draft's Warm Routes section. - diff --git a/quest/m3/cluster-routing/README.md b/quest/m3/cluster-routing/README.md index a3ee172a111..999b8ef7160 100644 --- a/quest/m3/cluster-routing/README.md +++ b/quest/m3/cluster-routing/README.md @@ -20,6 +20,14 @@ work, and no quest requires one in a later milestone, since a blocker ranks by its own milestone. ✅ m3 (rejected: m2, where the cut would wait on it; pulling the m2 blocker into m1). It ranks right after m2, when it can start. +Reopened 2026-11-15 (maintainer, /quest-plan on the whole route layer): +[cache-aware multicast](/quest/m2/multicast/README.md) replaces the route +layer this Plan describes, and where the two differ it wins. A routing node +is a node that forwards, not every publisher; a claim crosses a scoped grant +by a chain, not a qualified identity under a route key; and holds, doubt, +and the non-goal of cache-dependent route prices are gone. One model for CDN +tiers, outbound local clusters, and meshes stands. + Decided with the maintainer, 2026-10-13: - Keep per-origin ROUTEs and path-less ANNOUNCEs on the ordered announce @@ -154,6 +162,7 @@ setup after the capabilities land. ## Required +- [Cache-aware multicast](/quest/m2/multicast/README.md) - the route layer this builds on, replacing the path rule - [Routes and announces](/quest/m2/cluster-routes/README.md) - shared routing contract and implementation ## Related diff --git a/quest/m3/cluster-routing/federation.md b/quest/m3/cluster-routing/federation.md index 863f28599a2..129e3221667 100644 --- a/quest/m3/cluster-routing/federation.md +++ b/quest/m3/cluster-routing/federation.md @@ -9,6 +9,15 @@ cluster with two uplinks, partition, and healing. No bespoke federation router. ## Plan +Reopened 2026-11-15 (maintainer, /quest-plan on +[cache-aware multicast](/quest/m2/multicast/README.md)): the CDN relay +announces the local relay's claims under its own Node ID, with a chain of +the nodes that did so, and refuses a claim whose chain names it. No route, +qualified identity, or route key crosses the scoped grant, so the key file +below goes, and a relay behind the local relay may be scoped too. +[The switch](/quest/m2/multicast/switch.md) rewrites the paragraphs below +that say otherwise. + Decided 2026-10-13: this is a required use case, independently shippable from peer-to-peer routing. Existing cluster sessions are bidirectional and scoped peer admission exists; extend and prove those paths rather than creating a reverse diff --git a/quest/m3/route-health/README.md b/quest/m3/route-health/README.md new file mode 100644 index 00000000000..1a08030b568 --- /dev/null +++ b/quest/m3/route-health/README.md @@ -0,0 +1,73 @@ +# Route health + +## Goal + +A relay or endpoint that opts in and demands a track over one route watches +a second route to the same Instance with a metadata-only subscription, and +moves the track when the first stalls, without waiting on a QUIC keep-alive +or idle timeout. Today a route that dies without a close starves its readers +until the 10 s idle timeout fires, and a session that stays up while its +payload stalls is never recovered +(`rs/moq-net/tests/integration/route_health.rs`). + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): ✅ planned again, in m3, as the +secondary route of [cache-aware multicast](/quest/m2/multicast/README.md) +(requirement 13: "don't rely on QUIC keep-alives for fast failovers"). The +first epic was deleted 2026-11-09 for want of a consumer +([moq-dev/moq#8423](https://github.com/moq-dev/moq/pull/8423)); read it with +`git show cd560bde5^:quest/m5/route-health/README.md`. Its decisions stand, +made by the maintainer on 2026-10-13 and 2026-11-02 from the stalls +[moq-dev/moq#7096](https://github.com/moq-dev/moq/pull/7096) measured: + +- ✅ Evidence is comparison, never silence. Switch when the active route's + delivered media time lags what the standby reports by more than the + reader's max delay, or 2 s for a reader without one, while the reader + demands. A quiet track is not a dead one, and `largest()` values are never + compared. +- ✅ One standby per demanded track: the next-best route to the same + Instance. Only payload demand creates a standby, so metadata observers + never branch into a monitoring tree of their own. +- ✅ When neither route shows progress for that budget, the standby slot + moves to the next candidate, doubling the wait per move, and nothing is + failed. +- ✅ A demoted route never takes the track back while the working one + delivers (2026-11-02, reversing 2026-10-13's return after recovery). +- ✅ Per track, not per session: one stalled track never fails a session. +- ✅ Only between claims of one Instance the publisher named. A minted + Instance, as an older version or moq-transport gets, never resumes. + +Decided 2026-11-15 (maintainer, on +[moq-dev/moq#8873](https://github.com/moq-dev/moq/pull/8873)): ✅ opt in, +off by default: "We wouldn't do standbys inside our CDN". A standby about +doubles a track's subscription state and sends its headers down a second +path, which a mesh on links that stall wants and a CDN tier does not. The +node that would hold the standby turns it on, a relay in its configuration +and an endpoint on its client (rejected: a standby wherever a second route +exists). Turning it on for one broadcast, so an operator can offer it there, +is a later decision that [failover](/quest/m3/route-health/failover.md) +leaves room for. A node that has not opted in still finds a silent death by +the idle timeout alone and never recovers a stall on a live session. + +New on 2026-11-15, from the prior art this is closest to: + +- The standby is a route whose path does not name this node, and one that + does not cross the hop under suspicion ranks first. A fast-reroute + alternate often shares the failed element (RFC 7431 section 7). +- A relay stops asking its standby when the standby is the only thing it + serves that track to, so two relays never keep each other alive (the same + RFC's rule for its secondary). +- The answer to a metadata-only subscription is also where a neighbour says + whether it carries the Instance, if + [warmth](/quest/m2/multicast/warmth.md) picks that. + +This README's own work, once its children land: end-to-end validation across +a multi-hop relay network. Stalled video moves while healthy audio stays, a +stall with advancing control is caught, a recovered route does not take the +track back, two quiet routes stay unknown, a bottleneck shared by both routes +does not hide a working third, and relays deciding at once do not swap. +Report media overlap, control traffic, wakes, switches, and memory across +tracks, candidates, and readers. Revisit the relay's 10 s default QUIC idle +timeout with the result, since it stays the only detector wherever the +standby is off, and add a route-health section to the routing concept page. diff --git a/quest/m3/route-health/failover.md b/quest/m3/route-health/failover.md new file mode 100644 index 00000000000..8ef2f3d5080 --- /dev/null +++ b/quest/m3/route-health/failover.md @@ -0,0 +1,48 @@ +# [XL] A stalled route yields to its standby + +## Goal + +For each track with payload demand, a relay or endpoint that opted in holds +a metadata-only subscription on its next-best route to the same Instance, and +promotes it in place when the active route's delivered media falls behind +what the standby reports by more than the reader's budget. The track then +stays on the working route. A silent death and a payload stall on a live +session, which `rs/moq-net/tests/integration/route_health.rs` pins as lost +media and as never recovered, both recover within that budget. + +## Plan + +The [epic](/quest/m3/route-health/README.md) holds the decisions: comparison +as evidence, one standby per demanded track, the doubling exploration, no +return to a demoted route, per track, only for an Instance the publisher +named, and opt in. + +- The opt-in is off by default and belongs to the node that holds the + standby: a field in the relay's configuration and an option on the client, + named alike in Rust, JS, and the bindings. With it off, nothing here costs + a message or a subscription. Read the switch where a track's source is + chosen, so a later per-broadcast switch changes one lookup. + +- Each track chooses its source apart from the broadcast's shared route + selection: the front already holds every usable route to its Instance + (`rs/moq-net/src/model/origin.rs`, `model/front.rs`), and the router hands + it each alternate. The maintainer permits falling back to per-broadcast + failover if per-track proves disproportionate; say why if used. +- Promotion overlaps payload briefly while the new route is verified, and a + steady standby carries none. Bound the overlap. +- Two relays deciding at once must not swap onto each other: the standby is + never a route whose path names the relay, and a relay drops its standby + when the standby is all it serves the track to. +- Model the promotion with the `verify-failover` skill before building it, + since it adds an ordering between two subscriptions. +- Measure the interruption for a clean close, a silent death, and a payload + stall, against today's: about 0, the 10 s idle timeout, and never. + +Public API: the opt-in, in the relay's configuration and on the client, +beside [`Subscription.meta`](/quest/m3/route-health/meta.md). Wire: none +beyond it. + +## Required + +- [Metadata-only subscriptions](/quest/m3/route-health/meta.md) - what the standby observes +- [The switch](/quest/m2/multicast/switch.md) - the routes the standby chooses among diff --git a/quest/m3/route-health/meta.md b/quest/m3/route-health/meta.md new file mode 100644 index 00000000000..dbea1a38f6c --- /dev/null +++ b/quest/m3/route-health/meta.md @@ -0,0 +1,52 @@ +# [L] Metadata-only subscriptions + +## Goal + +Rust and JS subscribe to a track's group and frame progress without its +payload, through the shared subscription model and whichever lite wip is +current when this lands. A chain of relays forwards and aggregates that +demand without pulling the media behind it, and a metadata subscription +becomes a payload one in place, on the same stream. + +## Plan + +Decided 2026-10-13 and confirmed 2026-11-02 (maintainer), carried from the +quest deleted in [moq-dev/moq#8423](https://github.com/moq-dev/moq/pull/8423) +(`git show cd560bde5^:quest/m5/route-health/meta.md`): + +- ✅ A `meta: true` field on `Subscription`, carried by the existing + SUBSCRIBE: "SUBSCRIBE_META" was shorthand, not a new message. It is a live + stream: `largest()` on a payload-less standby is only the SUBSCRIBE_OK + snapshot, and a one-shot status races the active route. +- ✅ Each group stream marks its mode in its own header, so a stream opened + before a mode update is read under the mode it was opened in (rejected: + the mode only on the subscription, which races updates against open + streams). A metadata FRAME keeps its timestamp and declared size and + carries no bytes, so a real empty frame stays distinguishable. +- ✅ Promotion is a SUBSCRIBE_UPDATE on the same subscription, asking for + bytes from the reader's first missing frame, including positions it saw as + metadata. Metadata never satisfies a media reader, fills a cache as a + complete frame, or advances a resume cursor. +- ✅ A relay pulls payload upstream only for payload demand. Metadata demand + travels upstream as its own live metadata subscription, served from a + payload copy while the relay holds one. When only metadata demand remains, + the relay demotes its upstream subscription with an update, and drops it + when the last reader leaves. +- ✅ An older version never emulates it by fetching and discarding media: it + reports the mode unsupported. No moq-transport counterpart is needed. + +Mirror the field across Rust, JS, and every binding that exposes +`Subscription`. + +Tests: metadata-only traffic across several hops carries no payload; a relay +whose last payload reader leaves keeps forwarding headers to its metadata +readers; mixed demand; promotion and demotion during an open group; empty +frames; late frames in the old mode; resume at a position seen as metadata; +cancellation; refusal on an unsupported version. Sweep tracks, readers, and +relay depth for bytes, wakes, and memory. Update +`drafts/draft-lcurley-moq-lite.md` and both codecs together, and add the mode +to `just test interop --all`. + +Public API: `Subscription.meta` in Rust, JS, and the bindings. Wire: a +metadata mode in the current lite wip: the SUBSCRIBE field, a mode marker in +each group stream header, and payload-less FRAMEs. diff --git a/quest/m3/route-keep-next-hop.md b/quest/m3/route-keep-next-hop.md deleted file mode 100644 index d95dd202a06..00000000000 --- a/quest/m3/route-keep-next-hop.md +++ /dev/null @@ -1,53 +0,0 @@ -# [S] A relay keeps its next hop among equal routes - -## Goal - -A lite-07 relay that already routes to an origin through one neighbour stays -on it when another neighbour offers a route of the same rank: the same drain -state, priority, and metric. Today the lower key wins, so a relay moves for -nothing, and since routes name the nodes they crossed -([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)) every relay -downstream is sent a ROUTE_UPDATE for it. A test brings a link up beside an -equal one and counts the route messages sent on every other session: none. -The new link's own first advertisements do not count. - -## Plan - -Decided 2026-11-11 (maintainer, /quest-plan after #8487): ✅ plan it, in m3 -beside [split horizon](/quest/m3/route-split-horizon.md), since it trims -traffic and fixes nothing (rejected: dropping it). - -What it costs today, from #8487's measurements: a change of hops alone is -3.5% to 11% of route messages in the router's churn sweeps, about 3% of -control messages over real sessions, and on a 32-relay full mesh one link -coming up costs 592 route messages against 251 before the hops. - -[Cluster routing](/quest/m3/cluster-routing/README.md) already decided that -"a new connection alone does not outrank a different equal-cost route", so -this implements a settled rule. Keep the deterministic path hash among equal -routes for a request's first hop: that choice is per request and is not this -selection. - -- `selected` in `route/mod.rs` is a function of what the router holds, and - ranks by key last. Staying put makes it depend on the previous choice, so - keep the current next hop per identity and prefer it among equals. Mirror - it in `RouteHandoff.tla`, whose `Best` picks the lowest key, as the - `verify-route` skill says. -- Two routes of the same rank can still differ in whether they are - down-only, which decides whether an upstream neighbour is sent the route - at all. Stay put only among routes that agree on it, so a relay on a - down-only route still moves to an equal one it may export, and test that. -- Check that every router test that names the next hop it expects still - means what it says, rather than editing expectations to match. -- How a route is selected is being replaced - ([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27). - Apply this to the rule that stands when this starts. The scoped path router in - `rs/moq-net/src/route/path.rs` already stays on its next hop among equals. -- Measure with `Sim`'s `hop_updates` count and, if it has landed, - `mesh_sweep.rs`'s benchmark, before and after. - -Public API: none. Wire: none; behaviour only. - -## Related - -- [Relays split horizon per session](/quest/m3/route-split-horizon.md) - the other trim of route traffic diff --git a/quest/m3/route-split-horizon.md b/quest/m3/route-split-horizon.md deleted file mode 100644 index bc2e3fd269b..00000000000 --- a/quest/m3/route-split-horizon.md +++ /dev/null @@ -1,25 +0,0 @@ -# [S] Relays split horizon per session - -## Goal - -A relay never advertises a route back over the session it learned it from. -Since [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), each -neighbor holds two stream keys, so the router advertises a route back to its -source. That is loop-free, since the source refuses a route whose path names -it ([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27), but it -doubles route traffic between every pair of relays. - -## Plan - -Decided 2026-11-06 (maintainer, /quest-complete, a follow-up of #8023): ✅ -m3: lite-07 routing ships correct in the cut, and this trims its traffic -afterward. Not chosen: m2; m4. - -Suppress the advertisement per session rather than per neighbor, and measure -route messages between two relays before and after. - -Public API: none. Wire: none; behavior only. - -## Related - -- [A relay keeps its next hop among equal routes](/quest/m3/route-keep-next-hop.md) - the other trim of route traffic diff --git a/quest/m3/route-trace-validation.md b/quest/m3/route-trace-validation.md deleted file mode 100644 index a18518fcb96..00000000000 --- a/quest/m3/route-trace-validation.md +++ /dev/null @@ -1,47 +0,0 @@ -# [M] The exhaustive route search replaces RouteHandoff.tla - -## Goal - -One exhaustive check guards the lite-07 route rule, and it runs the compiled -code: the search over every delivery order on `path::Router` also checks -`Seen`, `StaleIsChased`, and `NoStaleReturn`, and `RouteHandoff.tla`, its -configs, and the route trace validation this quest first planned are gone, -with every skill, quest, and doc that pointed at the model pointing at the -search instead. - -## Plan - -Decided 2026-11-10 (maintainer, /quest-plan): trace validation as the -strongest guard against the route model drifting from the code -([moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447)), with resume, -reconnect, and subscribe each getting their own in m4. Moved to m3 on -2026-11-13 (maintainer, /quest-plan), since it waits on m2 work. - -Found 2026-11-14 by the [feature review](/quest/m3/release/feature-review.md): -the exhaustive route search -([moq-dev/moq#8729](https://github.com/moq-dev/moq/pull/8729), merged -2026-11-14) searches every delivery order on the compiled `path::Router`, -faster than TLC on the same bounds (0.44 s against 2 s on the triangle) and on -meshes TLC cannot reach. `RouteHandoff.tla` now mostly duplicates it: only -`Seen`, `StaleIsChased`, and `NoStaleReturn` stay outside it, and a search -over the compiled code cannot drift from it, which was this quest's whole -purpose for route. - -Decided 2026-11-14 (maintainer, pick on [moq-dev/moq#8764](https://github.com/moq-dev/moq/pull/8764)): -✅ keep the search and delete `RouteHandoff.tla` once #8729 lands (rejected: -keep the model and drop the search; keep both); #8729 has landed, so this is -ready. This replaces route trace -validation. - -- Fold the three rules into the search as properties it checks, and show - each one fails on a mutation that breaks it before deleting the model. -- Delete `rs/moq-net/verify/route/` and retarget or delete the - `verify-route` skill and its Coverage table, REVIEW.md's Models pointers, - and the quests that tell a PR to run `/verify-route`. -- The trace format and TLC recipe this quest was to build for the other - models now come from the first of their m4 quests: - [resume](/quest/m4/resume-trace-validation.md) builds them, and - [reconnect](/quest/m4/reconnect-trace-validation.md) and - [subscribe](/quest/m4/subscribe-trace-validation.md) reuse them. - -Public API: none. Wire: none. diff --git a/quest/m4/resume-trace-validation.md b/quest/m4/resume-trace-validation.md index 06c1f8731cf..e9d50cfe85d 100644 --- a/quest/m4/resume-trace-validation.md +++ b/quest/m4/resume-trace-validation.md @@ -17,7 +17,7 @@ can keep passing on code that has moved on. Milestone m4. [reconnect](/quest/m4/reconnect-trace-validation.md) and [subscribe](/quest/m4/subscribe-trace-validation.md) reuse: route's trace validation, which was to build them, gave way to the - [exhaustive route search](/quest/m3/route-trace-validation.md) (decided + exhaustive route search, which [the router](/quest/m2/multicast/router.md) now carries (decided 2026-11-14, maintainer, on [moq-dev/moq#8764](https://github.com/moq-dev/moq/pull/8764)). The sweep records each step as the model action it corresponds to (the `verify-resume` skill's Coverage table is the mapping) with the variables