From 5226196d20378bb58affe960d2a14f00cdea46bd Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 13:24:33 -0800 Subject: [PATCH 1/2] quest: plan cache-aware multicast routing Replaces lite-07's route layer before the cut: routes to forwarding nodes, evidence instead of holds and doubt, claims crossing trust boundaries by a chain, a measured pick of how warm copies are found, a mechanized convergence proof, and the metadata-only secondary in m3. Co-Authored-By: Claude Fable 5.2 --- quest/m2/README.md | 12 +- quest/m2/cluster-routes/README.md | 2 + quest/m2/cluster-routes/acceptance.md | 7 + quest/m2/cluster-routes/doubt-holds.md | 11 + quest/m2/cluster-routes/doubt-keepalive.md | 25 +- quest/m2/cluster-routes/held-after-close.md | 12 + quest/m2/cluster-routes/request-loop.md | 64 ----- quest/m2/cluster-routes/route-fanout.md | 13 + quest/m2/lite07-finalize.md | 12 +- quest/m2/multicast/README.md | 164 +++++++++++ quest/m2/multicast/accept.md | 46 +++ quest/m2/multicast/bakeoff.md | 90 ++++++ quest/m2/multicast/proof.md | 62 ++++ quest/m2/multicast/requests.md | 82 ++++++ quest/m2/multicast/router.md | 51 ++++ quest/m2/multicast/switch.md | 79 ++++++ quest/m2/multicast/warmth.md | 64 +++++ quest/m3/cache-routing/README.md | 297 ++++---------------- quest/m3/cache-routing/select.md | 86 +++--- quest/m3/cache-routing/transient-cycles.md | 41 --- quest/m3/cluster-routing/README.md | 9 + quest/m3/cluster-routing/federation.md | 9 + quest/m3/route-health/README.md | 61 ++++ quest/m3/route-health/failover.md | 41 +++ quest/m3/route-health/meta.md | 52 ++++ quest/m3/route-keep-next-hop.md | 53 ---- quest/m3/route-split-horizon.md | 25 -- quest/m3/route-trace-validation.md | 47 ---- quest/m4/resume-trace-validation.md | 2 +- 29 files changed, 981 insertions(+), 538 deletions(-) delete mode 100644 quest/m2/cluster-routes/request-loop.md create mode 100644 quest/m2/multicast/README.md create mode 100644 quest/m2/multicast/accept.md create mode 100644 quest/m2/multicast/bakeoff.md create mode 100644 quest/m2/multicast/proof.md create mode 100644 quest/m2/multicast/requests.md create mode 100644 quest/m2/multicast/router.md create mode 100644 quest/m2/multicast/switch.md create mode 100644 quest/m2/multicast/warmth.md delete mode 100644 quest/m3/cache-routing/transient-cycles.md create mode 100644 quest/m3/route-health/README.md create mode 100644 quest/m3/route-health/failover.md create mode 100644 quest/m3/route-health/meta.md delete mode 100644 quest/m3/route-keep-next-hop.md delete mode 100644 quest/m3/route-split-horizon.md delete mode 100644 quest/m3/route-trace-validation.md diff --git a/quest/m2/README.md b/quest/m2/README.md index 7fb904262da..44f31144ac8 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -9,6 +9,11 @@ complete it before cutting main into release. ## Plan +Decided 2026-11-15 (maintainer, /quest-plan on the whole route layer): ✅ the +cut waits on [cache-aware multicast](/quest/m2/multicast/README.md), which +replaces lite-07's route layer before it freezes (rejected: lite-08). The +routes chain stays the critical path, now through that epic. + Decided 2026-11-05 (maintainer, milestone re-cut): m2 holds the breaking half of what was "everything before the cut" (2026-10-20). Breaking covers every published surface, as the root [questline](/quest/README.md) defines: the @@ -222,10 +227,11 @@ maintainer asked that what remains be unblocked or punted, and picked these on [moq-dev/moq#8768](https://github.com/moq-dev/moq/pull/8768)). m2 keeps only what would need a breaking release had it landed right after the cut: -- ✅ Left m2: [the request loop](/quest/m2/cluster-routes/request-loop.md) +- ✅ Left m2: [the request loop](/quest/m2/multicast/requests.md) to m3's cluster-routing epic (a fixture and a fix, no wire), until it - returned to m2's cluster-routes epic on 2026-11-15 (maintainer, on - [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)); + returned to m2 on 2026-11-15 (maintainer, on + [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)) and became + the request quest of [cache-aware multicast](/quest/m2/multicast/README.md); [media consumers](/quest/m3/time-addressing-consumers.md) to m3 (its API breaks ride the model; dropping hang's markers is tolerated by readers); [moq-transport time addressing](/quest/m4/ietf-time-addressing.md) to m4 diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index f8dc4710b0a..f8b0b7ea5a3 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -82,6 +82,8 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 30. ✅ Decided 2026-11-12 (maintainer, /quest-audit before the cut), after a survey of BGP, EIGRP, and their successors against item 27: most of the path rule already matches that prior art (AS_PATH loop detection, root-cause withdrawal, graceful restart, End-of-RIB, outbound route filtering, and item 29's Only-To-Customer flag). Three gaps are taken, all in [the switch](https://github.com/moq-dev/moq/pull/8731) and [acceptance](/quest/m2/cluster-routes/acceptance.md): **a dead-hop message on a scoped session applies only to the routes heard on that session and casts no doubt**, since in BGP a neighbour withdraws only what it sent, and a scoped peer (a federated relay, later a browser) could otherwise name a core hop it only read in a path and drop healthy routes outside its scope; **holds are bounded in time**, a neighbour's missing answer counting as nothing after a deadline, and a session whose announce stream has stopped draining is closed, which avoids EIGRP's stuck-in-active and BGP's zombie routes (RFC 9687), and is safe because a hold protects presence, not loop freedom; and **three tests before any design change**: neighbours noticing a dead relay up to the 3 s keep-alive apart (to size the doubt timer, measured in hops of delay today), a three-relay priority dispute wheel (RFC 4264's wedgies, since priority ranks above cost), and a primary that fails and returns. No wire bytes change; the first defines what the dead-hop message means. Rejected: declaring the link relation in SETUP and refusing leaked routes on receipt (RFC 9234), since the accepting side already knows the relation from its grant and it adds wire for a misconfiguration; MRAI and route-flap damping, which exist because BGP delays announcements; link state; per-hop signatures. +31. Reopened 2026-11-15 (maintainer, /quest-plan on the whole route layer): ✅ **this rule is the baseline, and [cache-aware multicast](/quest/m2/multicast/README.md) replaces it on lite-07-wip before the cut.** An audit of what is built found requirements the path rule does not meet: its exhaustive checking stops at 3 relays and rests on a timer that outlasts every stale offer, route state follows publishers, nothing prefers a warm copy, and a stalled route is never left. Superseded there: item 18 (every publisher an origin node), item 19 (ROUTE_HOLD), item 26's route per scope, item 27's doubt and remembered dead hops, the Route Key and qualified Node IDs of the contract below, and item 14. Standing: the path with its self-naming rule and the withdrawal that names a dead hop (item 27), revisions and floors, one announce stream per side with its interest set (items 21 to 24), the sticky down-only flag (item 29), and drain, then priority, then cost. This epic's children still land: they are what the [bake-off](/quest/m2/multicast/bakeoff.md) measures against. Rejected: repairing this rule in place; freezing lite-07 with it and redesigning in lite-08. + The route layer's core landed in [moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676): a crate-private, sans-IO `Router` in `rs/moq-net/src/route/`, generic over the diff --git a/quest/m2/cluster-routes/acceptance.md b/quest/m2/cluster-routes/acceptance.md index dbaf6f6cae1..003f627db02 100644 --- a/quest/m2/cluster-routes/acceptance.md +++ b/quest/m2/cluster-routes/acceptance.md @@ -27,6 +27,12 @@ mount continuity (`mount.rs`), and operation-count regressions (`route/tests/cost.rs`, `route/node/tests.rs`). They found the flaws the quests under Required fix; the tests that pin each flaw say so in their docs. +Reordered 2026-11-15 (maintainer, /quest-plan on +[cache-aware multicast](/quest/m2/multicast/README.md)): the retirement below +waits for [the switch](/quest/m2/multicast/switch.md), which rewrites these +fixtures' assertions for the new rule and deletes the Required quests it +makes moot. + What remains: - Retire the quarantine, the dedup, and the hold once the Required quests @@ -48,6 +54,7 @@ What remains: ## Required +- [The switch](/quest/m2/multicast/switch.md) - the rule whose fixtures replace the mitigations, once the new route layer is on the wire - [Held after close](/quest/m2/cluster-routes/held-after-close.md) - a relay whose own session closed shows its held claims unavailable, not offline - [Doubt holds](/quest/m2/cluster-routes/doubt-holds.md) - a relay keeps holding while doubt alone keeps it off its routes - [Route fan-out](/quest/m2/cluster-routes/route-fanout.md) - a route change costs only the streams and broadcasts it touches diff --git a/quest/m2/cluster-routes/doubt-holds.md b/quest/m2/cluster-routes/doubt-holds.md index 2898d468152..bb494ab31df 100644 --- a/quest/m2/cluster-routes/doubt-holds.md +++ b/quest/m2/cluster-routes/doubt-holds.md @@ -8,6 +8,13 @@ go while its neighbours answer that they have nothing. ## Plan +Reordered 2026-11-15 (maintainer, /quest-plan on [cache-aware +multicast](/quest/m2/multicast/README.md)): this waits on the +[bake-off](/quest/m2/multicast/bakeoff.md). The new rule has neither holds nor +doubt, so nothing would be left for this to fix. If the new rule goes ahead, +[the switch](/quest/m2/multicast/switch.md) deletes this quest with the +fixtures that pin it; if it does not, this starts as written. + Kept 2026-11-15 (maintainer, /quest-complete on [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ keep it in m2 (rejected: drop; moving it to m3). @@ -34,6 +41,10 @@ neighbour that holds only because of doubt should answer with ROUTE_HOLD, not nothing. Check the rule in `RouteHandoff.tla` with the `verify-route` skill before changing `rs/moq-net/src/route/path.rs`. +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - decides whether the hold and the doubt this fixes remain + ## Related - [Routes and announces](/quest/m2/cluster-routes/README.md) - Plan item 27 and the hold rule diff --git a/quest/m2/cluster-routes/doubt-keepalive.md b/quest/m2/cluster-routes/doubt-keepalive.md index 859b9212d17..be6a172a659 100644 --- a/quest/m2/cluster-routes/doubt-keepalive.md +++ b/quest/m2/cluster-routes/doubt-keepalive.md @@ -7,6 +7,20 @@ apart its neighbours notice the death within the keep-alive. ## Plan +Reordered 2026-11-15 (maintainer, /quest-plan on [cache-aware +multicast](/quest/m2/multicast/README.md)): this waits on the +[bake-off](/quest/m2/multicast/bakeoff.md). The maintainer deferred how the +doubt covers a silent death on 2026-11-15, "not yet sure about the doubt +design", and decided the same day that evidence replaces the doubt. If the new +rule goes ahead, [the switch](/quest/m2/multicast/switch.md) deletes this +quest with the fixtures that pin it; if it does not, this starts as written. + +If the doubt stays, the decision returns to the maintainer: (a) the doubt +lasts at least the configured idle timeout, so it covers the spread (the +recommendation: no wire change, and a relay already knows its idle timeout); +(b) relay keep-alives shorten below the doubt, which costs traffic on every +relay link; (c) accept the window and keep the test pinning it. + Kept 2026-11-15 (maintainer, /quest-complete on [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ keep it in m2 (rejected: drop; moving it to m3). @@ -21,16 +35,9 @@ dead `X` at 1 s, holds again at 1.5 s, and moves onto `R3`'s route through at 3 s. Readers see each move as available for a moment, and a request in that window goes to a dead relay. A 1 s doubt does not cover a 3 s spread. -## Waiting +## Required -- Maintainer: how the doubt covers a silent death. Options: (a) the doubt - lasts at least the configured idle timeout, so it covers the spread (the - recommendation: no wire change, and a relay already knows its idle - timeout); (b) relay keep-alives shorten below the doubt, which costs - traffic on every relay link; (c) accept the window and keep the test - pinning it. Check with `/quest-grill` or the PR that picks it up. The - maintainer deferred this decision on 2026-11-15, not yet sure about the - doubt design. +- [The bake-off](/quest/m2/multicast/bakeoff.md) - decides whether the doubt this sizes remains ## Related diff --git a/quest/m2/cluster-routes/held-after-close.md b/quest/m2/cluster-routes/held-after-close.md index 829a4336926..561e0d9edf0 100644 --- a/quest/m2/cluster-routes/held-after-close.md +++ b/quest/m2/cluster-routes/held-after-close.md @@ -9,6 +9,14 @@ route recovers. ## Plan +Reordered 2026-11-15 (maintainer, /quest-plan on [cache-aware +multicast](/quest/m2/multicast/README.md)): this waits on the +[bake-off](/quest/m2/multicast/bakeoff.md). The new rule has no hold: a reader +debounces unavailable to offline locally, which answers the same symptom. If +the new rule goes ahead, [the switch](/quest/m2/multicast/switch.md) deletes +this quest with the fixtures that pin it; if it does not, this starts as +written. + Kept 2026-11-15 (maintainer, /quest-complete on [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ keep it in m2 (rejected: drop; moving it to m3). @@ -32,6 +40,10 @@ move it, unavailable, onto a stream that still hears the held route, or show it through the node itself, before the closing stream withdraws it. Turn the three pinned fixtures into the contract's held-then-available sequence. +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - decides whether the hold this shows remains + ## Related - [Routes and announces](/quest/m2/cluster-routes/README.md) - the hold rule and "Presence and availability are separate" diff --git a/quest/m2/cluster-routes/request-loop.md b/quest/m2/cluster-routes/request-loop.md deleted file mode 100644 index abf8f591f7d..00000000000 --- a/quest/m2/cluster-routes/request-loop.md +++ /dev/null @@ -1,64 +0,0 @@ -# [M] A request loop heals once routes settle - -## Goal - -A subscription that loops between relays while routes settle reaches the -publisher once the next hops stop looping, instead of hanging silently or -ending with `Unroutable` while its relay shows the broadcast available. -Under the path rule, selected next hops can loop for a moment -([epic](/quest/m2/cluster-routes/README.md) Plan item 27). A request -forwarded around that loop meets a relay already requesting the same track -upstream, and aggregation joins it to that request. Nothing errors, so a -pending request that never follows its route's change waits forever with -no data and no failure. - -## Plan - -Decided 2026-11-15 (maintainer, on -[moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ move it -back to m2, under the cluster-routes epic (rejected: keep it in m3). Relays -that show a broadcast available while ending readers' subscriptions should -block the milestone. - -Moved from m2 to m3 2026-11-14 (the m2 triage before the cut, under -/quest-afk, for the maintainer's review): ✅ m3, under the cluster-routing -epic (rejected: m2). A fixture and, if it hangs, a fix to where a relay's -pending request goes, with no wire field (the hop count was rejected below), -so it lands after the cut as a bug fix in shipped code. - -Decided 2026-11-13 (maintainer, /quest-plan on the BGP failure modes): - -- ✅ A fixture on real relays, with no TLA+ (rejected: adding pending - requests and aggregation to `RouteHandoff.tla`). The model's - `RequestsLoopFree` already checks next hops; this checks that the - relay's request state follows them. -- ✅ If the fixture hangs, a relay's pending upstream request follows its - route when the selected next hop changes, even before the request is - answered, so the loop clears with the routes (rejected: a hop count on - every request that fails one at the limit, which adds a field to SUBSCRIBE, - FETCH, and the other requests and turns a loop that would heal into an - error the subscriber sees). Epic Plan item 27 already rejected keeping a - request from going back through the relay that sent it. - -Found again by [acceptance](/quest/m2/cluster-routes/acceptance.md) once -[moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) merged: -`route_failover::live_mesh_link_failure_keeps_presence` cuts all but one of a -publishing relay's links on a 34-relay mesh, and 3 readers on relays that -stayed available end with `Unroutable` instead of hanging, all for one -publisher. Each such relay moved onto a neighbour's direct route that the -neighbour's own close made stale; the replaced doubt rule held them and -ended none. The test pins those readers: flip it once a pending request -follows its route and they read on. - -The fixture runs real lite-07 relays with mocked time. Form the loop the -churn sweeps found (two unrelated failures at once, so two relays briefly -route a publisher through each other), subscribe at both while it lasts, -then let the routes settle. Both subscribers receive the next group, and no -upstream request is left pending. Cover a request answered before the loop -and one still unanswered when it forms. Check which relay state carries the -pending request (the front in `rs/moq-net/src/model/front.rs`, and how -`route/node.rs` `home`s it) before choosing where the fix goes. - -## Related - -- [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) - its doubt rule lets a publisher leaving three or more relays at once loop a request until the news arrives diff --git a/quest/m2/cluster-routes/route-fanout.md b/quest/m2/cluster-routes/route-fanout.md index 09a134c7092..3b474bce219 100644 --- a/quest/m2/cluster-routes/route-fanout.md +++ b/quest/m2/cluster-routes/route-fanout.md @@ -8,6 +8,15 @@ remembered for sessions nothing was learned on. ## Plan +Reordered 2026-11-15 (maintainer, /quest-plan on [cache-aware +multicast](/quest/m2/multicast/README.md)): this waits on the +[bake-off](/quest/m2/multicast/bakeoff.md). Its first and third findings are +in `path::Router`'s indexes and its doubts and dead hops, which the new router +replaces; the second, per-broadcast rewrites in `route/node.rs`, the +[router](/quest/m2/multicast/router.md) takes as a requirement. If the new +rule goes ahead, [the switch](/quest/m2/multicast/switch.md) deletes this +quest with the fixtures that pin it; if it does not, this starts as written. + Kept 2026-11-15 (maintainer, /quest-complete on [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ keep it in m2 (rejected: drop; moving it to m3). @@ -43,6 +52,10 @@ once fixed: Build on [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734), which also edits `path.rs`. +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - decides whether the path router these findings are in remains + ## Related - [Route wakes](/quest/m4/route-wakes.md) - origin-side wakes per route change diff --git a/quest/m2/lite07-finalize.md b/quest/m2/lite07-finalize.md index 8f1c89affa5..858eac6c026 100644 --- a/quest/m2/lite07-finalize.md +++ b/quest/m2/lite07-finalize.md @@ -129,7 +129,7 @@ maintainer picked these on [moq-dev/moq#8768](https://github.com/moq-dev/moq/pull/8768)), applying the maintainer's rule that m2 keeps only what would need a breaking release had it landed right after the cut: -- ✅ [The request loop](/quest/m2/cluster-routes/request-loop.md) moved +- ✅ [The request loop](/quest/m2/multicast/requests.md) moved to m3 and no longer gates this: a fixture and a fix with no wire change. It returned to m2 on 2026-11-15 and still does not gate this. - ✅ [Route layer acceptance](/quest/m2/cluster-routes/acceptance.md) stays @@ -155,8 +155,18 @@ would need a breaking release had it landed right after the cut: both publishers already name a Group Stream the subscriber stopped in SUBSCRIBE_DROP, which the draft now states, even after a reliable reset. +Decided 2026-11-15 (maintainer, /quest-plan on the whole route layer): ✅ +lite-07 carries the redesigned route layer of +[cache-aware multicast](/quest/m2/multicast/README.md), and the cut waits +for it (rejected: freezing lite-07 with the path rule and redesigning in +lite-08; freezing lite-07 with no route layer). That redesign removes the +Route Key Parameter, so the 2026-11-10 wait on its symbolic model and the +route key lane's early signal above end when +[the switch](/quest/m2/multicast/switch.md) lands. + ## Required +- [Cache-aware multicast](/quest/m2/multicast/README.md) - lite-07 carries the redesigned route layer: routes to forwarding nodes, no holds or doubt, claims crossing trust boundaries by a chain, and no Route Key - [Routes and announces](/quest/m2/cluster-routes/README.md) - lite-07 carries the route layer: ROUTE per origin node and path-less ANNOUNCE, with the native hop list gone (only the temporary lite-06 compatibility chain remains) - [Lite-07 by start](/quest/m2/time-addressing/lite07.md) - lite-07 carries group starts, previous-start links, GROUP_END, and time bounds instead of sequence numbers - [@moq/net by start](/quest/m2/time-addressing/js.md) - JS speaks the same lite-07 wire diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md new file mode 100644 index 00000000000..6a5ed5576f9 --- /dev/null +++ b/quest/m2/multicast/README.md @@ -0,0 +1,164 @@ +# Cache-aware multicast routing + +## Goal + +moq-net gets one routing design, proven and measured before it replaces +lite-07's route layer. A subscription joins the nearest node already carrying +the same Instance, so no link carries a group twice, and moves between routes +of equal Instance without the reader noticing. State and messages grow with +forwarding nodes and demanded broadcasts, not publishers. There is no leader, +global view, or single point of failure, on CDN tiers and on a self-forming +mesh alike. + +Non-goals: choosing, dialing, and pricing links from measured quality, which +is Kixel's; browsers as transit; stitching different Instances together; and +moq-transport routing parity beyond each node using its nearest publisher. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan), after an audit of the lite-07 +route layer as built, the history of about thirty routing attempts since +2023, and a survey of the literature. +[Routes and announces](/quest/m2/cluster-routes/README.md) holds the rule +this replaces and its measurements. + +The maintainer's requirements, which every child is judged against: + +1. A subscription resumes only onto equal identity. +2. A publisher may hold two connections (a GOAWAY handover, redundant + ingest), and routes switch between them seamlessly. +3. Some connections do not proxy: an edge passes its clients to its cores + and its cores' routes to its clients, never one core's routes to another, + and a leaf, such as a relay on a camera, advertises only what its host + serves. +4. The specification stays simple, reusing messages and concepts. +5. Traffic grows as little as possible with the cluster. +6. No single point of failure. +7. It is proven to work. +8. Routing prefers warm caches: a route has a cold cost and a warm cost, and + routes coalesce onto a node that says it carries a broadcast, so no link + carries the same data twice. +9. Messages scale with the cluster: streams of messages, never one blob. +10. One stream carries what must stay ordered. +11. moq-transport is supported to a degree: identity does not carry, but two + publishers announcing one name on opposite sides of the network work. +12. No global view. With competing claims each node uses its closest, and + nothing elects a leader. +13. A secondary route may carry metadata only, so a stalled primary is found + without waiting on QUIC keep-alives. +14. No approach is assumed: earlier attempts are evidence, not constraints. + +What the audit found of today's layer: 6 and 12 hold, and 1 holds with no +model covering it; 4, 8, and 13 do not hold (16 message types on the announce +stream and about 57 concepts against lite-06's 5 messages, no cache term in +any ranking, and a payload stall on live sessions that is never recovered); +the rest hold in part. Exhaustive checking stops at 3 relays and one session +close, safety only, and rests on a timer that outlasts every stale offer, +which a 1 s doubt does not when a dead relay's neighbours notice up to 3 s +apart. Route state and churn follow publishers. + +Settled: + +1. ✅ **Replace, not repair.** Today's ROUTE, DEAD, hold, and doubt wire is a + candidate like any other (rejected: keeping the path router as the base + and adding cache awareness and the metadata secondary on top, which leaves + requirement 4 unmet and state following publishers; moq also owning link + formation and pricing for the mesh). +2. ✅ **lite-07 carries it, and the cut waits** (rejected: lite-08, which + publishes a wire we expect to replace and makes two route layers + interoperate for a release; lite-07 without a route layer, a large strip + that leaves lite-07 clusters with lite-06's hunting). The route stream and + the acceptance fixtures still land: they are the baseline the + [bake-off](/quest/m2/multicast/bakeoff.md) measures against, and one + announce stream per side is wanted by any design. The quests that only + polish holds, doubt, or the path router's indexes wait on the bake-off. +3. ✅ **A routing node is a node that forwards and is trusted to; claims, + not routes, cross a trust boundary.** Every CDN relay and every mesh + device routes under its own Node ID. An endpoint that never forwards (an + app, an encoder, a browser) is local to the relay it connects to, which + announces its broadcasts under the relay's Node ID. Meshes sharing a fleet + credential are one routing domain. Across a scoped grant the receiving + node announces the claim again under its own Node ID with a chain of the + nodes that did so, and refuses a claim whose chain names it, which is the + Lite-06 Chain made general. The maintainer first answered "we need every + publisher so meshes can communicate with other meshes"; the chain covers + that across any number of scoped hops, where a qualified identity crosses + one. Redundant ingest is two relays claiming one Instance. This removes + the Route Key Parameter with its proof, qualified Node IDs, and routes per + prefix, and reverses Plan items 18 and 26 of + [Routes and announces](/quest/m2/cluster-routes/README.md) (rejected: + every publisher an origin node, as today; measuring both, which doubles + the prototype). +4. ✅ **Evidence, not timers.** The route layer is a plain path vector with + the dead-hop withdrawal as an accelerator: no ROUTE_HOLD, no doubt, and no + dead hop remembered past the routes that name it. Routing state is a + hint. A request is confirmed only once its upstream is, data is identified + by Instance, track, group, and frame, and a stale or stalled route is left + on evidence ([requests](/quest/m2/multicast/requests.md), + [route health](/quest/m3/route-health/README.md)). What a hold protected, + a claim flickering while routes settle, becomes a reader's local debounce + from unavailable to offline, never on the wire. This reverses items 19 and + 27's doubt (rejected: keeping holds and doubt with the doubt tied to the + idle timeout; a doubt lifted by an epoch the surviving node floods). +5. ✅ **Three ways to find a warm copy are measured before one is picked** + ([warmth](/quest/m2/multicast/warmth.md)): joins that stop at the first + node carrying the Instance and nothing more; that plus a neighbour saying + on each SUBSCRIBE answer whether it carries the Instance and at what cost; + and carrying nodes advertising it. One constraint binds all three: a warm + cost is strictly positive and summed from the source along the path the + data takes. A warm discount lets two warm relays prefer each other, which + happened four times here. +6. ✅ **One mark per link.** A route learned over a marked link is never sent + over another marked link, as `upstream` works today. An edge marks its + core links and a leaf marks every relay link (rejected: a strict mark that + sends only what the node originates, which ends a LAN peer publishing + through a neighbour whose uplink survived; both switches). +7. ✅ **The proof is mechanized, with TLA+ and sweeps beside it** + ([proof](/quest/m2/multicast/proof.md); rejected: a written proof with + property tests, the kind of argument that passed review here and failed + for the echo, the deferred echo, and belief by distance; TLA+ and sweeps + alone, which say nothing about an arbitrary topology). +8. Carried from the route-health epic deleted 2026-11-09 + ([moq-dev/moq#8423](https://github.com/moq-dev/moq/pull/8423)), whose + decisions the maintainer made 2026-10-13 and 2026-11-02 and nothing + reversed: metadata-only delivery is a mode of SUBSCRIBE, never its own + message; a stall is found by comparing the two routes, never by silence; + and a demoted route never takes a track back while the working one + delivers. + +The shape those decisions give, for the children to confirm or improve; each +departure is a decision to list in its PR: + +- **Identity.** A path and an Instance name content, and a request names the + Instance. A relay holds one upstream subscription per Instance and track, + whichever origin node its downstream requests named, so a join stops at the + first node already carrying it. +- **Reachability.** One route per forwarding node: its path, drain, priority, + and cost. A node never uses a route whose path names it. Rank is not + draining, then the lowest priority on the path, then the summed cost, each + link adding at least 1, so extending a route never improves it. +- **Claims.** A prefix, Instance, Epoch, and cost, announced by a forwarding + node for itself and its local endpoints, ordered by the node's revision and + certified by floors as today, which measured no revival and no false END. +- **Boundaries.** The chain of item 3. +- **Requests.** A request names the origin node its first hop chose, so + relays that rank differently never loop it, and fails closed. + +Lessons from the history that bind every child: + +- A fact carries its own order: a revision, a path, a withdrawal that names + its cause. Holds, quarantines, hold-downs, and split horizon never stopped + a stale copy reviving a withdrawn fact. +- Every input to a rank travels in the offer and never improves when a route + is extended. An input that depends on demand needs its own argument. +- Each side says everything that must stay ordered on one stream. +- A failure costs messages per session or per forwarding node, never per + broadcast or per publisher behind it. +- Nothing ships unexercised: a loop check that never fired cost 32 bounces + per announce in production. + +## Related + +- [Routes and announces](/quest/m2/cluster-routes/README.md) - the rule this replaces, its contract, and its measurements +- [Cache-aware routing](/quest/m3/cache-routing/README.md) - implements the warmth pick +- [Route health](/quest/m3/route-health/README.md) - the metadata-only secondary diff --git a/quest/m2/multicast/accept.md b/quest/m2/multicast/accept.md new file mode 100644 index 00000000000..59d0e01d04e --- /dev/null +++ b/quest/m2/multicast/accept.md @@ -0,0 +1,46 @@ +# [L] The cases no test covers run on real sessions + +## Goal + +On real lite-07 sessions with mocked time, the switched route layer passes +the cases the 2026-11-15 audit found neither a model nor a test for, and the +PR reports the measured interruption for each kind of failure: + +- two publishers claiming one path from opposite sides of a mesh, where + each relay uses its nearest and none loops; +- one publisher ingesting on two relays under one Instance, losing either, + with readers across the mesh reading on; +- a publisher handed over by GOAWAY to another relay, with no group missed + or repeated; +- moq-transport and lite-06 publishers through the route layer, including + two announcing one namespace on different relays; +- three meshes joined by scoped links, in a chain and in a ring, where no + claim circles and a claim ended at its source ends everywhere; +- an edge that marks its core links and a leaf that marks every link, each + passing only what requirement 3 allows; +- a relay that dies silently while its neighbours notice seconds apart; +- failures at the same instant on a mesh no exhaustive check reaches; +- and no group crossing one link twice for one subscription. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): the redesign is accepted on +the implementation before lite-07 freezes, as the maintainer decided for the +path rule on 2026-11-14. + +[Route layer acceptance](/quest/m2/cluster-routes/acceptance.md) owns the +fixtures that exist, which the [switch](/quest/m2/multicast/switch.md) +rewrites for the new rule; this adds the cases above beside them in +`rs/moq-net/tests/integration/`. Impair links with bufferbloat, not random +loss. Where a case needs `js/net`, run it in `just test interop --all`. + +A case that fails is a flaw to fix on lite-07-wip, in the quest that owns +the code, before the cut. + +Public API: none. Wire: none expected. + +## Required + +- [The switch](/quest/m2/multicast/switch.md) - the sessions these cases run on +- [Requests](/quest/m2/multicast/requests.md) - a request fails closed and follows its Instance +- [The proof](/quest/m2/multicast/proof.md) - what these cases need not show diff --git a/quest/m2/multicast/bakeoff.md b/quest/m2/multicast/bakeoff.md new file mode 100644 index 00000000000..c6350036dd9 --- /dev/null +++ b/quest/m2/multicast/bakeoff.md @@ -0,0 +1,90 @@ +# [XL] The new route rule is measured against today's + +## Goal + +A test-only prototype of the [epic](/quest/m2/multicast/README.md)'s route +rule runs beside `route::path::Router` on the same meshes and failure +schedules, and the PR reports for both what each requirement costs: messages +and retained state as forwarding nodes, publishers, broadcasts, and links +grow; the time to a settled route and the time a reader goes unfed after each +kind of failure; and every loop, stale route, false END, and revival seen. +The maintainer picks from those numbers whether the rule goes ahead as +decided and with what changes, and the PR amends the +[router](/quest/m2/multicast/router.md), +[switch](/quest/m2/multicast/switch.md), and +[proof](/quest/m2/multicast/proof.md) quests with what it learned. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): the rule's direction is settled +in the epic (items 3, 4, and 6); this measures it before anything is built on +it, as [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533) measured +paths against Babel. + +Build on `rs/moq-net/src/route/tests/`: `Sim` drives a router over a mesh +with messages held and delivered out of step, `compare.rs` ran two routers +over one schedule, and `search.rs` walks every delivery order on a small +mesh. The prototype stays under `tests/`; nothing ships from this PR. + +The rule to prototype: + +- A route per forwarding node, ranked not draining, then the lowest priority + on its path, then cost. A node never uses a route whose path names it. +- A closed session is named once per neighbour, and every node drops every + route through that hop. No moved list, no hold, no doubt, and a dead hop + is forgotten once no heard route names it. +- An endpoint's claims belong to the relay it connects to. A claim crossing + a scoped grant is announced again with the chain of the epic's item 3. +- One mark per link, as `upstream` is today. +- A request is confirmed only once its upstream is, and one fed around a + loop reports no success. Count what that costs readers: this is the + accepted price of having no timer. + +Shapes, each at more than one size: + +- CDN tiers: edges on two cores per region with the cores meshed, up to at + least 32 relays and 123 links at diameter 4. +- Meshes: a ring with log N chords, a sparse random mesh, a clique, and a + mesh with leaves that mark every link. +- Two meshes joined by one scoped link and by two, and three meshes in a + chain and in a ring, where the chain must stop a claim circling. +- One broadcast per publisher, the case where routes per publisher helped + least. + +Failures: a clean close; a silent death its neighbours notice up to the idle +timeout apart; a relay with many neighbours dying; a link that flaps faster +than any timer; a publisher on two relays losing one, then both; a publisher +moving; a GOAWAY drain; a partition that heals; and unrelated failures at the +same instant. + +Today's numbers, to beat or to explain: + +| Case | Today | +| --- | --- | +| A relay with several neighbours dies, 24 relays | 124 route messages, no loop | +| A link closes in a clique, one publisher behind it, then eight | 44, then 198 messages | +| A detour, settled | 12 hops of delay, 6.2 avoiding the far end | +| A publisher moves to another relay, clique of 24 | 1,100 ms dark | +| Churn, 120,000 steps | 32 steps with a route loop | +| A silent death noticed over 3 s | three relays each move onto a dead route twice | +| A link comes up on a 32-relay full mesh | 592 route messages | +| One route change beside 1,000 unrelated streams | 1.09 ms, against 9.8 µs beside 10 | + +The known risk is the reason doubt exists. Paths with no dead-hop message +hunted for 65,219 messages when a publisher left a clique of 12, and with the +dead hop but no doubt, selected routes looped when a relay with several +neighbours died. The epic accepts a loop among selected routes while +requests fail closed; show what it costs in messages and unfed readers. If +that is worse than today, try evidence before any timer: for one, rank every +route through the far end of a closed session as draining until that node's +own word about the session arrives, which reuses the drain and needs no +clock. A timer in the rule goes back to the maintainer. + +Keep the gate fast: the fixtures that pin a property run in the merge gate, +and the long sweeps run nightly from `test/route-sweep.sh`. + +Public API: none. Wire: none. + +## Related + +- [Routes and announces](/quest/m2/cluster-routes/README.md) - where today's numbers were measured diff --git a/quest/m2/multicast/proof.md b/quest/m2/multicast/proof.md new file mode 100644 index 00000000000..0343dc184ac --- /dev/null +++ b/quest/m2/multicast/proof.md @@ -0,0 +1,62 @@ +# [M] The route rule is proven to converge on any topology + +## Goal + +The route rule's algebra is instantiated against the published Agda library +for path-vector convergence, with every axiom discharged: how a node ranks +routes, how a route changes crossing a link, the mark's export filter, the +rule that a path naming its receiver is invalid, and the chain across a trust +boundary. The theorem then applies on any topology: from any starting state, +under loss, reordering, and duplication, the nodes reach one stable state. A +Rust property test asserts that the router's own rank and extend functions +meet the same axioms, a nightly lane checks the Agda, and +`rs/moq-net/verify/README.md` says what is proven and what is not. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): ✅ mechanized, checked nightly +and not in the merge gate, with TLA+ and sweeps beside it (rejected: a +written proof with property tests; TLA+ and sweeps alone). Today's checking +is exhaustive only at 2 to 3 relays with one failure, and the maintainer +ended the unbounded TLA+ attempt on 2026-11-13. + +The theorem is Daggitt and Griffin's, "Formally Verified Convergence of +Policy-Rich DBF Routing Protocols" ([arXiv 2106.01184](https://arxiv.org/abs/2106.01184)), +Theorem 5: an increasing path algebra always converges. Its library is +[MatthewDaggitt/agda-routing](https://github.com/MatthewDaggitt/agda-routing); +pin it and Agda through Nix. + +What to show, each a lemma about one message handler: + +- Selection is a total order with a fixed tie-break. +- No change across a session makes a route preferred to what it was: adding + the link's cost, taking the lower priority, keeping a drain, appending the + hop. A filter maps a route to invalid. +- A route whose path names the receiver is invalid, and a path grows by + exactly the hop it crossed. +- What a node advertises is a function of the latest it heard from each + neighbour. The dead-hop withdrawal is extra: show it removes only routes + the iteration would remove, so it only arrives sooner. +- The chain is the same algebra at the grain of trust boundaries. + +State plainly what the theorem does not give, and where each is covered: + +- No loop at every instant: [requests](/quest/m2/multicast/requests.md) fail + closed instead. +- A cost that depends on who subscribed: + [warmth](/quest/m2/multicast/warmth.md) argues it separately. +- Liveness under churn that never stops. +- That the tree is loop-free once routes settle: one paragraph, since next + hops toward one origin node at a fixed point cannot cycle and a request + names its origin node. + +TLA+ stays for the session, subscription, and failover state machines, where +orderings matter. `RouteHandoff.tla` describes the rule being replaced and +goes with it in the [switch](/quest/m2/multicast/switch.md). Report how long +the Agda check takes. + +Public API: none. Wire: none. + +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - fixes the rank and the transformations to prove diff --git a/quest/m2/multicast/requests.md b/quest/m2/multicast/requests.md new file mode 100644 index 00000000000..c27dafbb5b2 --- /dev/null +++ b/quest/m2/multicast/requests.md @@ -0,0 +1,82 @@ +# [L] A request treats routing as a hint + +## Goal + +No subscription, fetch, or track request depends on routing state being +fresh. A relay confirms a request only once its upstream has, and serves +every request for one Instance and track from one upstream subscription, +whichever origin node the requests named. A request whose route ends, is +refused as unroutable, or loops moves to another route of its Instance +instead of ending the track, hanging, or ending with `Unroutable` while its +relay shows the broadcast available. No group crosses one link twice for one +subscription, and the old and new copies of a move overlap for a bounded +time. Tests on real lite-07 relays with mocked time pin each. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): this is the data half of +"evidence, not timers" ([epic](/quest/m2/multicast/README.md) item 4). With +no hold and no doubt, selected routes may loop or go stale while news is in +flight, and that is safe only if a request fails closed and follows its +Instance. It started as the request loop quest, which this absorbs. + +Decided 2026-11-15 (maintainer, on +[moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ m2 +(rejected: m3). Relays that show a broadcast available while ending readers' +subscriptions should block the milestone. + +Decided 2026-11-13 (maintainer, /quest-plan on the BGP failure modes): + +- ✅ A fixture on real relays, with no TLA+ for pending requests and + aggregation (rejected: adding them to the route model). +- ✅ A relay's pending upstream request follows its route when the selected + next hop changes, even before the request is answered, so a loop clears + with the routes (rejected: a hop count on every request that fails one at + the limit, which adds a field to SUBSCRIBE, FETCH, and the other requests + and turns a loop that would heal into an error the subscriber sees). + +Why a request cannot find its own loop: once relays aggregate requests, one +fed around a loop meets a relay already asking upstream and joins it, so +neither a nonce nor a carried path ever returns to its sender. Nothing +errors and nobody is fed. What holds instead: a pending request confirms +nothing downstream, and every relay re-asks when its selected route changes. + +Hazards to reproduce first, from the 2026-11-15 audit and the acceptance +fixtures: + +- `route_failover::live_mesh_link_failure_keeps_presence` cuts all but one + of a publishing relay's links on a 34-relay mesh, and 3 readers on relays + that stayed available end with `Unroutable`. Flip it once they read on. +- An upstream that loses its own route resets the subscription `Unroutable` + and sends the route news on another stream. If the reset is handled first, + the front aborts a track that has delivered and never splices it again + (`rs/moq-net/src/model/front.rs` `track_ended`), while + `rs/moq-net/src/model/resume.rs` treats an unroutable copy as waiting for + another route. Found by reading; reproduce it two hops below a live + upstream that loses its route, and for a fresh TRACK naming a route that + just ended. +- During a move the old and new upstream subscriptions overlap with no time + bound if the old route keeps winning, and the newest group is asked for + again. +- Two route entries for one claim through one session send two SUBSCRIBEs + that differ only in depth. + +The loop fixture: form the loop the churn sweeps found (two unrelated +failures at once, so two relays briefly route a publisher through each +other), subscribe at both while it lasts, then let the routes settle. Both +subscribers receive the next group and no upstream request is left pending. +Cover a request answered before the loop and one still unanswered when it +forms. + +Fix in the layer that survives +[the resume redesign](/quest/m4/resume-redesign.md), and run the +`verify-failover` and `verify-resume` skills on what they cover. + +Public API: none. Wire: none expected. If a refusal must tell "this route is +gone" from "no such track", that is a lite-07 wire change to report, which is +why this sits in m2. + +## Related + +- [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) - its doubt rule lets a publisher leaving three or more relays at once loop a request until the news arrives +- [Resume as producer replacement](/quest/m4/resume-redesign.md) - rewrites the per-reader path some of these hazards live in diff --git a/quest/m2/multicast/router.md b/quest/m2/multicast/router.md new file mode 100644 index 00000000000..4edfba9453b --- /dev/null +++ b/quest/m2/multicast/router.md @@ -0,0 +1,51 @@ +# [XL] The new route rule runs as a sans-IO router + +## Goal + +`rs/moq-net/src/route/` holds a production router for the +[epic](/quest/m2/multicast/README.md)'s rule beside `path::Router`, sans-IO +and driven only by tests until the [switch](/quest/m2/multicast/switch.md). +The [bake-off](/quest/m2/multicast/bakeoff.md)'s fixtures and sweeps run on +it, the exhaustive delivery-order search checks it, and a benchmark swept +over forwarding nodes, publishers, broadcasts, and sessions shows that an +event costs work in what it changed. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): the router first with no wire +change, then the switch, the order +[Routes and announces](/quest/m2/cluster-routes/README.md) Plan item 28 used +for the path rule. + +- Build the rule as the bake-off's pick leaves it: routes to forwarding nodes + only, the dead-hop withdrawal with no moved list, no hold, no doubt, no + timer, claims ordered by revision and certified by floors, the chain + across a scoped grant, and the one mark. +- Keep what is independent of the rule: the interest set edited in place + ([moq-dev/moq#8728](https://github.com/moq-dev/moq/pull/8728)), and + `Reader` and `View`. +- The reader debounces unavailable to offline locally, with mocked time, for + as long as the bake-off measured routes take to settle. It replaces what a + hold protected and is never on the wire. +- A relay stays on its next hop among routes of equal rank, and never sends + a route back over the session it learned it on. Both were quests of their + own until 2026-11-15: a change of hops alone was 3.5% to 11% of route + messages, one link coming up on a 32-relay full mesh cost 592 route + messages against 251, and advertising a route back to its source doubled + route traffic between every pair of relays. +- Per-event work follows what changed, never every answering stream or every + claim behind a route. [Route fan-out](/quest/m2/cluster-routes/route-fanout.md) + pins where the path router fails this. +- Bound what a peer can make the router hold by its own requests, not by a + constant. +- The exhaustive search over every delivery order + ([moq-dev/moq#8729](https://github.com/moq-dev/moq/pull/8729)) moves to + this router, and each property it checks is shown to fail on a mutation + that breaks it. The maintainer chose on 2026-11-14 to keep that search over + the compiled code and retire the TLA+ route model. + +Public API: none. Wire: none. + +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - the pick, and the fixtures this promotes diff --git a/quest/m2/multicast/switch.md b/quest/m2/multicast/switch.md new file mode 100644 index 00000000000..21ab958223a --- /dev/null +++ b/quest/m2/multicast/switch.md @@ -0,0 +1,79 @@ +# [XL] lite-07 sessions run the new route rule + +## Goal + +Every lite-07 session in Rust and `js/net` drives the +[new router](/quest/m2/multicast/router.md) on one announce stream per side, +the draft specifies it, and the old rule is gone: ROUTE_HOLD, the doubt, +DEAD's moved list, the Route Key Parameter with its proof and symbolic model, +qualified Node IDs, routes per prefix with their Route Suffix and Route +Depth, `path::Router`, and `RouteHandoff.tla`. The draft's routing text is +shorter than today's and the announce stream carries fewer message types. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): lite-07 carries the new rule +and the cut waits for it ([epic](/quest/m2/multicast/README.md) item 2). +lite-07 is `-wip`, so relays pinned to `main` roll together, as each earlier +lite-07 wire change did. + +Recommended wire, for the implementer to confirm or improve; each departure +is a decision to list in the PR: + +- A route is started, updated, and ended, and a closed session is named in + one fixed-size message. Nothing answers a hold, and no message lists moved + routes: the updates follow in order on the same stream. +- A claim carries its chain as a field of its own, replacing the temporary + Lite-06 Chain and its Setup Parameter: the nodes that announced it again + across a scoped grant or an older version. +- A request keeps its Instance and the origin node its first hop chose, and + loses Route Depth. +- An endpoint that never forwards declares no Node ID. A peer holding a + scoped publish grant sends claims and no routes, and the relay announces + them under its own Node ID. Peers that trust each other with the whole + tree exchange routes. +- The link mark stays as `upstream`. moq-net's per-link `proxy` switch, which + the relay hard-wires on, goes if nothing else needs it; that is a public + API change to report. + +Delete with the old rule: + +- `rs/moq-net/src/route/proof.rs`, `js/net/src/lite/route_key.ts`, + `rs/moq-net/verify/route-key/`, the `verify-route-key` skill, and the + interop route key lane (`test/interop/clients/js/route-key.ts`). +- `rs/moq-net/verify/route/` and the `verify-route` skill, or rewrite the + skill for the exhaustive search. +- [Firefox tabs prove their route key](/quest/m5/routes-browser-proof.md), + and the release cut's reason to wait on + [the Playwright bump](/quest/m1/playwright-pin.md). +- [Doubt holds](/quest/m2/cluster-routes/doubt-holds.md), + [doubt and keep-alive](/quest/m2/cluster-routes/doubt-keepalive.md), + [held after close](/quest/m2/cluster-routes/held-after-close.md), and + whatever of [route fan-out](/quest/m2/cluster-routes/route-fanout.md) the + new router answers, with the fixtures that pin them. + +Rewrite the assertions of the acceptance fixtures +(`tests/integration/route_acceptance.rs`, `route_failover.rs`, +`route_cluster.rs`) for the new rule, keeping their topologies. + +Quests written against the old wire, to retarget or delete in this PR: +[outbound federation](/quest/m3/cluster-routing/federation.md), +[the routing guide](/quest/m3/routing-guide.md), +[route wakes](/quest/m4/route-wakes.md), +[handoff routes](/quest/m4/relay-handoff-routes.md), +[regrant recheck](/quest/m4/regrant-route-recheck.md), +[announce interest memory](/quest/m4/announce-interest-memory.md), +[the mesh condition](/quest/m5/lite07-mesh.md), and +[route memory](/quest/m9/perf/cluster-route-memory.md). + +Update the draft's Announce, Floors, Routing, Dead Hops, and Resolution +sections, the message sections, the Setup Parameters, and the lite-07 +changelog, and `doc/bin/relay/cluster.md`. Run `just check`, +`just test interop --all`, and `just drafts check`. + +Public API: the per-link `proxy` switch, if removed. Wire: lite-07-wip only. + +## Required + +- [The router](/quest/m2/multicast/router.md) - the rule these sessions drive +- [The route stream](/quest/m2/cluster-routes/route-stream.md) - one announce stream per side, which this builds on diff --git a/quest/m2/multicast/warmth.md b/quest/m2/multicast/warmth.md new file mode 100644 index 00000000000..dcaae9aa4c6 --- /dev/null +++ b/quest/m2/multicast/warmth.md @@ -0,0 +1,64 @@ +# [L] Three ways to find a warm copy are measured + +## Goal + +The [bake-off](/quest/m2/multicast/bakeoff.md)'s harness runs three ways for +a relay to learn that a neighbour already carries a broadcast, over the +prototype route rule, and the PR reports for each: the copies each link +carries and the total link cost of each broadcast's tree, against plain +shortest-path joins and against the cheapest tree; control messages per +reader change; the time to a first frame; and unfed readers and pull cycles +under churn. The maintainer picks one, and the PR rewrites +[the implementation quest](/quest/m3/cache-routing/select.md) for it. + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): ✅ measure all three, then +pick (rejected: picking one unmeasured). The warm wire was stripped from +lite-07 on 2026-11-14 because its benefit over plain path routing was never +measured. + +The three: + +1. **On the path only.** A join stops at the first relay carrying the + Instance, and equal-cost next hops are chosen by the rendezvous hash, so + siblings facing the same choice converge on one. No warm signal. This is + the baseline the other two must beat. +2. **Ask on subscribe.** Each SUBSCRIBE answer, payload or metadata-only, + says whether the neighbour carries the Instance and its summed cost from + the source, and the relay takes the cheapest. The + [metadata-only secondary](/quest/m3/route-health/README.md) doubles as + the query, so nothing is sent for a broadcast nobody asks about. Sweep how + many neighbours a relay asks. +3. **Advertise.** A carrying relay announces it on the announce stream. + +The constraint the maintainer set for all three: a warm link costs its warm +price and a cold link its cold price, both at least 1, summed from the source +along the path the data takes, so a route never ranks better for being +extended. The stripped rule broke it: a carrier offered its copy at no cost +and a hash ranked carriers, and pulls formed a cycle in 3.3% of churn steps, +most leaving a reader unfed. + +A cost that depends on who subscribed is outside the +[proof](/quest/m2/multicast/proof.md)'s theorem. Argue it separately and +sweep it: taking a route only makes it cheaper for others, and leaving one +only worsens an offer nobody uses. A cycle that outlasts the messages in +flight fails the mechanism; count the transient ones with the readers they +leave unfed. + +Shapes where the three differ: two edges that each prefer a different core; +LAN relays with an uplink each; a ring with a far source; and one viewer per +broadcast, where the first must cost no message at all. Earlier sweeps to +compare with: a LAN mesh of 6 pulled one uplink copy per broadcast, a ring +of 6 kept 12 copies for 8 broadcasts, and warm routes cost 3.1 to 4.5 +messages per reader change. + +The cut waits on the pick, not on the implementation. An additive field +lands in whichever wip is current with its implementation, as decided +2026-11-14; the PR says whether the pick changes bytes lite-07 would freeze. + +Public API: none. Wire: none. + +## Required + +- [The bake-off](/quest/m2/multicast/bakeoff.md) - the harness and the route rule the three run over diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index 08e865e68b5..cfc86709e50 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -2,263 +2,62 @@ ## Goal -A relay stays on a broadcast's path only while it has a local reader, feeds -two or more downstream sessions, or is its one consumer's cheapest route -given the copies the others hold. Otherwise its child bypasses it, so a broadcast with one viewer -crosses fewer relays. A relay joins a neighbor -that already carries a broadcast instead of pulling a second copy, so a LAN -of relays with their own uplinks pulls one copy over one uplink. Routing -stays loop-free under stale state, converges once demand and topology stop -changing, and sends no per-broadcast message for an idle broadcast. +A relay pulls each broadcast from the cheapest copy: a neighbour already +carrying its Instance when that costs less than a fresh copy from the source. +A LAN of relays with their own uplinks pulls one copy over one uplink, edges +that prefer different cores share one copy between regions, and no link +carries a group twice. Routing converges once demand and topology stop +changing, and a broadcast nobody reads costs no message of its own. Non-goals: steering clients to the relay that holds a broadcast; decisions -from measured load or RTT (link quality, planned downstream in Kixel); -browsers serving copies they receive; a relay on neither consumer's cheapest -route becoming a shared stop; one copy across a multi-hop sparse overlay -rather than among direct neighbors; and creating links, since routing -chooses only among the sessions an operator configured. +from measured load or RTT; browsers serving copies they receive; and creating +links, since routing chooses only among the sessions an operator configured. ## Plan -Decided 2026-11-08 (maintainer, /quest-plan): - -1. ✅ **Node routes are the cold cost, and optional per-broadcast warm routes - are the warm cost.** The per-origin ROUTE metric of - [Cluster routing](/quest/m3/cluster-routing/README.md) stays the cold - cost, static and one per origin. A relay carrying a broadcast may - advertise a warm route for it. This revises Cluster routing's non-goal of - cache-dependent route prices. (rejected: per-broadcast forwarding with a - cache-only subscribe and no advertisement; keeping routing cache-agnostic) -2. ✅ **A warm route is per neighbor and travels one hop.** Reopened by - item 13; replaced by item 14, where a warm route propagates and a relay - offers each neighbor the best copy whose Hops name neither. It went only to - direct neighbors and was never forwarded. B was warm toward N only while B - had demand besides N's (a local reader or another downstream session), - and withdraws once that demand has been gone for the copy's existing - linger, with no new timer. B sends it only to neighbors whose cold metric - for the origin is at least its own, since no other neighbor may use it - (item 4). Without the per-neighbor rule, a relay whose reader left stays - warm because it still feeds N, and N never skips it again. (rejected: - broadcast-wide warmth, as the pre-#3225 design had; forwarding warmth k - hops, which floods every warm transition) -3. ✅ **One cost per link, priced marginally.** Replaced by item 14: a copy - ranks as a route does, drain then priority then cost, and its carrier offers - it at no cost, so joining a copy costs only the links it crosses. Pulling through a warm - neighbor costs the link alone, since that neighbor's cold metric is - already paid; a cold neighbor costs the link plus its cold metric. A tie - goes to the warm route, then to item 4's hash. A topology that wants both - behaviors prices a link that skips one relay at more than one neighbor - link and less than two: a cold broadcast then takes the skip, and once - the skipped relay pulls for its own reader, its neighbor joins it. With - uniform prices both cases tie and fall to the hash. (rejected: a warm and - a cold price per link; a warm price the carrier seeds) -4. **Eligibility, which the proof must hold.** Reopened by item 13. A relay - may pull broadcast P - through neighbor N only when N's advertised route to the request's pinned - origin has a newer seqno, or an equal seqno with - `(cold metric, hash(P, N))` below `(feasibility distance, hash(P, self))`. - A feasibility distance never exceeds what its node advertised at that - seqno, so it strictly decreases along every forwarding edge even when an - advertisement is stale: no loop, no rank, no hold timer. The pre-#3225 - rank was a cold cost each relay reported about itself, so a stale report - could let a ring of relays all let go at once, which its 500 ms hold only - narrowed. Route advertisements stay one selection per origin. -5. ✅ **Proof first:** model and sweeps before the implementation. - (rejected: implementing with targeted fixtures only) -6. ✅ **The proof picks the wire placement.** The warm message is designed - fresh rather than reviving ANNOUNCE_TAKEOVER, which - [moq-dev/moq#8063](https://github.com/moq-dev/moq/pull/8063) deleted in - favor of instance continuity. The proof sizes it and - picks lite-07 or an additive SETUP extension. (rejected: - reusing ANNOUNCE_TAKEOVER's slot) -7. ✅ m3, after the m2 route layer it ranks on. (rejected: m2; m4) Amended - 2026-11-09 (maintainer, /quest-plan from Kixel's audit of its fleet plan): - ✅ the proof alone moves to m2, ahead of the lite-07 cut, and lands the - message's draft section and codec there if it places it in lite-07 - (rejected: a design-only sizing quest in m2; leaving the proof in m3 and - accepting a SETUP extension). The implementation stays in m3. This - replaces item 6's rule that choosing lite-07 went back to the maintainer; - item 11 records the placement the maintainer chose. - -Proof results 2026-11-10 ([moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)), -modeled in `rs/moq-net/src/route/tests/cache.rs` over the real `Router`: - -8. **Item 4 holds.** No sweep step in 3000 churned scenarios formed a - per-broadcast cycle, and the fixtures fail when either the feasibility - check or the hash order is dropped. The router answers it with the - crate-private `Router::eligible`, which marks a route tied at the - feasibility distance so the caller applies the hash. The argument needs a - feasibility distance that never rises while a neighbor holds the advert; - the overtaking echo that - [moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447) found breaks - that, which [the implementation](/quest/m3/cache-routing/select.md) must - not ship before it is fixed. -Decided 2026-11-10 (maintainer, /quest-iterate on -[moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)), from the -proof's proposals: - -9. ✅ **Each pull names whom it serves**, amending item 2; replaced by item 14, - whose SUBSCRIBE says only whether it joins a copy, and lite-07 no longer - carries `Serves`: a trailing `Serves` field on SUBSCRIBE and SUBSCRIBE_UPDATE. (rejected: no field) - "Demand besides N's" can't be told from one hop: if B's only demand is a - relay pulling on N's behalf, B counts warm toward N, N joins B, B's demand - vanishes, and the cycle repeats every linger. `Serves` is alone (a local - reader, or two or more roots below it), a single root relay's Node ID, or - unknown (an endpoint that doesn't say, which warms nothing). N counts B - warm only when B serves alone or a root other than N and N's own root. A - narrowed warm route stands until the copy's linger runs out, as the - withdrawal already does. -10. ✅ **Price a dependent copy at its cost, and move only on strict - improvement**, amending item 3; replaced by item 14, whose offers do not - depend on who pulls what, and whose sweep settles every scenario with no - `UNSETTLED` list; the Goal's "cheapest cold route" became - "cheapest route given the copies the others hold". (rejected: dropping - mutual warmth so only a relay with a local reader advertises, which - converges trivially but loses the LAN sharing that feeding two sessions - gives; link-only pricing) Link-only pricing has no steady state when two - relays each keep the other warm (sweep scenario 2062 flips forever). A - warm route carries its sender's own price for its pull; a receiver the - copy depends on (its serves names only the receiver's root) pays link - plus that cost, an independent one pays the link alone, and a relay - moves only for a strictly lower `(draining, price)`. With exact - information that is a potential game on the forest's total link cost. - Under the sweep's churn, with the draft's Rendezvous Hash, 2999 of 3000 - scenarios settle; scenario 2251 never does, as two carriers keep a - narrowed route standing while two relays swap between them. The sweep names them in - `UNSETTLED` and checks each still never settles; item 14's rule settles - them. -11. ✅ **The wire is in moq-lite-07 itself** (its warm fields and `Serves` - replaced by item 14's), overturning the proof's choice - of a SETUP extension. (rejected: an additive Warm Routes SETUP - extension) Lite-07 is unpublished, so its shape can still change until - the cut, and a later change goes to the next version. The draft's - Warm Routes section and the Rust codec landed with the proof: - `WARM_START` (0xC) carries the Route ID it travels under, the path - suffix, the Instance and Minted encoded as in ANNOUNCE_START, then the - serves and cost, and assigns a Warm ID; `WARM_UPDATE` (0xD) replaces a - warm route's serves and cost by Warm ID, so a price change costs one - message rather than an end and a start; `WARM_END` (0xE) withdraws it. - A warm route follows the ROUTE_START it names on the announce stream the - carrier answers, and a ROUTE_END retires the warm routes under it. - SUBSCRIBE and SUBSCRIBE_UPDATE end with `Serves`: a type of unknown - (0x0), alone (0x1), or relay (0x2) followed by its Node ID. The hash is - the Resolution section's FNV-1a rendezvous hash of the path and the - neighbor's Node ID. Until the implementation, Rust reads warm routes and - serves and ignores them, and says unknown; `js/net` says alone, since it - forwards nothing, and skips the warm types as unknown, which is how an - endpoint that does not choose per broadcast ignores them. About 4.5 warm - messages per reader change across the sweep's churn, which also flaps - links and advances seqnos; an idle broadcast sends none. - -Measured by the proof: - -12. **Costs the sweeps measured.** A full LAN mesh of 6 relays pulls one - uplink copy per broadcast; a ring of 6 keeps 12 copies for 8 broadcasts, - one per hash local minimum. An origin seqno advance that reaches a child - before its warm parent makes the parent's older route ineligible, so the - child moves off and back (2 moves in the chain fixture); origins advance - on every announce stream open and close. Item 13's paths drop the seqno - ([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27), - and this churn with it. - -13. Reopened 2026-11-11 (maintainer, /quest-plan on the route redesign): - items 2 and 4. In the maintainer's words: "I think we need warm routes - propagating, that decision was made based on this flawed setup." ✅ **A - warm route is per broadcast and propagates**, for as long as it is - cheaper than the route its publisher's broadcasts share - ([epic](/quest/m2/cluster-routes/README.md) Plan item 26). A - broadcast no relay carries rides the shared route and costs no message of - its own. Item 4 rests safety on Babel's distance, which cannot let a relay - join a copy at a neighbour that is no closer to the publisher: the join - the Goal wants, and why item 2 stopped at one hop. Items 1 and 3 stand. - Decided 2026-11-11 (maintainer, on - [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533)): what keeps a warm - route free of loops and stale copies is its path, as for every route - (routes epic Plan item 27), and the proof then holds that rule. The - prototype in `rs/moq-net/src/route/tests/copies.rs` showed three things - the proof starts from. A warm route carries the path its copy took and - names the relay that carries it, and a relay refuses one whose path names - it, so it joins a copy at a neighbour farther from the publisher than - itself. Two relays that start carrying in the same instant trade copies - for good unless they are ranked, so a relay already pulling moves onto - another's copy only if that carrier ranks below it. And when a carrier's - uplink closes, it and a relay pulling from it point at each other until - the news crosses their link. - [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) proves items - 2 and 4 as first decided. - -14. **The proven rule** ([moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732)), - proven in `rs/moq-net/src/route/tests/cache.rs` over `route::path::Router` - ([moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)), in three - layers that each settle before the next depends on them: - - **Routes** are item 27's, which copies never change. - - **Offers.** A relay with a local reader is its copy's carrier and offers - every neighbor its copy, its own Hop ID the only hop, at no cost. A relay - with no reader offers each neighbor the best of its route and the copies - offered to it whose Hops name neither relay, when that best is a copy. - Copies rank as routes do, drain then priority then cost, so an offer - depends only on readers, routes, and other offers, and settles as a - monotone path vector does. - - **Pulls.** A carrier pulls through its best source among its route and - the copies of carriers ranked below it by the Rendezvous Hash. A relay - with no reader pulls through its best source whose Hops name no - neighbor it serves, so long as that is, for each such neighbor, the copy - it offered it or a lower-ranked carrier's; otherwise it follows its - route. SUBSCRIBE's `Warm` says whether a pull joins the copy offered or - follows the route, and a relay pulled along its route follows its own. - - Each join gets a copy whose carrier, then whose rank, is strictly better - than what it gives the relays joined to it, so once the offers and - subscriptions in flight are delivered the pulls are free of loops, as the - maintainer accepted on 2026-11-13. Unranked, two carriers join each other - and neither is fed. This replaces the rules of - [moq-dev/moq#8615](https://github.com/moq-dev/moq/pull/8615) (monotone - copy costs, carrier rank on every move, a relay bound by the carriers - behind it), which failed a few scenarios per seed because what a relay - offered depended on who pulled through it. - - Measured by `sweep_cache_routing_long` (3000 scenarios each with and - without held links, 6 relays, 2 publishers, priorities and drains), under - 12 seeds (72,000 scenarios): every scenario settles on the offers computed - centrally, an idle broadcast sends no warm message, and about 3.1 warm - messages go out per reader change. A pull lingering with no demand can hold - a copy its parent promised another neighbour, so the pulls around it may - move until the linger (30 s) runs out; settling counts from there. During - churn a broadcast's pulls form a cycle after 3.3% of steps, in about half - the scenarios, each gone within 142 deliveries. The sweep runs nightly - from `test/route-sweep.sh`. - - The wire: WARM_START and WARM_UPDATE carry a copy's Warm Cost, a Flags - byte whose low bit marks a draining copy, its priority, and its Hops, - carrier first and the sender last. SUBSCRIBE and SUBSCRIBE_UPDATE end with - a `Warm` byte. Rust's codec reads and writes both, and `js/net` the `Warm` - byte, which both send as 0 (follow the route) and ignore when read, as - `js/net` skips the warm messages; [the implementation](/quest/m3/cache-routing/select.md) - acts on them. - -Prior art: warm/cold route prices -([moq-dev/moq#2925](https://github.com/moq-dev/moq/pull/2925)) were deleted -by prefix routes ([moq-dev/moq#3225](https://github.com/moq-dev/moq/pull/3225)), -and the pop-skipping questline (warm advertise, rank, peer reconfigure) was -pruned in [moq-dev/moq#4393](https://github.com/moq-dev/moq/pull/4393). -Related designs: PIM-SM's switch from a shared tree to a shortest-path tree -([RFC 7761](https://www.rfc-editor.org/rfc/rfc7761.html)), NHRP shortcuts -([RFC 2332](https://www.rfc-editor.org/rfc/rfc2332.html)), and Babel's -feasible successors ([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966.html)). - -This README's own work, once its children land: +Reopened 2026-11-15 (maintainer, /quest-plan on +[cache-aware multicast](/quest/m2/multicast/README.md), requirement 8: +"there is a cold cost for a route and a warm cost", and routes coalesce onto +a node that says it carries a broadcast). What stands and what goes: + +- ✅ Joins stop at the first relay carrying the Instance, which the route + layer gives with no message + ([requests](/quest/m2/multicast/requests.md)). +- ✅ How a relay learns of a warm neighbour off its path is picked from + measurements ([warmth](/quest/m2/multicast/warmth.md)), which + [the implementation](/quest/m3/cache-routing/select.md) then builds. +- ✅ A warm cost is strictly positive and summed from the source along the + path the data takes. This replaces item 14's rule below, under which a + carrier offered its copy at no cost and a hash ranked carriers. +- The non-goal of "one copy across a multi-hop sparse overlay rather than + among direct neighbors" is dropped: a mesh is the product this serves. + +The earlier Plan, items 1 to 14 (2026-11-08 to 2026-11-14), is in +`git show bf04cecfe1:quest/m3/cache-routing/README.md`. What it measured: + +- Cache-aware cost was built four times before it (2026-07 to 2026-08), and + each time two warm relays adopted each other. +- Item 4's rule rested on Babel's feasibility distance, which cannot let a + relay join a copy at a neighbour no closer to the publisher. +- Item 14's rule settled every one of 72,000 sweep scenarios and sent about + 3.1 warm messages per reader change, but its pulls formed a cycle in 3.3% + of churn steps, about 9 in 10 of which left a reader unfed, and its gain + over plain path routing was never measured + ([moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732)). The + maintainer stripped its wire from lite-07 on 2026-11-14 + ([warm routes on the wire](/quest/m2/warm-routes-wire.md)). + +This README's own work, once its child lands: - End-to-end tests over real sessions, time mocked. A chain of three relays - plus a skip link, priced 3 and 5: a cold broadcast takes the skip, a reader - on the skipped relay makes the far relay join it, and that reader leaving - sends it back to the skip. Two LAN relays with their own upstream links and - readers on both pull one copy over one uplink, with the owner spread across - broadcasts by the hash. -- Docs: warm routes and how to price skip links in `doc/bin/relay/cluster.md`, - and a section on the routing concept page once the + plus a skip link: a cold broadcast takes the skip, a reader on the skipped + relay makes the far relay join it, and that reader leaving sends it back. + Two LAN relays with their own uplinks and readers on both pull one copy + over one uplink. +- Docs: warm and cold costs in `doc/bin/relay/cluster.md`, and a section on + the routing concept page once the [routing guide](/quest/m3/routing-guide.md) has added it. ## Related -- [Cluster routing](/quest/m3/cluster-routing/README.md) - the node routes and feasibility distances this ranks on -- [Relays split horizon per session](/quest/m3/route-split-horizon.md) - per-session suppression of node routes, beside warm routes' per-neighbor rule +- [Cluster routing](/quest/m3/cluster-routing/README.md) - the node routes this ranks on diff --git a/quest/m3/cache-routing/select.md b/quest/m3/cache-routing/select.md index 0700156904c..05685a6cf14 100644 --- a/quest/m3/cache-routing/select.md +++ b/quest/m3/cache-routing/select.md @@ -1,60 +1,56 @@ -# [XL] Warm routes and per-broadcast parent selection +# [XL] Relays pull each broadcast from the cheapest copy ## Goal -Relays send warm routes as the -[epic](/quest/m3/cache-routing/README.md) decides. Each relay pulls every -demanded broadcast through its cheapest eligible neighbor, and moves an -existing copy when a cheaper eligible one appears or its warm parent -withdraws. A peer that never sends warm routes sees only cold routes and -behaves as it does today. Relays export how many upstream copies they hold -and how many moves warm routes caused. +Relays learn of warm copies the way +[warmth](/quest/m2/multicast/warmth.md) picked, and each pulls every demanded +broadcast from the cheapest copy of its Instance, moving an existing pull +only when the route it uses gets worse or ends, or another is strictly +cheaper in summed cost. A peer that sends no warm signal is treated as cold +and behaves as it does today. Relays export how many upstream copies they +hold and how many moves a warm copy caused. ## Plan -- Implement the rule the [epic](/quest/m3/cache-routing/README.md)'s item 14 - records ([moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732)) by - moving `decide` and `offer_from` out of `rs/moq-net/src/route/tests/cache.rs` - into production code that the fixtures and sweep then drive, the way `Sim` - drives `path::Router`: today they test the rule against itself and pass - without touching the session. Add a check of the Goal's efficiency to - `steady()`, copies held against a cold-routing baseline, since a rule that - never offered a copy passes the sweep now. The rule covers the - warm messages and SUBSCRIBE's `Warm` byte (which return to the wire with - it, below), warm state derived from each front's demand, and the choice of - what to pull. The long sweep already runs nightly from - `test/route-sweep.sh`. +Rewritten 2026-11-15 (maintainer, /quest-plan on +[cache-aware multicast](/quest/m2/multicast/README.md)): the rule this was +to build, item 14 of the [epic](/quest/m3/cache-routing/README.md)'s earlier +Plan, is replaced. The warmth quest's PR fills in the mechanism here once +the maintainer has picked it. + +What holds whichever is picked: + +- A warm link costs its warm price and a cold one its cold price, both at + least 1, summed from the source along the path the data takes. - Extend today's selection rather than adding a second selector. `route_order` in `rs/moq-net/src/model/origin.rs` already hashes equal routes per path, and a front already moves to a better route of its own - instance and resplices (`rs/moq-net/src/model/front.rs`). -- A warm route names the origin and instance it carries, and a front moves - only to a copy of its own instance. -- Bound the warm state a peer can make a relay hold by design, as - announcements are bounded. -- Each pull sets SUBSCRIBE's `Warm` byte, which #8732 put in place of the - `Serves` field; `js/net`, which forwards nothing, sends 0 (follow the - route) and skips the warm types as unknown. -- Export upstream copies held, warm routes sent and received, and moves by - cause through the relay's existing metrics. + Instance and resplices (`rs/moq-net/src/model/front.rs`). A front moves + only to a copy of its own Instance. +- The rule runs in production code that the fixtures and sweeps drive. The + stripped rule lived only in a test simulator that checked it against + itself, so a rule that never offered a copy passed. +- Check the Goal's efficiency: copies held against plain path routing. +- Bound the warm state a peer can make a relay hold by design. +- Export upstream copies held and moves by cause through the relay's + existing metrics. - `rs/moq-net/AGENTS.md` says a relay behaves the same warm or cold. That - covers answers to requests; warm routes change only which neighbor a - request goes to. If the rule's wording contradicts warm routes, bring a - rewording to the maintainer. + covers answers to requests; this changes which neighbour a request goes + to. Bring a rewording to the maintainer if the two contradict. + +Decided 2026-11-14 (maintainer, pick on +[moq-dev/moq#8764](https://github.com/moq-dev/moq/pull/8764)): ✅ lite-07 +freezes without a warm wire, so whatever wire the pick needs lands in the +then-current wip version with this. + +Public API: report any relay configuration added, such as a warm cost per +link. Wire: whatever the pick needs, in the then-current wip version. -Decided 2026-11-14 (maintainer, pick on [moq-dev/moq#8764](https://github.com/moq-dev/moq/pull/8764)): -✅ lite-07 freezes without the warm wire -([warm routes on the wire](/quest/m2/warm-routes-wire.md)), so this quest -re-adds WARM_START, WARM_UPDATE, WARM_END, and SUBSCRIBE's `Warm` byte in the -then-current wip version, and first measures copies and relays crossed -against plain path routing. +## Required -Public API: report any relay configuration added for warm routes. Wire: the -warm messages and the `Warm` byte return in the then-current wip version, -with the draft's Warm Routes section. +- [Warmth](/quest/m2/multicast/warmth.md) - picks the mechanism this builds +- [The switch](/quest/m2/multicast/switch.md) - the route layer it ranks on ## Related -- [Routes and announces](/quest/m2/cluster-routes/README.md) - the route layer driving lite-07 sessions since [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023); its switch to path routes deletes the feasibility distances this rule once read -- [Warm routes on the wire](/quest/m2/warm-routes-wire.md) - may strip the warm wire from lite-07 or move this epic downstream before this starts -- [Transient cycles](/quest/m3/cache-routing/transient-cycles.md) - lowers the churn-time pull cycles the rule allows +- [Route health](/quest/m3/route-health/README.md) - its metadata-only subscription is the query if the pick is to ask on subscribe diff --git a/quest/m3/cache-routing/transient-cycles.md b/quest/m3/cache-routing/transient-cycles.md deleted file mode 100644 index d165450bed5..00000000000 --- a/quest/m3/cache-routing/transient-cycles.md +++ /dev/null @@ -1,41 +0,0 @@ -# [M] Fewer pull cycles while warm routes churn - -## Goal - -The cache-aware rule's transient pull cycles during churn happen less often -than the 3.3% of churn steps [moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732) -measured, or clear faster than its 142 deliveries, without slowing a join the -maintainer has not agreed to slow, and the sweep reports the rate so a change -that raises it shows. - -## Plan - -Decided 2026-11-14 (maintainer, /quest-complete on #8732): ✅ merge, and plan -a quest to lower the rate (rejected: accept the rate as is, hold the PR, -delete). Planned 2026-11-14 by the -[feature review](/quest/m3/release/feature-review.md), which found the pick -had no quest yet. - -What #8732 measured: about 9 in 10 cycle steps leave some reader unfed until -the crossing message lands; #8615's earlier sweep saw 0.39% and 53 -deliveries; at seed 3, dropping per-link priorities (2.8 to 3.0%) or letting -carriers exclude the neighbours they serve (2.9 to 3.1%) explained neither the -gap. Start by finding which of the sweep's churn or the per-neighbour offers -causes it, then propose fixes with their join-latency cost. - -A second baseline to account for: [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) -(the path rule for a flapping publisher) re-pinned -`route::tests::cache::a_ring_of_stale_offers_settles_on_the_link_left`'s cap -on cycle steps from 40 to 204 across 20 seeds, 4 to 22 per seed, each cycle -cleared within 20 deliveries; the rule it replaced gave 2 per seed, each -cleared within 2. Explain that rise along with #8732's rate, and measure any -fix against both the sweep and this fixture. - -If [warm routes on the wire](/quest/m2/warm-routes-wire.md) moves the epic -downstream, this moves with it. - -Decided 2026-11-14 (maintainer, pick on [moq-dev/moq#8764](https://github.com/moq-dev/moq/pull/8764)): ✅ a fix adds no join latency; one that only clears cycles faster comes first (rejected: a carrier waiting for a crossing message before it joins). - -Public API: none. Wire: none expected; a rule change that needs one updates -the draft's Warm Routes section. - diff --git a/quest/m3/cluster-routing/README.md b/quest/m3/cluster-routing/README.md index a3ee172a111..999b8ef7160 100644 --- a/quest/m3/cluster-routing/README.md +++ b/quest/m3/cluster-routing/README.md @@ -20,6 +20,14 @@ work, and no quest requires one in a later milestone, since a blocker ranks by its own milestone. ✅ m3 (rejected: m2, where the cut would wait on it; pulling the m2 blocker into m1). It ranks right after m2, when it can start. +Reopened 2026-11-15 (maintainer, /quest-plan on the whole route layer): +[cache-aware multicast](/quest/m2/multicast/README.md) replaces the route +layer this Plan describes, and where the two differ it wins. A routing node +is a node that forwards, not every publisher; a claim crosses a scoped grant +by a chain, not a qualified identity under a route key; and holds, doubt, +and the non-goal of cache-dependent route prices are gone. One model for CDN +tiers, outbound local clusters, and meshes stands. + Decided with the maintainer, 2026-10-13: - Keep per-origin ROUTEs and path-less ANNOUNCEs on the ordered announce @@ -154,6 +162,7 @@ setup after the capabilities land. ## Required +- [Cache-aware multicast](/quest/m2/multicast/README.md) - the route layer this builds on, replacing the path rule - [Routes and announces](/quest/m2/cluster-routes/README.md) - shared routing contract and implementation ## Related diff --git a/quest/m3/cluster-routing/federation.md b/quest/m3/cluster-routing/federation.md index 863f28599a2..129e3221667 100644 --- a/quest/m3/cluster-routing/federation.md +++ b/quest/m3/cluster-routing/federation.md @@ -9,6 +9,15 @@ cluster with two uplinks, partition, and healing. No bespoke federation router. ## Plan +Reopened 2026-11-15 (maintainer, /quest-plan on +[cache-aware multicast](/quest/m2/multicast/README.md)): the CDN relay +announces the local relay's claims under its own Node ID, with a chain of +the nodes that did so, and refuses a claim whose chain names it. No route, +qualified identity, or route key crosses the scoped grant, so the key file +below goes, and a relay behind the local relay may be scoped too. +[The switch](/quest/m2/multicast/switch.md) rewrites the paragraphs below +that say otherwise. + Decided 2026-10-13: this is a required use case, independently shippable from peer-to-peer routing. Existing cluster sessions are bidirectional and scoped peer admission exists; extend and prove those paths rather than creating a reverse diff --git a/quest/m3/route-health/README.md b/quest/m3/route-health/README.md new file mode 100644 index 00000000000..57b91c76d79 --- /dev/null +++ b/quest/m3/route-health/README.md @@ -0,0 +1,61 @@ +# Route health + +## Goal + +A relay or endpoint that demands a track over one route watches a second +route to the same Instance with a metadata-only subscription, and moves the +track when the first stalls, without waiting on a QUIC keep-alive or idle +timeout. Today a route that dies without a close starves its readers until +the 10 s idle timeout fires, and a session that stays up while its payload +stalls is never recovered +(`rs/moq-net/tests/integration/route_health.rs`). + +## Plan + +Decided 2026-11-15 (maintainer, /quest-plan): ✅ planned again, in m3, as the +secondary route of [cache-aware multicast](/quest/m2/multicast/README.md) +(requirement 13: "don't rely on QUIC keep-alives for fast failovers"). The +first epic was deleted 2026-11-09 for want of a consumer +([moq-dev/moq#8423](https://github.com/moq-dev/moq/pull/8423)); read it with +`git show cd560bde5^:quest/m5/route-health/README.md`. Its decisions stand, +made by the maintainer on 2026-10-13 and 2026-11-02 from the stalls +[moq-dev/moq#7096](https://github.com/moq-dev/moq/pull/7096) measured: + +- ✅ Evidence is comparison, never silence. Switch when the active route's + delivered media time lags what the standby reports by more than the + reader's max delay, or 2 s for a reader without one, while the reader + demands. A quiet track is not a dead one, and `largest()` values are never + compared. +- ✅ One standby per demanded track: the next-best route to the same + Instance. Only payload demand creates a standby, so metadata observers + never branch into a monitoring tree of their own. +- ✅ When neither route shows progress for that budget, the standby slot + moves to the next candidate, doubling the wait per move, and nothing is + failed. +- ✅ A demoted route never takes the track back while the working one + delivers (2026-11-02, reversing 2026-10-13's return after recovery). +- ✅ Per track, not per session: one stalled track never fails a session. +- ✅ Only between claims of one Instance the publisher named. A minted + Instance, as an older version or moq-transport gets, never resumes. + +New on 2026-11-15, from the prior art this is closest to: + +- The standby is a route whose path does not name this node, and one that + does not cross the hop under suspicion ranks first. A fast-reroute + alternate often shares the failed element (RFC 7431 section 7). +- A relay stops asking its standby when the standby is the only thing it + serves that track to, so two relays never keep each other alive (the same + RFC's rule for its secondary). +- The answer to a metadata-only subscription is also where a neighbour says + whether it carries the Instance, if + [warmth](/quest/m2/multicast/warmth.md) picks that. + +This README's own work, once its children land: end-to-end validation across +a multi-hop relay network. Stalled video moves while healthy audio stays, a +stall with advancing control is caught, a recovered route does not take the +track back, two quiet routes stay unknown, a bottleneck shared by both routes +does not hide a working third, and relays deciding at once do not swap. +Report media overlap, control traffic, wakes, switches, and memory across +tracks, candidates, and readers. Revisit the relay's 10 s default QUIC idle +timeout with the result, and add a route-health section to the routing +concept page. diff --git a/quest/m3/route-health/failover.md b/quest/m3/route-health/failover.md new file mode 100644 index 00000000000..074c917db5c --- /dev/null +++ b/quest/m3/route-health/failover.md @@ -0,0 +1,41 @@ +# [XL] A stalled route yields to its standby + +## Goal + +For each track with payload demand, a relay or endpoint holds a +metadata-only subscription on its next-best route to the same Instance, and +promotes it in place when the active route's delivered media falls behind +what the standby reports by more than the reader's budget. The track then +stays on the working route. A silent death and a payload stall on a live +session, which `rs/moq-net/tests/integration/route_health.rs` pins as lost +media and as never recovered, both recover within that budget. + +## Plan + +The [epic](/quest/m3/route-health/README.md) holds the decisions: comparison +as evidence, one standby per demanded track, the doubling exploration, no +return to a demoted route, per track, and only for an Instance the publisher +named. + +- Each track chooses its source apart from the broadcast's shared route + selection: the front already holds every usable route to its Instance + (`rs/moq-net/src/model/origin.rs`, `model/front.rs`), and the router hands + it each alternate. The maintainer permits falling back to per-broadcast + failover if per-track proves disproportionate; say why if used. +- Promotion overlaps payload briefly while the new route is verified, and a + steady standby carries none. Bound the overlap. +- Two relays deciding at once must not swap onto each other: the standby is + never a route whose path names the relay, and a relay drops its standby + when the standby is all it serves the track to. +- Model the promotion with the `verify-failover` skill before building it, + since it adds an ordering between two subscriptions. +- Measure the interruption for a clean close, a silent death, and a payload + stall, against today's: about 0, the 10 s idle timeout, and never. + +Public API: none expected beyond +[`Subscription.meta`](/quest/m3/route-health/meta.md). Wire: none beyond it. + +## Required + +- [Metadata-only subscriptions](/quest/m3/route-health/meta.md) - what the standby observes +- [The switch](/quest/m2/multicast/switch.md) - the routes the standby chooses among diff --git a/quest/m3/route-health/meta.md b/quest/m3/route-health/meta.md new file mode 100644 index 00000000000..dbea1a38f6c --- /dev/null +++ b/quest/m3/route-health/meta.md @@ -0,0 +1,52 @@ +# [L] Metadata-only subscriptions + +## Goal + +Rust and JS subscribe to a track's group and frame progress without its +payload, through the shared subscription model and whichever lite wip is +current when this lands. A chain of relays forwards and aggregates that +demand without pulling the media behind it, and a metadata subscription +becomes a payload one in place, on the same stream. + +## Plan + +Decided 2026-10-13 and confirmed 2026-11-02 (maintainer), carried from the +quest deleted in [moq-dev/moq#8423](https://github.com/moq-dev/moq/pull/8423) +(`git show cd560bde5^:quest/m5/route-health/meta.md`): + +- ✅ A `meta: true` field on `Subscription`, carried by the existing + SUBSCRIBE: "SUBSCRIBE_META" was shorthand, not a new message. It is a live + stream: `largest()` on a payload-less standby is only the SUBSCRIBE_OK + snapshot, and a one-shot status races the active route. +- ✅ Each group stream marks its mode in its own header, so a stream opened + before a mode update is read under the mode it was opened in (rejected: + the mode only on the subscription, which races updates against open + streams). A metadata FRAME keeps its timestamp and declared size and + carries no bytes, so a real empty frame stays distinguishable. +- ✅ Promotion is a SUBSCRIBE_UPDATE on the same subscription, asking for + bytes from the reader's first missing frame, including positions it saw as + metadata. Metadata never satisfies a media reader, fills a cache as a + complete frame, or advances a resume cursor. +- ✅ A relay pulls payload upstream only for payload demand. Metadata demand + travels upstream as its own live metadata subscription, served from a + payload copy while the relay holds one. When only metadata demand remains, + the relay demotes its upstream subscription with an update, and drops it + when the last reader leaves. +- ✅ An older version never emulates it by fetching and discarding media: it + reports the mode unsupported. No moq-transport counterpart is needed. + +Mirror the field across Rust, JS, and every binding that exposes +`Subscription`. + +Tests: metadata-only traffic across several hops carries no payload; a relay +whose last payload reader leaves keeps forwarding headers to its metadata +readers; mixed demand; promotion and demotion during an open group; empty +frames; late frames in the old mode; resume at a position seen as metadata; +cancellation; refusal on an unsupported version. Sweep tracks, readers, and +relay depth for bytes, wakes, and memory. Update +`drafts/draft-lcurley-moq-lite.md` and both codecs together, and add the mode +to `just test interop --all`. + +Public API: `Subscription.meta` in Rust, JS, and the bindings. Wire: a +metadata mode in the current lite wip: the SUBSCRIBE field, a mode marker in +each group stream header, and payload-less FRAMEs. diff --git a/quest/m3/route-keep-next-hop.md b/quest/m3/route-keep-next-hop.md deleted file mode 100644 index d95dd202a06..00000000000 --- a/quest/m3/route-keep-next-hop.md +++ /dev/null @@ -1,53 +0,0 @@ -# [S] A relay keeps its next hop among equal routes - -## Goal - -A lite-07 relay that already routes to an origin through one neighbour stays -on it when another neighbour offers a route of the same rank: the same drain -state, priority, and metric. Today the lower key wins, so a relay moves for -nothing, and since routes name the nodes they crossed -([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)) every relay -downstream is sent a ROUTE_UPDATE for it. A test brings a link up beside an -equal one and counts the route messages sent on every other session: none. -The new link's own first advertisements do not count. - -## Plan - -Decided 2026-11-11 (maintainer, /quest-plan after #8487): ✅ plan it, in m3 -beside [split horizon](/quest/m3/route-split-horizon.md), since it trims -traffic and fixes nothing (rejected: dropping it). - -What it costs today, from #8487's measurements: a change of hops alone is -3.5% to 11% of route messages in the router's churn sweeps, about 3% of -control messages over real sessions, and on a 32-relay full mesh one link -coming up costs 592 route messages against 251 before the hops. - -[Cluster routing](/quest/m3/cluster-routing/README.md) already decided that -"a new connection alone does not outrank a different equal-cost route", so -this implements a settled rule. Keep the deterministic path hash among equal -routes for a request's first hop: that choice is per request and is not this -selection. - -- `selected` in `route/mod.rs` is a function of what the router holds, and - ranks by key last. Staying put makes it depend on the previous choice, so - keep the current next hop per identity and prefer it among equals. Mirror - it in `RouteHandoff.tla`, whose `Best` picks the lowest key, as the - `verify-route` skill says. -- Two routes of the same rank can still differ in whether they are - down-only, which decides whether an upstream neighbour is sent the route - at all. Stay put only among routes that agree on it, so a relay on a - down-only route still moves to an equal one it may export, and test that. -- Check that every router test that names the next hop it expects still - means what it says, rather than editing expectations to match. -- How a route is selected is being replaced - ([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27). - Apply this to the rule that stands when this starts. The scoped path router in - `rs/moq-net/src/route/path.rs` already stays on its next hop among equals. -- Measure with `Sim`'s `hop_updates` count and, if it has landed, - `mesh_sweep.rs`'s benchmark, before and after. - -Public API: none. Wire: none; behaviour only. - -## Related - -- [Relays split horizon per session](/quest/m3/route-split-horizon.md) - the other trim of route traffic diff --git a/quest/m3/route-split-horizon.md b/quest/m3/route-split-horizon.md deleted file mode 100644 index bc2e3fd269b..00000000000 --- a/quest/m3/route-split-horizon.md +++ /dev/null @@ -1,25 +0,0 @@ -# [S] Relays split horizon per session - -## Goal - -A relay never advertises a route back over the session it learned it from. -Since [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), each -neighbor holds two stream keys, so the router advertises a route back to its -source. That is loop-free, since the source refuses a route whose path names -it ([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27), but it -doubles route traffic between every pair of relays. - -## Plan - -Decided 2026-11-06 (maintainer, /quest-complete, a follow-up of #8023): ✅ -m3: lite-07 routing ships correct in the cut, and this trims its traffic -afterward. Not chosen: m2; m4. - -Suppress the advertisement per session rather than per neighbor, and measure -route messages between two relays before and after. - -Public API: none. Wire: none; behavior only. - -## Related - -- [A relay keeps its next hop among equal routes](/quest/m3/route-keep-next-hop.md) - the other trim of route traffic diff --git a/quest/m3/route-trace-validation.md b/quest/m3/route-trace-validation.md deleted file mode 100644 index a18518fcb96..00000000000 --- a/quest/m3/route-trace-validation.md +++ /dev/null @@ -1,47 +0,0 @@ -# [M] The exhaustive route search replaces RouteHandoff.tla - -## Goal - -One exhaustive check guards the lite-07 route rule, and it runs the compiled -code: the search over every delivery order on `path::Router` also checks -`Seen`, `StaleIsChased`, and `NoStaleReturn`, and `RouteHandoff.tla`, its -configs, and the route trace validation this quest first planned are gone, -with every skill, quest, and doc that pointed at the model pointing at the -search instead. - -## Plan - -Decided 2026-11-10 (maintainer, /quest-plan): trace validation as the -strongest guard against the route model drifting from the code -([moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447)), with resume, -reconnect, and subscribe each getting their own in m4. Moved to m3 on -2026-11-13 (maintainer, /quest-plan), since it waits on m2 work. - -Found 2026-11-14 by the [feature review](/quest/m3/release/feature-review.md): -the exhaustive route search -([moq-dev/moq#8729](https://github.com/moq-dev/moq/pull/8729), merged -2026-11-14) searches every delivery order on the compiled `path::Router`, -faster than TLC on the same bounds (0.44 s against 2 s on the triangle) and on -meshes TLC cannot reach. `RouteHandoff.tla` now mostly duplicates it: only -`Seen`, `StaleIsChased`, and `NoStaleReturn` stay outside it, and a search -over the compiled code cannot drift from it, which was this quest's whole -purpose for route. - -Decided 2026-11-14 (maintainer, pick on [moq-dev/moq#8764](https://github.com/moq-dev/moq/pull/8764)): -✅ keep the search and delete `RouteHandoff.tla` once #8729 lands (rejected: -keep the model and drop the search; keep both); #8729 has landed, so this is -ready. This replaces route trace -validation. - -- Fold the three rules into the search as properties it checks, and show - each one fails on a mutation that breaks it before deleting the model. -- Delete `rs/moq-net/verify/route/` and retarget or delete the - `verify-route` skill and its Coverage table, REVIEW.md's Models pointers, - and the quests that tell a PR to run `/verify-route`. -- The trace format and TLC recipe this quest was to build for the other - models now come from the first of their m4 quests: - [resume](/quest/m4/resume-trace-validation.md) builds them, and - [reconnect](/quest/m4/reconnect-trace-validation.md) and - [subscribe](/quest/m4/subscribe-trace-validation.md) reuse them. - -Public API: none. Wire: none. diff --git a/quest/m4/resume-trace-validation.md b/quest/m4/resume-trace-validation.md index fb854733727..57082e86d80 100644 --- a/quest/m4/resume-trace-validation.md +++ b/quest/m4/resume-trace-validation.md @@ -17,7 +17,7 @@ can keep passing on code that has moved on. Milestone m4. [reconnect](/quest/m4/reconnect-trace-validation.md) and [subscribe](/quest/m4/subscribe-trace-validation.md) reuse: route's trace validation, which was to build them, gave way to the - [exhaustive route search](/quest/m3/route-trace-validation.md) (decided + exhaustive route search, which [the router](/quest/m2/multicast/router.md) now carries (decided 2026-11-14, maintainer, on [moq-dev/moq#8764](https://github.com/moq-dev/moq/pull/8764)). The sweep records each step as the model action it corresponds to (the `verify-resume` skill's Coverage table is the mapping) with the variables From cd0e34865c191c07bbbb0c7aa4f4caaa73dd646d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 13:55:02 -0800 Subject: [PATCH 2/2] quest: record the maintainer's answers on warmth, moq-transport, ingest, and standbys Warm means demand, with pop skipping and one uplink per group of drones as the two cases requirement 8 must deliver. A moq-transport session is an endpoint. Redundant ingest is one Instance from synchronized encoders. The metadata secondary is opt in. Also follows main's deletion of the resume redesign quest, which the requests quest linked to. Co-Authored-By: Claude Fable 5.2 --- quest/m2/multicast/README.md | 55 +++++++++++++++++++---- quest/m2/multicast/accept.md | 6 ++- quest/m2/multicast/requests.md | 14 ++++-- quest/m2/multicast/warmth.md | 72 ++++++++++++++++++++++++------- quest/m3/cache-routing/README.md | 19 +++++--- quest/m3/route-health/README.md | 28 ++++++++---- quest/m3/route-health/failover.md | 19 +++++--- 7 files changed, 162 insertions(+), 51 deletions(-) diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md index 6a5ed5576f9..a9c81669e2e 100644 --- a/quest/m2/multicast/README.md +++ b/quest/m2/multicast/README.md @@ -11,8 +11,9 @@ global view, or single point of failure, on CDN tiers and on a self-forming mesh alike. Non-goals: choosing, dialing, and pricing links from measured quality, which -is Kixel's; browsers as transit; stitching different Instances together; and -moq-transport routing parity beyond each node using its nearest publisher. +is Kixel's; browsers as transit; stitching different Instances together; +moq-transport routing parity beyond each node using its nearest publisher; +and a moq-transport node between two of our relays. ## Plan @@ -26,7 +27,8 @@ The maintainer's requirements, which every child is judged against: 1. A subscription resumes only onto equal identity. 2. A publisher may hold two connections (a GOAWAY handover, redundant - ingest), and routes switch between them seamlessly. + ingest), and routes switch between them seamlessly. Both serve one + Instance, so redundant encoders are synchronized. 3. Some connections do not proxy: an edge passes its clients to its cores and its cores' routes to its clients, never one core's routes to another, and a leaf, such as a relay on a camera, advertises only what its host @@ -35,17 +37,19 @@ The maintainer's requirements, which every child is judged against: 5. Traffic grows as little as possible with the cluster. 6. No single point of failure. 7. It is proven to work. -8. Routing prefers warm caches: a route has a cold cost and a warm cost, and - routes coalesce onto a node that says it carries a broadcast, so no link - carries the same data twice. +8. Routing prefers warm nodes: a route has a cold cost and a warm cost, and + routes coalesce onto a node that already has demand for a broadcast, so + no link carries the same data twice. A CDN join stops at no pop without + demand, and drones with an uplink each settle on one that reaches the CDN. 9. Messages scale with the cluster: streams of messages, never one blob. 10. One stream carries what must stay ordered. 11. moq-transport is supported to a degree: identity does not carry, but two - publishers announcing one name on opposite sides of the network work. + publishers announcing one name on opposite sides of the network work. A + moq-transport node is always an endpoint. 12. No global view. With competing claims each node uses its closest, and nothing elects a leader. 13. A secondary route may carry metadata only, so a stalled primary is found - without waiting on QUIC keep-alives. + without waiting on QUIC keep-alives, where a node opts in. 14. No approach is assumed: earlier attempts are evidence, not constraints. What the audit found of today's layer: 6 and 12 hold, and 1 holds with no @@ -126,6 +130,41 @@ Settled: and a demoted route never takes a track back while the working one delivers. +Decided 2026-11-15 (maintainer, answering the requirements the plan's review +flagged as costly, on +[moq-dev/moq#8873](https://github.com/moq-dev/moq/pull/8873)): + +9. ✅ **Warm means demand, and two cases define requirement 8.** A node is + warm for an Instance while something downstream of it asks for it, not + because groups sit in its cache: "downstream demand instead of a yes/no + cache state". Pop skipping: a join stops at no pop without demand. Drones + that "self-elect who is the one who reaches out to the CDN": with an + uplink each, "the warm cost to a drone beats the cold cost to the CDN" + and one uplink carries the broadcast (rejected: narrowing 8 to joins that + stop at a carrier already on the path, which delivers neither case). + [Warmth](/quest/m2/multicast/warmth.md) measures both. +10. ✅ **A moq-transport session is an endpoint, never transit.** Its wire + has no path, chain, or Instance, so a moq-transport relay announcing + back a name it learned from us cannot be told from a nearer publisher. + That is unsupported: it must cost bounded work and touch no other + broadcast ([requests](/quest/m2/multicast/requests.md)), and nothing + detects it (rejected: ignoring a claim for a name already announced to + that session, which also hides a real second publisher behind it). +11. ✅ **Redundant ingest is one Instance from synchronized encoders.** Two + connections claiming one Instance serve the same frames under the same + group and frame numbers, and a publisher that restarts hard mints a new + Instance, as the draft already requires. Nothing checks it: a pair that + breaks it hurts its own broadcast. Encoders that are not synchronized + use two Instances, and a move between them is a restart the reader sees; + our importers are that case, since same-epoch importers were dropped + 2026-10-19. moq-transport never gets redundant ingest: it has no way to + flag an encoder restart, so every claim over it is a minted Instance. +12. ✅ **The metadata secondary is opt in** + ([route health](/quest/m3/route-health/README.md)): a mesh on links that + stall wants it, and the CDN's own tiers run without one (rejected: a + standby wherever a second route exists, which about doubles a track's + subscription state across the cluster). + The shape those decisions give, for the children to confirm or improve; each departure is a decision to list in its PR: diff --git a/quest/m2/multicast/accept.md b/quest/m2/multicast/accept.md index 59d0e01d04e..95c220c747e 100644 --- a/quest/m2/multicast/accept.md +++ b/quest/m2/multicast/accept.md @@ -9,11 +9,13 @@ PR reports the measured interruption for each kind of failure: - two publishers claiming one path from opposite sides of a mesh, where each relay uses its nearest and none loops; - one publisher ingesting on two relays under one Instance, losing either, - with readers across the mesh reading on; + with readers across the mesh reading on, and a pair under two Instances, + where the move is a restart and nothing is spliced; - a publisher handed over by GOAWAY to another relay, with no group missed or repeated; - moq-transport and lite-06 publishers through the route layer, including - two announcing one namespace on different relays; + two announcing one namespace on different relays, each a minted Instance + that no reader is moved between mid-stream; - three meshes joined by scoped links, in a chain and in a ring, where no claim circles and a claim ended at its source ends everywhere; - an edge that marks its core links and a leaf that marks every link, each diff --git a/quest/m2/multicast/requests.md b/quest/m2/multicast/requests.md index c27dafbb5b2..dd28f3db54d 100644 --- a/quest/m2/multicast/requests.md +++ b/quest/m2/multicast/requests.md @@ -60,6 +60,12 @@ fixtures: again. - Two route entries for one claim through one session send two SUBSCRIBEs that differ only in depth. +- A moq-transport peer that announces back a name it learned from this + relay looks like a nearer publisher, and a request sent to it comes + straight back. Unsupported + ([epic](/quest/m2/multicast/README.md) item 10): pin that the pair of + requests stays pending without retrying in a loop, and that other + broadcasts on both sessions read on. The loop fixture: form the loop the churn sweeps found (two unrelated failures at once, so two relays briefly route a publisher through each @@ -68,9 +74,10 @@ subscribers receive the next group and no upstream request is left pending. Cover a request answered before the loop and one still unanswered when it forms. -Fix in the layer that survives -[the resume redesign](/quest/m4/resume-redesign.md), and run the -`verify-failover` and `verify-resume` skills on what they cover. +Fix in the per-reader path, which stays: the producer-replacement redesign +that would have deleted it was dropped 2026-11-15 (maintainer, +/quest-complete). Run the `verify-failover` and `verify-resume` skills on +what they cover. Public API: none. Wire: none expected. If a refusal must tell "this route is gone" from "no such track", that is a lite-07 wire change to report, which is @@ -79,4 +86,3 @@ why this sits in m2. ## Related - [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) - its doubt rule lets a publisher leaving three or more relays at once loop a request until the news arrives -- [Resume as producer replacement](/quest/m4/resume-redesign.md) - rewrites the per-reader path some of these hazards live in diff --git a/quest/m2/multicast/warmth.md b/quest/m2/multicast/warmth.md index dcaae9aa4c6..4302b7a7338 100644 --- a/quest/m2/multicast/warmth.md +++ b/quest/m2/multicast/warmth.md @@ -3,13 +3,15 @@ ## Goal The [bake-off](/quest/m2/multicast/bakeoff.md)'s harness runs three ways for -a relay to learn that a neighbour already carries a broadcast, over the -prototype route rule, and the PR reports for each: the copies each link -carries and the total link cost of each broadcast's tree, against plain -shortest-path joins and against the cheapest tree; control messages per -reader change; the time to a first frame; and unfed readers and pull cycles -under churn. The maintainer picks one, and the PR rewrites -[the implementation quest](/quest/m3/cache-routing/select.md) for it. +a relay to learn that a neighbour already has demand for a broadcast, over +the prototype route rule, and the PR reports for each: whether a CDN join +skips every pop without demand and whether drones with an uplink each settle +on one; the copies each link carries and the total link cost of each +broadcast's tree, against plain shortest-path joins and against the cheapest +tree; control messages per reader change; the time to a first frame; and +unfed readers and pull cycles under churn. The maintainer picks one, and the +PR rewrites [the implementation quest](/quest/m3/cache-routing/select.md) +for it. ## Plan @@ -18,19 +20,42 @@ pick (rejected: picking one unmeasured). The warm wire was stripped from lite-07 on 2026-11-14 because its benefit over plain path routing was never measured. +Decided 2026-11-15 (maintainer, on +[moq-dev/moq#8873](https://github.com/moq-dev/moq/pull/8873); +[epic](/quest/m2/multicast/README.md) item 9): + +- ✅ A node is warm for an Instance while it has demand for it: something + downstream asked and its own upstream subscription is confirmed. Groups + left in a cache do not make it warm (rejected: a yes/no cache state). + Whether the signal is that fact alone or says how much demand stands + behind it is for the measurement. +- ✅ Two cases must work, so the pick is the cheapest mechanism that + delivers both: + - **Pop skipping.** Pops joined by links that pass over other pops. A join + for a broadcast no pop in between has demand for takes the long link and + costs those pops nothing. Once one of them has demand, a farther pop + joins through it when that is cheaper than the long link, and goes back + when the demand ends. + - **One uplink per group of drones.** Drones meshed with each other, each + with a costly uplink to the CDN. The first with demand reaches the CDN, + and the rest join it over the mesh, because a warm drone costs less than + a cold uplink. Two that reach out at once end on one uplink, and when + the drone holding it loses it, another takes over. + The three: -1. **On the path only.** A join stops at the first relay carrying the +1. **On the path only.** A join stops at the first relay with demand for the Instance, and equal-cost next hops are chosen by the rendezvous hash, so - siblings facing the same choice converge on one. No warm signal. This is - the baseline the other two must beat. + siblings facing the same choice converge on one. No warm signal. A join + never learns of demand off its own path, so this delivers neither case: + it is the baseline for cost, not a candidate. 2. **Ask on subscribe.** Each SUBSCRIBE answer, payload or metadata-only, - says whether the neighbour carries the Instance and its summed cost from - the source, and the relay takes the cheapest. The - [metadata-only secondary](/quest/m3/route-health/README.md) doubles as + says whether the neighbour has demand for the Instance and its summed + cost from the source, and the relay takes the cheapest. A + [metadata-only subscription](/quest/m3/route-health/meta.md) doubles as the query, so nothing is sent for a broadcast nobody asks about. Sweep how many neighbours a relay asks. -3. **Advertise.** A carrying relay announces it on the announce stream. +3. **Advertise.** A relay with demand announces it on the announce stream. The constraint the maintainer set for all three: a warm link costs its warm price and a cold link its cold price, both at least 1, summed from the source @@ -39,6 +64,21 @@ extended. The stripped rule broke it: a carrier offered its copy at no cost and a hash ranked carriers, and pulls formed a cycle in 3.3% of churn steps, most leaving a reader unfed. +What that constraint gives and what it leaves: with each relay's cost summed +along the path its data takes, two relays can never each prefer the other, +since one would need `b + link < a` and the other `a + link < b`. For the +same reason two drones that reached out at once, each at the cost of its own +warm uplink, never merge. Merging them needs a rule both compute alike, such +as the lower rendezvous hash of Instance and Node ID keeping its uplink, and +a move that completes only once the new copy delivers. Show that rule adds +no cycle, or report that simultaneous starts stay on two uplinks and what +that costs. + +Joining the cheapest point of the tree that exists is the greedy online +Steiner tree, which stays within a logarithmic factor of the cheapest tree +when readers only join (Imase and Waxman, 1991). Leaves and moves are what +the sweep adds to that. + A cost that depends on who subscribed is outside the [proof](/quest/m2/multicast/proof.md)'s theorem. Argue it separately and sweep it: taking a route only makes it cheaper for others, and leaving one @@ -46,8 +86,8 @@ only worsens an offer nobody uses. A cycle that outlasts the messages in flight fails the mechanism; count the transient ones with the readers they leave unfed. -Shapes where the three differ: two edges that each prefer a different core; -LAN relays with an uplink each; a ring with a far source; and one viewer per +Shapes where the three differ: the two cases above; two edges that each +prefer a different core; a ring with a far source; and one viewer per broadcast, where the first must cost no message at all. Earlier sweeps to compare with: a LAN mesh of 6 pulled one uplink copy per broadcast, a ring of 6 kept 12 copies for 8 broadcasts, and warm routes cost 3.1 to 4.5 diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index cfc86709e50..63405116fcc 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -2,12 +2,13 @@ ## Goal -A relay pulls each broadcast from the cheapest copy: a neighbour already -carrying its Instance when that costs less than a fresh copy from the source. -A LAN of relays with their own uplinks pulls one copy over one uplink, edges -that prefer different cores share one copy between regions, and no link -carries a group twice. Routing converges once demand and topology stop -changing, and a broadcast nobody reads costs no message of its own. +A relay pulls each broadcast from the cheapest copy: a neighbour that +already has demand for its Instance when that costs less than a fresh copy +from the source. A join stops at no pop without demand, drones with their +own uplinks pull one copy over one uplink, edges that prefer different cores +share one copy between regions, and no link carries a group twice. Routing +converges once demand and topology stop changing, and a broadcast nobody +reads costs no message of its own. Non-goals: steering clients to the relay that holds a broadcast; decisions from measured load or RTT; browsers serving copies they receive; and creating @@ -29,6 +30,9 @@ a node that says it carries a broadcast). What stands and what goes: - ✅ A warm cost is strictly positive and summed from the source along the path the data takes. This replaces item 14's rule below, under which a carrier offered its copy at no cost and a hash ranked carriers. +- ✅ Warm means demand, not cache contents, and pop skipping and one uplink + per group of drones are the two cases that must work (maintainer, + 2026-11-15, [epic](/quest/m2/multicast/README.md) item 9). - The non-goal of "one copy across a multi-hop sparse overlay rather than among direct neighbors" is dropped: a mesh is the product this serves. @@ -53,7 +57,8 @@ This README's own work, once its child lands: plus a skip link: a cold broadcast takes the skip, a reader on the skipped relay makes the far relay join it, and that reader leaving sends it back. Two LAN relays with their own uplinks and readers on both pull one copy - over one uplink. + over one uplink, whether the readers arrive in turn or at once, and the + other uplink takes over when that one is lost. - Docs: warm and cold costs in `doc/bin/relay/cluster.md`, and a section on the routing concept page once the [routing guide](/quest/m3/routing-guide.md) has added it. diff --git a/quest/m3/route-health/README.md b/quest/m3/route-health/README.md index 57b91c76d79..1a08030b568 100644 --- a/quest/m3/route-health/README.md +++ b/quest/m3/route-health/README.md @@ -2,12 +2,12 @@ ## Goal -A relay or endpoint that demands a track over one route watches a second -route to the same Instance with a metadata-only subscription, and moves the -track when the first stalls, without waiting on a QUIC keep-alive or idle -timeout. Today a route that dies without a close starves its readers until -the 10 s idle timeout fires, and a session that stays up while its payload -stalls is never recovered +A relay or endpoint that opts in and demands a track over one route watches +a second route to the same Instance with a metadata-only subscription, and +moves the track when the first stalls, without waiting on a QUIC keep-alive +or idle timeout. Today a route that dies without a close starves its readers +until the 10 s idle timeout fires, and a session that stays up while its +payload stalls is never recovered (`rs/moq-net/tests/integration/route_health.rs`). ## Plan @@ -38,6 +38,18 @@ made by the maintainer on 2026-10-13 and 2026-11-02 from the stalls - ✅ Only between claims of one Instance the publisher named. A minted Instance, as an older version or moq-transport gets, never resumes. +Decided 2026-11-15 (maintainer, on +[moq-dev/moq#8873](https://github.com/moq-dev/moq/pull/8873)): ✅ opt in, +off by default: "We wouldn't do standbys inside our CDN". A standby about +doubles a track's subscription state and sends its headers down a second +path, which a mesh on links that stall wants and a CDN tier does not. The +node that would hold the standby turns it on, a relay in its configuration +and an endpoint on its client (rejected: a standby wherever a second route +exists). Turning it on for one broadcast, so an operator can offer it there, +is a later decision that [failover](/quest/m3/route-health/failover.md) +leaves room for. A node that has not opted in still finds a silent death by +the idle timeout alone and never recovers a stall on a live session. + New on 2026-11-15, from the prior art this is closest to: - The standby is a route whose path does not name this node, and one that @@ -57,5 +69,5 @@ track back, two quiet routes stay unknown, a bottleneck shared by both routes does not hide a working third, and relays deciding at once do not swap. Report media overlap, control traffic, wakes, switches, and memory across tracks, candidates, and readers. Revisit the relay's 10 s default QUIC idle -timeout with the result, and add a route-health section to the routing -concept page. +timeout with the result, since it stays the only detector wherever the +standby is off, and add a route-health section to the routing concept page. diff --git a/quest/m3/route-health/failover.md b/quest/m3/route-health/failover.md index 074c917db5c..8ef2f3d5080 100644 --- a/quest/m3/route-health/failover.md +++ b/quest/m3/route-health/failover.md @@ -2,8 +2,8 @@ ## Goal -For each track with payload demand, a relay or endpoint holds a -metadata-only subscription on its next-best route to the same Instance, and +For each track with payload demand, a relay or endpoint that opted in holds +a metadata-only subscription on its next-best route to the same Instance, and promotes it in place when the active route's delivered media falls behind what the standby reports by more than the reader's budget. The track then stays on the working route. A silent death and a payload stall on a live @@ -14,8 +14,14 @@ media and as never recovered, both recover within that budget. The [epic](/quest/m3/route-health/README.md) holds the decisions: comparison as evidence, one standby per demanded track, the doubling exploration, no -return to a demoted route, per track, and only for an Instance the publisher -named. +return to a demoted route, per track, only for an Instance the publisher +named, and opt in. + +- The opt-in is off by default and belongs to the node that holds the + standby: a field in the relay's configuration and an option on the client, + named alike in Rust, JS, and the bindings. With it off, nothing here costs + a message or a subscription. Read the switch where a track's source is + chosen, so a later per-broadcast switch changes one lookup. - Each track chooses its source apart from the broadcast's shared route selection: the front already holds every usable route to its Instance @@ -32,8 +38,9 @@ named. - Measure the interruption for a clean close, a silent death, and a payload stall, against today's: about 0, the 10 s idle timeout, and never. -Public API: none expected beyond -[`Subscription.meta`](/quest/m3/route-health/meta.md). Wire: none beyond it. +Public API: the opt-in, in the relay's configuration and on the client, +beside [`Subscription.meta`](/quest/m3/route-health/meta.md). Wire: none +beyond it. ## Required