From c342b4b8e0de7d5d65bd54bb100fbe0153db8ca5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 19:36:50 -0800 Subject: [PATCH 1/2] quest: plan cache-aware routing Co-Authored-By: Claude Opus 5.5 --- quest/m2/cluster-routing/README.md | 11 +-- quest/m3/README.md | 1 + quest/m3/cache-routing/README.md | 103 +++++++++++++++++++++++++++++ quest/m3/cache-routing/proof.md | 62 +++++++++++++++++ quest/m3/cache-routing/select.md | 45 +++++++++++++ quest/m4/p2p/README.md | 4 +- quest/m4/p2p/cost-scopes.md | 5 +- 7 files changed, 223 insertions(+), 8 deletions(-) create mode 100644 quest/m3/cache-routing/README.md create mode 100644 quest/m3/cache-routing/proof.md create mode 100644 quest/m3/cache-routing/select.md diff --git a/quest/m2/cluster-routing/README.md b/quest/m2/cluster-routing/README.md index b3a4bcd0bff..ebf98a6e62f 100644 --- a/quest/m2/cluster-routing/README.md +++ b/quest/m2/cluster-routing/README.md @@ -119,10 +119,12 @@ fixtures and controlled-time regressions, including residual stale revival, not only results from deployment tooling. Do not claim per-origin aggregation reduces origin count when every application publishes one broadcast. -Non-goals: measured RTT/loss pricing by default, cache-dependent route prices, -TURN, browser transit, full link-state topology, or cross-operator economic -exchange rates. Basic federation does not wait for -[routing cost domains](/quest/m5/routing-cost-domains.md). +Non-goals: measured RTT/loss pricing by default, TURN, browser transit, full +link-state topology, or cross-operator economic exchange rates. Basic +federation does not wait for +[routing cost domains](/quest/m5/routing-cost-domains.md). Node routes here +are the cold cost that [cache-aware routing](/quest/m3/cache-routing/README.md)'s +per-broadcast warm routes rank against (decided 2026-11-08, maintainer). Each implementation updates the docs it makes stale. The separate [routing guide](/quest/m1/routing-guide.md) explains the combined design and @@ -141,3 +143,4 @@ setup after the capabilities land. - [Routing guide](/quest/m1/routing-guide.md) - concept page and federation setup; P2P extends it in m4 - [Route wakes](/quest/m4/route-wakes.md) - local work follows affected fronts rather than the whole covered table - [Link quality](/quest/m5/link-quality.md) - optional measured link cost, distinct from delivery-health failover +- [Cache-aware routing](/quest/m3/cache-routing/README.md) - per-broadcast warm routes ranked against these node routes diff --git a/quest/m3/README.md b/quest/m3/README.md index 4536d3edef8..754ec9c4f9e 100644 --- a/quest/m3/README.md +++ b/quest/m3/README.md @@ -47,6 +47,7 @@ federation) land. - [Release pull requests read the R2 Rust cache](/quest/m3/mbx-r2-release-cache.md) - after the cut brings the platform Rust cache to `release` - [Binary release publishers run on their tag](/quest/m3/release-tag-publishers.md) - after the cut brings the workflows to `release` - [Relays split horizon per session](/quest/m3/route-split-horizon.md) - a relay never advertises a route back over the session it came from +- [Cache-aware routing](/quest/m3/cache-routing/README.md) - a relay stays on a broadcast's path only for a local reader or two downstream sessions, and joins a neighbor's copy instead of pulling another - [crates.io trusts the workflow](/quest/m3/crates-trusted-publisher.md) - maintainer action: the crates the cut first-publishes by token get a trusted publisher for `release-rs.yml` - [Rust releases publish through OIDC](/quest/m3/release-rs-oidc.md) - release-plz publishes with trusted publishing and `CARGO_REGISTRY_TOKEN` is revoked - [One policy for oversized container metadata](/quest/m3/import-metadata-policy.md) - fMP4 and MKV refuse or stream huge metadata by one documented rule diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md new file mode 100644 index 00000000000..45fa26fbc5d --- /dev/null +++ b/quest/m3/cache-routing/README.md @@ -0,0 +1,103 @@ +# Cache-aware routing + +## Goal + +A relay stays on a broadcast's path only while it has a local reader or +feeds two or more downstream sessions. Otherwise its child bypasses it, so a +broadcast with one viewer crosses fewer relays. A relay joins a neighbor +that already carries a broadcast instead of pulling a second copy, so a LAN +of relays with their own uplinks pulls one copy over one uplink. Routing +stays loop-free under stale state, converges once demand and topology stop +changing, and sends no per-broadcast message for an idle broadcast. + +Non-goals: steering clients to the relay that holds a broadcast; decisions +from measured load or RTT ([Link quality](/quest/m5/link-quality.md)); +browsers serving copies they receive; a relay on neither consumer's cheapest +route becoming a shared stop; one copy across a multi-hop sparse overlay +rather than among direct neighbors; and creating links, since routing +chooses only among the sessions an operator configured. + +## Plan + +Decided 2026-11-08 (maintainer, /quest-plan): + +1. ✅ **Node routes are the cold cost, and optional per-broadcast warm routes + are the warm cost.** The per-origin ROUTE metric of + [Cluster routing](/quest/m2/cluster-routing/README.md) stays the cold + cost, static and one per origin. A relay carrying a broadcast may + advertise a warm route for it. This revises Cluster routing's non-goal of + cache-dependent route prices. (rejected: per-broadcast forwarding with a + cache-only subscribe and no advertisement; keeping routing cache-agnostic) +2. ✅ **A warm route is per neighbor and travels one hop.** It goes only to + direct neighbors and is never forwarded. B is warm toward N only while B + has demand besides N's (a local reader or another downstream session), + and withdraws once that demand has been gone for the copy's existing + linger, with no new timer. B sends it only to neighbors whose cold metric + for the origin is at least its own, since no other neighbor may use it + (item 4). Without the per-neighbor rule, a relay whose reader left stays + warm because it still feeds N, and N never skips it again. (rejected: + broadcast-wide warmth, as the pre-#3225 design had; forwarding warmth k + hops, which floods every warm transition) +3. ✅ **One cost per link, priced marginally.** Pulling through a warm + neighbor costs the link alone, since that neighbor's cold metric is + already paid; a cold neighbor costs the link plus its cold metric. A tie + goes to the warm route, then to item 4's hash. A topology that wants both + behaviors prices a link that skips one relay at more than one neighbor + link and less than two: a cold broadcast then takes the skip, and once + the skipped relay pulls for its own reader, its neighbor joins it. With + uniform prices both cases tie and fall to the hash. (rejected: a warm and + a cold price per link; a warm price the carrier seeds) +4. **Eligibility, which the proof must hold.** A relay may pull broadcast P + through neighbor N only when N's advertised route to the request's pinned + origin has a newer seqno, or an equal seqno with + `(cold metric, hash(P, N))` below `(feasibility distance, hash(P, self))`. + A feasibility distance never exceeds what its node advertised at that + seqno, so it strictly decreases along every forwarding edge even when an + advertisement is stale: no loop, no rank, no hold timer. The pre-#3225 + rank was a cold cost each relay reported about itself, so a stale report + could let a ring of relays all let go at once, which its 500 ms hold only + narrowed. Route advertisements stay one selection per origin. +5. ✅ **Proof first:** model and sweeps before the implementation. + (rejected: implementing with targeted fixtures only) +6. ✅ **The proof picks the wire placement.** The warm message is designed + fresh rather than reviving ANNOUNCE_TAKEOVER, which + [moq-dev/moq#8063](https://github.com/moq-dev/moq/pull/8063) deleted in + favor of instance continuity. The proof sizes it and + picks lite-07 or an additive SETUP extension. The maintainer leans toward + lite-07 if the message is small; choosing lite-07 goes back to the + maintainer, since it moves the implementation into m2. (rejected: + reusing ANNOUNCE_TAKEOVER's slot) +7. ✅ m3, after the m2 route layer it ranks on. (rejected: m2; m4) + +Prior art: warm/cold route prices +([moq-dev/moq#2925](https://github.com/moq-dev/moq/pull/2925)) were deleted +by prefix routes ([moq-dev/moq#3225](https://github.com/moq-dev/moq/pull/3225)), +and the pop-skipping questline (warm advertise, rank, peer reconfigure) was +pruned in [moq-dev/moq#4393](https://github.com/moq-dev/moq/pull/4393). +Related designs: PIM-SM's switch from a shared tree to a shortest-path tree +([RFC 7761](https://www.rfc-editor.org/rfc/rfc7761.html)), NHRP shortcuts +([RFC 2332](https://www.rfc-editor.org/rfc/rfc2332.html)), and Babel's +feasible successors ([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966.html)). + +This README's own work, once its children land: + +- End-to-end tests over real sessions, time mocked. A chain of three relays + plus a skip link, priced 3 and 5: a cold broadcast takes the skip, a reader + on the skipped relay makes the far relay join it, and that reader leaving + sends it back to the skip. Two LAN relays with their own upstream links and + readers on both pull one copy over one uplink, with the owner spread across + broadcasts by the hash. +- Docs: warm routes and how to price skip links in `doc/bin/relay/cluster.md`, + and a section on the routing concept page once the + [routing guide](/quest/m1/routing-guide.md) has added it. + +## Required + +- [Prove cache-aware routing](/quest/m3/cache-routing/proof.md) - warm routes and the eligibility rule hold in route-layer sweeps, and the wire placement is chosen +- [Warm routes and per-broadcast parent selection](/quest/m3/cache-routing/select.md) - relays advertise warm routes and pull each broadcast through the cheapest eligible neighbor + +## Related + +- [Cluster routing](/quest/m2/cluster-routing/README.md) - the node routes and feasibility distances this ranks on +- [Relays split horizon per session](/quest/m3/route-split-horizon.md) - per-session suppression of node routes, beside warm routes' per-neighbor rule +- [Route health](/quest/m5/route-health/README.md) - stalled-delivery failover, separate from choosing where a copy is pulled diff --git a/quest/m3/cache-routing/proof.md b/quest/m3/cache-routing/proof.md new file mode 100644 index 00000000000..394927330dd --- /dev/null +++ b/quest/m3/cache-routing/proof.md @@ -0,0 +1,62 @@ +# [L] Prove cache-aware routing + +## Goal + +Sweeps over the route layer show that the [epic](/quest/m3/cache-routing/README.md)'s +warm routes and eligibility rule keep per-broadcast forwarding loop-free at +every step, under stale, reordered, and in-flight messages. Once demand and +topology stop changing, every demanded broadcast is delivered, parent choices +stop changing, and a relay carries a broadcast only when it has a local +reader, feeds two or more downstream sessions, or sits on a consumer's +cheapest cold route. The warm message's shape and wire placement are chosen +and recorded in the epic's Plan. + +## Plan + +Build on the sans-IO `Router` in `rs/moq-net/src/route/`, which each +origin's `route::Node` drives, and the harness behind its handoff sweeps, +not the reference model in +`rs/moq-net/tests/integration/routing_contracts/`, which +[Route layer acceptance](/quest/m2/cluster-routing/routes/acceptance.md) +deletes. The router exposes one selected route per origin; the proof needs +each origin's eligible neighbors as well, so add that as a crate-private +query. Add per-broadcast demand and forwarding on top of the router state: +readers joining and leaving, each relay's parent per broadcast, warm routes +per neighbor, and the existing linger. + +Fixtures, each a deterministic counterexample the rules must survive: + +- A chain of three relays plus a skip link, priced 3 and 5: cold skip, warm + join, and the return to the skip when the skipped relay's reader leaves. +- A relay whose only consumer is N withdraws its warm route toward N, and N + bypasses it. +- Two relays at equal cold metric, both warm: only the lower hash keeps its + upstream, the other joins it, and they never adopt each other. +- A ring of relays each holding a stale, lower report of a neighbor's cold + metric, the hazard behind the pre-#3225 hold: no loop and no broadcast left + without a source. +- A warm withdrawal in flight while N's request is on its way: the former + carrier forwards the request along its own route, since it answers the same + warm or cold, and forwarding stays loop-free. +- An origin seqno advance, a route loss, and a relay that drains (GOAWAY + makes a neighbor least preferred) while broadcasts are being pulled. +- A warm route for another instance of the same path is never joined. +- k LAN relays with their own uplinks: one uplink copy per broadcast in a + full mesh; record the copies a sparse ring keeps, which the epic accepts. + +Sweeps churn random meshes: readers come and go, sessions drop and return, +and messages are held and delivered out of step. Check at every step that the +per-broadcast forwarding graph has no cycle, and at quiescence the Goal's +steady state. Report warm messages per demand change and confirm an idle +broadcast sends none. + +Wire: design the warm message, which names at least the path, the origin's +Route ID, and the instance, and say which stream carries it and how a peer +that never sends one behaves. Size it against lite-07 and an additive SETUP +extension and record the choice and its reason in the epic's Plan. Choosing +lite-07 goes back to the maintainer, since it moves +[the implementation](/quest/m3/cache-routing/select.md) into m2. + +## Related + +- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the feasibility rules and handoff sweeps this extends diff --git a/quest/m3/cache-routing/select.md b/quest/m3/cache-routing/select.md new file mode 100644 index 00000000000..a3c9994b1c1 --- /dev/null +++ b/quest/m3/cache-routing/select.md @@ -0,0 +1,45 @@ +# [XL] Warm routes and per-broadcast parent selection + +## Goal + +Relays send warm routes to their direct neighbors as the +[epic](/quest/m3/cache-routing/README.md) decides. Each relay pulls every +demanded broadcast through its cheapest eligible neighbor, and moves an +existing copy when a cheaper eligible one appears or its warm parent +withdraws. A peer that never sends warm routes sees only cold routes and +behaves as it does today. Relays export how many upstream copies they hold +and how many moves warm routes caused. + +## Plan + +- Implement what [Prove cache-aware routing](/quest/m3/cache-routing/proof.md) + settles and pass its fixtures: the warm message where it placed it, warm + state per neighbor derived from each front's demand, and eligibility over + the router's feasibility distances. +- Extend today's selection rather than adding a second selector. + `route_order` in `rs/moq-net/src/model/origin.rs` already hashes equal + routes per path, and a front already moves to a better route of its own + instance and resplices (`rs/moq-net/src/model/front.rs`). +- A warm route names the origin and instance it carries, and a front moves + only to a copy of its own instance. +- Bound the warm state a peer can make a relay hold by design, as + announcements are bounded. +- Only relay sessions that negotiated warm routes exchange them. A browser + never parents, so `js/net` needs at most to skip the message. +- Export upstream copies held, warm routes sent and received, and moves by + cause through the relay's existing metrics. +- `rs/moq-net/AGENTS.md` says a relay behaves the same warm or cold. That + covers answers to requests; warm routes change only which neighbor a + request goes to. If the rule's wording contradicts warm routes, bring a + rewording to the maintainer. + +Public API: report any relay configuration added for warm routes. Wire: the +warm message as the proof placed it, with its draft section in the same PR. + +## Required + +- [Prove cache-aware routing](/quest/m3/cache-routing/proof.md) - the fixtures and wire placement this implements + +## Related + +- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the route layer driving lite-07 sessions since [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), whose feasibility distances eligibility reads diff --git a/quest/m4/p2p/README.md b/quest/m4/p2p/README.md index 2ee0ad1e4af..0f701d5c625 100644 --- a/quest/m4/p2p/README.md +++ b/quest/m4/p2p/README.md @@ -140,8 +140,8 @@ identity becomes a public key, independent of P2P. Decided 2026-10-13: priority then total cost selects among authorized routes; a direct connection does not always win. A watching tab never re-serves what it receives. Offloading a room of watchers is a local relay's job -(`moq-cli --p2p` is the small one), which keeps warm and cold costs and cache-aware routing out of the -protocol. [Peer route policy](/quest/m4/p2p/cost-scopes.md) sets the cost +(`moq-cli --p2p` is the small one); warm routes pass only between relays +([cache-aware routing](/quest/m3/cache-routing/README.md)). [Peer route policy](/quest/m4/p2p/cost-scopes.md) sets the cost knobs and defaults. Target the public `@moq/net` interface regardless of whether its diff --git a/quest/m4/p2p/cost-scopes.md b/quest/m4/p2p/cost-scopes.md index 1f777dc1e04..e1a99ae2e45 100644 --- a/quest/m4/p2p/cost-scopes.md +++ b/quest/m4/p2p/cost-scopes.md @@ -12,8 +12,9 @@ Decided 2026-10-13, replacing the unconditional direct-first policy: - Applications explicitly choose which discovered peers to connect to and assign per-connection priority. Priority wins before configured additive - cost; links cost one by default and zero is refused (decided 2026-10-13). No automatic dial-all, cache price, - warm/cold route state, or RTT-based repricing by default. + cost; links cost one by default and zero is refused (decided 2026-10-13). No automatic dial-all or + RTT-based repricing by default. A browser sends no warm routes; those pass + only between relays ([cache-aware routing](/quest/m3/cache-routing/README.md)). - Within a priority, compare complete route cost including the origin's prefix seed. A direct routing node loses when its total route is more expensive. Equal distinct routes retain deterministic path hashing; newest From f1ad28a2da51bbb29ac53494ee2ccdaf9f1e6f72 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 20:51:59 -0800 Subject: [PATCH 2/2] quest: one carrying rule for cache-aware routing Co-Authored-By: Claude Opus 5.5 --- quest/m3/README.md | 2 +- quest/m3/cache-routing/README.md | 7 ++++--- quest/m3/cache-routing/proof.md | 9 +++++---- 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/quest/m3/README.md b/quest/m3/README.md index 754ec9c4f9e..e1ba55da519 100644 --- a/quest/m3/README.md +++ b/quest/m3/README.md @@ -47,7 +47,7 @@ federation) land. - [Release pull requests read the R2 Rust cache](/quest/m3/mbx-r2-release-cache.md) - after the cut brings the platform Rust cache to `release` - [Binary release publishers run on their tag](/quest/m3/release-tag-publishers.md) - after the cut brings the workflows to `release` - [Relays split horizon per session](/quest/m3/route-split-horizon.md) - a relay never advertises a route back over the session it came from -- [Cache-aware routing](/quest/m3/cache-routing/README.md) - a relay stays on a broadcast's path only for a local reader or two downstream sessions, and joins a neighbor's copy instead of pulling another +- [Cache-aware routing](/quest/m3/cache-routing/README.md) - a relay stays on a broadcast's path only for a local reader, two downstream sessions, or its one consumer's cheapest cold route, and joins a neighbor's copy instead of pulling another - [crates.io trusts the workflow](/quest/m3/crates-trusted-publisher.md) - maintainer action: the crates the cut first-publishes by token get a trusted publisher for `release-rs.yml` - [Rust releases publish through OIDC](/quest/m3/release-rs-oidc.md) - release-plz publishes with trusted publishing and `CARGO_REGISTRY_TOKEN` is revoked - [One policy for oversized container metadata](/quest/m3/import-metadata-policy.md) - fMP4 and MKV refuse or stream huge metadata by one documented rule diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index 45fa26fbc5d..e54ff8d0c75 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -2,9 +2,10 @@ ## Goal -A relay stays on a broadcast's path only while it has a local reader or -feeds two or more downstream sessions. Otherwise its child bypasses it, so a -broadcast with one viewer crosses fewer relays. A relay joins a neighbor +A relay stays on a broadcast's path only while it has a local reader, feeds +two or more downstream sessions, or is its one consumer's cheapest cold +route. Otherwise its child bypasses it, so a broadcast with one viewer +crosses fewer relays. A relay joins a neighbor that already carries a broadcast instead of pulling a second copy, so a LAN of relays with their own uplinks pulls one copy over one uplink. Routing stays loop-free under stale state, converges once demand and topology stop diff --git a/quest/m3/cache-routing/proof.md b/quest/m3/cache-routing/proof.md index 394927330dd..db66e537b47 100644 --- a/quest/m3/cache-routing/proof.md +++ b/quest/m3/cache-routing/proof.md @@ -7,8 +7,8 @@ warm routes and eligibility rule keep per-broadcast forwarding loop-free at every step, under stale, reordered, and in-flight messages. Once demand and topology stop changing, every demanded broadcast is delivered, parent choices stop changing, and a relay carries a broadcast only when it has a local -reader, feeds two or more downstream sessions, or sits on a consumer's -cheapest cold route. The warm message's shape and wire placement are chosen +reader, feeds two or more downstream sessions, or is its one consumer's +cheapest cold route, as the epic's Goal states. The warm message's shape and wire placement are chosen and recorded in the epic's Plan. ## Plan @@ -28,8 +28,9 @@ Fixtures, each a deterministic counterexample the rules must survive: - A chain of three relays plus a skip link, priced 3 and 5: cold skip, warm join, and the return to the skip when the skipped relay's reader leaves. -- A relay whose only consumer is N withdraws its warm route toward N, and N - bypasses it. +- A relay whose only consumer is N withdraws its warm route toward N. N + bypasses it over a cheaper skip link, and stays when the relay is N's + cheapest cold route. - Two relays at equal cold metric, both warm: only the lower hash keeps its upstream, the other joins it, and they never adopt each other. - A ring of relays each holding a stale, lower report of a neighbor's cold