Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions quest/m2/cluster-routing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,10 +119,12 @@ fixtures and controlled-time regressions, including residual stale revival,
not only results from deployment tooling. Do not claim per-origin aggregation
reduces origin count when every application publishes one broadcast.

Non-goals: measured RTT/loss pricing by default, cache-dependent route prices,
TURN, browser transit, full link-state topology, or cross-operator economic
exchange rates. Basic federation does not wait for
[routing cost domains](/quest/m5/routing-cost-domains.md).
Non-goals: measured RTT/loss pricing by default, TURN, browser transit, full
link-state topology, or cross-operator economic exchange rates. Basic
federation does not wait for
[routing cost domains](/quest/m5/routing-cost-domains.md). Node routes here
are the cold cost that [cache-aware routing](/quest/m3/cache-routing/README.md)'s
per-broadcast warm routes rank against (decided 2026-11-08, maintainer).

Each implementation updates the docs it makes stale. The separate
[routing guide](/quest/m1/routing-guide.md) explains the combined design and
Expand All @@ -141,3 +143,4 @@ setup after the capabilities land.
- [Routing guide](/quest/m1/routing-guide.md) - concept page and federation setup; P2P extends it in m4
- [Route wakes](/quest/m4/route-wakes.md) - local work follows affected fronts rather than the whole covered table
- [Link quality](/quest/m5/link-quality.md) - optional measured link cost, distinct from delivery-health failover
- [Cache-aware routing](/quest/m3/cache-routing/README.md) - per-broadcast warm routes ranked against these node routes
1 change: 1 addition & 0 deletions quest/m3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ federation) land.
- [Release pull requests read the R2 Rust cache](/quest/m3/mbx-r2-release-cache.md) - after the cut brings the platform Rust cache to `release`
- [Binary release publishers run on their tag](/quest/m3/release-tag-publishers.md) - after the cut brings the workflows to `release`
- [Relays split horizon per session](/quest/m3/route-split-horizon.md) - a relay never advertises a route back over the session it came from
- [Cache-aware routing](/quest/m3/cache-routing/README.md) - a relay stays on a broadcast's path only for a local reader, two downstream sessions, or its one consumer's cheapest cold route, and joins a neighbor's copy instead of pulling another
- [crates.io trusts the workflow](/quest/m3/crates-trusted-publisher.md) - maintainer action: the crates the cut first-publishes by token get a trusted publisher for `release-rs.yml`
- [Rust releases publish through OIDC](/quest/m3/release-rs-oidc.md) - release-plz publishes with trusted publishing and `CARGO_REGISTRY_TOKEN` is revoked
- [One policy for oversized container metadata](/quest/m3/import-metadata-policy.md) - fMP4 and MKV refuse or stream huge metadata by one documented rule
Expand Down
104 changes: 104 additions & 0 deletions quest/m3/cache-routing/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# Cache-aware routing

## Goal

A relay stays on a broadcast's path only while it has a local reader, feeds
two or more downstream sessions, or is its one consumer's cheapest cold
route. Otherwise its child bypasses it, so a broadcast with one viewer
crosses fewer relays. A relay joins a neighbor
that already carries a broadcast instead of pulling a second copy, so a LAN
of relays with their own uplinks pulls one copy over one uplink. Routing
stays loop-free under stale state, converges once demand and topology stop
changing, and sends no per-broadcast message for an idle broadcast.

Non-goals: steering clients to the relay that holds a broadcast; decisions
from measured load or RTT ([Link quality](/quest/m5/link-quality.md));
browsers serving copies they receive; a relay on neither consumer's cheapest
route becoming a shared stop; one copy across a multi-hop sparse overlay
rather than among direct neighbors; and creating links, since routing
chooses only among the sessions an operator configured.

## Plan

Decided 2026-11-08 (maintainer, /quest-plan):

1. ✅ **Node routes are the cold cost, and optional per-broadcast warm routes
are the warm cost.** The per-origin ROUTE metric of
[Cluster routing](/quest/m2/cluster-routing/README.md) stays the cold
cost, static and one per origin. A relay carrying a broadcast may
advertise a warm route for it. This revises Cluster routing's non-goal of
cache-dependent route prices. (rejected: per-broadcast forwarding with a
cache-only subscribe and no advertisement; keeping routing cache-agnostic)
2. ✅ **A warm route is per neighbor and travels one hop.** It goes only to
direct neighbors and is never forwarded. B is warm toward N only while B
has demand besides N's (a local reader or another downstream session),
and withdraws once that demand has been gone for the copy's existing
linger, with no new timer. B sends it only to neighbors whose cold metric
for the origin is at least its own, since no other neighbor may use it
(item 4). Without the per-neighbor rule, a relay whose reader left stays
warm because it still feeds N, and N never skips it again. (rejected:
broadcast-wide warmth, as the pre-#3225 design had; forwarding warmth k
hops, which floods every warm transition)
3. ✅ **One cost per link, priced marginally.** Pulling through a warm
neighbor costs the link alone, since that neighbor's cold metric is
already paid; a cold neighbor costs the link plus its cold metric. A tie
goes to the warm route, then to item 4's hash. A topology that wants both
behaviors prices a link that skips one relay at more than one neighbor
link and less than two: a cold broadcast then takes the skip, and once
the skipped relay pulls for its own reader, its neighbor joins it. With
uniform prices both cases tie and fall to the hash. (rejected: a warm and
a cold price per link; a warm price the carrier seeds)
4. **Eligibility, which the proof must hold.** A relay may pull broadcast P
through neighbor N only when N's advertised route to the request's pinned
origin has a newer seqno, or an equal seqno with
`(cold metric, hash(P, N))` below `(feasibility distance, hash(P, self))`.
A feasibility distance never exceeds what its node advertised at that
seqno, so it strictly decreases along every forwarding edge even when an
advertisement is stale: no loop, no rank, no hold timer. The pre-#3225
rank was a cold cost each relay reported about itself, so a stale report
could let a ring of relays all let go at once, which its 500 ms hold only
narrowed. Route advertisements stay one selection per origin.
5. ✅ **Proof first:** model and sweeps before the implementation.
(rejected: implementing with targeted fixtures only)
6. ✅ **The proof picks the wire placement.** The warm message is designed
fresh rather than reviving ANNOUNCE_TAKEOVER, which
[moq-dev/moq#8063](https://github.com/moq-dev/moq/pull/8063) deleted in
favor of instance continuity. The proof sizes it and
picks lite-07 or an additive SETUP extension. The maintainer leans toward
lite-07 if the message is small; choosing lite-07 goes back to the
maintainer, since it moves the implementation into m2. (rejected:
reusing ANNOUNCE_TAKEOVER's slot)
7. ✅ m3, after the m2 route layer it ranks on. (rejected: m2; m4)

Prior art: warm/cold route prices
([moq-dev/moq#2925](https://github.com/moq-dev/moq/pull/2925)) were deleted
by prefix routes ([moq-dev/moq#3225](https://github.com/moq-dev/moq/pull/3225)),
and the pop-skipping questline (warm advertise, rank, peer reconfigure) was
pruned in [moq-dev/moq#4393](https://github.com/moq-dev/moq/pull/4393).
Related designs: PIM-SM's switch from a shared tree to a shortest-path tree
([RFC 7761](https://www.rfc-editor.org/rfc/rfc7761.html)), NHRP shortcuts
([RFC 2332](https://www.rfc-editor.org/rfc/rfc2332.html)), and Babel's
feasible successors ([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966.html)).

This README's own work, once its children land:

- End-to-end tests over real sessions, time mocked. A chain of three relays
plus a skip link, priced 3 and 5: a cold broadcast takes the skip, a reader
on the skipped relay makes the far relay join it, and that reader leaving
sends it back to the skip. Two LAN relays with their own upstream links and
readers on both pull one copy over one uplink, with the owner spread across
broadcasts by the hash.
- Docs: warm routes and how to price skip links in `doc/bin/relay/cluster.md`,
and a section on the routing concept page once the
[routing guide](/quest/m1/routing-guide.md) has added it.

## Required

- [Prove cache-aware routing](/quest/m3/cache-routing/proof.md) - warm routes and the eligibility rule hold in route-layer sweeps, and the wire placement is chosen
- [Warm routes and per-broadcast parent selection](/quest/m3/cache-routing/select.md) - relays advertise warm routes and pull each broadcast through the cheapest eligible neighbor

## Related

- [Cluster routing](/quest/m2/cluster-routing/README.md) - the node routes and feasibility distances this ranks on
- [Relays split horizon per session](/quest/m3/route-split-horizon.md) - per-session suppression of node routes, beside warm routes' per-neighbor rule
- [Route health](/quest/m5/route-health/README.md) - stalled-delivery failover, separate from choosing where a copy is pulled
63 changes: 63 additions & 0 deletions quest/m3/cache-routing/proof.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# [L] Prove cache-aware routing

## Goal

Sweeps over the route layer show that the [epic](/quest/m3/cache-routing/README.md)'s
warm routes and eligibility rule keep per-broadcast forwarding loop-free at
every step, under stale, reordered, and in-flight messages. Once demand and
topology stop changing, every demanded broadcast is delivered, parent choices
stop changing, and a relay carries a broadcast only when it has a local
reader, feeds two or more downstream sessions, or is its one consumer's
cheapest cold route, as the epic's Goal states. The warm message's shape and wire placement are chosen
and recorded in the epic's Plan.

## Plan

Build on the sans-IO `Router` in `rs/moq-net/src/route/`, which each
origin's `route::Node` drives, and the harness behind its handoff sweeps,
not the reference model in
`rs/moq-net/tests/integration/routing_contracts/`, which
[Route layer acceptance](/quest/m2/cluster-routing/routes/acceptance.md)
deletes. The router exposes one selected route per origin; the proof needs
each origin's eligible neighbors as well, so add that as a crate-private
query. Add per-broadcast demand and forwarding on top of the router state:
readers joining and leaving, each relay's parent per broadcast, warm routes
per neighbor, and the existing linger.

Fixtures, each a deterministic counterexample the rules must survive:

- A chain of three relays plus a skip link, priced 3 and 5: cold skip, warm
join, and the return to the skip when the skipped relay's reader leaves.
- A relay whose only consumer is N withdraws its warm route toward N. N
bypasses it over a cheaper skip link, and stays when the relay is N's
cheapest cold route.
- Two relays at equal cold metric, both warm: only the lower hash keeps its
upstream, the other joins it, and they never adopt each other.
- A ring of relays each holding a stale, lower report of a neighbor's cold
metric, the hazard behind the pre-#3225 hold: no loop and no broadcast left
without a source.
- A warm withdrawal in flight while N's request is on its way: the former
carrier forwards the request along its own route, since it answers the same
warm or cold, and forwarding stays loop-free.
- An origin seqno advance, a route loss, and a relay that drains (GOAWAY
makes a neighbor least preferred) while broadcasts are being pulled.
- A warm route for another instance of the same path is never joined.
- k LAN relays with their own uplinks: one uplink copy per broadcast in a
full mesh; record the copies a sparse ring keeps, which the epic accepts.

Sweeps churn random meshes: readers come and go, sessions drop and return,
and messages are held and delivered out of step. Check at every step that the
per-broadcast forwarding graph has no cycle, and at quiescence the Goal's
steady state. Report warm messages per demand change and confirm an idle
broadcast sends none.

Wire: design the warm message, which names at least the path, the origin's
Route ID, and the instance, and say which stream carries it and how a peer
that never sends one behaves. Size it against lite-07 and an additive SETUP
extension and record the choice and its reason in the epic's Plan. Choosing
lite-07 goes back to the maintainer, since it moves
[the implementation](/quest/m3/cache-routing/select.md) into m2.

## Related

- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the feasibility rules and handoff sweeps this extends
45 changes: 45 additions & 0 deletions quest/m3/cache-routing/select.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# [XL] Warm routes and per-broadcast parent selection

## Goal

Relays send warm routes to their direct neighbors as the
[epic](/quest/m3/cache-routing/README.md) decides. Each relay pulls every
demanded broadcast through its cheapest eligible neighbor, and moves an
existing copy when a cheaper eligible one appears or its warm parent
withdraws. A peer that never sends warm routes sees only cold routes and
behaves as it does today. Relays export how many upstream copies they hold
and how many moves warm routes caused.

## Plan

- Implement what [Prove cache-aware routing](/quest/m3/cache-routing/proof.md)
settles and pass its fixtures: the warm message where it placed it, warm
state per neighbor derived from each front's demand, and eligibility over
the router's feasibility distances.
- Extend today's selection rather than adding a second selector.
`route_order` in `rs/moq-net/src/model/origin.rs` already hashes equal
routes per path, and a front already moves to a better route of its own
instance and resplices (`rs/moq-net/src/model/front.rs`).
- A warm route names the origin and instance it carries, and a front moves
only to a copy of its own instance.
- Bound the warm state a peer can make a relay hold by design, as
announcements are bounded.
- Only relay sessions that negotiated warm routes exchange them. A browser
never parents, so `js/net` needs at most to skip the message.
- Export upstream copies held, warm routes sent and received, and moves by
cause through the relay's existing metrics.
- `rs/moq-net/AGENTS.md` says a relay behaves the same warm or cold. That
covers answers to requests; warm routes change only which neighbor a
request goes to. If the rule's wording contradicts warm routes, bring a
rewording to the maintainer.

Public API: report any relay configuration added for warm routes. Wire: the
warm message as the proof placed it, with its draft section in the same PR.

## Required

- [Prove cache-aware routing](/quest/m3/cache-routing/proof.md) - the fixtures and wire placement this implements

## Related

- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the route layer driving lite-07 sessions since [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), whose feasibility distances eligibility reads
4 changes: 2 additions & 2 deletions quest/m4/p2p/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,8 @@ identity becomes a public key, independent of P2P.
Decided 2026-10-13: priority then total cost selects among authorized routes;
a direct connection does not always win. A watching tab never re-serves what
it receives. Offloading a room of watchers is a local relay's job
(`moq-cli --p2p` is the small one), which keeps warm and cold costs and cache-aware routing out of the
protocol. [Peer route policy](/quest/m4/p2p/cost-scopes.md) sets the cost
(`moq-cli --p2p` is the small one); warm routes pass only between relays
([cache-aware routing](/quest/m3/cache-routing/README.md)). [Peer route policy](/quest/m4/p2p/cost-scopes.md) sets the cost
knobs and defaults.

Target the public `@moq/net` interface regardless of whether its
Expand Down
5 changes: 3 additions & 2 deletions quest/m4/p2p/cost-scopes.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ Decided 2026-10-13, replacing the unconditional direct-first policy:

- Applications explicitly choose which discovered peers to connect to and
assign per-connection priority. Priority wins before configured additive
cost; links cost one by default and zero is refused (decided 2026-10-13). No automatic dial-all, cache price,
warm/cold route state, or RTT-based repricing by default.
cost; links cost one by default and zero is refused (decided 2026-10-13). No automatic dial-all or
RTT-based repricing by default. A browser sends no warm routes; those pass
only between relays ([cache-aware routing](/quest/m3/cache-routing/README.md)).
- Within a priority, compare complete route cost including the origin's
prefix seed. A direct routing node loses when its total route is more
expensive. Equal distinct routes retain deterministic path hashing; newest
Expand Down