diff --git a/quest/m1/playwright-pin.md b/quest/m1/playwright-pin.md deleted file mode 100644 index b53cf4601f6..00000000000 --- a/quest/m1/playwright-pin.md +++ /dev/null @@ -1,63 +0,0 @@ -# [S] Playwright bundles Chromium's spec exporter - -## Goal - -The toolchain's Playwright moves to a release whose bundled Chromium is 158 -or later, the first build whose `exportKeyingMaterial` follows -draft-ietf-webtrans-http3, so the route key lane -(`test/interop/clients/js/route-key.ts`) proves the Chromium tab. The lane -already requires a proof from Chromium 158 on, so the bump changes no -expectation. Firefox's exporter departs from the draft in every build so far. - -[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) removed the route key and its lane, so nothing -waits on this bump any more: [the switch's leftovers](/quest/m2/multicast/switch.md) -list restating or deleting this quest. - -The root flake's `pkgs.playwright-driver` supplies `PLAYWRIGHT_BROWSERS_PATH` -and `PLAYWRIGHT_VERSION` (`flake.nix`), and the npm pins -(`test/interop/clients/js/package.json`, `bun.lock`) must match it -(`test/smoke/freshness.sh`). As of 2026-11-06 both are 1.64.1, bundling -Chromium 156 and Firefox 157. npm's latest, 1.65.0, bundles Chromium -158.0.8127.18 and Firefox 159.0; no release bundles Firefox 160 yet, and -nixpkgs still carries 1.64.1. - -## Plan - -Moved from m2 to m1 2026-11-14 (the m2 triage before the cut, under -/quest-afk, for the maintainer's review): ✅ m1, since a toolchain bump breaks -nothing and builds on no m2 work (rejected: m2, where the cut waited on -nixpkgs; m4 or m5, which hold it behind the cut or the next wire cut once -nixpkgs lands). [The release cut](/quest/m3/release/cut.md) requires it in -place of lite07-finalize, since the cut PR's interop lane runs with -`CI_STRICT` and fails the route key lane on a Chromium below 158; -[lite07-finalize](/quest/m2/lite07-finalize.md) runs the lane once with -npm's Playwright 1.65.0 before the rename as an early signal. This amends -the 2026-11-06 decision below that the lite-07 cut waits for the bump. - -Decided 2026-11-06 (maintainer, /quest-complete, on Pullfrog's review of -[moq-dev/moq#8037](https://github.com/moq-dev/moq/pull/8037)): ✅ bump -Playwright, and the lite-07 cut waits for it. Not chosen: running installed -Chrome and Firefox channels instead of the bundled builds; leaving the lane -informational so the cut does not wait. - -Take the bump once nixpkgs carries 1.65.0 or later, and check that the lane -passes the Chromium tab. Refresh the flake lock and the npm pins together, as -`smoke-toolchain.yml` describes. Overriding -`playwright-driver` in the flake was considered and not chosen: it would -rebuild and pin browser binaries outside nixpkgs. - -## Waiting - -- nixpkgs: `playwright-driver` at 1.65.0 or later on `nixpkgs-unstable`; - check `pkgs/development/web/playwright/driver.nix` there. playwright-python - v1.65.0 shipped 2026-11-09, and an agent opened the bump from the - kixelated/nixpkgs fork the same day, - [NixOS/nixpkgs#585789](https://github.com/NixOS/nixpkgs/pull/585789) (it also - fixes `update.sh` for 1.65's SHA-pinned driver version and rebases the - python driver patch). A reviewer approved it on 2026-11-12, so it waits - for a nixpkgs committer to merge it, then for it to reach - `nixpkgs-unstable`. Decided 2026-11-11 (maintainer, - /quest-unblock): an agent drafts each review fix on the fork's branch with - an `Assisted-by:` trailer, as nixpkgs' CONTRIBUTING.md requires, plus a separate AI-disclosure line in - the reply itself, which that policy requires apart from commits; the - maintainer reviews the diff and reply before anything is pushed or posted. diff --git a/quest/m2/README.md b/quest/m2/README.md index ceecb49d41c..97cc32b28cd 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -32,8 +32,8 @@ removal, a rename, or a wire, catalog, or format change that is free until returned from m4, and the two moq-transport interop quests in m3 were deleted. Kept as they are: datagram tracks with their lite-05/06 bridge, all seven lite versions, the hidden `.` path prefix, and the route plan - of [cache-aware multicast](/quest/m2/multicast/README.md), whose - [switch](/quest/m2/multicast/switch.md) settled the public route API in + of [cache-aware multicast](/quest/m2/multicast/README.md), whose switch + settled the public route API in [moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053). The moq-cluster draft stays, since third-party relays ship it, rewritten to what the binding sends. @@ -103,7 +103,9 @@ Decided 2026-11-12 (maintainer, /quest-audit before the cut): the capture selections. None is on the routes chain, which stays the critical path. - ✅ Under the path rule a route keeps one sticky down-only flag beside its - path; see the [routes epic](/quest/m2/cluster-routes/README.md). + path; see the [routes epic](/quest/m2/cluster-routes/README.md). The + route layer that replaced it keeps the flag as a route's mark + ([cache-aware multicast](/quest/m2/multicast/README.md) item 6). Carried from m1's Plan before the re-cut, where "m1" means the whole pre-cut milestone, now m1 and m2: @@ -289,9 +291,10 @@ what would need a breaking release had it landed right after the cut: [moq-transport time addressing](https://github.com/moq-dev/moq/pull/8918) to m4 (low-priority compatibility, no named consumer), until it returned to m2 on 2026-11-16; and the - [Playwright bump](/quest/m1/playwright-pin.md) to m1, which the release - cut requires in place of lite07-finalize (its strict interop lane needs - Chromium 158), with lite07-finalize checking Chromium 158 once by hand. The E2EE nonce + Playwright bump to m1, which the release cut required in place of + lite07-finalize (its strict interop lane needed Chromium 158), with + lite07-finalize checking Chromium 158 once by hand; deleted with the route + key lane it served. The E2EE nonce quest was removed: the nonce already binds the wire ID, which becomes the start, and lite-07 by start rewords the draft. Rejected for each: staying in m2. diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index 10ed4200108..47e75f5c81f 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -2,527 +2,50 @@ ## Goal -Routing info splits into two layers on the session's announce stream, in -lite-07 (the current wip version, decided 2026-10-05), for every session. A ROUTE advertises reachability of one -origin node; an ANNOUNCE says a prefix lives at a route's node and carries no -path. A link flap or relay loss sends one ROUTE change per origin whose best -route changed, never a re-announce per broadcast; withdrawing an origin-owned -claim does not revive stale copies or end other origins serving the same epoch; failover to another neighbour with -the same epoch preserves continuity; and no stale neighbor copy revives a -claim whose withdrawal is already known. +Routing info travels in two layers on lite-07's Announce Stream, for every +session: a ROUTE advertises reachability of a routing node, and a claim says +a prefix is served there. A link flap or relay loss sends route changes per +routing node, never a re-announce per broadcast; no stale copy revives a +claim whose withdrawal is already known; and a subscription moves between +claims of one Instance without a restart. The route layer is accepted on +real sessions, and the lite-06 churn mitigations it replaces are retired. ## Plan -1. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **The draft describes the target before the code switches.** A hop-less ANNOUNCE cannot run before the router drives sessions; the quest's own Plan says so. So the draft now specifies the whole lite-07 route wire, while main's lite-07 session keeps sending hop-based ANNOUNCEs until route session integration ([moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023)) lands. lite-07 is unpublished `-wip`, off by default, and the epic lands in one release before the cut. The other option was to put only the new message sections in the draft and leave ANNOUNCE hop-based. I rejected it because ANNOUNCE_FLOOR's "for that route" and the Route ID would then be undefined in the draft. -2. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **The request field is `Node ID (i)`, with 0 meaning "the publisher selects".** It is added to every request that resolves a path: TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS. Node ID 0 is reserved everywhere. -3. ~~✅ Decided 2026-11-05 (maintainer, /quest-plan): **ROUTE_REQUEST names a Node ID, not a Route ID.** A request can then never race a ROUTE_END into a protocol violation.~~ Superseded by item 27: ROUTE_REQUEST is gone ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)). -4. ~~A held ROUTE_UPDATE carries a seqno and down-only bit of 0.~~ Superseded by item 14: a held route is its own message and carries no seqno or bit. -5. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Floor regions have no order, and a repeated region is a violation.** A sorted order would need Rust's `Path` ordering reproduced exactly in JS. Decoding into a map rejects duplicates at the same cost. Superseded the same day by item 10, which removes regions. -6. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **The Hop setup parameter and ANNOUNCE_OK's Hop ID are dropped from lite-07.** Their only roles were path-vector exclusion and the implicit trailing hop. ANNOUNCE_OK reserves no body for a route-key proof: [moq-dev/moq#7937](https://github.com/moq-dev/moq/pull/7937), accepted the same day, proves the route key once per session in SETUP. The live hop-based codec still switches in session integration, as choice 1 requires. -7. ~~✅ Decided 2026-11-05 (maintainer, /quest-plan): **An echo naming a Route ID that was never ended, or was already echoed, is a violation at the publisher.** The publisher can check this from its live and unacknowledged ids alone, with no growing state.~~ Superseded by item 27: the ROUTE_END echo is gone, and a subscriber sends nothing after its ANNOUNCE_REQUEST ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)). -8. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ANNOUNCE_TAKEOVER keeps its shape, with a Route ID in place of hops, and its predecessor must be under the same route.** [moq-dev/moq#8063](https://github.com/moq-dev/moq/pull/8063) deletes the message, so this keeps the change minimal. Decided 2026-11-06 (maintainer, /quest-complete on [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023)), asked whether session integration ports the message or lets lite-07 answer a claim as a restart or start until announce-takeover lands: ✅ **port it**, so a job answering a claim stays a takeover across relays. Rejected: accepting the restart and start. -9. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Unknown announce types stay skipped by length on lite-06+**, as before. An unknown reply type is refused. -10. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ANNOUNCE_FLOOR is a marker, not a region list:** `{Route ID, Revision}`. It closes a batch and certifies the stream's claims for that route across the whole requested prefix as the origin's at that Revision. A publisher that cannot vouch for the whole requested prefix at one Revision still closes the batch, with a floor that certifies nothing, encoded per item 13. Regions, the replacing list, and the 65,535-byte overflow go away. Accepted cost: a relay with partial knowledge of an origin certifies nothing for that route, so a receiver lets its stale claims go only on ROUTE_END. Rejected: one region per message; a multipart atomic list. -11. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Clamped claims collapse to the best one.** When several of a route's claims cover a request's prefix (`live` at cost 10 and `live/a` at cost 1, request `live/a/x`), the publisher presents only the most specific, and when it is withdrawn moves the presented claim to the next covering one with ANNOUNCE_UPDATE or ANNOUNCE_RESTART instead of ending it. No wire change; one live claim per prefix per route holds, and a receiver merges claims at the prefix each stream presents them. Rejected: carrying each claim's ancestor depth. -12. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **A floor certifies only what its own request shows.** A stream whose request did not set `Hidden` certifies nothing about hidden claims. No new wire field. Rejected: isolating differently filtered views. -13. ✅ Decided 2026-11-05 (maintainer): **ANNOUNCE_FLOOR encodes `Revision + 1`, with 0 meaning the floor certifies nothing,** the same offset as `Max Age`. Rust decodes it to `Option`. It replaces a Revision of 2^64-1 as the marker: a huge sentinel is a bad pattern and costs 9 bytes as a lite-07 varint. Rejected: a separate message; a boolean field. -14. ✅ Decided 2026-11-05 (maintainer: "use 0 as a sentinel, or make a new message type"; the agent picked the message): **A held route is ROUTE_HOLD (0xB), `{Route ID}`,** replacing a ROUTE_UPDATE with a Metric of 2^64-1. A metric of 0 is real (the origin's own route), so 0 cannot mark held, and offsetting every metric by one would still leave a held update's seqno and down-only bit meaningless and refused. The message carries only what a hold means, item 4's rule goes away, and a metric is now only bounded by overflow, so the wire refuses no metric. The 2^64-1 Seqno and Revision bounds stay: they are bounds, not markers, leaving room for a ROUTE_REQUEST's successor and for the floor's offset. -15. ✅ Decided 2026-11-06 (maintainer, /quest-complete on [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023)), asked whether to amend the handoff contract's "a sender that certifies anything for a route answers every stream" or keep it: ✅ **send a route on a stream only while it meets the stream's interest**, with a floor certifying the last withdrawal before the route ends. Withdrawals still END, and what a node keeps per stream it answers is bounded by the origins that stream reads, at a ROUTE_END round trip when an origin leaves an interest. Rejected: keep answering every stream and bound only by per-peer caps, which is simpler but carries the whole node table on every stream. [moq-dev/moq#8293](https://github.com/moq-dev/moq/pull/8293) implements it. -16. ✅ Decided 2026-11-09 (maintainer, /quest-unblock on [moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375)), asked how a token-scoped lite-07 session certifies withdrawals so a clean finish ENDs across a relay: ✅ **the subscriber narrows its ANNOUNCE_REQUEST to its scope's single region** (where its limit and its subscribe grant meet), and reopens the request when that region changes so a widening renewal isn't lost. Floors keep their exact whole-prefix meaning, with no wire change and one SHOULD in the lite-07 draft. A multi-region scope still certifies nothing. Rejected: floors carrying regions on the wire, which reverses item 10; landing with the finish lanes accepting offline. -17. ✅ Decided 2026-11-09 (maintainer, /quest-unblock on [moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375)), asked how item 16's reopen counts the session's announce cap while the old request keeps serving until the new one is SYNCED: ✅ **the old request gets its own budget for the handover**, moving what it holds onto a separate counter with the same cap, so the session's count belongs wholly to the new request. A peer can make the session hold at most twice the cap, and only during one handover. Rejected: dropping the old request at once on every region change, which flickers claims offline; holding a client's first request until its credential is answered and dropping the old one on later renewals. - -The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing), which reopened the route layer's rules after the echo ordering bug. - -18. ✅ Asked who is an origin node: **every publisher stays an origin node.** Its identity is for signing, for one node across several links (redundant publishers), and because browsers may forward (peer-to-peer routing, outside this repository since 2026-11-14). Rejected: a relay originating, under its own Node ID, the claims of any peer that sends no Node ID or holds a scoped publish grant, as it already does for lite-06 and moq-transport publishers. That would make route state follow relays rather than publishers and remove qualified Node IDs, the Route Key, and the echo, but a publisher would have no identity of its own. Accepted cost: routes, holds, and seqno requests scale with publishers, so a node must still forget dead identities, and the echo that makes forgetting safe has to be sound. -19. ✅ Asked whether to keep them: **ROUTE_HOLD stays, and the newest floor wins per identity.** Rejected: dropping holds, where each stall takes every claim behind the node offline and re-sends them all; and claims that follow only the selected next hop, which re-pulls a node's whole claim table on each next-hop change and showed revivals and false ENDs in the sweeps below. -20. ~~✅ Asked when an origin advances its seqno unprompted: **only when a route link changes**, implemented as a bump when a route link closes. Rejected: only to answer a ROUTE_REQUEST; keeping the per-stream bump.~~ Superseded by item 27: there is no seqno; a closed route link is reported by the DEAD naming its hop ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)). -21. ✅ Asked how one side's routing messages travel so none overtakes another: **one unidirectional Announce Stream per side**, carrying its requests, its answers, and its echoes in order ([the route stream](https://github.com/moq-dev/moq/pull/8835)). It replaces "There MAY be multiple Announce Streams". Rejected: one bidirectional Announce Stream per direction with the echo moved to the answering half; one bidirectional stream per session. -22. ✅ **A side's interest is a set of prefixes edited in place.** The live set never nests, and one edit may swap prefixes for a wider or narrower one without re-sending what both cover ([the interest set](https://github.com/moq-dev/moq/pull/8728)). A token-scope change is such an edit, with one prefix per region of the scope. This supersedes items 16 and 17. -23. ✅ Asked what a floor covers with several prefixes on one stream: **one prefix, named by its id.** This amends item 10: the floor is still a marker and never a region list. Rejected: a floor over the whole set. -24. ✅ Asked how a side learns a newly added prefix's initial set is complete: **SYNCED per prefix, naming its id**, plus one marker per edit where it takes effect for both sides. Rejected: one SYNCED per edit, counted in order. -25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region, unless it changes whether the node may use the route: by crossing it, or by reaching the hop limit. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes. Its echo and distance drain are superseded by item 27, which keeps the self-naming rule and drops the distance, the seqno, the request, and the echo ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)). - - -26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **a route is shared per publisher and per scope, and a relay believes a route by that route's own path** ([the prototype](https://github.com/moq-dev/moq/pull/8533), the model (`rs/moq-net/verify/route/RouteHandoff.tla`), then [the router](https://github.com/moq-dev/moq/pull/8622)). A scope is the prefix a relay asked a neighbour for when its grant there is narrower than its interest. Today only a feasible neighbour counts, one strictly cheaper than the distance the relay remembers, so a relay's claims move whenever a cost or seqno does: it can forget a claim for good, a mesh with two grants never settles after one seqno advance, and behind a scoped neighbour it never shows what a farther one knows. Only a broadcast some relay carries gets its own route, and that route propagates ([cache-aware routing](/quest/m3/cache-routing/README.md) Plan item 8). What keeps a route free of loops and stale copies is item 27. The contract's bullet on which neighbours certify now states the rule the model settled. First decided the same day as any neighbour whose path to the publisher does not pass through the relay; tried on the router, two relays behind a scoped relay then keep a dead claim alive through each other, and the relay behind a scoped neighbour shows a claim it has no route to request. Rejected: that rule; believing a neighbour that is cheaper than the relay is now, which fixed the storm and the forgotten claim with no wire change but rests belief on distance; a path per broadcast as lite-06 has; link-state among relays; routes to relays. - -27. ✅ Decided 2026-11-11 (maintainer, on [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533), which measured it), asked what keeps a route free of loops and stale copies: **the route's path, a withdrawal that names the hop that died, holds, and a timed doubt. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo go.** The model (`rs/moq-net/verify/route/RouteHandoff.tla`) checks it, with dead hops kept until their timer fires, and [the router](https://github.com/moq-dev/moq/pull/8622) builds it as `route::path::Router` beside today's, which only the tests drive. [The switch](https://github.com/moq-dev/moq/pull/8731) put it on the wire (item 28) and deleted Babel's router, and the contract below describes it as built. The rule, as `rs/moq-net/src/route/path.rs` runs it: - - A hop names a relay and the session it learned the route on. A relay uses the cheapest route whose path does not name it, at once. - - A relay that loses a session names that hop, and every relay drops every route through it, whoever offered it. One message per session covers every publisher and lists the routes that moved to another path. - - A relay with no route left holds its claims, unavailable, until each neighbour that learned the route from it has answered: with another way, or with nothing once those behind it have answered. Then it lets go. - - A closed session casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time. No relay moves to a route through that node until a timer lifts the doubt. The route a relay already uses is exempt, and so is a route that begins at that node, unless it comes over a session to that node opened since the doubt began (below). - - The maintainer delegated these decisions to the coordinator while away, recorded 2026-11-13: "The user is away; the coordinator takes recommendations on their behalf, open to change in review." - - Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was to be modeled and measured separately; the decision below covers it. - - Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](https://github.com/moq-dev/moq/pull/8734)): ✅ **doubt blocks a route through a node, not one that begins at it, except the node's own route on a session to it that opened since the doubt began.** A publisher on two relays whose link to one flaps faster than the timer stays routed through the other: a relay that already hears a route through the other home that does not name it keeps one the whole time, as every relay does in the measured mesh (the two homes joined, a third relay behind both), and a relay whose neighbours all route through it is routed again once one of them re-offers, a round trip after the news, instead of after the timer. A publisher that moves to another relay is routed in 50 ms in a 24-relay clique and 325 ms in a ring, where it waited 1,100 and 1,850 ms. The returning link still waits out the timer, so no relay hears it until the relay it returned to takes it, which keeps the timer's damping of a flapping link: in 400 churn cases (120,000 steps) with the publisher on one relay, the rule costs 3% more route messages than timer-only doubt with no fewer relays routed, and with the publisher on two relays it routes relays 15% more often for 22% more route messages, 6% more claims sent, and 4% more claims taken offline. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`). The two-home churn, which flaps random links and restarts relays but never takes a publisher off three relays, shows a route loop in 51 of its 120,000 steps. With two homes the model finds no loop and no claim dropped (`TwoHomes.cfg`), given a timer that outlasts every stale offer as well as the DEAD (`Patient`). This also settles the fresh-publisher quest, which proposed the opposite exemption and is deleted: its measurements showed that taking a returning link at once costs more traffic and claims offline than it saves. Rejected: keeping timer-only doubt, under which a dual-homed publisher's other link is unusable for the whole timer; sparing every route that begins at the doubted node, which routes relays 67% more often with two homes but costs 121% more route messages, 76% more claims sent, and 69% more claims offline, since each return of a flapping link reaches every relay; sparing only links heard before the doubt began, which keeps the damping but rarely helps a relay whose neighbours had not chosen the other link; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waited on [the switch](https://github.com/moq-dev/moq/pull/8731); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good. - - Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). - - Declined 2026-11-14 (maintainer), timer-only doubt retained over it: a fresh publisher's exemption, under which a relay moves at once onto a doubted publisher's own route when an authenticated session the publisher opened after the doubt began offers it. [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) modeled and measured it. The model found no broken rule (1.8M states, every ordering of the publisher rejoining two joined relays), but the exemption frees only the publisher's own relay: every other relay still doubts the publisher until its timer fires, and a publisher that moves to another relay gains nothing. Under churn it shows each flap of a publisher's link that the timer damps. Measured on 24 relays (sparse: 16 meshes; churn: 400 cases), recovery in ms, and dark counting observations of a reachable publisher with no route: - - | Case | Rule | Recovery | Dark | Messages | Claims | - | --- | --- | ---: | ---: | ---: | ---: | - | Clique rejoin | timer | 1,050 | 984 | 1,105 | 96 | - | Clique rejoin | exempt | 1,025 | 921 | 1,105 | 96 | - | Ring rejoin | timer | 1,875 | 1,512 | 118 | 96 | - | Ring rejoin | exempt | 1,800 | 1,380 | 120 | 88 | - | Sparse rejoin | timer | 1,372 | 1,194 | 164 | 100 | - | Sparse rejoin | exempt | 1,285 | 1,056 | 187 | 91 | - | Clique move | either | 1,100 | 1,032 | 1,612 | 2,124 | - | Ring move | either | 1,850 | 1,488 | 117 | 92 | - | Churn | timer | | 162,654 | 8,785 | 5,698 | - | Churn | exempt | | 116,404 | 13,646 | 10,524 | - - Neither rule looped, and churn's stale observations stayed near 3,300 under both. - -28. ✅ Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel), asked how item 27's removal is split and ordered: **the router first with no wire change, then a switch of the sessions, and the switch lands before the route stream.** [The router](https://github.com/moq-dev/moq/pull/8622) runs the path rule with a route per scope, sans-IO, beside today's router. [The switch](https://github.com/moq-dev/moq/pull/8731) puts it on the lite-07-wip wire, the draft, the Rust sessions, and `js/net`, and deletes Babel's router. [The interest set](https://github.com/moq-dev/moq/pull/8728) then edits the path router only, and [the route stream](https://github.com/moq-dev/moq/pull/8835) follows the switch with no echo or route request to carry. The switch's recommended wire puts the answer that ends a hold on the publisher half, behind the routes its sender sent first, so the path rule needs no order between streams that today's sessions lack for one prefix. [Acceptance](/quest/m2/cluster-routes/acceptance.md) is of the switched sessions, and the inductive invariant stays in m1, rewritten for the path, without `LoopFree` as an invariant. Rejected: the router quest in m1; one quest for router, wire, draft, sessions, and `js/net`; the route stream first, which builds the echo and ROUTE_REQUEST on the new stream and then deletes them; one wire change for the path rule and the one stream together; moving the inductive invariant to m4; deleting it. Reversed 2026-11-13 (maintainer, /quest-complete): ✅ the inductive invariant is deleted, since Apalache cannot step even one transition of the route model ([moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564)), no Rust TLA+ checker qualifies ([moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600), [moq-dev/moq#8669](https://github.com/moq-dev/moq/pull/8669)), and no more JVM model checking is wanted here (rejected: keeping it paused). - -29. ✅ Decided 2026-11-12 (maintainer, /quest-audit before the cut), asked how the path rule keeps the `upstream` guarantee, which the prototype left out and [the switch](https://github.com/moq-dev/moq/pull/8731) had listed for removal with Babel's machinery: **a route keeps one sticky down-only flag beside its path.** A relay sets it on a route it learns over a link marked `upstream`, it is never cleared on a later hop, and a relay never sends a flagged route over an upstream link. The flag is export policy, not loop avoidance, so the path does not replace it: a relay that hears a CDN route from a LAN peer learned it on a link that is not upstream, and only the flag tells it the route came down from an upstream. That case is a local cluster with more than one uplink ([outbound federation](/quest/m3/cluster-routing/federation.md)), where a relay whose own uplink died reaches the CDN through a peer; an edge that dials only its cores needs just the local rule. This is the Only-To-Customer attribute of BGP roles (RFC 9234). The router (`rs/moq-net/src/route/path.rs`, built in [#8622](https://github.com/moq-dev/moq/pull/8622)) carries the flag and its export check, [the switch](https://github.com/moq-dev/moq/pull/8731) adds them to `RouteHandoff.tla` and keeps the flag on ROUTE_START and ROUTE_UPDATE, and [acceptance](/quest/m2/cluster-routes/acceptance.md)'s two-uplink mesh proves it. Sender side only, as today. Rejected: a mark on each hop of the path, which gives the same answer from more bytes and model state; the local rule with no flag, which either leaks through a LAN peer or gives up the dead-uplink fallback; also refusing a flagged route that arrives from a link whose grant marks the sender as a customer, which can be added later without a wire change. - -30. ✅ Decided 2026-11-12 (maintainer, /quest-audit before the cut), after a survey of BGP, EIGRP, and their successors against item 27: most of the path rule already matches that prior art (AS_PATH loop detection, root-cause withdrawal, graceful restart, End-of-RIB, outbound route filtering, and item 29's Only-To-Customer flag). Three gaps are taken, all in [the switch](https://github.com/moq-dev/moq/pull/8731) and [acceptance](/quest/m2/cluster-routes/acceptance.md): **a dead-hop message on a scoped session applies only to the routes heard on that session and casts no doubt**, since in BGP a neighbour withdraws only what it sent, and a scoped peer (a federated relay, later a browser) could otherwise name a core hop it only read in a path and drop healthy routes outside its scope; **holds are bounded in time**, a neighbour's missing answer counting as nothing after a deadline, and a session whose announce stream has stopped draining is closed, which avoids EIGRP's stuck-in-active and BGP's zombie routes (RFC 9687), and is safe because a hold protects presence, not loop freedom; and **three tests before any design change**: neighbours noticing a dead relay up to the 3 s keep-alive apart (to size the doubt timer, measured in hops of delay today), a three-relay priority dispute wheel (RFC 4264's wedgies, since priority ranks above cost), and a primary that fails and returns. No wire bytes change; the first defines what the dead-hop message means. Rejected: declaring the link relation in SETUP and refusing leaked routes on receipt (RFC 9234), since the accepting side already knows the relation from its grant and it adds wire for a misconfiguration; MRAI and route-flap damping, which exist because BGP delays announcements; link state; per-hop signatures. - -31. Reopened 2026-11-15 (maintainer, /quest-plan on the whole route layer): ✅ **this rule is the baseline, and [cache-aware multicast](/quest/m2/multicast/README.md) replaces it on lite-07-wip before the cut.** An audit of what is built found requirements the path rule does not meet: its exhaustive checking stops at 3 relays and rests on a timer that outlasts every stale offer, route state follows publishers, nothing prefers a warm copy, and a stalled route is never left. Superseded there: item 18 (every publisher an origin node), item 19 (ROUTE_HOLD), item 26's route per scope, item 27's doubt and remembered dead hops, the Route Key and qualified Node IDs of the contract below, and item 14. Standing: the path with its self-naming rule and the withdrawal that names a dead hop (item 27), revisions and floors, one announce stream per side with its interest set (items 21 to 24), the sticky down-only flag (item 29), and drain, then priority, then cost. This epic's children still land: they are what the [bake-off](https://github.com/moq-dev/moq/pull/8880) measures against. Rejected: repairing this rule in place; freezing lite-07 with it and redesigning in lite-08. - -The route layer's core landed in -[moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676): a -crate-private, sans-IO `Router` in `rs/moq-net/src/route/`, generic over the -neighbor key and the claim attributes, with every holding rule of the -contract below. Each message recomputes only the identity it names. Since -[the switch](https://github.com/moq-dev/moq/pull/8731), every -lite-07 session in Rust and `js/net` drives the path router -(`route::path::Router`, through `route::node::Node`), and Babel's router is -deleted with its distance, seqno requests, and echo. The handoff fixtures, -both sweeps, and the route-layer fixtures of the contract run against it in -`route/tests/`, and real sessions with mocked time in -`tests/integration/path_routes.rs`. The forwarding fixtures run on the router -in `route/tests/forward.rs`, and the topology, failover, and cluster -acceptance fixtures on real sessions in `tests/integration/route_acceptance.rs`, -`route_failover.rs`, and `route_cluster.rs`; the Babel-era reference model is -deleted. The children below carry the rest, in order, and land in -one release: lite-07 carries the route layer as a whole -([lite-07 finalize](/quest/m2/lite07-finalize.md) waits on this epic). This -README holds the routing contract every child implements. - -Decided 2026-11-03 (maintainer, on -[moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676)): land the core -and split the quest into this epic. Options were: ✅ land the core and split; -one PR; the core plus the Rust wire there, with JS as a child. - -Settled by the core, for review: - -- Every lite-07 claim covers its own path and everything below it, so the - model's exact-versus-wildcard distinction is gone: a claim above an - interest overlaps it (`a_claim_above_an_interest_overlaps_it`). -- Letting go of a region keeps the proof's rule: keep every still-certified - region, or rebuild them all if a lost floor is newer than a kept one, a - collection whose replay is bounded like any other. The core and the proof - keep per-region floors internally; the wire carries only item 10's - whole-interest marker, which session integration ([moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023)) - projects the view onto. -- A rotated identity ends the old one with ROUTE_END on every stream rather - than closing sessions, since the own identity is no longer proven per - session. - -Keep the 2026-10-01 two-layer direction, reconciled with the maintainer's -2026-10-13 [decisions](/quest/m3/cluster-routing/README.md) and the routing -contract below. The handoff contract below, decided 2026-10-16, settles claim freshness: - -The lite-07 wire is specified in `drafts/draft-lcurley-moq-lite.md` (Announce, -Floors, and Routing), and `rs/moq-net/src/lite/announce.rs` encodes and decodes -its new messages ([moq-dev/moq#7764](https://github.com/moq-dev/moq/pull/7764)); -the bullets below are the contract it carries. - -- The announce cost is the origin's per-prefix seed (`transcode/**` at 10 and - `transcode/foobar` at 1 from one node, decided 2026-10-01) and nobody - changes it in transit; a path's cost through a route is that seed plus the - route metric. Link changes therefore touch only ROUTE, and routes rank on - the route metric alone, after drain and priority. This is today's single route cost, which stops - accumulating per hop once this lands. -- Decided 2026-10-13: the metric is configured additive cost with strictly - positive link increments, defaulting to one per link. Refuse zero-cost links; - origin prefix seeds may still be zero. Overflow is unreachable, never - saturated, using checked arithmetic. No hop-count component is needed: - each link strictly increases the metric, so extending a route never ranks - it better and selection converges, where a zero-cost link would let two - relays prefer each other's paths (`zero_cost_links_are_refused`, - `overflow_is_unreachable`). Update `?cost=0` examples with - the implementation and report the breaking configuration change. -- Decided 2026-10-13: GOAWAY gives the peer lowest preference, below every - eligible non-draining peer, and triggers failover. Retain it as a last resort - while connected; do not encode drain by saturating the metric. Using a - draining peer's routes locally without advertising them was rejected: - downstream nodes would lose their only route during the drain. As built - (decided 2026-11-13): a route drains when any session on its path drains, - carried in each offer and ranked before priority and cost, and a relay that - hears GOAWAY sends a DRAIN naming the leaving node in a DEAD to each - neighbour it sent a route through it. Every relay doubts that node and - moves at once only onto paths that avoid it: a draining path ranks worst - but stays usable, and no request loops through offers that predate the - drain (`goaway_moves_a_relay_to_a_path_that_does_not_drain`). Existing - readers resume only across a matching explicit epoch; epochless changes - preserve Restart. -- Fresh routing identity per process/tab restart, declared by the origin and - shared across its live sessions, decided 2026-10-13. It is unverified. - Decided 2026-11-02 (maintainer, /quest-unblock): peer grants are dropped and no key proof is - planned; a provable identity, if a use case needs one, makes this route - identity a public key, independent of P2P. Decided 2026-11-04 (maintainer, - /quest-unblock): scoped forgery is that use case; a peer proves a route key - in its lite-07 SETUP (the Route Key parameter, bound to the session by its - exporter), and every identity it sends while scoped, its own included, is - qualified under it. The own identity stays a fresh random id per process: - peers cannot check one derived from the key and a private salt, so deriving - it adds nothing. - Decided 2026-10-26 (maintainer): receivers do not stamp a fresh identity - per session. In a mesh, two relays stamping their sessions from one origin - forward its routes as two identities a third relay cannot recognize as - one, and a declared identity kept beside the stamp as a loop hint would - stay unverified. - Switching between paths resumes only across a matching explicit epoch. - Keep administrative identity and broadcast epoch separate. A node names - itself in paths by a hop fresh per process, since its session numbers - restart with it (`Router::new`); the origin's `Config::node` is both that - hop and its declared identity, so one set by hand must be fresh per - process too. A random identity alone does not keep a dead incarnation - away. What bounds it is ordered sessions, state scoped to the session that - delivered it, never advertising an unreachable route, the DEAD that drops - every route through a closed session's hop, the dead hop each node keeps - while a heard route names it, and the doubt on the far end: a dead - incarnation then reappears only from a message already in flight, which - delivers its withdrawal next. - `cluster.id` stops naming a routing identity. -- Decided 2026-10-14: every identity a scoped connection sends is - qualified: filed under a key the receiver derives, rather than its bare - id, when the connection's publish grant is scoped, and bare when it is - unscoped (publish `**` at the root). Its own identity is no exception - (2026-10-26): the bare exemption relied on the dropped proof, so an - unproven own identity could otherwise claim a victim's, as the fixture - below's forger does. The key has its own hash domain, so - it cannot collide with a key-derived identity, and is collision-resistant - at the identity width, since the forwarder picks `id` and could otherwise - grind another peer's key. The relay that files it translates it back when - forwarding a request into that connection. Claims under it are refused - outside the grant's scope. Routes are shared by all of an identity's - claims, so otherwise a scoped peer forging an identity, or declaring a - victim's id as its own, captures claims outside its namespace - (`a_forged_identity_is_contained_by_a_scoped_admission`); with it, a - forger or a misconfigured uplink only affects its own scope. The `peer` - claim no longer decides transit trust. Legitimate scoped delivery survives - losing one of two uplinks under one key - (`scoped_delivery_through_two_uplinks_reaches_one_key`). -- Decided 2026-11-04 (maintainer): a scoped peer's routing identities are - keyed by the route key the peer proves, so relays configured with the same - key file fail over as one; a scoped peer that proves no key, such as one on - qmux, gets a per-session key - ([moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023)). - The router takes the key from the caller (`Scope::qualify`). -- Decided 2026-10-14: qualified keys carry a marker every relay recognizes, - and a scoped connection forwarding an already qualified identity has that - route refused, with no wire change. Otherwise a key sent down through a - misconfigured uplink comes back and is filed again as a new identity on - every lap (`a_qualified_identity_is_never_qualified_again`). - Nested scoped federation is therefore out: the inner level must be - unscoped. Revisit with a wire change if a use case appears. Rejected: - keying a forwarded identity by its bare id and the narrowest scope it - crossed, which needs the scope in ROUTE, and accepting growth bounded only - by metric overflow. -- Loop freedom is the path rule (Plan item 27). A route carries its path, - each hop a node and the session it learned the route on, and a node never - uses a route whose path names it. A node that loses a session sends one - DEAD per neighbour naming that hop and listing the routes that moved, and - every node drops every route through the hop, whoever offered it, and keeps - the hop while a heard route names it. The far end of a closed session is - doubted for a second, since a dead relay is reported one neighbour at a - time: no node moves onto a route through it until the timer fires, and the - route a node already uses is exempt. On a scoped session a DEAD applies - only to the routes heard there and casts no doubt (Plan item 30). Selected - routes can loop only while the news that ends a stale route is on its way - (`sweep_no_route_outgrows_a_loop_free_path`). -- Holds replace starvation recovery. A node with no usable route holds its - claims, unavailable, and sends ROUTE_HOLD while the neighbour it learned - the route from holds it on the path this node's route extended, while a - neighbour offers a route that only the doubt keeps it off, or until each - neighbour that learned the route from it has answered, with another route, - a ROUTE_HOLD, or a ROUTE_END. A neighbour that holds under the node answers - with ROUTE_HOLD, so holds never wait on each other in a cycle. A hold lasts - 10 s at most, and a session whose announce stream stops draining for 10 s - is closed (Plan item 30): a hold protects presence, not loop freedom - (`holds_retire_once_nothing_can_recover`, and on real sessions - `a_hold_ends_behind_an_answering_neighbour`, - `a_hold_ends_behind_a_closed_session`, - `a_hold_ends_at_its_deadline_behind_a_silent_neighbour`, and - `a_peer_that_stops_reading_is_closed`). -- A detour that avoids the far end of a closed link is taken at once, and one - through it waits out the doubt - (`a_detour_avoiding_the_far_end_is_taken_at_once`, - `a_detour_through_the_far_end_waits_out_the_doubt`); a relay that dies - reaches every relay as withdrawals, and none moves onto a route through it - early (`a_dead_relay_reaches_every_neighbour_as_withdrawals`). Decided - 2026-10-13: instant failover is not a requirement; measure transport - failure detection, the doubt, and total interruption separately. -- Claim freshness follows the handoff contract below, independently of the - requested path's forwarding choice. -- Decided 2026-10-13: preserve application priority among transit routes, - then compare additive cost. Revised for the path rule on 2026-11-13 - (maintainer, [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)): - a local preference alone lets two relays prefer each other's paths forever, - so a route's priority is the lowest link priority on its path, and a route - drains if any session on it drains; drain, then priority, rank before cost, - and ROUTE_START and ROUTE_UPDATE carry both - (`mutually_preferred_links_settle`). -- Decided 2026-10-14: one selection per identity and prefix, keeping the `upstream` - link mark and its down-only bit. A node advertises its best route whether - or not anything subscribes. A route learned on an upstream link, or through - a neighbor's down-only route, is down-only, and a down-only route is never - sent to an upstream link, so CDN relays never shortcut between each other - through a local network. CDN content still flows into the network and local - content out, including through a peer whose own uplink died. The relay that - accepted a request from an upstream link ends that forwarded subscription - when its selected route turns down-only, and the first hop re-resolves; no - request bit goes on the wire. One selection hides a cheaper exportable - route whenever the preferred route is down-only - (`a_preferred_down_only_route_hides_the_exportable_alternative`); this is - accepted, and ranking local peers above a backup uplink avoids it, though a - route's priority is the lowest on its path, so it must outrank the whole - path. The two-view model it replaces lost its export restriction one LAN hop - after an upstream ingress, and identity-only aggregation deadlocked across - views (`the_cdn_never_reaches_itself_through_the_lan`, - `an_upstream_request_stops_once_the_route_turns_down_only`, - `a_dead_uplink_consumes_and_publishes_through_a_lan_peer`). A - misconfigured uplink that leaks CDN routes back is contained by - qualified identities rather than a downstream mark - (`misconfigured_uplink_is_contained_by_qualified_identities`). -- Decided 2026-10-14: a local per-connection `proxy` preference decides - whether a connection is sent routes learned from other connections; it is - never on the wire. Relays default on; applications choose per P2P - connection; browser tabs stay off until the Rust-to-TypeScript port. -- A plain client (`proxy` off) advertises its own origin and publications only, under - the one identity it declares on every live session. Clients receive only authorized, interested claims; opaque ids expose no - backbone addresses, but are not themselves authorization. A client - advertises nothing it receives. - -Handoff contract, proven 2026-10-15 by the fixtures in `handoff.rs` against -next-hop authority and a snapshot barrier alone. A next-hop change adopts the -newest knowledge any session has, never an older complete snapshot, and route -loss never reports an END: - -- Each origin numbers the changes to its own claims with a revision, a - varint that starts at zero for each routing identity. An origin never - advertises the last value, 2^64-1 (the largest lite-07 varint): a claim - change that would reach it rotates the identity instead, ending the old one - everywhere and announcing the claims again under a fresh one, so nothing - wraps (`exhausted_counters_rotate_the_identity`). - A session sends an origin's announcements in batches, each closed by its - floor: the revision at which the stream's announcements for that route, - across the route's prefix, are the origin's, or nothing when the sender - does not know them across that prefix at one revision (Plan item 10). -- As built, settled by `rs/moq-net/verify/route/RouteHandoff.tla` with Plan - item 27's rule: a route is keyed by its publisher and the prefix it covers, - and carries its own path. A node that asks a neighbour for a narrower prefix - gets one route there, covering that prefix: the neighbour's live route for - that prefix if it has one, else its route for a prefix above it. A node - believes a route that is live, whose path does not name it, and whose - stream answers its current interest, and keeps one view per prefix: the - newest floor any believed route covering that prefix gave it, kept whole - while it holds a route covering the prefix, so a lagging route cannot - revive a withdrawal another delivered, and a route that ends takes only - what no other route covers. While no believed route certifies the prefix, - the view shows what the selected route carries there, uncertified. A claim - shows from the narrowest prefix's view and is available while a live route - covers it. A request starts on the narrowest live route covering its claim, - names its publisher and that route's depth, follows the route each node - offered the one it came from, and is never forwarded along a route whose - path names the node it came from. A node remembers a dead hop while a - neighbour's route names it and until the timer the hop's close started - fires, so no route through a hop it knows is dead returns. Selected routes - can loop only while the news that ends a stale route is on its way, and - only through at least one stale route. What its timer costs fired early, - and the three-failure loop regression `Loop.cfg` pins, are in the - `verify-route` skill. A sender sends a route on a stream only while the - route meets the stream's interest (Plan item 15), and a node presents each - stream only the claims that changed. An interest change is one edit of the - prefixes on the session's Announce Stream, and at its ANNOUNCE_OK both ends - carry over the routes both sets cover and retire whatever lies outside the - new set ([the route stream](https://github.com/moq-dev/moq/pull/8835), - Plan item 22). -- Per identity and prefix, a node is usable (a live selected route: - announcements shown available), held, or let go. It holds as the path rule - says (above), showing the announcements as unavailable and sending - ROUTE_HOLD. Otherwise the node lets go: ROUTE_END to every neighbour it sent - the route to, and its reader sees the announcements go offline, a distinct - event from an END the origin sent; while held, the reader sees them - unavailable. A dead origin or a total partition goes offline everywhere - within the hold's bound (`holds_retire_once_nothing_can_recover`). No - promise is made of learning a withdrawal during a partition. -- A node may collect an identity's views and floors once it holds no route - for it; correctness does not depend on when. A dead hop stays while a heard - route names it and until its timer fires, and a doubt until its timer - fires; nothing else outlives the routes. After collection, or a process - restart, a stale view returns only from a sender that has not yet received - the withdrawal, and only until it does. Retained state is each session's - routes and their last views, one view per held identity and prefix within - the current interest, and those dead hops and doubts; neither withdrawal - history nor interest history leaves anything. One claim change costs one - batch per session carrying the route and one event per watching node - (`announcement_cost_follows_sessions_and_live_claims`). Without split - horizon a relay echoes an origin's claims back to it. -- A fresh identity per restart also starts a fresh revision space; reusing - an identity would need its revision persisted, and its hop's sessions - numbered past the old process's - (`origin_restart_takes_a_fresh_identity_for_its_claims`). -- Presence and availability are separate. A forwarded subscription follows - only a usable route, so a held route ends it as unreachability does while - the reader's announcement stays, unavailable. When the selected route - moves between usable next hops, keep the same-epoch subscription on the old - one until the new one delivers; epochless changes remain Restart. The model - does not cover the data path; the route-change acceptance tests do. -- `sweep_handoff` churns 150 random meshes per rule: two origins and a - client, claims changing, sessions dropping and returning, messages held - and delivered out of step, interest changing, origins restarting with - fresh identities, and route-less identities collected. Merged floors keep - every view equal to some revision of the origin's truth after every step, - and once churn stops every node shows each live origin's current claims and - holds nothing for a dead one. `sweep_scoped_handoff` churns a client - behind two scoped grants across interests inside, above, and outside both - scopes, with no false END. Over `sweep_handoff`'s scenarios, measured on - Babel's router on 2026-10-16: - - | Rule | Revivals | False ENDs | Re-announcements | Reader events | Claim KiB | - | --- | --- | --- | --- | --- | --- | - | Next-hop authority | 386 | 424 | 393 | 8853 | 856 | - | Snapshot barrier | 231 | 175 | 260 | 9086 | 825 | - | Merged floors | 0 | 0 | 30 | 9020 | 799 | - - Merged floors' re-announcements follow an offline period or a collection, - never a known withdrawal. Byte counts are estimates in virtual time, and - charge each batch its full region list, - with paths as suffixes of the interest. - -Decided with the maintainer 2026-10-16, from the handoff proof: - -1. Floor encoding: ✅ one ANNOUNCE_FLOOR closes each batch, and ANNOUNCEs - carry no revision. Its regions came with decisions 7 and 8. Rejected: a revision on every ANNOUNCE, - which adds bytes and removes no state, since a receiver still needs each - session's last view to apply a batch from a sender whose own view jumped; - and the floor on ROUTE_UPDATE, which ties claim churn to route processing. -2. Held encoding: ~~an infinite metric (2^64-1) means held and ROUTE_END - means let go, with no new field. Rejected: an explicit held bit.~~ - Superseded 2026-11-05 by Plan item 14: ROUTE_HOLD means held, and - ROUTE_END still means let go. -3. Reader API: ✅ an announcement carries availability, and letting go is a - distinct offline event, separate from the origin's END, in Rust and JS. - Rejected: presence alone, which gives a reader nothing to re-request on - when a held route recovers; let-go reported as END, as today; and - availability with let-go kept as END, which hides whether the origin - ended or the network lost it. -4. ~~Collection: ✅ collect routes and announcements once no neighbor's route - entry remains, held entries included. `sweep_handoff` checks selected - routes stay loop-free at every step; collecting while a neighbor's entry - remains loops at once. Rejected: Babel's source GC timer. - Revised after review: an empty route table does not prove that an echo of - the node's own advertisement has drained from every session, and - accepting that echo loops - (`collection_keeps_the_feasibility_distance_against_a_queued_echo`), so - decision 6 drains the feasibility distance instead.~~ - Superseded by Plan item 27 ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)): with no distance, a node collects an identity once it holds no route for it, and keeps only dead hops and doubts past their routes. -5. ~~Seqno requests: ✅ one pending request per origin at every hop, resent on - a next-hop change, with no timer and no hop count. Rejected: the - bounded-retry sketch, which strands a quiet origin's node. - Revised 2026-10-16 after review: a hop forwards to its next hop even when - that neighbour asked it, since excluding requesters strands requests - (`sweep_requests_reach_a_quiet_origin`).~~ - Superseded by Plan item 27 ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)): there is no seqno or request; holds and the doubt cover recovery. -6. ~~Feasibility-distance retention: ✅ drain it. A receiver echoes every - ROUTE_END back on the same stream once applied, and a node drops the - distance once it routes and advertises the identity nowhere and every - echo is back, a closed session counting as one. Reusing ROUTE_END adds no - message type. The neighbour's own ROUTE_END cannot be the reply: a client - or a node whose copy is infeasible never starts a route back, so the - distance would leak for every identity sent to a client. A per-neighbour - echo is enough, with no diffusing computation: once none holds the - advertisement, no node selects through this one, and every other node - keeps its own distance. A restart is the same drain, since its sessions - close with the process - (`restart_drops_the_feasibility_distance_with_every_session`). - `sweep_handoff` interleaves echoes freely with routes, stays loop-free at - every step, and leaves no distance for a dead identity; dropping the - distance before the echoes return fails the queued echo fixture, and - dropping it at no entry loops where the echoed condition never does - (`sweep_forgetting_distances`). Revised 2026-11-10 (Plan item 25): the - echo is not enough alone, since a third node can hold a copy derived from - the advertisement and hand it back after the drain. Each route names the - nodes it crossed, and a node never selects one that names itself - (`sweep_no_route_outgrows_a_loop_free_path`). Rejected: keeping the distance forever, - which grows with publisher churn times uptime on every relay that saw an - identity, about 16 bytes each, until a restart; and Babel's source GC - timer, which assumes stale echoes are gone by then rather than proving it.~~ - Superseded by Plan item 27 ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)): there is no distance to retain and no echo; the path's self-naming rule from item 25 stays. -7. ~~Scope coverage under one key: ✅ per-region floors (superseded on the - wire 2026-11-05 by Plan item 10; the core keeps regions internally). ANNOUNCE_FLOOR - names the region it certifies, a suffix of the stream's interest prefix - (the scope met with the interest), each node tracks which regions each - feasible neighbour's last view certifies, and a region none certifies - goes offline. One key can arrive through grants whose pattern unions - differ, and without this a scope whose last session closed stays shown, - available, with no way to learn its withdrawal; a relay forwarding such a - key also needs one floor per scope on a stream. Rejected: one pattern - union per key, which refuses or re-files a refresh that changes the - union and cannot be checked across relays; and deferring to the - implementation, which leaves the proof's freshness claim false for scoped - delivery.~~ - Superseded by Plan item 27 ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)): a route covers a prefix, and a node keeps one view per prefix from the newest floor any believed route covering it gave. -8. Region-set framing (superseded 2026-11-05 by Plan item 10, a - whole-interest marker): ✅ one ANNOUNCE_FLOOR per batch lists every region - the sender certifies for the route: (Route ID, count, then a region suffix - and revision per region). The list replaces the last, so a sender stops - certifying a region it lost by leaving it out, and a count of zero answers - an interest it cannot certify, which its peer needs to stop waiting - (`a_disjoint_interest_is_answered_with_nothing`). The common case is one - empty suffix, one byte over (Route ID, revision). Rejected: one region per - ANNOUNCE_FLOOR with revision 0 dropping a region, which gives revision 0 a - meaning and costs a floor per dropped region; and an end-of-batch marker, - a message for what one floor frames. - -Why not a hold-down: [moq-dev/moq#4644](https://github.com/moq-dev/moq/pull/4644) retracted at once and re-announced a replacement -after 1 s, which ended a downstream front whose only path ran through the -relay, dropping subscribers for about a second while the relay kept serving; -holding without retracting brought hunting back in measurements, and -#4642's 300 ms cursor hold was the earlier mitigation. Main also contains -[moq-dev/moq#5375](https://github.com/moq-dev/moq/pull/5375)'s two-second retraction quarantine and -[moq-dev/moq#5391](https://github.com/moq-dev/moq/pull/5391)'s same-clock queue dedup. -#5375 measured residual stale revivals, so neither longer holds nor one passing -mesh establish correctness. Separating reachability from claim freshness is -the direction, not yet a proof. Retire mitigations only after the incident -fixtures and failover tests demonstrate their replacement. - -Simulator findings (2026-10-12), measured with seed 1 on a tiered -45-node topology (four regions of two upstream nodes and their edges, plus -five single-node regions), tiered synthetic graphs of 347 and 1024 nodes, -and a six-node radio mesh with two uplinks. Bytes are uncompressed estimates -with eight-byte ids; these are protocol metrics in virtual time, not runtime -benchmarks. - -- Starvation: a five-node ring loses its preferred origin link while a - working four-hop backup remains. The backup's same-seqno route stays - infeasible for good, and restarting the backup's session does not help; - only an origin seqno advance does. Hence Babel's seqno request, which - Plan item 27 removed with the distance: the path rule takes the backup once - the doubt on the origin, the closed link's far end, lifts. -- Ending a broadcast on the 45-node topology costs 124.8 KiB against path - vector's 2817.0 KiB (1304.0 KiB with #4399's withdrawal hiding), but still - 2000 client updates against a 900 once-only minimum (up to 11 per node - and path), with 550 re-announcements. The synthetic graphs reach about - three times the minimum (20640 against 6940, 62182 against 20480). - Next-hop authority alone does not meet the Goal's "reaches each node - about once". -- Losing one of an edge's two upstream links leaves live broadcasts reported - as gone for 722.8 aggregate seconds before restoration, where path vector - reports none: the route moves, but the announces behind it trail. -- The down-only bit holds: with an uplink edge cut off from both of its - upstream nodes but still linked to the mesh, path vector routes its - subscriptions through the radio mesh and the split refuses, and after a - partition each island reaches the other through its uplink. +Superseded 2026-11-15 (maintainer, /quest-plan, item 31 of the earlier Plan): +[cache-aware multicast](/quest/m2/multicast/README.md) replaced this epic's +path rule on lite-07-wip before the cut, and +[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) shipped the +replacement. Gone with the old router: every publisher an origin node (item +18), ROUTE_HOLD (items 14 and 19), routes per scope (item 26), the doubt and +remembered dead hops (item 27), the Route Key and qualified Node IDs, and +the handoff contract's holds. The earlier Plan, items 1 to 31 with the +routing contract, the handoff contract and its fixtures, and the simulator +findings, is in `git show 4e73451999:quest/m2/cluster-routes/README.md`. + +Standing from it, as item 31 listed and the lite-07 draft carries them: a +route's path with its self-naming rule and the withdrawal that names a dead +hop (DEAD); revisions and floors, so the newest copy of a node's claims +wins; one Announce Stream per side with its interest set edited in place +and SYNCED per prefix (items 21 to 24); the sticky down-only flag (item 29), +now a route's mark; and drain, then priority, then cost. A request names the +routing node its first hop chose, and one naming none lets the publisher +select (item 2). + +Why not a hold-down: [moq-dev/moq#4644](https://github.com/moq-dev/moq/pull/4644) +retracted at once and re-announced a replacement after 1 s, which ended a +downstream front whose only path ran through the relay, dropping +subscribers for about a second while the relay kept serving; holding +without retracting brought hunting back in measurements, and #4642's 300 ms +cursor hold was the earlier mitigation. Main also contains +[moq-dev/moq#5375](https://github.com/moq-dev/moq/pull/5375)'s two-second +retraction quarantine and +[moq-dev/moq#5391](https://github.com/moq-dev/moq/pull/5391)'s same-clock +queue dedup. #5375 measured residual stale revivals, so neither longer holds +nor one passing mesh establish correctness. Retire the mitigations only +after the incident fixtures and failover tests demonstrate their +replacement, as [acceptance](/quest/m2/cluster-routes/acceptance.md) does. ## Related -- [Route wakes](/quest/m4/route-wakes.md) - index work follows the settled source-selection model +- [Route wakes](/quest/m4/route-wakes.md) - the origin's side of a route change diff --git a/quest/m2/cluster-routes/acceptance.md b/quest/m2/cluster-routes/acceptance.md index 65f87f7a3ba..849fd746ae8 100644 --- a/quest/m2/cluster-routes/acceptance.md +++ b/quest/m2/cluster-routes/acceptance.md @@ -26,21 +26,19 @@ fixing late routing flaws in lite-08). A flaw found here is fixed on lite-07-wip before it freezes. The fixtures landed in [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832), -and the reference model is deleted: forwarding on the router -(`rs/moq-net/src/route/tests/forward.rs`), and on real sessions with time mocked -the topology and prior-art fixtures (`tests/integration/route_acceptance.rs`), +and the reference model is deleted. On real sessions with time mocked: the +topology and prior-art fixtures (`tests/integration/route_acceptance.rs`), failover liveness and the live-graph tests of [moq-dev/moq#4644](https://github.com/moq-dev/moq/pull/4644) (`route_failover.rs`), the tiered cluster, partition, and scoped-interest checks (`route_cluster.rs`), -mount continuity (`mount.rs`), and operation-count regressions -(`route/tests/cost.rs`, `route/node/tests.rs`). They found the flaws the -quests under Required fix; the tests that pin each flaw say so in their docs. - -Reordered 2026-11-15 (maintainer, /quest-plan on -[cache-aware multicast](/quest/m2/multicast/README.md)): the retirement below -waits for [the switch](/quest/m2/multicast/switch.md), which rewrites these -fixtures' assertions for the new rule and deletes the Required quests it -makes moot. +and mount continuity (`mount.rs`). +[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) put lite-07 on +`route::relay::Router` and rewrote their assertions for it; the old +router's forwarding and operation-count fixtures went with it, and +`route/tests/multicast*.rs` covers the new one. The route fan-out quest +those pinned was deleted +with the old rule, its one surviving cost moved to +[route benches](/quest/m3/route-benches.md). Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit): the switch makes `origin::Config::update_hold` crate-private, since nothing @@ -54,8 +52,8 @@ path is part of retiring them below. What remains: -- Retire the quarantine, the dedup, and the hold once the Required quests - turn their pinned fixtures into the contract's behaviour. The quarantine +- Retire the quarantine, the dedup, and the hold, now that the fixtures + run on the route layer that replaces them. The quarantine is what turns the flaws into a 2 s gap: a relay that shows a held claim offline announces it again only after `RETRACT_QUARANTINE`. The hold and the quarantine sit on the announce cursor, which every version shares, so @@ -71,11 +69,6 @@ What remains: a node, so only publisher unit tests cover a relay answering 0. Cover it where `js/net` sends it, in `just test interop`. -## Required - -- [The switch](/quest/m2/multicast/switch.md) - the rule whose fixtures replace the mitigations, once the new route layer is on the wire -- [Route fan-out](/quest/m2/cluster-routes/route-fanout.md) - a route change costs only the streams and broadcasts it touches - ## Related - [Routes and announces](/quest/m2/cluster-routes/README.md) - the contract and its fixtures diff --git a/quest/m2/cluster-routes/route-fanout.md b/quest/m2/cluster-routes/route-fanout.md deleted file mode 100644 index 19af7b79c3d..00000000000 --- a/quest/m2/cluster-routes/route-fanout.md +++ /dev/null @@ -1,64 +0,0 @@ -# [L] Route fan-out - -## Goal - -A route change costs a relay work in the streams and broadcasts it touches: -not every answering stream, not every claim behind the route, and not state -remembered for sessions nothing was learned on. - -## Plan - -Reordered 2026-11-15 (maintainer, /quest-plan on [cache-aware -multicast](/quest/m2/multicast/README.md)): this waits on the -[bake-off](https://github.com/moq-dev/moq/pull/8880). Its first and third findings are -in `path::Router`'s indexes and its doubts and dead hops, which the new router -replaces; the second, per-broadcast rewrites in `route/node.rs`, the -router (`route::relay::Router`, from -[moq-dev/moq#8901](https://github.com/moq-dev/moq/pull/8901)) takes as a requirement. If the new -rule goes ahead, [the switch](/quest/m2/multicast/switch.md) deletes this -quest with the fixtures that pin it; if it does not, this starts as written. - -Kept 2026-11-15 (maintainer, /quest-complete on -[moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ keep it in -m2 (rejected: drop; moving it to m3). - -Found by [acceptance](/quest/m2/cluster-routes/acceptance.md)'s -operation-count regressions, each pinned by a `residual_*` test that fails -once fixed: - -- Unrelated streams (`route::tests::cost::residual_a_route_change_visits_every_unrelated_stream`, - `residual_closing_a_stream_revisits_every_other_stream`): every route change - visits every answering stream for each identity it recomputes, and changes - with no single route in scope (DEAD, drain, close) plan every prefix of - each stream. Closing one viewer costs the origins it watched times all - other streams, so viewer churn is quadratic. Measured (bench - `routes/path/repath`): one repath takes 9.8 µs beside 10 unrelated streams - and 1.09 ms beside 1,000. Recommended: index the answering streams in - `rs/moq-net/src/route/path.rs` by the prefixes they ask for and by the - routes sent to them. -- Per-broadcast rewrites (`route::node::tests::residual_a_route_update_reserves_every_claim`): - in `rs/moq-net/src/route/node.rs`, any change to a route re-serves every - claim behind it into the origin (`Node::home`), a route-table write per - broadcast that wakes the fronts below it. The wire - stays flat. Recommended: one shared record per route in the origin that its - broadcasts refer to, coordinated with [route wakes](/quest/m4/route-wakes.md). -- History-bound state (`residual_same_clock_churn_is_remembered_until_its_timer`, - `route::node::tests::residual_each_closed_session_is_remembered_until_the_doubt_timer`): - dead hops, doubts, and held routes grow with the churn inside one timer, - and a closed viewer session that sent no route still leaves a dead hop (and - a doubt if its client declared a node). Skipping the dead hop for a session - nothing was learned on looks safe, since no path can name it; check it with - the `verify-route` skill first. - -Build on [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734), -which also edits `path.rs`. - -## Required - -- [The switch](/quest/m2/multicast/switch.md) - deletes this with the old rule, unless the maintainer keeps today's rule on [moq-dev/moq#8880](https://github.com/moq-dev/moq/pull/8880) - - -## Related - -- [Route wakes](/quest/m4/route-wakes.md) - origin-side wakes per route change -- [Routes and announces](/quest/m2/cluster-routes/README.md) - the contract's cost bullets diff --git a/quest/m2/lite07-finalize.md b/quest/m2/lite07-finalize.md index 1ce18fe0bac..b80d9c8b9bf 100644 --- a/quest/m2/lite07-finalize.md +++ b/quest/m2/lite07-finalize.md @@ -24,13 +24,14 @@ until the maintainer calls the cut. This quest is the mechanical rename and checks done at that call; the Required list is the work already known to land first, not a closed freeze set. -The route layer of [Cluster routing](/quest/m3/cluster-routing/README.md) -(ROUTE_START/UPDATE/END, path-less ANNOUNCE, the native hop list dropped) lands in -lite-07, decided 2026-10-05. That removes lite-07's `Hop Base`/`Hop Keep` -announce compression along with the hop list, and moq.pro's lite-07 rollout -has to plan for a lite-07 without native hop lists; only the temporary Lite-06 -Chain, negotiated by its Setup Parameter, rides announcements until the lite-07 -to lite-08 migration. Only +A route layer lands in lite-07, decided 2026-10-05 and redesigned 2026-11-15 +(below). As shipped by [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987), +relays exchange ROUTE, ROUTE_END, DEAD, and DRAIN, every claim belongs to the +relay that announces it, and the native hop list is gone, with +`Path Base`/`Path Keep` compression in place of `Hop Base`/`Hop Keep`; a claim +from a lite-06 peer or across a scoped grant carries a chain instead, and +moq.pro's lite-07 rollout plans for that wire +([lite07-mesh](/quest/m5/lite07-mesh.md)). Only [Routes and announces](/quest/m2/cluster-routes/README.md) gates the cut, not the later federation/P2P integrations. Decided 2026-10-13: routes now owns the tier/partition correctness tests, so they run before the cut. @@ -41,8 +42,9 @@ wip is current when they land. Decided 2026-10-25: no routing identity proof gates the cut, and lite-07 routing identities stay declared and unverified (2026-11-02: peer grants are dropped and no key proof is planned). Reversed 2026-11-04 (maintainer, /quest-unblock): a peer proves a route key before the -cut, which landed as lite-07's Route Key Setup Parameter; its routes are filed -under it in [Routes and announces](/quest/m2/cluster-routes/README.md). Decided 2026-10-26 +cut, which landed as lite-07's Route Key Setup Parameter. Reversed again +2026-11-15 by the redesign below, which removed the parameter in +[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987). Decided 2026-10-26 (maintainer): receivers do not stamp per-session identities either, since that splits one origin into several identities across a relay mesh. @@ -123,6 +125,7 @@ moq-net): ✅ the cut waits for the symbolic model of the Route Key Parameter (rejected: neither proof gating, as the TLA+ models do not; the inductive routing invariant gating too, which sits behind the mesh model). The signed transcript freezes at the cut and a flaw in it can only be fixed on the wire. +Ended 2026-11-15 with the Route Key (below). Decided 2026-11-14 (the m2 triage before the cut, under /quest-afk; the maintainer picked these on @@ -137,7 +140,7 @@ would need a breaking release had it landed right after the cut: still waits on it (maintainer, /quest-complete on [moq-dev/moq#8768](https://github.com/moq-dev/moq/pull/8768); rejected: moving it to m3 and fixing late routing flaws in lite-08). -- ✅ [The Playwright bump](/quest/m1/playwright-pin.md) moved to m1 and +- ✅ The Playwright bump moved to m1 and gates [the release cut](/quest/m3/release/cut.md) instead of this: the cut PR's interop lane runs with `CI_STRICT` (`.github/workflows/interop.yml`), which fails the route key lane on a Chromium below 158, so only the pinned @@ -147,7 +150,9 @@ would need a breaking release had it landed right after the cut: the PR. If the tab fails, stop and report it, since only the wire can fix the signed transcript. Rejected: keeping the pin in m2, where the freeze waits on nixpkgs as well as the cut; dropping the check, which would find a - transcript flaw only after the freeze. + transcript flaw only after the freeze. Ended with the route key lane in + [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987): the + Playwright quest was deleted, since no lane needs Chromium 158. - ✅ The wire half of group drops split out as its own quest, which this waited on in place of the public drop signal, an API change that doesn't touch the wire. It settled without a @@ -161,8 +166,8 @@ lite-07 carries the redesigned route layer of for it (rejected: freezing lite-07 with the path rule and redesigning in lite-08; freezing lite-07 with no route layer). That redesign removes the Route Key Parameter, so the 2026-11-10 wait on its symbolic model and the -route key lane's early signal above end when -[the switch](/quest/m2/multicast/switch.md) lands. +route key lane's early signal above ended when +[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) landed. Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit, which also audited the lite-07 draft for what it could lose): @@ -197,7 +202,7 @@ the version that ships them. ## Required - [Cache-aware multicast](/quest/m2/multicast/README.md) - lite-07 carries the redesigned route layer: routes to forwarding nodes, no holds or doubt, claims crossing trust boundaries by a chain, and no Route Key -- [Routes and announces](/quest/m2/cluster-routes/README.md) - lite-07 carries the route layer: ROUTE per origin node and path-less ANNOUNCE, with the native hop list gone (only the temporary lite-06 compatibility chain remains) +- [Routes and announces](/quest/m2/cluster-routes/README.md) - the route layer's fixtures pass on real lite-07 sessions, and the lite-06 churn mitigations they replace are retired ## Related diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md index 96ffcfde827..0196142d0c8 100644 --- a/quest/m2/multicast/README.md +++ b/quest/m2/multicast/README.md @@ -2,8 +2,8 @@ ## Goal -moq-net gets one routing design, proven and measured before it replaces -lite-07's route layer. A subscription joins the nearest node already carrying +moq-net has one routing design, proven and measured, as lite-07's route +layer. A subscription joins the nearest node already carrying the same Instance, so no link carries a group twice, and moves between routes of equal Instance without the reader noticing. State and messages grow with forwarding nodes and demanded broadcasts, not publishers. There is no leader, @@ -17,11 +17,35 @@ and a moq-transport node between two of our relays. ## Plan +Where it stands (2026-11-18). The route layer shipped: +[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) put every +lite-07 session in Rust and `js/net` on `route::relay::Router`, rewrote the +draft's Routing, made `js/net` an endpoint, and deleted the path rule with +its holds, doubt, route key, qualified Node IDs, and routes per prefix; +[moq-dev/moq#9052](https://github.com/moq-dev/moq/pull/9052) made a relay +wait for a claim's route before answering a request for it; and +[moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053) made the +test-only route API crate-private, kept adopted claims off upstreams under +a marked Node ID, and marked the lite-06 churn mitigations; and +[moq-dev/moq#9082](https://github.com/moq-dev/moq/pull/9082) made a claim's +chain name its announcing relay, then each grant's exporter and adopter. +Its proof is +`rs/moq-net/verify/route-algebra/`, and the quests written against the old +wire were retargeted or deleted. The lite-07 cut still waits on this epic's +last child, [the uncovered cases](/quest/m2/multicast/accept.md), and on +[route layer acceptance](/quest/m2/cluster-routes/acceptance.md) retiring +the lite-06 mitigations. After the cut come warmth's wire +([cache-aware routing](/quest/m3/cache-routing/README.md)), +[route benches](/quest/m3/route-benches.md), +[the proof binding](/quest/m3/route-proof-binding.md), +[route health](/quest/m3/route-health/README.md), and +[outbound federation](/quest/m3/cluster-routing/federation.md). + Decided 2026-11-15 (maintainer, /quest-plan), after an audit of the lite-07 route layer as built, the history of about thirty routing attempts since 2023, and a survey of the literature. -[Routes and announces](/quest/m2/cluster-routes/README.md) holds the rule -this replaces and its measurements. +The rule this replaced, its contract, and its measurements are in +`git show 4e73451999:quest/m2/cluster-routes/README.md`. The maintainer's requirements, which every child is judged against: @@ -52,7 +76,7 @@ The maintainer's requirements, which every child is judged against: without waiting on QUIC keep-alives, where a node opts in. 14. No approach is assumed: earlier attempts are evidence, not constraints. -What the audit found of today's layer: 6 and 12 hold, and 1 holds with no +What the audit found of the path rule lite-07 then ran: 6 and 12 hold, and 1 holds with no model covering it; 4, 8, and 13 do not hold (16 message types on the announce stream and about 57 concepts against lite-06's 5 messages, no cache term in any ranking, and a payload stall on live sessions that is never recovered); @@ -75,7 +99,7 @@ Settled: the acceptance fixtures still land: they are the baseline the [bake-off](https://github.com/moq-dev/moq/pull/8880) measures against, and one announce stream per side is wanted by any design. The quests that only - polish holds, doubt, or the path router's indexes wait on the bake-off. + polished holds, doubt, or the path router's indexes went with it. 3. ✅ **A routing node is a node that forwards and is trusted to; claims, not routes, cross a trust boundary.** Every CDN relay and every mesh device routes under its own Node ID. An endpoint that never forwards (an @@ -248,19 +272,21 @@ judged the route layer at this plan's end state): layer; keeping the plan with a fallback written down now). The audit's case was the schedule, three XL quests in series before the freeze on a rule re-decided on six days, and that an endpoint must parse a protocol - only relays speak. What it leaves is in - [the switch](/quest/m2/multicast/switch.md): the public route API and - config that would freeze by omission, the half `js/net` implements, what - a non-forwarding endpoint must parse, and the moq-cluster draft, which - specified the warm discount item 5 forbids until it was rewritten to - what ships, since third-party relays ship it). + only relays speak. What it left went into the switch, which + [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) and + [moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053) finished: + the public route API and config that would freeze by omission, the half + `js/net` implements, and what a non-forwarding endpoint must parse. The + moq-cluster draft specified the warm discount item 5 forbids until it + was rewritten to what ships, since third-party relays ship it. -The shape those decisions give. The bake-off confirmed or changed each line; -a further departure is a decision to list in its PR: +The shape those decisions give, as +[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) built it; a +further departure is a decision to list in its PR: - **Identity.** A path and an Instance name content, and a request names the Instance. A relay holds one upstream subscription per Instance and track, - whichever origin node its downstream requests named, so a join stops at the + whichever routing node its downstream requests named, so a join stops at the first node already carrying it. - **Reachability.** One route per announcing relay, offered once it has a claim: its path, drain, priority, and cost. A node never uses a route @@ -281,7 +307,7 @@ a further departure is a decision to list in its PR: changed, and a withdrawal says whether the publisher ended the claim or its announcer lost it. - **Boundaries.** The chain of item 3. -- **Requests.** A request names the origin node its first hop chose, so +- **Requests.** A request names the routing node its first hop chose, so relays that rank differently never loop it, and fails closed. Lessons from the history that bind every child: @@ -299,6 +325,6 @@ Lessons from the history that bind every child: ## Related -- [Routes and announces](/quest/m2/cluster-routes/README.md) - the rule this replaces, its contract, and its measurements +- [Routes and announces](/quest/m2/cluster-routes/README.md) - the path rule this replaced, and the acceptance fixtures the cut still waits on - [Cache-aware routing](/quest/m3/cache-routing/README.md) - implements the warmth pick - [Route health](/quest/m3/route-health/README.md) - the metadata-only secondary diff --git a/quest/m2/multicast/accept.md b/quest/m2/multicast/accept.md index 52338211531..8a6138ca2b2 100644 --- a/quest/m2/multicast/accept.md +++ b/quest/m2/multicast/accept.md @@ -31,8 +31,9 @@ the implementation before lite-07 freezes, as the maintainer decided for the path rule on 2026-11-14. [Route layer acceptance](/quest/m2/cluster-routes/acceptance.md) owns the -fixtures that exist, which the [switch](/quest/m2/multicast/switch.md) -rewrites for the new rule; this adds the cases above beside them in +fixtures that exist, which +[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) rewrote for +the new rule; this adds the cases above beside them in `rs/moq-net/tests/integration/`. Impair links with bufferbloat, not random loss. Where a case needs `js/net`, run it in `just test interop --all`. @@ -40,7 +41,3 @@ A case that fails is a flaw to fix on lite-07-wip, in the quest that owns the code, before the cut. Public API: none. Wire: none expected. - -## Required - -- [The switch](/quest/m2/multicast/switch.md) - the sessions these cases run on diff --git a/quest/m2/multicast/switch.md b/quest/m2/multicast/switch.md deleted file mode 100644 index 6693d4c38c5..00000000000 --- a/quest/m2/multicast/switch.md +++ /dev/null @@ -1,49 +0,0 @@ -# [M] What the route switch left - -## Goal - -No quest still describes the old route rule or the Route Key. - -[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) put every -lite-07 session on `route::relay::Router`, rewrote the draft, made `js/net` -an endpoint, and deleted the old rule with its route key. -[moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053) made the -test-only route API crate-private, closed the adopted-claim gap with a -marked identity, marked the announce-churn mitigations, and removed -`poll::Session::export_keying_material`. What is left here is the quests -written against the old wire. - -## Plan - -Decided 2026-11-17 (maintainer, on -[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987#issuecomment-6958210502)): -✅ this quest is kept. - -Decided 2026-11-18 (maintainer, on -[moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053#issuecomment-6975349236)): -✅ kept as rewritten here, retargeting the old-wire quests (rejected: dropping it). - -Retarget each to the route rule that ships (`rs/moq-net/src/route/relay.rs` -and the lite-07 draft's Routing section), or delete it when the new rule -makes it moot: -[outbound federation](/quest/m3/cluster-routing/federation.md), -[the routing guide](/quest/m3/routing-guide.md), -[route wakes](/quest/m4/route-wakes.md), -[handoff routes](/quest/m4/relay-handoff-routes.md), -[regrant recheck](/quest/m4/regrant-route-recheck.md), -[announce interest memory](/quest/m4/announce-interest-memory.md), -[the mesh condition](/quest/m5/lite07-mesh.md), -[route memory](/quest/m9/perf/cluster-route-memory.md), -[route fan-out](/quest/m2/cluster-routes/route-fanout.md), and -[the Playwright bump](/quest/m1/playwright-pin.md), whose only reason was the -route key lane. - -Each retargeted quest also drops its link to this one. - -Public API: none. Wire: none. - -## Related - -- [Cache-aware multicast](/quest/m2/multicast/README.md) - the epic -- [Route benches](/quest/m3/route-benches.md) - the path-rule benches and costs the switch left -- [Proof binding](/quest/m3/route-proof-binding.md) - binds the route proof to the router diff --git a/quest/m3/cache-routing/select.md b/quest/m3/cache-routing/select.md index 8a4dffd5e86..ff6528835b5 100644 --- a/quest/m3/cache-routing/select.md +++ b/quest/m3/cache-routing/select.md @@ -46,8 +46,9 @@ Decided 2026-11-17 (maintainer, same comment): ✅ **warmth ships in the next wip version, after the lite-07 cut** (rejected: holding lite-07 for the port; freezing the rank and advert fields in lite-07 before the port has exercised them). The 2026-11-14 decision stands: lite-07 freezes without a -warm wire. The cut waits on [the switch](/quest/m2/multicast/switch.md), and -this port is the first item after it. +warm wire. The route layer it ranks on shipped in +[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987), and this port +is the first item after the cut. **Why ranked readers settle.** A relay's choice reads only copies feeding lower ranks, and what those feed never depends on a higher rank's choice. @@ -232,7 +233,7 @@ outright. - **The path and the mark.** Every offer names the relays the data passes from the publisher's relay to the sender, and carries the mark: set once the copy came down a link its receiver marked upstream anywhere on that - path, and kept on every hop after, as a route's down-only flag is. + path, and kept on every hop after, as a route's mark is. - **Adverts.** While R has demand for an Instance besides N, N has not marked R upstream, and R has not both marked N upstream and a copy whose mark is set, it tells N its offer and the lowest rank it feeds besides N on @@ -412,7 +413,6 @@ lite-07 freezes changes. ## Required -- [The switch](/quest/m2/multicast/switch.md) - the route layer it ranks on - [Lite-07 finalized](/quest/m2/lite07-finalize.md) - warmth's wire lands in the first wip version after the cut (maintainer, 2026-11-17) ## Related diff --git a/quest/m3/cluster-routing/README.md b/quest/m3/cluster-routing/README.md index 999b8ef7160..ea7b26e3840 100644 --- a/quest/m3/cluster-routing/README.md +++ b/quest/m3/cluster-routing/README.md @@ -4,7 +4,7 @@ One routing model serves CDN tiers, outbound-connected local clusters, and application-selected peer connections. Topology changes update reachability -per affected origin rather than replaying every advertisement behind it. +per affected routing node rather than replaying every advertisement behind it. Known advertisements remain distinct from current reachability, and stale copies cannot revive a claim after its withdrawal is known. @@ -21,90 +21,47 @@ by its own milestone. ✅ m3 (rejected: m2, where the cut would wait on it; pulling the m2 blocker into m1). It ranks right after m2, when it can start. Reopened 2026-11-15 (maintainer, /quest-plan on the whole route layer): -[cache-aware multicast](/quest/m2/multicast/README.md) replaces the route -layer this Plan describes, and where the two differ it wins. A routing node +[cache-aware multicast](/quest/m2/multicast/README.md) replaced the route +layer this Plan described, and where the two differ it wins. A routing node is a node that forwards, not every publisher; a claim crosses a scoped grant by a chain, not a qualified identity under a route key; and holds, doubt, and the non-goal of cache-dependent route prices are gone. One model for CDN tiers, outbound local clusters, and meshes stands. -Decided with the maintainer, 2026-10-13: +What stands of the 2026-10-13 decisions (maintainer), restated for the route +layer [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) shipped. +The rest of the earlier Plan (Babel's feasibility, every publisher an origin +node with a declared identity, qualified identities under a route key, holds, +and doubt) went with the old router; it is in +`git show 4e73451999:quest/m3/cluster-routing/README.md`. -- Keep per-origin ROUTEs and path-less ANNOUNCEs on the ordered announce - stream in lite-07-wip. Reachability and origin-owned claims have separate - lifetimes. A wildcard claim is a capability, not a list of live broadcasts; - withdrawing one origin's claim does not end other origins' copies of an epoch. +- Routes and claims have separate lifetimes on the ordered Announce Stream. + A wildcard claim is a capability, not a list of live broadcasts; + withdrawing one claim does not end other claims of its Instance. - Applications choose connections and assign each a local priority, whether - it reaches a CDN or a peer. Among Babel-feasible routes, priority precedes - additive cost; feasibility can override preference, and the resulting - arrival-order dependence is accepted. Link costs are strictly positive - (default one per link): zero is refused, overflow is unreachable, and there - is no hop-count component. Preserve deterministic path hashing among equal - routes and newest-first only for otherwise identical routes. A new - connection alone does not outrank a different equal-cost route. Cost - includes the origin's prefix seed, which may be zero. -- The first hop selects the origin; forwarded requests name that identity + it reaches a CDN or a peer. A route ranks by drain, then the lowest + priority on its path, then additive cost. Link costs are strictly positive + (default one per link): zero is refused, and there is no hop-count + component. Equal lite-07 claims spread by the draft's Rendezvous Hash + (a claim from an older version compares its hop chain's length first), + and among otherwise identical routes a node stays on its next hop. +- The first hop selects the claim's routing node; forwarded requests name it and aggregation preserves it. -- GOAWAY gives the peer lowest preference and fails over to an eligible - alternative, keeping it as a last resort while connected. Drain is a - preference state, never a saturated metric. -- Keep Babel. A detour longer than the lost route may wait for a sequence - refresh from the origin; instant failover is not required. Measure failure - detection, additional routing recovery, and total interruption separately. - Reopened 2026-11-11 (maintainer, /quest-plan after - [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews): - the distance refuses a longer way round until the publisher advances its - seqno, and refuses a cached copy at a neighbour no closer to the - publisher, which [cache-aware routing](/quest/m3/cache-routing/README.md) - wants to join. Decided the same day: a route is shared per publisher and - per scope, and a relay believes a route by that route's own path. Rejected: - link-state among relays, and routes to relays in place of publishers. - Decided 2026-11-11 (maintainer, on the measurements of - [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533)): Babel goes. A route is - kept free of loops by its path, a withdrawal that names the hop that died, - holds, and a timed doubt - ([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27). A - detour no longer waits on the publisher, and - [the switch](https://github.com/moq-dev/moq/pull/8731) deleted Babel's router. "Feasible" in this Plan - reads as: the route's path does not name the relay and no doubt keeps the - relay off it. -- An origin declares its own routing identity, fresh per process or tab - restart and shared across its live sessions. It is unverified. Decided 2026-11-02 (maintainer, /quest-unblock): no - key proof is planned; if a use case needs a provable identity, moq-lite-07's - route identity becomes a public key, independent of P2P. Decided 2026-11-04 - (maintainer, /quest-unblock): scoped forgery is that use case. A peer proves - a route key in its lite-07 SETUP, bound to the session, and every identity it - sends while scoped, its own included, is qualified under it. - Receiver-stamped - per-session identities were dropped 2026-10-26 (maintainer): in a mesh, - two relays stamping their sessions from one origin forward its routes as - two identities a third relay cannot recognize as one. Keep administrative - identity separate; `cluster.id` is removed. Switching between paths - resumes only across a matching explicit epoch. -- Every identity a connection sends, its own and those it forwards, is - qualified when its publish grant is scoped, and stays bare when it is - unscoped, so a forger only damages routes inside its own scope. (Decided - 2026-10-14; own identities included 2026-10-26, since they are no longer - proven.) The issuer-qualified subject key was dropped 2026-11-02 - (maintainer); since 2026-11-04 a scoped peer's identities are keyed by the - route key it proves ([moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023)). -- Matching explicit epochs authorize resume. Epochless changes retain - Restart and fresh resolution; neither node identity nor a healthy - alternate grants permission to splice content. -- During uncertain handoff, preserve known advertisement presence while - representing unavailable forwarding separately. Never manufacture an END - from route loss, imply successful delivery while unreachable, or retain - unbounded stale state. -- One route selection per identity. Each node advertises its best route, - with or without demand. Keep the `upstream` link mark and its down-only bit - so CDN relays never shortcut between each other through a local network; - CDN content still flows into the network and local content out, including - through a peer whose own uplink died. A cheaper exportable alternative - hidden behind a preferred down-only route is accepted; rank local peers - above a backup uplink to avoid it. The relay that accepted a request from - an upstream link ends that forwarded subscription when its route turns - down-only; no request bit on the wire. (Decided 2026-10-14, superseding - the two-view export plan.) +- GOAWAY makes every route through the node draining, ranked below every + route that avoids it and kept as a last resort while connected. Drain is a + rank, never a saturated metric. Instant failover is not required: measure + failure detection, routing recovery, and total interruption separately. +- A subscription moves only between claims of one Instance; another Instance + is a restart, and neither a node's identity nor a healthy alternate grants + permission to splice content. +- While no route reaches a claim it stays shown, unavailable, until the + reader's debounce lets it go; route loss never manufactures an END, and + no stale state is kept unbounded. +- The `upstream` link mark, now a route's mark, keeps CDN relays from + shortcutting through a local network while CDN content still flows in and + local content out, including through a peer whose own uplink died. A + strictly cheaper marked route still wins; rank local peers above a backup + uplink by priority to keep one copy per broadcast. - A local per-connection `proxy` preference decides whether a connection is sent routes learned from other connections. Relays default on; applications choose per peer connection (on for redundancy, off for @@ -115,19 +72,8 @@ Decided with the maintainer, 2026-10-13: peers outrank a backup uplink, and origin aggregation joins the rest. - A peer-to-peer consumer, outside this repository, owns proving that its CDN session and discovery stay connected and unused CDN media demand is released - after a handoff. m1 provides and tests the shared routing primitives without - requiring an actual peer-to-peer integration. - -The handoff proof showed a complete neighbor snapshot does not prove its -claims are current; receivers merge origin-versioned views by floor instead. -The routing contract is recorded in [Routes and announces](/quest/m2/cluster-routes/README.md) -with deterministic counterexamples: path-only subscriptions that reselect -origins at every hop loop and deadlock, so forwarded requests name their -identity; feasibility makes transit priorities depend on arrival order; -export keeps one selection with a down-only bit; and scoped forwarding is -contained by qualified identities. - -The new routing model and finalized lite-07 belong to the breaking release. + after a handoff. + Existing mesh deployments remain supported; the full edge/core topology migration is independent. Basic outbound federation proves the common routing contract without requiring that deployment migration. @@ -138,16 +84,9 @@ cases; [moq-dev/moq#5391](https://github.com/moq-dev/moq/pull/5391) bounded its queue under repeated same-clock churn. [moq-dev/moq#5382](https://github.com/moq-dev/moq/pull/5382) and [moq-dev/moq#5393](https://github.com/moq-dev/moq/pull/5393) separately -reduced idle-session polling. -The 300 ms update hold alone is no longer the baseline. Keep regression -coverage for announcement churn, idle work, active continuity, and retained -state separately; fewer control bytes does not prove cheaper runtime work. - -The earlier simulator estimates remain in [Routes and announces](/quest/m2/cluster-routes/README.md). -They do not prove the handoff correct; its fixtures do. Preserve public synthetic -fixtures and controlled-time regressions, including residual stale revival, -not only results from deployment tooling. Do not claim per-origin aggregation -reduces origin count when every application publishes one broadcast. +reduced idle-session polling. Keep regression coverage for announcement +churn, idle work, active continuity, and retained state separately; fewer +control bytes does not prove cheaper runtime work. Non-goals: measured RTT/loss pricing by default, TURN, browser transit, full link-state topology, or cross-operator economic exchange rates. Basic @@ -163,7 +102,7 @@ setup after the capabilities land. ## Required - [Cache-aware multicast](/quest/m2/multicast/README.md) - the route layer this builds on, replacing the path rule -- [Routes and announces](/quest/m2/cluster-routes/README.md) - shared routing contract and implementation +- [Routes and announces](/quest/m2/cluster-routes/README.md) - the route layer's acceptance on real sessions ## Related diff --git a/quest/m3/cluster-routing/federation.md b/quest/m3/cluster-routing/federation.md index 129e3221667..9cf1ff871c6 100644 --- a/quest/m3/cluster-routing/federation.md +++ b/quest/m3/cluster-routing/federation.md @@ -10,38 +10,38 @@ cluster with two uplinks, partition, and healing. No bespoke federation router. ## Plan Reopened 2026-11-15 (maintainer, /quest-plan on -[cache-aware multicast](/quest/m2/multicast/README.md)): the CDN relay -announces the local relay's claims under its own Node ID, with a chain of -the nodes that did so, and refuses a claim whose chain names it. No route, -qualified identity, or route key crosses the scoped grant, so the key file -below goes, and a relay behind the local relay may be scoped too. -[The switch](/quest/m2/multicast/switch.md) rewrites the paragraphs below -that say otherwise. +[cache-aware multicast](/quest/m2/multicast/README.md), item 3): claims, not +routes, cross a trust boundary. The CDN relay scopes the local relay's +connection to a path prefix, so the session carries claims alone (the +lite-07 draft's Routing: a peer confined to part of the tree exchanges no +routes). The CDN relay announces each claim it takes there under its own +Node ID with a chain of the relays that did so, and refuses a claim whose +chain names it. No route, qualified identity, or route key crosses the grant, +so a forger declaring a victim's identity captures nothing: the claim +belongs to the relay that announced it again. Nested scoped federation is in: +a relay behind the local relay may be scoped too, and the chain bounds the +nesting at 32 relays. The route key, qualified Node IDs, and their key file +went with [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987). +Since [moq-dev/moq#9082](https://github.com/moq-dev/moq/pull/9082) the chain +names the announcing relay, then each grant's exporter and adopter, and a +relay refuses a claim whose chain would name any relay twice once it adds +itself. Decided 2026-10-13: this is a required use case, independently shippable from peer-to-peer routing. Existing cluster sessions are bidirectional and scoped peer admission exists; extend and prove those paths rather than creating a reverse connection protocol. Do not require an inbound connection to the local relay. -Use authenticated, namespace-scoped admission: the CDN scopes the local -relay's connection to a path prefix, and identities it sends, its own and its -local publishers', are qualified, so a forger or a misconfigured uplink only -affects its own scope (decided 2026-10-14; see [Routes and -announces](/quest/m2/cluster-routes/README.md)). Decided 2026-11-04 -(maintainer): those identities are keyed by the route key the peer proves, -read from a key file the local relay's uplinks share -([moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023)). Without a key, a -scoped forger declaring a victim's id captures claims outside its scope -(`a_forged_identity_is_contained_by_a_scoped_admission`). The scope itself -(`hash(scope, id)`) was rejected 2026-10-15, and the grant's issuer-qualified -subject (`hash(subject, id)`) was dropped 2026-11-02 (maintainer). The key -must survive token refresh and be shared by the local relay's two uplinks. A -scoped connection cannot forward an already qualified identity, so nested -scoped federation is out: a relay behind the local relay connects unscoped. -The local relay marks its CDN links `upstream`, so CDN relays never shortcut -through the local network, and keeps its local `proxy` preference on toward -the CDN. One route selection per identity; a cheaper exportable alternative -hidden behind a preferred down-only route is accepted (decided 2026-10-14). +The local relay marks its CDN links `upstream`, the draft's marked sessions, +so a claim or route learned over one never goes out another and CDN relays +never shortcut through the local network; it keeps its local `proxy` +preference on toward the CDN. At equal rank an unmarked route beats a marked +one, but a strictly cheaper marked route still wins, so an operator who wants +local peers ahead of a backup uplink gives them a higher priority +(multicast item 13; decided 2026-10-14 as one selection with the cheaper +exportable alternative accepted). Two uplinks to two CDN relays are two +claims of one Instance, one per CDN relay, so a reader on the CDN moves +between them without a restart when one is lost. CDN content still flows into the local network and local content out: a local relay whose own uplink died reaches the CDN, in both directions, through a @@ -53,12 +53,15 @@ Basic federation requires neither economic cost translation nor cross-operator c Controlled-time integration tests: publish/consume in both directions through one outbound session; two local uplinks without CDN transit through the LAN; local partition reached through CDN, then restoration of the cheaper eligible -local route; overlapping namespace grants and rejected identity/scope spoofing; -restart and retained neighbor state; a CDN broadcast watched on both local -relays crossing one uplink once, failing over when that uplink dies; a local -relay with a dead uplink publishing and consuming through its peer. Test -equivalent epochs and epochless Restart separately. A client may retain a CDN session while consuming over a -better authorized local connection, with no unused duplicate media demand. +local route; overlapping namespace grants, a claim outside the grant refused, +and a claim that comes back to a relay its chain names refused; a scoped +relay behind the scoped local relay; restart and retained neighbor state; a +CDN broadcast watched on both local relays crossing one uplink once, failing +over when that uplink dies; a local relay with a dead uplink publishing and +consuming through its peer. Test a move between claims of one Instance and a +restart under a new Instance separately. A client may retain a CDN session +while consuming over a better authorized local connection, with no unused +duplicate media demand. Measure origins/broadcasts and peers/readers independently under reconnect and uplink churn. Reuse route-layer fixtures and ensure these run in CI. @@ -67,20 +70,18 @@ one-copy-per-broadcast priority recipe) and affected auth/config examples in the same PR; the combined concept/setup guide follows separately. Decided 2026-11-06 (maintainer; the agent picked the children): require -only the route children federation builds on, -session integration (landed in [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023) and [moq-dev/moq#8293](https://github.com/moq-dev/moq/pull/8293)) and +only the route children federation builds on, session integration (landed +in [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023) and +[moq-dev/moq#8293](https://github.com/moq-dev/moq/pull/8293)) and [acceptance](/quest/m2/cluster-routes/acceptance.md), not the whole [route epic](/quest/m2/cluster-routes/README.md). Federation is -relay-to-relay, so browser route-key proofs, the Playwright pin they wait on, -and the mixed-version follow-up (mount takeover) gates the -lite-07 cut, not this. Its consumer, Kixel's mesh relay node (downstream, in -kixel-ai/kixel.ai), is a Rust relay too. Acceptance still brings the js/net route layer for its -Rust/JS interop checks. Rejected: session integration alone, which drops the -proven contract the wire rule below relies on. +relay-to-relay: its consumer, Kixel's mesh relay node (downstream, in +kixel-ai/kixel.ai), is a Rust relay too. Rejected: session integration +alone, which drops the proven contract the wire rule below relies on. -Public API: report any peer configuration or admission change. Wire: use the -proven lite-07 routing contract; any additional change requires draft and -Rust/JS parity. Run `just check`, `just test interop --all`, and `just drafts check`. +Public API: report any peer configuration or admission change. Wire: the +lite-07 draft's Claims and Routing as they ship; any change requires draft +and Rust/JS parity. Run `just check`, `just test interop --all`, and `just drafts check`. ## Required diff --git a/quest/m3/release/cut.md b/quest/m3/release/cut.md index db69dd81422..53d74e17bf4 100644 --- a/quest/m3/release/cut.md +++ b/quest/m3/release/cut.md @@ -250,5 +250,4 @@ use the manual `just test wire-compat --remeasure` procedure while the automated - [Interop and Video pass on main](/quest/m0/interop-green/README.md) - the cut PR's release gate runs `interop.yml`, which failed on `main` every day through 2026-11-12 - [Nightly passes on main](/quest/m0/nightly/README.md) - the release gate runs `features.yml`, red in the 2026-11-12 Nightly - [Claim new npm names](/quest/m3/release/npm-first-publish.md) - trusted publishing can publish every new `@moq/*` package -- [The Playwright bump](/quest/m1/playwright-pin.md) - the cut PR's interop lane runs with `CI_STRICT`, which fails the route key lane on a Chromium below 158 - [Pre-cut correctness fixes](/quest/m1/precut-fixes/README.md) - both pre-cut audits' required fixes have landed diff --git a/quest/m3/route-benches.md b/quest/m3/route-benches.md index f690d54ed93..09341f86380 100644 --- a/quest/m3/route-benches.md +++ b/quest/m3/route-benches.md @@ -10,6 +10,8 @@ with a number that says why. Found by [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) and proposed by [moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053). +The origin rewrite below is what survived of the route fan-out quest, whose +other findings were the old path router's indexes and doubts. Decided 2026-11-18 (maintainer, on [moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053#issuecomment-6975349236)): @@ -27,7 +29,14 @@ multicast epic the cut waits on. - an edit swapping one prefix for another tells again the claims both cover; - a relay whose only route closes ends its subscriptions while the claim - stays shown. + stays shown; + - a change to a route rewrites every claim behind it into the origin: + `route::node` `State::flush` relanes each broadcast the route's node + claims, and `Served` carries the cost, so a re-price is one origin + write per broadcast, each waking the fronts below it. Recommended: one + shared record per route in the origin that its broadcasts refer to, + built with [route wakes](/quest/m4/route-wakes.md), which owns the + fronts' side. - [moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053) sends a whole copy of a node's claims wherever its filtered copy used to go (`withheld` is gone), so a claim moving between a node's two identities @@ -40,4 +49,3 @@ which lands before the lite-07 cut. ## Related - [Cache-aware multicast](/quest/m2/multicast/README.md) - the route layer these measure -- [What the route switch left](/quest/m2/multicast/switch.md) - where these came from diff --git a/quest/m3/route-health/failover.md b/quest/m3/route-health/failover.md index 63f568f013f..c48d01fb66e 100644 --- a/quest/m3/route-health/failover.md +++ b/quest/m3/route-health/failover.md @@ -48,4 +48,3 @@ none beyond it. ## Required - [Metadata-only subscriptions](/quest/m3/route-health/meta.md) - what the standby observes -- [The switch](/quest/m2/multicast/switch.md) - the routes the standby chooses among diff --git a/quest/m3/route-health/meta.md b/quest/m3/route-health/meta.md index de7ba07f515..3862a3220d4 100644 --- a/quest/m3/route-health/meta.md +++ b/quest/m3/route-health/meta.md @@ -91,7 +91,3 @@ to `just test interop --all`. Public API: a metadata subscription in Rust, JS, and the bindings. Wire: in the current lite wip, the mode on SUBSCRIBE (or a message of its own) and payload-less FRAMEs. - -## Required - -- [The switch](/quest/m2/multicast/switch.md) - the route layer this rides on, and nothing uses it sooner diff --git a/quest/m3/route-proof-binding.md b/quest/m3/route-proof-binding.md index b9f143cfe93..c36dd453a0b 100644 --- a/quest/m3/route-proof-binding.md +++ b/quest/m3/route-proof-binding.md @@ -54,4 +54,3 @@ Public API: none. Wire: none. ## Related - [Cache-aware multicast](/quest/m2/multicast/README.md) - the route layer it proves -- [What the route switch left](/quest/m2/multicast/switch.md) - where these came from diff --git a/quest/m3/routing-guide.md b/quest/m3/routing-guide.md index 93749bd9dad..f95f8d590b6 100644 --- a/quest/m3/routing-guide.md +++ b/quest/m3/routing-guide.md @@ -17,13 +17,18 @@ above. Every feature PR still fixes docs it makes stale. Walk an outbound-only local relay through scoped admission, two-way delivery, two uplinks, and partition/healing. Explain application-owned connection -selection and priority, cost comparison within priority, and deterministic -equal-route selection. - -Distinguish routing identity, broadcast epoch, advertisement presence, -forwarding, and reachability. Resume requires matching explicit epochs; -epochless changes restart. Keep economic domain aggregation and measured -link-cost policy outside the basic setup. +selection and priority, and the rank lite-07 routes and claims take as +shipped ([moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987), the +draft's Routing and Resolution): drain, then priority, then cost, an +unmarked route ahead of a marked one at equal rank, and the Rendezvous Hash +that spreads equal claims. + +Distinguish a relay's Node ID, a broadcast's Instance and Epoch, a claim +shown unavailable or lost, and a route. A subscription moves between claims +of one Instance; another Instance is a restart. Explain `upstream` (marked) +links, the chain a claim carries across a scoped grant, and that a +non-forwarding endpoint's claims belong to its relay. Keep economic domain +aggregation and measured link-cost policy outside the basic setup. Use generic public examples and validate CLI/config examples against the implemented help. Cross-link `doc/bin/relay/cluster.md` and identity guidance diff --git a/quest/m4/announce-interest-memory.md b/quest/m4/announce-interest-memory.md index f5b12e5e8f3..f0ab6a4c0a3 100644 --- a/quest/m4/announce-interest-memory.md +++ b/quest/m4/announce-interest-memory.md @@ -28,8 +28,10 @@ is about 10k times N times a few hundred bytes. Incoming announces are bounded by `session::Limits`; outgoing interest streams are not. Since [one route stream](https://github.com/moq-dev/moq/pull/8835), -lite-07 holds one interest stream per side, so what remains exposed is -lite-05/06 and moq-transport. +lite-07 holds one interest stream per side, and since +[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) its sessions +read the route layer (`route::node`) rather than an origin cursor each, so +what remains exposed is lite-05/06 and moq-transport. Decided 2026-11-15 (maintainer, /quest-complete): ✅ a `session::Limits` cap on interest streams for lite-05/06 and moq-transport now, counted like @@ -46,4 +48,4 @@ Public API: additive, a new `session::Limits` field. Wire: none. ## Related -- [Cluster routing](/quest/m3/cluster-routing/README.md) - reshapes the route table this walks +- [Route wakes](/quest/m4/route-wakes.md) - the origin route table these cursors walk diff --git a/quest/m4/regrant-route-recheck.md b/quest/m4/regrant-route-recheck.md index 91aab8cfd61..910479fedde 100644 --- a/quest/m4/regrant-route-recheck.md +++ b/quest/m4/regrant-route-recheck.md @@ -17,6 +17,8 @@ Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-a Re-scoped 2026-11-12 (/quest-audit): [Each side says everything about routing on one stream](https://github.com/moq-dev/moq/pull/8835) gives a lite-07 side one Announce Stream and turns a token-scope change into an edit of its interest set, so on lite-07 the factor K becomes one and the per-stream `regrant` this plan targets remains only for lite-06 and older. Re-measure the regrant (or interest-set edit) cost on lite-07 with the benches below, and apply the steps that follow to whichever path still pays routes × members. If only the older versions still pay it, say so in the PR and leave them. +As shipped by [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987), lite-07 regrants in `Routed::poll_permit` (`rs/moq-net/src/lite/publisher/routed.rs`): a routing peer's grant must stay whole or the session closes, and on a claims-only session every claim told or live is re-synced against the new permit, so lite-07 still pays its claims times grant members once per session. The `AnnounceRun::regrant` path above now serves lite-06 and older only. + - Derive the change once per `AUTH_OK`, not per stream: the members added and removed between the old and new grant. Each stream then re-checks only the routes under a changed member's prefix, using the origin's prefix structure to find them instead of walking every route. A member's prefix here is its literal segments before its first wildcard (`*` or `**`), since grant members are `Patterns` (`Permit` in `rs/moq-net/src/auth.rs`); a member that starts with a wildcard has no literal prefix and re-checks every route under the stream's prefix, as today. Bench that shape on its own row, and test wildcard members, removed members, and old and new grants that overlap. - A grown grant needs only routes under the added members, not a whole-origin snapshot per stream. - Keep today's semantics: a revoked route retracts, a newly covered route advertises, and a route both grants cover is untouched. @@ -27,4 +29,4 @@ Public API: none. Wire: none. ## Related - [Bound peer grant processing by design](/quest/m5/peer-grant-bound.md) - may change the received-grant type this compares against; whichever lands second adapts -- [Announce interest streams share one route table](/quest/m4/announce-interest-memory.md) - the per-stream route state that this walk reads +- [Cap the legacy announce interest streams](/quest/m4/announce-interest-memory.md) - the per-stream route state that this walk reads on lite-06 and older diff --git a/quest/m4/relay-handoff-routes.md b/quest/m4/relay-handoff-routes.md index e6f42b07c98..c3d4811095a 100644 --- a/quest/m4/relay-handoff-routes.md +++ b/quest/m4/relay-handoff-routes.md @@ -18,9 +18,13 @@ relay's drain ends. between them already exists; the new relay runs a moq-lite session to the old one over a socket it carries, and each treats the other as a cluster peer until the old one exits, so routes cross between generations as they - cross between nodes. Check first how the route rule treats two processes - that share the node's identity, since a shared hop would read each other's - routes as loops. + cross between nodes. The two share only the node's stats name + (`rs/moq-relay/src/handoff.rs`): each process routes under its own random + Relay ID (`origin::Config::default`, which `Cluster::new` uses), so + lite-07's rule that a node never uses a route whose Path names its own + relay treats them as two routing nodes, and neither refuses the other's + routes. A test pins that, so a later change that shares the Relay ID + across a handoff fails it. - Rejected: keep a peer's old session past idle until the old relay closes it. That holds the old relay to its drain deadline again, which is what handoff removes. diff --git a/quest/m4/route-wakes.md b/quest/m4/route-wakes.md index 907b8a789bc..4b7e4f8b57d 100644 --- a/quest/m4/route-wakes.md +++ b/quest/m4/route-wakes.md @@ -26,9 +26,20 @@ Facts from `main` (`rs/moq-net/src/model/origin.rs`, since Wildcard landed in `select` under the table read lock (`retain_routes`, then `best_route`, which hashes every pool member), then the driver polls every track and rescans deadlines. `Pin` is gone (#4741). -- `route_order` is rendezvous-style: FNV over the path and hop ids, lowest - wins. A join moves only paths the newcomer wins, a leave only paths the - leaver was winning. +- `route_order` ranks, in order: the most specific claim; the claim the + route layer marks primary; a route neither draining nor unavailable; the + higher link priority; an identified hop chain before an anonymous one, + whatever the cost; the cheaper cost; a broadcast published on this origin; + the shorter hop chain; then, on lite-07, the draft's Rendezvous Hash of + the path and the claim's Instance (`rendezvous`) and the claimant's + relay, or on older versions a hash of the Instance and then of the chain, + lowest first; and last the newest entry. A broadcast published here for + exactly the path skips both hashes. A join moves only paths the newcomer + wins, a leave only paths the leaver was winning. +- On lite-07 the route layer writes into this table through `route::node`, + one write per broadcast whose way changed; a change to one route rewrites + every broadcast behind it, which [route benches](/quest/m3/route-benches.md) + owns. This quest is the fronts' half: what each write wakes. Decisions: @@ -53,24 +64,25 @@ Decisions: paths. - A serving front follows the best route among those it may resume across (decided 2026-10-04, replacing Stay; narrowed 2026-10-07 after #4942 and - Restart): routes with its epoch. A - front resolved without one, or replaced by a newer epoch, stays on its route - for its subscribers until that route goes and never splices onto another - (maintainer, 2026-10-08); new requests take a fresh front on the winner - instead of waking it. An idle front (every track + Restart; restated for lite-07's Instance): claims of its Instance, at the + newest Epoch among them. A front whose Instance another Broadcast replaced + at the path stays on its claims for its subscribers until they go and never + splices onto another Instance (maintainer, 2026-10-08); new requests take a + fresh front on the winner instead of waking it. An idle front (every track forgotten, no consumer holding its broadcast, not waiting for coverage) - ends instead of moving. A join or re-price must rehash - every epoch front below the prefix, since rendezvous moves exactly the paths - the changed route now wins; only those fronts re-select. A leave wakes only + ends instead of moving. A join or re-price must rehash every front below + the prefix that may resume, since rendezvous moves exactly the paths the + changed route now wins; only those fronts re-select. A leave wakes only the fronts the leaver served or was requesting through. - Builds on fronts keyed by effective exclusion, so the index hangs off the final front identity. (2026-09-30) -Decided 2026-10-13: coordinate the index with the per-origin route model and -per-track active/standby selection. Routes and announces records identity -and the filtered candidate views. Do not build a second route table per track or -turn one origin metric update into per-broadcast mutations. Whichever -implementation lands first preserves these operation-count tests for the next. +Decided 2026-10-13: coordinate the index with the route model and per-track +active/standby selection. Do not build a second route table per track or +turn one route's cost change into per-broadcast mutations. Whichever +implementation lands first, this or the origin record in +[route benches](/quest/m3/route-benches.md), preserves these operation-count +tests for the next. Verification: `origin/pool_churn` leaves flat in served paths at every pool width it already sweeps, and a unit test that counts `select` calls per @@ -82,4 +94,8 @@ Public API: none. Wire: none. ## Required -- [Routes and announces](/quest/m2/cluster-routes/README.md) - settles the source-selection model the index serves, so it is built once on the final selection +- [Cache-aware routing](/quest/m3/cache-routing/README.md) - adds the warm pick to how a relay chooses its source, so the index is built once on the final selection + +## Related + +- [Route benches](/quest/m3/route-benches.md) - the route layer's side: one origin record per route instead of a write per broadcast diff --git a/quest/m5/lite07-mesh.md b/quest/m5/lite07-mesh.md index 5a680be302d..2c067bc31a3 100644 --- a/quest/m5/lite07-mesh.md +++ b/quest/m5/lite07-mesh.md @@ -13,8 +13,13 @@ this quest and every `Required` entry that links it. moq.pro's lite-07 rollout waits on [Finalize moq-lite-07](/quest/m2/lite07-finalize.md), so this condition cannot clear before a release carries the final version. -Decided 2026-10-05: [Cluster routing](/quest/m3/cluster-routing/README.md)'s -route layer lands in lite-07, so the final lite-07 carries ROUTE and -path-less ANNOUNCE instead of native hop lists (a temporary lite-06 -compatibility chain rides until lite-08), and loses the `Hop Base`/`Hop Keep` -announce compression. moq.pro's rollout must plan for that wire. +Decided 2026-10-05, and redesigned 2026-11-15 by +[cache-aware multicast](/quest/m2/multicast/README.md): the final lite-07 +carries the route layer of [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987). +Relays route to each other by ROUTE, ROUTE_END, DEAD, and DRAIN under a +Node ID with a Start, and every claim belongs to the relay that announces +it, with no hop list on the wire; a claim from a lite-06 peer, or across a +scoped grant, carries a chain of the relays that announced it again. Announce +compression is `Path Base`/`Path Keep`. moq.pro's rollout must plan for that +wire, and for its newest and previous releases interoperating as relays +while a lite-06 relay is still in the mesh. diff --git a/quest/m9/kio-channel-contention/README.md b/quest/m9/kio-channel-contention/README.md index 542496db149..d9b53cfa57d 100644 --- a/quest/m9/kio-channel-contention/README.md +++ b/quest/m9/kio-channel-contention/README.md @@ -454,7 +454,7 @@ Decided (maintainer, 2026-11-01, cluster spike): route it learned to every other peer, and hop chains prevent loops. How routes from outside the relay (other relays' peer links) reach every worker without re-announce is for the planning to settle. - [Routes and announces](/quest/m2/cluster-routes/README.md) plans a + [Cluster routing](/quest/m3/cluster-routing/README.md) keeps a per-connection `proxy` preference (decided 2026-10-14, relays default on) that decides whether a connection is sent routes learned from other connections; a worker link with it off is this condition. diff --git a/quest/m9/kio-channel-contention/relay-per-thread.md b/quest/m9/kio-channel-contention/relay-per-thread.md index 83d0d164d10..e8d65680221 100644 --- a/quest/m9/kio-channel-contention/relay-per-thread.md +++ b/quest/m9/kio-channel-contention/relay-per-thread.md @@ -28,7 +28,7 @@ routes. The epic README holds both spikes' numbers. announce delay at 1 to 16 workers (the cluster spike's `spike_announce` table), plus the data-plane shapes (chat, fanout, video-heavy). - Settle how other relays' routes reach every worker without re-announce, - for example through [Routes and announces](/quest/m2/cluster-routes/README.md)' + for example through [cluster routing](/quest/m3/cluster-routing/README.md)'s per-connection `proxy` preference turned off on worker links. - Compare against the replica spike ([moq-dev/moq#7533](https://github.com/moq-dev/moq/pull/7533)): replicate diff --git a/quest/m9/perf/cluster-route-memory.md b/quest/m9/perf/cluster-route-memory.md index 1b6933033dd..6dc9af90cd0 100644 --- a/quest/m9/perf/cluster-route-memory.md +++ b/quest/m9/perf/cluster-route-memory.md @@ -15,7 +15,7 @@ Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-a Moved to lab 2026-11-12 (maintainer, /quest-audit): a measurement with no public API or wire change. -- **Route memory.** #7534 meshed W origins over moq-lite-06 `memory` sessions, as dialed cluster peers do. Those numbers predate the lite-07 route layer (`rs/moq-net/src/route/`), so measure again on lite-07 before breaking anything down. Announce heap per broadcast went from 4.9 KB with one origin to 45 KB at 4 workers and 886 KB at 16 (240 routes, about 3.7 KB each), and allocations from 64 to 14,600; 10,000 broadcasts at 16 would hold about 9 GB. Real full-mesh relay clusters pay the same per route. Measure it on a relay cluster with the counting allocator (#7534's `rs/moq-relay/examples/spike_announce.rs` is in its prototype commit [`ddd8b04`](https://github.com/moq-dev/moq/commit/ddd8b042ab82c7a4d53957e31fe35942957b7dfb), not on `main`), break the 3.7 KB down by structure (origin route entry, per-session announce state, hop lists, cursors), and shrink what's duplicated per route. Check against [cluster routing](/quest/m3/cluster-routing/README.md)'s per-origin routes first, since they change what a route holds. +- **Route memory.** #7534 meshed W origins over moq-lite-06 `memory` sessions, as dialed cluster peers do. Those numbers predate the lite-07 route layer (`rs/moq-net/src/route/`), so measure again on lite-07 before breaking anything down. Announce heap per broadcast went from 4.9 KB with one origin to 45 KB at 4 workers and 886 KB at 16 (240 routes, about 3.7 KB each), and allocations from 64 to 14,600; 10,000 broadcasts at 16 would hold about 9 GB. Real full-mesh relay clusters pay the same per route. Measure it on a relay cluster with the counting allocator (#7534's `rs/moq-relay/examples/spike_announce.rs` is in its prototype commit [`ddd8b04`](https://github.com/moq-dev/moq/commit/ddd8b042ab82c7a4d53957e31fe35942957b7dfb), not on `main`), break the 3.7 KB down by structure (origin route entry, per-session announce state, hop lists on lite-06, cursors), and shrink what's duplicated per route. On lite-07 a route leads to a relay rather than a publisher ([moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987)), so the per-broadcast state to break down is each claim's copy per neighbour in `route::relay::Router`, its slot per claimant and session in `route::node`, and its origin route entry, with no hop list. - **Hop cost.** About 140 thousand user instructions per one-frame group across one in-memory hop; fanout's 64 subscribers spread over 15 hops raised user instructions 19%, while chat amortized it. Take a `cpu` profile of one hop and attribute it before fixing; a per-reader part belongs to [mesh session cost](/quest/m9/perf/mesh-session-cost.md), which owns the per-reader group cost. - **`bench-profile lock`**: moved to [its own quest](/quest/m9/perf/profile-lock-switches.md) on 2026-11-03 (maintainer, /quest-plan of the follow-ups from [moq-dev/moq#7742](https://github.com/moq-dev/moq/pull/7742)), which keeps refusing nonalternating switch records and finds their cause instead of tolerating them. @@ -24,4 +24,4 @@ Public API: none expected. Wire: none. ## Related - [kio channel contention](/quest/m9/kio-channel-contention/README.md) - the epic both spikes reported to -- [Announce interest streams share one route table](/quest/m4/announce-interest-memory.md) - the same per-stream announce state, from a subscriber's side +- [Cap the legacy announce interest streams](/quest/m4/announce-interest-memory.md) - the same per-stream announce state, from a subscriber's side