From 2b4e50596059d455d7cc0690790a51f389788a66 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 20:04:25 -0800 Subject: [PATCH 1/5] quest: plan lite-07's one routing stream per side and its interest set Records the 2026-11-10 /quest-plan on lite-07 routing: every publisher stays an origin node, holds and merged floors stay, and each side says everything about routing on one unidirectional Announce Stream whose interest is a set of prefixes edited in place. Splits the work into the interest set (model and router) and the route stream (wire and sessions), narrows echo-order to its model half, and plans the seqno bump fix, two offline-not-END fixes, the browser mesh, and core and edge roles in the mesh overlay. Co-Authored-By: Claude --- quest/m1/offline-cursor-fallback.md | 40 ++++++ quest/m1/offline-js-handle.md | 39 +++++ quest/m1/route-seqno-bumps.md | 56 ++++++++ quest/m2/cluster-routing/routes/README.md | 10 ++ quest/m2/cluster-routing/routes/echo-order.md | 40 ++++-- .../m2/cluster-routing/routes/interest-set.md | 102 +++++++++++++ quest/m2/cluster-routing/routes/js.md | 10 ++ .../m2/cluster-routing/routes/route-stream.md | 136 ++++++++++++++++++ quest/m3/mesh/overlay.md | 46 ++++-- quest/m4/p2p/README.md | 6 + quest/m5/browser-mesh.md | 65 +++++++++ 11 files changed, 531 insertions(+), 19 deletions(-) create mode 100644 quest/m1/offline-cursor-fallback.md create mode 100644 quest/m1/offline-js-handle.md create mode 100644 quest/m1/route-seqno-bumps.md create mode 100644 quest/m2/cluster-routing/routes/interest-set.md create mode 100644 quest/m2/cluster-routing/routes/route-stream.md create mode 100644 quest/m5/browser-mesh.md diff --git a/quest/m1/offline-cursor-fallback.md b/quest/m1/offline-cursor-fallback.md new file mode 100644 index 00000000000..921589a9c15 --- /dev/null +++ b/quest/m1/offline-cursor-fallback.md @@ -0,0 +1,40 @@ +# [S] A scoped cursor reports a lost route as offline when a broader route takes over + +## Goal + +A reader on a scoped announcement cursor whose exact route is lost with its +session, while a broader route still covers the same presented path, sees the +lost route go offline, never an END. An END still means the publisher +withdrew. + +## Plan + +Planned 2026-11-10 (maintainer, /quest-plan on lite-07 routing). Milestone: +✅ m1 (rejected: m2 in the routes epic; m0). + +Found by the automated review of +[moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375), posted fifteen +minutes before it merged and never addressed. With a covering cursor rooted +at `room/alice`, an exact forwarded route presents with captures `Some([])` +and the broader `room` fallback with `None`. Losing the exact route's session +passes `Gone::Offline` into the update, but the capture-change branch +(`rs/moq-net/src/model/origin.rs`, about line 1753, the arm that replaces a +presented route whose captures differ) retracts the old identity with +`Gone::End`. Readers then see End, as if the publisher had withdrawn. + +Reproduce it first by extending +`capture_change_retracts_before_reannouncing_a_presented_prefix` with a +forwarded route lost with its session. Then carry the actual removal reason +through that replacement instead of a fixed `Gone::End`. + +Check the bindings that surface the reason (`moq-ffi`, and the Swift, Kotlin, +Python, and C wrappers of `announced`) need no change beyond the Rust fix. + +Caused by: [moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375), which +introduced the offline reason and left this branch on End. + +Public API: none. Wire: none. + +## Related + +- [A scoped JS handle that closes reports its routes offline](/quest/m1/offline-js-handle.md) - the same misreport in `js/net` diff --git a/quest/m1/offline-js-handle.md b/quest/m1/offline-js-handle.md new file mode 100644 index 00000000000..f663cdb86bd --- /dev/null +++ b/quest/m1/offline-js-handle.md @@ -0,0 +1,39 @@ +# [S] A scoped JS handle that closes reports its routes offline + +## Goal + +In `@moq/net`, routes a session received through a scoped `Producer` go +offline when that scoped handle closes while the parent origin stays open. +The parent's announcement reader emits `offline`, not `end` or `absent`, +unless the announcer ended the route on the wire. + +## Plan + +Planned 2026-11-10 (maintainer, /quest-plan on lite-07 routing). Milestone: +✅ m1 (rejected: m2 in the routes epic; m0). + +Found by the automated review of +[moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375), posted fifteen +minutes before it merged and never addressed. Closing the scoped handle runs +the raw callbacks `#hold` registered (`js/net/src/origin.ts`, about line +1695) and skips `close(lost)`, so received entries keep `lost = false` and +the reader reports a withdrawal. A later route close cannot repair it, +because `ended` is already true. + +Reproduce it first in `js/net/src/origin.test.ts`: receive a route through a +scoped handle, close the handle, and read the parent's announcements. Then +put the reason in the shared release transition, so a received route's +teardown defaults to offline and only an explicit wire end selects +withdrawal. + +Compare with Rust's scoped teardown and say in the PR whether it has the same +gap; if it does, fix it in the same PR, as `moq-net` and `js/net` stay in +step. + +Caused by: [moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375). + +Public API: none. Wire: none. + +## Related + +- [A scoped cursor reports a lost route as offline when a broader route takes over](/quest/m1/offline-cursor-fallback.md) - the same misreport in the Rust cursor diff --git a/quest/m1/route-seqno-bumps.md b/quest/m1/route-seqno-bumps.md new file mode 100644 index 00000000000..1987501db87 --- /dev/null +++ b/quest/m1/route-seqno-bumps.md @@ -0,0 +1,56 @@ +# [S] A node advances its route seqno only when a route link changes + +## Goal + +A lite-07 node raises its own route Seqno unprompted only when a session it +exchanges routes over opens or closes, and otherwise only to answer a +ROUTE_REQUEST. A client connecting to or leaving a relay no longer makes the +relay send a ROUTE_UPDATE for its own identity on every link that carries it. +A test counts the ROUTE messages a mesh sends per client session and holds +them at zero. + +## Plan + +Decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing), asked when +an origin should advance its seqno unprompted: ✅ only when a route link +changes (rejected: only to answer a ROUTE_REQUEST, which makes every path +change that lengthens a route wait a round trip to the origin; keeping +today's per-stream bump until a sweep measures it). Milestone: ✅ m1 +(rejected: m2 in the routes epic; m0), since relays run lite-07 in production +now and this changes no wire. + +Why it exists: relays that remember a short distance to a node refuse its +new, longer path until the node issues a higher seqno. Bumping when its links +change saves the ROUTE_REQUEST round trip. + +What is wrong today: `register` and `Drop for Stream` in +`rs/moq-net/src/route/node.rs` call `router.bump()` for every announce stream +of every lite-07 session, and `Routed::limit` in +`rs/moq-net/src/lite/subscriber/routed.rs` registers again on a permit +change. A relay originates the claims of its lite-06 and moq-transport +publishers under its own identity, so each client session bumps that identity +at least four times. This was read from the code, not measured: reproduce it +first with a count of ROUTE messages per client connect and disconnect on a +three-relay mesh. + +Open for the implementer, with a recommendation: what counts as a route +link. Recommended: a session whose peer forwards, known because it has sent a +route to a node other than itself, plus every session of a node that does not +forward (all of a publisher's links are route links). A session that only +ever carries claims, or only the peer's own route, never bumps. Confirm the +choice against `reconnect_without_restart_needs_a_seqno_advance` and +`restart_reusing_identity_is_refused` in `route/tests/`, which the bump +exists for, and say in the PR if a narrower rule (bump on close only, since a +new link lengthens no path) also keeps them passing. + +Mirror any change to `bump`'s callers in `RouteHandoff.tla` as the +`verify-route` skill says, and state the ROUTE budget per session event so +[the mesh sweep](/quest/m1/route-mesh-sweep.md) can assert it. + +Public API: none. Wire: none. + +## Related + +- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the contract's seqno rule this narrows +- [Seeded lite-07 meshes hold the route rules and an announce budget](/quest/m1/route-mesh-sweep.md) - the budget that would have caught this +- [Each side says everything about routing on one stream](/quest/m2/cluster-routing/routes/route-stream.md) - registers one neighbour per session, which changes where the bump is called diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index e77823753b9..648917c8eb2 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -31,6 +31,16 @@ claim whose withdrawal is already known. 16. ✅ Decided 2026-11-09 (maintainer, /quest-unblock on [moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375)), asked how a token-scoped lite-07 session certifies withdrawals so a clean finish ENDs across a relay: ✅ **the subscriber narrows its ANNOUNCE_REQUEST to its scope's single region** (where its limit and its subscribe grant meet), and reopens the request when that region changes so a widening renewal isn't lost. Floors keep their exact whole-prefix meaning, with no wire change and one SHOULD in the lite-07 draft. A multi-region scope still certifies nothing. Rejected: floors carrying regions on the wire, which reverses item 10; landing with the finish lanes accepting offline. 17. ✅ Decided 2026-11-09 (maintainer, /quest-unblock on [moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375)), asked how item 16's reopen counts the session's announce cap while the old request keeps serving until the new one is SYNCED: ✅ **the old request gets its own budget for the handover**, moving what it holds onto a separate counter with the same cap, so the session's count belongs wholly to the new request. A peer can make the session hold at most twice the cap, and only during one handover. Rejected: dropping the old request at once on every region change, which flickers claims offline; holding a client's first request until its credential is answered and dropping the old one on later renewals. +The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing), which reopened the route layer's rules after the echo ordering bug. + +18. ✅ Asked who is an origin node: **every publisher stays an origin node.** Its identity is for signing, for one node across several links (redundant publishers), and because browsers will forward ([browser mesh](/quest/m5/browser-mesh.md)). Rejected: a relay originating, under its own Node ID, the claims of any peer that sends no Node ID or holds a scoped publish grant, as it already does for lite-06 and moq-transport publishers. That would make route state follow relays rather than publishers and remove qualified Node IDs, the Route Key, and the echo, but a publisher would have no identity of its own. Accepted cost: routes, holds, and seqno requests scale with publishers, so a node must still forget dead identities, and the echo that makes forgetting safe has to be sound. +19. ✅ Asked whether to keep them: **ROUTE_HOLD stays, and the newest floor wins per identity.** Rejected: dropping holds, where each stall takes every claim behind the node offline and re-sends them all; and claims that follow only the selected next hop, which re-pulls a node's whole claim table on each next-hop change and showed revivals and false ENDs in the sweeps below. +20. ✅ Asked when an origin advances its seqno unprompted: **only when a route link changes**, which [the seqno bumps](/quest/m1/route-seqno-bumps.md) implements. Today every announce stream open or close bumps it. Rejected: only to answer a ROUTE_REQUEST; keeping the per-stream bump. +21. ✅ Asked how one side's routing messages travel so none overtakes another: **one unidirectional Announce Stream per side**, carrying its requests, its answers, and its echoes in order ([the route stream](/quest/m2/cluster-routing/routes/route-stream.md)). It replaces "There MAY be multiple Announce Streams". Rejected: one bidirectional Announce Stream per direction with the echo moved to the answering half; one bidirectional stream per session. +22. ✅ **A side's interest is a set of prefixes edited in place.** The live set never nests, and one edit may swap prefixes for a wider or narrower one without re-sending what both cover ([the interest set](/quest/m2/cluster-routing/routes/interest-set.md)). A token-scope change is such an edit, with one prefix per region of the scope. This supersedes items 16 and 17, and the handoff contract's "An interest change opens a new stream" below, once the route stream lands. +23. ✅ Asked what a floor covers with several prefixes on one stream: **one prefix, named by its id.** This amends item 10: the floor is still a marker and never a region list. Rejected: a floor over the whole set. +24. ✅ Asked how a side learns a newly added prefix's initial set is complete: **SYNCED per prefix, naming its id**, plus one marker per edit where it takes effect for both sides. Rejected: one SYNCED per edit, counted in order. + The route layer's core landed in [moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676): a diff --git a/quest/m2/cluster-routing/routes/echo-order.md b/quest/m2/cluster-routing/routes/echo-order.md index 91fb40a3531..75e909762f6 100644 --- a/quest/m2/cluster-routing/routes/echo-order.md +++ b/quest/m2/cluster-routing/routes/echo-order.md @@ -2,10 +2,13 @@ ## Goal -A lite-07 node drops its feasibility distance for an identity only once no -advertisement derived from its own can still reach it. Two relays never count -a route to infinity, and a relay never shows a broadcast available through a -route that loops back to itself. +In the route model and `route::tests::Sim`, a ROUTE_END echo travels on the +FIFO toward the END's sender, behind every route the echoer sent first. A +node there drops its feasibility distance for an identity only once no +advertisement derived from its own can still reach it, so two relays never +count a route to infinity. Production gets the same order from +[the route stream](/quest/m2/cluster-routing/routes/route-stream.md), where +one side's echoes and routes share a stream. ## Plan @@ -24,19 +27,36 @@ Decided 2026-11-10 (maintainer, /quest-plan): ✅ the neighbour sends the echo on the stream that carries its routes to the node, queued behind every route it sent earlier (rejected: a barrier field naming the last route sent, which keeps the stream but adds state and a field; leaving the choice to the -implementer). This is a lite-07 wire change; lite-07 is still `-wip`, so it -changes freely, and the draft changes in the same PR. Mirror it in js/net's -port ([js/net route layer](/quest/m2/cluster-routing/routes/js.md)); whichever -lands second adapts. +implementer). + +Decided 2026-11-10 (maintainer, a later /quest-plan on lite-07 routing), +asked how the echo reaches "the stream that carries the echoer's routes" +when a session holds several Announce Streams each way, or none back: +✅ **each side says everything about routing on one unidirectional stream**, +so there is exactly one such stream and the order holds by construction +(rejected: a session-scoped echo sent on every stream the echoer answers; +one bidirectional Announce Stream per direction with the echo moved to the +answering half; the barrier again). That wire change is +[the route stream](/quest/m2/cluster-routing/routes/route-stream.md)'s, and +it is where the draft, the session code, and js/net change. This quest is +the model and `Sim` half, which +[moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487) holds, and it +lands first so [the interest set](/quest/m2/cluster-routing/routes/interest-set.md) +can build on it. Done when `residual_an_overtaking_echo_counts_to_infinity` becomes a test that the metric never passes the longest loop-free path, `NoCountToInfinity` moves into `RouteHandoff.cfg`, `CountToInfinity.cfg` is deleted, and the `verify-route` skill's open finding and sync commit are updated. The epic's -loop-freedom bullet then states the echo ordering it relies on. +loop-freedom bullet then states the echo ordering it relies on, and that +production does not have it until the route stream lands. Caused by: [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), which put the echo on the answering stream; the drain rule itself came from [moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676). -Public API: none. Wire: lite-07-wip only. +Public API: none. Wire: none. + +## Related + +- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - checks the same drain under unequal link costs, which the triangle with unit costs cannot reach diff --git a/quest/m2/cluster-routing/routes/interest-set.md b/quest/m2/cluster-routing/routes/interest-set.md new file mode 100644 index 00000000000..00e40e4723a --- /dev/null +++ b/quest/m2/cluster-routing/routes/interest-set.md @@ -0,0 +1,102 @@ +# [L] The route model and router take an interest as an edited set of prefixes + +## Goal + +The route model (`rs/moq-net/verify/route/RouteHandoff.tla`) and the router +(`rs/moq-net/src/route/`) treat a neighbour as one session, keyed once for +both directions, whose interest is a set of prefixes edited in place. TLC +checks every ordering of an edit against the routes, batches, floors, and +echoes sharing one FIFO per direction, and the existing rules hold: an edit +that swaps prefixes for a wider or narrower one keeps every claim both cover, +ends nothing, revives nothing, and sends nothing twice. Nothing on the wire +or in the sessions changes here; [the route stream](/quest/m2/cluster-routing/routes/route-stream.md) +carries it. + +## Plan + +Decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing). The model +goes first because the set's edit rule is the one new rule in this redesign, +and every earlier routing bug was an ordering nobody had enumerated. + +Settled: + +1. ✅ **A stamp covers one prefix, named by its id** (rejected: a stamp + covering the whole set). A floor is `{route, prefix, revision}`, so each + prefix is certified exactly as one stream per prefix was, and the merge + rule is unchanged. A whole-set stamp needs the receiver to track which + prefixes each stamp covers anyway, since a stamp sent before a prefix was + added says nothing about it, and a sender that knows only part of the set + could certify nothing. This amends Plan item 10 of the + [epic](/quest/m2/cluster-routing/routes/README.md): the floor stays a + marker, never a region list, and gains the prefix it names. +2. ✅ **SYNCED is per prefix, naming its id** (rejected: one SYNCED per edit, + counted in order, which makes a small prefix wait behind a large one). + Each edit also gets one marker in the answering side's stream, the point + where it takes effect for both sides. +3. ✅ **The live set never holds a prefix inside another, and one edit may + swap prefixes without re-sending what both cover.** In the maintainer's + words: "we might have fleet/r7 and fleet/r2, then expand it to fleet ... + somehow we should support fleet absorbing the previous two without + retransmitting everything". Rejected: nested prefixes that each list + independently, where a peer multiplies a relay's work by repeating a + prefix, bounded only by message size; and removing the old prefixes only + after the new one is SYNCED, which holds every claim twice during the + handover and is what Plan items 16 and 17 of the epic did across streams. + +The edit rule those three imply, to prove rather than assume: + +- An edit removes live prefixes by id and adds prefixes, each taking the next + id. An edit that would leave one live prefix inside another is refused. +- At the edit's marker, a claim of a removed prefix stays live, with its id + and attributes, when an added prefix covers it and does not hide it; it is + that prefix's claim from then on. Every other claim of a removed prefix is + dropped. A claim presented at a removed prefix itself may stand for a + broader claim (Plan item 11), which the receiver cannot tell apart, so it + is always dropped and sent again if the new set still presents it. +- The sender then sends what the added prefixes newly cover, one floor per + route for each added prefix, and SYNCED for it. +- Widening: a kept claim stays certified by its old prefix's last floor until + its route's first floor for the new prefix. Narrowing: the wider floor + carries to each added prefix inside it. Decide, and prove, what a floor + that certifies nothing does to the narrower floors it replaces; + recommended: it replaces them, so there is one floor per route per prefix. +- A route stays on the stream while some live prefix presents a claim of it + (Plan item 15, over the set). One whose last claim an edit drops ends with + ROUTE_END after the marker, echoed as usual, with no floor first, since + nothing was withdrawn. + +Work: + +- Start from the model and `Sim` with the echo on the neighbour's FIFO + ([the echo order](/quest/m2/cluster-routing/routes/echo-order.md)). +- Give the model at least two prefixes and edits in flight: add, remove, + widen, narrow, and an edit that crosses a claim change, a route change, a + ROUTE_END, and its echo. The model fixes interest at the root today, so + this is new state; keep it tractable by bounding edits as it bounds flaps. +- In the router, `Peer::asked` and `Peer::watch` become sets with a floor and + a synced flag per prefix, `batch` takes the prefix its floor names, and a + neighbour is one key for both directions, as `Sim` already has it. The + test-only `watch` and interest generation go: an interest change is an + edit, and a batch in flight before the marker applies to the old set + instead of being discarded. +- Redo the handoff fixtures that an interest change touches + (`widening_interest_needs_floors_scoped_to_what_they_cover`, + `cancelling_interest_retires_its_region`, + `widening_hides_what_was_frozen_until_a_covering_floor_arrives`, + `a_snapshot_queued_before_widening_keeps_the_frozen_rest_hidden`, and the + two scoped-grant widening fixtures) and both sweeps over edited sets. +- Update the `verify-route` skill's coverage table, its "Not covered" list, + and its sync commit. + +The session layer keeps one prefix per neighbour until the route stream +lands, so this changes no behaviour on its own. + +Public API: none (the router is crate-private). Wire: none. + +## Required + +- [A ROUTE_END echo arrives behind the echoer's routes](/quest/m2/cluster-routing/routes/echo-order.md) - the model and `Sim` with the echo on the neighbour's FIFO, which this builds on + +## Related + +- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - grows the same spec to larger meshes and unequal costs; whichever lands second adapts diff --git a/quest/m2/cluster-routing/routes/js.md b/quest/m2/cluster-routing/routes/js.md index 210a0bd9699..6bda1e56927 100644 --- a/quest/m2/cluster-routing/routes/js.md +++ b/quest/m2/cluster-routing/routes/js.md @@ -40,6 +40,16 @@ the cheapest one; fix it with the route-selection rework here. [moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058) resolves it. The local drain/datagram recipes and `just test media --media-cases replacement` remain available to validate the port. +- Decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing): every + publisher stays an origin node, a tab included, so this port keeps the + origin side (its own Node ID, seqno, and answering ROUTE_REQUEST). A tab + that forwards is planned separately, on the Rust router: + [browser mesh](/quest/m5/browser-mesh.md). +- The Announce Stream becomes one one-way stream per side with a set of + prefixes ([the route stream](/quest/m2/cluster-routing/routes/route-stream.md)). + Whichever of the two lands second adapts: JS then shares one stream across + its `announced()` calls, with the covering-prefix logic Rust's `Discovery` + has, instead of one stream per call. ## Related diff --git a/quest/m2/cluster-routing/routes/route-stream.md b/quest/m2/cluster-routing/routes/route-stream.md new file mode 100644 index 00000000000..31289d3259b --- /dev/null +++ b/quest/m2/cluster-routing/routes/route-stream.md @@ -0,0 +1,136 @@ +# [XL] Each side says everything about routing on one stream + +## Goal + +In a lite-07 session each endpoint opens at most one unidirectional Announce +Stream and sends everything it says about routing on it, in order: the +prefixes it wants, the routes, claims, floors, and SYNCED that answer the +peer's prefixes, its ROUTE_END echoes, and its route requests. No routing +message can overtake another, so production is the route model's one FIFO per +direction by construction, and a ROUTE_END echo always arrives behind the +routes its sender sent first. A change of token scope edits the prefix set +instead of reopening a request. `announced()` keeps its API in Rust and JS, +and lite-06 and older and moq-transport are untouched. + +## Plan + +Decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing). + +Why: today a side's requests ride the streams it opened and its answers ride +the streams its peer opened, and QUIC orders no stream against another. The +count to infinity [the echo order](/quest/m2/cluster-routing/routes/echo-order.md) +found is one message overtaking another across that split, and the router +keys each stream as its own neighbour (`route/node.rs`), where the model and +`Sim` have one neighbour per session. + +Settled: + +1. ✅ **One one-way stream per side** (rejected: one bidirectional Announce + Stream per direction with the echo moved to the answering half, which + still splits a side's messages over two streams, needs a rule for a peer + that opened none, and makes the model grow a second queue per direction; + one bidirectional stream per session opened by the client, which the + client must open even when it wants nothing). +2. ✅ **Its interest is a set of prefixes edited in place**, with a floor per + prefix, SYNCED per prefix, and swaps that keep what both sets cover: + [the interest set](/quest/m2/cluster-routing/routes/interest-set.md) + settles and proves those rules, and this puts them on the wire. +3. ✅ **A token-scope change is an edit.** The subscriber's set is its local + interests met with its scope's regions, one prefix per region, so a + multi-region scope is certifiable. Plan items 16 and 17 of the + [epic](/quest/m2/cluster-routing/routes/README.md) (reopen and replace, + and the old request's own budget) are superseded, and the overlapping + reopened requests that could hold three times the announce cap + ([moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375)'s open + review finding) cannot occur. + +Recommended wire, for the implementer to confirm or improve in the PR; each +departure is a choice to list there: + +- A unidirectional stream type, Announce. An endpoint opens it when it first + has something to say, opens at most one, and keeps it for the session. A + second one, or one that ends before the session, closes the session. No + reopening means no generation of streams to order against each other. +- The bidirectional Announce stream leaves lite-07. +- What a side sends as a subscriber: ANNOUNCE_REQUEST, now an edit (prefix + ids to remove, then prefixes to add, each with its `Hidden`), ROUTE_REQUEST, + and the echo. What it sends as a publisher: one ANNOUNCE_OK per edit it + applied, the marker where the edit takes effect, then ROUTE_START, UPDATE, + HOLD, and END, the ANNOUNCE messages, ANNOUNCE_FLOOR with a Prefix ID, and + SYNCED with a Prefix ID. ANNOUNCE_START names the Prefix ID its suffix is + relative to, and a compression base shares it. +- The echo needs its own message type. Both sides' Route IDs now meet on one + stream, so a ROUTE_END for the sender's own route and the echo of the + peer's cannot share a type. +- A side's Route IDs, Announce IDs, and Prefix IDs are one counter each per + stream. A message names the peer's id where it answers the peer: a Prefix + ID in ANNOUNCE_START, ANNOUNCE_FLOOR, and SYNCED, and a Route ID in the + echo. +- A removed prefix may still be named until the ANNOUNCE_OK for the edit that + removed it, and never after. The subscriber holds a removed prefix's claims + against its cap until then. +- The live set may be empty, never nests, and once encoded fits one + ANNOUNCE_REQUEST (65,535 bytes), as SUBSCRIBE's Ranges do. Each claim is + then presented once, plus one covering claim per route per prefix, so what + a peer can make the other hold follows its own request, not a constant. +- A prefix outside the subscriber's grant lists nothing and is SYNCED at + once; there is no per-prefix refusal. The local library refuses an + `announced()` outside its known scope, as it does today. + +Code this replaces, for lite-07 only: + +- `SubscriberDriver`'s `Tasks` and `Discovery`'s open and + close of covering prefixes (`rs/moq-net/src/lite/subscriber.rs`, + `rs/moq-net/src/discovery.rs`) become one stream machine whose edits follow + `Discovery`. A narrower request no longer stays open under a broader one: + one edit swaps them. +- `AnnouncePrefix::reopen`, `replacing`, `Routed::hand_over`, and + `Slots::split` go, with their tests in `routed.rs` and + `tests/integration/auth.rs`. Delete each outright if no other version uses + it. +- `Routed` (`lite/subscriber/routed.rs`) and `Answer` + (`lite/publisher/routed.rs`) become one per session, holding the prefixes. + `Answer` already reads several node streams (one per mount). +- `route::Node` registers one neighbour per session for both directions. +- `js/net`: each `announced()` call opens its own stream today + (`js/net/src/lite/subscriber.ts`), so JS gains the covering-prefix logic + `Discovery` has. [moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443) + ports the route layer with per-stream state; whichever lands second + adapts, and if it has landed, this PR changes `js/net` too. + +Also touching the same files, to merge around: +[moq-dev/moq#8458](https://github.com/moq-dev/moq/pull/8458) (its +per-announce-stream timer becomes per session) and +[moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) (warm routes on +the Announce Stream). + +Tests: every ordering hazard the old shape had, on real sessions with mocked +time. An echo never overtakes a route; a swap edit shows no claim offline and +re-sends nothing both sets cover; a narrowing edit drops only what it +uncovers; a scope renewal that changes the region keeps every still-covered +claim shown; a second Announce stream, a nested prefix, and a set over the +size limit each close the session; a peer that withholds ANNOUNCE_OK or +echoes holds no more than the announce cap. Replace +`tests/integration/lazy_discovery.rs`'s two-stream cases with edits. + +Update the draft's Streams tables, Announce, Floors, Routing, the message +sections, and the lite-07 changelog in the same PR, and the `verify-route` +skill's note on `node.rs`'s per-stream keys. Run `just check`, +`just test interop --all`, and `just drafts check`. + +Rollout: lite-07 is `-wip`, so a relay on this wire refuses one on the old +by misparse, not by version. Relays pinned to `main` roll together, as each +earlier lite-07 wire change did. + +Public API: none expected. Wire: lite-07-wip only. + +## Required + +- [The route model and router take an interest as an edited set of prefixes](/quest/m2/cluster-routing/routes/interest-set.md) - the edit rule proven in the model and the router that implements it + +## Related + +- [js/net route layer](/quest/m2/cluster-routing/routes/js.md) - the JS side of the same wire +- [A node advances its route seqno only when a route link changes](/quest/m1/route-seqno-bumps.md) - changes the same registration path +- [Announce interest streams share one route table](/quest/m4/announce-interest-memory.md) - lite-07 then holds one interest stream per side; the bound still matters for older versions +- [A regrant re-checks only the routes it can change](/quest/m4/regrant-route-recheck.md) - its per-stream product loses a factor on lite-07 diff --git a/quest/m3/mesh/overlay.md b/quest/m3/mesh/overlay.md index 16cead2cb3e..5714c5adc95 100644 --- a/quest/m3/mesh/overlay.md +++ b/quest/m3/mesh/overlay.md @@ -8,8 +8,11 @@ same overlay from the member list, and forwards for the rest. Any two nodes in a reachable group reach each other through the overlay in about log N hops, a departed node's links heal, an optional per-node degree cap bounds a constrained robot's links, and traffic crosses the relay uplink, which every -relay node keeps, when the overlay partitions. An application embedding -`Endpoint` reaches the overlay through a relay node. +relay node keeps, when the overlay partitions. Every node is a core unless +its application marks it an edge: an edge links to the cores it can reach +and its links cost more, so traffic between edges crosses a core while one +is reachable, and edges ring among themselves when none is. An application +embedding `Endpoint` reaches the overlay through a relay node. ## Plan @@ -32,13 +35,38 @@ join this quest: a loop, and the links and hops are reported. A node with no uplink has no fallback, so report whether the overlay stays connected whenever the reach graph is, and record what such a node can know about being cut off. -- Nodes that do not forward. An embedding application may turn a node's - `proxy` preference off so it carries only its own traffic (a camera or a - constrained robot; Kixel plans a `--leaf` flag for it). Such a node cannot - be a finger that forwards for the rest. Recommendation, to confirm in the - PR: it is not a ring member, links to a small number of members, and is - never chosen as a finger. Expose that preference per node in the relay - library if the embedding application cannot set it today. +- ~~Nodes that do not forward.~~ Superseded 2026-11-10 by the roles below: + Kixel dropped its `--leaf` flag, and every node forwards. + +Decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing, asked how a +beefier node should attract traffic once Kixel's non-forwarding leaf is +gone): ✅ **two roles, core and edge, and both link shaping and link +pricing.** In the maintainer's words: "Kinda like how moq.pro wants to have +edges and cores. Edges should prioritize connecting to cores, but if there's +no cores, do a ring of edges? I just want this to work by default, which I +guess means everything is a core unless stated otherwise." Rejected: pricing +alone; shaping alone; ranking links to cheaper nodes by priority, whose +outcome under the feasibility rule depends on which route arrived first. + +- Every node is a core unless the embedding application says it is an edge. + Every node forwards; the role only changes which links it keeps and what + they cost. +- Shaping: cores ring among themselves as below. An edge links to the cores + it can reach, and rings with other edges only when it reaches none, so the + mesh works with no core at all. Settle in the PR what an edge that reaches + a core does for an edge that reaches none (partial reach); recommended: it + accepts that edge's link. +- Pricing: a node declares a cost, 1 for a core and more for an edge, and a + link costs the product of its two ends' costs. A cost added per hop cannot + move traffic onto a core: a direct link is always cheaper than the same + link plus another. With a core at 1 and an edge at 3, edge to edge costs 9 + and edge to core to edge costs 6, whichever route arrived first. This is + local pricing of each peer's declared cost + ([link costs](/doc/bin/relay/cluster.md#link-costs)) and needs no wire + change; confirm that, and the edge's default cost, in the PR. +- Recommended, to confirm in the PR: an edge marks its links to cores + `upstream`, as moq.pro's edges do, so it never carries one core's routes to + another. - Ring: order the reachable peers (found through mDNS or [address hints](/quest/m3/mesh/addr.md)) by a hash of their endpoint ID, and diff --git a/quest/m4/p2p/README.md b/quest/m4/p2p/README.md index 367f179bc19..ae0e8e438b6 100644 --- a/quest/m4/p2p/README.md +++ b/quest/m4/p2p/README.md @@ -136,6 +136,12 @@ it yet. Which roster peer serves an announce stays the application's explicit selection. If identity or signing is wanted later, moq-lite-07's route identity becomes a public key, independent of P2P. +Decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing): a browser +that forwards for its peers on a local network is planned separately, in m5 +behind the Rust-to-TypeScript port +([browser mesh](/quest/m5/browser-mesh.md)). Browsers in this epic stay +non-transit until it lands. + Decided 2026-10-13: priority then total cost selects among authorized routes; a direct connection does not always win. A watching tab never re-serves what it receives. Offloading a room of watchers is a local relay's job diff --git a/quest/m5/browser-mesh.md b/quest/m5/browser-mesh.md new file mode 100644 index 00000000000..a4b67e18e8d --- /dev/null +++ b/quest/m5/browser-mesh.md @@ -0,0 +1,65 @@ +# [L] Browser tabs forward for their peers on a local network + +## Goal + +A page that opts in forwards broadcasts between the peers it holds direct +sessions with, so browsers on one local network form a mesh: a broadcast one +tab publishes reaches a tab it has no direct session with through a tab that +has both, and keeps flowing when the site's internet drops. A tab is then a +routing node like any relay, and the relay session stays the fallback route. + +## Plan + +Decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing): "Web clients +might forward too, honestly it would be lit to have P2P meshes for browsers +on the local network so please plan that." + +- ✅ The forwarding engine is the Rust router, run in the tab through the + [Rust-to-TypeScript port](/quest/m5/rs2ts-js.md) (rejected: handwriting + feasibility, route requests, holds, and echoes in `js/net`, a second + implementation of the part of the protocol that has produced the most + bugs; pulling the port forward to m4). So the tab runs the router the route + model checks, and the [P2P epic](/quest/m4/p2p/README.md)'s "no second + routing implementation" holds. +- ✅ m5, behind that port. + +This lifts three statements once it lands: the P2P epic's "A browser never +carries another peer's traffic", the +[cluster routing](/quest/m2/cluster-routing/README.md) plan's browser tabs +keeping `proxy` off, and the [js/net route layer](/quest/m2/cluster-routing/routes/js.md)'s +non-transit tab. Until then each stays true. + +It also settles why every publisher stays an origin node (Plan item 18 of +[Routes and announces](/quest/m2/cluster-routing/routes/README.md)): a tab +that forwards needs its own identity. + +To settle when this starts: + +- Which sessions a tab meshes over. The P2P epic's data channels pair on a + LAN with host and mDNS candidates and an empty ICE server list; a tab can + also dial a local relay by WebTransport with a pinned certificate hash. + Recommended: reuse the P2P epic's `Peers`, with `proxy` turned on per + connection, rather than a second connection type. +- Trust. A forwarding tab can alter what it forwards, and a scoped peer's + identities are qualified, so a second scoped hop is refused today. Say + which grant LAN peers give each other, and whether the mesh needs more + than one scoped hop. +- Limits a tab needs that a relay does not: how many peers it forwards for, + and what it does on battery or a metered link. + +Test in the browser suite: three tabs in a line, where the ends share no +session, deliver a broadcast end to end, reroute when the middle tab closes +and a relay path exists, and never loop. + +Public API: a per-connection forwarding option in `@moq/net` or `@moq/p2p`. +Wire: none expected; a forwarding tab speaks the lite-07 route layer as a +relay does. + +## Required + +- [Ship JS-only @moq/net from Rust](/quest/m5/rs2ts-js.md) - the Rust router and model running in the tab +- [P2P](/quest/m4/p2p/README.md) - direct browser sessions, their signaling, and `Peers` + +## Related + +- [js/net route layer](/quest/m2/cluster-routing/routes/js.md) - the handwritten, non-forwarding route layer this replaces in the tab From 360a82a23bb80088ed61553fa3d98c9e62ac6f4d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 20:36:22 -0800 Subject: [PATCH 2/5] quest: a relay echoes a ROUTE_END only when its own downstream has The route model still counts to infinity with the echo on the neighbour's FIFO once the origin sits behind a relay. Records the decision (epic item 25), grows echo-order to own the sound drain in the model, Sim, and router, and keeps per-link costs and that shape in the mesh model. Co-Authored-By: Claude --- quest/m1/route-mesh-model.md | 6 + quest/m2/cluster-routing/routes/README.md | 9 +- quest/m2/cluster-routing/routes/echo-order.md | 121 ++++++++++++------ .../m2/cluster-routing/routes/interest-set.md | 5 +- .../m2/cluster-routing/routes/route-stream.md | 4 +- 5 files changed, 97 insertions(+), 48 deletions(-) diff --git a/quest/m1/route-mesh-model.md b/quest/m1/route-mesh-model.md index cc0e93fc9b5..02803c5d86f 100644 --- a/quest/m1/route-mesh-model.md +++ b/quest/m1/route-mesh-model.md @@ -30,6 +30,12 @@ are routing's confidence gates for finalizing lite-07. - Count to infinity stays in its own open-finding config until [the echo fix](/quest/m2/cluster-routing/routes/echo-order.md) lands, so the other rules still run on larger meshes. +- Decided 2026-11-10 (maintainer, a later /quest-plan on lite-07 routing): + the model takes a cost per link, since deployments price links, and always + includes an origin that is not adjacent to every relay. The triangle with + the origin as one of its nodes cannot reach the count to infinity found + that day, which needs only an origin behind a relay; the echo fix adds the + costs and that shape, and this keeps both on every larger mesh. Replay every counterexample on `route::tests::Sim` before calling it a bug, as the `verify-route` skill says, and pin confirmed ones as `residual_*` tests. diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index 648917c8eb2..45c170ed6cf 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -40,6 +40,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 22. ✅ **A side's interest is a set of prefixes edited in place.** The live set never nests, and one edit may swap prefixes for a wider or narrower one without re-sending what both cover ([the interest set](/quest/m2/cluster-routing/routes/interest-set.md)). A token-scope change is such an edit, with one prefix per region of the scope. This supersedes items 16 and 17, and the handoff contract's "An interest change opens a new stream" below, once the route stream lands. 23. ✅ Asked what a floor covers with several prefixes on one stream: **one prefix, named by its id.** This amends item 10: the floor is still a marker and never a region list. Rejected: a floor over the whole set. 24. ✅ Asked how a side learns a newly added prefix's initial set is complete: **SYNCED per prefix, naming its id**, plus one marker per edit where it takes effect for both sides. Rejected: one SYNCED per edit, counted in order. +25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **a relay echoes a ROUTE_END from the neighbour it routed through only when its own ROUTE_ENDs have been echoed**, and any other neighbour's at once ([the echo fix](/quest/m2/cluster-routing/routes/echo-order.md), which holds the counterexample). This supersedes the "per neighbour, not a diffusing computation" claim in the handoff contract and in decision 6 below. Rejected: never dropping the distance; a timer; relays as the only origin nodes. The route layer's core landed in @@ -190,8 +191,10 @@ the bullets below are the contract it carries. accept a route if its seqno is newer, or equal with a metric below the feasibility distance. Only the origin advances its seqno. Retraction is ROUTE_HOLD or ROUTE_END. The route model found a count to infinity - (2026-11-10): an echo can overtake a route derived from the echoed one; - [the echo fix](/quest/m2/cluster-routing/routes/echo-order.md) orders it. + (2026-11-10): an echo can overtake a route derived from the echoed one, + and even in order a per-neighbour echo misses a derived copy held by a + third relay; [the echo fix](/quest/m2/cluster-routing/routes/echo-order.md) + makes the echo wait for the echoer's own downstream (Plan item 25). - Starvation recovery is Babel's seqno request (RFC 8966 s3.8, decided 2026-10-12). A node whose routes to an origin are all infeasible sends ROUTE_REQUEST for that origin, with its feasibility seqno plus one, to the @@ -555,7 +558,7 @@ benchmarks. - [Route layer acceptance](/quest/m2/cluster-routing/routes/acceptance.md) - forwarding fixtures ported and the model deleted, incident fixtures, end-to-end checks, and benchmarks - [Firefox tabs prove their route key](/quest/m2/cluster-routing/routes/browser-proof.md) - a current Firefox tab opens a lite-07 session with a route-key proof the relay verifies, once neqo's exporter follows the draft - [Playwright bundles Chromium's spec exporter](/quest/m2/cluster-routing/routes/playwright-pin.md) - the toolchain's Playwright bundles Chromium 158+, so the route key lane proves the Chromium tab -- [A ROUTE_END echo arrives behind the echoer's routes](/quest/m2/cluster-routing/routes/echo-order.md) - no node drops its feasibility distance while a route derived from its own can still arrive, so routes never count to infinity +- [A node forgets a route only once nothing derived from it remains](/quest/m2/cluster-routing/routes/echo-order.md) - a node echoes a ROUTE_END only when its own downstream has, so no relay drops its feasibility distance while a derived route exists and routes never count to infinity - [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - every ordering on 4-5 relay meshes with two origins and clients, with an announce budget and convergence - [Seeded lite-07 meshes hold the route rules and an announce budget](/quest/m1/route-mesh-sweep.md) - the route rules and message budget checked on real sessions over random meshes diff --git a/quest/m2/cluster-routing/routes/echo-order.md b/quest/m2/cluster-routing/routes/echo-order.md index 75e909762f6..28e92e4ea0c 100644 --- a/quest/m2/cluster-routing/routes/echo-order.md +++ b/quest/m2/cluster-routing/routes/echo-order.md @@ -1,14 +1,13 @@ -# [M] A ROUTE_END echo arrives behind the echoer's routes +# [L] A node forgets a route only once nothing derived from it remains ## Goal -In the route model and `route::tests::Sim`, a ROUTE_END echo travels on the -FIFO toward the END's sender, behind every route the echoer sent first. A -node there drops its feasibility distance for an identity only once no -advertisement derived from its own can still reach it, so two relays never -count a route to infinity. Production gets the same order from -[the route stream](/quest/m2/cluster-routing/routes/route-stream.md), where -one side's echoes and routes share a stream. +A lite-07 node drops its feasibility distance for an identity only once no +advertisement derived from its own exists anywhere, on a link or held by +another relay. The router, `route::tests::Sim`, and the route model agree on +the rule, and TLC finds no count to infinity on meshes where the origin sits +behind a relay, with unit and with unequal link costs. A dead publisher goes +offline everywhere and routing goes quiet. ## Plan @@ -18,45 +17,85 @@ Found 2026-11-10 by the route model ([moq-dev/moq#8447](https://github.com/moq-d echoed its ROUTE_END (`update` in `route/mod.rs`). But the echo rides the announce stream the node answers, while the neighbour's own routes ride the stream the node asked for (`route/node.rs`), so the echo can overtake an older -route the neighbour derived from the node's. The node then accepts that route -as a fresh, longer one. `residual_an_overtaking_echo_counts_to_infinity` in -`route/tests/routing.rs` replays it on the router: the metric climbs 3 to 8 -over six rounds. +route the neighbour derived from the node's. Decided 2026-11-10 (maintainer, /quest-plan): ✅ the neighbour sends the echo on the stream that carries its routes to the node, queued behind every route -it sent earlier (rejected: a barrier field naming the last route sent, which -keeps the stream but adds state and a field; leaving the choice to the -implementer). - -Decided 2026-11-10 (maintainer, a later /quest-plan on lite-07 routing), -asked how the echo reaches "the stream that carries the echoer's routes" -when a session holds several Announce Streams each way, or none back: -✅ **each side says everything about routing on one unidirectional stream**, -so there is exactly one such stream and the order holds by construction -(rejected: a session-scoped echo sent on every stream the echoer answers; -one bidirectional Announce Stream per direction with the echo moved to the -answering half; the barrier again). That wire change is -[the route stream](/quest/m2/cluster-routing/routes/route-stream.md)'s, and -it is where the draft, the session code, and js/net change. This quest is -the model and `Sim` half, which -[moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487) holds, and it -lands first so [the interest set](/quest/m2/cluster-routing/routes/interest-set.md) -can build on it. - -Done when `residual_an_overtaking_echo_counts_to_infinity` becomes a test -that the metric never passes the longest loop-free path, `NoCountToInfinity` -moves into `RouteHandoff.cfg`, `CountToInfinity.cfg` is deleted, and the -`verify-route` skill's open finding and sync commit are updated. The epic's -loop-freedom bullet then states the echo ordering it relies on, and that -production does not have it until the route stream lands. +it sent earlier (rejected: a barrier field naming the last route sent; +leaving the choice to the implementer). +[moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487) did that in the +model and `Sim`. In production the order comes from one unidirectional +Announce Stream per side +([the route stream](/quest/m2/cluster-routing/routes/route-stream.md), +decided the same day), not from moving the echo between today's streams. + +Found 2026-11-10, a later /quest-plan on lite-07 routing: **that order is +not enough.** With the echo on the neighbour's FIFO, the model still counts +to infinity as soon as the origin is not adjacent to every relay. An echo +proves only that nothing derived is in flight on that one link; a derived +copy can sit at a third relay and arrive over two hops. The shortest case, +an origin behind relay A with A, B, and C in a triangle and unequal costs: + +1. The origin announces. A advertises to B and C, and B and C to each other. +2. The origin's session closes. A sends ROUTE_END to B and C. +3. B and C each apply it and echo. Their earlier ROUTEs to each other are + still queued. +4. A has both echoes and drops its distance. +5. B receives C's old ROUTE, which is feasible there, selects it, and + advertises it to A, which accepts it. +6. The route laps A, C, B, A, each relay draining and accepting again, and + the metric climbs without bound. + +With unit costs the same count needs only a delayed ROUTE and three relays +that drain in turn. #8487's triangle passes because its origin is one of the +three nodes and drops what it hears for its own identity. The branch +`route-drain-counterexamples` (commit `c6ccf3f71c`, built on #8487's head) +holds per-link costs in the model, the two failing configs (`DrainCosts`, 20 +states, and `DrainUnit`, 23 states), and four `Sim` replays that follow the +traces step for step. No forwarding loop closed in them; the harm is a dead +origin shown and routed to forever. + +Decided 2026-11-10 (maintainer, the same /quest-plan), asked how a relay +safely forgets a publisher that is gone: ✅ **a relay acknowledges a +ROUTE_END only when its own downstream has.** A node that was routing +through the sender echoes its ROUTE_END only once every ROUTE_END it sent in +turn has been echoed; a ROUTE_END from any other neighbour is echoed at +once, so a cycle cannot deadlock. A node then drains only when nothing +derived from its route remains anywhere. Rejected: never dropping the +distance, about 16 bytes per publisher ever seen, which grows without bound +since every publisher is an origin node (epic item 18) and which a peer can +grow on purpose; a timer as in RFC 8966, whose safety rests on stale +messages being gone by then; and making only relays origin nodes so that +never forgetting costs nothing, rejected again. + +That rule is not modelled yet, and it is the most intricate of the options, +so the model goes first: + +- Put the rule in `RouteHandoff.tla` with per-link costs, add the origin + behind a triangle (both cost sets) to the checked configs, and run until + `NoCountToInfinity`, `LoopFree`, and `QuietIsSettled` pass at full bounds. + State exactly which neighbours a node waits on when its fallback is + feasible and it sends no ROUTE_END of its own; that case is where the + count above starts. +- Check a ring of four with no origin leaf and a second origin placement, as + far as explicit-state runs allow, and say what stays unexplored. +- Prove a peer that withholds echoes still holds no more than today: one end + per node per stream. +- Then the router and `Sim`, with the four replays turned into tests that the + metric never passes the longest loop-free path. If the rule cannot be made + to pass, stop and report it rather than weakening a rule. +- Update the draft's Routing text for when an echo is sent; the message and + its fields do not change. Update the `verify-route` skill's open finding, + coverage, and sync commit, and the epic's loop-freedom bullet and its + decision 6, which says a per-neighbour echo is enough. Caused by: [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), which -put the echo on the answering stream; the drain rule itself came from -[moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676). +put the echo on the answering stream, and +[moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676), whose drain +rule assumed a per-neighbour echo proves no derived route remains. -Public API: none. Wire: none. +Public API: none. Wire: lite-07-wip, the timing of the echo only. ## Related -- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - checks the same drain under unequal link costs, which the triangle with unit costs cannot reach +- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - grows the same spec to larger meshes; it keeps the per-link costs this adds diff --git a/quest/m2/cluster-routing/routes/interest-set.md b/quest/m2/cluster-routing/routes/interest-set.md index 00e40e4723a..61dea28bc81 100644 --- a/quest/m2/cluster-routing/routes/interest-set.md +++ b/quest/m2/cluster-routing/routes/interest-set.md @@ -68,7 +68,8 @@ The edit rule those three imply, to prove rather than assume: Work: - Start from the model and `Sim` with the echo on the neighbour's FIFO - ([the echo order](/quest/m2/cluster-routing/routes/echo-order.md)). + and a sound drain + ([the echo fix](/quest/m2/cluster-routing/routes/echo-order.md)). - Give the model at least two prefixes and edits in flight: add, remove, widen, narrow, and an edit that crosses a claim change, a route change, a ROUTE_END, and its echo. The model fixes interest at the root today, so @@ -95,7 +96,7 @@ Public API: none (the router is crate-private). Wire: none. ## Required -- [A ROUTE_END echo arrives behind the echoer's routes](/quest/m2/cluster-routing/routes/echo-order.md) - the model and `Sim` with the echo on the neighbour's FIFO, which this builds on +- [A node forgets a route only once nothing derived from it remains](/quest/m2/cluster-routing/routes/echo-order.md) - the model, `Sim`, and router with the echo on the neighbour's FIFO, per-link costs, and a sound drain, which this builds on ## Related diff --git a/quest/m2/cluster-routing/routes/route-stream.md b/quest/m2/cluster-routing/routes/route-stream.md index 31289d3259b..c603d3afa90 100644 --- a/quest/m2/cluster-routing/routes/route-stream.md +++ b/quest/m2/cluster-routing/routes/route-stream.md @@ -18,8 +18,8 @@ Decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing). Why: today a side's requests ride the streams it opened and its answers ride the streams its peer opened, and QUIC orders no stream against another. The -count to infinity [the echo order](/quest/m2/cluster-routing/routes/echo-order.md) -found is one message overtaking another across that split, and the router +first count to infinity [the echo fix](/quest/m2/cluster-routing/routes/echo-order.md) +records is one message overtaking another across that split, and the router keys each stream as its own neighbour (`route/node.rs`), where the model and `Sim` have one neighbour per session. From 7daa3267fdbf10f09918017397774a546ac55114 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 23:40:33 -0800 Subject: [PATCH 3/5] quest: a route names the relays it crossed, replacing the deferred echo The route model ran the deferred echo: it fails when a session closes before answering. A hop list per origin route passes the same shapes, with the echo ordered or not, so the fix lands on today's streams. Co-Authored-By: Claude --- quest/m1/route-mesh-model.md | 6 - quest/m2/cluster-routing/routes/README.md | 9 +- quest/m2/cluster-routing/routes/echo-order.md | 214 +++++++++++------- .../m2/cluster-routing/routes/interest-set.md | 14 +- quest/m2/cluster-routing/routes/js.md | 1 + .../m2/cluster-routing/routes/route-stream.md | 7 +- 6 files changed, 154 insertions(+), 97 deletions(-) diff --git a/quest/m1/route-mesh-model.md b/quest/m1/route-mesh-model.md index 02803c5d86f..cc0e93fc9b5 100644 --- a/quest/m1/route-mesh-model.md +++ b/quest/m1/route-mesh-model.md @@ -30,12 +30,6 @@ are routing's confidence gates for finalizing lite-07. - Count to infinity stays in its own open-finding config until [the echo fix](/quest/m2/cluster-routing/routes/echo-order.md) lands, so the other rules still run on larger meshes. -- Decided 2026-11-10 (maintainer, a later /quest-plan on lite-07 routing): - the model takes a cost per link, since deployments price links, and always - includes an origin that is not adjacent to every relay. The triangle with - the origin as one of its nodes cannot reach the count to infinity found - that day, which needs only an origin behind a relay; the echo fix adds the - costs and that shape, and this keeps both on every larger mesh. Replay every counterexample on `route::tests::Sim` before calling it a bug, as the `verify-route` skill says, and pin confirmed ones as `residual_*` tests. diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index 45c170ed6cf..4c5e032a32c 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -40,7 +40,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 22. ✅ **A side's interest is a set of prefixes edited in place.** The live set never nests, and one edit may swap prefixes for a wider or narrower one without re-sending what both cover ([the interest set](/quest/m2/cluster-routing/routes/interest-set.md)). A token-scope change is such an edit, with one prefix per region of the scope. This supersedes items 16 and 17, and the handoff contract's "An interest change opens a new stream" below, once the route stream lands. 23. ✅ Asked what a floor covers with several prefixes on one stream: **one prefix, named by its id.** This amends item 10: the floor is still a marker and never a region list. Rejected: a floor over the whole set. 24. ✅ Asked how a side learns a newly added prefix's initial set is complete: **SYNCED per prefix, naming its id**, plus one marker per edit where it takes effect for both sides. Rejected: one SYNCED per edit, counted in order. -25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **a relay echoes a ROUTE_END from the neighbour it routed through only when its own ROUTE_ENDs have been echoed**, and any other neighbour's at once ([the echo fix](/quest/m2/cluster-routing/routes/echo-order.md), which holds the counterexample). This supersedes the "per neighbour, not a diffusing computation" claim in the handoff contract and in decision 6 below. Rejected: never dropping the distance; a timer; relays as the only origin nodes. +25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the relays it crossed, and a node never uses a route that names itself** ([the fix](/quest/m2/cluster-routing/routes/echo-order.md), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes. The route layer's core landed in @@ -193,8 +193,9 @@ the bullets below are the contract it carries. ROUTE_HOLD or ROUTE_END. The route model found a count to infinity (2026-11-10): an echo can overtake a route derived from the echoed one, and even in order a per-neighbour echo misses a derived copy held by a - third relay; [the echo fix](/quest/m2/cluster-routing/routes/echo-order.md) - makes the echo wait for the echoer's own downstream (Plan item 25). + third relay; [the fix](/quest/m2/cluster-routing/routes/echo-order.md) + adds the relays a route crossed, and a node refuses one naming itself + (Plan item 25). - Starvation recovery is Babel's seqno request (RFC 8966 s3.8, decided 2026-10-12). A node whose routes to an origin are all infeasible sends ROUTE_REQUEST for that origin, with its feasibility seqno plus one, to the @@ -558,7 +559,7 @@ benchmarks. - [Route layer acceptance](/quest/m2/cluster-routing/routes/acceptance.md) - forwarding fixtures ported and the model deleted, incident fixtures, end-to-end checks, and benchmarks - [Firefox tabs prove their route key](/quest/m2/cluster-routing/routes/browser-proof.md) - a current Firefox tab opens a lite-07 session with a route-key proof the relay verifies, once neqo's exporter follows the draft - [Playwright bundles Chromium's spec exporter](/quest/m2/cluster-routing/routes/playwright-pin.md) - the toolchain's Playwright bundles Chromium 158+, so the route key lane proves the Chromium tab -- [A node forgets a route only once nothing derived from it remains](/quest/m2/cluster-routing/routes/echo-order.md) - a node echoes a ROUTE_END only when its own downstream has, so no relay drops its feasibility distance while a derived route exists and routes never count to infinity +- [A node never takes back a route that crossed it](/quest/m2/cluster-routing/routes/echo-order.md) - each route names the relays it crossed and a node refuses one naming itself, in the model, the router, and the wire - [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - every ordering on 4-5 relay meshes with two origins and clients, with an announce budget and convergence - [Seeded lite-07 meshes hold the route rules and an announce budget](/quest/m1/route-mesh-sweep.md) - the route rules and message budget checked on real sessions over random meshes diff --git a/quest/m2/cluster-routing/routes/echo-order.md b/quest/m2/cluster-routing/routes/echo-order.md index 28e92e4ea0c..ac1615cb427 100644 --- a/quest/m2/cluster-routing/routes/echo-order.md +++ b/quest/m2/cluster-routing/routes/echo-order.md @@ -1,40 +1,34 @@ -# [L] A node forgets a route only once nothing derived from it remains +# [L] A node never takes back a route that crossed it ## Goal -A lite-07 node drops its feasibility distance for an identity only once no -advertisement derived from its own exists anywhere, on a link or held by -another relay. The router, `route::tests::Sim`, and the route model agree on -the rule, and TLC finds no count to infinity on meshes where the origin sits -behind a relay, with unit and with unequal link costs. A dead publisher goes -offline everywhere and routing goes quiet. +A lite-07 route names the relays it crossed, and a node never uses a route +that names itself. A node that has forgotten an identity therefore cannot +take back a copy of its own old route, however long that copy was in flight +and whatever sessions closed meanwhile. The router, `route::tests::Sim`, the +route model, and the draft agree on the rule, and TLC finds no count to +infinity on meshes where the origin sits behind a relay, with unit and with +unequal link costs, with the echo travelling as it does on today's streams. +A dead publisher goes offline everywhere and routing goes quiet. ## Plan Found 2026-11-10 by the route model ([moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447)): `rs/moq-net/verify/route/CountToInfinity.cfg` breaks `NoCountToInfinity` in -9 steps with no partition. A node drops its distance once every neighbour -echoed its ROUTE_END (`update` in `route/mod.rs`). But the echo rides the -announce stream the node answers, while the neighbour's own routes ride the -stream the node asked for (`route/node.rs`), so the echo can overtake an older -route the neighbour derived from the node's. - -Decided 2026-11-10 (maintainer, /quest-plan): ✅ the neighbour sends the echo -on the stream that carries its routes to the node, queued behind every route -it sent earlier (rejected: a barrier field naming the last route sent; -leaving the choice to the implementer). -[moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487) did that in the -model and `Sim`. In production the order comes from one unidirectional -Announce Stream per side -([the route stream](/quest/m2/cluster-routing/routes/route-stream.md), -decided the same day), not from moving the echo between today's streams. - -Found 2026-11-10, a later /quest-plan on lite-07 routing: **that order is -not enough.** With the echo on the neighbour's FIFO, the model still counts -to infinity as soon as the origin is not adjacent to every relay. An echo -proves only that nothing derived is in flight on that one link; a derived -copy can sit at a third relay and arrive over two hops. The shortest case, -an origin behind relay A with A, B, and C in a triangle and unequal costs: +9 steps with no partition. A node drops its feasibility distance once every +neighbour echoed its ROUTE_END (`update` in `route/mod.rs`). But the echo +rides the announce stream the node answers, while the neighbour's own routes +ride the stream the node asked for (`route/node.rs`), so the echo can +overtake an older route the neighbour derived from the node's. + +Found the same day, a /quest-plan on lite-07 routing: **putting the echo in +order is not enough.** With the echo behind the echoer's routes, as +[moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487) has it in the +model and `Sim`, the model still counts to infinity once the origin is not +adjacent to every relay. An echo proves only that nothing derived is in +flight on that one link; a derived copy can sit at a third relay and arrive +over two hops. The shortest case, an origin behind relay A with A, B, and C +in a triangle and unequal costs: 1. The origin announces. A advertises to B and C, and B and C to each other. 2. The origin's session closes. A sends ROUTE_END to B and C. @@ -46,56 +40,122 @@ an origin behind relay A with A, B, and C in a triangle and unequal costs: 6. The route laps A, C, B, A, each relay draining and accepting again, and the metric climbs without bound. -With unit costs the same count needs only a delayed ROUTE and three relays -that drain in turn. #8487's triangle passes because its origin is one of the -three nodes and drops what it hears for its own identity. The branch -`route-drain-counterexamples` (commit `c6ccf3f71c`, built on #8487's head) -holds per-link costs in the model, the two failing configs (`DrainCosts`, 20 -states, and `DrainUnit`, 23 states), and four `Sim` replays that follow the -traces step for step. No forwarding loop closed in them; the harm is a dead -origin shown and routed to forever. +#8487's triangle passes because its origin is one of the three nodes and +drops what it hears for its own identity. Decided 2026-11-10 (maintainer, the same /quest-plan), asked how a relay -safely forgets a publisher that is gone: ✅ **a relay acknowledges a -ROUTE_END only when its own downstream has.** A node that was routing -through the sender echoes its ROUTE_END only once every ROUTE_END it sent in -turn has been echoed; a ROUTE_END from any other neighbour is echoed at -once, so a cycle cannot deadlock. A node then drains only when nothing -derived from its route remains anywhere. Rejected: never dropping the -distance, about 16 bytes per publisher ever seen, which grows without bound -since every publisher is an origin node (epic item 18) and which a peer can -grow on purpose; a timer as in RFC 8966, whose safety rests on stale -messages being gone by then; and making only relays origin nodes so that -never forgetting costs nothing, rejected again. - -That rule is not modelled yet, and it is the most intricate of the options, -so the model goes first: - -- Put the rule in `RouteHandoff.tla` with per-link costs, add the origin - behind a triangle (both cost sets) to the checked configs, and run until - `NoCountToInfinity`, `LoopFree`, and `QuietIsSettled` pass at full bounds. - State exactly which neighbours a node waits on when its fallback is - feasible and it sends no ROUTE_END of its own; that case is where the - count above starts. -- Check a ring of four with no origin leaf and a second origin placement, as - far as explicit-state runs allow, and say what stays unexplored. -- Prove a peer that withholds echoes still holds no more than today: one end - per node per stream. -- Then the router and `Sim`, with the four replays turned into tests that the - metric never passes the longest loop-free path. If the rule cannot be made - to pass, stop and report it rather than weakening a rule. -- Update the draft's Routing text for when an echo is sent; the message and - its fields do not change. Update the `verify-route` skill's open finding, - coverage, and sync commit, and the epic's loop-freedom bullet and its - decision 6, which says a per-neighbour echo is enough. - -Caused by: [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), which -put the echo on the answering stream, and -[moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676), whose drain -rule assumed a per-neighbour echo proves no derived route remains. - -Public API: none. Wire: lite-07-wip, the timing of the echo only. +safely forgets a publisher that is gone: a relay acknowledges a ROUTE_END +only when its own downstream has. **Superseded the same day, after the model +ran it.** That rule needed three sub-rules, each forced by a counterexample +(a held route counts as routing through the sender; a holding node defers +until the hold resolves; a deferred echo waits only on the ROUTE_ENDs +outstanding when it was deferred). It then passed with one flap and failed +with two: a closed session counts as that neighbour's echo while its last +advertisement is still in flight, so a relay takes back one stale copy of +its own route. The copy stayed within two laps, but no rule that works by +proving "nothing is left" survives a session that closes before answering. +It also makes an upstream wait on its whole subtree before it may +re-advertise, and lets a peer that withholds its echo stall that upstream. + +Decided 2026-11-10 (maintainer, the same /quest-plan, with the model results +in hand): ✅ **each route names the relays it crossed, and a node never uses +a route that names itself.** Feasibility still selects routes and still +stops hunting; the list is only a refusal, and it covers the one case +feasibility cannot, a node that no longer remembers. It is one list per +origin route, not the per-claim Hop ID list lite-07 removed, so a link +change still sends one ROUTE_UPDATE per origin, and the Rendezvous Hash and +per-broadcast choices among neighbours are unaffected: a node keeps one +entry per origin per neighbour as today. The echo and the drain rule stay +exactly as they are. Rejected: the deferred echo above; never dropping the +distance, about 16 bytes per publisher ever seen, which a peer can grow on +purpose; a timer as in RFC 8966, whose safety rests on stale messages being +gone by then; and making only relays origin nodes, rejected again. + +What the model showed, all in +`rs/moq-net/verify/route/DRAIN.md` on the branch +`route-drain-counterexamples` (`5e7e3eb434`), which also holds the model +with all three variants behind switches, per-link costs, the configs that +reproduce each result, and four `Sim` replays of the original count: + +- With the list, the triangle passes at full bounds, and the origin behind + a triangle passes with two flaps at unit cost and at four unequal cost + sets. The same holds with the echo free to overtake its sender's other + messages, which is today's wire; without the list that ordering counts to + infinity in 11 states. +- The echo cannot go: with the list and no echo, a node forgets while a + neighbour still routes through its advertisement, and `LoopFree` fails in + 944 states. +- A self-naming route is kept as an unusable entry until it ends. Dropping + it outright lets collection run while the neighbour's route is still open. +- Not run: full bounds on the 4-relay shapes, any 5-relay shape, and the + runs listed there as stopped. No message count was measured. + +So this lands on today's Announce Streams and does not wait for +[the route stream](/quest/m2/cluster-routing/routes/route-stream.md), which +later puts the echo in order by construction. A recommendation the +maintainer has not confirmed; say so in the PR. + +Work, model first: + +- In `RouteHandoff.tla`, keep the echo on its own queue as on `main`, add + the list and a cost per link, add the origin behind a triangle (unit and + unequal costs) to the checked configs, and turn `NoCountToInfinity` on in + every config. Delete `CountToInfinity.cfg` as an open finding. If + [moq-dev/moq#8479](https://github.com/moq-dev/moq/pull/8479) has landed, + that includes its line, ring, and chord, with costs; if not, whichever + lands second adapts. +- **The list's one cost is unmeasured, so measure it before the router.** + An upstream path change at equal metric now sends a ROUTE_UPDATE to + everyone downstream, where today it stops. Run the list against #8479's + announce budget (4 routing messages per session direction per external + change) on its meshes. If it breaks the budget or brings hunting back, + stop and report it rather than raising the budget. +- Say what stays unexplored. Run TLC one config at a time on the recipe's + settings; the 4-relay shapes with the origin inside a cycle did not finish + in 40 minutes on a loaded host. +- Router: an advertisement and a route entry carry the list; `route` marks + an entry that names this node; `selected`, the infeasible set, and the + certifiers skip it, so it never holds a route or asks for a seqno; a node + appends its own Node ID when it advertises. Turn the four replays and the + two-flap trace into tests that the metric never passes the longest + loop-free path. +- Wire, for the implementer to confirm or improve; list each departure in + the PR. ROUTE_START and ROUTE_UPDATE gain a count and that many Node IDs, + in order from the origin, the sender last and the origin itself left out. + A node checks the list for its own Node ID as received, before it files + anything a scoped peer sent under a qualified Node ID. A list longer than + a fixed limit makes the route unusable; reuse lite-06's hop limit unless + there is a reason not to. A node that forwards declares a Node ID, since + it has to name itself. +- Draft: the two messages, Routing (a route that names the receiver, and + that forgetting is safe because of it, not because of the echo alone), + and the lite-07 changelog. `js/net` has no route layer yet, so + [its quest](/quest/m2/cluster-routing/routes/js.md) decodes the list and + sends an empty one for its own route. +- Update the `verify-route` skill's open finding, coverage, and sync + commit, and the epic's loop-freedom bullet and decision 6, which say the + echo alone makes forgetting safe. +- [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)'s cache + sweep runs on the Rust router and depends on the feasibility check; rerun + it with the list if it has landed. +- Run `just check`, `just test interop --all`, and `just drafts check`. + +#8487 holds this quest's branch with the echo moved behind the echoer's +routes in the model and `Sim`. That change is no longer needed here and +belongs to the route stream; start again from `main`'s model. + +Rollout: lite-07 is `-wip`, so relays pinned to `main` roll together, as +each earlier lite-07 wire change did. + +Caused by: [moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676), +whose drain rule assumed a per-neighbour echo proves no derived route +remains, and [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), +which put the echo on the answering stream. + +Public API: none. Wire: lite-07-wip, a hop list on ROUTE_START and +ROUTE_UPDATE. ## Related -- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - grows the same spec to larger meshes; it keeps the per-link costs this adds +- [Each side says everything about routing on one stream](/quest/m2/cluster-routing/routes/route-stream.md) - puts the echo in order later; this does not wait for it +- [js/net route layer](/quest/m2/cluster-routing/routes/js.md) - decodes the list in JS diff --git a/quest/m2/cluster-routing/routes/interest-set.md b/quest/m2/cluster-routing/routes/interest-set.md index 61dea28bc81..a6f8c4c2f02 100644 --- a/quest/m2/cluster-routing/routes/interest-set.md +++ b/quest/m2/cluster-routing/routes/interest-set.md @@ -67,9 +67,11 @@ The edit rule those three imply, to prove rather than assume: Work: -- Start from the model and `Sim` with the echo on the neighbour's FIFO - and a sound drain - ([the echo fix](/quest/m2/cluster-routing/routes/echo-order.md)). +- Start from the model and `Sim` with per-link costs and the hop list + ([the fix](/quest/m2/cluster-routing/routes/echo-order.md)), and move the + echo from its own queue onto the neighbour's FIFO, as + [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487) did: with one + stream per side nothing overtakes anything. - Give the model at least two prefixes and edits in flight: add, remove, widen, narrow, and an edit that crosses a claim change, a route change, a ROUTE_END, and its echo. The model fixes interest at the root today, so @@ -96,8 +98,4 @@ Public API: none (the router is crate-private). Wire: none. ## Required -- [A node forgets a route only once nothing derived from it remains](/quest/m2/cluster-routing/routes/echo-order.md) - the model, `Sim`, and router with the echo on the neighbour's FIFO, per-link costs, and a sound drain, which this builds on - -## Related - -- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - grows the same spec to larger meshes and unequal costs; whichever lands second adapts +- [A node never takes back a route that crossed it](/quest/m2/cluster-routing/routes/echo-order.md) - the model, `Sim`, and router with per-link costs and the hop list, which this builds on diff --git a/quest/m2/cluster-routing/routes/js.md b/quest/m2/cluster-routing/routes/js.md index 6bda1e56927..aef64e866a7 100644 --- a/quest/m2/cluster-routing/routes/js.md +++ b/quest/m2/cluster-routing/routes/js.md @@ -18,6 +18,7 @@ Decided 2026-10-27 (maintainer), from the cheapest one; fix it with the route-selection rework here. - Feed the availability API (`RouteInfo.available` and the `offline` route kind) from the route layer. - Rust/JS interop covers the route messages, the floor marker, and the echo. +- Decode the hop list on ROUTE_START and ROUTE_UPDATE and send an empty one for the tab's own route, once [the fix](/quest/m2/cluster-routing/routes/echo-order.md) adds it. A tab forwards nothing, so it never appends itself; the [browser mesh](/quest/m5/browser-mesh.md) does. - Encode lite-07's Route ID (ANNOUNCE_START) and Node ID (every TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS request, 0 when the publisher selects the node) as Rust does, and pin `js/net/src/lite/announce.test.ts`'s goldens to `lite::compress`'s `GOLDEN` and `JS_LITERAL`. - Leave the temporary Lite-06 Chain Setup Parameter unsent: JS translates nothing, so it keeps no chain, and a Rust peer sends the chain only to a peer that offers it. - Restore the native JS lite-07 lanes. Decided 2026-11-06 (maintainer, diff --git a/quest/m2/cluster-routing/routes/route-stream.md b/quest/m2/cluster-routing/routes/route-stream.md index c603d3afa90..379c72b2c8b 100644 --- a/quest/m2/cluster-routing/routes/route-stream.md +++ b/quest/m2/cluster-routing/routes/route-stream.md @@ -18,10 +18,13 @@ Decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing). Why: today a side's requests ride the streams it opened and its answers ride the streams its peer opened, and QUIC orders no stream against another. The -first count to infinity [the echo fix](/quest/m2/cluster-routing/routes/echo-order.md) +first count to infinity [the fix](/quest/m2/cluster-routing/routes/echo-order.md) records is one message overtaking another across that split, and the router keys each stream as its own neighbour (`route/node.rs`), where the model and -`Sim` have one neighbour per session. +`Sim` have one neighbour per session. That fix no longer rests on this: the +hop list it adds holds with the echo unordered, so it lands first, on +today's streams. This removes the ordering hazard as a class and makes a +token-scope change an edit. Settled: From be4049e55c38d4cae48c4bae5152414510e3d6c5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 23:44:00 -0800 Subject: [PATCH 4/5] quest: name JS's routes(), say who merges the neighbour key, list the browser mesh Co-Authored-By: Claude --- quest/m2/cluster-routing/routes/interest-set.md | 7 +++++-- quest/m2/cluster-routing/routes/js.md | 2 +- quest/m2/cluster-routing/routes/route-stream.md | 6 +++--- quest/m5/README.md | 1 + 4 files changed, 10 insertions(+), 6 deletions(-) diff --git a/quest/m2/cluster-routing/routes/interest-set.md b/quest/m2/cluster-routing/routes/interest-set.md index a6f8c4c2f02..d346d049493 100644 --- a/quest/m2/cluster-routing/routes/interest-set.md +++ b/quest/m2/cluster-routing/routes/interest-set.md @@ -91,8 +91,11 @@ Work: - Update the `verify-route` skill's coverage table, its "Not covered" list, and its sync commit. -The session layer keeps one prefix per neighbour until the route stream -lands, so this changes no behaviour on its own. +The session layer keeps one prefix per neighbour, and `route::Node` keeps +registering each announce stream under its own key, until the route stream +lands. The router here only gains the ability to hold both directions under +one key, which `Sim` and the tests use, so this changes no behaviour on its +own. Public API: none (the router is crate-private). Wire: none. diff --git a/quest/m2/cluster-routing/routes/js.md b/quest/m2/cluster-routing/routes/js.md index aef64e866a7..70f2ffbabc7 100644 --- a/quest/m2/cluster-routing/routes/js.md +++ b/quest/m2/cluster-routing/routes/js.md @@ -49,7 +49,7 @@ the cheapest one; fix it with the route-selection rework here. - The Announce Stream becomes one one-way stream per side with a set of prefixes ([the route stream](/quest/m2/cluster-routing/routes/route-stream.md)). Whichever of the two lands second adapts: JS then shares one stream across - its `announced()` calls, with the covering-prefix logic Rust's `Discovery` + its `routes()` calls, with the covering-prefix logic Rust's `Discovery` has, instead of one stream per call. ## Related diff --git a/quest/m2/cluster-routing/routes/route-stream.md b/quest/m2/cluster-routing/routes/route-stream.md index 379c72b2c8b..d1046676ef9 100644 --- a/quest/m2/cluster-routing/routes/route-stream.md +++ b/quest/m2/cluster-routing/routes/route-stream.md @@ -9,7 +9,7 @@ peer's prefixes, its ROUTE_END echoes, and its route requests. No routing message can overtake another, so production is the route model's one FIFO per direction by construction, and a ROUTE_END echo always arrives behind the routes its sender sent first. A change of token scope edits the prefix set -instead of reopening a request. `announced()` keeps its API in Rust and JS, +instead of reopening a request. `announced()` in Rust and `routes()` in JS keep their APIs, and lite-06 and older and moq-transport are untouched. ## Plan @@ -94,8 +94,8 @@ Code this replaces, for lite-07 only: - `Routed` (`lite/subscriber/routed.rs`) and `Answer` (`lite/publisher/routed.rs`) become one per session, holding the prefixes. `Answer` already reads several node streams (one per mount). -- `route::Node` registers one neighbour per session for both directions. -- `js/net`: each `announced()` call opens its own stream today +- `route::Node` registers one neighbour per session for both directions. Today `receive` and `answer` each take a fresh key (`route/node.rs`), so one session is two neighbours in the router; the interest set makes the router accept one key for both and this makes the sessions use it. +- `js/net`: each `routes()` call opens its own stream today (`js/net/src/lite/subscriber.ts`), so JS gains the covering-prefix logic `Discovery` has. [moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443) ports the route layer with per-stream state; whichever lands second diff --git a/quest/m5/README.md b/quest/m5/README.md index 4395de569af..7886c4fdf1b 100644 --- a/quest/m5/README.md +++ b/quest/m5/README.md @@ -37,3 +37,4 @@ consumer asks. ## Required - [Miri's GenMC mode explores every schedule of kio's lock-free code](/quest/m5/miri-genmc.md) - waiting on rust-lang/miri to run `cargo miri test` in GenMC mode, then Nightly checks kio's log and channel exhaustively +- [Browser tabs forward for their peers on a local network](/quest/m5/browser-mesh.md) - tabs on one LAN form a mesh and forward for each other, on the Rust-to-TypeScript port From 470f8e41e99228658149317ce1453817e5960531 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 23:49:06 -0800 Subject: [PATCH 5/5] quest: hop-list entries are never qualified, and two more places to update Co-Authored-By: Claude --- quest/m2/cluster-routing/routes/echo-order.md | 28 +++++++++++++------ 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/quest/m2/cluster-routing/routes/echo-order.md b/quest/m2/cluster-routing/routes/echo-order.md index ac1615cb427..6a4a16b7018 100644 --- a/quest/m2/cluster-routing/routes/echo-order.md +++ b/quest/m2/cluster-routing/routes/echo-order.md @@ -122,19 +122,31 @@ Work, model first: - Wire, for the implementer to confirm or improve; list each departure in the PR. ROUTE_START and ROUTE_UPDATE gain a count and that many Node IDs, in order from the origin, the sender last and the origin itself left out. - A node checks the list for its own Node ID as received, before it files - anything a scoped peer sent under a qualified Node ID. A list longer than - a fixed limit makes the route unusable; reuse lite-06's hop limit unless + List entries are never qualified: a relay files a scoped peer's origin + Node ID under a qualified one as today, and carries the list's entries on + as received, each a bare Node ID. A relay cannot compute another relay's + qualification, so a qualified entry could never match the node it names. + A scoped peer can then add or drop entries only on routes filed under its + own qualified identities, which harms nothing but those routes, and a lap + through it is still stopped by refusing to requalify a qualified identity + (`scoped_connections_cannot_requalify_a_qualified_identity`). The model + has no scoped grants, so cover this in `Sim` with a route that crosses a + relay, a scoped peer, and the same relay again. A list longer than a + fixed limit makes the route unusable; reuse lite-06's hop limit unless there is a reason not to. A node that forwards declares a Node ID, since it has to name itself. -- Draft: the two messages, Routing (a route that names the receiver, and - that forgetting is safe because of it, not because of the echo alone), - and the lite-07 changelog. `js/net` has no route layer yet, so +- Draft: the two messages, Routing (a route that names the receiver, that + list entries are never qualified, and that forgetting is safe because of + the list, not because of the echo alone), the Node ID Parameter, which + today says an endpoint that originates no routes omits it and must now + say a node that forwards sends it, and the lite-07 changelog. `js/net` has no route layer yet, so [its quest](/quest/m2/cluster-routing/routes/js.md) decodes the list and sends an empty one for its own route. - Update the `verify-route` skill's open finding, coverage, and sync - commit, and the epic's loop-freedom bullet and decision 6, which say the - echo alone makes forgetting safe. + commit, and the three places in the epic that say the echo alone makes + forgetting safe: the loop-freedom bullet, decision 6, and the handoff + contract's bullet on dropping the distance ("as safe as a fresh node + joining"). - [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)'s cache sweep runs on the Rust router and depends on the feasibility check; rerun it with the list if it has landed.