diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 5771104341a..476df24bd4d 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -42,7 +42,7 @@ just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/rout All three pass at the sync commit. Each takes minutes on 4 workers, so run them last. First run what answers in seconds: - `just rs test -p moq-net --lib -E 'test(/route::/)'`. `sweep_no_route_outgrows_a_loop_free_path` delivers every message and echo in a random order on random meshes with unequal costs, flaps, restarts, and collections, and checks the metric bound, loop freedom, and the quiet state. Raise its case count for a wider search: 40,000 cases run in 15 seconds in release. -- TLC's simulation mode on the config in question, which samples deep runs the exhaustive bounds do not reach: add `-simulate num=3000 -depth 80`. +- TLC's simulation mode, which samples runs the exhaustive bounds do not reach. The config's `MaxRev`, `MaxFlaps`, and `MaxCollects` still bound a simulated run, so raise them in a copy of the config first, such as three session changes, two claim changes, and a collection, then add `-simulate num=3000 -depth 150`. Both behind-a-triangle configs fail `SimplePaths` within seconds if `Crossed` is made false, which is the count to infinity the hops fix: a relay drops its distance once its neighbours echoed, and takes back a copy of its own route that a third relay still held. diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 32ce09bb922..4a523af7cd2 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -495,6 +495,7 @@ A metric that overflows 64 bits makes the route unusable. A route carries its Hops: the Hop ID of every node it crossed, the origin node first and the sender last. A Hop ID is a non-zero 64-bit value a node names itself with for as long as its process runs, such as the Node ID it started with; a rotation does not change it. +A receiver tolerates a zero or repeated Hop ID instead of closing the session: such a route harms only itself. A node sending its selected route appends its own Hop ID to the Hops it received, and an origin's own route starts with its Hop ID alone. A relay forwards the other entries unchanged and never qualifies them. A route that arrives with 32 Hops is unusable. @@ -2273,7 +2274,7 @@ The `Message Length` describes the payload size on the wire. - Added announce compression: ANNOUNCE_START gains `Path Base` and `Path Keep` to copy the head of a live claim's suffix. - Split routing into two layers on the Announce Stream. ROUTE_START (0x6), ROUTE_UPDATE (0x7), and ROUTE_END (0x8) advertise how the publisher reaches an origin node, named by a Node ID, with a seqno, an additive metric, and a down-only bit, and loops are prevented by Babel's feasibility condition. Every ANNOUNCE_START names its route's Route ID in place of the Hop ID list, so a link change sends one ROUTE_UPDATE per origin rather than an update per claim. Removed the Hop ID list and ANNOUNCE_OK's Hop ID, and replaced the SETUP Hop parameter (0x5) with a [Node ID parameter](#node-id-parameter), so an endpoint that reaches itself detects it at setup. - ROUTE_START and ROUTE_UPDATE carry Flags, whose 0x2 bit adds an optional RTT: the informational sum of each link's lowest smoothed RTT over the last 30 seconds to the origin node, which never selects a route. -- Added Hops to ROUTE_START and ROUTE_UPDATE: the Hop ID of every node the route crossed, at most 32. A node never selects a route whose Hops name it, so one that dropped its feasibility distance cannot take back a copy of its own route. +- Added Hops to ROUTE_START and ROUTE_UPDATE: the Hop ID of every node the route crossed, at most 32. A node never selects a route whose Hops name it, so one that dropped its feasibility distance cannot take back a copy of its own route. A zero or repeated Hop ID in a route's Hops is tolerated, where a repeated non-zero Hop ID in moq-lite-06's Hop ID list is a PROTOCOL_VIOLATION. - Added ROUTE_HOLD (0xB), which holds a route while it recovers; ROUTE_END lets it go. Neither withdraws the origin's claims, and the subscriber echoes every ROUTE_END back. - Added ROUTE_REQUEST (0xA), sent by the subscriber on the Announce Stream to ask for a route with a newer seqno when every route it has is infeasible. - Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the requested prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows. diff --git a/quest/m1/route-inductive.md b/quest/m1/route-inductive.md index ac1efca1f11..a5390de7085 100644 --- a/quest/m1/route-inductive.md +++ b/quest/m1/route-inductive.md @@ -36,8 +36,9 @@ stay constants the config states. Say both bounds in the `verify-route` skill. replay it on `route::tests::Sim` as the skill says before calling it a bug. - Apalache needs type annotations. Add them to the spec TLC already checks, so both tools read one file and neither drifts. -- `NoCountToInfinity` follows from `SimplePaths`, which is inductive on its - own: every route sent adds its sender to hops that did not name it +- `NoCountToInfinity` follows from `SimplePaths`, which is inductive once + the invariant also says a route is held only from a node the holder shares + a link with: every route sent adds its sender to hops that did not name it ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)). Start there. - Run it through the `tla` recipes with Apalache from the flake. If five relays does not fit Nightly, record the largest mesh that does and its time. @@ -47,6 +48,7 @@ Public API: none. Wire: none. ## Required - [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - the spec parametrised over mesh shapes that this invariant is stated on +- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the rule for whose claims a relay believes, which the claim rules rest on; an invariant found before it is found for a rule about to go ## Related diff --git a/quest/m1/route-trace-validation.md b/quest/m1/route-trace-validation.md index cf16769e271..bd7a7742912 100644 --- a/quest/m1/route-trace-validation.md +++ b/quest/m1/route-trace-validation.md @@ -40,3 +40,7 @@ Related: [the mesh route model](/quest/m1/route-mesh-model.md), which grows the spec these traces are checked against. Public API: none. Wire: none. + +## Required + +- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - replaces who a relay believes and adds two prefixes and a scoped relay, so traces are validated against the rule that stays diff --git a/quest/m2/cache-routing-proof.md b/quest/m2/cache-routing-proof.md index e95de051d54..a864b4ee6e9 100644 --- a/quest/m2/cache-routing-proof.md +++ b/quest/m2/cache-routing-proof.md @@ -27,6 +27,16 @@ it; a SETUP extension needs neither before the cut, and from its list. Either placement is now this quest's to record, with no second trip to the maintainer for choosing lite-07. +Reopened 2026-11-11 (maintainer, /quest-plan on the route redesign, epic +Plan item 8): a warm route propagates, and the rule that keeps it free of +loops and stale copies is +[the prototype](/quest/m2/cluster-routing/routes/stale-copies.md)'s to pick. +The eligibility rule and the fixtures below are for the one-hop warm route +first decided, which [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) +proves. Once the prototype's pick is recorded, rewrite both for it: the +fixtures keep their meshes, and gain a relay that joins a copy at a neighbour +farther from the publisher than itself. + Build on the sans-IO `Router` in `rs/moq-net/src/route/`, which each origin's `route::Node` drives, and the harness behind its handoff sweeps, not the reference model in @@ -70,6 +80,10 @@ Route ID, and the instance, and say which stream carries it and how a peer that never sends one behaves. Size it against lite-07 and an additive SETUP extension and record the choice and its reason in the epic's Plan. +## Required + +- [A prototype settles what keeps a route free of stale copies](/quest/m2/cluster-routing/routes/stale-copies.md) - the rule a propagating warm route rests on, which replaces the one this proved + ## Related - [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the feasibility rules and handoff sweeps this extends diff --git a/quest/m2/cluster-routing/README.md b/quest/m2/cluster-routing/README.md index d6e3fb3bd68..8704e22e123 100644 --- a/quest/m2/cluster-routing/README.md +++ b/quest/m2/cluster-routing/README.md @@ -37,6 +37,18 @@ Decided with the maintainer, 2026-10-13: - Keep Babel. A detour longer than the lost route may wait for a sequence refresh from the origin; instant failover is not required. Measure failure detection, additional routing recovery, and total interruption separately. + Reopened 2026-11-11 (maintainer, /quest-plan after + [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews): + whether Babel's distance stays is + [the prototype](/quest/m2/cluster-routing/routes/stale-copies.md)'s to + measure, against paths alone with a withdrawal that names the hop that + died. The distance refuses a longer way round until the publisher advances + its seqno, and refuses a cached copy at a neighbour no closer to the + publisher, which [cache-aware routing](/quest/m3/cache-routing/README.md) + wants to join. Decided the same day: a route is shared per publisher and + per scope, and a relay believes a route by that route's own path, whichever + rule wins. Rejected: link-state among relays, and routes to relays in + place of publishers. - An origin declares its own routing identity, fresh per process or tab restart and shared across its live sessions. It is unverified. Decided 2026-11-02 (maintainer, /quest-unblock): no key proof is planned; if a use case needs a provable identity, moq-lite-07's diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index 52cf1df74f5..63f05aea260 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -40,9 +40,11 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 22. ✅ **A side's interest is a set of prefixes edited in place.** The live set never nests, and one edit may swap prefixes for a wider or narrower one without re-sending what both cover ([the interest set](/quest/m2/cluster-routing/routes/interest-set.md)). A token-scope change is such an edit, with one prefix per region of the scope. This supersedes items 16 and 17, and the handoff contract's "An interest change opens a new stream" below, once the route stream lands. 23. ✅ Asked what a floor covers with several prefixes on one stream: **one prefix, named by its id.** This amends item 10: the floor is still a marker and never a region list. Rejected: a floor over the whole set. 24. ✅ Asked how a side learns a newly added prefix's initial set is complete: **SYNCED per prefix, naming its id**, plus one marker per edit where it takes effect for both sides. Rejected: one SYNCED per edit, counted in order. -25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes. +25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region, unless it changes whether the node may use the route: by crossing it, or by reaching the hop limit. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes. +26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **a route is shared per publisher and per scope, and a relay believes a route by that route's own path** ([the prototype](/quest/m2/cluster-routing/routes/stale-copies.md), [the model](/quest/m2/cluster-routing/routes/scope-model.md), then [the router](/quest/m2/cluster-routing/routes/scope.md)). A scope is the prefix a relay asked a neighbour for when its grant there is narrower than its interest. Today only a feasible neighbour counts, one strictly cheaper than the distance the relay remembers, so a relay's claims move whenever a cost or seqno does: it can forget a claim for good, a mesh with two grants never settles after one seqno advance, and behind a scoped neighbour it never shows what a farther one knows. Only a broadcast some relay carries gets its own route, and that route propagates ([cache-aware routing](/quest/m3/cache-routing/README.md) Plan item 8). Open, for the prototype to measure and the maintainer to pick: whether Babel's distance or the path alone, with a withdrawal that names the hop that died, keeps a route free of loops and stale copies. The contract's bullet on which neighbours certify stands until the model quest rewrites it. First decided the same day as any neighbour whose path to the publisher does not pass through the relay; tried on the router, two relays behind a scoped relay then keep a dead claim alive through each other, and the relay behind a scoped neighbour shows a claim it has no route to request. Rejected: that rule; believing a neighbour that is cheaper than the relay is now, which fixed the storm and the forgotten claim with no wire change but rests belief on distance; a path per broadcast as lite-06 has; link-state among relays; routes to relays. + The route layer's core landed in [moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676): a crate-private, sans-IO `Router` in `rs/moq-net/src/route/`, generic over the diff --git a/quest/m2/cluster-routing/routes/handoff-revival.md b/quest/m2/cluster-routing/routes/handoff-revival.md new file mode 100644 index 00000000000..04cbe0ff62a --- /dev/null +++ b/quest/m2/cluster-routing/routes/handoff-revival.md @@ -0,0 +1,37 @@ +# [M] The handoff sweep revives no ended claim + +## Goal + +`sweep_handoff` in `rs/moq-net/src/route/tests/handoff.rs` reports no revival +over 20,000 scenarios: no reader sees one life of a claim start again after +it saw that life end, unless it forgot the claim in between. A publisher that +withdraws a claim and announces it again starts a new life, which is no +revival. The scenario that revives one today is a plain test. + +## Plan + +Found 2026-11-11 by running the sweep past its 150 scenarios: scenario 16316 +revives a claim, on `main` at `688312b9eb`, and no earlier one does. It is +the same whichever neighbours the router believes. It has not been traced. +Decided the same day (maintainer, /quest-plan): ✅ plan it. + +- Reproduce it: the sweep draws every scenario from one random stream, so run + the first 16,317. Record the scenario's steps as it runs and remove them + one at a time while the revival remains, as + [the false END](/quest/m2/cluster-routing/routes/widen-false-end.md) was + shrunk. +- Decide whether it is a revival the contract forbids or one it excuses: a + replay after a collection or a restart is allowed only until the + withdrawal behind it lands + ([epic](/quest/m2/cluster-routing/routes/README.md), handoff contract). If + the fixture's `revivals` count misses an excuse, fix the count and say so. +- Otherwise fix the cause and keep the shrunk case as a test. +- If the cause is in what + [the route per scope](/quest/m2/cluster-routing/routes/scope.md) deletes, + pin it as a `residual_` test naming that quest instead of fixing it twice. + +Public API: none. Wire: none. + +## Related + +- [Widening an interest ends no live claim](/quest/m2/cluster-routing/routes/widen-false-end.md) - found the same way, and adds the way to run more scenarios diff --git a/quest/m2/cluster-routing/routes/interest-set.md b/quest/m2/cluster-routing/routes/interest-set.md index ba70d6921fe..fd471af3227 100644 --- a/quest/m2/cluster-routing/routes/interest-set.md +++ b/quest/m2/cluster-routing/routes/interest-set.md @@ -71,10 +71,13 @@ Work: ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)), and move the echo from its own queue onto the neighbour's FIFO, as that PR's first commit (`81c0698295`) did: with one stream per side nothing overtakes anything. -- Give the model at least two prefixes and edits in flight: add, remove, - widen, narrow, and an edit that crosses a claim change, a route change, a - ROUTE_END, and its echo. The model fixes interest at the root today, so - this is new state; keep it tractable by bounding edits as it bounds flaps. +- The model has two prefixes and a scoped relay since + [it has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md). + Give it edits in flight: add, remove, widen, narrow, and an edit that + crosses a claim change, a route change, a ROUTE_END, and its echo. Keep it + tractable by bounding edits as it bounds flaps. Where Settled item 1 says + the merge rule is unchanged, read it as that quest's rule: a floor names + its route's prefix, and the newest floor wins among one publisher's routes. - In the router, `Peer::asked` and `Peer::watch` become sets with a floor and a synced flag per prefix, `batch` takes the prefix its floor names, and a neighbour is one key for both directions, as `Sim` already has it. The @@ -97,3 +100,12 @@ one key, which `Sim` and the tests use, so this changes no behaviour on its own. Public API: none (the router is crate-private). Wire: none. + +## Required + +- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the two prefixes and the rule for whose claims a relay believes, which an edit must preserve + +## Related + +- [The router keeps a route per scope](/quest/m2/cluster-routing/routes/scope.md) - rewrites how the router keeps a floor per prefix; whichever lands second adapts +- [Widening an interest ends no live claim](/quest/m2/cluster-routing/routes/widen-false-end.md) - a false END on an interest change today, which the edit in place may remove diff --git a/quest/m2/cluster-routing/routes/scope-model.md b/quest/m2/cluster-routing/routes/scope-model.md new file mode 100644 index 00000000000..31f70e0165e --- /dev/null +++ b/quest/m2/cluster-routing/routes/scope-model.md @@ -0,0 +1,99 @@ +# [L] The route model has a route per scope + +## Goal + +`rs/moq-net/verify/route/RouteHandoff.tla` gains two prefixes and relays that +ask a neighbour for only one of them. A route is keyed by its publisher and +the prefix it covers, carries its own path, and is believed by a relay it did +not cross. TLC shows on it: + +- A relay shows a claim exactly while it holds a route that covers the claim + and did not cross the relay, and the newest revision the relay knows still + has the claim. It shows the claim as it is at that revision. +- Once nothing is in flight, no relay shows a claim that no path from its + publisher carries. +- A change of path or seqno alone starts, ends, and changes no claim a reader + sees. Whether the claim is available may change. +- A request for a shown claim follows next hops to the publisher without a + loop, also where one route's prefix lies inside another's. So a relay + behind a scoped neighbour reaches what a farther, unscoped neighbour + offers. + +The model uses the rule +[the prototype](/quest/m2/cluster-routing/routes/stale-copies.md) picked for +keeping a route free of stale copies. The rule this settles is written into +the epic's contract for +[the router](/quest/m2/cluster-routing/routes/scope.md) to implement. The +router's behaviour, the wire, and the draft do not change here. + +## Plan + +Decided 2026-11-11 (maintainer, /quest-plan). The decisions and what was +tried are in [the prototype](/quest/m2/cluster-routing/routes/stale-copies.md)'s +Plan and the [epic](/quest/m2/cluster-routing/routes/README.md)'s Plan item +26. + +Why a route per scope: a relay that asked a neighbour for prefix `a` knows a +publisher's claims only inside `a`. Today that partial knowledge lives inside +one view per publisher, as a floor per region, with claims no floor covers +carried loose, and a rule for which neighbours vouch for which region. Every +open finding is in that machinery. With the prefix on the route, every +neighbour that offers a route knows all of it, and the route ends when its +path does. + +Recommendations the model may overturn. Say in the PR what it settled on and +why: + +- **The prefix a route covers travels with the route**, as a field of + ROUTE_START. On a stream that asked for a narrower prefix the route covers + that prefix, so narrowing composes. Rejected so far: deriving a Node ID + from the publisher and the prefix, which needs no wire change but cannot + be narrowed a second time, since a relay cannot read the prefix back. +- **The newest floor wins among one publisher's routes.** A claim both a + narrow and a wide route cover is shown once, at the newer revision, and a + relay keeps what it learned while any route covers the claim + (epic Plan item 19). Nothing is pruned by an instant: a route that ends + takes only what no other route covers. +- **A request takes the narrowest route that covers its claim, and is never + forwarded along a route whose path names the neighbour it came from.** + This is the unsolved part: prove the selected next hops cannot loop when + the narrow and the wide route lead different ways, under stale messages, + or find the rule that does. +- Keep the counterexample that broke the first plan: two relays behind a + scoped relay must not keep a wider route alive through each other after + its path closed. +- Keep the lagging neighbour: a relay learns a withdrawal from one route, + that route ends, and a second route still carries the claim at the older + revision. The relay must not show the claim again + (`an_older_complete_snapshot_revives_no_known_withdrawal`). Give it a + config, with the second route covering the same prefix and a wider one. + +Work: + +- In the model, replace the fixed root interest with two prefixes and a + `Scoped` set of relays that ask for one. Key `rx`, the selection, and what + is sent by route. Replace `Certifiers`, `Certified`, `Prune`, and the + stored merged `view` with one floor per route and the newest-floor rule. + Add the Goal's four rules: `Seen`, an extended `QuietIsSettled`, `Stable`, + and `Reaches`. +- Add configs for the dead claim, the blind relay, and the meshes in the two + reproducer comments the prototype pins, at bounds that finish in minutes on + the recipe's settings. Run TLC one config at a time, simulation first. +- Rewrite the epic's bullet that begins "A node tracks the regions each + neighbour's last view certifies" to the settled rule, marked as not yet + built. +- Update the `verify-route` skill's rules table, coverage table, and + "Not covered" list. + +[The interest set](/quest/m2/cluster-routing/routes/interest-set.md) adds +edits to the two prefixes this introduces. + +Public API: none. Wire: none. + +## Required + +- [A prototype settles what keeps a route free of stale copies](/quest/m2/cluster-routing/routes/stale-copies.md) - the rule for loops and stale copies that this model is written on + +## Related + +- [The router keeps a route per scope](/quest/m2/cluster-routing/routes/scope.md) - implements the rule this settles diff --git a/quest/m2/cluster-routing/routes/scope.md b/quest/m2/cluster-routing/routes/scope.md new file mode 100644 index 00000000000..97280cf01a3 --- /dev/null +++ b/quest/m2/cluster-routing/routes/scope.md @@ -0,0 +1,65 @@ +# [XL] The router keeps a route per scope + +## Goal + +`route::Router` keys a route by its publisher and the prefix it covers, and +believes a route that did not cross it, as +[the model](/quest/m2/cluster-routing/routes/scope-model.md) settled. A relay +shows a claim exactly while such a route covers it and the newest revision +the relay knows still has the claim, two relays never keep a +dead route alive through each other, a change of path or seqno alone changes +no claim a reader sees, and a relay behind a scoped neighbour reaches what a +farther one offers. The `residual_` tests +[the prototype](/quest/m2/cluster-routing/routes/stale-copies.md) pinned pass +as plain tests, a sweep over meshes with scoped relays holds the rules at +every quiet point, and the draft and the epic's contract say what the router +does. + +## Plan + +Decided 2026-11-11 (maintainer, /quest-plan): the prototype picks the rule +for stale copies, the model settles the route per scope, and the router comes +last (rejected: the router first, patched case by case; recording the +findings as known limits and waiting for federation). + +Work: + +- Key `rx`, the selection, `sent`, and the views in `route/mod.rs` by route. + A view then has one floor, so delete `certifiers`, `certified`, `prune`, + `loose`, and the per-region floors, and keep the newest floor among one + publisher's routes as the model has it. Keep the reader's view a function + of what the router holds, computed incrementally the way `show` is today, + with the test build comparing it against the whole computation. +- Remove what #8487 added to work around the old rule: `recompute`'s `prune` + switch, which skips pruning on a change of hops or RTT alone. Keep the + qualified-key test in `infeasible` unless the new rule makes it moot, and + say which. +- Serve a request through the route the model's rule picks, in + `route/node.rs`'s `home`, in place of every feasible route whose neighbour + `carries` the claim. +- Flip each `residual_` test, and add the review harness's scoped and + two-prefix families as a sweep beside + `sweep_no_route_outgrows_a_loop_free_path`, checking at every quiet point + that each relay shows what the rule says and nothing waits. Compare a few + thousand scripts against the router before this change and explain every + difference in the PR. +- Wire (lite-07-wip): whatever the model settled for naming a route's + prefix. List it under Impact and run `just test interop --all`. +- Draft: Routing says a route covers a prefix, and Floors says a floor + certifies its route's prefix. Replace the Floors condition "every peer + whose last advertised route to it is feasible here has answered the + current request" with the settled rule. The hold rule in Routing is not + about belief and stays unless the prototype's pick removes holds. Add a + changelog bullet. +- If [js/net's route layer](/quest/m2/cluster-routing/routes/js.md) has + landed, change it in the same PR. +- Rewrite the epic's bullet on who certifies as built, and update the + `verify-route` skill's coverage and sync commit. +- Measure route and claim messages per change with + `rs/moq-net/tests/integration/mesh_sweep.rs`'s benchmark before and after. + +Public API: none (the router is crate-private). Wire: lite-07-wip. + +## Required + +- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the rule, proven on two prefixes and a scoped relay diff --git a/quest/m2/cluster-routing/routes/stale-copies.md b/quest/m2/cluster-routing/routes/stale-copies.md new file mode 100644 index 00000000000..d3c6356fea2 --- /dev/null +++ b/quest/m2/cluster-routing/routes/stale-copies.md @@ -0,0 +1,167 @@ +# [L] A prototype settles what keeps a route free of stale copies + +## Goal + +A test-only router beside `route::tests::Sim` runs lite-07 routing on paths +alone: a relay uses any route whose path does not name it, and a withdrawal +names the hop that died, so every route through that hop goes at once. The +same seeded meshes and scripts run on it and on today's router, which keeps +Babel's distance. The PR reports for both: + +- whether selected routes ever loop, and for how many steps; +- for how many steps a relay still shows a route whose path is gone, after a + link closes, a relay dies, and a publisher leaves; +- how many route messages each of those changes costs on 3, 6, 12, and 24 + relays; +- how many steps a relay with a longer way round takes to use it; +- whether a relay can pull a broadcast from a neighbour that already carries + it and is no closer to the publisher than the relay. + +The PR recommends one of the two rules, and the maintainer's pick is recorded +in the [epic](/quest/m2/cluster-routing/routes/README.md)'s Plan. Each bug the +reviews of [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487) found +is pinned as a `residual_` test on today's router. The production router, the +wire, and the draft do not change here. + +## Plan + +Decided 2026-11-11 (maintainer, /quest-plan after #8487's two adversarial +reviews, revised the same day once the first plan's rule was tried on the +router). + +Why: today a relay believes a neighbour's copy of a publisher's claims only +while that neighbour is "feasible", which means strictly cheaper than the +lowest cost the relay ever advertised (Babel's distance). That set moves +whenever a cost or a seqno moves, and the relay's claims move with it. The +reviews found no loop and no stuck relay in route selection, and no remaining +bug on a mesh where every relay may see everything. Every finding needs +neighbours that know different parts of a publisher's claims, which happens +when a relay asks a neighbour for less than it wants because its grant on +that session is narrower. Call the prefix it asked for the **scope**. + +What was tried, each as a few lines behind a switch on `main` at `688312b9eb`: + +| | Today | Believe any neighbour whose route did not cross the relay | Believe a neighbour cheaper than the relay is now, ignoring the seqno, and keep what it sent | +| --- | --- | --- | --- | +| The seqno storm mesh settles | no | no | yes | +| A returning neighbour brings its claims back | yes | yes | yes | +| A seqno advance changes nothing shown | yes | yes | yes | +| Two relays behind a scoped relay let a dead claim go | yes | no, kept for good | yes | +| The blind relay shows a farther neighbour's claim | no | yes, as available, with no route to request it | no | +| The rest of `moq-net` | pass | 1 fails | 1 fails | + +On lite-06 (`release`), over real sessions, the blind relay sees and resolves +the far claim, and the two relays retract the dead one once. lite-06 carries +a path on every announcement, so believing by path is sound there. lite-07 +has one path per publisher, which cannot vouch for claims that reached the +sender another way. + +Settled: + +1. ✅ **A route is shared per publisher and per scope, and a relay believes a + route by that route's own path.** Every claim on a route is then known + completely for the route's prefix, and came along the path the route + names. Where every relay may see everything this is one route per + publisher, as today. + [The model](/quest/m2/cluster-routing/routes/scope-model.md) and + [the router](/quest/m2/cluster-routing/routes/scope.md) build it. Rejected: + believing by the publisher's one path, the first plan, which the table + shows is unsound. Rejected: believing by cost alone, the third column, + which fixed the storm in 25 lines with no wire change; the maintainer does + not want belief to rest on distance, and the blind relay stays blind. + Rejected: a path per broadcast as lite-06 has, which re-announces every + broadcast when a path changes. +2. ✅ **The blind relay is fixed by routing, not by showing more.** A relay + whose closest neighbour is scoped does not show what a farther neighbour + knows, and today that is consistent: it believes exactly the neighbours it + can forward a request to, and the farther one is not feasible. With a + route per scope the wider route is its own destination, so the relay can + both show the claim and reach it. +3. ✅ **Only a broadcast some relay carries gets its own route, and that + route propagates.** This reopens + [cache-aware routing](/quest/m3/cache-routing/README.md)'s one-hop warm + route (its Plan item 8). +4. ✅ **What keeps a route free of loops and stale copies is this quest's to + measure, not yet decided.** The hops stop loops, not stale copies. + Two candidates: + - **Distance, as today.** A relay uses a neighbour only when the + neighbour's cost is below the lowest cost the relay ever advertised. + Anything routed through the relay costs more than that, so such a + neighbour's route is no copy of the relay's own. It can still be stale + another way: a neighbour whose path closed stays usable until its + withdrawal arrives. It never loops, even for a moment. It refuses + good routes: a longer way round waits for the publisher to advance its + seqno, and a relay cannot join a copy carried by a neighbour that is no + closer to the publisher, which is the join cache-aware routing wants. + The seqno, ROUTE_REQUEST, ROUTE_HOLD, and the ROUTE_END echo all exist + to recover what this check refuses. + - **Paths alone, with a withdrawal that names what died.** A relay uses + any route whose path does not name it. Each hop in a path names the + relay and the session it learned the route on. When that session + closes, the relay's withdrawal names that hop, and every relay drops + every route through it at once, whichever neighbour offered it. This is + root cause notification from BGP research + ([Pei et al. 2005](https://www.cs.colostate.edu/~massey/pubs/journal/massey_comnet05.pdf)), + which bounds convergence by the mesh's diameter. Its known gap: when a + relay with several neighbours dies, each neighbour reports its own + session, so one stale switch per neighbour can remain. A route can also + loop for a moment while a path change is in flight. It needs no seqno, + request, hold, or echo. lite-06 instead holds stale paths back with a + 300 ms hold and a 2 s quarantine, and + [moq-dev/moq#5375](https://github.com/moq-dev/moq/pull/5375) still + measured stale revivals. + + The October measurement that chose Babel compared it with a path per + broadcast, not with shared routes, so its message counts do not answer + this. +5. ✅ **No routes to relays.** A relay keeps the cheapest path to each + publisher's scope, not to a relay that publisher is attached to. + +Work: + +- Write the paths-alone router as test-only code under `route/tests/`, as + small as it can be: routes shared per publisher, each with a path and a + cost; the withdrawal that names a hop; and claims that ride a route with + one floor, newest floor winning. Leave out what the comparison does not + need: scoped grants, qualified keys, interest changes, GOAWAY, and RTT. +- A relay remembers a dead hop only while one of its neighbours still offers + a route through it. Report whether a stale route can return after that, + and for how long. +- Drive both routers from one script generator, the one behind + `sweep_no_route_outgrows_a_loop_free_path`, and from the mesh shapes of + `rs/moq-net/tests/integration/mesh_sweep.rs`'s benchmark. Check at every + step and at every quiet point, and count as the Goal lists. +- Report what a relay's reader sees when its only route goes. Today it holds + the claims as unavailable while a seqno request is out + ([epic](/quest/m2/cluster-routing/routes/README.md) Plan item 19); on + paths alone there is nothing to wait for, so it lets go at once. +- Build the cached-copy case for both: a relay whose neighbour carries a + broadcast and is farther from the publisher. Two relays that each carry a + copy must not adopt each other, and a copy that goes must take every route + through it along. +- Pin each finding as a `residual_` test in `route/tests/`, asserting what + today's router does, with a comment naming + [the router quest](/quest/m2/cluster-routing/routes/scope.md): the seqno + storm ([reproducer](https://github.com/moq-dev/moq/pull/8487#issuecomment-6798722861)), + the forgotten claim and the two tests its failed fix broke + ([reproducers](https://github.com/moq-dev/moq/pull/8487#issuecomment-6802083874)), + and the blind relay: a publisher behind a relay that asked for `a` and + behind a two-relay chain that asked for everything, both joined to one + relay, which shows `a/x` and not `b/x`. +- Add the dead claim as a plain test, since today's router passes it and any + new rule must: a relay whose only way to the publisher asked for `a`, with + two relays joined to it and to each other, one of which also has a dear + link straight to the publisher. That link closes and the publisher + withdraws `b/x`; neither relay may still show it. +- Record the maintainer's pick in the epic's Plan item 26 and in + [Cluster routing](/quest/m2/cluster-routing/README.md)'s "Keep Babel" + bullet. If the pick is paths alone, plan removing the distance, the seqno, + ROUTE_REQUEST, ROUTE_HOLD, and the echo with `/quest-plan` before the + router quest starts. + +Public API: none. Wire: none. + +## Related + +- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - models the rule this picks +- [Cache-aware routing](/quest/m3/cache-routing/README.md) - its warm routes rest on the rule this picks diff --git a/quest/m2/cluster-routing/routes/widen-false-end.md b/quest/m2/cluster-routing/routes/widen-false-end.md new file mode 100644 index 00000000000..f02f902167a --- /dev/null +++ b/quest/m2/cluster-routing/routes/widen-false-end.md @@ -0,0 +1,93 @@ +# [M] Widening an interest ends no live claim + +## Goal + +A client that widens its interest never sees END for a claim its publisher +still has. `sweep_scoped_handoff` in `rs/moq-net/src/route/tests/handoff.rs` +reports no false END over 20,000 scenarios, and the case below is a plain +test. + +## Plan + +Found 2026-11-11 by running the sweep past its 150 scenarios: 25 of 20,000 +end a live claim, the first at scenario 750, on `main` at `688312b9eb`. It is +the same whichever neighbours the router believes, so +[the route per scope](/quest/m2/cluster-routing/routes/scope.md) does not fix +it. Decided the same day (maintainer, /quest-plan): ✅ plan it (rejected: +leaving it to the interest set). + +Scenario 750, shrunk to 13 steps. The client sits behind two relays, each +under a grant for one prefix, and sees the publisher's `live` at whichever +prefix its interest and a grant meet. It narrows to `live/b/z`, then widens +to everything while its sessions and the publisher's flap. Its reader sees +`live/b/z` start, go offline, start, and END, where the claim only moved up +to `live/b`: + +```rust +let (origin, r1, r2, client) = (0, 1, 2, 3); +let mut sim = Sim::new(4); +sim.leaf(origin); +sim.leaf(client); +sim.join(origin, r1, 1); +sim.join(origin, r2, 1); +sim.join(r1, r2, 1); +for (relay, scope) in [(r1, "live/a/"), (r2, "live/b/")] { + let grant = Grant { subject: "local", scope }; + sim.link(client, relay, SimLink::cost(1).grant(grant), SimLink::cost(1)); +} +sim.originate(origin, 7, 0); +assert!(sim.run()); + +sim.announce(origin, "live/", 0, None); +sim.settle(1); +sim.deliver(r1, origin); +sim.settle(1); +sim.disconnect(client, r1); +sim.settle(1); +sim.disconnect(origin, r1); +sim.settle(1); +sim.deliver(r2, r1); +sim.settle(1); +sim.disconnect(origin, r2); +sim.settle(1); +sim.connect(origin, r1); +sim.settle(1); +sim.watch(client, "live/b/z"); +sim.settle(1); +sim.disconnect(origin, r1); +sim.settle(1); +sim.announce(origin, "live/", 1, None); +sim.settle(1); +sim.connect(origin, r2); +sim.settle(1); +sim.watch(client, ""); +sim.settle(1); +sim.disconnect(client, r2); +sim.settle(1); + +sim.connect(origin, r1); +sim.connect(client, r1); +sim.connect(client, r2); +assert!(sim.run()); +assert_eq!(false_ends(&sim.events), 0); +``` + +`show` in `route/mod.rs` reports a claim that left as offline, not ended, +when a claim above it with the same item is still shown. Here that check +missed. Find why before changing it: the order the widened stream's first +batch and the session close arrive in is the likely cause. + +- Shrink further if it helps, then fix the cause and keep the case as a + test. +- Give the sweep a way to run more scenarios than the gate's 150 without + editing it, and say in the PR how many it ran clean. +- [The interest set](/quest/m2/cluster-routing/routes/interest-set.md) + replaces "an interest change opens a new stream" with an edit in place. If + it lands first, run the case and the 20,000 scenarios on it, and delete + this quest if both are clean. + +Public API: none. Wire: none. + +## Related + +- [The route model and router take an interest as an edited set of prefixes](/quest/m2/cluster-routing/routes/interest-set.md) - rewrites interest changes, and may remove the case diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index 66f04a670f2..c8baa12eebf 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -29,7 +29,8 @@ Decided 2026-11-08 (maintainer, /quest-plan): advertise a warm route for it. This revises Cluster routing's non-goal of cache-dependent route prices. (rejected: per-broadcast forwarding with a cache-only subscribe and no advertisement; keeping routing cache-agnostic) -2. ✅ **A warm route is per neighbor and travels one hop.** It goes only to +2. ✅ **A warm route is per neighbor and travels one hop.** Reopened by + item 8. It goes only to direct neighbors and is never forwarded. B is warm toward N only while B has demand besides N's (a local reader or another downstream session), and withdraws once that demand has been gone for the copy's existing @@ -48,7 +49,8 @@ Decided 2026-11-08 (maintainer, /quest-plan): the skipped relay pulls for its own reader, its neighbor joins it. With uniform prices both cases tie and fall to the hash. (rejected: a warm and a cold price per link; a warm price the carrier seeds) -4. **Eligibility, which the proof must hold.** A relay may pull broadcast P +4. **Eligibility, which the proof must hold.** Reopened by item 8. A relay + may pull broadcast P through neighbor N only when N's advertised route to the request's pinned origin has a newer seqno, or an equal seqno with `(cold metric, hash(P, N))` below `(feasibility distance, hash(P, self))`. @@ -74,6 +76,22 @@ Decided 2026-11-08 (maintainer, /quest-plan): accepting a SETUP extension). The implementation stays in m3. This replaces item 6's rule that choosing lite-07 went back to the maintainer. +8. Reopened 2026-11-11 (maintainer, /quest-plan on the route redesign): + items 2 and 4. In the maintainer's words: "I think we need warm routes + propagating, that decision was made based on this flawed setup." ✅ **A + warm route is per broadcast and propagates**, for as long as it is + cheaper than the route its publisher's broadcasts share + ([epic](/quest/m2/cluster-routing/routes/README.md) Plan item 26). A + broadcast no relay carries rides the shared route and costs no message of + its own. Item 4 rests safety on Babel's distance, which cannot let a relay + join a copy at a neighbour that is no closer to the publisher: the join + the Goal wants, and why item 2 stopped at one hop. What keeps a warm route + free of loops and stale copies is + [the prototype](/quest/m2/cluster-routing/routes/stale-copies.md)'s to + measure, and the proof then holds that rule. Items 1 and 3 stand. + [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) proves items + 2 and 4 as first decided. + Prior art: warm/cold route prices ([moq-dev/moq#2925](https://github.com/moq-dev/moq/pull/2925)) were deleted by prefix routes ([moq-dev/moq#3225](https://github.com/moq-dev/moq/pull/3225)), diff --git a/quest/m3/route-keep-next-hop.md b/quest/m3/route-keep-next-hop.md new file mode 100644 index 00000000000..2c5c3120709 --- /dev/null +++ b/quest/m3/route-keep-next-hop.md @@ -0,0 +1,52 @@ +# [S] A relay keeps its next hop among equal routes + +## Goal + +A lite-07 relay that already routes to an origin through one neighbour stays +on it when another neighbour offers a route of the same rank: the same drain +state, priority, and metric. Today the lower key wins, so a relay moves for +nothing, and since routes name the nodes they crossed +([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)) every relay +downstream is sent a ROUTE_UPDATE for it. A test brings a link up beside an +equal one and counts the route messages sent on every other session: none. +The new link's own first advertisements do not count. + +## Plan + +Decided 2026-11-11 (maintainer, /quest-plan after #8487): ✅ plan it, in m3 +beside [split horizon](/quest/m3/route-split-horizon.md), since it trims +traffic and fixes nothing (rejected: dropping it). + +What it costs today, from #8487's measurements: a change of hops alone is +3.5% to 11% of route messages in the router's churn sweeps, about 3% of +control messages over real sessions, and on a 32-relay full mesh one link +coming up costs 592 route messages against 251 before the hops. + +[Cluster routing](/quest/m2/cluster-routing/README.md) already decided that +"a new connection alone does not outrank a different equal-cost route", so +this implements a settled rule. Keep the deterministic path hash among equal +routes for a request's first hop: that choice is per request and is not this +selection. + +- `selected` in `route/mod.rs` is a function of what the router holds, and + ranks by key last. Staying put makes it depend on the previous choice, so + keep the current next hop per identity and prefer it among equals. Mirror + it in `RouteHandoff.tla`, whose `Best` picks the lowest key, as the + `verify-route` skill says. +- Two routes of the same rank can still differ in whether they are + down-only, which decides whether an upstream neighbour is sent the route + at all. Stay put only among routes that agree on it, so a relay on a + down-only route still moves to an equal one it may export, and test that. +- Check that every router test that names the next hop it expects still + means what it says, rather than editing expectations to match. +- [The prototype](/quest/m2/cluster-routing/routes/stale-copies.md) may + replace how a route is selected. If it has, apply this to the rule that + stands. +- Measure with `Sim`'s `hop_updates` count and, if it has landed, + `mesh_sweep.rs`'s benchmark, before and after. + +Public API: none. Wire: none; behaviour only. + +## Related + +- [Relays split horizon per session](/quest/m3/route-split-horizon.md) - the other trim of route traffic diff --git a/quest/m3/route-split-horizon.md b/quest/m3/route-split-horizon.md index 6a1978ba6f6..e2199c49e65 100644 --- a/quest/m3/route-split-horizon.md +++ b/quest/m3/route-split-horizon.md @@ -18,3 +18,7 @@ Suppress the advertisement per session rather than per neighbor, and measure route messages between two relays before and after. Public API: none. Wire: none; behavior only. + +## Related + +- [A relay keeps its next hop among equal routes](/quest/m3/route-keep-next-hop.md) - the other trim of route traffic diff --git a/rs/moq-net/src/route/tests/routing.rs b/rs/moq-net/src/route/tests/routing.rs index 51a770b03e0..05be36f9dbb 100644 --- a/rs/moq-net/src/route/tests/routing.rs +++ b/rs/moq-net/src/route/tests/routing.rs @@ -1184,6 +1184,11 @@ fn a_scoped_copy_that_crossed_a_relay_asks_for_no_seqno() { let key = qualified(LOCAL.subject, 7); assert_eq!(next(&sim, cdn, key), Some(far)); assert_eq!(next(&sim, near, key), Some(cdn)); + assert_eq!( + sim.shows(near, key, "local/x"), + Some(true), + "the far relay's copy is a route" + ); sim.release(near, cdn); assert!(sim.run()); @@ -1214,6 +1219,7 @@ fn a_scoped_copy_that_crossed_a_relay_is_not_waited_on() { let key = qualified(LOCAL.subject, 7); assert_eq!(next(&sim, cdn, key), Some(far)); assert_eq!(sim.route(near, key).map(|route| route.metric), Some(7)); + assert_eq!(sim.shows(near, key, "local/x"), Some(true)); sim.disconnect(far, cdn); assert!(sim.run());