From ccccfcbd89e87a82ba433b750966c02637c9911e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 10:43:52 -0800 Subject: [PATCH 1/4] quest: plan certifying claims by path, and a relay keeping its next hop among equals Two quests in the routes epic replace how a relay decides whose copy of a publisher's claims to believe: the rule in the route model first, then the router. A third, in m3, stops a relay moving between equal routes. The interest set and two m1 model quests wait on the model. Also answers the review notes that reached #8487 as it was queued: the skill's simulation advice, the draft on a zero or repeated Hop ID, one epic sentence, the inductive invariant's missing premise, and a positive check under a qualified key. Co-Authored-By: Claude --- .claude/skills/verify-route/SKILL.md | 2 +- drafts/draft-lcurley-moq-lite.md | 1 + quest/m1/route-inductive.md | 6 +- quest/m1/route-trace-validation.md | 4 + quest/m2/cluster-routing/README.md | 7 + quest/m2/cluster-routing/routes/README.md | 4 +- .../cluster-routing/routes/certify-model.md | 124 ++++++++++++++++++ quest/m2/cluster-routing/routes/certify.md | 57 ++++++++ .../m2/cluster-routing/routes/interest-set.md | 18 ++- quest/m3/route-keep-next-hop.md | 45 +++++++ quest/m3/route-split-horizon.md | 4 + rs/moq-net/src/route/tests/routing.rs | 6 + 12 files changed, 270 insertions(+), 8 deletions(-) create mode 100644 quest/m2/cluster-routing/routes/certify-model.md create mode 100644 quest/m2/cluster-routing/routes/certify.md create mode 100644 quest/m3/route-keep-next-hop.md diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 5771104341a..476df24bd4d 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -42,7 +42,7 @@ just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/rout All three pass at the sync commit. Each takes minutes on 4 workers, so run them last. First run what answers in seconds: - `just rs test -p moq-net --lib -E 'test(/route::/)'`. `sweep_no_route_outgrows_a_loop_free_path` delivers every message and echo in a random order on random meshes with unequal costs, flaps, restarts, and collections, and checks the metric bound, loop freedom, and the quiet state. Raise its case count for a wider search: 40,000 cases run in 15 seconds in release. -- TLC's simulation mode on the config in question, which samples deep runs the exhaustive bounds do not reach: add `-simulate num=3000 -depth 80`. +- TLC's simulation mode, which samples runs the exhaustive bounds do not reach. The config's `MaxRev`, `MaxFlaps`, and `MaxCollects` still bound a simulated run, so raise them in a copy of the config first, such as three session changes, two claim changes, and a collection, then add `-simulate num=3000 -depth 150`. Both behind-a-triangle configs fail `SimplePaths` within seconds if `Crossed` is made false, which is the count to infinity the hops fix: a relay drops its distance once its neighbours echoed, and takes back a copy of its own route that a third relay still held. diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 4cac9f93a4e..35be1c9644c 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -494,6 +494,7 @@ A metric that overflows 64 bits makes the route unusable. A route carries its Hops: the Hop ID of every node it crossed, the origin node first and the sender last. A Hop ID is a non-zero 64-bit value a node names itself with for as long as its process runs, such as the Node ID it started with; a rotation does not change it. +A receiver tolerates a zero or repeated Hop ID instead of closing the session: such a route harms only itself. A node sending its selected route appends its own Hop ID to the Hops it received, and an origin's own route starts with its Hop ID alone. A relay forwards the other entries unchanged and never qualifies them. A route that arrives with 32 Hops is unusable. diff --git a/quest/m1/route-inductive.md b/quest/m1/route-inductive.md index ac1efca1f11..ad453c9d968 100644 --- a/quest/m1/route-inductive.md +++ b/quest/m1/route-inductive.md @@ -36,8 +36,9 @@ stay constants the config states. Say both bounds in the `verify-route` skill. replay it on `route::tests::Sim` as the skill says before calling it a bug. - Apalache needs type annotations. Add them to the spec TLC already checks, so both tools read one file and neither drifts. -- `NoCountToInfinity` follows from `SimplePaths`, which is inductive on its - own: every route sent adds its sender to hops that did not name it +- `NoCountToInfinity` follows from `SimplePaths`, which is inductive once + the invariant also says a route is held only from a node the holder shares + a link with: every route sent adds its sender to hops that did not name it ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)). Start there. - Run it through the `tla` recipes with Apalache from the flake. If five relays does not fit Nightly, record the largest mesh that does and its time. @@ -47,6 +48,7 @@ Public API: none. Wire: none. ## Required - [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - the spec parametrised over mesh shapes that this invariant is stated on +- [The route model certifies claims by path](/quest/m2/cluster-routing/routes/certify-model.md) - the certification rule the claim rules rest on; an invariant found before it is found for a rule about to go ## Related diff --git a/quest/m1/route-trace-validation.md b/quest/m1/route-trace-validation.md index cf16769e271..dabec5a30de 100644 --- a/quest/m1/route-trace-validation.md +++ b/quest/m1/route-trace-validation.md @@ -40,3 +40,7 @@ Related: [the mesh route model](/quest/m1/route-mesh-model.md), which grows the spec these traces are checked against. Public API: none. Wire: none. + +## Required + +- [The route model certifies claims by path](/quest/m2/cluster-routing/routes/certify-model.md) - replaces the certification rule and adds two prefixes and a scoped relay, so traces are validated against the rule that stays diff --git a/quest/m2/cluster-routing/README.md b/quest/m2/cluster-routing/README.md index d6e3fb3bd68..afd333e91ca 100644 --- a/quest/m2/cluster-routing/README.md +++ b/quest/m2/cluster-routing/README.md @@ -37,6 +37,13 @@ Decided with the maintainer, 2026-10-13: - Keep Babel. A detour longer than the lost route may wait for a sequence refresh from the origin; instant failover is not required. Measure failure detection, additional routing recovery, and total interruption separately. + Reaffirmed 2026-11-11 (maintainer, /quest-plan after + [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), + asked whether to replace it: ✅ keep it for route selection, with the hops + that PR added, and replace only how a relay decides whose copy of a + publisher's claims to believe + ([the model](/quest/m2/cluster-routing/routes/certify-model.md)). Rejected: + a path-vector without distances, and link-state among relays. - An origin declares its own routing identity, fresh per process or tab restart and shared across its live sessions. It is unverified. Decided 2026-11-02 (maintainer, /quest-unblock): no key proof is planned; if a use case needs a provable identity, moq-lite-07's diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index 80a4fb11698..3bd1f7650b3 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -40,9 +40,11 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 22. ✅ **A side's interest is a set of prefixes edited in place.** The live set never nests, and one edit may swap prefixes for a wider or narrower one without re-sending what both cover ([the interest set](/quest/m2/cluster-routing/routes/interest-set.md)). A token-scope change is such an edit, with one prefix per region of the scope. This supersedes items 16 and 17, and the handoff contract's "An interest change opens a new stream" below, once the route stream lands. 23. ✅ Asked what a floor covers with several prefixes on one stream: **one prefix, named by its id.** This amends item 10: the floor is still a marker and never a region list. Rejected: a floor over the whole set. 24. ✅ Asked how a side learns a newly added prefix's initial set is complete: **SYNCED per prefix, naming its id**, plus one marker per edit where it takes effect for both sides. Rejected: one SYNCED per edit, counted in order. -25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes. +25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region, unless it changes whether the node may use the route: by crossing it, or by reaching the hop limit. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes. +26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **any neighbour whose path to the publisher does not pass through the relay, closer or not** ([the model](/quest/m2/cluster-routing/routes/certify-model.md), then [the router](/quest/m2/cluster-routing/routes/certify.md)). Today only a feasible neighbour counts, one strictly closer than the distance the relay remembers, so a relay's claims move whenever a distance or seqno does: it can forget a claim for good, and behind a scoped neighbour it never shows what a farther one certifies. Route selection keeps Babel and the hops. The contract's bullet on which neighbours certify stands until the model quest rewrites it. Rejected: only strictly closer neighbours; replacing Babel with a path-vector; link-state among relays. + The route layer's core landed in [moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676): a crate-private, sans-IO `Router` in `rs/moq-net/src/route/`, generic over the diff --git a/quest/m2/cluster-routing/routes/certify-model.md b/quest/m2/cluster-routing/routes/certify-model.md new file mode 100644 index 00000000000..f1d617b6320 --- /dev/null +++ b/quest/m2/cluster-routing/routes/certify-model.md @@ -0,0 +1,124 @@ +# [L] The route model certifies claims by path, not by distance + +## Goal + +`rs/moq-net/verify/route/RouteHandoff.tla` decides which neighbour's copy of a +publisher's claims a relay believes by that neighbour's path: its hops do not +name the relay. The remembered distance no longer decides it. The model gains +two prefixes and a relay whose grant is scoped to one of them, and TLC shows on +it: + +- A relay shows a claim exactly while some neighbour certifies it through a + path that does not pass through the relay itself. +- What a relay shows depends only on what its neighbours currently send, never + on the order messages arrived. +- A seqno advance alone changes nothing a reader sees. +- A relay behind a scoped neighbour shows what a farther, unscoped neighbour + certifies. + +Each bug the reviews of [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487) +found is pinned as a `residual_` test on today's router, and the rule the +model settles on is written into the epic's contract for +[the router](/quest/m2/cluster-routing/routes/certify.md) to implement. The +router's behaviour, the wire, and the draft do not change here. Route +selection does not change at all: Babel and the hops stay. + +## Plan + +Decided 2026-11-11 (maintainer, /quest-plan after #8487's two adversarial +reviews). + +Why: a relay believes a neighbour's copy of a publisher's claims only while +that neighbour is "feasible", which means strictly closer to the publisher +than the distance the relay remembers. That set moves whenever a distance or a +seqno moves, and the relay's claims move with it. After the hops landed, the +reviews found no loop and no stuck relay in route selection, and no difference +from the old router on meshes where every relay may see everything. Every +remaining finding needs neighbours that certify different prefixes, which +happens when a relay with a scoped grant forwards for others or has uplinks +under different scopes: + +- A relay forgets a claim for good once the neighbour that certified it stops + being feasible for a moment and returns + ([reproducers](https://github.com/moq-dev/moq/pull/8487#issuecomment-6802083874)). +- Taking the returning neighbour's view back, the obvious fix, flaps a relay's + claims on every seqno advance and keeps one static mesh asking for seqnos + for good (same comment). A neighbour at the relay's own distance is feasible + for one message during an advance. +- A relay whose closest neighbour is scoped never shows what a farther + neighbour certifies, although a request for it routes. +- A mesh with two grants, an upstream link, and two prefixes never settles + after one seqno bump under round-robin delivery + ([reproducer](https://github.com/moq-dev/moq/pull/8487#issuecomment-6798722861)). + +Settled: + +1. ✅ **Replace only how claims are certified; keep Babel and the hops.** + Rejected: replacing Babel with a path-vector, where routes can loop for a + moment while converging, a withdrawal hunts through stale paths, and + [cache-aware routing](/quest/m3/cache-routing/README.md)'s eligibility rule + loses the distance it rests on. Rejected: link-state among relays, since + relays apply their own rules (a priority per connection, upstream marks, + scoped peers) and a shared map stays loop-free only if all compute alike. +2. ✅ **The goal is the four outcomes above.** Rejected: keeping "strictly + closer" and patching each case, which leaves the rule tied to distances. + Rejected: also promising that no reader ever sees a claim go offline and + come back unless its publisher was unreachable, which may need timers. +3. ✅ **A relay believes any neighbour whose path to the publisher does not + pass through it, closer or not.** Rejected: only strictly closer ones, + which keeps a relay behind a scoped neighbour blind. +4. ✅ **The seqno storm is an acceptance test here, not a quest.** The + flapping that feeds it is what this removes. If it survives the new rule, + say so in the PR and it gets its own quest. + +The rule to prove, as a recommendation the model may overturn. Say in the PR +what it settled on and why: + +- **One source per prefix.** For each prefix some neighbour certifies, the + relay takes that prefix's claims from one neighbour: the best ranked among + those whose route does not cross it. It merges nothing across neighbours, + except that it never shows an older revision than one it already knows. +- **What it shows is computed, not accumulated.** Define the reader's view as + an operator of the routes and views the relay holds, with no stored merged + view to prune. Order dependence then cannot be written down. +- **What a relay certifies must be checkable by its receiver.** The hops a + stream carries are the sender's selected route's. When the sender's source + for a prefix is another neighbour, those hops say nothing about where that + prefix's claims came from. Decide between a route per prefix on the stream, + each with its own hops, and certifying nothing for such a prefix. Keep this + counterexample in the model: three relays behind one scoped relay each + hold a wider floor from a path that has since died, and must not keep it + alive by certifying it to each other. +- **Try the hops alone first.** Certification forwards no request, so a + moment's wrong belief costs a late let-go, not a loop. Add a distance per + prefix only if the model needs it. + +Work: + +- In the model, replace the fixed root interest with two prefixes and a + `Scoped` set of relays that ask for one of them, and replace `Certifiers`, + `Certified`, `Prune`, and the stored `view` with the rule above. Add the + rules `Stable` (a seqno advance changes no shown claim) and `Seen` (a claim + a neighbour certifies through a path that does not cross the relay is + shown), and extend `QuietIsSettled` to say nobody shows a claim that no + path from the origin certifies. +- Add configs for the meshes in the two reproducer comments and for the blind + relay, at bounds that finish in minutes on the recipe's settings. Run TLC + one config at a time, simulation first. +- Pin each reproducer as a `residual_` test in `route/tests/`, asserting what + today's router does, with a comment naming this quest's successor. +- Rewrite the epic's bullet that begins "A node tracks the regions each + neighbour's last view certifies" to the settled rule, marked as not yet + built, and record the rule as Plan item 26 there. +- Update the `verify-route` skill's rules table, coverage table, and + "Not covered" list. + +[The interest set](/quest/m2/cluster-routing/routes/interest-set.md) adds +edits to the two prefixes this introduces. + +Public API: none. Wire: none. + +## Related + +- [The router certifies claims by path](/quest/m2/cluster-routing/routes/certify.md) - implements the rule this settles +- [Cache-aware routing](/quest/m3/cache-routing/README.md) - why the feasibility distance stays diff --git a/quest/m2/cluster-routing/routes/certify.md b/quest/m2/cluster-routing/routes/certify.md new file mode 100644 index 00000000000..c4d705c6e8d --- /dev/null +++ b/quest/m2/cluster-routing/routes/certify.md @@ -0,0 +1,57 @@ +# [XL] The router certifies claims by path, not by distance + +## Goal + +`route::Router` believes a neighbour's copy of a publisher's claims when that +neighbour's hops do not name the router, as +[the model](/quest/m2/cluster-routing/routes/certify-model.md) settled. A +relay shows a claim exactly while such a neighbour certifies it, what it shows +depends only on what its neighbours currently send, and a seqno advance alone +changes nothing a reader sees. The `residual_` tests that quest pinned pass as +plain tests, a sweep over meshes with scoped relays holds the rules at every +quiet point, and the draft and the epic's contract say what the router does. + +## Plan + +Decided 2026-11-11 (maintainer, /quest-plan after +[moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews): the +work is two quests, the rule in the model first and the router second +(rejected: the model alone, deciding the router from what it shows; recording +the findings as known limits and waiting for federation). The decisions and +the rule are in the model quest's Plan and the epic's Plan item 26. + +Work: + +- Replace `certifiers`, `certified`, `prune`, `loose`, and the merged `views` + in `route/mod.rs` with the settled rule. Keep the reader's view a function + of what the router holds, computed incrementally the way `show` is today, + with the test build comparing it against the whole computation. +- Remove what #8487 added to work around the old rule: `recompute`'s `prune` + switch, which skips pruning on a change of hops or RTT alone. Keep the + qualified-key test in `infeasible` unless the new rule makes it moot, and + say which. +- Flip each `residual_` test from the model quest, and add the review + harness's scoped and two-prefix families as a sweep beside + `sweep_no_route_outgrows_a_loop_free_path`, checking at every quiet point + that each relay shows what the rule says and nothing waits. Compare a few + thousand scripts against the router before this change and explain every + difference in the PR. +- Draft: Routing and Floors say who certifies, in place of "a peer holds a + route whose last advertisement is feasible here" wherever that sentence + means certification. Add a changelog bullet. If the rule changes what a + ROUTE or ANNOUNCE_FLOOR on a stream means, that is a lite-07-wip wire + change: list it under Impact and run `just test interop --all`. +- If [js/net's route layer](/quest/m2/cluster-routing/routes/js.md) has + landed, change it in the same PR. +- Rewrite the epic's certification bullet as built, and update the + `verify-route` skill's coverage and sync commit. +- Measure route and claim messages per change with + `rs/moq-net/tests/integration/mesh_sweep.rs`'s benchmark before and after, + if it has landed. + +Public API: none (the router is crate-private). Wire: lite-07-wip, only if the +model quest's rule needs it. + +## Required + +- [The route model certifies claims by path](/quest/m2/cluster-routing/routes/certify-model.md) - the rule, proven on two prefixes and a scoped relay, and the tests that pin today's bugs diff --git a/quest/m2/cluster-routing/routes/interest-set.md b/quest/m2/cluster-routing/routes/interest-set.md index ba70d6921fe..5ab9ce4ba7a 100644 --- a/quest/m2/cluster-routing/routes/interest-set.md +++ b/quest/m2/cluster-routing/routes/interest-set.md @@ -71,10 +71,12 @@ Work: ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)), and move the echo from its own queue onto the neighbour's FIFO, as that PR's first commit (`81c0698295`) did: with one stream per side nothing overtakes anything. -- Give the model at least two prefixes and edits in flight: add, remove, - widen, narrow, and an edit that crosses a claim change, a route change, a - ROUTE_END, and its echo. The model fixes interest at the root today, so - this is new state; keep it tractable by bounding edits as it bounds flaps. +- The model has two prefixes and a scoped relay since + [it certifies by path](/quest/m2/cluster-routing/routes/certify-model.md). + Give it edits in flight: add, remove, widen, narrow, and an edit that + crosses a claim change, a route change, a ROUTE_END, and its echo. Keep it + tractable by bounding edits as it bounds flaps. Where a rule below says the + merge rule is unchanged, read it as that quest's rule. - In the router, `Peer::asked` and `Peer::watch` become sets with a floor and a synced flag per prefix, `batch` takes the prefix its floor names, and a neighbour is one key for both directions, as `Sim` already has it. The @@ -97,3 +99,11 @@ one key, which `Sim` and the tests use, so this changes no behaviour on its own. Public API: none (the router is crate-private). Wire: none. + +## Required + +- [The route model certifies claims by path](/quest/m2/cluster-routing/routes/certify-model.md) - the two prefixes and the certification rule that an edit must preserve + +## Related + +- [The router certifies claims by path](/quest/m2/cluster-routing/routes/certify.md) - rewrites how the router keeps a floor per prefix; whichever lands second adapts diff --git a/quest/m3/route-keep-next-hop.md b/quest/m3/route-keep-next-hop.md new file mode 100644 index 00000000000..84f80f3cb95 --- /dev/null +++ b/quest/m3/route-keep-next-hop.md @@ -0,0 +1,45 @@ +# [S] A relay keeps its next hop among equal routes + +## Goal + +A lite-07 relay that already routes to an origin through one neighbour stays +on it when another neighbour offers a route of the same rank: the same drain +state, priority, and metric. Today the lower key wins, so a relay moves for +nothing, and since routes name the nodes they crossed +([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)) every relay +downstream is sent a ROUTE_UPDATE for it. A test counts route messages for a +link coming up beside an equal one and holds the count where it was before +the link. + +## Plan + +Decided 2026-11-11 (maintainer, /quest-plan after #8487): ✅ plan it, in m3 +beside [split horizon](/quest/m3/route-split-horizon.md), since it trims +traffic and fixes nothing (rejected: dropping it). + +What it costs today, from #8487's measurements: a change of hops alone is +3.5% to 11% of route messages in the router's churn sweeps, about 3% of +control messages over real sessions, and on a 32-relay full mesh one link +coming up costs 592 route messages against 251 before the hops. + +[Cluster routing](/quest/m2/cluster-routing/README.md) already decided that +"a new connection alone does not outrank a different equal-cost route", so +this implements a settled rule. Keep the deterministic path hash among equal +routes for a request's first hop: that choice is per request and is not this +selection. + +- `selected` in `route/mod.rs` is a function of what the router holds, and + ranks by key last. Staying put makes it depend on the previous choice, so + keep the current next hop per identity and prefer it among equals. Mirror + it in `RouteHandoff.tla`, whose `Best` picks the lowest key, as the + `verify-route` skill says. +- Check that every router test that names the next hop it expects still + means what it says, rather than editing expectations to match. +- Measure with `Sim`'s `hop_updates` count and, if it has landed, + `mesh_sweep.rs`'s benchmark, before and after. + +Public API: none. Wire: none; behaviour only. + +## Related + +- [Relays split horizon per session](/quest/m3/route-split-horizon.md) - the other trim of route traffic diff --git a/quest/m3/route-split-horizon.md b/quest/m3/route-split-horizon.md index 6a1978ba6f6..e2199c49e65 100644 --- a/quest/m3/route-split-horizon.md +++ b/quest/m3/route-split-horizon.md @@ -18,3 +18,7 @@ Suppress the advertisement per session rather than per neighbor, and measure route messages between two relays before and after. Public API: none. Wire: none; behavior only. + +## Related + +- [A relay keeps its next hop among equal routes](/quest/m3/route-keep-next-hop.md) - the other trim of route traffic diff --git a/rs/moq-net/src/route/tests/routing.rs b/rs/moq-net/src/route/tests/routing.rs index 51a770b03e0..05be36f9dbb 100644 --- a/rs/moq-net/src/route/tests/routing.rs +++ b/rs/moq-net/src/route/tests/routing.rs @@ -1184,6 +1184,11 @@ fn a_scoped_copy_that_crossed_a_relay_asks_for_no_seqno() { let key = qualified(LOCAL.subject, 7); assert_eq!(next(&sim, cdn, key), Some(far)); assert_eq!(next(&sim, near, key), Some(cdn)); + assert_eq!( + sim.shows(near, key, "local/x"), + Some(true), + "the far relay's copy is a route" + ); sim.release(near, cdn); assert!(sim.run()); @@ -1214,6 +1219,7 @@ fn a_scoped_copy_that_crossed_a_relay_is_not_waited_on() { let key = qualified(LOCAL.subject, 7); assert_eq!(next(&sim, cdn, key), Some(far)); assert_eq!(sim.route(near, key).map(|route| route.metric), Some(7)); + assert_eq!(sim.shows(near, key, "local/x"), Some(true)); sim.disconnect(far, cdn); assert!(sim.run()); From 1b960ba89bcb83fb1b294f8cf39fe105d6a74c48 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 12:35:34 -0800 Subject: [PATCH 2/4] quest: replan the route redesign as a route per scope The first plan's rule, believing any neighbour whose route did not cross the relay, was tried on the router: two relays behind a scoped relay keep a dead claim alive through each other, the seqno storm does not settle, and the blind relay shows a claim it has no route to request. A route is now shared per publisher and per scope and believed by its own path. A prototype first measures what keeps a route free of stale copies: Babel's distance, or paths alone with a withdrawal that names the hop that died. Cache-aware routing's one-hop warm route is reopened, and two bugs found by running the sweeps deeper get quests. Co-Authored-By: Claude --- drafts/draft-lcurley-moq-lite.md | 2 +- quest/m1/route-inductive.md | 2 +- quest/m1/route-trace-validation.md | 2 +- quest/m2/cache-routing-proof.md | 14 ++ quest/m2/cluster-routing/README.md | 19 +- quest/m2/cluster-routing/routes/README.md | 2 +- .../cluster-routing/routes/certify-model.md | 124 ------------- quest/m2/cluster-routing/routes/certify.md | 57 ------ .../cluster-routing/routes/handoff-revival.md | 36 ++++ .../m2/cluster-routing/routes/interest-set.md | 12 +- .../m2/cluster-routing/routes/scope-model.md | 93 ++++++++++ quest/m2/cluster-routing/routes/scope.md | 64 +++++++ .../m2/cluster-routing/routes/stale-copies.md | 165 ++++++++++++++++++ .../cluster-routing/routes/widen-false-end.md | 93 ++++++++++ quest/m3/cache-routing/README.md | 22 ++- quest/m3/route-keep-next-hop.md | 13 +- 16 files changed, 518 insertions(+), 202 deletions(-) delete mode 100644 quest/m2/cluster-routing/routes/certify-model.md delete mode 100644 quest/m2/cluster-routing/routes/certify.md create mode 100644 quest/m2/cluster-routing/routes/handoff-revival.md create mode 100644 quest/m2/cluster-routing/routes/scope-model.md create mode 100644 quest/m2/cluster-routing/routes/scope.md create mode 100644 quest/m2/cluster-routing/routes/stale-copies.md create mode 100644 quest/m2/cluster-routing/routes/widen-false-end.md diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 35be1c9644c..601b0d8e1fe 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -2271,7 +2271,7 @@ The `Message Length` describes the payload size on the wire. - Added announce compression: ANNOUNCE_START gains `Path Base` and `Path Keep` to copy the head of a live claim's suffix. - Split routing into two layers on the Announce Stream. ROUTE_START (0x6), ROUTE_UPDATE (0x7), and ROUTE_END (0x8) advertise how the publisher reaches an origin node, named by a Node ID, with a seqno, an additive metric, and a down-only bit, and loops are prevented by Babel's feasibility condition. Every ANNOUNCE_START names its route's Route ID in place of the Hop ID list, so a link change sends one ROUTE_UPDATE per origin rather than an update per claim. Removed the Hop ID list and ANNOUNCE_OK's Hop ID, and replaced the SETUP Hop parameter (0x5) with a [Node ID parameter](#node-id-parameter), so an endpoint that reaches itself detects it at setup. - ROUTE_START and ROUTE_UPDATE carry Flags, whose 0x2 bit adds an optional RTT: the informational sum of each link's lowest smoothed RTT over the last 30 seconds to the origin node, which never selects a route. -- Added Hops to ROUTE_START and ROUTE_UPDATE: the Hop ID of every node the route crossed, at most 32. A node never selects a route whose Hops name it, so one that dropped its feasibility distance cannot take back a copy of its own route. +- Added Hops to ROUTE_START and ROUTE_UPDATE: the Hop ID of every node the route crossed, at most 32. A node never selects a route whose Hops name it, so one that dropped its feasibility distance cannot take back a copy of its own route. A zero or repeated Hop ID in a route's Hops is tolerated, where a repeated non-zero Hop ID in moq-lite-06's Hop ID list is a PROTOCOL_VIOLATION. - Added ROUTE_HOLD (0xB), which holds a route while it recovers; ROUTE_END lets it go. Neither withdraws the origin's claims, and the subscriber echoes every ROUTE_END back. - Added ROUTE_REQUEST (0xA), sent by the subscriber on the Announce Stream to ask for a route with a newer seqno when every route it has is infeasible. - Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the requested prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows. diff --git a/quest/m1/route-inductive.md b/quest/m1/route-inductive.md index ad453c9d968..a5390de7085 100644 --- a/quest/m1/route-inductive.md +++ b/quest/m1/route-inductive.md @@ -48,7 +48,7 @@ Public API: none. Wire: none. ## Required - [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - the spec parametrised over mesh shapes that this invariant is stated on -- [The route model certifies claims by path](/quest/m2/cluster-routing/routes/certify-model.md) - the certification rule the claim rules rest on; an invariant found before it is found for a rule about to go +- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the rule for whose claims a relay believes, which the claim rules rest on; an invariant found before it is found for a rule about to go ## Related diff --git a/quest/m1/route-trace-validation.md b/quest/m1/route-trace-validation.md index dabec5a30de..bd7a7742912 100644 --- a/quest/m1/route-trace-validation.md +++ b/quest/m1/route-trace-validation.md @@ -43,4 +43,4 @@ Public API: none. Wire: none. ## Required -- [The route model certifies claims by path](/quest/m2/cluster-routing/routes/certify-model.md) - replaces the certification rule and adds two prefixes and a scoped relay, so traces are validated against the rule that stays +- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - replaces who a relay believes and adds two prefixes and a scoped relay, so traces are validated against the rule that stays diff --git a/quest/m2/cache-routing-proof.md b/quest/m2/cache-routing-proof.md index e95de051d54..a864b4ee6e9 100644 --- a/quest/m2/cache-routing-proof.md +++ b/quest/m2/cache-routing-proof.md @@ -27,6 +27,16 @@ it; a SETUP extension needs neither before the cut, and from its list. Either placement is now this quest's to record, with no second trip to the maintainer for choosing lite-07. +Reopened 2026-11-11 (maintainer, /quest-plan on the route redesign, epic +Plan item 8): a warm route propagates, and the rule that keeps it free of +loops and stale copies is +[the prototype](/quest/m2/cluster-routing/routes/stale-copies.md)'s to pick. +The eligibility rule and the fixtures below are for the one-hop warm route +first decided, which [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) +proves. Once the prototype's pick is recorded, rewrite both for it: the +fixtures keep their meshes, and gain a relay that joins a copy at a neighbour +farther from the publisher than itself. + Build on the sans-IO `Router` in `rs/moq-net/src/route/`, which each origin's `route::Node` drives, and the harness behind its handoff sweeps, not the reference model in @@ -70,6 +80,10 @@ Route ID, and the instance, and say which stream carries it and how a peer that never sends one behaves. Size it against lite-07 and an additive SETUP extension and record the choice and its reason in the epic's Plan. +## Required + +- [A prototype settles what keeps a route free of stale copies](/quest/m2/cluster-routing/routes/stale-copies.md) - the rule a propagating warm route rests on, which replaces the one this proved + ## Related - [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the feasibility rules and handoff sweeps this extends diff --git a/quest/m2/cluster-routing/README.md b/quest/m2/cluster-routing/README.md index afd333e91ca..8704e22e123 100644 --- a/quest/m2/cluster-routing/README.md +++ b/quest/m2/cluster-routing/README.md @@ -37,13 +37,18 @@ Decided with the maintainer, 2026-10-13: - Keep Babel. A detour longer than the lost route may wait for a sequence refresh from the origin; instant failover is not required. Measure failure detection, additional routing recovery, and total interruption separately. - Reaffirmed 2026-11-11 (maintainer, /quest-plan after - [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), - asked whether to replace it: ✅ keep it for route selection, with the hops - that PR added, and replace only how a relay decides whose copy of a - publisher's claims to believe - ([the model](/quest/m2/cluster-routing/routes/certify-model.md)). Rejected: - a path-vector without distances, and link-state among relays. + Reopened 2026-11-11 (maintainer, /quest-plan after + [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews): + whether Babel's distance stays is + [the prototype](/quest/m2/cluster-routing/routes/stale-copies.md)'s to + measure, against paths alone with a withdrawal that names the hop that + died. The distance refuses a longer way round until the publisher advances + its seqno, and refuses a cached copy at a neighbour no closer to the + publisher, which [cache-aware routing](/quest/m3/cache-routing/README.md) + wants to join. Decided the same day: a route is shared per publisher and + per scope, and a relay believes a route by that route's own path, whichever + rule wins. Rejected: link-state among relays, and routes to relays in + place of publishers. - An origin declares its own routing identity, fresh per process or tab restart and shared across its live sessions. It is unverified. Decided 2026-11-02 (maintainer, /quest-unblock): no key proof is planned; if a use case needs a provable identity, moq-lite-07's diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index 3bd1f7650b3..60fd029ee4c 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -43,7 +43,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region, unless it changes whether the node may use the route: by crossing it, or by reaching the hop limit. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes. -26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **any neighbour whose path to the publisher does not pass through the relay, closer or not** ([the model](/quest/m2/cluster-routing/routes/certify-model.md), then [the router](/quest/m2/cluster-routing/routes/certify.md)). Today only a feasible neighbour counts, one strictly closer than the distance the relay remembers, so a relay's claims move whenever a distance or seqno does: it can forget a claim for good, and behind a scoped neighbour it never shows what a farther one certifies. Route selection keeps Babel and the hops. The contract's bullet on which neighbours certify stands until the model quest rewrites it. Rejected: only strictly closer neighbours; replacing Babel with a path-vector; link-state among relays. +26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **a route is shared per publisher and per scope, and a relay believes a route by that route's own path** ([the prototype](/quest/m2/cluster-routing/routes/stale-copies.md), [the model](/quest/m2/cluster-routing/routes/scope-model.md), then [the router](/quest/m2/cluster-routing/routes/scope.md)). A scope is the prefix a relay asked a neighbour for when its grant there is narrower than its interest. Today only a feasible neighbour counts, one strictly cheaper than the distance the relay remembers, so a relay's claims move whenever a cost or seqno does: it can forget a claim for good, a mesh with two grants never settles after one seqno advance, and behind a scoped neighbour it never shows what a farther one knows. Only a broadcast some relay carries gets its own route, and that route propagates ([cache-aware routing](/quest/m3/cache-routing/README.md) Plan item 8). Open, for the prototype to measure and the maintainer to pick: whether Babel's distance or the path alone, with a withdrawal that names the hop that died, keeps a route free of loops and stale copies. The contract's bullet on which neighbours certify stands until the model quest rewrites it. First decided the same day as any neighbour whose path to the publisher does not pass through the relay; tried on the router, two relays behind a scoped relay then keep a dead claim alive through each other, and the relay behind a scoped neighbour shows a claim it has no route to request. Rejected: that rule; believing a neighbour that is cheaper than the relay is now, which fixed the storm and the forgotten claim with no wire change but rests belief on distance; a path per broadcast as lite-06 has; link-state among relays; routes to relays. The route layer's core landed in [moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676): a diff --git a/quest/m2/cluster-routing/routes/certify-model.md b/quest/m2/cluster-routing/routes/certify-model.md deleted file mode 100644 index f1d617b6320..00000000000 --- a/quest/m2/cluster-routing/routes/certify-model.md +++ /dev/null @@ -1,124 +0,0 @@ -# [L] The route model certifies claims by path, not by distance - -## Goal - -`rs/moq-net/verify/route/RouteHandoff.tla` decides which neighbour's copy of a -publisher's claims a relay believes by that neighbour's path: its hops do not -name the relay. The remembered distance no longer decides it. The model gains -two prefixes and a relay whose grant is scoped to one of them, and TLC shows on -it: - -- A relay shows a claim exactly while some neighbour certifies it through a - path that does not pass through the relay itself. -- What a relay shows depends only on what its neighbours currently send, never - on the order messages arrived. -- A seqno advance alone changes nothing a reader sees. -- A relay behind a scoped neighbour shows what a farther, unscoped neighbour - certifies. - -Each bug the reviews of [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487) -found is pinned as a `residual_` test on today's router, and the rule the -model settles on is written into the epic's contract for -[the router](/quest/m2/cluster-routing/routes/certify.md) to implement. The -router's behaviour, the wire, and the draft do not change here. Route -selection does not change at all: Babel and the hops stay. - -## Plan - -Decided 2026-11-11 (maintainer, /quest-plan after #8487's two adversarial -reviews). - -Why: a relay believes a neighbour's copy of a publisher's claims only while -that neighbour is "feasible", which means strictly closer to the publisher -than the distance the relay remembers. That set moves whenever a distance or a -seqno moves, and the relay's claims move with it. After the hops landed, the -reviews found no loop and no stuck relay in route selection, and no difference -from the old router on meshes where every relay may see everything. Every -remaining finding needs neighbours that certify different prefixes, which -happens when a relay with a scoped grant forwards for others or has uplinks -under different scopes: - -- A relay forgets a claim for good once the neighbour that certified it stops - being feasible for a moment and returns - ([reproducers](https://github.com/moq-dev/moq/pull/8487#issuecomment-6802083874)). -- Taking the returning neighbour's view back, the obvious fix, flaps a relay's - claims on every seqno advance and keeps one static mesh asking for seqnos - for good (same comment). A neighbour at the relay's own distance is feasible - for one message during an advance. -- A relay whose closest neighbour is scoped never shows what a farther - neighbour certifies, although a request for it routes. -- A mesh with two grants, an upstream link, and two prefixes never settles - after one seqno bump under round-robin delivery - ([reproducer](https://github.com/moq-dev/moq/pull/8487#issuecomment-6798722861)). - -Settled: - -1. ✅ **Replace only how claims are certified; keep Babel and the hops.** - Rejected: replacing Babel with a path-vector, where routes can loop for a - moment while converging, a withdrawal hunts through stale paths, and - [cache-aware routing](/quest/m3/cache-routing/README.md)'s eligibility rule - loses the distance it rests on. Rejected: link-state among relays, since - relays apply their own rules (a priority per connection, upstream marks, - scoped peers) and a shared map stays loop-free only if all compute alike. -2. ✅ **The goal is the four outcomes above.** Rejected: keeping "strictly - closer" and patching each case, which leaves the rule tied to distances. - Rejected: also promising that no reader ever sees a claim go offline and - come back unless its publisher was unreachable, which may need timers. -3. ✅ **A relay believes any neighbour whose path to the publisher does not - pass through it, closer or not.** Rejected: only strictly closer ones, - which keeps a relay behind a scoped neighbour blind. -4. ✅ **The seqno storm is an acceptance test here, not a quest.** The - flapping that feeds it is what this removes. If it survives the new rule, - say so in the PR and it gets its own quest. - -The rule to prove, as a recommendation the model may overturn. Say in the PR -what it settled on and why: - -- **One source per prefix.** For each prefix some neighbour certifies, the - relay takes that prefix's claims from one neighbour: the best ranked among - those whose route does not cross it. It merges nothing across neighbours, - except that it never shows an older revision than one it already knows. -- **What it shows is computed, not accumulated.** Define the reader's view as - an operator of the routes and views the relay holds, with no stored merged - view to prune. Order dependence then cannot be written down. -- **What a relay certifies must be checkable by its receiver.** The hops a - stream carries are the sender's selected route's. When the sender's source - for a prefix is another neighbour, those hops say nothing about where that - prefix's claims came from. Decide between a route per prefix on the stream, - each with its own hops, and certifying nothing for such a prefix. Keep this - counterexample in the model: three relays behind one scoped relay each - hold a wider floor from a path that has since died, and must not keep it - alive by certifying it to each other. -- **Try the hops alone first.** Certification forwards no request, so a - moment's wrong belief costs a late let-go, not a loop. Add a distance per - prefix only if the model needs it. - -Work: - -- In the model, replace the fixed root interest with two prefixes and a - `Scoped` set of relays that ask for one of them, and replace `Certifiers`, - `Certified`, `Prune`, and the stored `view` with the rule above. Add the - rules `Stable` (a seqno advance changes no shown claim) and `Seen` (a claim - a neighbour certifies through a path that does not cross the relay is - shown), and extend `QuietIsSettled` to say nobody shows a claim that no - path from the origin certifies. -- Add configs for the meshes in the two reproducer comments and for the blind - relay, at bounds that finish in minutes on the recipe's settings. Run TLC - one config at a time, simulation first. -- Pin each reproducer as a `residual_` test in `route/tests/`, asserting what - today's router does, with a comment naming this quest's successor. -- Rewrite the epic's bullet that begins "A node tracks the regions each - neighbour's last view certifies" to the settled rule, marked as not yet - built, and record the rule as Plan item 26 there. -- Update the `verify-route` skill's rules table, coverage table, and - "Not covered" list. - -[The interest set](/quest/m2/cluster-routing/routes/interest-set.md) adds -edits to the two prefixes this introduces. - -Public API: none. Wire: none. - -## Related - -- [The router certifies claims by path](/quest/m2/cluster-routing/routes/certify.md) - implements the rule this settles -- [Cache-aware routing](/quest/m3/cache-routing/README.md) - why the feasibility distance stays diff --git a/quest/m2/cluster-routing/routes/certify.md b/quest/m2/cluster-routing/routes/certify.md deleted file mode 100644 index c4d705c6e8d..00000000000 --- a/quest/m2/cluster-routing/routes/certify.md +++ /dev/null @@ -1,57 +0,0 @@ -# [XL] The router certifies claims by path, not by distance - -## Goal - -`route::Router` believes a neighbour's copy of a publisher's claims when that -neighbour's hops do not name the router, as -[the model](/quest/m2/cluster-routing/routes/certify-model.md) settled. A -relay shows a claim exactly while such a neighbour certifies it, what it shows -depends only on what its neighbours currently send, and a seqno advance alone -changes nothing a reader sees. The `residual_` tests that quest pinned pass as -plain tests, a sweep over meshes with scoped relays holds the rules at every -quiet point, and the draft and the epic's contract say what the router does. - -## Plan - -Decided 2026-11-11 (maintainer, /quest-plan after -[moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews): the -work is two quests, the rule in the model first and the router second -(rejected: the model alone, deciding the router from what it shows; recording -the findings as known limits and waiting for federation). The decisions and -the rule are in the model quest's Plan and the epic's Plan item 26. - -Work: - -- Replace `certifiers`, `certified`, `prune`, `loose`, and the merged `views` - in `route/mod.rs` with the settled rule. Keep the reader's view a function - of what the router holds, computed incrementally the way `show` is today, - with the test build comparing it against the whole computation. -- Remove what #8487 added to work around the old rule: `recompute`'s `prune` - switch, which skips pruning on a change of hops or RTT alone. Keep the - qualified-key test in `infeasible` unless the new rule makes it moot, and - say which. -- Flip each `residual_` test from the model quest, and add the review - harness's scoped and two-prefix families as a sweep beside - `sweep_no_route_outgrows_a_loop_free_path`, checking at every quiet point - that each relay shows what the rule says and nothing waits. Compare a few - thousand scripts against the router before this change and explain every - difference in the PR. -- Draft: Routing and Floors say who certifies, in place of "a peer holds a - route whose last advertisement is feasible here" wherever that sentence - means certification. Add a changelog bullet. If the rule changes what a - ROUTE or ANNOUNCE_FLOOR on a stream means, that is a lite-07-wip wire - change: list it under Impact and run `just test interop --all`. -- If [js/net's route layer](/quest/m2/cluster-routing/routes/js.md) has - landed, change it in the same PR. -- Rewrite the epic's certification bullet as built, and update the - `verify-route` skill's coverage and sync commit. -- Measure route and claim messages per change with - `rs/moq-net/tests/integration/mesh_sweep.rs`'s benchmark before and after, - if it has landed. - -Public API: none (the router is crate-private). Wire: lite-07-wip, only if the -model quest's rule needs it. - -## Required - -- [The route model certifies claims by path](/quest/m2/cluster-routing/routes/certify-model.md) - the rule, proven on two prefixes and a scoped relay, and the tests that pin today's bugs diff --git a/quest/m2/cluster-routing/routes/handoff-revival.md b/quest/m2/cluster-routing/routes/handoff-revival.md new file mode 100644 index 00000000000..716df8fd7d9 --- /dev/null +++ b/quest/m2/cluster-routing/routes/handoff-revival.md @@ -0,0 +1,36 @@ +# [M] The handoff sweep revives no ended claim + +## Goal + +`sweep_handoff` in `rs/moq-net/src/route/tests/handoff.rs` reports no revival +over 20,000 scenarios: no reader sees a claim start again after it saw the +claim end, unless it forgot the claim in between. The scenario that revives +one today is a plain test. + +## Plan + +Found 2026-11-11 by running the sweep past its 150 scenarios: scenario 16316 +revives a claim, on `main` at `688312b9eb`, and no earlier one does. It is +the same whichever neighbours the router believes. It has not been traced. +Decided the same day (maintainer, /quest-plan): ✅ plan it. + +- Reproduce it: the sweep draws every scenario from one random stream, so run + the first 16,317. Record the scenario's steps as it runs and remove them + one at a time while the revival remains, as + [the false END](/quest/m2/cluster-routing/routes/widen-false-end.md) was + shrunk. +- Decide whether it is a revival the contract forbids or one it excuses: a + replay after a collection or a restart is allowed only until the + withdrawal behind it lands + ([epic](/quest/m2/cluster-routing/routes/README.md), handoff contract). If + the fixture's `revivals` count misses an excuse, fix the count and say so. +- Otherwise fix the cause and keep the shrunk case as a test. +- If the cause is in what + [the route per scope](/quest/m2/cluster-routing/routes/scope.md) deletes, + pin it as a `residual_` test naming that quest instead of fixing it twice. + +Public API: none. Wire: none. + +## Related + +- [Widening an interest ends no live claim](/quest/m2/cluster-routing/routes/widen-false-end.md) - found the same way, and adds the way to run more scenarios diff --git a/quest/m2/cluster-routing/routes/interest-set.md b/quest/m2/cluster-routing/routes/interest-set.md index 5ab9ce4ba7a..fd471af3227 100644 --- a/quest/m2/cluster-routing/routes/interest-set.md +++ b/quest/m2/cluster-routing/routes/interest-set.md @@ -72,11 +72,12 @@ Work: that PR's first commit (`81c0698295`) did: with one stream per side nothing overtakes anything. - The model has two prefixes and a scoped relay since - [it certifies by path](/quest/m2/cluster-routing/routes/certify-model.md). + [it has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md). Give it edits in flight: add, remove, widen, narrow, and an edit that crosses a claim change, a route change, a ROUTE_END, and its echo. Keep it - tractable by bounding edits as it bounds flaps. Where a rule below says the - merge rule is unchanged, read it as that quest's rule. + tractable by bounding edits as it bounds flaps. Where Settled item 1 says + the merge rule is unchanged, read it as that quest's rule: a floor names + its route's prefix, and the newest floor wins among one publisher's routes. - In the router, `Peer::asked` and `Peer::watch` become sets with a floor and a synced flag per prefix, `batch` takes the prefix its floor names, and a neighbour is one key for both directions, as `Sim` already has it. The @@ -102,8 +103,9 @@ Public API: none (the router is crate-private). Wire: none. ## Required -- [The route model certifies claims by path](/quest/m2/cluster-routing/routes/certify-model.md) - the two prefixes and the certification rule that an edit must preserve +- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the two prefixes and the rule for whose claims a relay believes, which an edit must preserve ## Related -- [The router certifies claims by path](/quest/m2/cluster-routing/routes/certify.md) - rewrites how the router keeps a floor per prefix; whichever lands second adapts +- [The router keeps a route per scope](/quest/m2/cluster-routing/routes/scope.md) - rewrites how the router keeps a floor per prefix; whichever lands second adapts +- [Widening an interest ends no live claim](/quest/m2/cluster-routing/routes/widen-false-end.md) - a false END on an interest change today, which the edit in place may remove diff --git a/quest/m2/cluster-routing/routes/scope-model.md b/quest/m2/cluster-routing/routes/scope-model.md new file mode 100644 index 00000000000..a31ed03572a --- /dev/null +++ b/quest/m2/cluster-routing/routes/scope-model.md @@ -0,0 +1,93 @@ +# [L] The route model has a route per scope + +## Goal + +`rs/moq-net/verify/route/RouteHandoff.tla` gains two prefixes and relays that +ask a neighbour for only one of them. A route is keyed by its publisher and +the prefix it covers, carries its own path, and is believed by a relay it did +not cross. TLC shows on it: + +- A relay shows a claim exactly while it holds a route that covers the claim + and did not cross the relay, at the newest revision the relay knows. +- Once nothing is in flight, no relay shows a claim that no path from its + publisher carries. +- A change of path or seqno alone starts, ends, and changes no claim a reader + sees. Whether the claim is available may change. +- A request for a shown claim follows next hops to the publisher without a + loop, also where one route's prefix lies inside another's. So a relay + behind a scoped neighbour reaches what a farther, unscoped neighbour + offers. + +The model uses the rule +[the prototype](/quest/m2/cluster-routing/routes/stale-copies.md) picked for +keeping a route free of stale copies. The rule this settles is written into +the epic's contract for +[the router](/quest/m2/cluster-routing/routes/scope.md) to implement. The +router's behaviour, the wire, and the draft do not change here. + +## Plan + +Decided 2026-11-11 (maintainer, /quest-plan). The decisions and what was +tried are in [the prototype](/quest/m2/cluster-routing/routes/stale-copies.md)'s +Plan and the [epic](/quest/m2/cluster-routing/routes/README.md)'s Plan item +26. + +Why a route per scope: a relay that asked a neighbour for prefix `a` knows a +publisher's claims only inside `a`. Today that partial knowledge lives inside +one view per publisher, as a floor per region, with claims no floor covers +carried loose, and a rule for which neighbours vouch for which region. Every +open finding is in that machinery. With the prefix on the route, every +neighbour that offers a route knows all of it, and the route ends when its +path does. + +Recommendations the model may overturn. Say in the PR what it settled on and +why: + +- **The prefix a route covers travels with the route**, as a field of + ROUTE_START. On a stream that asked for a narrower prefix the route covers + that prefix, so narrowing composes. Rejected so far: deriving a Node ID + from the publisher and the prefix, which needs no wire change but cannot + be narrowed a second time, since a relay cannot read the prefix back. +- **The newest floor wins among one publisher's routes.** A claim both a + narrow and a wide route cover is shown once, at the newer revision, and a + relay keeps what it learned while any route covers the claim + (epic Plan item 19). Nothing is pruned by an instant: a route that ends + takes only what no other route covers. +- **A request takes the narrowest route that covers its claim, and is never + forwarded along a route whose path names the neighbour it came from.** + This is the unsolved part: prove the selected next hops cannot loop when + the narrow and the wide route lead different ways, under stale messages, + or find the rule that does. +- Keep the counterexample that broke the first plan: two relays behind a + scoped relay must not keep a wider route alive through each other after + its path closed. + +Work: + +- In the model, replace the fixed root interest with two prefixes and a + `Scoped` set of relays that ask for one. Key `rx`, the selection, and what + is sent by route. Replace `Certifiers`, `Certified`, `Prune`, and the + stored merged `view` with one floor per route and the newest-floor rule. + Add the Goal's four rules: `Seen`, an extended `QuietIsSettled`, `Stable`, + and `Reaches`. +- Add configs for the dead claim, the blind relay, and the meshes in the two + reproducer comments the prototype pins, at bounds that finish in minutes on + the recipe's settings. Run TLC one config at a time, simulation first. +- Rewrite the epic's bullet that begins "A node tracks the regions each + neighbour's last view certifies" to the settled rule, marked as not yet + built. +- Update the `verify-route` skill's rules table, coverage table, and + "Not covered" list. + +[The interest set](/quest/m2/cluster-routing/routes/interest-set.md) adds +edits to the two prefixes this introduces. + +Public API: none. Wire: none. + +## Required + +- [A prototype settles what keeps a route free of stale copies](/quest/m2/cluster-routing/routes/stale-copies.md) - the rule for loops and stale copies that this model is written on + +## Related + +- [The router keeps a route per scope](/quest/m2/cluster-routing/routes/scope.md) - implements the rule this settles diff --git a/quest/m2/cluster-routing/routes/scope.md b/quest/m2/cluster-routing/routes/scope.md new file mode 100644 index 00000000000..02b0d10b795 --- /dev/null +++ b/quest/m2/cluster-routing/routes/scope.md @@ -0,0 +1,64 @@ +# [XL] The router keeps a route per scope + +## Goal + +`route::Router` keys a route by its publisher and the prefix it covers, and +believes a route that did not cross it, as +[the model](/quest/m2/cluster-routing/routes/scope-model.md) settled. A relay +shows a claim exactly while such a route covers it, two relays never keep a +dead route alive through each other, a change of path or seqno alone changes +no claim a reader sees, and a relay behind a scoped neighbour reaches what a +farther one offers. The `residual_` tests +[the prototype](/quest/m2/cluster-routing/routes/stale-copies.md) pinned pass +as plain tests, a sweep over meshes with scoped relays holds the rules at +every quiet point, and the draft and the epic's contract say what the router +does. + +## Plan + +Decided 2026-11-11 (maintainer, /quest-plan): the prototype picks the rule +for stale copies, the model settles the route per scope, and the router comes +last (rejected: the router first, patched case by case; recording the +findings as known limits and waiting for federation). + +Work: + +- Key `rx`, the selection, `sent`, and the views in `route/mod.rs` by route. + A view then has one floor, so delete `certifiers`, `certified`, `prune`, + `loose`, and the per-region floors, and keep the newest floor among one + publisher's routes as the model has it. Keep the reader's view a function + of what the router holds, computed incrementally the way `show` is today, + with the test build comparing it against the whole computation. +- Remove what #8487 added to work around the old rule: `recompute`'s `prune` + switch, which skips pruning on a change of hops or RTT alone. Keep the + qualified-key test in `infeasible` unless the new rule makes it moot, and + say which. +- Serve a request through the route the model's rule picks, in + `route/node.rs`'s `home`, in place of every feasible route whose neighbour + `carries` the claim. +- Flip each `residual_` test, and add the review harness's scoped and + two-prefix families as a sweep beside + `sweep_no_route_outgrows_a_loop_free_path`, checking at every quiet point + that each relay shows what the rule says and nothing waits. Compare a few + thousand scripts against the router before this change and explain every + difference in the PR. +- Wire (lite-07-wip): whatever the model settled for naming a route's + prefix. List it under Impact and run `just test interop --all`. +- Draft: Routing says a route covers a prefix, and Floors says a floor + certifies its route's prefix. Replace the Floors condition "every peer + whose last advertised route to it is feasible here has answered the + current request" with the settled rule. The hold rule in Routing is not + about belief and stays unless the prototype's pick removes holds. Add a + changelog bullet. +- If [js/net's route layer](/quest/m2/cluster-routing/routes/js.md) has + landed, change it in the same PR. +- Rewrite the epic's bullet on who certifies as built, and update the + `verify-route` skill's coverage and sync commit. +- Measure route and claim messages per change with + `rs/moq-net/tests/integration/mesh_sweep.rs`'s benchmark before and after. + +Public API: none (the router is crate-private). Wire: lite-07-wip. + +## Required + +- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the rule, proven on two prefixes and a scoped relay diff --git a/quest/m2/cluster-routing/routes/stale-copies.md b/quest/m2/cluster-routing/routes/stale-copies.md new file mode 100644 index 00000000000..98ecbb58bac --- /dev/null +++ b/quest/m2/cluster-routing/routes/stale-copies.md @@ -0,0 +1,165 @@ +# [L] A prototype settles what keeps a route free of stale copies + +## Goal + +A test-only router beside `route::tests::Sim` runs lite-07 routing on paths +alone: a relay uses any route whose path does not name it, and a withdrawal +names the hop that died, so every route through that hop goes at once. The +same seeded meshes and scripts run on it and on today's router, which keeps +Babel's distance. The PR reports for both: + +- whether selected routes ever loop, and for how many steps; +- for how many steps a relay still shows a route whose path is gone, after a + link closes, a relay dies, and a publisher leaves; +- how many route messages each of those changes costs on 3, 6, 12, and 24 + relays; +- how many steps a relay with a longer way round takes to use it; +- whether a relay can pull a broadcast from a neighbour that already carries + it and is no closer to the publisher than the relay. + +The PR recommends one of the two rules, and the maintainer's pick is recorded +in the [epic](/quest/m2/cluster-routing/routes/README.md)'s Plan. Each bug the +reviews of [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487) found +is pinned as a `residual_` test on today's router. The production router, the +wire, and the draft do not change here. + +## Plan + +Decided 2026-11-11 (maintainer, /quest-plan after #8487's two adversarial +reviews, revised the same day once the first plan's rule was tried on the +router). + +Why: today a relay believes a neighbour's copy of a publisher's claims only +while that neighbour is "feasible", which means strictly cheaper than the +lowest cost the relay ever advertised (Babel's distance). That set moves +whenever a cost or a seqno moves, and the relay's claims move with it. The +reviews found no loop and no stuck relay in route selection, and no remaining +bug on a mesh where every relay may see everything. Every finding needs +neighbours that know different parts of a publisher's claims, which happens +when a relay asks a neighbour for less than it wants because its grant on +that session is narrower. Call the prefix it asked for the **scope**. + +What was tried, each as a few lines behind a switch on `main` at `688312b9eb`: + +| | Today | Believe any neighbour whose route did not cross the relay | Believe a neighbour cheaper than the relay is now, ignoring the seqno, and keep what it sent | +| --- | --- | --- | --- | +| The seqno storm mesh settles | no | no | yes | +| A returning neighbour brings its claims back | yes | yes | yes | +| A seqno advance changes nothing shown | yes | yes | yes | +| Two relays behind a scoped relay let a dead claim go | yes | no, kept for good | yes | +| The blind relay shows a farther neighbour's claim | no | yes, as available, with no route to request it | no | +| The rest of `moq-net` | pass | 1 fails | 1 fails | + +On lite-06 (`release`), over real sessions, the blind relay sees and resolves +the far claim, and the two relays retract the dead one once. lite-06 carries +a path on every announcement, so believing by path is sound there. lite-07 +has one path per publisher, which cannot vouch for claims that reached the +sender another way. + +Settled: + +1. ✅ **A route is shared per publisher and per scope, and a relay believes a + route by that route's own path.** Every claim on a route is then known + completely for the route's prefix, and came along the path the route + names. Where every relay may see everything this is one route per + publisher, as today. + [The model](/quest/m2/cluster-routing/routes/scope-model.md) and + [the router](/quest/m2/cluster-routing/routes/scope.md) build it. Rejected: + believing by the publisher's one path, the first plan, which the table + shows is unsound. Rejected: believing by cost alone, the third column, + which fixed the storm in 25 lines with no wire change; the maintainer does + not want belief to rest on distance, and the blind relay stays blind. + Rejected: a path per broadcast as lite-06 has, which re-announces every + broadcast when a path changes. +2. ✅ **The blind relay is fixed by routing, not by showing more.** A relay + whose closest neighbour is scoped does not show what a farther neighbour + knows, and today that is consistent: it believes exactly the neighbours it + can forward a request to, and the farther one is not feasible. With a + route per scope the wider route is its own destination, so the relay can + both show the claim and reach it. +3. ✅ **Only a broadcast some relay carries gets its own route, and that + route propagates.** This reopens + [cache-aware routing](/quest/m3/cache-routing/README.md)'s one-hop warm + route (its Plan item 8). +4. ✅ **What keeps a route free of loops and stale copies is this quest's to + measure, not yet decided.** The hops stop loops, not stale copies. + Two candidates: + - **Distance, as today.** A relay uses a neighbour only when the + neighbour's cost is below the lowest cost the relay ever advertised. + Anything routed through the relay costs more than that, so such a + neighbour cannot be stale. It never loops, even for a moment. It refuses + good routes: a longer way round waits for the publisher to advance its + seqno, and a relay cannot join a copy carried by a neighbour that is no + closer to the publisher, which is the join cache-aware routing wants. + The seqno, ROUTE_REQUEST, ROUTE_HOLD, and the ROUTE_END echo all exist + to recover what this check refuses. + - **Paths alone, with a withdrawal that names what died.** A relay uses + any route whose path does not name it. Each hop in a path names the + relay and the session it learned the route on. When that session + closes, the relay's withdrawal names that hop, and every relay drops + every route through it at once, whichever neighbour offered it. This is + root cause notification from BGP research + ([Pei et al. 2005](https://www.cs.colostate.edu/~massey/pubs/journal/massey_comnet05.pdf)), + which bounds convergence by the mesh's diameter. Its known gap: when a + relay with several neighbours dies, each neighbour reports its own + session, so one stale switch per neighbour can remain. A route can also + loop for a moment while a path change is in flight. It needs no seqno, + request, hold, or echo. lite-06 instead holds stale paths back with a + 300 ms hold and a 2 s quarantine, and + [moq-dev/moq#5375](https://github.com/moq-dev/moq/pull/5375) still + measured stale revivals. + + The October measurement that chose Babel compared it with a path per + broadcast, not with shared routes, so its message counts do not answer + this. +5. ✅ **No routes to relays.** A relay keeps the cheapest path to each + publisher's scope, not to a relay that publisher is attached to. + +Work: + +- Write the paths-alone router as test-only code under `route/tests/`, as + small as it can be: routes shared per publisher, each with a path and a + cost; the withdrawal that names a hop; and claims that ride a route with + one floor, newest floor winning. Leave out what the comparison does not + need: scoped grants, qualified keys, interest changes, GOAWAY, and RTT. +- A relay remembers a dead hop only while one of its neighbours still offers + a route through it. Report whether a stale route can return after that, + and for how long. +- Drive both routers from one script generator, the one behind + `sweep_no_route_outgrows_a_loop_free_path`, and from the mesh shapes of + `rs/moq-net/tests/integration/mesh_sweep.rs`'s benchmark. Check at every + step and at every quiet point, and count as the Goal lists. +- Report what a relay's reader sees when its only route goes. Today it holds + the claims as unavailable while a seqno request is out + ([epic](/quest/m2/cluster-routing/routes/README.md) Plan item 19); on + paths alone there is nothing to wait for, so it lets go at once. +- Build the cached-copy case for both: a relay whose neighbour carries a + broadcast and is farther from the publisher. Two relays that each carry a + copy must not adopt each other, and a copy that goes must take every route + through it along. +- Pin each finding as a `residual_` test in `route/tests/`, asserting what + today's router does, with a comment naming + [the router quest](/quest/m2/cluster-routing/routes/scope.md): the seqno + storm ([reproducer](https://github.com/moq-dev/moq/pull/8487#issuecomment-6798722861)), + the forgotten claim and the two tests its failed fix broke + ([reproducers](https://github.com/moq-dev/moq/pull/8487#issuecomment-6802083874)), + and the blind relay: a publisher behind a relay that asked for `a` and + behind a two-relay chain that asked for everything, both joined to one + relay, which shows `a/x` and not `b/x`. +- Add the dead claim as a plain test, since today's router passes it and any + new rule must: a relay whose only way to the publisher asked for `a`, with + two relays joined to it and to each other, one of which also has a dear + link straight to the publisher. That link closes and the publisher + withdraws `b/x`; neither relay may still show it. +- Record the maintainer's pick in the epic's Plan item 26 and in + [Cluster routing](/quest/m2/cluster-routing/README.md)'s "Keep Babel" + bullet. If the pick is paths alone, plan removing the distance, the seqno, + ROUTE_REQUEST, ROUTE_HOLD, and the echo with `/quest-plan` before the + router quest starts. + +Public API: none. Wire: none. + +## Related + +- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - models the rule this picks +- [Cache-aware routing](/quest/m3/cache-routing/README.md) - its warm routes rest on the rule this picks diff --git a/quest/m2/cluster-routing/routes/widen-false-end.md b/quest/m2/cluster-routing/routes/widen-false-end.md new file mode 100644 index 00000000000..f02f902167a --- /dev/null +++ b/quest/m2/cluster-routing/routes/widen-false-end.md @@ -0,0 +1,93 @@ +# [M] Widening an interest ends no live claim + +## Goal + +A client that widens its interest never sees END for a claim its publisher +still has. `sweep_scoped_handoff` in `rs/moq-net/src/route/tests/handoff.rs` +reports no false END over 20,000 scenarios, and the case below is a plain +test. + +## Plan + +Found 2026-11-11 by running the sweep past its 150 scenarios: 25 of 20,000 +end a live claim, the first at scenario 750, on `main` at `688312b9eb`. It is +the same whichever neighbours the router believes, so +[the route per scope](/quest/m2/cluster-routing/routes/scope.md) does not fix +it. Decided the same day (maintainer, /quest-plan): ✅ plan it (rejected: +leaving it to the interest set). + +Scenario 750, shrunk to 13 steps. The client sits behind two relays, each +under a grant for one prefix, and sees the publisher's `live` at whichever +prefix its interest and a grant meet. It narrows to `live/b/z`, then widens +to everything while its sessions and the publisher's flap. Its reader sees +`live/b/z` start, go offline, start, and END, where the claim only moved up +to `live/b`: + +```rust +let (origin, r1, r2, client) = (0, 1, 2, 3); +let mut sim = Sim::new(4); +sim.leaf(origin); +sim.leaf(client); +sim.join(origin, r1, 1); +sim.join(origin, r2, 1); +sim.join(r1, r2, 1); +for (relay, scope) in [(r1, "live/a/"), (r2, "live/b/")] { + let grant = Grant { subject: "local", scope }; + sim.link(client, relay, SimLink::cost(1).grant(grant), SimLink::cost(1)); +} +sim.originate(origin, 7, 0); +assert!(sim.run()); + +sim.announce(origin, "live/", 0, None); +sim.settle(1); +sim.deliver(r1, origin); +sim.settle(1); +sim.disconnect(client, r1); +sim.settle(1); +sim.disconnect(origin, r1); +sim.settle(1); +sim.deliver(r2, r1); +sim.settle(1); +sim.disconnect(origin, r2); +sim.settle(1); +sim.connect(origin, r1); +sim.settle(1); +sim.watch(client, "live/b/z"); +sim.settle(1); +sim.disconnect(origin, r1); +sim.settle(1); +sim.announce(origin, "live/", 1, None); +sim.settle(1); +sim.connect(origin, r2); +sim.settle(1); +sim.watch(client, ""); +sim.settle(1); +sim.disconnect(client, r2); +sim.settle(1); + +sim.connect(origin, r1); +sim.connect(client, r1); +sim.connect(client, r2); +assert!(sim.run()); +assert_eq!(false_ends(&sim.events), 0); +``` + +`show` in `route/mod.rs` reports a claim that left as offline, not ended, +when a claim above it with the same item is still shown. Here that check +missed. Find why before changing it: the order the widened stream's first +batch and the session close arrive in is the likely cause. + +- Shrink further if it helps, then fix the cause and keep the case as a + test. +- Give the sweep a way to run more scenarios than the gate's 150 without + editing it, and say in the PR how many it ran clean. +- [The interest set](/quest/m2/cluster-routing/routes/interest-set.md) + replaces "an interest change opens a new stream" with an edit in place. If + it lands first, run the case and the 20,000 scenarios on it, and delete + this quest if both are clean. + +Public API: none. Wire: none. + +## Related + +- [The route model and router take an interest as an edited set of prefixes](/quest/m2/cluster-routing/routes/interest-set.md) - rewrites interest changes, and may remove the case diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index 66f04a670f2..c8baa12eebf 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -29,7 +29,8 @@ Decided 2026-11-08 (maintainer, /quest-plan): advertise a warm route for it. This revises Cluster routing's non-goal of cache-dependent route prices. (rejected: per-broadcast forwarding with a cache-only subscribe and no advertisement; keeping routing cache-agnostic) -2. ✅ **A warm route is per neighbor and travels one hop.** It goes only to +2. ✅ **A warm route is per neighbor and travels one hop.** Reopened by + item 8. It goes only to direct neighbors and is never forwarded. B is warm toward N only while B has demand besides N's (a local reader or another downstream session), and withdraws once that demand has been gone for the copy's existing @@ -48,7 +49,8 @@ Decided 2026-11-08 (maintainer, /quest-plan): the skipped relay pulls for its own reader, its neighbor joins it. With uniform prices both cases tie and fall to the hash. (rejected: a warm and a cold price per link; a warm price the carrier seeds) -4. **Eligibility, which the proof must hold.** A relay may pull broadcast P +4. **Eligibility, which the proof must hold.** Reopened by item 8. A relay + may pull broadcast P through neighbor N only when N's advertised route to the request's pinned origin has a newer seqno, or an equal seqno with `(cold metric, hash(P, N))` below `(feasibility distance, hash(P, self))`. @@ -74,6 +76,22 @@ Decided 2026-11-08 (maintainer, /quest-plan): accepting a SETUP extension). The implementation stays in m3. This replaces item 6's rule that choosing lite-07 went back to the maintainer. +8. Reopened 2026-11-11 (maintainer, /quest-plan on the route redesign): + items 2 and 4. In the maintainer's words: "I think we need warm routes + propagating, that decision was made based on this flawed setup." ✅ **A + warm route is per broadcast and propagates**, for as long as it is + cheaper than the route its publisher's broadcasts share + ([epic](/quest/m2/cluster-routing/routes/README.md) Plan item 26). A + broadcast no relay carries rides the shared route and costs no message of + its own. Item 4 rests safety on Babel's distance, which cannot let a relay + join a copy at a neighbour that is no closer to the publisher: the join + the Goal wants, and why item 2 stopped at one hop. What keeps a warm route + free of loops and stale copies is + [the prototype](/quest/m2/cluster-routing/routes/stale-copies.md)'s to + measure, and the proof then holds that rule. Items 1 and 3 stand. + [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) proves items + 2 and 4 as first decided. + Prior art: warm/cold route prices ([moq-dev/moq#2925](https://github.com/moq-dev/moq/pull/2925)) were deleted by prefix routes ([moq-dev/moq#3225](https://github.com/moq-dev/moq/pull/3225)), diff --git a/quest/m3/route-keep-next-hop.md b/quest/m3/route-keep-next-hop.md index 84f80f3cb95..2c5c3120709 100644 --- a/quest/m3/route-keep-next-hop.md +++ b/quest/m3/route-keep-next-hop.md @@ -7,9 +7,9 @@ on it when another neighbour offers a route of the same rank: the same drain state, priority, and metric. Today the lower key wins, so a relay moves for nothing, and since routes name the nodes they crossed ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)) every relay -downstream is sent a ROUTE_UPDATE for it. A test counts route messages for a -link coming up beside an equal one and holds the count where it was before -the link. +downstream is sent a ROUTE_UPDATE for it. A test brings a link up beside an +equal one and counts the route messages sent on every other session: none. +The new link's own first advertisements do not count. ## Plan @@ -33,8 +33,15 @@ selection. keep the current next hop per identity and prefer it among equals. Mirror it in `RouteHandoff.tla`, whose `Best` picks the lowest key, as the `verify-route` skill says. +- Two routes of the same rank can still differ in whether they are + down-only, which decides whether an upstream neighbour is sent the route + at all. Stay put only among routes that agree on it, so a relay on a + down-only route still moves to an equal one it may export, and test that. - Check that every router test that names the next hop it expects still means what it says, rather than editing expectations to match. +- [The prototype](/quest/m2/cluster-routing/routes/stale-copies.md) may + replace how a route is selected. If it has, apply this to the rule that + stands. - Measure with `Sim`'s `hop_updates` count and, if it has landed, `mesh_sweep.rs`'s benchmark, before and after. From a0060f5eaa59b08a2ae4e6f8581398774f9e02f3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 12:51:18 -0800 Subject: [PATCH 3/4] quest: the scope model keeps the lagging-neighbour case A relay that learned a withdrawal from a route that then ended must not show the claim again from a route still at the older revision. Co-Authored-By: Claude --- quest/m2/cluster-routing/routes/scope-model.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/quest/m2/cluster-routing/routes/scope-model.md b/quest/m2/cluster-routing/routes/scope-model.md index a31ed03572a..90d9a11deb4 100644 --- a/quest/m2/cluster-routing/routes/scope-model.md +++ b/quest/m2/cluster-routing/routes/scope-model.md @@ -61,6 +61,11 @@ why: - Keep the counterexample that broke the first plan: two relays behind a scoped relay must not keep a wider route alive through each other after its path closed. +- Keep the lagging neighbour: a relay learns a withdrawal from one route, + that route ends, and a second route still carries the claim at the older + revision. The relay must not show the claim again + (`an_older_complete_snapshot_revives_no_known_withdrawal`). Give it a + config, with the second route covering the same prefix and a wider one. Work: From 3e70b8dcb9465fc3ec5c75abe9e38df766fefecc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 13:03:00 -0800 Subject: [PATCH 4/4] quest: answer the review of the rewritten route plan A claim is shown only while the newest known revision still has it, a revival is within one life of a claim, and Babel's distance rules out a copy of the relay's own route, not every stale neighbour. Co-Authored-By: Claude --- quest/m2/cluster-routing/routes/handoff-revival.md | 7 ++++--- quest/m2/cluster-routing/routes/scope-model.md | 3 ++- quest/m2/cluster-routing/routes/scope.md | 3 ++- quest/m2/cluster-routing/routes/stale-copies.md | 4 +++- 4 files changed, 11 insertions(+), 6 deletions(-) diff --git a/quest/m2/cluster-routing/routes/handoff-revival.md b/quest/m2/cluster-routing/routes/handoff-revival.md index 716df8fd7d9..04cbe0ff62a 100644 --- a/quest/m2/cluster-routing/routes/handoff-revival.md +++ b/quest/m2/cluster-routing/routes/handoff-revival.md @@ -3,9 +3,10 @@ ## Goal `sweep_handoff` in `rs/moq-net/src/route/tests/handoff.rs` reports no revival -over 20,000 scenarios: no reader sees a claim start again after it saw the -claim end, unless it forgot the claim in between. The scenario that revives -one today is a plain test. +over 20,000 scenarios: no reader sees one life of a claim start again after +it saw that life end, unless it forgot the claim in between. A publisher that +withdraws a claim and announces it again starts a new life, which is no +revival. The scenario that revives one today is a plain test. ## Plan diff --git a/quest/m2/cluster-routing/routes/scope-model.md b/quest/m2/cluster-routing/routes/scope-model.md index 90d9a11deb4..31f70e0165e 100644 --- a/quest/m2/cluster-routing/routes/scope-model.md +++ b/quest/m2/cluster-routing/routes/scope-model.md @@ -8,7 +8,8 @@ the prefix it covers, carries its own path, and is believed by a relay it did not cross. TLC shows on it: - A relay shows a claim exactly while it holds a route that covers the claim - and did not cross the relay, at the newest revision the relay knows. + and did not cross the relay, and the newest revision the relay knows still + has the claim. It shows the claim as it is at that revision. - Once nothing is in flight, no relay shows a claim that no path from its publisher carries. - A change of path or seqno alone starts, ends, and changes no claim a reader diff --git a/quest/m2/cluster-routing/routes/scope.md b/quest/m2/cluster-routing/routes/scope.md index 02b0d10b795..97280cf01a3 100644 --- a/quest/m2/cluster-routing/routes/scope.md +++ b/quest/m2/cluster-routing/routes/scope.md @@ -5,7 +5,8 @@ `route::Router` keys a route by its publisher and the prefix it covers, and believes a route that did not cross it, as [the model](/quest/m2/cluster-routing/routes/scope-model.md) settled. A relay -shows a claim exactly while such a route covers it, two relays never keep a +shows a claim exactly while such a route covers it and the newest revision +the relay knows still has the claim, two relays never keep a dead route alive through each other, a change of path or seqno alone changes no claim a reader sees, and a relay behind a scoped neighbour reaches what a farther one offers. The `residual_` tests diff --git a/quest/m2/cluster-routing/routes/stale-copies.md b/quest/m2/cluster-routing/routes/stale-copies.md index 98ecbb58bac..d3c6356fea2 100644 --- a/quest/m2/cluster-routing/routes/stale-copies.md +++ b/quest/m2/cluster-routing/routes/stale-copies.md @@ -87,7 +87,9 @@ Settled: - **Distance, as today.** A relay uses a neighbour only when the neighbour's cost is below the lowest cost the relay ever advertised. Anything routed through the relay costs more than that, so such a - neighbour cannot be stale. It never loops, even for a moment. It refuses + neighbour's route is no copy of the relay's own. It can still be stale + another way: a neighbour whose path closed stays usable until its + withdrawal arrives. It never loops, even for a moment. It refuses good routes: a longer way round waits for the publisher to advance its seqno, and a relay cannot join a copy carried by a neighbour that is no closer to the publisher, which is the join cache-aware routing wants.