From b45982504a5595fc74f659bf9de07ddc0b0040e0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 03:14:03 -0800 Subject: [PATCH 01/12] chore(quest): claim quest/m2/cluster-routes/flapping-publisher.md Quest-Claim: 33d40786d1a7732f From b72e395ebc88fc8a0374701eb37d17fe0e98464b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 05:55:32 -0800 Subject: [PATCH 02/12] feat(route): doubt spares a route that begins at the doubted node A dual-homed publisher whose link to one relay flaps faster than the doubt timer stays routed at every relay through its other link. The model shows the cost: a publisher leaving three or more relays at once can loop a request until the news arrives (Homes.cfg, LoopsLostThePublisher), and two publisher links failing within one timer can drop a claim (TwoHomes.cfg). Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 13 +- quest/m2/cluster-routes/README.md | 6 +- quest/m2/cluster-routes/flapping-publisher.md | 74 +++------ quest/m2/cluster-routes/fresh-publisher.md | 51 ------ rs/moq-net/src/route/path.rs | 18 ++- rs/moq-net/src/route/tests/compare.rs | 5 +- rs/moq-net/src/route/tests/flap.rs | 146 ++++++++++++++++++ rs/moq-net/src/route/tests/mod.rs | 12 ++ rs/moq-net/verify/route/Flap.cfg | 37 +++++ rs/moq-net/verify/route/Homes.cfg | 37 +++++ rs/moq-net/verify/route/Loop.cfg | 1 + rs/moq-net/verify/route/Ring.cfg | 1 + rs/moq-net/verify/route/RouteHandoff.cfg | 1 + rs/moq-net/verify/route/RouteHandoff.tla | 49 +++++- rs/moq-net/verify/route/Twice.cfg | 1 + rs/moq-net/verify/route/TwoHomes.cfg | 39 +++++ 16 files changed, 371 insertions(+), 120 deletions(-) delete mode 100644 quest/m2/cluster-routes/fresh-publisher.md create mode 100644 rs/moq-net/src/route/tests/flap.rs create mode 100644 rs/moq-net/verify/route/Flap.cfg create mode 100644 rs/moq-net/verify/route/Homes.cfg create mode 100644 rs/moq-net/verify/route/TwoHomes.cfg diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index c26aa3e6551..11ac6209c3e 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -5,7 +5,7 @@ description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteH # Verify the route rule -`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. +`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires, keeping relays off routes through it but not off a route that begins at it. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. ## Rules @@ -26,8 +26,10 @@ Each rule is an INVARIANT; the harm says why it matters. | `QuietIsSettled` | Once nothing is in flight and every timer fired: each relay shows exactly the claims a path of open sessions carries to it, available, and a request for each reaches the publisher; no relay holds, keeps a stale route, or remembers a dead hop. | A hold never ends, a claim is shown that no path carries, or a relay behind a scoped neighbour never reaches what a farther one offers. | | `Budget` | From one quiet point to the next, the messages sent stay within 4 each way over each configured session per prefix and per claim, for each external change. | One change sets off announce spam, or routing never goes quiet. | | `RequestsLoopFree` | A request never loops. | A request cycles through relays instead of reaching the publisher. | +| `LoopsLostThePublisher` | A request that loops goes round relays whose routes each begin on a closed session of the publisher. | A loop forms while the publisher is still reachable from the relays on it. | +| `Routed` | A relay that knows a route over open sessions crossing no relay it doubts uses some route. | Doubt about the publisher alone darkens a relay with a way to it, as when one of a dual-homed publisher's links flaps. | -Every config checks `RequestsLoopFree`; `Loop.cfg` pins the three-failure regression. +Every config but `Homes.cfg` checks `RequestsLoopFree`; `Loop.cfg` pins the three-failure regression. `Homes.cfg` checks `LoopsLostThePublisher` instead: doubt spares a route that begins at the doubted node, so a publisher leaving three relays at once can send a request round them until the news arrives. `Flaky` names the links that open and close; `Flap.cfg`, `TwoHomes.cfg`, and `Homes.cfg` narrow it to the publisher's own (`OwnLinks`) or one of them (`FirstLink`). No config checks a liveness property: `Spec` has no fairness, so TLC does not show that a hold ends. That one does is an argument over what TLC does check. `Budget` bounds the messages a burst sends, each delivery takes one message off a queue, and a doubt starts only at a close or a delivery, so a burst is a finite number of steps. A delivery is enabled while a queue holds a message and a timer may fire once no news about its node is in flight, so a burst left alone reaches the state with nothing in flight and no doubt. `QuietIsSettled` says no relay holds there. @@ -79,6 +81,9 @@ change; faster checkers are separate work. | `Drain.cfg` | The publisher joined to two relays joined to each other; both hear its GOAWAY. Ranking only a relay's own drain exceeded `Budget`. | 1 claim change, 1 session change, GOAWAY on each session | 648,218 | two minutes | | `DrainRing.cfg` | The publisher joined to three relays in a ring that is cheap one way round (`Ring`); all hear its GOAWAY. Without the DRAIN news and its doubt, a request looped round the ring with no session closed, breaking `LoopsAreStale`. Simulated. | 1 claim change, GOAWAY on each session | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | | `Prefer.cfg` | The same triangle, where both relays prefer the link between them. Ranking only a relay's own link priority exceeded `Budget`. | 1 claim change, 1 session change | 14,630 | seconds | +| `Flap.cfg` | The publisher on relays 1 and 2, joined to each other, and relay 3 behind both; only the publisher's link to relay 1 opens and closes. Under the replaced rule, where doubt also kept relays off routes beginning at the publisher, `Routed` failed at once. Simulated. | 1 claim change, 4 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | +| `TwoHomes.cfg` | The same, where both of the publisher's links open and close. Drops `Stable`: a relay can take an offer older than the timer and drop a claim it showed. Simulated. | 1 claim change, 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | +| `Homes.cfg` | The publisher on each of three relays in a ring that is cheap one way round, leaving all three. `RequestsLoopFree` fails within seconds; checks `LoopsLostThePublisher`. Simulated. | 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | | `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, seed 1 | a few minutes | Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. @@ -89,13 +94,13 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on th ## Coverage -Sync commit: `31b215b112fb649cc3f63d4de3c76b5e87769a06`, the base of the scoped path router in [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622). Run `git log 31b215b11.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `12537d1c2272c39db154fbd210bf60b8f438da4d`, the base of [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734), which made doubt spare the node a route begins at. Run `git log 12537d1c2.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model; the comparison-only `Rule::ORIGIN` lets doubt block a route that begins at the doubted node too, which `Flap.cfg` fails with `Doubted` defined as it was. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | | `rs/moq-net/src/route/path.rs` `Router::run`, `Router::select`, `Router::retain` | `Select` plans live routes and their reflected sources before `SelectKey` decides holds. | | `rs/moq-net/src/route/path.rs` `Offer::rank`, `Router::goaway`, `Router::drains`, `Router::report_draining`, `Router::dead` (`Dead::Draining`) | `Rank`, `Below`, `PathPrio`, `CostAt`, `GoAway`, `MarkDrain`, `Credited`, `ReportDrain`, `Recv` | -| `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted` | +| `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted`, `RelayDoubted`: a path's first node does not count | | `rs/moq-net/src/route/path.rs` `Router::src` | `SrcKey` | | `rs/moq-net/src/route/path.rs` `Router::want`, `Router::advertise`, `Router::report` | `Want`, `Advertise`, `Report` | | `rs/moq-net/src/route/path.rs` `Router::route`, `Router::batch`, `Router::dead`, `Router::fail`, `Router::disconnect`, `Router::connect`, `Router::change` | `Recv`, `Fail`, `Disconnect`, `Connect`, `ChangeClaim` | diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index 2e7e945cab0..a2ca493760c 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -49,11 +49,13 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout - A hop names a relay and the session it learned the route on. A relay uses the cheapest route whose path does not name it, at once. - A relay that loses a session names that hop, and every relay drops every route through it, whoever offered it. One message per session covers every publisher and lists the routes that moved to another path. - A relay with no route left holds its claims, unavailable, until each neighbour that learned the route from it has answered: with another way, or with nothing once those behind it have answered. Then it lets go. - - A closed session casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time. No relay moves to a route through that node until a timer lifts the doubt. The route a relay already uses is exempt. + - A closed session casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time. No relay moves to a route through that node until a timer lifts the doubt. The route a relay already uses is exempt, and so is a route that begins at that node (below). The maintainer delegated these decisions to the coordinator while away, recorded 2026-11-13: "The user is away; the coordinator takes recommendations on their behalf, open to change in review." - Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. [The fresh-publisher exemption](/quest/m2/cluster-routes/fresh-publisher.md) is modeled and measured separately before deciding whether to adopt it. + Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was to be modeled and measured separately; the decision below covers it. + + Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](/quest/m2/cluster-routes/flapping-publisher.md)): ✅ **doubt blocks a route through a node, not one that begins at it.** A publisher on two relays whose link to one flaps faster than the timer stays routed at every relay through the other: in 400 churn cases (120,000 steps), relays that knew a way but had no route fell from 162,654 to 74,945 observations, with no loop in either, for 22,828 route messages instead of 8,785 and 11,450 stale-route observations instead of 3,312, since relays now move where they used to wait. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`). With two homes no request loops (`TwoHomes.cfg`), but when both links fail within one timer a relay can take an offer older than the timer and drop a claim it showed, where the replaced rule held it dark (`TwoHomes.cfg` drops `Stable`). The exemption also covers a publisher's own new direct session, which the fresh-publisher quest proposed exempting alone, so that quest is deleted. Rejected: keeping timer-only doubt; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waits on [the switch](/quest/m2/cluster-routes/switch.md); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good. Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). diff --git a/quest/m2/cluster-routes/flapping-publisher.md b/quest/m2/cluster-routes/flapping-publisher.md index 25e70f59aef..9e26c1cff55 100644 --- a/quest/m2/cluster-routes/flapping-publisher.md +++ b/quest/m2/cluster-routes/flapping-publisher.md @@ -1,68 +1,34 @@ -# [L] A dual-homed publisher stays reachable while one link flaps +# [S] A dual-homed publisher stays reachable on real sessions while one link flaps ## Goal -A publisher connected to two relays whose link to one of them closes and -reopens faster than the doubt timer stays reachable at every relay through -its other link, the whole time. The path rule's doubt still keeps relays -off routes through a dead relay until its timer fires. +On real lite-07 sessions, a publisher connected to two relays whose link to +one of them closes and reopens faster than the doubt timer stays reachable at +every relay through its other link, the whole time. The draft says doubt +spares a route that begins at the doubted node. ## Plan -Decided 2026-11-13 (maintainer, /quest-plan on the BGP failure modes): ✅ fix -it, checked in TLA+ as well as on real sessions (rejected: pinning today's -behaviour as an accepted cost). +[moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) settled the rule +([epic](/quest/m2/cluster-routes/README.md) Plan item 27, decided 2026-11-14): +doubt blocks a route through a node, not one that begins at it. The model +checks it (`Flap.cfg`, `TwoHomes.cfg`, `Homes.cfg`, and `Routed`), the path +router applies it, and `route/tests/flap.rs` pins it on the router. Sessions +still run Babel's distance, so what is left waits on the switch: -The bug, as the path rule stands ([epic](/quest/m2/cluster-routes/README.md) -Plan item 27): a closed session casts doubt on the node at its far end, and -every relay that hears the DEAD doubts that node. When the closed session is -the publisher's own, the doubted node is the publisher, and every route to -its claims names it, the route through its healthy link included. A relay -that was using the closed link cannot move to the healthy one, and each flap -opens a new session whose close extends the doubt. While the link keeps -flapping faster than the timer, those relays have no route although the -publisher was never unreachable. [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622) -measured the general cost: 2.87× more reachable-but-unrouted observations -under timer-only doubt. - -✅ Model first: doubt blocks a route that passes through a node, not one that -ends at it as its origin (rejected: capping a doubt at one timer from the -first close, which still leaves a gap after each flap; modeling both). Doubt -exists because a dead relay is reported one neighbour at a time, so a relay -could move onto a stale route through it and loop, and a route cannot loop -through its own endpoint. The model has to show what the rule gives up: when -the origin really dies, a relay may move onto a stale route to it that a -neighbour's DEAD has not yet withdrawn. Check that such a route is still -chased out (`StaleIsChased`) and that requests stay loop-free -(`RequestsLoopFree`). Decide in the PR what counts as the origin when a -relay originates a route for a broadcast it carries -([cache-aware routing](/quest/m3/cache-routing/README.md)). - -`fresh-publisher.md`, added by #8622, exempts only a publisher's own new -direct session from doubt, so it never helps a relay further away. If this -rule holds, it covers that case too: fold that quest into this one or delete -it, whichever has landed first. - -Work: - -- `rs/moq-net/verify/route/RouteHandoff.tla`: a config with a publisher on - two relays and flaps on one link, and a property that a relay with a path - of sessions that never closed to the publisher routes to it once that path - is known. Run the existing configs under the new rule. -- The path router (`route::path`, from - [the router](https://github.com/moq-dev/moq/pull/8622)) applies the rule, with the - model's counterexamples reproduced there first. - A fixture on real sessions with mocked time: the dual-homed publisher with one link flapping at half the timer, where a subscriber at a far relay receives every group while it flaps. -- Re-run #8622's reconnect and churn measurements under the new rule and - report reachable-but-unrouted observations, stale routes, and loops beside - timer-only doubt. -- Update the epic's Plan item 27, the draft's Routing section if - [the switch](/quest/m2/cluster-routes/switch.md) has landed, and the - `verify-route` skill. +- The draft's Routing section: doubt keeps a relay off a route through the + far end of a closed session, but not off one that begins there, and a + publisher on three or more relays that leaves them all at once can loop a + request until the news arrives. + +## Required + +- [lite-07 sessions run the path router](/quest/m2/cluster-routes/switch.md) - sessions that run the path rule, and the draft's Routing section to amend ## Related - [A relay that refuses a forwarded route drops the route, not the session](/quest/m1/refuse-route.md) - another failure mode from the same BGP comparison -- [A request loop heals once routes settle](/quest/m2/cluster-routes/request-loop.md) - another failure mode from the same BGP comparison +- [A request loop heals once routes settle](/quest/m2/cluster-routes/request-loop.md) - heals the loop a publisher leaving three relays at once can cause diff --git a/quest/m2/cluster-routes/fresh-publisher.md b/quest/m2/cluster-routes/fresh-publisher.md deleted file mode 100644 index 10e5a978847..00000000000 --- a/quest/m2/cluster-routes/fresh-publisher.md +++ /dev/null @@ -1,51 +0,0 @@ -# [L] Model and measure a fresh publisher's doubt exemption - -## Goal - -Determine whether a fresh authenticated direct publisher session can recover -its own route before the one-second doubt expires, without admitting a stale -return or a request loop. Compare it with timer-only doubt before deciding -whether to adopt it. - -## Plan - -Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, -while the maintainer was away): -timer-only doubt is the baseline in -[moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622), and this exemption -is modeled and measured separately. Early collection is rejected because it -permits the request loop recorded by `Loop.cfg`. These choices remain open -to change in review. - -The fixed 24-relay reconnect cases take 1,050 ms in a clique and 1,875 ms in -a ring, including propagation at 25 ms per hop, versus 50 ms and 325 ms for -the distance rule. Both path policies wait because heard routes still name -the doubted publisher. The exemption could remove that wait for a publisher -returning directly, but not for a relay presenting somebody else's route. - -Model what authenticates the publisher and binds the offer to the new direct -session, including delayed offers from the old session, reconnects, and -concurrent link failures. Keep the timer-only model as the control. Reproduce -counterexamples on the router and measure the same reconnect and churn -scripts on both policies, reporting traffic, request recovery, stale routes, -and reachable but unrouted observations. Preserve the baseline's safety -checks; a faster reconnect alone does not justify the exemption. - -Recommend limiting any exemption to the new authenticated publisher's own -direct offer. A forwarded offer or a new transport session alone does not -establish that the origin is alive. Record the evidence and adoption decision -before changing the default, and mirror an adopted rule in Rust, JS, and the -draft wherever the wire switch has already landed. - -## Waiting - -- Maintainer: after the model and measurements, adopt the narrowly authenticated - direct-publisher exemption or retain timer-only doubt. Recommend adoption only - if it preserves the safety checks and improves the measured reconnect cases; - otherwise retain the baseline. An agent can gather that evidence, but adopting - an exception changes the coordinator's accepted routing policy and needs an - explicit decision. - -## Related - -- [lite-07 sessions run the path router](/quest/m2/cluster-routes/switch.md) - supplies the authenticated session boundary when it puts the rule on the wire diff --git a/rs/moq-net/src/route/path.rs b/rs/moq-net/src/route/path.rs index 0c318bfa13c..2447380ff64 100644 --- a/rs/moq-net/src/route/path.rs +++ b/rs/moq-net/src/route/path.rs @@ -20,7 +20,10 @@ //! holds only the path it last sent, so holds never close a cycle. A closed //! session casts doubt on the node at its far end, since a dead relay is reported //! one neighbour at a time: no node moves to a route through it until a timer -//! lifts the doubt. There is no distance, seqno, route request, or echo. +//! lifts the doubt. A route that begins at the doubted node is exempt, so a +//! publisher stays reachable through its other links while one flaps; when it +//! leaves three or more relays at once, requests can go round them until the +//! news arrives. There is no distance, seqno, route request, or echo. //! //! A route covers a prefix: the one its receiver asked for, or a narrower one a //! node on its path asked for. A stream answering a prefix carries the sender's @@ -180,6 +183,9 @@ pub(crate) struct Doubt { pub timer: Duration, #[cfg(test)] pub lift: Lift, + /// The replaced comparison rule: doubt also blocks a route that begins at the node. + #[cfg(test)] + pub origin: bool, } impl Doubt { @@ -188,6 +194,8 @@ impl Doubt { timer: Duration::from_secs(1), #[cfg(test)] lift: Lift::Timer, + #[cfg(test)] + origin: false, }; } @@ -1295,9 +1303,15 @@ impl Router { path.iter().any(|hop| self.draining.contains(&hop.node)) } - /// Whether `path` crosses a node in doubt. + /// Whether `path` crosses a node in doubt. The node a route begins at does not + /// count: a route cannot loop through its own end. fn doubts_path(&self, path: &[Hop]) -> bool { + #[cfg(test)] + let skip = usize::from(!self.doubt.origin); + #[cfg(not(test))] + let skip = 1; path.iter() + .skip(skip) .any(|hop| matches!(self.doubted.get(&hop.node), Some(Suspicion::Doubted(_)))) } diff --git a/rs/moq-net/src/route/tests/compare.rs b/rs/moq-net/src/route/tests/compare.rs index 400997f50a1..26efe9876dd 100644 --- a/rs/moq-net/src/route/tests/compare.rs +++ b/rs/moq-net/src/route/tests/compare.rs @@ -1,6 +1,7 @@ //! One script on both rules: [`Router`], which keeps Babel's distance, and //! [`path::Router`], which keeps its doubt until the timer fires. The ignored -//! benchmarks also compare the rejected early-collection policy. +//! benchmarks also compare the rejected early-collection policy, and the replaced +//! rule where doubt also kept relays off routes to a doubted publisher. //! //! A [`Lab`] builds the same mesh on each, applies one change at a time, and //! delivers in rounds: every message queued when a round starts crosses its link @@ -17,7 +18,7 @@ use super::*; const RULES: [Rule; 2] = [Rule::Distance, Rule::PATH]; /// The rules compared, in the report's order. -pub(super) const COMPARED: [Rule; 3] = [Rule::Distance, Rule::EARLY, Rule::PATH]; +pub(super) const COMPARED: [Rule; 4] = [Rule::Distance, Rule::EARLY, Rule::ORIGIN, Rule::PATH]; /// A queue of messages, or of echoed ROUTE_ENDs, from one node to another. #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] diff --git a/rs/moq-net/src/route/tests/flap.rs b/rs/moq-net/src/route/tests/flap.rs new file mode 100644 index 00000000000..e48fde313bd --- /dev/null +++ b/rs/moq-net/src/route/tests/flap.rs @@ -0,0 +1,146 @@ +//! A publisher's own sessions closing: doubt keeps relays off routes through the +//! far end, but not off a route that begins there. + +use super::*; + +/// The neighbour `node` routes 7 through, following next hops until the +/// publisher: `Ok(true)` at it, `Ok(false)` stopped short, `Err` round a loop. +fn walk(sim: &Sim, node: Node) -> Result> { + let mut seen = vec![node]; + let mut at = node; + loop { + match sim.toward(at, 7) { + None => return Ok(false), + Some(None) => return Ok(true), + Some(Some(next)) if seen.contains(&next) => { + seen.push(next); + return Err(seen); + } + Some(Some(next)) => { + seen.push(next); + at = next; + } + } + } +} + +/// The publisher on relays 1 and 2, and relay 3 behind both, routing through 1. +fn dual_homed(rule: Rule) -> Sim { + let (origin, r1, r2, r3) = (0, 1, 2, 3); + let mut sim = Sim::with(rule, 4); + sim.leaf(origin); + sim.join(origin, r1, 1); + sim.join(origin, r2, 2); + sim.join(r1, r2, 1); + sim.join(r1, r3, 1); + sim.join(r2, r3, 1); + sim.originate(origin, 7, 0); + sim.announce(origin, "live/x", 0, None); + assert!(sim.run()); + assert_eq!(sim.next_hop(r3, 7), Some(r1)); + sim +} + +/// The publisher's link to relay 1 closes and reopens every half doubt timer. +/// Every relay keeps a route that reaches the publisher through relay 2 the +/// whole time, and shows the claim available. +#[test] +fn a_flapping_link_keeps_a_dual_homed_publisher_routed() { + let (origin, r1, r2, r3) = (0, 1, 2, 3); + let half = Doubt::DEFAULT.timer / 2; + let mut sim = dual_homed(Rule::PATH); + for flap in 0..8 { + match flap % 2 { + 0 => sim.disconnect(origin, r1), + _ => sim.connect(origin, r1), + } + // Each step, before any message moves and after each round of them. + loop { + for relay in [r1, r2, r3] { + assert_eq!(walk(&sim, relay), Ok(true), "flap {flap}: relay {relay}"); + assert_eq!(sim.shows(relay, 7, "live/x"), Some(true), "flap {flap}: relay {relay}"); + } + if sim.busy().is_empty() { + break; + } + // One round; whether it went quiet is the loop's test. + sim.settle(1); + } + sim.wait(half); + } + assert!(sim.run()); + assert_eq!(sim.next_hop(r3, 7), Some(r1), "back on the cheaper link"); +} + +/// Under the replaced rule, the relays that used the closed link have no route +/// while the link flaps, though the publisher never left. +#[test] +fn a_flapping_link_darkened_the_publisher_under_the_replaced_rule() { + let (origin, r1, r3) = (0, 1, 3); + let half = Doubt::DEFAULT.timer / 2; + let mut sim = dual_homed(Rule::ORIGIN); + for flap in 0..8 { + match flap % 2 { + 0 => sim.disconnect(origin, r1), + _ => sim.connect(origin, r1), + } + assert!(sim.flow()); + assert_eq!(walk(&sim, r3), Ok(false), "flap {flap}"); + sim.wait(half); + } +} + +/// The publisher on each of three relays in a ring that is cheap one way round, +/// whose sessions all close before any relay hears another's news. +fn ring_of_homes(rule: Rule) -> Sim { + let (origin, r1, r2, r3) = (0, 1, 2, 3); + let mut sim = Sim::with(rule, 4); + sim.leaf(origin); + for relay in [r1, r2, r3] { + sim.join(origin, relay, 1); + } + for (a, b) in [(r1, r2), (r2, r3), (r3, r1)] { + sim.link(a, b, SimLink::cost(1), SimLink::cost(3)); + } + sim.originate(origin, 7, 0); + sim.announce(origin, "live/x", 0, None); + assert!(sim.run()); + for relay in [r1, r2, r3] { + assert_eq!(sim.toward(relay, 7), Some(Some(origin)), "relay {relay} direct"); + } + sim +} + +/// The publisher leaves all three relays at once. Under the path rule each moves +/// onto its cheaper neighbour's direct route, which that neighbour's own close +/// made stale, so requests go round the ring until the news arrives: the cost of +/// keeping a publisher's other links usable. The replaced rule held every relay +/// instead. Either way every relay lets go once the news has spread. +#[test] +fn a_publisher_leaving_every_relay_at_once_loops_until_the_news_arrives() { + let (origin, r1, r2, r3) = (0, 1, 2, 3); + for (rule, loops) in [(Rule::ORIGIN, 0), (Rule::PATH, 3)] { + let mut sim = ring_of_homes(rule); + for relay in [r1, r2, r3] { + sim.disconnect(origin, relay); + } + let looped = [r1, r2, r3] + .into_iter() + .filter(|&relay| walk(&sim, relay).is_err()) + .count(); + assert_eq!(looped, loops, "{}", rule.as_str()); + // One hop of delay: each relay hears its neighbours' news. + sim.settle(1); + for relay in [r1, r2, r3] { + assert!( + walk(&sim, relay).is_ok(), + "{}: relay {relay} after one hop", + rule.as_str() + ); + } + assert!(sim.run()); + for relay in [r1, r2, r3] { + assert_eq!(sim.toward(relay, 7), None, "{}: relay {relay}", rule.as_str()); + } + } +} diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index 85f7f5d9667..6a08949d097 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -11,6 +11,7 @@ mod cache; mod compare; +mod flap; mod handoff; mod paths; mod residual; @@ -49,6 +50,12 @@ impl Rule { /// The comparison rule lifting doubt once no heard route names the node. pub const EARLY: Self = Self::Path(Doubt { lift: Lift::Unnamed, + origin: true, + ..Doubt::DEFAULT + }); + /// The replaced rule, where doubt also blocks every route to a doubted publisher. + pub const ORIGIN: Self = Self::Path(Doubt { + origin: true, ..Doubt::DEFAULT }); @@ -58,6 +65,11 @@ impl Rule { Self::Path(Doubt { lift: Lift::Unnamed, .. }) => "path+early", + Self::Path(Doubt { + lift: Lift::Timer, + origin: true, + .. + }) => "path+origin", Self::Path(Doubt { lift: Lift::Timer, .. }) => "path", } } diff --git a/rs/moq-net/verify/route/Flap.cfg b/rs/moq-net/verify/route/Flap.cfg new file mode 100644 index 00000000000..567b4133a63 --- /dev/null +++ b/rs/moq-net/verify/route/Flap.cfg @@ -0,0 +1,37 @@ +\* The publisher (0) on relays 1 and 2, joined to each other, and relay 3 behind +\* both. Only the publisher's link to relay 1 opens and closes, so the only doubt is +\* about the publisher: its link to relay 1 flaps while relay 2's stays up, and +\* every relay that knows the way through relay 2 keeps a route the whole time. +\* Simulate: -simulate num=1000 -depth 150 -seed 1 +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {1, 2}, {1, 3}, {2, 3}} + Dear = {} + Ring = {} + Preferred = {} + Leaving = {} + Scoped = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 4 + Patient = TRUE + Flaky <- FirstLink + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + Routed + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Homes.cfg b/rs/moq-net/verify/route/Homes.cfg new file mode 100644 index 00000000000..9688ad2107f --- /dev/null +++ b/rs/moq-net/verify/route/Homes.cfg @@ -0,0 +1,37 @@ +\* The publisher (0) on each of three relays in a ring that is cheap one way +\* round, leaving all three. Each relay may move onto its cheaper neighbour's +\* direct route before that neighbour's news arrives, so a request can go round +\* the ring (`RequestsLoopFree` fails): only `LoopsLostThePublisher` holds. +\* Simulate: -simulate num=1000 -depth 150 -seed 1 +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {0, 3}, {1, 2}, {2, 3}, {1, 3}} + Dear = {} + Ring = {1, 2, 3} + Preferred = {} + Leaving = {} + Scoped = {} + Claims = {"x"} + Narrow = {} + MaxRev = 0 + MaxFlaps = 3 + Patient = TRUE + Flaky <- OwnLinks + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + LoopsLostThePublisher + Routed + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Loop.cfg b/rs/moq-net/verify/route/Loop.cfg index 5dc279c44d1..97941ab80bb 100644 --- a/rs/moq-net/verify/route/Loop.cfg +++ b/rs/moq-net/verify/route/Loop.cfg @@ -30,5 +30,6 @@ INVARIANTS LoopsAreStale StaleIsChased RequestsLoopFree + Routed QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/Ring.cfg b/rs/moq-net/verify/route/Ring.cfg index c8234dd7ae1..5cfce5fd1b9 100644 --- a/rs/moq-net/verify/route/Ring.cfg +++ b/rs/moq-net/verify/route/Ring.cfg @@ -30,5 +30,6 @@ INVARIANTS LoopsAreStale StaleIsChased RequestsLoopFree + Routed QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/RouteHandoff.cfg b/rs/moq-net/verify/route/RouteHandoff.cfg index 0c5dcdd120d..64f3ae9473c 100644 --- a/rs/moq-net/verify/route/RouteHandoff.cfg +++ b/rs/moq-net/verify/route/RouteHandoff.cfg @@ -30,6 +30,7 @@ INVARIANTS LoopsAreStale StaleIsChased RequestsLoopFree + Routed NoStaleReturn QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index e5f2ec9ae35..340dd1d3691 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -19,8 +19,9 @@ (* - A relay with no route left holds its claims, unavailable, until each *) (* neighbour that learned the route from it has answered. *) (* - A closed session casts doubt on the node at its far end: no relay *) -(* moves to a route through it until a timer lifts the doubt. The timer *) -(* is the step `Lift`, which may fire at any moment. *) +(* moves to a route through it until a timer lifts the doubt. A route *) +(* that begins at it is exempt: it cannot loop through its own end. The *) +(* timer is the step `Lift`, which may fire at any moment. *) (* *) (* Each node's every entry point is one atomic step ending in `Update`, *) (* as in the router. Messages travel on one FIFO per direction. A *) @@ -47,6 +48,12 @@ CONSTANTS Patient \* TRUE: a doubt lifts only once no news about its node is in flight Relays == Nodes \ {Origin} +\* The links whose sessions open and close. A config may narrow them to the +\* publisher's own (`OwnLinks`) or to one of them (`FirstLink`). +Flaky == Links +OwnLinks == {l \in Links : Origin \in l} +\* The publisher's link to the lowest relay. +FirstLink == {{Origin, CHOOSE r \in Relays : \A x \in Relays : r <= x}} Cost == [l \in Links |-> IF l \in Dear THEN 2 ELSE 1] \* The relay after n round `Ring`, the lowest after the highest. @@ -162,7 +169,12 @@ Live(r) == r.on /\ ~r.held ----------------------------------------------------------------------------- (* Selection on node n's state s (path.rs: `select`, `pending`, `hopes`). *) -Doubted(s, path) == \E i \in DOMAIN path : path[i][1] \in s.doubt +\* The path crosses a relay in doubt. Its first node, where the route begins, +\* does not count: a route cannot loop through its own end. +RelayDoubted(s, path) == \E i \in DOMAIN path : i > 1 /\ path[i][1] \in s.doubt +\* Doubt keeps a node off a route through a doubted relay, not off one that +\* begins at a doubted publisher. +Doubted(s, path) == RelayDoubted(s, path) \* The path crosses a node this node knows sent GOAWAY. Drains(s, path) == \E i \in DOMAIN path : path[i][1] \in s.draining @@ -673,7 +685,8 @@ Connect(l) == rs == [x \in l |-> LET y == CHOOSE y \in l : y # x IN Update(x, Closed(st[x], y), OV, up1, gen1, ended[x], NoReport, FALSE)] - IN /\ l \notin up + IN /\ l \in Flaky + /\ l \notin up /\ flaps < MaxFlaps /\ up' = up1 /\ gen' = gen1 @@ -694,7 +707,8 @@ Disconnect(l) == THEN Update(x, s, OV, up1, gen, ended[x], NoReport, FALSE) ELSE Update(x, Fail(x, s, hop, None, {}, FALSE), OV, up1, gen, ended[x], DeadReport(hop), FALSE)] - IN /\ l \in up + IN /\ l \in Flaky + /\ l \in up /\ flaps < MaxFlaps /\ up' = up1 /\ flaps' = flaps + 1 @@ -826,6 +840,31 @@ LoopsAreStale == \* A request never loops. It does while news is in flight: `Loop.cfg` pins it. RequestsLoopFree == \A n \in Relays, c \in Claims : Request(n, c) # "loop" +\* A request that loops goes round relays whose routes each begin on a closed +\* session of the publisher: it left every one of them, and their news is in +\* flight. Moving onto a route that begins at a doubted node allows that loop; +\* doubt about a relay still keeps every other one out. +LoopsLostThePublisher == + \A n \in Relays, c \in Claims : + Request(n, c) = "loop" => + LET w == Walk(n, IF c \in Narrow /\ Live(st[n].rt["a"]) THEN "a" ELSE "root", None, Cardinality(Nodes)) + \* The positions inside a stretch of the walk that comes back to its start. + round == {i \in DOMAIN w : \E j, k \in DOMAIN w : j <= i /\ i < k /\ w[j].node = w[k].node} + IN \A i \in round : Len(w[i].path) >= 2 /\ Gone(w[i].path[2]) + +\* A route a relay knows over sessions that are still open, crossing no relay +\* it doubts. +Fresh(n, m, k) == + /\ Usable(n, st[n], m, k) + /\ ~Stale(st[n].rx[m][k].path) + /\ ~RelayDoubted(st[n], st[n].rx[m][k].path) + +\* Doubt about the publisher alone never leaves a relay without a route: one +\* that knows a fresh route uses some route, so a request stops only at a relay +\* that knows no way on. Which route it uses may still be stale while the news +\* that ends it is in flight, which `StaleIsChased` bounds. +Routed == \A n \in Relays, k \in Keys : (\E m \in Nodes : Fresh(n, m, k)) => Live(st[n].rt[k]) + \* No node may use a route through a hop it once learned had died: after a \* dead hop is forgotten, no stale route returns. It holds only for one failure \* and a patient timer; `StaleIsChased` bounds the rest. diff --git a/rs/moq-net/verify/route/Twice.cfg b/rs/moq-net/verify/route/Twice.cfg index 2b277a48b58..9cf29804494 100644 --- a/rs/moq-net/verify/route/Twice.cfg +++ b/rs/moq-net/verify/route/Twice.cfg @@ -31,5 +31,6 @@ INVARIANTS LoopsAreStale StaleIsChased RequestsLoopFree + Routed QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/TwoHomes.cfg b/rs/moq-net/verify/route/TwoHomes.cfg new file mode 100644 index 00000000000..82c37f7536c --- /dev/null +++ b/rs/moq-net/verify/route/TwoHomes.cfg @@ -0,0 +1,39 @@ +\* Flap.cfg where both of the publisher's sessions open and close. A relay may +\* move onto an offer older than the doubt timer, naming a hop it has already +\* forgotten died, where doubt about the publisher kept it off before. When the +\* next message on that link names a hop it knows died, the relay lets go: +\* claims it showed go offline, then return once another route arrives. The +\* replaced rule kept the relay dark for the whole timer instead, so this +\* config drops `Stable`. +\* Simulate: -simulate num=1000 -depth 150 -seed 1 +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {1, 2}, {1, 3}, {2, 3}} + Dear = {} + Ring = {} + Preferred = {} + Leaving = {} + Scoped = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 3 + Patient = TRUE + Flaky <- OwnLinks + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + Routed + QuietIsSettled + Budget From 8ceade84981f63091cb37d54c534b76d04c7973b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 07:31:06 -0800 Subject: [PATCH 03/12] feat(route): a link that returns during the doubt waits it out Sparing every route that begins at a doubted publisher showed each return of a flapping link to every relay: in two-home churn, 121% more route messages and 69% more claims offline than timer-only doubt. Doubt now still blocks the publisher's own route on a session opened since the doubt began, so the relay it returned to takes it only after the timer and no other relay hears it before then, while the publisher's other links stay usable. The churn benchmark gains a two-home variant. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 9 +- quest/m2/cluster-routes/README.md | 4 +- quest/m2/cluster-routes/flapping-publisher.md | 14 +-- rs/moq-net/src/route/path.rs | 98 +++++++++++++------ rs/moq-net/src/route/tests/compare.rs | 42 ++++++-- rs/moq-net/src/route/tests/flap.rs | 20 +++- rs/moq-net/src/route/tests/mod.rs | 23 +++-- rs/moq-net/verify/route/RouteHandoff.tla | 36 +++++-- rs/moq-net/verify/route/TwoHomes.cfg | 3 +- 9 files changed, 184 insertions(+), 65 deletions(-) diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 11ac6209c3e..db420a6f4de 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -5,7 +5,7 @@ description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteH # Verify the route rule -`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires, keeping relays off routes through it but not off a route that begins at it. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. +`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires, keeping relays off routes through it but not off a route that begins at it, unless that route comes over a relay's own session to it opened since the doubt began. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. ## Rules @@ -27,7 +27,7 @@ Each rule is an INVARIANT; the harm says why it matters. | `Budget` | From one quiet point to the next, the messages sent stay within 4 each way over each configured session per prefix and per claim, for each external change. | One change sets off announce spam, or routing never goes quiet. | | `RequestsLoopFree` | A request never loops. | A request cycles through relays instead of reaching the publisher. | | `LoopsLostThePublisher` | A request that loops goes round relays whose routes each begin on a closed session of the publisher. | A loop forms while the publisher is still reachable from the relays on it. | -| `Routed` | A relay that knows a route over open sessions crossing no relay it doubts uses some route. | Doubt about the publisher alone darkens a relay with a way to it, as when one of a dual-homed publisher's links flaps. | +| `Routed` | A relay that knows a route over open sessions, crossing no relay it doubts and not a link to the publisher that came back during its doubt, uses some route. | Doubt about the publisher alone darkens a relay with a way to it, as when one of a dual-homed publisher's links flaps. | Every config but `Homes.cfg` checks `RequestsLoopFree`; `Loop.cfg` pins the three-failure regression. `Homes.cfg` checks `LoopsLostThePublisher` instead: doubt spares a route that begins at the doubted node, so a publisher leaving three relays at once can send a request round them until the news arrives. `Flaky` names the links that open and close; `Flap.cfg`, `TwoHomes.cfg`, and `Homes.cfg` narrow it to the publisher's own (`OwnLinks`) or one of them (`FirstLink`). @@ -94,13 +94,14 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on th ## Coverage -Sync commit: `12537d1c2272c39db154fbd210bf60b8f438da4d`, the base of [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734), which made doubt spare the node a route begins at. Run `git log 12537d1c2.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model; the comparison-only `Rule::ORIGIN` lets doubt block a route that begins at the doubted node too, which `Flap.cfg` fails with `Doubted` defined as it was. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `12537d1c2272c39db154fbd210bf60b8f438da4d`, the base of [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734), which made doubt spare the node a route begins at. Run `git log 12537d1c2.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model; the comparison-only `Rule::ORIGIN` lets doubt block a route that begins at the doubted node too, which `Flap.cfg` fails with `Doubted` defined as it was, and `Rule::SPARED` lets it block none, even a returning link. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | | `rs/moq-net/src/route/path.rs` `Router::run`, `Router::select`, `Router::retain` | `Select` plans live routes and their reflected sources before `SelectKey` decides holds. | | `rs/moq-net/src/route/path.rs` `Offer::rank`, `Router::goaway`, `Router::drains`, `Router::report_draining`, `Router::dead` (`Dead::Draining`) | `Rank`, `Below`, `PathPrio`, `CostAt`, `GoAway`, `MarkDrain`, `Credited`, `ReportDrain`, `Recv` | -| `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted`, `RelayDoubted`: a path's first node does not count | +| `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted`, `RelayDoubted` (a path's first node does not count), `Returned` | +| `rs/moq-net/src/route/path.rs` `Router::suspect` | `Since`: a doubt on the publisher that begins notes the generation of the relay's own session to it | | `rs/moq-net/src/route/path.rs` `Router::src` | `SrcKey` | | `rs/moq-net/src/route/path.rs` `Router::want`, `Router::advertise`, `Router::report` | `Want`, `Advertise`, `Report` | | `rs/moq-net/src/route/path.rs` `Router::route`, `Router::batch`, `Router::dead`, `Router::fail`, `Router::disconnect`, `Router::connect`, `Router::change` | `Recv`, `Fail`, `Disconnect`, `Connect`, `ChangeClaim` | diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index a2ca493760c..87ca730f969 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -49,13 +49,13 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout - A hop names a relay and the session it learned the route on. A relay uses the cheapest route whose path does not name it, at once. - A relay that loses a session names that hop, and every relay drops every route through it, whoever offered it. One message per session covers every publisher and lists the routes that moved to another path. - A relay with no route left holds its claims, unavailable, until each neighbour that learned the route from it has answered: with another way, or with nothing once those behind it have answered. Then it lets go. - - A closed session casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time. No relay moves to a route through that node until a timer lifts the doubt. The route a relay already uses is exempt, and so is a route that begins at that node (below). + - A closed session casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time. No relay moves to a route through that node until a timer lifts the doubt. The route a relay already uses is exempt, and so is a route that begins at that node, unless it comes over a session to that node opened since the doubt began (below). The maintainer delegated these decisions to the coordinator while away, recorded 2026-11-13: "The user is away; the coordinator takes recommendations on their behalf, open to change in review." Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was to be modeled and measured separately; the decision below covers it. - Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](/quest/m2/cluster-routes/flapping-publisher.md)): ✅ **doubt blocks a route through a node, not one that begins at it.** A publisher on two relays whose link to one flaps faster than the timer stays routed at every relay through the other: in 400 churn cases (120,000 steps), relays that knew a way but had no route fell from 162,654 to 74,945 observations, with no loop in either, for 22,828 route messages instead of 8,785 and 11,450 stale-route observations instead of 3,312, since relays now move where they used to wait. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`). With two homes no request loops (`TwoHomes.cfg`), but when both links fail within one timer a relay can take an offer older than the timer and drop a claim it showed, where the replaced rule held it dark (`TwoHomes.cfg` drops `Stable`). The exemption also covers a publisher's own new direct session, which the fresh-publisher quest proposed exempting alone, so that quest is deleted. Rejected: keeping timer-only doubt; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waits on [the switch](/quest/m2/cluster-routes/switch.md); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good. + Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](/quest/m2/cluster-routes/flapping-publisher.md)): ✅ **doubt blocks a route through a node, not one that begins at it, except the node's own route on a session to it that opened since the doubt began.** A publisher on two relays whose link to one flaps faster than the timer stays routed at every relay through the other, and a publisher that moves to another relay is routed in 50 ms in a 24-relay clique and 325 ms in a ring, where it waited 1,100 and 1,850 ms. The returning link still waits out the timer, so no relay hears it until the relay it returned to takes it, which keeps the timer's damping of a flapping link: in 400 churn cases (120,000 steps) with the publisher on one relay, the rule costs 3% more route messages than timer-only doubt with no fewer relays routed, and with the publisher on two relays it routes relays 14% more often for 22% more route messages, 6% more claims sent, and 3% more claims taken offline. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`; 44 of the two-relay churn's 120,000 steps). With two homes the model finds no loop (`TwoHomes.cfg`), but when both links fail within one timer a relay can take an offer older than the timer and drop a claim it showed, where the replaced rule held it dark (`TwoHomes.cfg` drops `Stable`). This also settles the fresh-publisher quest, which proposed the opposite exemption and is deleted: its measurements showed that taking a returning link at once costs more traffic and claims offline than it saves. Rejected: keeping timer-only doubt, under which a dual-homed publisher's other link is unusable for the whole timer; sparing every route that begins at the doubted node, which routes relays 67% more often with two homes but costs 121% more route messages, 76% more claims sent, and 69% more claims offline, since each return of a flapping link reaches every relay; sparing only links heard before the doubt began, which keeps the damping but rarely helps a relay whose neighbours had not chosen the other link; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waits on [the switch](/quest/m2/cluster-routes/switch.md); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good. Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). diff --git a/quest/m2/cluster-routes/flapping-publisher.md b/quest/m2/cluster-routes/flapping-publisher.md index 9e26c1cff55..08accc57ef5 100644 --- a/quest/m2/cluster-routes/flapping-publisher.md +++ b/quest/m2/cluster-routes/flapping-publisher.md @@ -4,14 +4,15 @@ On real lite-07 sessions, a publisher connected to two relays whose link to one of them closes and reopens faster than the doubt timer stays reachable at -every relay through its other link, the whole time. The draft says doubt -spares a route that begins at the doubted node. +every relay through its other link, the whole time, while the flapping link +waits out the timer each time it returns. The draft states the rule. ## Plan [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) settled the rule ([epic](/quest/m2/cluster-routes/README.md) Plan item 27, decided 2026-11-14): -doubt blocks a route through a node, not one that begins at it. The model +doubt blocks a route through a node, not one that begins at it, except the +node's own route on a session to it opened since the doubt began. The model checks it (`Flap.cfg`, `TwoHomes.cfg`, `Homes.cfg`, and `Routed`), the path router applies it, and `route/tests/flap.rs` pins it on the router. Sessions still run Babel's distance, so what is left waits on the switch: @@ -20,9 +21,10 @@ still run Babel's distance, so what is left waits on the switch: one link flapping at half the timer, where a subscriber at a far relay receives every group while it flaps. - The draft's Routing section: doubt keeps a relay off a route through the - far end of a closed session, but not off one that begins there, and a - publisher on three or more relays that leaves them all at once can loop a - request until the news arrives. + far end of a closed session, and off that node's own route on a session to + it opened since, but not off another route that begins there; a publisher on + three or more relays that leaves them all at once can loop a request until + the news arrives. ## Required diff --git a/rs/moq-net/src/route/path.rs b/rs/moq-net/src/route/path.rs index 2447380ff64..489344cd00a 100644 --- a/rs/moq-net/src/route/path.rs +++ b/rs/moq-net/src/route/path.rs @@ -21,9 +21,11 @@ //! session casts doubt on the node at its far end, since a dead relay is reported //! one neighbour at a time: no node moves to a route through it until a timer //! lifts the doubt. A route that begins at the doubted node is exempt, so a -//! publisher stays reachable through its other links while one flaps; when it -//! leaves three or more relays at once, requests can go round them until the -//! news arrives. There is no distance, seqno, route request, or echo. +//! publisher stays reachable through its other links while one flaps, except on +//! a session to it opened since the doubt began: a link that comes back waits +//! out the timer, which damps its flaps. When a publisher leaves three or more +//! relays at once, requests can go round them until the news arrives. There is +//! no distance, seqno, route request, or echo. //! //! A route covers a prefix: the one its receiver asked for, or a narrower one a //! node on its path asked for. A stream answering a prefix carries the sender's @@ -176,6 +178,19 @@ pub(crate) enum Lift { Timer, } +/// What doubt does to a route that begins at the doubted node. +#[cfg(test)] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum AtOrigin { + /// The replaced comparison rule: it blocks the route, as it does one through the node. + Blocked, + /// The rejected comparison rule: it never blocks the route. + Spared, + /// It blocks the route only on this node's own session to the node, opened + /// since the doubt began. + Own, +} + /// How a closed session's doubt works. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) struct Doubt { @@ -183,9 +198,8 @@ pub(crate) struct Doubt { pub timer: Duration, #[cfg(test)] pub lift: Lift, - /// The replaced comparison rule: doubt also blocks a route that begins at the node. #[cfg(test)] - pub origin: bool, + pub origin: AtOrigin, } impl Doubt { @@ -195,7 +209,7 @@ impl Doubt { #[cfg(test)] lift: Lift::Timer, #[cfg(test)] - origin: false, + origin: AtOrigin::Own, }; } @@ -259,19 +273,23 @@ impl Death { /// A node's timed doubt. Tests also represent early collection before the /// timer retires. -#[derive(Clone, Copy, Debug)] +#[derive(Clone, Debug)] enum Suspicion { - Doubted(Instant), + Doubted { + until: Instant, + /// The last session this node opened before the doubt began. + since: u64, + }, #[cfg(test)] Lifted(Instant), } impl Suspicion { - fn deadline(self) -> Instant { + fn deadline(&self) -> Instant { match self { - Self::Doubted(at) => at, + Self::Doubted { until, .. } => *until, #[cfg(test)] - Self::Lifted(at) => at, + Self::Lifted(at) => *at, } } } @@ -1303,16 +1321,42 @@ impl Router { path.iter().any(|hop| self.draining.contains(&hop.node)) } - /// Whether `path` crosses a node in doubt. The node a route begins at does not - /// count: a route cannot loop through its own end. - fn doubts_path(&self, path: &[Hop]) -> bool { + /// Whether doubt keeps this node off the route `peer` offered with `path`: one + /// through a node in doubt, or the doubted node's own route on a session to it + /// that opened since the doubt began. A route cannot loop through its own end, so + /// the publisher's other links stay usable while one of them flaps; a link that + /// comes back waits out the timer, which damps the flaps, and no other node + /// hears it until this one takes it. + fn doubts_path(&self, peer: N, path: &[Hop]) -> bool { + let doubted = |node: Id| matches!(self.doubted.get(&node), Some(Suspicion::Doubted { .. })); + let Some((first, rest)) = path.split_first() else { + return false; + }; + if rest.iter().any(|hop| doubted(hop.node)) { + return true; + } + let Some(Suspicion::Doubted { since, .. }) = self.doubted.get(&first.node) else { + return false; + }; #[cfg(test)] - let skip = usize::from(!self.doubt.origin); - #[cfg(not(test))] - let skip = 1; - path.iter() - .skip(skip) - .any(|hop| matches!(self.doubted.get(&hop.node), Some(Suspicion::Doubted(_)))) + match self.doubt.origin { + AtOrigin::Blocked => return true, + AtOrigin::Spared => return false, + AtOrigin::Own => {} + } + rest.is_empty() && self.sessions[&peer].number > *since + } + + /// Doubt `node` until `at`, or later if it already is. A doubt that begins notes + /// the last session opened so far, so a session to `node` opened since waits. + fn suspect(&mut self, node: Id, at: Instant) { + if let Some(Suspicion::Doubted { until, .. }) = self.doubted.get_mut(&node) { + *until = (*until).max(at); + return; + } + let until = self.doubted.get(&node).map_or(at, |state| state.deadline().max(at)); + let since = self.opened; + self.doubted.insert(node, Suspicion::Doubted { until, since }); } fn hear(&mut self, key: RouteId, from: N, heard: Heard) { @@ -1404,9 +1448,7 @@ impl Router { if !self.draining.insert(node) { return BTreeSet::new(); } - let at = now.saturating_add(self.doubt.timer); - let at = self.doubted.get(&node).map_or(at, |state| state.deadline().max(at)); - self.doubted.insert(node, Suspicion::Doubted(at)); + self.suspect(node, now.saturating_add(self.doubt.timer)); self.naming(node) } @@ -1475,9 +1517,7 @@ impl Router { } } if far != self.hop { - let at = now.saturating_add(self.doubt.timer); - let at = self.doubted.get(&far).map_or(at, |state| state.deadline().max(at)); - self.doubted.insert(far, Suspicion::Doubted(at)); + self.suspect(far, now.saturating_add(self.doubt.timer)); // Any route through that node may have become unusable. ids.extend(self.naming(far)); } @@ -1528,7 +1568,7 @@ impl Router { let Some(metric) = heard.offer.metric.checked_add(session.link.cost) else { continue; }; - if Some(peer) != using && self.doubts_path(&heard.offer.path) { + if Some(peer) != using && self.doubts_path(peer, &heard.offer.path) { // It may come back once the doubt lifts. continue; } @@ -1644,7 +1684,7 @@ impl Router { heard.state == State::Live && !names(&heard.offer.path, self.hop) && heard.offer.metric.checked_add(self.sessions[peer].link.cost).is_some() - && self.doubts_path(&heard.offer.path) + && self.doubts_path(*peer, &heard.offer.path) }) } @@ -2615,7 +2655,7 @@ impl Router { .doubted .iter() .filter_map(|(node, state)| { - (matches!(state, Suspicion::Doubted(_)) && !self.named(*node)).then_some(*node) + (matches!(state, Suspicion::Doubted { .. }) && !self.named(*node)).then_some(*node) }) .collect(); for node in lifted { diff --git a/rs/moq-net/src/route/tests/compare.rs b/rs/moq-net/src/route/tests/compare.rs index 26efe9876dd..f4a3870cbfd 100644 --- a/rs/moq-net/src/route/tests/compare.rs +++ b/rs/moq-net/src/route/tests/compare.rs @@ -1,7 +1,7 @@ //! One script on both rules: [`Router`], which keeps Babel's distance, and //! [`path::Router`], which keeps its doubt until the timer fires. The ignored //! benchmarks also compare the rejected early-collection policy, and the replaced -//! rule where doubt also kept relays off routes to a doubted publisher. +//! rule where doubt kept relays off every route to a doubted publisher. //! //! A [`Lab`] builds the same mesh on each, applies one change at a time, and //! delivers in rounds: every message queued when a round starts crosses its link @@ -788,8 +788,9 @@ pub(super) struct Churn { /// random meshes of three to five relays where links flap, the publisher's most of /// all, relays restart, and every message is delivered in a random order. Each rule /// sees the same meshes and the same changes at the same steps. Once the churn -/// stops the mesh must settle and be right. -pub(super) fn churn(rule: Rule, cases: std::ops::Range) -> Churn { +/// stops the mesh must settle and be right. With `two_homes` the publisher also +/// joins a second relay, and its flaps pick either link. +pub(super) fn churn(rule: Rule, cases: std::ops::Range, two_homes: bool) -> Churn { let mut total = Churn::default(); for case in cases { let mut script = XorShift(0x0bad_c0de + case as u64 * 0x9e37_79b9); @@ -811,11 +812,18 @@ pub(super) fn churn(rule: Rule, cases: std::ops::Range) -> Churn { }; let mut lab = Lab::new(rule, &mesh); let (node, _) = mesh.publisher(0); + let mut homes = vec![mesh.homes[0]]; + if two_homes { + homes.push((mesh.homes[0] + 1 + script.next(relays as u64 - 1) as usize) % relays); + lab.attach(node, homes[1]); + assert!(lab.settle()); + } let before = lab.net.sent(); for _ in 0..300 { match script.next(40) { 0 | 1 => { let (a, b) = match script.chance(60) { + true if two_homes => (node, homes[script.next(2) as usize]), true => (node, mesh.homes[0]), false => { let (a, b, _) = mesh.links[script.next(mesh.links.len() as u64) as usize]; @@ -864,9 +872,11 @@ pub(super) fn churn(rule: Rule, cases: std::ops::Range) -> Churn { #[test] fn every_rule_settles_after_churn() { for rule in RULES { - let cost = churn(rule, 0..60); - if rule == Rule::Distance { - assert_eq!(cost.loops, 0); + for two_homes in [false, true] { + let cost = churn(rule, 0..60, two_homes); + if rule == Rule::Distance { + assert_eq!(cost.loops, 0); + } } } } @@ -881,7 +891,7 @@ fn benchmark_route_churn() { println!("rule,route_messages,claim_messages"); eprintln!("rule,steps,loops,stale,dark,msgs,items,claims,offline,unavailable"); for rule in COMPARED { - let churn = churn(rule, 0..400); + let churn = churn(rule, 0..400, false); let (rule, steps, sent) = (rule.as_str(), churn.steps, churn.sent); println!("{rule},{},{}", sent.msgs, sent.claims); eprintln!( @@ -891,6 +901,24 @@ fn benchmark_route_churn() { } } +/// The same churn with the publisher on two relays: what keeping its other link +/// usable while one flaps costs, per rule. +#[test] +#[ignore = "a benchmark; Nightly reports it"] +fn benchmark_route_churn_two_homes() { + println!("rule,homes,route_messages,claim_messages"); + eprintln!("rule,steps,loops,stale,dark,msgs,items,claims,offline,unavailable"); + for rule in COMPARED { + let churn = churn(rule, 0..400, true); + let (rule, steps, sent) = (rule.as_str(), churn.steps, churn.sent); + println!("{rule},2,{},{}", sent.msgs, sent.claims); + eprintln!( + "{rule},{steps},{},{},{},{},{},{},{},{}", + churn.loops, churn.stale, churn.dark, sent.msgs, sent.items, sent.claims, sent.offline, sent.unavailable, + ); + } +} + /// Each rule's router carries every change on every shape, and is right whenever /// it settles. #[test] diff --git a/rs/moq-net/src/route/tests/flap.rs b/rs/moq-net/src/route/tests/flap.rs index e48fde313bd..e088a831227 100644 --- a/rs/moq-net/src/route/tests/flap.rs +++ b/rs/moq-net/src/route/tests/flap.rs @@ -72,6 +72,24 @@ fn a_flapping_link_keeps_a_dual_homed_publisher_routed() { assert_eq!(sim.next_hop(r3, 7), Some(r1), "back on the cheaper link"); } +/// A link that comes back while its publisher is in doubt waits out the timer, so +/// a flapping link does not pull every relay back and forth with each flap; the +/// rejected rule that never doubts a route's origin takes it at once. +#[test] +fn a_link_that_comes_back_waits_out_the_doubt() { + let (origin, r1) = (0, 1); + for (rule, back) in [(Rule::PATH, false), (Rule::SPARED, true)] { + let mut sim = dual_homed(rule); + sim.disconnect(origin, r1); + assert!(sim.flow()); + sim.connect(origin, r1); + assert!(sim.flow()); + assert_eq!(sim.next_hop(r1, 7) == Some(origin), back, "{}", rule.as_str()); + assert!(sim.run()); + assert_eq!(sim.next_hop(r1, 7), Some(origin), "{}: after the timer", rule.as_str()); + } +} + /// Under the replaced rule, the relays that used the closed link have no route /// while the link flaps, though the publisher never left. #[test] @@ -119,7 +137,7 @@ fn ring_of_homes(rule: Rule) -> Sim { #[test] fn a_publisher_leaving_every_relay_at_once_loops_until_the_news_arrives() { let (origin, r1, r2, r3) = (0, 1, 2, 3); - for (rule, loops) in [(Rule::ORIGIN, 0), (Rule::PATH, 3)] { + for (rule, loops) in [(Rule::ORIGIN, 0), (Rule::SPARED, 3), (Rule::PATH, 3)] { let mut sim = ring_of_homes(rule); for relay in [r1, r2, r3] { sim.disconnect(origin, relay); diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index 6a08949d097..6a5ffc971a3 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -27,7 +27,7 @@ use std::{ use kio::clock::Instant; -pub(super) use super::path::{Doubt, Lift}; +pub(super) use super::path::{AtOrigin, Doubt, Lift}; use super::*; pub(super) type Node = usize; @@ -50,12 +50,18 @@ impl Rule { /// The comparison rule lifting doubt once no heard route names the node. pub const EARLY: Self = Self::Path(Doubt { lift: Lift::Unnamed, - origin: true, + origin: AtOrigin::Blocked, ..Doubt::DEFAULT }); /// The replaced rule, where doubt also blocks every route to a doubted publisher. pub const ORIGIN: Self = Self::Path(Doubt { - origin: true, + origin: AtOrigin::Blocked, + ..Doubt::DEFAULT + }); + /// The rejected rule, where doubt never blocks a route to a doubted publisher, even + /// on a session to it that opened during the doubt. + pub const SPARED: Self = Self::Path(Doubt { + origin: AtOrigin::Spared, ..Doubt::DEFAULT }); @@ -66,11 +72,16 @@ impl Rule { lift: Lift::Unnamed, .. }) => "path+early", Self::Path(Doubt { - lift: Lift::Timer, - origin: true, + origin: AtOrigin::Blocked, .. }) => "path+origin", - Self::Path(Doubt { lift: Lift::Timer, .. }) => "path", + Self::Path(Doubt { + origin: AtOrigin::Spared, + .. + }) => "path+spared", + Self::Path(Doubt { + origin: AtOrigin::Own, .. + }) => "path", } } } diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index 340dd1d3691..9b9706ed7dc 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -172,9 +172,18 @@ Live(r) == r.on /\ ~r.held \* The path crosses a relay in doubt. Its first node, where the route begins, \* does not count: a route cannot loop through its own end. RelayDoubted(s, path) == \E i \in DOMAIN path : i > 1 /\ path[i][1] \in s.doubt -\* Doubt keeps a node off a route through a doubted relay, not off one that -\* begins at a doubted publisher. -Doubted(s, path) == RelayDoubted(s, path) +\* The generation of n's session to the publisher, or None without a link. +OwnGen(n) == IF {n, Origin} \in Links THEN gen[{n, Origin}] ELSE None +\* The doubted publisher's own route, on a session to it that opened since the +\* doubt began: a link that comes back waits out the timer, damping its flaps. +Returned(n, s, m, path) == + /\ Len(path) = 1 + /\ Origin \in s.doubt + /\ m = Origin + /\ OwnGen(n) > s.since +\* Doubt keeps a node off a route through a doubted relay, and off a returned +\* link, but not off another route that begins at the doubted publisher. +Doubted(n, s, m, path) == RelayDoubted(s, path) \/ Returned(n, s, m, path) \* The path crosses a node this node knows sent GOAWAY. Drains(s, path) == \E i \in DOMAIN path : path[i][1] \in s.draining @@ -211,7 +220,7 @@ Pending(n, s, k) == /\ s.tx[p][j].src = k \* A neighbour offers a route that only doubt keeps this node off. -Hopes(n, s, k) == \E m \in Nodes : Usable(n, s, m, k) /\ Doubted(s, s.rx[m][k].path) +Hopes(n, s, k) == \E m \in Nodes : Usable(n, s, m, k) /\ Doubted(n, s, m, s.rx[m][k].path) \* `select` for one key: the best-ranked usable route, staying on the next hop \* among equals, then the lowest key. Doubt keeps a node from moving to a route, @@ -220,7 +229,7 @@ SelectKey(n, s, k, gens) == LET was == s.rt[k] current == IF was.on THEN was.next ELSE None using == IF Live(was) THEN was.next ELSE None - C == {m \in Nodes : Usable(n, s, m, k) /\ (m = using \/ ~Doubted(s, s.rx[m][k].path))} + C == {m \in Nodes : Usable(n, s, m, k) /\ (m = using \/ ~Doubted(n, s, m, s.rx[m][k].path))} Path(m) == Append(s.rx[m][k].path, <>) Drain(m) == s.rx[m][k].drain \/ m \in s.drain \/ Drains(s, s.rx[m][k].path) R(m) == Rank(Path(m), Drain(m)) @@ -510,6 +519,7 @@ InitNode == [ view |-> [k \in Keys |-> NoView], dead |-> {}, doubt |-> {}, + since |-> None, \* the generation of the session to the publisher when its doubt began wait |-> {}, expiry |-> {}, \* Pending timers belong to individual dead hops. drain |-> {}, \* the neighbours that sent GOAWAY on the session open to them @@ -565,6 +575,10 @@ Unannounce(c) == rev < MaxRev /\ own[c] # None /\ ChangeClaim(c, None) ----------------------------------------------------------------------------- (* A message arrives (path.rs: `recv`, `fail`). *) +\* `suspect`: a doubt on the publisher that begins notes n's session to it, so +\* one opened since waits. +Since(n, s, x) == IF x = Origin /\ Origin \notin s.doubt THEN OwnGen(n) ELSE s.since + \* `fail`: the session `hop` names closed. `from` reported it with what became \* of its own routes through it, or this node saw it close (`from` is None). \* It casts doubt on the node at the session's far end, `hop[2]`. @@ -585,13 +599,15 @@ Fail(n, s, hop, from, moved, gone) == ELSE IF h.state = "live" THEN [h EXCEPT !.state = "dead"] ELSE h]] IN [s EXCEPT !.rx = rx1, !.dead = @ \cup {hop}, !.expiry = IF hop \in s.dead THEN @ ELSE @ \cup {hop}, - !.doubt = @ \cup ({hop[2]} \ {n}), !.wait = @ \cup ({hop[2]} \ {n})] + !.doubt = @ \cup ({hop[2]} \ {n}), !.wait = @ \cup ({hop[2]} \ {n}), + !.since = Since(n, s, hop[2])] \* `drains`: x sent GOAWAY. The first word of it casts doubt on x, so no relay \* takes a stale offer through it until the timer fires. MarkDrain(n, s, x) == IF x \in s.draining \/ x = n THEN s - ELSE [s EXCEPT !.draining = @ \cup {x}, !.doubt = @ \cup {x}, !.wait = @ \cup {x}] + ELSE [s EXCEPT !.draining = @ \cup {x}, !.doubt = @ \cup {x}, !.wait = @ \cup {x}, + !.since = Since(n, s, x)] \* A DRAIN counts only for a node on a path `from` offered. Credited(s, from, x) == \E k \in Keys : s.rx[from][k].on /\ Names(s.rx[from][k].path, x) @@ -635,6 +651,7 @@ Deliver(d) == \* the news to cross the mesh. Lift(n, x) == LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ {x}, !.wait = @ \ {x}, + !.since = IF x = Origin THEN None ELSE @, !.expiry = {h \in @ : h[2] # x}], OV, up, gen, ended[n], NoReport, TRUE)] IN /\ x \in st[n].wait @@ -853,13 +870,14 @@ LoopsLostThePublisher == IN \A i \in round : Len(w[i].path) >= 2 /\ Gone(w[i].path[2]) \* A route a relay knows over sessions that are still open, crossing no relay -\* it doubts. +\* it doubts, and not a link that came back during the doubt. Fresh(n, m, k) == /\ Usable(n, st[n], m, k) /\ ~Stale(st[n].rx[m][k].path) /\ ~RelayDoubted(st[n], st[n].rx[m][k].path) + /\ ~Returned(n, st[n], m, st[n].rx[m][k].path) -\* Doubt about the publisher alone never leaves a relay without a route: one +\* Doubt about the publisher alone never keeps a relay off its other links: one \* that knows a fresh route uses some route, so a request stops only at a relay \* that knows no way on. Which route it uses may still be stale while the news \* that ends it is in flight, which `StaleIsChased` bounds. diff --git a/rs/moq-net/verify/route/TwoHomes.cfg b/rs/moq-net/verify/route/TwoHomes.cfg index 82c37f7536c..35c0c14fb33 100644 --- a/rs/moq-net/verify/route/TwoHomes.cfg +++ b/rs/moq-net/verify/route/TwoHomes.cfg @@ -1,6 +1,7 @@ \* Flap.cfg where both of the publisher's sessions open and close. A relay may \* move onto an offer older than the doubt timer, naming a hop it has already -\* forgotten died, where doubt about the publisher kept it off before. When the +\* forgotten died, where doubt about the publisher kept it off before: the +\* exemption for a route beginning at the publisher allows it. When the \* next message on that link names a hop it knows died, the relay lets go: \* claims it showed go offline, then return once another route arrives. The \* replaced rule kept the relay dark for the whole timer instead, so this From b3ee511c08f4fb647e4fb6d08438cde6541f5382 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 09:49:52 -0800 Subject: [PATCH 04/12] fix(route): a restart reconnects a publisher's second home; credit drain ranking The two-home churn's restart left the publisher's attached link closed until a random flap reopened it. A restart now reconnects the links the mesh lacks, and the churn checks it. The publisher's GOAWAY stays loop-free because every path to it drains alike, not because of doubt; the comments now say so. The model's header names the returning-link exception, and the verify-route skill runs the flap fixtures. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 4 ++-- quest/m2/cluster-routes/README.md | 2 +- rs/moq-net/src/route/path.rs | 10 +++++++--- rs/moq-net/src/route/tests/compare.rs | 14 +++++++++++++- rs/moq-net/src/route/tests/handoff.rs | 7 ++++--- rs/moq-net/verify/route/DrainRing.cfg | 4 ++-- rs/moq-net/verify/route/RouteHandoff.tla | 9 ++++++--- 7 files changed, 35 insertions(+), 15 deletions(-) diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index db420a6f4de..beab5eee663 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -79,14 +79,14 @@ change; faster checkers are separate work. | `Overlap.cfg` | Five relays, three asking for `"a"`, with overlapping cheap broad and dear narrow paths. Simulated past the message budget. | 1 claim change, no session change | 1,000 behaviours per worker (4,000 on four workers), depth 500 | simulation | | `Loop.cfg` | The publisher with a session to each of three relays in a triangle, all closing. Samples `RequestsLoopFree` with timer-only doubt. | 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | simulation | | `Drain.cfg` | The publisher joined to two relays joined to each other; both hear its GOAWAY. Ranking only a relay's own drain exceeded `Budget`. | 1 claim change, 1 session change, GOAWAY on each session | 648,218 | two minutes | -| `DrainRing.cfg` | The publisher joined to three relays in a ring that is cheap one way round (`Ring`); all hear its GOAWAY. Without the DRAIN news and its doubt, a request looped round the ring with no session closed, breaking `LoopsAreStale`. Simulated. | 1 claim change, GOAWAY on each session | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | +| `DrainRing.cfg` | The publisher joined to three relays in a ring that is cheap one way round (`Ring`); all hear its GOAWAY. Without the DRAIN news, which ranks every path to the publisher as draining alike, a request looped round the ring with no session closed, breaking `LoopsAreStale`. Simulated. | 1 claim change, GOAWAY on each session | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | | `Prefer.cfg` | The same triangle, where both relays prefer the link between them. Ranking only a relay's own link priority exceeded `Budget`. | 1 claim change, 1 session change | 14,630 | seconds | | `Flap.cfg` | The publisher on relays 1 and 2, joined to each other, and relay 3 behind both; only the publisher's link to relay 1 opens and closes. Under the replaced rule, where doubt also kept relays off routes beginning at the publisher, `Routed` failed at once. Simulated. | 1 claim change, 4 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | | `TwoHomes.cfg` | The same, where both of the publisher's links open and close. Drops `Stable`: a relay can take an offer older than the timer and drop a claim it showed. Simulated. | 1 claim change, 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | | `Homes.cfg` | The publisher on each of three relays in a ring that is cheap one way round, leaving all three. `RequestsLoopFree` fails within seconds; checks `LoopsLostThePublisher`. Simulated. | 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | | `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, seed 1 | a few minutes | -Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. +Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare|flap)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. ## Reading a failure diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index 37778abdaae..a12ea3a6aa2 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -55,7 +55,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was to be modeled and measured separately; the decision below covers it. - Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](/quest/m2/cluster-routes/flapping-publisher.md)): ✅ **doubt blocks a route through a node, not one that begins at it, except the node's own route on a session to it that opened since the doubt began.** A publisher on two relays whose link to one flaps faster than the timer stays routed at every relay through the other, and a publisher that moves to another relay is routed in 50 ms in a 24-relay clique and 325 ms in a ring, where it waited 1,100 and 1,850 ms. The returning link still waits out the timer, so no relay hears it until the relay it returned to takes it, which keeps the timer's damping of a flapping link: in 400 churn cases (120,000 steps) with the publisher on one relay, the rule costs 3% more route messages than timer-only doubt with no fewer relays routed, and with the publisher on two relays it routes relays 14% more often for 22% more route messages, 6% more claims sent, and 3% more claims taken offline. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`; 44 of the two-relay churn's 120,000 steps). With two homes the model finds no loop (`TwoHomes.cfg`), but when both links fail within one timer a relay can take an offer older than the timer and drop a claim it showed, where the replaced rule held it dark (`TwoHomes.cfg` drops `Stable`). This also settles the fresh-publisher quest, which proposed the opposite exemption and is deleted: its measurements showed that taking a returning link at once costs more traffic and claims offline than it saves. Rejected: keeping timer-only doubt, under which a dual-homed publisher's other link is unusable for the whole timer; sparing every route that begins at the doubted node, which routes relays 67% more often with two homes but costs 121% more route messages, 76% more claims sent, and 69% more claims offline, since each return of a flapping link reaches every relay; sparing only links heard before the doubt began, which keeps the damping but rarely helps a relay whose neighbours had not chosen the other link; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waits on [the switch](/quest/m2/cluster-routes/switch.md); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good. + Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](/quest/m2/cluster-routes/flapping-publisher.md)): ✅ **doubt blocks a route through a node, not one that begins at it, except the node's own route on a session to it that opened since the doubt began.** A publisher on two relays whose link to one flaps faster than the timer stays routed at every relay through the other, and a publisher that moves to another relay is routed in 50 ms in a 24-relay clique and 325 ms in a ring, where it waited 1,100 and 1,850 ms. The returning link still waits out the timer, so no relay hears it until the relay it returned to takes it, which keeps the timer's damping of a flapping link: in 400 churn cases (120,000 steps) with the publisher on one relay, the rule costs 3% more route messages than timer-only doubt with no fewer relays routed, and with the publisher on two relays it routes relays 15% more often for 22% more route messages, 6% more claims sent, and 4% more claims taken offline. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`). The two-home churn, which flaps random links and restarts relays but never takes a publisher off three relays, shows a route loop in 51 of its 120,000 steps. With two homes the model finds no loop (`TwoHomes.cfg`), but when both links fail within one timer a relay can take an offer older than the timer and drop a claim it showed, where the replaced rule held it dark (`TwoHomes.cfg` drops `Stable`). This also settles the fresh-publisher quest, which proposed the opposite exemption and is deleted: its measurements showed that taking a returning link at once costs more traffic and claims offline than it saves. Rejected: keeping timer-only doubt, under which a dual-homed publisher's other link is unusable for the whole timer; sparing every route that begins at the doubted node, which routes relays 67% more often with two homes but costs 121% more route messages, 76% more claims sent, and 69% more claims offline, since each return of a flapping link reaches every relay; sparing only links heard before the doubt began, which keeps the damping but rarely helps a relay whose neighbours had not chosen the other link; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waits on [the switch](/quest/m2/cluster-routes/switch.md); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good. Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). diff --git a/rs/moq-net/src/route/path.rs b/rs/moq-net/src/route/path.rs index 489344cd00a..ee9e2658713 100644 --- a/rs/moq-net/src/route/path.rs +++ b/rs/moq-net/src/route/path.rs @@ -10,7 +10,9 @@ //! leaving node in a [`Dead::Draining`] to each neighbour it sent a route through //! it, which passes it on the same way, and each node doubts the leaving node as //! a closed session's far end is doubted, so no node moves onto a stale offer -//! through it and a request cannot loop through offers that predate the drain. When a session closes, +//! through it and a request cannot loop through offers that predate the drain. +//! When the leaving node is the publisher, every path to it ranks as draining +//! alike, so no node moves at all. When a session closes, //! the node that lost it names that hop in one [`Dead::Closed`] per neighbour, which //! lists the routes that moved to another path, and every node drops every route //! through the hop, whoever offered it. A node with no route left holds its @@ -135,7 +137,8 @@ pub(crate) enum Dead { /// `node` sent GOAWAY: every path through it ranks as draining, the worst rank, /// and stays usable. The news casts doubt on the node, so no relay moves onto a /// stale offer through it until the timer fires, and a request cannot loop through - /// offers that predate the drain. + /// offers that predate the drain. A publisher's drain needs no doubt: every path to + /// it ranks as draining alike, so no relay moves. Draining { node: Id }, } @@ -1442,7 +1445,8 @@ impl Router { } /// `node` sent GOAWAY, as of `now`. The first word of it casts doubt on the node, - /// so no stale offer through it is taken before the timer fires. Returns the + /// so no stale offer through it is taken before the timer fires; a route that + /// begins at it ends there and cannot loop. Returns the /// identities whose routes rank anew. fn drains(&mut self, node: Id, now: Instant) -> BTreeSet { if !self.draining.insert(node) { diff --git a/rs/moq-net/src/route/tests/compare.rs b/rs/moq-net/src/route/tests/compare.rs index f4a3870cbfd..bce7c107c97 100644 --- a/rs/moq-net/src/route/tests/compare.rs +++ b/rs/moq-net/src/route/tests/compare.rs @@ -584,6 +584,14 @@ impl Lab { } self.net.reset(relay); self.revive(relay); + // Unlike after a crash, the links the mesh lacks reconnect too, such as a + // publisher's second home. + let linked: Vec<(Node, Node)> = (self.linked.iter().copied()) + .filter(|&(a, b)| (a == relay || b == relay) && !self.is_up(a, b)) + .collect(); + for (a, b) in linked { + self.mend(a, b); + } } /// Deliver one message from a queue `pick` chooses. False if none waits. @@ -835,7 +843,11 @@ pub(super) fn churn(rule: Rule, cases: std::ops::Range, two_homes: bool) false => lab.mend(a, b), } } - 2 if case % 4 == 0 && script.chance(25) => lab.restart(script.next(relays as u64) as usize), + 2 if case % 4 == 0 && script.chance(25) => { + let relay = script.next(relays as u64) as usize; + lab.restart(relay); + assert!(homes.iter().all(|&home| home != relay || lab.is_up(node, home))); + } _ => { lab.deliver(&mut pick); } diff --git a/rs/moq-net/src/route/tests/handoff.rs b/rs/moq-net/src/route/tests/handoff.rs index 1d1ddcebebf..b4f196a7f42 100644 --- a/rs/moq-net/src/route/tests/handoff.rs +++ b/rs/moq-net/src/route/tests/handoff.rs @@ -1751,9 +1751,10 @@ fn goaway_moves_a_relay_to_a_path_that_does_not_drain() { /// joined to `hub`, which reaches the publisher. A GOAWAY from `leaving` reaches /// the hub's side before any ring relay hears the others re-rank. Ranking drain /// alone, each moved onto the next relay's old, undrained offer, and a request went -/// round the ring with no session closed. The drain news casts doubt on the leaving -/// node, so no relay moves onto a stale offer through it, and every request reaches -/// the publisher after each delivery and timer. +/// round the ring with no session closed. When the hub leaves, the drain news casts +/// doubt on it, so no relay moves onto a stale offer through it. When the publisher +/// leaves, every path names it and ranks as draining alike, so no relay moves. Either +/// way every request reaches the publisher after each delivery and timer. fn goaway_around_a_ring(sim: &mut Sim, hub: Node, leaving: Node) { let (origin, ring) = (0, [1, 2, 3]); for relay in ring { diff --git a/rs/moq-net/verify/route/DrainRing.cfg b/rs/moq-net/verify/route/DrainRing.cfg index 2698259d236..dc3f647d21d 100644 --- a/rs/moq-net/verify/route/DrainRing.cfg +++ b/rs/moq-net/verify/route/DrainRing.cfg @@ -4,8 +4,8 @@ \* publisher's GOAWAY may reach every relay before any hears the others re-rank. \* Ranking drain alone, each moved onto the next relay's old, undrained offer, \* and a request went round the ring with no session closed, breaking -\* RequestsLoopFree and LoopsAreStale. The DRAIN news casts doubt on the -\* publisher, so no relay takes a stale offer through it. +\* RequestsLoopFree and LoopsAreStale. Every path names the publisher, so +\* each ranks as draining alike and no relay moves off its route. \* The Rust fixture is a_goaway_in_a_ring_never_loops_a_request. CONSTANTS Nodes = {0, 1, 2, 3} diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index 9b9706ed7dc..8513443e0c4 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -20,8 +20,10 @@ (* neighbour that learned the route from it has answered. *) (* - A closed session casts doubt on the node at its far end: no relay *) (* moves to a route through it until a timer lifts the doubt. A route *) -(* that begins at it is exempt: it cannot loop through its own end. The *) -(* timer is the step `Lift`, which may fire at any moment. *) +(* that begins at it is exempt, since it cannot loop through its own *) +(* end, except over a session to it opened since the doubt began *) +(* (`Returned`): a link that comes back waits out the timer. The timer *) +(* is the step `Lift`, which may fire at any moment. *) (* *) (* Each node's every entry point is one atomic step ending in `Update`, *) (* as in the router. Messages travel on one FIFO per direction. A *) @@ -603,7 +605,8 @@ Fail(n, s, hop, from, moved, gone) == !.since = Since(n, s, hop[2])] \* `drains`: x sent GOAWAY. The first word of it casts doubt on x, so no relay -\* takes a stale offer through it until the timer fires. +\* takes a stale offer through it until the timer fires. A route that begins at +\* x needs none: every path to x drains alike, so no relay moves. MarkDrain(n, s, x) == IF x \in s.draining \/ x = n THEN s ELSE [s EXCEPT !.draining = @ \cup {x}, !.doubt = @ \cup {x}, !.wait = @ \cup {x}, From 30b3e36451511c1946507e266653fea16f321c61 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 12:44:32 -0800 Subject: [PATCH 05/12] test(route): pin the stale-ring cache budget to the path rule's loop Main's a_ring_of_stale_offers_settles_on_the_link_left pinned 40 cycle steps, the replaced rule's exact total (2 per seed). Under this rule a relay that lost its publisher link takes a neighbour's direct route that neighbour's own close made stale, until that DEAD crosses the link: the accepted transient loop. The seeded total is 204 (4 to 22 per seed, a cycle lasting at most 20 deliveries), and every seed still settles with one copy fed through the link left. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/route/tests/cache.rs | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/rs/moq-net/src/route/tests/cache.rs b/rs/moq-net/src/route/tests/cache.rs index 53bdc015200..d60b3785d50 100644 --- a/rs/moq-net/src/route/tests/cache.rs +++ b/rs/moq-net/src/route/tests/cache.rs @@ -1146,7 +1146,10 @@ fn equal_relays_pull_one_copy_owned_by_the_lower_hash() { /// but one of those links while every link of the ring is held, so each relay acts on /// what it last heard. Delivered in any order, once quiet every reader is fed through /// the one link left, and the pulls formed a cycle only while a change was still -/// crossing a link. +/// crossing a link. A relay that lost its link may take a neighbour's direct route +/// that the neighbour's own close made stale, until that close's DEAD crosses the +/// link between them: the transient loop the routes epic's Plan item 27 accepts. The +/// budget pins the seeded total, so a change in the rate shows here. #[test] fn a_ring_of_stale_offers_settles_on_the_link_left() { const RING: usize = 5; @@ -1192,7 +1195,7 @@ fn a_ring_of_stale_offers_settles_on_the_link_left() { assert_eq!(mesh.parent(relays[0], &x), Some(o), "seed {seed}"); loops += mesh.loops; } - assert!(loops <= 40, "{loops} steps with a cycle across 20 seeds"); + assert!(loops <= 204, "{loops} steps with a cycle across 20 seeds"); } /// `b` withdraws its copy while `c`'s subscription to it is on its way. The From 29f754488ebf8e9afedc5c25e91b2de9d2489cf2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 15:33:39 -0800 Subject: [PATCH 06/12] fix(route): fingerprint a doubt's session mark, after merging main Main's exhaustive search (#8729) fingerprints each doubt as doubted or lifted. Here a doubt also notes the last session opened when it began, which decides whether a returning link waits, so the fingerprint carries it. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/route/path.rs | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/rs/moq-net/src/route/path.rs b/rs/moq-net/src/route/path.rs index 2b8e78a4e80..c7ac6ea87fe 100644 --- a/rs/moq-net/src/route/path.rs +++ b/rs/moq-net/src/route/path.rs @@ -1321,7 +1321,15 @@ impl Router { .map(|(hop, death)| (hop, death.far(), death.deadline().map(left))) .collect(); let doubted: Vec<_> = (doubted.iter()) - .map(|(node, state)| (node, matches!(state, Suspicion::Doubted(_)), left(state.deadline()))) + .map(|(node, state)| { + // `since` decides whether a returning link waits, so it is state. + let since = match state { + Suspicion::Doubted { since, .. } => Some(*since), + #[cfg(test)] + Suspicion::Lifted(_) => None, + }; + (node, since, left(state.deadline())) + }) .collect(); super::digest(hasher, &(hop, doubt, opened, sessions, answering, origins, rx)); super::digest(hasher, &(rx_hops, tx_hops, tx_nodes, reflects, waiting, routes, views)); From a0473ae80ce8e7cc905ea41a211133803c08643d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 22:30:42 -0800 Subject: [PATCH 07/12] test(route): a link opened before a doubt is taken at once Kills the two mutants of doubts_path's session check that the PR's tests missed: a relay doubting the publisher moves onto its own link to it when that link opened before the doubt, even as the last session it opened. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/route/tests/flap.rs | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/rs/moq-net/src/route/tests/flap.rs b/rs/moq-net/src/route/tests/flap.rs index e088a831227..4d92b190b06 100644 --- a/rs/moq-net/src/route/tests/flap.rs +++ b/rs/moq-net/src/route/tests/flap.rs @@ -162,3 +162,32 @@ fn a_publisher_leaving_every_relay_at_once_loops_until_the_news_arrives() { } } } + +/// A relay doubting the publisher still moves at once onto its own link to it, +/// opened before the doubt began, even as the last session it opened; only the +/// replaced rule holds it until the timer. +#[test] +fn a_link_opened_before_the_doubt_is_taken_at_once() { + let (origin, r1, r2) = (0, 1, 2); + for (rule, moves) in [(Rule::PATH, true), (Rule::ORIGIN, false)] { + let mut sim = Sim::with(rule, 3); + sim.leaf(origin); + sim.join(origin, r1, 1); + sim.join(r1, r2, 1); + sim.join(origin, r2, 5); + sim.originate(origin, 7, 0); + sim.announce(origin, "live/x", 0, None); + assert!(sim.run()); + assert_eq!(sim.toward(r2, 7), Some(Some(r1)), "{}", rule.as_str()); + sim.disconnect(origin, r1); + assert!(sim.flow()); + assert_eq!(sim.toward(r2, 7) == Some(Some(origin)), moves, "{}", rule.as_str()); + assert!(sim.run()); + assert_eq!( + sim.toward(r2, 7), + Some(Some(origin)), + "{}: after the timer", + rule.as_str() + ); + } +} From 9be566b9600f4e94a32e083c3fbdc0d6a9f1bc65 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 01:47:10 -0800 Subject: [PATCH 08/12] docs(quest): the draft text keeps a relay's current route exempt from doubt Co-Authored-By: Claude Opus 5.5 --- quest/m2/cluster-routes/flapping-publisher.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/quest/m2/cluster-routes/flapping-publisher.md b/quest/m2/cluster-routes/flapping-publisher.md index 5d0bc9f50bc..f4ab58e7eb4 100644 --- a/quest/m2/cluster-routes/flapping-publisher.md +++ b/quest/m2/cluster-routes/flapping-publisher.md @@ -22,7 +22,8 @@ still run Babel's distance, so what is left waits on the switch: receives every group while it flaps. - The draft's Routing section: doubt keeps a relay off a route through the far end of a closed session, and off that node's own route on a session to - it opened since, but not off another route that begins there; a publisher on + it opened since, but not off the route it already uses or another route that + begins there; a publisher on three or more relays that leaves them all at once can loop a request until the news arrives. From 599fa8d70db163015d788e8f526d4f5ebb82b1de Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 03:23:03 -0800 Subject: [PATCH 09/12] feat(route): sessions spare a doubted origin's other links; the draft says so Since the switch, sessions run this rule: restore the unannounce route-change tests the old rule had replaced, drop catalog_handover's doubt wait, add the flapping-link fixture on real sessions, state the rule in the draft's Dead Hops section, and finish the flapping-publisher quest. Qualify the epic's claim that every relay stays routed. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 2 +- drafts/draft-lcurley-moq-lite.md | 8 +-- quest/m2/cluster-routes/README.md | 2 +- quest/m2/cluster-routes/flapping-publisher.md | 35 ------------- quest/m3/cluster-routing/request-loop.md | 2 +- .../tests/integration/catalog_handover.rs | 9 ++-- rs/moq-net/tests/integration/path_routes.rs | 45 ++++++++++++++++- rs/moq-net/tests/integration/route_change.rs | 50 ++++--------------- 8 files changed, 65 insertions(+), 88 deletions(-) delete mode 100644 quest/m2/cluster-routes/flapping-publisher.md diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 0a22d308c4c..8cc777beedc 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -80,7 +80,7 @@ change; faster checkers are separate work. | `Blind.cfg` | A relay whose nearest neighbour asks for `"a"` alone, and a farther one knows everything. Simulated. | 2 claim changes, 1 session change | 40,000 behaviours, 4.8M states | ten minutes | | `Overlap.cfg` | Five relays, three asking for `"a"`, with overlapping cheap broad and dear narrow paths. Simulated past the message budget. | 1 claim change, no session change | 1,000 behaviours per worker (4,000 on four workers), depth 500 | simulation | | `Loop.cfg` | The publisher with a session to each of three relays in a triangle, all closing. Samples `RequestsLoopFree` with timer-only doubt. | 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | simulation | -| `Drain.cfg` | The publisher joined to two relays joined to each other; both hear its GOAWAY. Ranking only a relay's own drain exceeded `Budget`. | 1 claim change, 1 session change, GOAWAY on each session | 511,147 | two minutes | +| `Drain.cfg` | The publisher joined to two relays joined to each other; both hear its GOAWAY. Ranking only a relay's own drain exceeded `Budget`. | 1 claim change, 1 session change, GOAWAY on each session | 575,887 | two minutes | | `DrainRing.cfg` | The publisher joined to three relays in a ring that is cheap one way round (`Ring`); all hear its GOAWAY. Without the DRAIN news, which ranks every path to the publisher as draining alike, a request looped round the ring with no session closed, breaking `LoopsAreStale`. Simulated. | 1 claim change, GOAWAY on each session | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | | `Prefer.cfg` | The same triangle, where both relays prefer the link between them. Ranking only a relay's own link priority exceeded `Budget`. | 1 claim change, 1 session change | 15,380 | seconds | | `Upstream.cfg` | Two uplinks: the publisher on a CDN relay joined by a dear link to another, and a local mesh of two relays, each with an upstream link to one CDN relay. With the CDN link down, the second CDN relay has no route rather than one through the local mesh. The Rust fixture is `two_uplinks_never_carry_cdn_transit`. Without the export check, or with a flag that a LAN hop clears, `QuietIsSettled` and `DownOnly` fail within seconds. | 1 claim change, 1 session change | 76,992 | seconds | diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 37e6584c5bb..16af6695659 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -542,7 +542,7 @@ Its priority is the lowest local priority of a session on its Path: a receiver t A node ranks its usable routes to an origin node at a prefix: one that is not draining first, then the highest priority, then the lowest metric. Extending a route never clears its drain, never raises its priority, and always raises its metric, so no node ranks a route above how the neighbour it came from ranked it, and selection converges. A node selects its best-ranked usable route at once. -A route is unusable while its Path names the node, it is held, a hop on its Path is dead, a node on its Path is in doubt (see [Dead Hops](#dead-hops)) unless it is the route the node already uses, it arrived with 32 hops, or its metric overflows. +A route is unusable while its Path names the node, it is held, a hop on its Path is dead, the doubt keeps the node off it (see [Dead Hops](#dead-hops)), it arrived with 32 hops, or its metric overflows. A node MAY mark a session upstream. A route is down-only when it arrived on an upstream session or with its down-only flag set; a node sends a down-only route with the flag set, and never on an upstream session. @@ -575,7 +575,9 @@ Then every route through the hop is dead, whoever sent it: the sender's routes c The receiver remembers the dead hop, so a route through it that arrives late stays unusable. A closed session also casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time: a neighbour that has not heard yet may still offer a stale route through it. -A receiver doubts the far node for one second, during which it moves onto no route through that node; the route it already uses stays usable. +A receiver doubts the far node for one second, during which it moves onto no route whose Path names that node after the origin node's hop, and onto no route the doubted node originates whose Path is that node's hop alone, received on a session that opened since the doubt began; the route it already uses stays usable. +Any other route the doubted node originates stays usable, since a route cannot loop through its own origin: an origin on several relays stays reachable through the others while its session to one of them flaps, and a session that reopens waits out the doubt, which damps the flaps. +A node cannot tell an origin leaving from its session to one relay closing, so an origin on three or more relays that leaves them all at once can send a request round them, each relay on a neighbour's route that the neighbour's own close made stale, until that neighbour's DEAD arrives. Nothing on the wire carries the doubt. A node that sees a session close doubts its far end in the same way. @@ -2552,7 +2554,7 @@ An application-specific payload. - ROUTE_START and ROUTE_UPDATE carry Flags, whose 0x2 bit adds an optional RTT: the informational sum of each link's lowest smoothed RTT over the last 30 seconds to the origin node, which never selects a route. - ROUTE_START and ROUTE_UPDATE carry a Route Offer that leads with the route's Path: one hop per node the route crossed, each a Hop ID and the Session Number that node learned the route on, at most 32. A node never uses a route whose Path names it. ROUTE_START also carries a Route Suffix naming the prefix the route covers, and the offer carries the route's drain (Flags 0x4) and priority, which rank before its metric. - Added ROUTE_HOLD (0xB), which holds a route while it may recover, for at most 10 seconds; ROUTE_END lets it go. Neither withdraws the origin's claims. -- Added DEAD (0xF): a node that loses a session names that hop once to each neighbour it sent a route through it, listing the routes that moved, and every node drops every route through the hop and doubts the node at the far end for a second; a DRAIN names a node that sent a GOAWAY, whose routes rank draining. The path, the dead hop, holds, and the doubt keep routes free of loops and stale copies, with no distance, sequence number, route request, or echo. On a scoped session a DEAD applies only to the routes heard there. +- Added DEAD (0xF): a node that loses a session names that hop once to each neighbour it sent a route through it, listing the routes that moved, and every node drops every route through the hop and doubts the node at the far end for a second, keeping nodes off routes through it but not off one it originates, except over a session to it opened since; a DRAIN names a node that sent a GOAWAY, whose routes rank draining. The path, the dead hop, holds, and the doubt keep routes free of stale copies, and of loops except briefly when an origin on three or more relays leaves them all at once, with no distance, sequence number, route request, or echo. On a scoped session a DEAD applies only to the routes heard there. - A node closes a session whose peer leaves an Announce Stream undrained for 10 seconds, and the subscriber sends nothing on the stream after ANNOUNCE_REQUEST. - Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the route's prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows. - A route presents only its most specific claim covering the requested prefix, moved to the next covering claim when it is withdrawn rather than ended. diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index 46539c723b9..45fafc1b72d 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -55,7 +55,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was to be modeled and measured separately; the decision below covers it. - Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](/quest/m2/cluster-routes/flapping-publisher.md)): ✅ **doubt blocks a route through a node, not one that begins at it, except the node's own route on a session to it that opened since the doubt began.** A publisher on two relays whose link to one flaps faster than the timer stays routed through the other: a relay that already hears a route through the other home that does not name it keeps one the whole time, as every relay does in the measured mesh (the two homes joined, a third relay behind both), and a relay whose neighbours all route through it is routed again once one of them re-offers, a round trip after the news, instead of after the timer. A publisher that moves to another relay is routed in 50 ms in a 24-relay clique and 325 ms in a ring, where it waited 1,100 and 1,850 ms. The returning link still waits out the timer, so no relay hears it until the relay it returned to takes it, which keeps the timer's damping of a flapping link: in 400 churn cases (120,000 steps) with the publisher on one relay, the rule costs 3% more route messages than timer-only doubt with no fewer relays routed, and with the publisher on two relays it routes relays 15% more often for 22% more route messages, 6% more claims sent, and 4% more claims taken offline. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`). The two-home churn, which flaps random links and restarts relays but never takes a publisher off three relays, shows a route loop in 51 of its 120,000 steps. With two homes the model finds no loop (`TwoHomes.cfg`), but when both links fail within one timer a relay can take an offer older than the timer and drop a claim it showed, where the replaced rule held it dark (`TwoHomes.cfg` drops `Stable`). This also settles the fresh-publisher quest, which proposed the opposite exemption and is deleted: its measurements showed that taking a returning link at once costs more traffic and claims offline than it saves. Rejected: keeping timer-only doubt, under which a dual-homed publisher's other link is unusable for the whole timer; sparing every route that begins at the doubted node, which routes relays 67% more often with two homes but costs 121% more route messages, 76% more claims sent, and 69% more claims offline, since each return of a flapping link reaches every relay; sparing only links heard before the doubt began, which keeps the damping but rarely helps a relay whose neighbours had not chosen the other link; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waited on [the switch](https://github.com/moq-dev/moq/pull/8731); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good. + Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](https://github.com/moq-dev/moq/pull/8734)): ✅ **doubt blocks a route through a node, not one that begins at it, except the node's own route on a session to it that opened since the doubt began.** A publisher on two relays whose link to one flaps faster than the timer stays routed through the other: a relay that already hears a route through the other home that does not name it keeps one the whole time, as every relay does in the measured mesh (the two homes joined, a third relay behind both), and a relay whose neighbours all route through it is routed again once one of them re-offers, a round trip after the news, instead of after the timer. A publisher that moves to another relay is routed in 50 ms in a 24-relay clique and 325 ms in a ring, where it waited 1,100 and 1,850 ms. The returning link still waits out the timer, so no relay hears it until the relay it returned to takes it, which keeps the timer's damping of a flapping link: in 400 churn cases (120,000 steps) with the publisher on one relay, the rule costs 3% more route messages than timer-only doubt with no fewer relays routed, and with the publisher on two relays it routes relays 15% more often for 22% more route messages, 6% more claims sent, and 4% more claims taken offline. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`). The two-home churn, which flaps random links and restarts relays but never takes a publisher off three relays, shows a route loop in 51 of its 120,000 steps. With two homes the model finds no loop (`TwoHomes.cfg`), but when both links fail within one timer a relay can take an offer older than the timer and drop a claim it showed, where the replaced rule held it dark (`TwoHomes.cfg` drops `Stable`). This also settles the fresh-publisher quest, which proposed the opposite exemption and is deleted: its measurements showed that taking a returning link at once costs more traffic and claims offline than it saves. Rejected: keeping timer-only doubt, under which a dual-homed publisher's other link is unusable for the whole timer; sparing every route that begins at the doubted node, which routes relays 67% more often with two homes but costs 121% more route messages, 76% more claims sent, and 69% more claims offline, since each return of a flapping link reaches every relay; sparing only links heard before the doubt began, which keeps the damping but rarely helps a relay whose neighbours had not chosen the other link; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waited on [the switch](https://github.com/moq-dev/moq/pull/8731); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good. Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). diff --git a/quest/m2/cluster-routes/flapping-publisher.md b/quest/m2/cluster-routes/flapping-publisher.md deleted file mode 100644 index 6b5ed70c18a..00000000000 --- a/quest/m2/cluster-routes/flapping-publisher.md +++ /dev/null @@ -1,35 +0,0 @@ -# [S] A dual-homed publisher stays reachable on real sessions while one link flaps - -## Goal - -On real lite-07 sessions, a publisher connected to two relays whose link to -one of them closes and reopens faster than the doubt timer stays reachable -through its other link, while the flapping link waits out the timer each time -it returns. A relay that already hears a route through the other link that -does not name it stays routed the whole time; any other relay is routed again -once a neighbour re-offers, a round trip after the news, instead of after the -timer. The draft states the rule. - -## Plan - -[moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) settled the rule -([epic](/quest/m2/cluster-routes/README.md) Plan item 27, decided 2026-11-14): -doubt blocks a route through a node, not one that begins at it, except the -node's own route on a session to it opened since the doubt began. The model -checks it (`Flap.cfg`, `TwoHomes.cfg`, `Homes.cfg`, and `Routed`), the path -router applies it, and `route/tests/flap.rs` pins it on the router. -[The switch](https://github.com/moq-dev/moq/pull/8731) put lite-07 sessions on -the path router, so what is left is: - -- A fixture on real sessions with mocked time: the dual-homed publisher with - one link flapping at half the timer, where a subscriber at a far relay - receives every group while it flaps. -- The draft's Routing section: doubt keeps a relay off a route through the - far end of a closed session, and off that node's own route on a session to - it opened since, but not off the route it already uses or another route that - begins there; a publisher on three or more relays that leaves them all at - once can loop a request until the news arrives. - -## Related - -- [A request loop heals once routes settle](/quest/m3/cluster-routing/request-loop.md) - heals the loop a publisher leaving three relays at once can cause diff --git a/quest/m3/cluster-routing/request-loop.md b/quest/m3/cluster-routing/request-loop.md index b50a15f1fdb..34be16918a9 100644 --- a/quest/m3/cluster-routing/request-loop.md +++ b/quest/m3/cluster-routing/request-loop.md @@ -44,4 +44,4 @@ pending request (the front in `rs/moq-net/src/model/front.rs`, and how ## Related -- [A dual-homed publisher stays reachable while one link flaps](/quest/m2/cluster-routes/flapping-publisher.md) - another failure mode from the same BGP comparison +- [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) - its doubt rule lets a publisher leaving three or more relays at once loop a request until the news arrives diff --git a/rs/moq-net/tests/integration/catalog_handover.rs b/rs/moq-net/tests/integration/catalog_handover.rs index 2587a5ac480..c4163b182cc 100644 --- a/rs/moq-net/tests/integration/catalog_handover.rs +++ b/rs/moq-net/tests/integration/catalog_handover.rs @@ -154,12 +154,9 @@ impl Topology { async fn trigger(&mut self, trigger: Trigger) { match trigger { Trigger::Disconnect => abort(self.a_to_r.take().unwrap()), - Trigger::Unannounce => { - abort(self.p_to_a.take().unwrap()); - // `P` is the far end of `A`'s closed session, and `B`'s route crosses it too: - // on lite-07 `R` holds the claim until the doubt lifts, then moves to `B`. - kio::clock::sleep(DOUBT).await; - } + // `P` is the far end of `A`'s closed session, but `B`'s route begins at `P`, so + // doubt about `P` does not keep `R` off it. + Trigger::Unannounce => abort(self.p_to_a.take().unwrap()), Trigger::BetterRoute => { let direct = link(self.version, &self.publisher, &self.subscriber, None).await; self.links.push(direct); diff --git a/rs/moq-net/tests/integration/path_routes.rs b/rs/moq-net/tests/integration/path_routes.rs index a5db1c75806..fbb3bc2720a 100644 --- a/rs/moq-net/tests/integration/path_routes.rs +++ b/rs/moq-net/tests/integration/path_routes.rs @@ -9,8 +9,9 @@ //! unavailable, until the neighbours that learned the route from it answer, or for //! [`HOLD`] at most. -use crate::route_change::{link, produce_origin}; +use crate::route_change::{link, next, payload, produce_origin, read, timed}; use crate::support; +use support::{AppendNext, AtStart}; use std::time::Duration; @@ -168,6 +169,48 @@ async fn a_detour_through_the_far_end_waits_out_the_doubt() { assert!(usable.at >= held.at + DOUBT, "not before the doubt lifts: {seen:?}"); } +/// `P` is on `A` and `B`, which peer with each other, and `W` is behind both. `P`'s session +/// to `A` closes and reopens every half doubt. Doubt about `P` keeps no relay off `B`'s +/// route, which begins at `P`, so `W` shows the claim available throughout and its +/// subscriber reads each group at once. The reopened session waits out the doubt. +#[kio::test] +async fn a_flapping_link_keeps_a_dual_homed_publisher_routed() { + let version = lite07(); + let [p, a, b, w] = [1, 2, 3, 4].map(produce_origin); + let live = publish(&p, "live"); + let track = live.broadcast().create_track("video", timed()).unwrap(); + let mut p_a = peer(version, &p, &a).await; + let _p_b = peer(version, &p, &b).await; + let _a_b = peer(version, &a, &b).await; + let _a_w = peer(version, &a, &w).await; + let _b_w = peer(version, &b, &w).await; + quiesce().await; + + let mut watch = record(&w, "live").await; + let broadcast = w.consume().request("live", None).await.unwrap(); + let mut rx = read(broadcast.track("video").unwrap().subscribe(None).await.unwrap()); + let start = kio::clock::now(); + for flap in 0..8 { + match flap % 2 { + 0 => cut(&p_a), + _ => p_a = peer(version, &p, &a).await, + } + let sent = kio::clock::now(); + let mut group = track.append_next().unwrap(); + group.write_frame(group.at_start(), payload(flap, 0)).unwrap(); + group.close().unwrap(); + assert_eq!(next(&mut rx).await, (flap, payload(flap, 0)), "flap {flap}"); + let late = kio::clock::now().saturating_duration_since(sent); + assert!(late < DOUBT / 2, "flap {flap}: read {late:?} after it was written"); + kio::clock::sleep(DOUBT / 2).await; + } + let seen = watch.seen(start); + assert!( + seen.iter().all(|seen| seen.available && !withdrawn(seen)), + "available throughout: {seen:?}" + ); +} + /// `X` is `P`'s only way out and neighbours `R1`, `R2` and `R3`, which mesh with each other. /// `X` dies: each relay hears its own session close and its neighbours' DEADs, and none /// moves onto another's route through `X` on the way. Each withdraws the claim once. diff --git a/rs/moq-net/tests/integration/route_change.rs b/rs/moq-net/tests/integration/route_change.rs index f2ee1a5fb73..fa67d9da085 100644 --- a/rs/moq-net/tests/integration/route_change.rs +++ b/rs/moq-net/tests/integration/route_change.rs @@ -230,7 +230,7 @@ pub(super) fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver) -> (u64, Vec) { +pub(super) async fn next(rx: &mut mpsc::UnboundedReceiver) -> (u64, Vec) { let (group, _, payload) = next_timed(rx).await; (group, payload) } @@ -563,41 +563,6 @@ async fn better_route_restarts(version: &str) { ); } -/// A route through the far end of a closed link waits out the doubt. `A` loses its -/// session to `P`, the far end, and `B`'s route names `P` too: `R` cannot tell `P` from a -/// dead relay whose news is still on its way, so it holds the claim and moves to `B` only -/// once the doubt lifts. The subscription through `A` ends meanwhile; a new one reads on -/// through `B`. -async fn waits_out_the_doubt(version: &str) { - let version: Version = version.parse().unwrap(); - let (mut topology, sub) = Topology::new(version).await; - let mut rx = read(sub); - topology.standby().await; - - let mut group = topology.track.append_next().unwrap(); - group.write_frame(group.at_start(), payload(0, 0)).unwrap(); - group.close().unwrap(); - assert_eq!(next(&mut rx).await, (0, payload(0, 0))); - - topology.trigger(Trigger::Unannounce).await; - kio::clock::sleep(Duration::from_secs(1)).await; - let mut group = topology.track.append_next().unwrap(); - group.write_frame(group.at_start(), payload(1, 0)).unwrap(); - group.close().unwrap(); - let remote = topology.subscriber.consume().request("live", None).await.unwrap(); - let mut fresh = read(remote.track("video").unwrap().subscribe(None).await.unwrap()); - assert_eq!(next(&mut fresh).await, (1, payload(1, 0)), "{version}: through B"); - match kio::clock::timeout(Duration::from_secs(10), rx.next()) - .await - .expect("the subscription through A hung") - { - Some((_, Err(_))) => {} - None => panic!("{version}: the subscription through A ended cleanly"), - Some((group, Ok(_))) => panic!("{version}: the subscription through A read group {group}"), - } - assert!(rx.try_recv().is_err(), "{version}: a delivery after the error"); -} - macro_rules! route_change_tests { ($($name:ident: $version:literal,)*) => { $( @@ -634,11 +599,16 @@ macro_rules! route_change_tests { run(Trigger::Disconnect, Position::MidGroup).await; } + /// `B`'s route begins at `P`, the far end of the closed link, so doubt about + /// `P` does not keep `R` off it: the subscription moves at once. #[kio::test] - async fn unannounce_waits_out_the_doubt() { - kio::clock::timeout(TEST_TIMEOUT, waits_out_the_doubt($version)) - .await - .expect("timed out"); + async fn unannounce_between_groups() { + run(Trigger::Unannounce, Position::BetweenGroups).await; + } + + #[kio::test] + async fn unannounce_mid_group() { + run(Trigger::Unannounce, Position::MidGroup).await; } #[kio::test] From e034e5c9251b16e3e943c3f7777037400db51244 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 04:33:55 -0800 Subject: [PATCH 10/12] fix(route): a session's later stream shares its generation under doubt A new permit files another stream of the same session; comparing stream numbers took it for a link that returned during the doubt and blocked the publisher's healthy route until the timer fired. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 2 +- rs/moq-net/src/route/path.rs | 12 ++++++++++- rs/moq-net/src/route/tests/routing.rs | 31 +++++++++++++++++++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 8cc777beedc..d59fc1887bf 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -116,7 +116,7 @@ Sync commit: `cbb742cecae437ee94fbf25848521ac11b145fbc`, where [moq-dev/moq#8734 | `rs/moq-net/src/route/path.rs` `Router::run`, `Router::select`, `Router::retain` | `Select` plans live routes and their reflected sources before `SelectKey` decides holds. | | `rs/moq-net/src/route/path.rs` `Offer::rank`, `Router::goaway`, `Router::drains`, `Router::report_draining`, `Router::dead` (`Dead::Draining`) | `Rank`, `Below`, `PathPrio`, `CostAt`, `GoAway`, `MarkDrain`, `Credited`, `ReportDrain`, `Recv` | | `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted`, `RelayDoubted` (a path's first node does not count), `Returned` | -| `rs/moq-net/src/route/path.rs` `Router::suspect`, `Router::doubt_until` | `Since`: a doubt on the publisher that begins notes the generation of the relay's own session to it | +| `rs/moq-net/src/route/path.rs` `Router::suspect`, `Router::doubt_until` | `Since`: a doubt on the publisher that begins notes the generation of the relay's own session to it. The router shares one generation across a session's streams (`Session::generation`), which the model's one stream per session leaves out. | | `rs/moq-net/src/route/path.rs` `Router::src` | `SrcKey` | | `rs/moq-net/src/route/path.rs` `Router::usable_offer`, `Offer` | `SelectKey`'s `Down`, `Want0`'s export check, and `DownOnly`: the down-only flag is set on a route learned over an upstream link or from a down-only one, and never sent over an upstream link. | | `rs/moq-net/src/route/path.rs` `Router::want`, `Router::stream`, `Router::advertise`, `Router::report` | `Want`, `Want0`, `Streams`, `Advertise`, `Report` | diff --git a/rs/moq-net/src/route/path.rs b/rs/moq-net/src/route/path.rs index 25b0b523077..56806670bed 100644 --- a/rs/moq-net/src/route/path.rs +++ b/rs/moq-net/src/route/path.rs @@ -462,6 +462,10 @@ impl Route { struct Session { /// How this node names the session in the hops of the routes it learns there. number: u64, + /// The `number` of the first stream of the neighbour's session still registered, + /// which its later streams share: doubt asks whether the session, not the stream, + /// opened since it began. + generation: u64, /// How the neighbour names itself in a path. hop: Id, link: Link, @@ -911,8 +915,14 @@ impl Router { return Err(Error::ZeroCost); } self.opened += 1; + // A later stream of a session already open, such as one a new permit files, is + // no returning link. The tests' `Sim` leaves `link.session` 0: a session per stream. + let generation = (self.sessions.values()) + .find(|other| link.session != 0 && other.link.session == link.session) + .map_or(self.opened, |other| other.generation); let session = Session { number: self.opened, + generation, hop, link, rtt: None, @@ -1987,7 +1997,7 @@ impl Router { AtOrigin::Spared => return false, AtOrigin::Own => {} } - rest.is_empty() && self.sessions[&peer].number > *since + rest.is_empty() && self.sessions[&peer].generation > *since } /// Doubt `node` until `at`, or later if it already is. A doubt that begins notes diff --git a/rs/moq-net/src/route/tests/routing.rs b/rs/moq-net/src/route/tests/routing.rs index a2f3b1ac7fa..38291824f2b 100644 --- a/rs/moq-net/src/route/tests/routing.rs +++ b/rs/moq-net/src/route/tests/routing.rs @@ -809,3 +809,34 @@ fn a_route_change_alone_names_its_identity() { router.route(up, 7, p(""), Some(path::Reach::Finite(along(&hops, 2)))); assert_eq!(router.changed(), BTreeMap::from([(7, BTreeSet::new())])); } + +/// A doubt about the publisher begins while this node has a session to it. That session +/// then files a second stream, as a new permit does, and the first retires. The second +/// stream is no returning link, so the publisher's own route stays usable on it. +#[test] +fn a_later_stream_of_a_session_the_doubt_predates_is_not_doubted() { + let (publisher, first, other, later) = (7, 1, 2, 3); + let session = |tag| Link { + session: tag, + ..Link::default() + }; + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + for (key, tag) in [(first, 5), (other, 6)] { + router.connect(key, publisher, session(tag)).unwrap(); + router.edited(key); + router.route(key, publisher, p(""), offer([publisher], 1)); + router.batch(key, publisher, p(""), vec![(p("live/x"), Some(1))], Some(1)); + } + // The other session to the publisher closes, casting doubt on it. + router.disconnect([other], Instant::ZERO); + router.connect(later, publisher, session(5)).unwrap(); + router.edited(later); + router.route(later, publisher, p(""), offer([publisher], 1)); + router.batch(later, publisher, p(""), vec![(p("live/x"), Some(1))], Some(1)); + router.retire(first); + assert_eq!( + router.route_for(publisher, &p("")).and_then(path::Route::next), + Some(later) + ); + assert!(router.shown()[&(publisher, p("live/x"))].available); +} From 6898b2710f13349daeeda8c49f85fd817a45d455 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 09:20:41 -0800 Subject: [PATCH 11/12] test(verify): TwoHomes.cfg checks Stable again under a patient timer Main's patient timer (#8834) waits for stale offers as well as DEAD news, which closes the hole that let a relay take an offer naming a hop it had forgotten died. TwoHomes.cfg with Stable passes 40,000 traces on seed 1, where the pre-merge model failed within a minute. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 4 ++-- quest/m2/cluster-routes/README.md | 2 +- rs/moq-net/verify/route/TwoHomes.cfg | 15 +++++++-------- 3 files changed, 10 insertions(+), 11 deletions(-) diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index f40e52ae155..07fcca1890b 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -70,7 +70,7 @@ change; faster checkers are separate work. | Config | Mesh | Bounds | Previous distinct states | Previous run | | --- | --- | --- | --- | --- | -| `RouteHandoff.cfg` | The publisher behind relay 1, in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 476,434 | a minute or two | +| `RouteHandoff.cfg` | The publisher behind relay 1, in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 434,956 | a minute or two | | `Early.cfg` | The same, with a timer that fires at any moment. | 1 claim change, 1 session change | 14,220,217 | half an hour | | `Twice.cfg` | The same, where two sessions close, as when relay 1 dies. Simulated: `num` counts behaviours per worker. | 1 claim change, 2 session changes | 40,000 behaviours, 4.0M states | a few minutes | | `Ring.cfg` | The publisher and three relays in a ring, with a timer that fires at any moment. | 1 claim change, 1 session change | 813,075 | a minute or two | @@ -89,7 +89,7 @@ change; faster checkers are separate work. | `EditWiden.cfg` | The same, where relay 2 watches `"a"` and widens to everything or drops it once. | 1 claim change, 1 session change, 1 edit | 3,655,573 | a quarter of an hour | | `Edit.cfg` | The same with two edits, such as widening then narrowing back, or dropping everything then asking again. Before the rekey kept the route the sender would reflect, a widening then a narrowing crossing the close of relay 2's own session kept a route relay 1 learned through it and took a claim offline, breaking `Stable`. Simulated; an exhaustive pass covered every behaviour to depth 18 (10.6M states) before it was stopped. | 1 claim change, 1 session change, 2 edits | 40,000 behaviours, 4.8M states | four minutes | | `Flap.cfg` | The publisher on relays 1 and 2, joined to each other, and relay 3 behind both; only the publisher's link to relay 1 opens and closes. Under the replaced rule, where doubt also kept relays off routes beginning at the publisher, `Routed` failed at once. Simulated. | 1 claim change, 4 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | -| `TwoHomes.cfg` | The same, where both of the publisher's links open and close. Drops `Stable`: a relay can take an offer older than the timer and drop a claim it showed. Simulated. | 1 claim change, 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | +| `TwoHomes.cfg` | The same, where both of the publisher's links open and close. Before `Patient` waited for stale offers too, a relay could take an offer naming a hop it had forgotten died and drop a claim it showed, breaking `Stable` within a minute. Simulated. | 1 claim change, 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | | `Homes.cfg` | The publisher on each of three relays in a ring that is cheap one way round, leaving all three. `RequestsLoopFree` fails within seconds; checks `LoopsLostThePublisher`. Simulated. | 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | | `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, a new seed each run | a few minutes | diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index 45fafc1b72d..7b28561ff31 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -55,7 +55,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was to be modeled and measured separately; the decision below covers it. - Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](https://github.com/moq-dev/moq/pull/8734)): ✅ **doubt blocks a route through a node, not one that begins at it, except the node's own route on a session to it that opened since the doubt began.** A publisher on two relays whose link to one flaps faster than the timer stays routed through the other: a relay that already hears a route through the other home that does not name it keeps one the whole time, as every relay does in the measured mesh (the two homes joined, a third relay behind both), and a relay whose neighbours all route through it is routed again once one of them re-offers, a round trip after the news, instead of after the timer. A publisher that moves to another relay is routed in 50 ms in a 24-relay clique and 325 ms in a ring, where it waited 1,100 and 1,850 ms. The returning link still waits out the timer, so no relay hears it until the relay it returned to takes it, which keeps the timer's damping of a flapping link: in 400 churn cases (120,000 steps) with the publisher on one relay, the rule costs 3% more route messages than timer-only doubt with no fewer relays routed, and with the publisher on two relays it routes relays 15% more often for 22% more route messages, 6% more claims sent, and 4% more claims taken offline. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`). The two-home churn, which flaps random links and restarts relays but never takes a publisher off three relays, shows a route loop in 51 of its 120,000 steps. With two homes the model finds no loop (`TwoHomes.cfg`), but when both links fail within one timer a relay can take an offer older than the timer and drop a claim it showed, where the replaced rule held it dark (`TwoHomes.cfg` drops `Stable`). This also settles the fresh-publisher quest, which proposed the opposite exemption and is deleted: its measurements showed that taking a returning link at once costs more traffic and claims offline than it saves. Rejected: keeping timer-only doubt, under which a dual-homed publisher's other link is unusable for the whole timer; sparing every route that begins at the doubted node, which routes relays 67% more often with two homes but costs 121% more route messages, 76% more claims sent, and 69% more claims offline, since each return of a flapping link reaches every relay; sparing only links heard before the doubt began, which keeps the damping but rarely helps a relay whose neighbours had not chosen the other link; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waited on [the switch](https://github.com/moq-dev/moq/pull/8731); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good. + Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](https://github.com/moq-dev/moq/pull/8734)): ✅ **doubt blocks a route through a node, not one that begins at it, except the node's own route on a session to it that opened since the doubt began.** A publisher on two relays whose link to one flaps faster than the timer stays routed through the other: a relay that already hears a route through the other home that does not name it keeps one the whole time, as every relay does in the measured mesh (the two homes joined, a third relay behind both), and a relay whose neighbours all route through it is routed again once one of them re-offers, a round trip after the news, instead of after the timer. A publisher that moves to another relay is routed in 50 ms in a 24-relay clique and 325 ms in a ring, where it waited 1,100 and 1,850 ms. The returning link still waits out the timer, so no relay hears it until the relay it returned to takes it, which keeps the timer's damping of a flapping link: in 400 churn cases (120,000 steps) with the publisher on one relay, the rule costs 3% more route messages than timer-only doubt with no fewer relays routed, and with the publisher on two relays it routes relays 15% more often for 22% more route messages, 6% more claims sent, and 4% more claims taken offline. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`). The two-home churn, which flaps random links and restarts relays but never takes a publisher off three relays, shows a route loop in 51 of its 120,000 steps. With two homes the model finds no loop and no claim dropped (`TwoHomes.cfg`), given a timer that outlasts every stale offer as well as the DEAD (`Patient`). This also settles the fresh-publisher quest, which proposed the opposite exemption and is deleted: its measurements showed that taking a returning link at once costs more traffic and claims offline than it saves. Rejected: keeping timer-only doubt, under which a dual-homed publisher's other link is unusable for the whole timer; sparing every route that begins at the doubted node, which routes relays 67% more often with two homes but costs 121% more route messages, 76% more claims sent, and 69% more claims offline, since each return of a flapping link reaches every relay; sparing only links heard before the doubt began, which keeps the damping but rarely helps a relay whose neighbours had not chosen the other link; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waited on [the switch](https://github.com/moq-dev/moq/pull/8731); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good. Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). diff --git a/rs/moq-net/verify/route/TwoHomes.cfg b/rs/moq-net/verify/route/TwoHomes.cfg index 8ad59c5032b..9e60c41bf2f 100644 --- a/rs/moq-net/verify/route/TwoHomes.cfg +++ b/rs/moq-net/verify/route/TwoHomes.cfg @@ -1,11 +1,9 @@ -\* Flap.cfg where both of the publisher's sessions open and close. A relay may -\* move onto an offer older than the doubt timer, naming a hop it has already -\* forgotten died, where doubt about the publisher kept it off before: the -\* exemption for a route beginning at the publisher allows it. When the -\* next message on that link names a hop it knows died, the relay lets go: -\* claims it showed go offline, then return once another route arrives. The -\* replaced rule kept the relay dark for the whole timer instead, so this -\* config drops `Stable`. +\* Flap.cfg where both of the publisher's sessions open and close. The exemption +\* for a route beginning at the publisher lets a relay move onto an offer through +\* its other link while doubt about the publisher stands. Before a patient timer +\* waited for stale offers as well as DEAD news, a relay could forget a dead hop +\* while an offer naming it was still queued, take that offer, and drop a claim it +\* showed once the next message named the hop again, breaking `Stable`. \* Simulate: -simulate num=1000 -depth 150 -seed 1 CONSTANTS Nodes = {0, 1, 2, 3} @@ -42,3 +40,4 @@ INVARIANTS Routed QuietIsSettled Budget + Stable From da31c0981d7274abf65ef2af975218bfd367d5ee Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 09:40:53 -0800 Subject: [PATCH 12/12] docs(draft): the request-loop guarantee names the three-home exception Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 16af6695659..689d147b636 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -553,7 +553,7 @@ An origin never advertises a Revision of 2^64-1: a change that would reach it in A SUBSCRIBE, FETCH, TRACK, TRACKS, or DATAGRAMS request names the origin node that serves it, and the Route Depth of the route it follows: how many leading segments of its Broadcast Path make up the route's prefix. A subscriber that names 0 lets the publisher select one (see [Resolution](#resolution)), and a relay forwarding a request names the origin node and Route Depth of the route it selected or was given. -A relay forwards a request naming an origin node only along the route it sent the requester to that node at that prefix, or a SUBSCRIBE through the neighbor [Warm Routes](#warm-routes) chooses for its broadcast, and resets it with UNROUTABLE when that route is no longer usable or its Path names the requester, so the request never reaches another origin or loops. +A relay forwards a request naming an origin node only along the route it sent the requester to that node at that prefix, or a SUBSCRIBE through the neighbor [Warm Routes](#warm-routes) chooses for its broadcast, and resets it with UNROUTABLE when that route is no longer usable or its Path names the requester, so the request never reaches another origin, and loops only briefly after an origin on three or more relays leaves them all at once (see [Dead Hops](#dead-hops)). Claims with the same Instance serve the same Broadcast, so a relay MAY move a live subscription between them, including onto a higher Epoch, continuing from the first frame the subscriber lacks, and a route change (a reconnect, a cheaper path, a draining session) is invisible to the subscriber. A relay MUST NOT move a subscription to a claim with a different Instance: it stays on its Instance's claims and ends with the last of them.