diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 0e5723ea3a2..07fcca1890b 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -5,7 +5,7 @@ description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteH # Verify the route rule -`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. A route learned on a link its learner marks upstream, or from a down-only route, is down-only, and no relay sends one over a link it marks upstream (`Upstream`, the links whose higher-numbered end marks them), so no relay carries traffic between two upstreams. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` start out watching alone), and keeps dead hops until their timer fires. A relay asks each neighbour for the set of prefixes it watches, none inside another; a relay in `Editors` may change that set up to `MaxEdits` times. Each change is an edit, sent as an ask on the FIFO and answered by a marker on the neighbour's FIFO, where both sides rekey the routes between them alike (`Rekey`): a key the new set covers stays, the route at a removed `"a"` moves to an added `"root"` without its floor, and the route at a removed `"root"` carries its floor, presented at `"a"`, to an added `"a"`. A heard route is believed for a prefix only while the relay asked for it at the last marker and in every edit still in flight. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no message naming a closed session it covers is in flight: no DEAD, and no stale offer sent before the news. Every lite-07 session drives the path router through `route::node::Node` since [the switch](https://github.com/moq-dev/moq/pull/8731), and `route::tests::Sim` drives it under `Rule::PATH` (timer-only doubt). +`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires, keeping relays off routes through it but not off a route that begins at it, unless that route comes over a relay's own session to it opened since the doubt began. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. A route learned on a link its learner marks upstream, or from a down-only route, is down-only, and no relay sends one over a link it marks upstream (`Upstream`, the links whose higher-numbered end marks them), so no relay carries traffic between two upstreams. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` start out watching alone), and keeps dead hops until their timer fires. A relay asks each neighbour for the set of prefixes it watches, none inside another; a relay in `Editors` may change that set up to `MaxEdits` times. Each change is an edit, sent as an ask on the FIFO and answered by a marker on the neighbour's FIFO, where both sides rekey the routes between them alike (`Rekey`): a key the new set covers stays, the route at a removed `"a"` moves to an added `"root"` without its floor, and the route at a removed `"root"` carries its floor, presented at `"a"`, to an added `"a"`. A heard route is believed for a prefix only while the relay asked for it at the last marker and in every edit still in flight. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no message naming a closed session it covers is in flight: no DEAD, and no stale offer sent before the news. Every lite-07 session drives the path router through `route::node::Node` since [the switch](https://github.com/moq-dev/moq/pull/8731), and `route::tests::Sim` drives it under `Rule::PATH` (timer-only doubt). ## Rules @@ -28,8 +28,10 @@ Each rule is an INVARIANT; the harm says why it matters. | `QuietIsSettled` | Once nothing is in flight and every timer fired: each relay shows exactly the claims a path of open sessions through relays watching them carries to it, never up a link its sender marks upstream after one its learner does, available, and a request for each reaches the publisher; no relay holds, keeps a stale route, or remembers a dead hop; and both ends of each session agree on what it asks for. | A hold never ends, a claim is shown that no path carries, a relay behind a scoped neighbour never reaches what a farther one offers, or an edit never takes effect. | | `Budget` | From one quiet point to the next, the messages sent stay within 4 each way over each configured session per prefix and per claim, for each external change. | One change sets off announce spam, or routing never goes quiet. | | `RequestsLoopFree` | A request never loops. | A request cycles through relays instead of reaching the publisher. | +| `LoopsLostThePublisher` | A request that loops goes round relays whose routes each begin on a closed session of the publisher. | A loop forms while the publisher is still reachable from the relays on it. | +| `Routed` | A relay that knows a route over open sessions, crossing no relay it doubts and not a link to the publisher that came back during its doubt, uses some route. | Doubt about the publisher alone darkens a relay with a way to it, as when one of a dual-homed publisher's links flaps. | -Every config checks `RequestsLoopFree`; `Loop.cfg` pins the three-failure regression. +Every config but `Homes.cfg` checks `RequestsLoopFree`; `Loop.cfg` pins the three-failure regression. `Homes.cfg` checks `LoopsLostThePublisher` instead: doubt spares a route that begins at the doubted node, so a publisher leaving three relays at once can send a request round them until the news arrives. `Flaky` names the links that open and close; `Flap.cfg`, `TwoHomes.cfg`, and `Homes.cfg` narrow it to the publisher's own (`OwnLinks`) or one of them (`FirstLink`). No config checks a liveness property: `Spec` has no fairness, so TLC does not show that a hold ends. That one does is an argument over what TLC does check. `Budget` bounds the messages a burst sends, each delivery takes one message off a queue, and a doubt starts only at a close or a delivery, so a burst is a finite number of steps. A delivery is enabled while a queue holds a message and a timer may fire once no news about its node is in flight, so a burst left alone reaches the state with nothing in flight and no doubt. `QuietIsSettled` says no relay holds there. @@ -68,7 +70,7 @@ change; faster checkers are separate work. | Config | Mesh | Bounds | Previous distinct states | Previous run | | --- | --- | --- | --- | --- | -| `RouteHandoff.cfg` | The publisher behind relay 1, in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 476,434 | a minute or two | +| `RouteHandoff.cfg` | The publisher behind relay 1, in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 434,956 | a minute or two | | `Early.cfg` | The same, with a timer that fires at any moment. | 1 claim change, 1 session change | 14,220,217 | half an hour | | `Twice.cfg` | The same, where two sessions close, as when relay 1 dies. Simulated: `num` counts behaviours per worker. | 1 claim change, 2 session changes | 40,000 behaviours, 4.0M states | a few minutes | | `Ring.cfg` | The publisher and three relays in a ring, with a timer that fires at any moment. | 1 claim change, 1 session change | 813,075 | a minute or two | @@ -78,17 +80,20 @@ change; faster checkers are separate work. | `Blind.cfg` | A relay whose nearest neighbour asks for `"a"` alone, and a farther one knows everything. Simulated. | 2 claim changes, 1 session change | 40,000 behaviours, 4.8M states | ten minutes | | `Overlap.cfg` | Five relays, three asking for `"a"`, with overlapping cheap broad and dear narrow paths. Simulated past the message budget. | 1 claim change, no session change | 1,000 behaviours per worker (4,000 on four workers), depth 500 | simulation | | `Loop.cfg` | The publisher with a session to each of three relays in a triangle, all closing. Samples `RequestsLoopFree` with timer-only doubt. | 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | simulation | -| `Drain.cfg` | The publisher joined to two relays joined to each other; both hear its GOAWAY. Ranking only a relay's own drain exceeded `Budget`. | 1 claim change, 1 session change, GOAWAY on each session | 511,147 | two minutes | -| `DrainRing.cfg` | The publisher joined to three relays in a ring that is cheap one way round (`Ring`); all hear its GOAWAY. Without the DRAIN news and its doubt, a request looped round the ring with no session closed, breaking `LoopsAreStale`. Simulated. | 1 claim change, GOAWAY on each session | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | -| `Prefer.cfg` | The same triangle, where both relays prefer the link between them. Ranking only a relay's own link priority exceeded `Budget`. | 1 claim change, 1 session change | 12,654 | seconds | -| `Upstream.cfg` | Two uplinks: the publisher on a CDN relay joined by a dear link to another, and a local mesh of two relays, each with an upstream link to one CDN relay. With the CDN link down, the second CDN relay has no route rather than one through the local mesh. The Rust fixture is `two_uplinks_never_carry_cdn_transit`. Without the export check, or with a flag that a LAN hop clears, `QuietIsSettled` and `DownOnly` fail within seconds. | 1 claim change, 1 session change | 68,717 | seconds | -| `UpstreamScoped.cfg` | The same, where the second CDN relay asks for `"a"` alone, so the stream to it over an upstream link picks among covering routes that are all down-only and ends. | 1 claim change, 1 session change | 69,576 | seconds | +| `Drain.cfg` | The publisher joined to two relays joined to each other; both hear its GOAWAY. Ranking only a relay's own drain exceeded `Budget`. | 1 claim change, 1 session change, GOAWAY on each session | 575,887 | two minutes | +| `DrainRing.cfg` | The publisher joined to three relays in a ring that is cheap one way round (`Ring`); all hear its GOAWAY. Without the DRAIN news, which ranks every path to the publisher as draining alike, a request looped round the ring with no session closed, breaking `LoopsAreStale`. Simulated. | 1 claim change, GOAWAY on each session | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | +| `Prefer.cfg` | The same triangle, where both relays prefer the link between them. Ranking only a relay's own link priority exceeded `Budget`. | 1 claim change, 1 session change | 15,380 | seconds | +| `Upstream.cfg` | Two uplinks: the publisher on a CDN relay joined by a dear link to another, and a local mesh of two relays, each with an upstream link to one CDN relay. With the CDN link down, the second CDN relay has no route rather than one through the local mesh. The Rust fixture is `two_uplinks_never_carry_cdn_transit`. Without the export check, or with a flag that a LAN hop clears, `QuietIsSettled` and `DownOnly` fail within seconds. | 1 claim change, 1 session change | 76,992 | seconds | +| `UpstreamScoped.cfg` | The same, where the second CDN relay asks for `"a"` alone, so the stream to it over an upstream link picks among covering routes that are all down-only and ends. | 1 claim change, 1 session change | 71,680 | seconds | | `EditNarrow.cfg` | The publisher with sessions to relays 1 and 2, the one to relay 2 dear; relay 2 watches everything and narrows to `"a"` or drops it once. | 1 claim change, 1 session change, 1 edit | 768,483 | three minutes | | `EditWiden.cfg` | The same, where relay 2 watches `"a"` and widens to everything or drops it once. | 1 claim change, 1 session change, 1 edit | 3,655,573 | a quarter of an hour | | `Edit.cfg` | The same with two edits, such as widening then narrowing back, or dropping everything then asking again. Before the rekey kept the route the sender would reflect, a widening then a narrowing crossing the close of relay 2's own session kept a route relay 1 learned through it and took a claim offline, breaking `Stable`. Simulated; an exhaustive pass covered every behaviour to depth 18 (10.6M states) before it was stopped. | 1 claim change, 1 session change, 2 edits | 40,000 behaviours, 4.8M states | four minutes | +| `Flap.cfg` | The publisher on relays 1 and 2, joined to each other, and relay 3 behind both; only the publisher's link to relay 1 opens and closes. Under the replaced rule, where doubt also kept relays off routes beginning at the publisher, `Routed` failed at once. Simulated. | 1 claim change, 4 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | +| `TwoHomes.cfg` | The same, where both of the publisher's links open and close. Before `Patient` waited for stale offers too, a relay could take an offer naming a hop it had forgotten died and drop a claim it showed, breaking `Stable` within a minute. Simulated. | 1 claim change, 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | +| `Homes.cfg` | The publisher on each of three relays in a ring that is cheap one way round, leaving all three. `RequestsLoopFree` fails within seconds; checks `LoopsLostThePublisher`. Simulated. | 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | | `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, a new seed each run | a few minutes | -Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. +Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare|flap)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. ## Search the router @@ -104,13 +109,14 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on th ## Coverage -Sync commit: `857f2eea4ac4cc7a3fb821f7d250637eb464b248`, where [the switch](https://github.com/moq-dev/moq/pull/8731) took in [the interest set](https://github.com/moq-dev/moq/pull/8728). Run `git log 857f2eea4.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SelectKey` sets a route's down-only flag as `Router::usable_offer` does, and `Want0` ends a live down-only route toward an upstream link. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, leaving out down-only routes toward an upstream link and falling back to a held one, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `cbb742cecae437ee94fbf25848521ac11b145fbc`, where [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734), which made doubt spare the node a route begins at, took in [the switch](https://github.com/moq-dev/moq/pull/8731). Run `git log cbb742cec.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SelectKey` sets a route's down-only flag as `Router::usable_offer` does, and `Want0` ends a live down-only route toward an upstream link. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, leaving out down-only routes toward an upstream link and falling back to a held one, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model; the comparison-only `Rule::ORIGIN` lets doubt block a route that begins at the doubted node too, which `Flap.cfg` fails with `Doubted` defined as it was, and `Rule::SPARED` lets it block none, even a returning link. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | | `rs/moq-net/src/route/path.rs` `Router::run`, `Router::select`, `Router::retain` | `Select` plans live routes and their reflected sources before `SelectKey` decides holds. | | `rs/moq-net/src/route/path.rs` `Offer::rank`, `Router::goaway`, `Router::drains`, `Router::report_draining`, `Router::dead` (`Dead::Draining`) | `Rank`, `Below`, `PathPrio`, `CostAt`, `GoAway`, `MarkDrain`, `Credited`, `ReportDrain`, `Recv` | -| `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted` | +| `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted`, `RelayDoubted` (a path's first node does not count), `Returned` | +| `rs/moq-net/src/route/path.rs` `Router::suspect`, `Router::doubt_until` | `Since`: a doubt on the publisher that begins notes the generation of the relay's own session to it. The router shares one generation across a session's streams (`Session::generation`), which the model's one stream per session leaves out. | | `rs/moq-net/src/route/path.rs` `Router::src` | `SrcKey` | | `rs/moq-net/src/route/path.rs` `Router::usable_offer`, `Offer` | `SelectKey`'s `Down`, `Want0`'s export check, and `DownOnly`: the down-only flag is set on a route learned over an upstream link or from a down-only one, and never sent over an upstream link. | | `rs/moq-net/src/route/path.rs` `Router::want`, `Router::stream`, `Router::advertise`, `Router::report` | `Want`, `Want0`, `Streams`, `Advertise`, `Report` | diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 37e6584c5bb..689d147b636 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -542,7 +542,7 @@ Its priority is the lowest local priority of a session on its Path: a receiver t A node ranks its usable routes to an origin node at a prefix: one that is not draining first, then the highest priority, then the lowest metric. Extending a route never clears its drain, never raises its priority, and always raises its metric, so no node ranks a route above how the neighbour it came from ranked it, and selection converges. A node selects its best-ranked usable route at once. -A route is unusable while its Path names the node, it is held, a hop on its Path is dead, a node on its Path is in doubt (see [Dead Hops](#dead-hops)) unless it is the route the node already uses, it arrived with 32 hops, or its metric overflows. +A route is unusable while its Path names the node, it is held, a hop on its Path is dead, the doubt keeps the node off it (see [Dead Hops](#dead-hops)), it arrived with 32 hops, or its metric overflows. A node MAY mark a session upstream. A route is down-only when it arrived on an upstream session or with its down-only flag set; a node sends a down-only route with the flag set, and never on an upstream session. @@ -553,7 +553,7 @@ An origin never advertises a Revision of 2^64-1: a change that would reach it in A SUBSCRIBE, FETCH, TRACK, TRACKS, or DATAGRAMS request names the origin node that serves it, and the Route Depth of the route it follows: how many leading segments of its Broadcast Path make up the route's prefix. A subscriber that names 0 lets the publisher select one (see [Resolution](#resolution)), and a relay forwarding a request names the origin node and Route Depth of the route it selected or was given. -A relay forwards a request naming an origin node only along the route it sent the requester to that node at that prefix, or a SUBSCRIBE through the neighbor [Warm Routes](#warm-routes) chooses for its broadcast, and resets it with UNROUTABLE when that route is no longer usable or its Path names the requester, so the request never reaches another origin or loops. +A relay forwards a request naming an origin node only along the route it sent the requester to that node at that prefix, or a SUBSCRIBE through the neighbor [Warm Routes](#warm-routes) chooses for its broadcast, and resets it with UNROUTABLE when that route is no longer usable or its Path names the requester, so the request never reaches another origin, and loops only briefly after an origin on three or more relays leaves them all at once (see [Dead Hops](#dead-hops)). Claims with the same Instance serve the same Broadcast, so a relay MAY move a live subscription between them, including onto a higher Epoch, continuing from the first frame the subscriber lacks, and a route change (a reconnect, a cheaper path, a draining session) is invisible to the subscriber. A relay MUST NOT move a subscription to a claim with a different Instance: it stays on its Instance's claims and ends with the last of them. @@ -575,7 +575,9 @@ Then every route through the hop is dead, whoever sent it: the sender's routes c The receiver remembers the dead hop, so a route through it that arrives late stays unusable. A closed session also casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time: a neighbour that has not heard yet may still offer a stale route through it. -A receiver doubts the far node for one second, during which it moves onto no route through that node; the route it already uses stays usable. +A receiver doubts the far node for one second, during which it moves onto no route whose Path names that node after the origin node's hop, and onto no route the doubted node originates whose Path is that node's hop alone, received on a session that opened since the doubt began; the route it already uses stays usable. +Any other route the doubted node originates stays usable, since a route cannot loop through its own origin: an origin on several relays stays reachable through the others while its session to one of them flaps, and a session that reopens waits out the doubt, which damps the flaps. +A node cannot tell an origin leaving from its session to one relay closing, so an origin on three or more relays that leaves them all at once can send a request round them, each relay on a neighbour's route that the neighbour's own close made stale, until that neighbour's DEAD arrives. Nothing on the wire carries the doubt. A node that sees a session close doubts its far end in the same way. @@ -2552,7 +2554,7 @@ An application-specific payload. - ROUTE_START and ROUTE_UPDATE carry Flags, whose 0x2 bit adds an optional RTT: the informational sum of each link's lowest smoothed RTT over the last 30 seconds to the origin node, which never selects a route. - ROUTE_START and ROUTE_UPDATE carry a Route Offer that leads with the route's Path: one hop per node the route crossed, each a Hop ID and the Session Number that node learned the route on, at most 32. A node never uses a route whose Path names it. ROUTE_START also carries a Route Suffix naming the prefix the route covers, and the offer carries the route's drain (Flags 0x4) and priority, which rank before its metric. - Added ROUTE_HOLD (0xB), which holds a route while it may recover, for at most 10 seconds; ROUTE_END lets it go. Neither withdraws the origin's claims. -- Added DEAD (0xF): a node that loses a session names that hop once to each neighbour it sent a route through it, listing the routes that moved, and every node drops every route through the hop and doubts the node at the far end for a second; a DRAIN names a node that sent a GOAWAY, whose routes rank draining. The path, the dead hop, holds, and the doubt keep routes free of loops and stale copies, with no distance, sequence number, route request, or echo. On a scoped session a DEAD applies only to the routes heard there. +- Added DEAD (0xF): a node that loses a session names that hop once to each neighbour it sent a route through it, listing the routes that moved, and every node drops every route through the hop and doubts the node at the far end for a second, keeping nodes off routes through it but not off one it originates, except over a session to it opened since; a DRAIN names a node that sent a GOAWAY, whose routes rank draining. The path, the dead hop, holds, and the doubt keep routes free of stale copies, and of loops except briefly when an origin on three or more relays leaves them all at once, with no distance, sequence number, route request, or echo. On a scoped session a DEAD applies only to the routes heard there. - A node closes a session whose peer leaves an Announce Stream undrained for 10 seconds, and the subscriber sends nothing on the stream after ANNOUNCE_REQUEST. - Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the route's prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows. - A route presents only its most specific claim covering the requested prefix, moved to the next covering claim when it is withdrawn rather than ended. diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index 01ff7e29b8e..7b28561ff31 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -49,12 +49,14 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout - A hop names a relay and the session it learned the route on. A relay uses the cheapest route whose path does not name it, at once. - A relay that loses a session names that hop, and every relay drops every route through it, whoever offered it. One message per session covers every publisher and lists the routes that moved to another path. - A relay with no route left holds its claims, unavailable, until each neighbour that learned the route from it has answered: with another way, or with nothing once those behind it have answered. Then it lets go. - - A closed session casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time. No relay moves to a route through that node until a timer lifts the doubt. The route a relay already uses is exempt. + - A closed session casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time. No relay moves to a route through that node until a timer lifts the doubt. The route a relay already uses is exempt, and so is a route that begins at that node, unless it comes over a session to that node opened since the doubt began (below). The maintainer delegated these decisions to the coordinator while away, recorded 2026-11-13: "The user is away; the coordinator takes recommendations on their behalf, open to change in review." Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was to be modeled and measured separately; the decision below covers it. + Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](https://github.com/moq-dev/moq/pull/8734)): ✅ **doubt blocks a route through a node, not one that begins at it, except the node's own route on a session to it that opened since the doubt began.** A publisher on two relays whose link to one flaps faster than the timer stays routed through the other: a relay that already hears a route through the other home that does not name it keeps one the whole time, as every relay does in the measured mesh (the two homes joined, a third relay behind both), and a relay whose neighbours all route through it is routed again once one of them re-offers, a round trip after the news, instead of after the timer. A publisher that moves to another relay is routed in 50 ms in a 24-relay clique and 325 ms in a ring, where it waited 1,100 and 1,850 ms. The returning link still waits out the timer, so no relay hears it until the relay it returned to takes it, which keeps the timer's damping of a flapping link: in 400 churn cases (120,000 steps) with the publisher on one relay, the rule costs 3% more route messages than timer-only doubt with no fewer relays routed, and with the publisher on two relays it routes relays 15% more often for 22% more route messages, 6% more claims sent, and 4% more claims taken offline. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`). The two-home churn, which flaps random links and restarts relays but never takes a publisher off three relays, shows a route loop in 51 of its 120,000 steps. With two homes the model finds no loop and no claim dropped (`TwoHomes.cfg`), given a timer that outlasts every stale offer as well as the DEAD (`Patient`). This also settles the fresh-publisher quest, which proposed the opposite exemption and is deleted: its measurements showed that taking a returning link at once costs more traffic and claims offline than it saves. Rejected: keeping timer-only doubt, under which a dual-homed publisher's other link is unusable for the whole timer; sparing every route that begins at the doubted node, which routes relays 67% more often with two homes but costs 121% more route messages, 76% more claims sent, and 69% more claims offline, since each return of a flapping link reaches every relay; sparing only links heard before the doubt began, which keeps the damping but rarely helps a relay whose neighbours had not chosen the other link; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waited on [the switch](https://github.com/moq-dev/moq/pull/8731); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good. + Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). Declined 2026-11-14 (maintainer), timer-only doubt retained over it: a fresh publisher's exemption, under which a relay moves at once onto a doubted publisher's own route when an authenticated session the publisher opened after the doubt began offers it. [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) modeled and measured it. The model found no broken rule (1.8M states, every ordering of the publisher rejoining two joined relays), but the exemption frees only the publisher's own relay: every other relay still doubts the publisher until its timer fires, and a publisher that moves to another relay gains nothing. Under churn it shows each flap of a publisher's link that the timer damps. Measured on 24 relays (sparse: 16 meshes; churn: 400 cases), recovery in ms, and dark counting observations of a reachable publisher with no route: diff --git a/quest/m2/cluster-routes/flapping-publisher.md b/quest/m2/cluster-routes/flapping-publisher.md deleted file mode 100644 index 43a34e0127c..00000000000 --- a/quest/m2/cluster-routes/flapping-publisher.md +++ /dev/null @@ -1,66 +0,0 @@ -# [L] A dual-homed publisher stays reachable while one link flaps - -## Goal - -A publisher connected to two relays whose link to one of them closes and -reopens faster than the doubt timer stays reachable at every relay through -its other link, the whole time. The path rule's doubt still keeps relays -off routes through a dead relay until its timer fires. - -## Plan - -Decided 2026-11-13 (maintainer, /quest-plan on the BGP failure modes): ✅ fix -it, checked in TLA+ as well as on real sessions (rejected: pinning today's -behaviour as an accepted cost). - -The bug, as the path rule stands ([epic](/quest/m2/cluster-routes/README.md) -Plan item 27): a closed session casts doubt on the node at its far end, and -every relay that hears the DEAD doubts that node. When the closed session is -the publisher's own, the doubted node is the publisher, and every route to -its claims names it, the route through its healthy link included. A relay -that was using the closed link cannot move to the healthy one, and each flap -opens a new session whose close extends the doubt. While the link keeps -flapping faster than the timer, those relays have no route although the -publisher was never unreachable. [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622) -measured the general cost: 2.87× more reachable-but-unrouted observations -under timer-only doubt. - -✅ Model first: doubt blocks a route that passes through a node, not one that -ends at it as its origin (rejected: capping a doubt at one timer from the -first close, which still leaves a gap after each flap; modeling both). Doubt -exists because a dead relay is reported one neighbour at a time, so a relay -could move onto a stale route through it and loop, and a route cannot loop -through its own endpoint. The model has to show what the rule gives up: when -the origin really dies, a relay may move onto a stale route to it that a -neighbour's DEAD has not yet withdrawn. Check that such a route is still -chased out (`StaleIsChased`) and that requests stay loop-free -(`RequestsLoopFree`). Decide in the PR what counts as the origin when a -relay originates a route for a broadcast it carries -([cache-aware routing](/quest/m3/cache-routing/README.md)). - -`fresh-publisher.md`, added by #8622, exempts only a publisher's own new -direct session from doubt, so it never helps a relay further away. If this -rule holds, it covers that case too: fold that quest into this one or delete -it, whichever has landed first. - -Work: - -- `rs/moq-net/verify/route/RouteHandoff.tla`: a config with a publisher on - two relays and flaps on one link, and a property that a relay with a path - of sessions that never closed to the publisher routes to it once that path - is known. Run the existing configs under the new rule. -- The path router (`route::path`, from - [the router](https://github.com/moq-dev/moq/pull/8622)) applies the rule, with the - model's counterexamples reproduced there first. -- A fixture on real sessions with mocked time: the dual-homed publisher with - one link flapping at half the timer, where a subscriber at a far relay - receives every group while it flaps. -- Re-run #8622's reconnect and churn measurements under the new rule and - report reachable-but-unrouted observations, stale routes, and loops beside - timer-only doubt. -- Update the epic's Plan item 27, the draft's Routing section, and the - `verify-route` skill. - -## Related - -- [A request loop heals once routes settle](/quest/m3/cluster-routing/request-loop.md) - another failure mode from the same BGP comparison diff --git a/quest/m3/cluster-routing/request-loop.md b/quest/m3/cluster-routing/request-loop.md index b50a15f1fdb..34be16918a9 100644 --- a/quest/m3/cluster-routing/request-loop.md +++ b/quest/m3/cluster-routing/request-loop.md @@ -44,4 +44,4 @@ pending request (the front in `rs/moq-net/src/model/front.rs`, and how ## Related -- [A dual-homed publisher stays reachable while one link flaps](/quest/m2/cluster-routes/flapping-publisher.md) - another failure mode from the same BGP comparison +- [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) - its doubt rule lets a publisher leaving three or more relays at once loop a request until the news arrives diff --git a/rs/moq-net/src/route/path.rs b/rs/moq-net/src/route/path.rs index 2707df16d36..56806670bed 100644 --- a/rs/moq-net/src/route/path.rs +++ b/rs/moq-net/src/route/path.rs @@ -10,7 +10,9 @@ //! leaving node in a [`Dead::Draining`] to each neighbour it sent a route through //! it, which passes it on the same way, and each node doubts the leaving node as //! a closed session's far end is doubted, so no node moves onto a stale offer -//! through it and a request cannot loop through offers that predate the drain. When a session closes, +//! through it and a request cannot loop through offers that predate the drain. +//! When the leaving node is the publisher, every path to it ranks as draining +//! alike, so no node moves at all. When a session closes, //! the node that lost it names that hop in one [`Dead::Closed`] per neighbour, which //! lists the routes that moved to another path, and every node drops every route //! through the hop, whoever offered it. A node with no route left holds its @@ -20,7 +22,12 @@ //! holds only the path it last sent, so holds never close a cycle. A closed //! session casts doubt on the node at its far end, since a dead relay is reported //! one neighbour at a time: no node moves to a route through it until a timer -//! lifts the doubt. There is no distance, seqno, route request, or echo. +//! lifts the doubt. A route that begins at the doubted node is exempt, so a +//! publisher stays reachable through its other links while one flaps, except on +//! a session to it opened since the doubt began: a link that comes back waits +//! out the timer, which damps its flaps. When a publisher leaves three or more +//! relays at once, requests can go round them until the news arrives. There is +//! no distance, seqno, route request, or echo. //! //! A route covers a prefix: the one its receiver asked for, or a narrower one a //! node on its path asked for. A stream answering a prefix carries the sender's @@ -140,7 +147,8 @@ pub(crate) enum Dead { /// `node` sent GOAWAY: every path through it ranks as draining, the worst rank, /// and stays usable. The news casts doubt on the node, so no relay moves onto a /// stale offer through it until the timer fires, and a request cannot loop through - /// offers that predate the drain. + /// offers that predate the drain. A publisher's drain needs no doubt: every path to + /// it ranks as draining alike, so no relay moves. Draining { node: Id }, } @@ -202,6 +210,19 @@ pub(crate) enum Lift { Timer, } +/// What doubt does to a route that begins at the doubted node. +#[cfg(test)] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum AtOrigin { + /// The replaced comparison rule: it blocks the route, as it does one through the node. + Blocked, + /// The rejected comparison rule: it never blocks the route. + Spared, + /// It blocks the route only on this node's own session to the node, opened + /// since the doubt began. + Own, +} + /// How a closed session's doubt works. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) struct Doubt { @@ -209,6 +230,8 @@ pub(crate) struct Doubt { pub timer: Duration, #[cfg(test)] pub lift: Lift, + #[cfg(test)] + pub origin: AtOrigin, } impl Doubt { @@ -217,6 +240,8 @@ impl Doubt { timer: Duration::from_secs(1), #[cfg(test)] lift: Lift::Timer, + #[cfg(test)] + origin: AtOrigin::Own, }; } @@ -286,19 +311,23 @@ impl Death { /// A node's timed doubt. Tests also represent early collection before the /// timer retires. -#[derive(Clone, Copy, Debug)] +#[derive(Clone, Debug)] enum Suspicion { - Doubted(Instant), + Doubted { + until: Instant, + /// The last session this node opened before the doubt began. + since: u64, + }, #[cfg(test)] Lifted(Instant), } impl Suspicion { - fn deadline(self) -> Instant { + fn deadline(&self) -> Instant { match self { - Self::Doubted(at) => at, + Self::Doubted { until, .. } => *until, #[cfg(test)] - Self::Lifted(at) => at, + Self::Lifted(at) => *at, } } } @@ -433,6 +462,10 @@ impl Route { struct Session { /// How this node names the session in the hops of the routes it learns there. number: u64, + /// The `number` of the first stream of the neighbour's session still registered, + /// which its later streams share: doubt asks whether the session, not the stream, + /// opened since it began. + generation: u64, /// How the neighbour names itself in a path. hop: Id, link: Link, @@ -882,8 +915,14 @@ impl Router { return Err(Error::ZeroCost); } self.opened += 1; + // A later stream of a session already open, such as one a new permit files, is + // no returning link. The tests' `Sim` leaves `link.session` 0: a session per stream. + let generation = (self.sessions.values()) + .find(|other| link.session != 0 && other.link.session == link.session) + .map_or(self.opened, |other| other.generation); let session = Session { number: self.opened, + generation, hop, link, rtt: None, @@ -1831,7 +1870,15 @@ impl Router { .map(|(hop, death)| (hop, death.far(), death.deadline().map(left))) .collect(); let doubted: Vec<_> = (doubted.iter()) - .map(|(node, state)| (node, matches!(state, Suspicion::Doubted(_)), left(state.deadline()))) + .map(|(node, state)| { + // `since` decides whether a returning link waits, so it is state. + let since = match state { + Suspicion::Doubted { since, .. } => Some(*since), + #[cfg(test)] + Suspicion::Lifted(_) => None, + }; + (node, since, left(state.deadline())) + }) .collect(); let held: Vec<_> = (held_until.iter()).map(|(key, at)| (key, left(*at))).collect(); super::digest(hasher, &(hop, doubt, opened, sessions, answering, origins, rx)); @@ -1927,10 +1974,42 @@ impl Router { path.iter().any(|hop| self.draining.contains(&hop.node)) } - /// Whether `path` crosses a node in doubt. - fn doubts_path(&self, path: &[Hop]) -> bool { - path.iter() - .any(|hop| matches!(self.doubted.get(&hop.node), Some(Suspicion::Doubted(_)))) + /// Whether doubt keeps this node off the route `peer` offered with `path`: one + /// through a node in doubt, or the doubted node's own route on a session to it + /// that opened since the doubt began. A route cannot loop through its own end, so + /// the publisher's other links stay usable while one of them flaps; a link that + /// comes back waits out the timer, which damps the flaps, and no other node + /// hears it until this one takes it. + fn doubts_path(&self, peer: N, path: &[Hop]) -> bool { + let doubted = |node: Id| matches!(self.doubted.get(&node), Some(Suspicion::Doubted { .. })); + let Some((first, rest)) = path.split_first() else { + return false; + }; + if rest.iter().any(|hop| doubted(hop.node)) { + return true; + } + let Some(Suspicion::Doubted { since, .. }) = self.doubted.get(&first.node) else { + return false; + }; + #[cfg(test)] + match self.doubt.origin { + AtOrigin::Blocked => return true, + AtOrigin::Spared => return false, + AtOrigin::Own => {} + } + rest.is_empty() && self.sessions[&peer].generation > *since + } + + /// Doubt `node` until `at`, or later if it already is. A doubt that begins notes + /// the last session opened so far, so a session to `node` opened since waits. + fn doubt_until(&mut self, node: Id, at: Instant) { + if let Some(Suspicion::Doubted { until, .. }) = self.doubted.get_mut(&node) { + *until = (*until).max(at); + return; + } + let until = self.doubted.get(&node).map_or(at, |state| state.deadline().max(at)); + let since = self.opened; + self.doubted.insert(node, Suspicion::Doubted { until, since }); } fn hear(&mut self, key: RouteId, from: N, heard: Heard) { @@ -2016,15 +2095,14 @@ impl Router { } /// `node` sent GOAWAY, as of `now`. The first word of it casts doubt on the node, - /// so no stale offer through it is taken before the timer fires. Returns the + /// so no stale offer through it is taken before the timer fires; a route that + /// begins at it ends there and cannot loop. Returns the /// identities whose routes rank anew. fn drains(&mut self, node: Id, now: Instant) -> BTreeSet { if !self.draining.insert(node) { return BTreeSet::new(); } - let at = now.saturating_add(self.doubt.timer); - let at = self.doubted.get(&node).map_or(at, |state| state.deadline().max(at)); - self.doubted.insert(node, Suspicion::Doubted(at)); + self.doubt_until(node, now.saturating_add(self.doubt.timer)); self.naming(node) } @@ -2079,9 +2157,7 @@ impl Router { if far == self.hop { return BTreeSet::new(); } - let at = now.saturating_add(self.doubt.timer); - let at = self.doubted.get(&far).map_or(at, |state| state.deadline().max(at)); - self.doubted.insert(far, Suspicion::Doubted(at)); + self.doubt_until(far, now.saturating_add(self.doubt.timer)); // Any route through that node may have become unusable. self.naming(far) } @@ -2200,7 +2276,7 @@ impl Router { } let session = &self.sessions[&peer]; let metric = heard.offer.metric.checked_add(session.link.cost)?; - if Some(peer) != using && self.doubts_path(&heard.offer.path) { + if Some(peer) != using && self.doubts_path(peer, &heard.offer.path) { // It may come back once the doubt lifts. return None; } @@ -2331,7 +2407,7 @@ impl Router { heard.state == State::Live && !names(&heard.offer.path, self.hop) && heard.offer.metric.checked_add(self.sessions[peer].link.cost).is_some() - && self.doubts_path(&heard.offer.path) + && self.doubts_path(*peer, &heard.offer.path) }) } @@ -3381,7 +3457,9 @@ impl Router { let lifted: Vec = self .doubted .iter() - .filter_map(|(node, state)| (matches!(state, Suspicion::Doubted(_)) && !self.named(*node)).then_some(*node)) + .filter_map(|(node, state)| { + (matches!(state, Suspicion::Doubted { .. }) && !self.named(*node)).then_some(*node) + }) .collect(); for node in lifted { let at = self.doubted[&node].deadline(); diff --git a/rs/moq-net/src/route/tests/cache.rs b/rs/moq-net/src/route/tests/cache.rs index 192b4a580ba..b270e4e2149 100644 --- a/rs/moq-net/src/route/tests/cache.rs +++ b/rs/moq-net/src/route/tests/cache.rs @@ -1146,7 +1146,10 @@ fn equal_relays_pull_one_copy_owned_by_the_lower_hash() { /// but one of those links while every link of the ring is held, so each relay acts on /// what it last heard. Delivered in any order, once quiet every reader is fed through /// the one link left, and the pulls formed a cycle only while a change was still -/// crossing a link. +/// crossing a link. A relay that lost its link may take a neighbour's direct route +/// that the neighbour's own close made stale, until that close's DEAD crosses the +/// link between them: the transient loop the routes epic's Plan item 27 accepts. The +/// budget pins the seeded total, so a change in the rate shows here. #[test] fn a_ring_of_stale_offers_settles_on_the_link_left() { const RING: usize = 5; @@ -1192,7 +1195,7 @@ fn a_ring_of_stale_offers_settles_on_the_link_left() { assert_eq!(mesh.parent(relays[0], &x), Some(o), "seed {seed}"); loops += mesh.loops; } - assert!(loops <= 40, "{loops} steps with a cycle across 20 seeds"); + assert!(loops <= 204, "{loops} steps with a cycle across 20 seeds"); } /// `b` withdraws its copy while `c`'s subscription to it is on its way. The diff --git a/rs/moq-net/src/route/tests/compare.rs b/rs/moq-net/src/route/tests/compare.rs index 9e9e777147d..85daa23d486 100644 --- a/rs/moq-net/src/route/tests/compare.rs +++ b/rs/moq-net/src/route/tests/compare.rs @@ -1,5 +1,6 @@ //! One script on [`path::Router`], which keeps its doubt until the timer fires. The -//! ignored benchmarks also compare the rejected early-collection policy. +//! ignored benchmarks also compare the rejected early-collection policy, and the +//! replaced rule where doubt kept relays off every route to a doubted publisher. //! //! A [`Lab`] builds the same mesh on each, applies one change at a time, and //! delivers in rounds: every message queued when a round starts crosses its link @@ -16,7 +17,7 @@ use super::*; const RULES: [Rule; 1] = [Rule::PATH]; /// The rules compared, in the report's order. -pub(super) const COMPARED: [Rule; 2] = [Rule::EARLY, Rule::PATH]; +pub(super) const COMPARED: [Rule; 3] = [Rule::EARLY, Rule::ORIGIN, Rule::PATH]; /// A queue of messages from one node to another. pub(super) type Chan = (Node, Node); @@ -559,6 +560,14 @@ impl Lab { } self.net.reset(relay); self.revive(relay); + // Unlike after a crash, the links the mesh lacks reconnect too, such as a + // publisher's second home. + let linked: Vec<(Node, Node)> = (self.linked.iter().copied()) + .filter(|&(a, b)| (a == relay || b == relay) && !self.is_up(a, b)) + .collect(); + for (a, b) in linked { + self.mend(a, b); + } } /// Deliver one message from a queue `pick` chooses. False if none waits. @@ -763,8 +772,9 @@ pub(super) struct Churn { /// random meshes of three to five relays where links flap, the publisher's most of /// all, relays restart, and every message is delivered in a random order. Each rule /// sees the same meshes and the same changes at the same steps. Once the churn -/// stops the mesh must settle and be right. -pub(super) fn churn(rule: Rule, cases: std::ops::Range) -> Churn { +/// stops the mesh must settle and be right. With `two_homes` the publisher also +/// joins a second relay, and its flaps pick either link. +pub(super) fn churn(rule: Rule, cases: std::ops::Range, two_homes: bool) -> Churn { let mut total = Churn::default(); for case in cases { let mut script = XorShift(0x0bad_c0de + case as u64 * 0x9e37_79b9); @@ -786,11 +796,18 @@ pub(super) fn churn(rule: Rule, cases: std::ops::Range) -> Churn { }; let mut lab = Lab::new(rule, &mesh); let (node, _) = mesh.publisher(0); + let mut homes = vec![mesh.homes[0]]; + if two_homes { + homes.push((mesh.homes[0] + 1 + script.next(relays as u64 - 1) as usize) % relays); + lab.attach(node, homes[1]); + assert!(lab.settle()); + } let before = lab.net.sent(); for _ in 0..300 { match script.next(40) { 0 | 1 => { let (a, b) = match script.chance(60) { + true if two_homes => (node, homes[script.next(2) as usize]), true => (node, mesh.homes[0]), false => { let (a, b, _) = mesh.links[script.next(mesh.links.len() as u64) as usize]; @@ -802,7 +819,11 @@ pub(super) fn churn(rule: Rule, cases: std::ops::Range) -> Churn { false => lab.mend(a, b), } } - 2 if case % 4 == 0 && script.chance(25) => lab.restart(script.next(relays as u64) as usize), + 2 if case % 4 == 0 && script.chance(25) => { + let relay = script.next(relays as u64) as usize; + lab.restart(relay); + assert!(homes.iter().all(|&home| home != relay || lab.is_up(node, home))); + } _ => { lab.deliver(&mut pick); } @@ -838,7 +859,9 @@ pub(super) fn churn(rule: Rule, cases: std::ops::Range) -> Churn { #[test] fn every_rule_settles_after_churn() { for rule in RULES { - churn(rule, 0..60); + for two_homes in [false, true] { + churn(rule, 0..60, two_homes); + } } } @@ -852,7 +875,7 @@ fn benchmark_route_churn() { println!("rule,route_messages,claim_messages"); eprintln!("rule,steps,loops,stale,dark,msgs,items,claims,offline,unavailable"); for rule in COMPARED { - let churn = churn(rule, 0..400); + let churn = churn(rule, 0..400, false); let (rule, steps, sent) = (rule.as_str(), churn.steps, churn.sent); println!("{rule},{},{}", sent.msgs, sent.claims); eprintln!( @@ -862,6 +885,24 @@ fn benchmark_route_churn() { } } +/// The same churn with the publisher on two relays: what keeping its other link +/// usable while one flaps costs, per rule. +#[test] +#[ignore = "a benchmark; Nightly reports it"] +fn benchmark_route_churn_two_homes() { + println!("rule,homes,route_messages,claim_messages"); + eprintln!("rule,steps,loops,stale,dark,msgs,items,claims,offline,unavailable"); + for rule in COMPARED { + let churn = churn(rule, 0..400, true); + let (rule, steps, sent) = (rule.as_str(), churn.steps, churn.sent); + println!("{rule},2,{},{}", sent.msgs, sent.claims); + eprintln!( + "{rule},{steps},{},{},{},{},{},{},{},{}", + churn.loops, churn.stale, churn.dark, sent.msgs, sent.items, sent.claims, sent.offline, sent.unavailable, + ); + } +} + /// Each rule's router carries every change on every shape, and is right whenever /// it settles. #[test] diff --git a/rs/moq-net/src/route/tests/flap.rs b/rs/moq-net/src/route/tests/flap.rs new file mode 100644 index 00000000000..c123bd5231a --- /dev/null +++ b/rs/moq-net/src/route/tests/flap.rs @@ -0,0 +1,193 @@ +//! A publisher's own sessions closing: doubt keeps relays off routes through the +//! far end, but not off a route that begins there. + +use super::*; + +/// The neighbour `node` routes 7 through, following next hops until the +/// publisher: `Ok(true)` at it, `Ok(false)` stopped short, `Err` round a loop. +fn walk(sim: &Sim, node: Node) -> Result> { + let mut seen = vec![node]; + let mut at = node; + loop { + match sim.toward(at, 7) { + None => return Ok(false), + Some(None) => return Ok(true), + Some(Some(next)) if seen.contains(&next) => { + seen.push(next); + return Err(seen); + } + Some(Some(next)) => { + seen.push(next); + at = next; + } + } + } +} + +/// The publisher on relays 1 and 2, and relay 3 behind both, routing through 1. +fn dual_homed(rule: Rule) -> Sim { + let (origin, r1, r2, r3) = (0, 1, 2, 3); + let mut sim = Sim::with(rule, 4); + sim.leaf(origin); + sim.join(origin, r1, 1); + sim.join(origin, r2, 2); + sim.join(r1, r2, 1); + sim.join(r1, r3, 1); + sim.join(r2, r3, 1); + sim.originate(origin, 7); + sim.announce(origin, "live/x", 0, None); + assert!(sim.run()); + assert_eq!(sim.next_hop(r3, 7), Some(r1)); + sim +} + +/// The publisher's link to relay 1 closes and reopens every half doubt timer. +/// Every relay keeps a route that reaches the publisher through relay 2 the +/// whole time, and shows the claim available. +#[test] +fn a_flapping_link_keeps_a_dual_homed_publisher_routed() { + let (origin, r1, r2, r3) = (0, 1, 2, 3); + let half = Doubt::DEFAULT.timer / 2; + let mut sim = dual_homed(Rule::PATH); + for flap in 0..8 { + match flap % 2 { + 0 => sim.disconnect(origin, r1), + _ => sim.connect(origin, r1), + } + // Each step, before any message moves and after each round of them. + loop { + for relay in [r1, r2, r3] { + assert_eq!(walk(&sim, relay), Ok(true), "flap {flap}: relay {relay}"); + assert_eq!(sim.shows(relay, 7, "live/x"), Some(true), "flap {flap}: relay {relay}"); + } + if sim.busy().is_empty() { + break; + } + // One round; whether it went quiet is the loop's test. + sim.settle(1); + } + sim.wait(half); + } + assert!(sim.run()); + assert_eq!(sim.next_hop(r3, 7), Some(r1), "back on the cheaper link"); +} + +/// A link that comes back while its publisher is in doubt waits out the timer, so +/// a flapping link does not pull every relay back and forth with each flap; the +/// rejected rule that never doubts a route's origin takes it at once. +#[test] +fn a_link_that_comes_back_waits_out_the_doubt() { + let (origin, r1) = (0, 1); + for (rule, back) in [(Rule::PATH, false), (Rule::SPARED, true)] { + let mut sim = dual_homed(rule); + sim.disconnect(origin, r1); + assert!(sim.flow()); + sim.connect(origin, r1); + assert!(sim.flow()); + assert_eq!(sim.next_hop(r1, 7) == Some(origin), back, "{}", rule.as_str()); + assert!(sim.run()); + assert_eq!(sim.next_hop(r1, 7), Some(origin), "{}: after the timer", rule.as_str()); + } +} + +/// Under the replaced rule, the relays that used the closed link have no route +/// while the link flaps, though the publisher never left. +#[test] +fn a_flapping_link_darkened_the_publisher_under_the_replaced_rule() { + let (origin, r1, r3) = (0, 1, 3); + let half = Doubt::DEFAULT.timer / 2; + let mut sim = dual_homed(Rule::ORIGIN); + for flap in 0..8 { + match flap % 2 { + 0 => sim.disconnect(origin, r1), + _ => sim.connect(origin, r1), + } + assert!(sim.flow()); + assert_eq!(walk(&sim, r3), Ok(false), "flap {flap}"); + sim.wait(half); + } +} + +/// The publisher on each of three relays in a ring that is cheap one way round, +/// whose sessions all close before any relay hears another's news. +fn ring_of_homes(rule: Rule) -> Sim { + let (origin, r1, r2, r3) = (0, 1, 2, 3); + let mut sim = Sim::with(rule, 4); + sim.leaf(origin); + for relay in [r1, r2, r3] { + sim.join(origin, relay, 1); + } + for (a, b) in [(r1, r2), (r2, r3), (r3, r1)] { + sim.link(a, b, SimLink::cost(1), SimLink::cost(3)); + } + sim.originate(origin, 7); + sim.announce(origin, "live/x", 0, None); + assert!(sim.run()); + for relay in [r1, r2, r3] { + assert_eq!(sim.toward(relay, 7), Some(Some(origin)), "relay {relay} direct"); + } + sim +} + +/// The publisher leaves all three relays at once. Under the path rule each moves +/// onto its cheaper neighbour's direct route, which that neighbour's own close +/// made stale, so requests go round the ring until the news arrives: the cost of +/// keeping a publisher's other links usable. The replaced rule held every relay +/// instead. Either way every relay lets go once the news has spread. +#[test] +fn a_publisher_leaving_every_relay_at_once_loops_until_the_news_arrives() { + let (origin, r1, r2, r3) = (0, 1, 2, 3); + for (rule, loops) in [(Rule::ORIGIN, 0), (Rule::SPARED, 3), (Rule::PATH, 3)] { + let mut sim = ring_of_homes(rule); + for relay in [r1, r2, r3] { + sim.disconnect(origin, relay); + } + let looped = [r1, r2, r3] + .into_iter() + .filter(|&relay| walk(&sim, relay).is_err()) + .count(); + assert_eq!(looped, loops, "{}", rule.as_str()); + // One hop of delay: each relay hears its neighbours' news. + sim.settle(1); + for relay in [r1, r2, r3] { + assert!( + walk(&sim, relay).is_ok(), + "{}: relay {relay} after one hop", + rule.as_str() + ); + } + assert!(sim.run()); + for relay in [r1, r2, r3] { + assert_eq!(sim.toward(relay, 7), None, "{}: relay {relay}", rule.as_str()); + } + } +} + +/// A relay doubting the publisher still moves at once onto its own link to it, +/// opened before the doubt began, even as the last session it opened; only the +/// replaced rule holds it until the timer. +#[test] +fn a_link_opened_before_the_doubt_is_taken_at_once() { + let (origin, r1, r2) = (0, 1, 2); + for (rule, moves) in [(Rule::PATH, true), (Rule::ORIGIN, false)] { + let mut sim = Sim::with(rule, 3); + sim.leaf(origin); + sim.join(origin, r1, 1); + sim.join(r1, r2, 1); + sim.join(origin, r2, 5); + sim.originate(origin, 7); + sim.announce(origin, "live/x", 0, None); + assert!(sim.run()); + assert_eq!(sim.toward(r2, 7), Some(Some(r1)), "{}", rule.as_str()); + sim.disconnect(origin, r1); + assert!(sim.flow()); + assert_eq!(sim.toward(r2, 7) == Some(Some(origin)), moves, "{}", rule.as_str()); + assert!(sim.run()); + assert_eq!( + sim.toward(r2, 7), + Some(Some(origin)), + "{}: after the timer", + rule.as_str() + ); + } +} diff --git a/rs/moq-net/src/route/tests/handoff.rs b/rs/moq-net/src/route/tests/handoff.rs index e2740830f0a..51261f840dd 100644 --- a/rs/moq-net/src/route/tests/handoff.rs +++ b/rs/moq-net/src/route/tests/handoff.rs @@ -1646,9 +1646,10 @@ fn goaway_moves_a_relay_to_a_path_that_does_not_drain() { /// joined to `hub`, which reaches the publisher. A GOAWAY from `leaving` reaches /// the hub's side before any ring relay hears the others re-rank. Ranking drain /// alone, each moved onto the next relay's old, undrained offer, and a request went -/// round the ring with no session closed. The drain news casts doubt on the leaving -/// node, so no relay moves onto a stale offer through it, and every request reaches -/// the publisher after each delivery and timer. +/// round the ring with no session closed. When the hub leaves, the drain news casts +/// doubt on it, so no relay moves onto a stale offer through it. When the publisher +/// leaves, every path names it and ranks as draining alike, so no relay moves. Either +/// way every request reaches the publisher after each delivery and timer. fn goaway_around_a_ring(sim: &mut Sim, hub: Node, leaving: Node) { let (origin, ring) = (0, [1, 2, 3]); for relay in ring { diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index f52b824c8ea..e0d23679375 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -10,6 +10,7 @@ mod cache; mod compare; +mod flap; mod handoff; mod interest; mod residual; @@ -26,7 +27,7 @@ use std::{ use kio::clock::Instant; -pub(super) use super::path::{Doubt, Lift}; +pub(super) use super::path::{AtOrigin, Doubt, Lift}; use super::*; pub(super) type Node = usize; @@ -46,6 +47,18 @@ impl Rule { /// The comparison rule lifting doubt once no heard route names the node. pub const EARLY: Self = Self::Path(Doubt { lift: Lift::Unnamed, + origin: AtOrigin::Blocked, + ..Doubt::DEFAULT + }); + /// The replaced rule, where doubt also blocks every route to a doubted publisher. + pub const ORIGIN: Self = Self::Path(Doubt { + origin: AtOrigin::Blocked, + ..Doubt::DEFAULT + }); + /// The rejected rule, where doubt never blocks a route to a doubted publisher, even + /// on a session to it that opened during the doubt. + pub const SPARED: Self = Self::Path(Doubt { + origin: AtOrigin::Spared, ..Doubt::DEFAULT }); @@ -54,7 +67,17 @@ impl Rule { Self::Path(Doubt { lift: Lift::Unnamed, .. }) => "path+early", - Self::Path(Doubt { lift: Lift::Timer, .. }) => "path", + Self::Path(Doubt { + origin: AtOrigin::Blocked, + .. + }) => "path+origin", + Self::Path(Doubt { + origin: AtOrigin::Spared, + .. + }) => "path+spared", + Self::Path(Doubt { + origin: AtOrigin::Own, .. + }) => "path", } } } diff --git a/rs/moq-net/src/route/tests/routing.rs b/rs/moq-net/src/route/tests/routing.rs index a2f3b1ac7fa..38291824f2b 100644 --- a/rs/moq-net/src/route/tests/routing.rs +++ b/rs/moq-net/src/route/tests/routing.rs @@ -809,3 +809,34 @@ fn a_route_change_alone_names_its_identity() { router.route(up, 7, p(""), Some(path::Reach::Finite(along(&hops, 2)))); assert_eq!(router.changed(), BTreeMap::from([(7, BTreeSet::new())])); } + +/// A doubt about the publisher begins while this node has a session to it. That session +/// then files a second stream, as a new permit does, and the first retires. The second +/// stream is no returning link, so the publisher's own route stays usable on it. +#[test] +fn a_later_stream_of_a_session_the_doubt_predates_is_not_doubted() { + let (publisher, first, other, later) = (7, 1, 2, 3); + let session = |tag| Link { + session: tag, + ..Link::default() + }; + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + for (key, tag) in [(first, 5), (other, 6)] { + router.connect(key, publisher, session(tag)).unwrap(); + router.edited(key); + router.route(key, publisher, p(""), offer([publisher], 1)); + router.batch(key, publisher, p(""), vec![(p("live/x"), Some(1))], Some(1)); + } + // The other session to the publisher closes, casting doubt on it. + router.disconnect([other], Instant::ZERO); + router.connect(later, publisher, session(5)).unwrap(); + router.edited(later); + router.route(later, publisher, p(""), offer([publisher], 1)); + router.batch(later, publisher, p(""), vec![(p("live/x"), Some(1))], Some(1)); + router.retire(first); + assert_eq!( + router.route_for(publisher, &p("")).and_then(path::Route::next), + Some(later) + ); + assert!(router.shown()[&(publisher, p("live/x"))].available); +} diff --git a/rs/moq-net/tests/integration/catalog_handover.rs b/rs/moq-net/tests/integration/catalog_handover.rs index 2587a5ac480..c4163b182cc 100644 --- a/rs/moq-net/tests/integration/catalog_handover.rs +++ b/rs/moq-net/tests/integration/catalog_handover.rs @@ -154,12 +154,9 @@ impl Topology { async fn trigger(&mut self, trigger: Trigger) { match trigger { Trigger::Disconnect => abort(self.a_to_r.take().unwrap()), - Trigger::Unannounce => { - abort(self.p_to_a.take().unwrap()); - // `P` is the far end of `A`'s closed session, and `B`'s route crosses it too: - // on lite-07 `R` holds the claim until the doubt lifts, then moves to `B`. - kio::clock::sleep(DOUBT).await; - } + // `P` is the far end of `A`'s closed session, but `B`'s route begins at `P`, so + // doubt about `P` does not keep `R` off it. + Trigger::Unannounce => abort(self.p_to_a.take().unwrap()), Trigger::BetterRoute => { let direct = link(self.version, &self.publisher, &self.subscriber, None).await; self.links.push(direct); diff --git a/rs/moq-net/tests/integration/path_routes.rs b/rs/moq-net/tests/integration/path_routes.rs index a5db1c75806..fbb3bc2720a 100644 --- a/rs/moq-net/tests/integration/path_routes.rs +++ b/rs/moq-net/tests/integration/path_routes.rs @@ -9,8 +9,9 @@ //! unavailable, until the neighbours that learned the route from it answer, or for //! [`HOLD`] at most. -use crate::route_change::{link, produce_origin}; +use crate::route_change::{link, next, payload, produce_origin, read, timed}; use crate::support; +use support::{AppendNext, AtStart}; use std::time::Duration; @@ -168,6 +169,48 @@ async fn a_detour_through_the_far_end_waits_out_the_doubt() { assert!(usable.at >= held.at + DOUBT, "not before the doubt lifts: {seen:?}"); } +/// `P` is on `A` and `B`, which peer with each other, and `W` is behind both. `P`'s session +/// to `A` closes and reopens every half doubt. Doubt about `P` keeps no relay off `B`'s +/// route, which begins at `P`, so `W` shows the claim available throughout and its +/// subscriber reads each group at once. The reopened session waits out the doubt. +#[kio::test] +async fn a_flapping_link_keeps_a_dual_homed_publisher_routed() { + let version = lite07(); + let [p, a, b, w] = [1, 2, 3, 4].map(produce_origin); + let live = publish(&p, "live"); + let track = live.broadcast().create_track("video", timed()).unwrap(); + let mut p_a = peer(version, &p, &a).await; + let _p_b = peer(version, &p, &b).await; + let _a_b = peer(version, &a, &b).await; + let _a_w = peer(version, &a, &w).await; + let _b_w = peer(version, &b, &w).await; + quiesce().await; + + let mut watch = record(&w, "live").await; + let broadcast = w.consume().request("live", None).await.unwrap(); + let mut rx = read(broadcast.track("video").unwrap().subscribe(None).await.unwrap()); + let start = kio::clock::now(); + for flap in 0..8 { + match flap % 2 { + 0 => cut(&p_a), + _ => p_a = peer(version, &p, &a).await, + } + let sent = kio::clock::now(); + let mut group = track.append_next().unwrap(); + group.write_frame(group.at_start(), payload(flap, 0)).unwrap(); + group.close().unwrap(); + assert_eq!(next(&mut rx).await, (flap, payload(flap, 0)), "flap {flap}"); + let late = kio::clock::now().saturating_duration_since(sent); + assert!(late < DOUBT / 2, "flap {flap}: read {late:?} after it was written"); + kio::clock::sleep(DOUBT / 2).await; + } + let seen = watch.seen(start); + assert!( + seen.iter().all(|seen| seen.available && !withdrawn(seen)), + "available throughout: {seen:?}" + ); +} + /// `X` is `P`'s only way out and neighbours `R1`, `R2` and `R3`, which mesh with each other. /// `X` dies: each relay hears its own session close and its neighbours' DEADs, and none /// moves onto another's route through `X` on the way. Each withdraws the claim once. diff --git a/rs/moq-net/tests/integration/route_change.rs b/rs/moq-net/tests/integration/route_change.rs index f2ee1a5fb73..fa67d9da085 100644 --- a/rs/moq-net/tests/integration/route_change.rs +++ b/rs/moq-net/tests/integration/route_change.rs @@ -230,7 +230,7 @@ pub(super) fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver) -> (u64, Vec) { +pub(super) async fn next(rx: &mut mpsc::UnboundedReceiver) -> (u64, Vec) { let (group, _, payload) = next_timed(rx).await; (group, payload) } @@ -563,41 +563,6 @@ async fn better_route_restarts(version: &str) { ); } -/// A route through the far end of a closed link waits out the doubt. `A` loses its -/// session to `P`, the far end, and `B`'s route names `P` too: `R` cannot tell `P` from a -/// dead relay whose news is still on its way, so it holds the claim and moves to `B` only -/// once the doubt lifts. The subscription through `A` ends meanwhile; a new one reads on -/// through `B`. -async fn waits_out_the_doubt(version: &str) { - let version: Version = version.parse().unwrap(); - let (mut topology, sub) = Topology::new(version).await; - let mut rx = read(sub); - topology.standby().await; - - let mut group = topology.track.append_next().unwrap(); - group.write_frame(group.at_start(), payload(0, 0)).unwrap(); - group.close().unwrap(); - assert_eq!(next(&mut rx).await, (0, payload(0, 0))); - - topology.trigger(Trigger::Unannounce).await; - kio::clock::sleep(Duration::from_secs(1)).await; - let mut group = topology.track.append_next().unwrap(); - group.write_frame(group.at_start(), payload(1, 0)).unwrap(); - group.close().unwrap(); - let remote = topology.subscriber.consume().request("live", None).await.unwrap(); - let mut fresh = read(remote.track("video").unwrap().subscribe(None).await.unwrap()); - assert_eq!(next(&mut fresh).await, (1, payload(1, 0)), "{version}: through B"); - match kio::clock::timeout(Duration::from_secs(10), rx.next()) - .await - .expect("the subscription through A hung") - { - Some((_, Err(_))) => {} - None => panic!("{version}: the subscription through A ended cleanly"), - Some((group, Ok(_))) => panic!("{version}: the subscription through A read group {group}"), - } - assert!(rx.try_recv().is_err(), "{version}: a delivery after the error"); -} - macro_rules! route_change_tests { ($($name:ident: $version:literal,)*) => { $( @@ -634,11 +599,16 @@ macro_rules! route_change_tests { run(Trigger::Disconnect, Position::MidGroup).await; } + /// `B`'s route begins at `P`, the far end of the closed link, so doubt about + /// `P` does not keep `R` off it: the subscription moves at once. #[kio::test] - async fn unannounce_waits_out_the_doubt() { - kio::clock::timeout(TEST_TIMEOUT, waits_out_the_doubt($version)) - .await - .expect("timed out"); + async fn unannounce_between_groups() { + run(Trigger::Unannounce, Position::BetweenGroups).await; + } + + #[kio::test] + async fn unannounce_mid_group() { + run(Trigger::Unannounce, Position::MidGroup).await; } #[kio::test] diff --git a/rs/moq-net/verify/route/DrainRing.cfg b/rs/moq-net/verify/route/DrainRing.cfg index 3f1af4ca367..d405f8b1f1d 100644 --- a/rs/moq-net/verify/route/DrainRing.cfg +++ b/rs/moq-net/verify/route/DrainRing.cfg @@ -4,8 +4,8 @@ \* publisher's GOAWAY may reach every relay before any hears the others re-rank. \* Ranking drain alone, each moved onto the next relay's old, undrained offer, \* and a request went round the ring with no session closed, breaking -\* RequestsLoopFree and LoopsAreStale. The DRAIN news casts doubt on the -\* publisher, so no relay takes a stale offer through it. +\* RequestsLoopFree and LoopsAreStale. Every path names the publisher, so +\* each ranks as draining alike and no relay moves off its route. \* The Rust fixture is a_goaway_in_a_ring_never_loops_a_request. CONSTANTS Nodes = {0, 1, 2, 3} diff --git a/rs/moq-net/verify/route/Flap.cfg b/rs/moq-net/verify/route/Flap.cfg new file mode 100644 index 00000000000..5c80b6cbd28 --- /dev/null +++ b/rs/moq-net/verify/route/Flap.cfg @@ -0,0 +1,41 @@ +\* The publisher (0) on relays 1 and 2, joined to each other, and relay 3 behind +\* both. Only the publisher's link to relay 1 opens and closes, so the only doubt is +\* about the publisher: its link to relay 1 flaps while relay 2's stays up, and +\* every relay that knows the way through relay 2 keeps a route the whole time. +\* Simulate: -simulate num=1000 -depth 150 -seed 1 +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {1, 2}, {1, 3}, {2, 3}} + Dear = {} + Ring = {} + Preferred = {} + Upstream = {} + Leaving = {} + Scoped = {} + Editors = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 4 + MaxEdits = 0 + Patient = TRUE + Flaky <- FirstLink + +SPECIFICATION Spec + +INVARIANTS + Truthful + Mirrored + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + Routed + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Homes.cfg b/rs/moq-net/verify/route/Homes.cfg new file mode 100644 index 00000000000..14bbd7820fc --- /dev/null +++ b/rs/moq-net/verify/route/Homes.cfg @@ -0,0 +1,41 @@ +\* The publisher (0) on each of three relays in a ring that is cheap one way +\* round, leaving all three. Each relay may move onto its cheaper neighbour's +\* direct route before that neighbour's news arrives, so a request can go round +\* the ring (`RequestsLoopFree` fails): only `LoopsLostThePublisher` holds. +\* Simulate: -simulate num=1000 -depth 150 -seed 1 +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {0, 3}, {1, 2}, {2, 3}, {1, 3}} + Dear = {} + Ring = {1, 2, 3} + Preferred = {} + Upstream = {} + Leaving = {} + Scoped = {} + Editors = {} + Claims = {"x"} + Narrow = {} + MaxRev = 0 + MaxFlaps = 3 + MaxEdits = 0 + Patient = TRUE + Flaky <- OwnLinks + +SPECIFICATION Spec + +INVARIANTS + Truthful + Mirrored + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + LoopsLostThePublisher + Routed + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Loop.cfg b/rs/moq-net/verify/route/Loop.cfg index cb660433b81..ef55ee3e829 100644 --- a/rs/moq-net/verify/route/Loop.cfg +++ b/rs/moq-net/verify/route/Loop.cfg @@ -34,6 +34,7 @@ INVARIANTS LoopsAreStale StaleIsChased RequestsLoopFree + Routed QuietIsSettled Budget NoStaleReturn diff --git a/rs/moq-net/verify/route/Ring.cfg b/rs/moq-net/verify/route/Ring.cfg index 682095cfed3..0493962427b 100644 --- a/rs/moq-net/verify/route/Ring.cfg +++ b/rs/moq-net/verify/route/Ring.cfg @@ -34,5 +34,6 @@ INVARIANTS LoopsAreStale StaleIsChased RequestsLoopFree + Routed QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/RouteHandoff.cfg b/rs/moq-net/verify/route/RouteHandoff.cfg index 9b96f8f9d4b..9a4cb8ee296 100644 --- a/rs/moq-net/verify/route/RouteHandoff.cfg +++ b/rs/moq-net/verify/route/RouteHandoff.cfg @@ -34,6 +34,7 @@ INVARIANTS LoopsAreStale StaleIsChased RequestsLoopFree + Routed NoStaleReturn QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index 443bfe54d07..f55ddf9ab4b 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -28,8 +28,11 @@ (* - A relay with no route left holds its claims, unavailable, until each *) (* neighbour that learned the route from it has answered. *) (* - A closed session casts doubt on the node at its far end: no relay *) -(* moves to a route through it until a timer lifts the doubt. The timer *) -(* is the step `Lift`, which may fire at any moment. *) +(* moves to a route through it until a timer lifts the doubt. A route *) +(* that begins at it is exempt, since it cannot loop through its own *) +(* end, except over a session to it opened since the doubt began *) +(* (`Returned`): a link that comes back waits out the timer. The timer *) +(* is the step `Lift`, which may fire at any moment. *) (* *) (* Each node's every entry point is one atomic step ending in `Update`, *) (* as in the router. Messages travel on one FIFO per direction. A *) @@ -59,6 +62,12 @@ CONSTANTS Patient \* TRUE: a timer fires only once no news or stale offer about its node is in flight Relays == Nodes \ {Origin} +\* The links whose sessions open and close. A config may narrow them to the +\* publisher's own (`OwnLinks`) or to one of them (`FirstLink`). +Flaky == Links +OwnLinks == {l \in Links : Origin \in l} +\* The publisher's link to the lowest relay. +FirstLink == {{Origin, CHOOSE r \in Relays : \A x \in Relays : r <= x}} Cost == [l \in Links |-> IF l \in Dear THEN 2 ELSE 1] \* The relay after n round `Ring`, the lowest after the highest. @@ -186,7 +195,21 @@ Live(r) == r.on /\ ~r.held ----------------------------------------------------------------------------- (* Selection on node n's state s (path.rs: `select`, `pending`, `hopes`). *) -Doubted(s, path) == \E i \in DOMAIN path : path[i][1] \in s.doubt +\* The path crosses a relay in doubt. Its first node, where the route begins, +\* does not count: a route cannot loop through its own end. +RelayDoubted(s, path) == \E i \in DOMAIN path : i > 1 /\ path[i][1] \in s.doubt +\* The generation of n's session to the publisher, or None without a link. +OwnGen(n) == IF {n, Origin} \in Links THEN gen[{n, Origin}] ELSE None +\* The doubted publisher's own route, on a session to it that opened since the +\* doubt began: a link that comes back waits out the timer, damping its flaps. +Returned(n, s, m, path) == + /\ Len(path) = 1 + /\ Origin \in s.doubt + /\ m = Origin + /\ OwnGen(n) > s.since +\* Doubt keeps a node off a route through a doubted relay, and off a returned +\* link, but not off another route that begins at the doubted publisher. +Doubted(n, s, m, path) == RelayDoubted(s, path) \/ Returned(n, s, m, path) \* The path crosses a node this node knows sent GOAWAY. Drains(s, path) == \E i \in DOMAIN path : path[i][1] \in s.draining @@ -232,7 +255,7 @@ Pending(n, s, k) == /\ s.tx[p][j].src = k \* A neighbour offers a route that only doubt keeps this node off. -Hopes(n, s, k) == \E m \in Nodes : Usable(n, s, m, k) /\ Doubted(s, s.rx[m][k].path) +Hopes(n, s, k) == \E m \in Nodes : Usable(n, s, m, k) /\ Doubted(n, s, m, s.rx[m][k].path) \* `select` for one key: the best-ranked usable route, staying on the next hop \* among equals, then the lowest key. Doubt keeps a node from moving to a route, @@ -241,7 +264,7 @@ SelectKey(n, s, k, gens) == LET was == s.rt[k] current == IF was.on THEN was.next ELSE None using == IF Live(was) THEN was.next ELSE None - C == {m \in Nodes : Usable(n, s, m, k) /\ (m = using \/ ~Doubted(s, s.rx[m][k].path))} + C == {m \in Nodes : Usable(n, s, m, k) /\ (m = using \/ ~Doubted(n, s, m, s.rx[m][k].path))} Path(m) == Append(s.rx[m][k].path, <>) Drain(m) == s.rx[m][k].drain \/ m \in s.drain \/ Drains(s, s.rx[m][k].path) \* Sticky: learned on an upstream link, or from a down-only route. @@ -562,6 +585,7 @@ InitNode(n) == [ view |-> [k \in Keys |-> NoView], dead |-> {}, doubt |-> {}, + since |-> None, \* the generation of the session to the publisher when its doubt began wait |-> {}, expiry |-> {}, \* Pending timers belong to individual dead hops. drain |-> {}, \* the neighbours that sent GOAWAY on the session open to them @@ -618,6 +642,10 @@ Unannounce(c) == rev < MaxRev /\ own[c] # None /\ ChangeClaim(c, None) ----------------------------------------------------------------------------- (* A message arrives (path.rs: `recv`, `fail`). *) +\* `suspect`: a doubt on the publisher that begins notes n's session to it, so +\* one opened since waits. +Since(n, s, x) == IF x = Origin /\ Origin \notin s.doubt THEN OwnGen(n) ELSE s.since + \* `fail`: the session `hop` names closed. `from` reported it with what became \* of its own routes through it, or this node saw it close (`from` is None). \* It casts doubt on the node at the session's far end, `hop[2]`. @@ -638,13 +666,16 @@ Fail(n, s, hop, from, moved, gone) == ELSE IF h.state = "live" THEN [h EXCEPT !.state = "dead"] ELSE h]] IN [s EXCEPT !.rx = rx1, !.dead = @ \cup {hop}, !.expiry = IF hop \in s.dead THEN @ ELSE @ \cup {hop}, - !.doubt = @ \cup ({hop[2]} \ {n}), !.wait = @ \cup ({hop[2]} \ {n})] + !.doubt = @ \cup ({hop[2]} \ {n}), !.wait = @ \cup ({hop[2]} \ {n}), + !.since = Since(n, s, hop[2])] \* `drains`: x sent GOAWAY. The first word of it casts doubt on x, so no relay -\* takes a stale offer through it until the timer fires. +\* takes a stale offer through it until the timer fires. A route that begins at +\* x needs none: every path to x drains alike, so no relay moves. MarkDrain(n, s, x) == IF x \in s.draining \/ x = n THEN s - ELSE [s EXCEPT !.draining = @ \cup {x}, !.doubt = @ \cup {x}, !.wait = @ \cup {x}] + ELSE [s EXCEPT !.draining = @ \cup {x}, !.doubt = @ \cup {x}, !.wait = @ \cup {x}, + !.since = Since(n, s, x)] \* A DRAIN counts only for a node on a path `from` offered. Credited(s, from, x) == \E k \in Keys : s.rx[from][k].on /\ Names(s.rx[from][k].path, x) @@ -769,6 +800,7 @@ InFlight(P(_)) == \E d \in Dirs : \E i \in DOMAIN q[d] : \E h \in MsgHops(q[d][i \* offer sent before the news reached its sender. Lift(n, x) == LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ {x}, !.wait = @ \ {x}, + !.since = IF x = Origin THEN None ELSE @, !.expiry = {h \in @ : h[2] # x}], OV, up, gen, ended[n], NoReport, Claims)] IN /\ x \in st[n].wait @@ -830,7 +862,8 @@ Connect(l) == a == Asks(Closed(st[x], y), y) u == Update(x, a.s, OV, up1, gen1, ended[x], NoReport, {}) IN [u EXCEPT !.out[y] = a.out \o @]] - IN /\ l \notin up + IN /\ l \in Flaky + /\ l \notin up /\ flaps < MaxFlaps /\ up' = up1 /\ gen' = gen1 @@ -851,7 +884,8 @@ Disconnect(l) == THEN Update(x, s, OV, up1, gen, ended[x], NoReport, {}) ELSE Update(x, Fail(x, s, hop, None, {}, FALSE), OV, up1, gen, ended[x], DeadReport(hop), {})] - IN /\ l \in up + IN /\ l \in Flaky + /\ l \in up /\ flaps < MaxFlaps /\ up' = up1 /\ flaps' = flaps + 1 @@ -1042,6 +1076,32 @@ LoopsAreStale == \* A request never loops. It does while news is in flight: `Loop.cfg` pins it. RequestsLoopFree == \A n \in Relays, c \in Claims : Request(n, c) # "loop" +\* A request that loops goes round relays whose routes each begin on a closed +\* session of the publisher: it left every one of them, and their news is in +\* flight. Moving onto a route that begins at a doubted node allows that loop; +\* doubt about a relay still keeps every other one out. +LoopsLostThePublisher == + \A n \in Relays, c \in Claims : + Request(n, c) = "loop" => + LET w == Walk(n, IF c \in Narrow /\ Live(st[n].rt["a"]) THEN "a" ELSE "root", None, Cardinality(Nodes)) + \* The positions inside a stretch of the walk that comes back to its start. + round == {i \in DOMAIN w : \E j, k \in DOMAIN w : j <= i /\ i < k /\ w[j].node = w[k].node} + IN \A i \in round : Len(w[i].path) >= 2 /\ Gone(w[i].path[2]) + +\* A route a relay knows over sessions that are still open, crossing no relay +\* it doubts, and not a link that came back during the doubt. +Fresh(n, m, k) == + /\ Usable(n, st[n], m, k) + /\ ~Stale(st[n].rx[m][k].path) + /\ ~RelayDoubted(st[n], st[n].rx[m][k].path) + /\ ~Returned(n, st[n], m, st[n].rx[m][k].path) + +\* Doubt about the publisher alone never keeps a relay off its other links: one +\* that knows a fresh route uses some route, so a request stops only at a relay +\* that knows no way on. Which route it uses may still be stale while the news +\* that ends it is in flight, which `StaleIsChased` bounds. +Routed == \A n \in Relays, k \in Keys : (\E m \in Nodes : Fresh(n, m, k)) => Live(st[n].rt[k]) + \* No node may use a route through a hop it once learned had died: after a \* dead hop is forgotten, no stale route returns. It holds only for one failure \* and a patient timer; `StaleIsChased` bounds the rest. diff --git a/rs/moq-net/verify/route/Twice.cfg b/rs/moq-net/verify/route/Twice.cfg index 6ed75be9bdd..71ec5379ed1 100644 --- a/rs/moq-net/verify/route/Twice.cfg +++ b/rs/moq-net/verify/route/Twice.cfg @@ -35,6 +35,7 @@ INVARIANTS LoopsAreStale StaleIsChased RequestsLoopFree + Routed QuietIsSettled Budget NoStaleReturn diff --git a/rs/moq-net/verify/route/TwoHomes.cfg b/rs/moq-net/verify/route/TwoHomes.cfg new file mode 100644 index 00000000000..9e60c41bf2f --- /dev/null +++ b/rs/moq-net/verify/route/TwoHomes.cfg @@ -0,0 +1,43 @@ +\* Flap.cfg where both of the publisher's sessions open and close. The exemption +\* for a route beginning at the publisher lets a relay move onto an offer through +\* its other link while doubt about the publisher stands. Before a patient timer +\* waited for stale offers as well as DEAD news, a relay could forget a dead hop +\* while an offer naming it was still queued, take that offer, and drop a claim it +\* showed once the next message named the hop again, breaking `Stable`. +\* Simulate: -simulate num=1000 -depth 150 -seed 1 +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {1, 2}, {1, 3}, {2, 3}} + Dear = {} + Ring = {} + Preferred = {} + Upstream = {} + Leaving = {} + Scoped = {} + Editors = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 3 + MaxEdits = 0 + Patient = TRUE + Flaky <- OwnLinks + +SPECIFICATION Spec + +INVARIANTS + Truthful + Mirrored + NoRevival + NoFalseEnd + NoRegression + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + Routed + QuietIsSettled + Budget + Stable