diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 6d60ca610dc..c26aa3e6551 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -1,11 +1,11 @@ --- name: verify-route -description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteHandoff.tla`, the rule the routes epic's Plan item 27 picked, as `rs/moq-net/src/route/tests/paths.rs` prototypes it) with TLA+. Use when a change touches route paths, the DEAD message for a closed session, holds, the doubt and its timer, forgetting dead hops, routes per prefix and scoped streams, the newest-floor views, what a reader is shown, or where a request goes, or when a routing bug needs every message ordering checked rather than a random sweep. +description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteHandoff.tla`, the rule the routes epic's Plan item 27 picked, as `route::path::Router` in `rs/moq-net/src/route/path.rs` runs it) with TLA+. Use when a change touches route paths, the DEAD message for a closed session, holds, the doubt and its timer, forgetting dead hops, routes per prefix and scoped streams, the newest-floor views, what a reader is shown, or where a request goes, or when a routing bug needs every message ordering checked rather than a random sweep. --- # Verify the route rule -`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and a closed session casts doubt on its far end until a timer fires. It follows the prototype `Relay` in `rs/moq-net/src/route/tests/paths.rs` under `Rules::waits`, adds a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The timer is the step `Lift`, which may fire at any moment, or with `Patient` only once no DEAD about its node is in flight. `route::Router` still runs Babel's distance; [the router quest](/quest/m2/cluster-routes/scope.md) builds this rule. +`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. ## Rules @@ -25,7 +25,9 @@ Each rule is an INVARIANT; the harm says why it matters. | `NoStaleReturn` | No relay may use a route through a hop it once learned had died. | A dead path comes back after a relay forgot it. | | `QuietIsSettled` | Once nothing is in flight and every timer fired: each relay shows exactly the claims a path of open sessions carries to it, available, and a request for each reaches the publisher; no relay holds, keeps a stale route, or remembers a dead hop. | A hold never ends, a claim is shown that no path carries, or a relay behind a scoped neighbour never reaches what a farther one offers. | | `Budget` | From one quiet point to the next, the messages sent stay within 4 each way over each configured session per prefix and per claim, for each external change. | One change sets off announce spam, or routing never goes quiet. | -| `RequestsLoopFree` | A request never loops. | Only `Loop.cfg` checks it, as an open finding. | +| `RequestsLoopFree` | A request never loops. | A request cycles through relays instead of reaching the publisher. | + +Every config checks `RequestsLoopFree`; `Loop.cfg` pins the three-failure regression. No config checks a liveness property: `Spec` has no fairness, so TLC does not show that a hold ends. That one does is an argument over what TLC does check. `Budget` bounds the messages a burst sends, each delivery takes one message off a queue, and a doubt starts only at a close or a delivery, so a burst is a finite number of steps. A delivery is enabled while a queue holds a message and a timer may fire once no news about its node is in flight, so a burst left alone reaches the state with nothing in flight and no doubt. `QuietIsSettled` says no relay holds there. @@ -38,39 +40,75 @@ quest gate just rs tla moq-net/verify/route/RouteHandoff.tla quest gate just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Ring.cfg ``` -Run the configs the change touches, one at a time. Nightly runs every config (`just test tla rs/moq-net/verify/route`), simulating each one with a `Simulate:` line and expecting `Loop.cfg` to break. Times are on the recipe's 4 workers. `Patient` is TRUE unless the table says otherwise. +Choose the config affected by the change; do not run the directory locally. +Nightly runs every config through `just test tla`, simulating each with a +`Simulate:` line. The recipe uses four workers. `Loop.cfg` checks the timer-only +doubt regression. When calling `just rs tla` +directly, pass the arguments from that line explicitly. For covering-source +selection, for example: + +```sh +quest gate just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Overlap.cfg -simulate num=1000 -depth 500 +``` -| Config | Mesh | Bounds | Distinct states | Run | +The older exhaustive counts and times below predate independent hop expiry +(`Expire`); they are historical baselines, not validation of the current model. +The Loop row gives its current simulation parameters. Times used four workers. +`Patient` is TRUE unless the table says otherwise. + +`Loop.cfg` used to stop at the first early-collection counterexample. With +timer-only doubt, the attempted exhaustive check was stopped by the user after +76 minutes: 32.1M distinct states, 19.3M still queued at depth 14, and a growing +queue. It did not complete. Routine checks now use its bounded `Simulate:` line, +as they do for the other large configs. This samples the invariants; it is not +an exhaustive proof. Do not rerun the full route spec exhaustively for this +change; faster checkers are separate work. + +| Config | Mesh | Bounds | Previous distinct states | Previous run | | --- | --- | --- | --- | --- | -| `RouteHandoff.cfg` | The publisher behind relay 1, in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 407,240 | a minute or two | -| `Early.cfg` | The same, with a timer that fires at any moment. | 1 claim change, 1 session change | 4,178,590 | ten minutes | +| `RouteHandoff.cfg` | The publisher behind relay 1, in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 476,515 | a minute or two | +| `Early.cfg` | The same, with a timer that fires at any moment. | 1 claim change, 1 session change | 14,220,298 | half an hour | | `Twice.cfg` | The same, where two sessions close, as when relay 1 dies. Simulated: `num` counts behaviours per worker. | 1 claim change, 2 session changes | 40,000 behaviours, 4.0M states | a few minutes | -| `Ring.cfg` | The publisher and three relays in a ring, with a timer that fires at any moment. | 1 claim change, 1 session change | 426,409 | a minute or two | -| `Lag.cfg` | Relay 3 hears the publisher through relays 1 and 2: the lagging neighbour. | 2 claim changes, 1 session change | 1,299,405 | a few minutes | -| `LagWide.cfg` | The same, with relay 1 asking for `"a"` alone. | 2 claim changes, 1 session change | 4,194,747 | ten minutes | +| `Ring.cfg` | The publisher and three relays in a ring, with a timer that fires at any moment. | 1 claim change, 1 session change | 890,827 | a minute or two | +| `Lag.cfg` | Relay 3 hears the publisher through relays 1 and 2: the lagging neighbour. | 2 claim changes, 1 session change | 1,926,671 | a few minutes | +| `LagWide.cfg` | The same, with relay 1 asking for `"a"` alone. | 2 claim changes, 1 session change | 4,675,558 | a quarter of an hour | | `Dead.cfg` | Two relays behind a relay whose only uplink asks for `"a"` alone, one with a dear session of its own to the publisher. Simulated. | 3 claim changes, 1 session change | 40,000 behaviours, 4.8M states | ten minutes | | `Blind.cfg` | A relay whose nearest neighbour asks for `"a"` alone, and a farther one knows everything. Simulated. | 2 claim changes, 1 session change | 40,000 behaviours, 4.8M states | ten minutes | -| `Loop.cfg` | The publisher with a session to each of three relays in a triangle, all closing. Open finding: `RequestsLoopFree` breaks. | 3 session changes | 979,474 when it breaks | a few minutes | +| `Overlap.cfg` | Five relays, three asking for `"a"`, with overlapping cheap broad and dear narrow paths. Simulated past the message budget. | 1 claim change, no session change | 1,000 behaviours per worker (4,000 on four workers), depth 500 | simulation | +| `Loop.cfg` | The publisher with a session to each of three relays in a triangle, all closing. Samples `RequestsLoopFree` with timer-only doubt. | 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | simulation | +| `Drain.cfg` | The publisher joined to two relays joined to each other; both hear its GOAWAY. Ranking only a relay's own drain exceeded `Budget`. | 1 claim change, 1 session change, GOAWAY on each session | 648,218 | two minutes | +| `DrainRing.cfg` | The publisher joined to three relays in a ring that is cheap one way round (`Ring`); all hear its GOAWAY. Without the DRAIN news and its doubt, a request looped round the ring with no session closed, breaking `LoopsAreStale`. Simulated. | 1 claim change, GOAWAY on each session | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | +| `Prefer.cfg` | The same triangle, where both relays prefer the link between them. Ranking only a relay's own link priority exceeded `Budget`. | 1 claim change, 1 session change | 14,630 | seconds | +| `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, seed 1 | a few minutes | -Run them last, one at a time. First run what answers in seconds: the prototype's tests (`just rs test -p moq-net --lib -E 'test(/route::tests::(paths|compare|copies)/)'`), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. +Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. ## Reading a failure -Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the prototype with `PathSim` (`deliver`, `disconnect`, `tick`), one call per step, except a timer. `Lift(n, x)` fires one timer on one relay, where `tick` moves every running relay's clock by one hop of delay: call it until that doubt lifts, and where another doubt lifted with it, the replay took a different path, which is no defect in the model. +Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the router with `Sim::with(Rule::PATH, nodes)` (`deliver`, `disconnect`, `connect`, `announce`, `wait`), one call per step, except a timer. `Lift(n, x)` fires one timer on one relay, where `wait(HOP)` moves every relay's clock by one hop of delay: call it until that doubt lifts, and where another doubt lifted with it, the replay took a different path, which is no defect in the model. ## Coverage -Sync commit: `d3b6c2573a30541402d396f8080488646f673594`. Run `git log d3b6c2573a.. -- rs/moq-net/src/route/tests/paths.rs` to see what changed since. The model describes the prototype, extended with a route per prefix and with dead hops kept until their timer fires; `route::Router` does not run this rule yet. `Relay::collect` forgets a dead hop as soon as no route names it, so `NoStaleReturn` covers the rule the router is to run, not the prototype's. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `31b215b112fb649cc3f63d4de3c76b5e87769a06`, the base of the scoped path router in [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622). Run `git log 31b215b11.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | -| `rs/moq-net/src/route/tests/paths.rs` `Relay::select`, `Relay::pending`, `Relay::hopes`, `Relay::doubted` | `SelectKey`, `Pending`, `Hopes`, `Doubted` | -| `rs/moq-net/src/route/tests/paths.rs` `Relay::want`, `Relay::advertise`, `Relay::report` | `Want`, `Advertise`, `Report` | -| `rs/moq-net/src/route/tests/paths.rs` `Relay::recv`, `Relay::fail`, `Relay::disconnect`, `Relay::connect`, `Relay::change` | `Recv`, `Fail`, `Disconnect`, `Connect`, `ChangeClaim` | -| `rs/moq-net/src/route/tests/paths.rs` `Relay::collect`, `Relay::tick` | `Collect`, `Lift` | -| `rs/moq-net/src/route/tests/paths.rs` `Relay::show` | `Know`, `ShownNow`, `Events` | -| `rs/moq-net/src/route/tests/paths.rs` `PathSim::deliver` | `Deliver` | +| `rs/moq-net/src/route/path.rs` `Router::run`, `Router::select`, `Router::retain` | `Select` plans live routes and their reflected sources before `SelectKey` decides holds. | +| `rs/moq-net/src/route/path.rs` `Offer::rank`, `Router::goaway`, `Router::drains`, `Router::report_draining`, `Router::dead` (`Dead::Draining`) | `Rank`, `Below`, `PathPrio`, `CostAt`, `GoAway`, `MarkDrain`, `Credited`, `ReportDrain`, `Recv` | +| `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted` | +| `rs/moq-net/src/route/path.rs` `Router::src` | `SrcKey` | +| `rs/moq-net/src/route/path.rs` `Router::want`, `Router::advertise`, `Router::report` | `Want`, `Advertise`, `Report` | +| `rs/moq-net/src/route/path.rs` `Router::route`, `Router::batch`, `Router::dead`, `Router::fail`, `Router::disconnect`, `Router::connect`, `Router::change` | `Recv`, `Fail`, `Disconnect`, `Connect`, `ChangeClaim` | +| `rs/moq-net/src/route/path.rs` `Router::collect_doubt`, `Router::timeout` | `Collect`, `Lift`, `Expire`: each dead hop's own timer retires it independently of later failures of the same node; reporting an already retained hop again does not restart its expiry. | +| `rs/moq-net/src/route/path.rs` `Router::know`, `Router::shown_now`, `Router::show` | `Know`, `ShownNow` | +| `rs/moq-net/src/route/reader.rs` `Reader::apply` | `Events` | +| `rs/moq-net/src/route/tests/mod.rs` `Sim::deliver` | `Deliver` | ## Not covered -One publisher, so routes for several publishers, their copies (`Relay::warm`, `Relay::source`), and the copy rank are left out; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, and a scoped relay asks every neighbour for the narrow one: no wildcards, interest edits, grants, qualified keys, or hidden claims. Leaves out restarts, GOAWAY, priorities, costs other than 1 and 2, RTT, and the hop limit. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. A route is sent on every stream whether or not the publisher has a claim there. The publisher ignores what it hears. No exhaustive pass has more than four relays or two session changes; the prototype's sweeps in `compare.rs` cover larger meshes and churn. +The router's current watch contract supports one interest prefix per neighbour. +Concurrent disjoint regions of one permit need the +[interest set](/quest/m2/cluster-routes/interest-set.md), including per-prefix +SYNCED; neither this model nor the current scoped sweep covers them. + +One publisher, so routes for several publishers are left out, and so are the copies and copy rank of cache-aware routing, which `rs/moq-net/src/route/tests/paths.rs` prototypes; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, and a scoped relay asks every neighbour for the narrow one: no wildcards, interest edits, stream generations or cancellation, grants, qualified keys, or hidden claims. Leaves out restarts, priorities other than 0 and 1, a priority that differs between a link's two ends, costs other than 1 and 2, RTT, upstream/down-only links, and the hop limit. Messages come only from honest routers: validation of an unrelated DEAD or its far node, and empty or overlong replacement paths, is outside the model's input domain. Rust fixtures cover those guards and exportable covering-source selection, including keeping a covering hold when every live source is unexportable. Interest-edit fixtures also cover a cancelled stream retaining its pending child replies; the model has no stream cancellation. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. A route is sent on every stream whether or not the publisher has a claim there. The publisher ignores what it hears. `Overlap.cfg` also simulates five relays with every session up. No exhaustive pass has more than four relays or two session changes; `Sim`'s sweeps on the path rule (`sweep_handoff`, `sweep_scoped_handoff`, `sweep_scoped_meshes_show_what_the_rule_says`, and `compare.rs`'s churn) cover larger meshes, grants, qualified keys, interest edits, restarts, and churn. diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 6c0bd3af103..e2ac69a1828 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -434,16 +434,18 @@ jobs: strategy: fail-fast: false matrix: - # The largest models get a runner each, and the route model's two - # largest configs (about 30 and 15 minutes) share one; the `route` - # and `rest` legs take every other config, so a new one is checked - # without editing this list. sh/gh/nightly.test.py fails on a path - # that no longer exists. + # The largest models get a runner each, as do the route model's two + # largest configs (Early about 14M distinct states, LagWide about + # 4.7M); the `route` and `rest` legs take every other config, so a + # new one is checked without editing this list. sh/gh/nightly.test.py + # fails on a path that no longer exists. include: - name: resume paths: rs/moq-net/verify/resume - - name: route large - paths: rs/moq-net/verify/route/Early.cfg rs/moq-net/verify/route/LagWide.cfg + - name: route early + paths: rs/moq-net/verify/route/Early.cfg + - name: route lagwide + paths: rs/moq-net/verify/route/LagWide.cfg - name: route paths: "rs/moq-net/verify/route :!rs/moq-net/verify/route/Early.cfg :!rs/moq-net/verify/route/LagWide.cfg" - name: rest diff --git a/quest/m1/route-trace-validation.md b/quest/m1/route-trace-validation.md index 0b537f71e94..3f03f1f46e0 100644 --- a/quest/m1/route-trace-validation.md +++ b/quest/m1/route-trace-validation.md @@ -43,7 +43,3 @@ Run it on the sweeps' existing seeds in CI only if it stays fast; otherwise in Nightly with the sweeps' fresh seeds. Public API: none. Wire: none. - -## Required - -- [The router runs the path rule with a route per scope](/quest/m2/cluster-routes/scope.md) - the router `Sim` records, running the rule the model has diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index cd25d34fbaf..2e7e945cab0 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -43,17 +43,21 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region, unless it changes whether the node may use the route: by crossing it, or by reaching the hop limit. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes. -26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **a route is shared per publisher and per scope, and a relay believes a route by that route's own path** ([the prototype](https://github.com/moq-dev/moq/pull/8533), the model (`rs/moq-net/verify/route/RouteHandoff.tla`), then [the router](/quest/m2/cluster-routes/scope.md)). A scope is the prefix a relay asked a neighbour for when its grant there is narrower than its interest. Today only a feasible neighbour counts, one strictly cheaper than the distance the relay remembers, so a relay's claims move whenever a cost or seqno does: it can forget a claim for good, a mesh with two grants never settles after one seqno advance, and behind a scoped neighbour it never shows what a farther one knows. Only a broadcast some relay carries gets its own route, and that route propagates ([cache-aware routing](/quest/m3/cache-routing/README.md) Plan item 8). What keeps a route free of loops and stale copies is item 27. The contract's bullet on which neighbours certify now states the rule the model settled. First decided the same day as any neighbour whose path to the publisher does not pass through the relay; tried on the router, two relays behind a scoped relay then keep a dead claim alive through each other, and the relay behind a scoped neighbour shows a claim it has no route to request. Rejected: that rule; believing a neighbour that is cheaper than the relay is now, which fixed the storm and the forgotten claim with no wire change but rests belief on distance; a path per broadcast as lite-06 has; link-state among relays; routes to relays. +26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **a route is shared per publisher and per scope, and a relay believes a route by that route's own path** ([the prototype](https://github.com/moq-dev/moq/pull/8533), the model (`rs/moq-net/verify/route/RouteHandoff.tla`), then [the router](https://github.com/moq-dev/moq/pull/8622)). A scope is the prefix a relay asked a neighbour for when its grant there is narrower than its interest. Today only a feasible neighbour counts, one strictly cheaper than the distance the relay remembers, so a relay's claims move whenever a cost or seqno does: it can forget a claim for good, a mesh with two grants never settles after one seqno advance, and behind a scoped neighbour it never shows what a farther one knows. Only a broadcast some relay carries gets its own route, and that route propagates ([cache-aware routing](/quest/m3/cache-routing/README.md) Plan item 8). What keeps a route free of loops and stale copies is item 27. The contract's bullet on which neighbours certify now states the rule the model settled. First decided the same day as any neighbour whose path to the publisher does not pass through the relay; tried on the router, two relays behind a scoped relay then keep a dead claim alive through each other, and the relay behind a scoped neighbour shows a claim it has no route to request. Rejected: that rule; believing a neighbour that is cheaper than the relay is now, which fixed the storm and the forgotten claim with no wire change but rests belief on distance; a path per broadcast as lite-06 has; link-state among relays; routes to relays. -27. ✅ Decided 2026-11-11 (maintainer, on [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533), which measured it), asked what keeps a route free of loops and stale copies: **the route's path, a withdrawal that names the hop that died, holds, and a timed doubt. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo go.** None of it is built: the model (`rs/moq-net/verify/route/RouteHandoff.tla`) checks it, with dead hops kept until their timer fires, [the router](/quest/m2/cluster-routes/scope.md) builds it, [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire (item 28), and the contract below describes today's router until they do. The rule, as the prototype in `rs/moq-net/src/route/tests/paths.rs` runs it: +27. ✅ Decided 2026-11-11 (maintainer, on [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533), which measured it), asked what keeps a route free of loops and stale copies: **the route's path, a withdrawal that names the hop that died, holds, and a timed doubt. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo go.** The model (`rs/moq-net/verify/route/RouteHandoff.tla`) checks it, with dead hops kept until their timer fires, and [the router](https://github.com/moq-dev/moq/pull/8622) builds it as `route::path::Router` beside today's, which only the tests drive. [The switch](/quest/m2/cluster-routes/switch.md) puts it on the wire (item 28), and the contract below describes today's router until it does. The rule, as `rs/moq-net/src/route/path.rs` runs it: - A hop names a relay and the session it learned the route on. A relay uses the cheapest route whose path does not name it, at once. - A relay that loses a session names that hop, and every relay drops every route through it, whoever offered it. One message per session covers every publisher and lists the routes that moved to another path. - A relay with no route left holds its claims, unavailable, until each neighbour that learned the route from it has answered: with another way, or with nothing once those behind it have answered. Then it lets go. - A closed session casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time. No relay moves to a route through that node until a timer lifts the doubt. The route a relay already uses is exempt. + The maintainer delegated these decisions to the coordinator while away, recorded 2026-11-13: "The user is away; the coordinator takes recommendations on their behalf, open to change in review." + + Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. [The fresh-publisher exemption](/quest/m2/cluster-routes/fresh-publisher.md) is modeled and measured separately before deciding whether to adopt it. + Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). -28. ✅ Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel), asked how item 27's removal is split and ordered: **the router first with no wire change, then a switch of the sessions, and the switch lands before the route stream.** [The router](/quest/m2/cluster-routes/scope.md) runs the path rule with a route per scope, sans-IO, beside today's router. [The switch](/quest/m2/cluster-routes/switch.md) puts it on the lite-07-wip wire, the draft, the Rust sessions, and `js/net`, and deletes Babel's router. [The interest set](/quest/m2/cluster-routes/interest-set.md) then edits the path router only, and [the route stream](/quest/m2/cluster-routes/route-stream.md) follows the switch with no echo or route request to carry. The switch's recommended wire puts the answer that ends a hold on the publisher half, behind the routes its sender sent first, so the path rule needs no order between streams that today's sessions lack for one prefix. [Acceptance](/quest/m2/cluster-routes/acceptance.md) is of the switched sessions, and the inductive invariant stays in m1, rewritten for the path, without `LoopFree` as an invariant. Rejected: the router quest in m1; one quest for router, wire, draft, sessions, and `js/net`; the route stream first, which builds the echo and ROUTE_REQUEST on the new stream and then deletes them; one wire change for the path rule and the one stream together; moving the inductive invariant to m4; deleting it. Reversed 2026-11-13 (maintainer, /quest-complete): ✅ the inductive invariant is deleted, since Apalache cannot step even one transition of the route model ([moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564)), no Rust TLA+ checker qualifies ([moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600), [moq-dev/moq#8669](https://github.com/moq-dev/moq/pull/8669)), and no more JVM model checking is wanted here (rejected: keeping it paused). +28. ✅ Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel), asked how item 27's removal is split and ordered: **the router first with no wire change, then a switch of the sessions, and the switch lands before the route stream.** [The router](https://github.com/moq-dev/moq/pull/8622) runs the path rule with a route per scope, sans-IO, beside today's router. [The switch](/quest/m2/cluster-routes/switch.md) puts it on the lite-07-wip wire, the draft, the Rust sessions, and `js/net`, and deletes Babel's router. [The interest set](/quest/m2/cluster-routes/interest-set.md) then edits the path router only, and [the route stream](/quest/m2/cluster-routes/route-stream.md) follows the switch with no echo or route request to carry. The switch's recommended wire puts the answer that ends a hold on the publisher half, behind the routes its sender sent first, so the path rule needs no order between streams that today's sessions lack for one prefix. [Acceptance](/quest/m2/cluster-routes/acceptance.md) is of the switched sessions, and the inductive invariant stays in m1, rewritten for the path, without `LoopFree` as an invariant. Rejected: the router quest in m1; one quest for router, wire, draft, sessions, and `js/net`; the route stream first, which builds the echo and ROUTE_REQUEST on the new stream and then deletes them; one wire change for the path rule and the one stream together; moving the inductive invariant to m4; deleting it. Reversed 2026-11-13 (maintainer, /quest-complete): ✅ the inductive invariant is deleted, since Apalache cannot step even one transition of the route model ([moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564)), no Rust TLA+ checker qualifies ([moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600), [moq-dev/moq#8669](https://github.com/moq-dev/moq/pull/8669)), and no more JVM model checking is wanted here (rejected: keeping it paused). The route layer's core landed in [moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676): a @@ -129,7 +133,12 @@ the bullets below are the contract it carries. (`goaway_keeps_serving_until_a_refresh_makes_the_alternative_feasible` pins both orders). Using a draining peer's routes locally without advertising them was rejected: downstream nodes would lose their only route during the - drain. Existing readers resume only + drain. The path rule carries drain in each offer instead (decided + 2026-11-13, with priority below), and, decided the same day, a relay that + hears GOAWAY sends a DRAIN naming the leaving node in the DEAD message, so + every relay doubts it and moves at once only onto paths that avoid it: a + draining path ranks worst but stays usable, and no request loops through + offers that predate the drain. Existing readers resume only across a matching explicit epoch; epochless changes preserve Restart. - Fresh routing identity per process/tab restart, declared by the origin and shared across its live sessions, decided 2026-10-13. It is unverified. @@ -261,6 +270,12 @@ the bullets below are the contract it carries. (`transit_priority_depends_on_arrival_order`). This limitation is accepted; priority remains local policy, not a propagated metric tier. Endpoints that advertise no transit routes hold no transit feasibility distance. + Reversed for the path rule on 2026-11-13 (maintainer, + [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)): with no + feasibility, a local preference lets two relays prefer each other's paths + forever. A route's priority is the lowest link priority on its path, and a + route drains if any session on it drains; drain, then priority, rank before + cost. The switch carries both on the wire. - Decided 2026-10-14: one selection per identity, keeping the `upstream` link mark and its down-only bit. A node advertises its best route whether or not anything subscribes. A route learned on an upstream link, or through @@ -352,8 +367,8 @@ loss never reports an END: since it may certify a wildcard above both at a later revision; and withdrawing one origin's wildcard leaves another origin's copy (`wildcard_claims_end_per_origin_and_are_certified_below`). -- Not yet built ([the router](/quest/m2/cluster-routes/scope.md) - builds it and [the switch](/quest/m2/cluster-routes/switch.md) puts +- Not yet on the wire ([the router](https://github.com/moq-dev/moq/pull/8622) + built it and [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire; the rest of this contract describes today's router until then), settled by `rs/moq-net/verify/route/RouteHandoff.tla` with Plan item 27's @@ -373,7 +388,7 @@ loss never reports an END: neighbour's route names it and until the timer the hop's close started fires, so no route through a hop it knows is dead returns. Selected routes can loop only while the news that ends a stale route is on its way, and - only through at least one stale route. What its timer costs fired early, and the loop `Loop.cfg` pins, are in the `verify-route` skill. + only through at least one stale route. What its timer costs fired early, and the three-failure loop regression `Loop.cfg` pins, are in the `verify-route` skill. A sender sends a route on a stream only while the route meets the stream's interest (Plan item 15), and a node presents each stream only the claims that changed. diff --git a/quest/m2/cluster-routes/flapping-publisher.md b/quest/m2/cluster-routes/flapping-publisher.md index fcdc6320b8a..25e70f59aef 100644 --- a/quest/m2/cluster-routes/flapping-publisher.md +++ b/quest/m2/cluster-routes/flapping-publisher.md @@ -50,7 +50,7 @@ Work: of sessions that never closed to the publisher routes to it once that path is known. Run the existing configs under the new rule. - The path router (`route::path`, from - [the router](/quest/m2/cluster-routes/scope.md)) applies the rule, with the + [the router](https://github.com/moq-dev/moq/pull/8622)) applies the rule, with the model's counterexamples reproduced there first. - A fixture on real sessions with mocked time: the dual-homed publisher with one link flapping at half the timer, where a subscriber at a far relay @@ -62,10 +62,6 @@ Work: [the switch](/quest/m2/cluster-routes/switch.md) has landed, and the `verify-route` skill. -## Required - -- [The router runs the path rule with a route per scope](/quest/m2/cluster-routes/scope.md) - the doubt this changes - ## Related - [A relay that refuses a forwarded route drops the route, not the session](/quest/m1/refuse-route.md) - another failure mode from the same BGP comparison diff --git a/quest/m2/cluster-routes/fresh-publisher.md b/quest/m2/cluster-routes/fresh-publisher.md new file mode 100644 index 00000000000..10e5a978847 --- /dev/null +++ b/quest/m2/cluster-routes/fresh-publisher.md @@ -0,0 +1,51 @@ +# [L] Model and measure a fresh publisher's doubt exemption + +## Goal + +Determine whether a fresh authenticated direct publisher session can recover +its own route before the one-second doubt expires, without admitting a stale +return or a request loop. Compare it with timer-only doubt before deciding +whether to adopt it. + +## Plan + +Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, +while the maintainer was away): +timer-only doubt is the baseline in +[moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622), and this exemption +is modeled and measured separately. Early collection is rejected because it +permits the request loop recorded by `Loop.cfg`. These choices remain open +to change in review. + +The fixed 24-relay reconnect cases take 1,050 ms in a clique and 1,875 ms in +a ring, including propagation at 25 ms per hop, versus 50 ms and 325 ms for +the distance rule. Both path policies wait because heard routes still name +the doubted publisher. The exemption could remove that wait for a publisher +returning directly, but not for a relay presenting somebody else's route. + +Model what authenticates the publisher and binds the offer to the new direct +session, including delayed offers from the old session, reconnects, and +concurrent link failures. Keep the timer-only model as the control. Reproduce +counterexamples on the router and measure the same reconnect and churn +scripts on both policies, reporting traffic, request recovery, stale routes, +and reachable but unrouted observations. Preserve the baseline's safety +checks; a faster reconnect alone does not justify the exemption. + +Recommend limiting any exemption to the new authenticated publisher's own +direct offer. A forwarded offer or a new transport session alone does not +establish that the origin is alive. Record the evidence and adoption decision +before changing the default, and mirror an adopted rule in Rust, JS, and the +draft wherever the wire switch has already landed. + +## Waiting + +- Maintainer: after the model and measurements, adopt the narrowly authenticated + direct-publisher exemption or retain timer-only doubt. Recommend adoption only + if it preserves the safety checks and improves the measured reconnect cases; + otherwise retain the baseline. An agent can gather that evidence, but adopting + an exception changes the coordinator's accepted routing policy and needs an + explicit decision. + +## Related + +- [lite-07 sessions run the path router](/quest/m2/cluster-routes/switch.md) - supplies the authenticated session boundary when it puts the rule on the wire diff --git a/quest/m2/cluster-routes/interest-set.md b/quest/m2/cluster-routes/interest-set.md index 0a2ece649cf..5ecc2b01bdd 100644 --- a/quest/m2/cluster-routes/interest-set.md +++ b/quest/m2/cluster-routes/interest-set.md @@ -21,7 +21,7 @@ and every earlier routing bug was an ordering nobody had enumerated. Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel): ✅ this edits the path router, after -[the route per scope](/quest/m2/cluster-routes/scope.md) builds it, +[the route per scope](https://github.com/moq-dev/moq/pull/8622) builds it, and the model as it has a route per prefix (`rs/moq-net/verify/route/RouteHandoff.tla`) (rejected: landing first on Babel's router, which @@ -80,6 +80,13 @@ The edit rule those three imply, to prove rather than assume: Work: +- Extend the current single-prefix-per-neighbour interface to all the disjoint + regions of one permit. `peer_interest` currently replaces its one answer and + SYNCED names only the neighbour; several calls under that key cannot serve + several regions. Update `watch`/`synced`'s contract and cover a permit with + `fleet/r2` and `fleet/r7` together, including an edit that absorbs both into + `fleet`. The router and switch preceding this quest claim only one region + per neighbour. - Start from the model and the router with the path rule. Everything a side says to a neighbour shares one FIFO per direction, the hold answer included: with one stream per side nothing overtakes anything. @@ -114,10 +121,6 @@ own. Public API: none (the router is crate-private). Wire: none. -## Required - -- [The router runs the path rule with a route per scope](/quest/m2/cluster-routes/scope.md) - the router this edits, with a floor per route - ## Related - [lite-07 sessions run the path router](/quest/m2/cluster-routes/switch.md) - deletes Babel's router; whichever lands second adapts diff --git a/quest/m2/cluster-routes/route-exhaustive-sim.md b/quest/m2/cluster-routes/route-exhaustive-sim.md index 46fd4bfa82b..e9e6dbff8a4 100644 --- a/quest/m2/cluster-routes/route-exhaustive-sim.md +++ b/quest/m2/cluster-routes/route-exhaustive-sim.md @@ -12,7 +12,7 @@ does not fit, the PR records the numbers and lands no test. ## Plan Decided 2026-11-10 (maintainer, /quest-plan), as a spike: `RouteHandoff.tla` -is a hand-kept mirror of the rule `route/tests/paths.rs` prototypes, which TLC interprets, while `Sim` +is a hand-kept mirror of `route::path::Router` in `rs/moq-net/src/route/path.rs`, which TLC interprets, while `Sim` already drives the router one atomic step at a time. Searching `Sim` itself runs compiled code, has no model to drift, and reaches what the model leaves out. @@ -32,8 +32,8 @@ out. rule at the step that would break it rather than from that log. - The check that the search works: run on today's router before the switch to the path rule, it must find the storm - `residual_a_seqno_advance_storms_between_two_grants` replays, or another - `residual_` test in `route/tests/residual.rs`. Those fixtures use 6 to 10 + `distance::a_seqno_advance_storms_between_two_grants` replays, or another + of today's bugs in `route/tests/residual.rs`. Those fixtures use 6 to 10 nodes, beyond the triangle, so the self-check searches the fixture's own mesh, bounded by depth and by the fixture's scripted claims, rather than the triangle's bounds; the search code takes the mesh as a parameter. @@ -51,18 +51,14 @@ search covers what a model check does, recommend in the PR what to do with the route model's TLC runs. Decided 2026-11-12 (maintainer, pick round): the search runs on the `Router` -after [the router quest](/quest/m2/cluster-routes/scope.md)'s rewrite to -the path rule, so it checks the code that ships and nothing is built twice +after [the router quest](https://github.com/moq-dev/moq/pull/8622)'s rewrite to +the path rule (`route::path::Router`), so it checks the code that ships and nothing is built twice (rejected: today's Babel `Router`, which the rewrite throws away; the test-only `paths.rs` prototype). The storm self-check above then runs against a router from before the rewrite. Public API: none. Wire: none. -## Required - -- [The router quest](/quest/m2/cluster-routes/scope.md) - rewrites `route::Router` to the path rule, the router this searches - ## Related - [Route sweeps are checked against the TLA+ model](/quest/m1/route-trace-validation.md) - guards the model against the code; this checks the code itself diff --git a/quest/m2/cluster-routes/scope.md b/quest/m2/cluster-routes/scope.md deleted file mode 100644 index f8a8d553657..00000000000 --- a/quest/m2/cluster-routes/scope.md +++ /dev/null @@ -1,115 +0,0 @@ -# [XL] The router runs the path rule with a route per scope - -## Goal - -`route::Router` runs the rule the model -(`rs/moq-net/verify/route/RouteHandoff.tla`) settled, sans-IO, and -`route::tests::Sim` drives it. A route is keyed by its publisher and the -prefix it covers and carries its path. A relay uses the cheapest route whose -path does not name it, drops every route through a hop a withdrawal names, -holds its claims while the neighbours that learned a route from it have yet -to answer, and stays off a node in doubt until the timer lifts it. It keeps -no distance, seqno, route request, or echo. - -A relay shows a claim exactly while such a route covers it and the newest -revision a covering route gave it still has the claim, two relays never keep -a dead route alive through each other, a change of path alone changes no -claim a reader sees, and a relay behind a scoped neighbour reaches what a -farther one offers. The `residual_` tests in -`rs/moq-net/src/route/tests/residual.rs` pass as plain tests on it, and a -sweep over meshes with scoped relays holds the rules at every quiet point. - -Sessions, the wire, the draft, and `js/net` keep today's router until -[the switch](/quest/m2/cluster-routes/switch.md), so nothing a peer -or an application sees changes here. - -## Plan - -Decided 2026-11-11 (maintainer, /quest-plan): the prototype picks the rule -for stale copies, the model settles the route per scope, and the router comes -last (rejected: the router first, patched case by case; recording the -findings as known limits and waiting for federation). - -Decided 2026-11-11 (maintainer, on -[moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533)): the route rule -changes too. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END -echo go, for the rule in the -[epic](/quest/m2/cluster-routes/README.md)'s Plan item 27. - -Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for -Babel), asked how that removal is split: ✅ two quests in this epic. This one -builds the router with no wire change, and -[the switch](/quest/m2/cluster-routes/switch.md) moves the sessions, -the wire, the draft, and `js/net` onto it and deletes Babel's router -(rejected: the same split with this quest in m1, since it breaks nothing; -one quest for everything, which is one PR across the router, the wire, the -draft, the Rust sessions, and `js/net`). - -Open, from the model -([moq-dev/moq#8541](https://github.com/moq-dev/moq/pull/8541)): how long a -relay keeps its doubt about a node. As the rule stands the doubt goes once no -route names the node, and a request can then loop while the news of three -closed sessions is still on its way: `Loop.cfg` breaks `RequestsLoopFree` at -depth 11 and is kept as that open finding. Keeping the doubt until its timer -fires showed no loop on that mesh in 7.6M states, to depth 13, and costs a -wait: no relay moves to a route through the doubted node until the timer -fires, a publisher reconnecting through another relay included, though a -relay keeps one it already has. Measure that reconnect delay on -the prototype's meshes under both rules and put the two to the maintainer -with the numbers. Recommended: keep the doubt, since the wait is bounded by -the timer and a looping request is not bounded by anything the relay -controls. If the rule changes, -`Loop.cfg` passes and `RequestsLoopFree` joins the rules every config checks. - -The two rules share a tree until the switch. Pick how, and say why in the -PR: the prototype (`route/tests/paths.rs`, `compare.rs`, `copies.rs`) -promoted out of `tests/` into a second router, or one router generic over -the rule. Either way the claim side is one implementation, not a copy: the -views, floors, `show`, and the interest handling are what the fixtures of -the last month fixed, and the rule changes only which routes feed them. -Sessions construct today's rule and nothing on the wire changes. Whatever -of the prototype the router replaces is deleted here. - -Work: - -- Route selection by the rule of the epic's Plan item 27, as the model - states it: the path with a session in each hop, the withdrawal naming a - dead hop and listing the routes that moved, holds and their answers, and - the doubt with its timer on the router's mocked clock. The timer's length - starts from the maintainer's 1 s; the model and the sweeps settle it. -- Key `rx`, the selection, `sent`, and the views by route. A view then has - one floor, so `certifiers`, `certified`, `prune`, `loose`, and the - per-region floors have no counterpart, and the newest floor wins among one - publisher's routes as the model has it. Keep the reader's view a function - of what the router holds, computed incrementally the way `show` is today, - with the test build comparing it against the whole computation. -- Keep a session's teardown about linear in the prefixes it held, as the - routes bench's `teardown` and `refollow` cells measure - (`rs/moq-net/benches/routes.rs`). -- What #8487 added to work around the old rule has no counterpart either: - `recompute`'s `prune` switch, which skips pruning on a change of hops or - RTT alone. Keep the qualified-key test in `infeasible` only if the new - rule still needs it, and say which. -- Expose the route the model's rule picks for a request, as a crate-private - query, so the switch can serve a request through it in `route/node.rs`'s - `home`. `Sim` checks it reaches the publisher without a loop (`Reaches`). -- Run the handoff fixtures and both sweeps on the new rule. Flip each - `residual_` test, and add the review harness's scoped and two-prefix - families as a sweep beside `sweep_no_route_outgrows_a_loop_free_path`, - checking at every quiet point that each relay shows what the rule says and - nothing waits. Compare a few thousand scripts against today's router and - explain every difference in the PR. -- Report route and claim messages per change on `Sim` for both rules, on the - meshes `compare.rs` holds. The switch measures real sessions. -- Update the `verify-route` skill's coverage and sync commit for the rule - `Sim` now runs. - -Left to [the switch](/quest/m2/cluster-routes/switch.md): the wire, -the draft, `route/node.rs` and the sessions, `js/net`, the epic's contract -as built, and the deletion of today's router. - -Public API: none (the router is crate-private). Wire: none. - -## Related - -- [lite-07 sessions run the path router](/quest/m2/cluster-routes/switch.md) - puts this router on the wire and deletes the one it replaces diff --git a/quest/m2/cluster-routes/switch.md b/quest/m2/cluster-routes/switch.md index 692083d0496..e71904da705 100644 --- a/quest/m2/cluster-routes/switch.md +++ b/quest/m2/cluster-routes/switch.md @@ -4,8 +4,8 @@ Every lite-07 session, in Rust and in `js/net`, routes by the rule of the [epic](/quest/m2/cluster-routes/README.md)'s Plan item 27, on the -router [the route per scope](/quest/m2/cluster-routes/scope.md) -built. A route on the wire carries its path with a session in each hop and +router [the route per scope](https://github.com/moq-dev/moq/pull/8622) +built (`route::path::Router`). A route on the wire carries its path with a session in each hop and the prefix it covers. A relay that loses a session names that hop in one message, and a relay answers a neighbour that holds for it. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo are gone from @@ -33,6 +33,27 @@ Babel): routing message on the subscriber half, so the ordering the route stream was planned to give the echo is not needed here. +Decided 2026-11-13 (maintainer, +[moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)): + +- ✅ ROUTE_START and ROUTE_UPDATE carry the route's drain bit, set when any + session on its path drains after GOAWAY, and its priority, the lowest link + priority on its path. Both rank before the metric, as `path::Offer` + carries them (rejected: drain and priority as tie-breakers below cost; a + drain cost penalty that drops priority). The router sends an update when + either changes, and a DEAD's moved routes carry them too. +- ✅ The message that carries DEAD gains a DRAIN entry naming a node that + sent GOAWAY (decided 2026-11-13, maintainer, /quest-complete). The relay + that hears GOAWAY sends it to each neighbour it sent a route through that + node, and each relay passes it on the same way, once per neighbour while + such a route stays sent. Every relay ranks a path through the node as + draining, the worst rank, but keeps it usable and never removes it. It + doubts the node until the existing timer fires, so it moves at once only + onto paths that avoid the node, and a request never loops through offers + that predate the drain. A relay accepts a DRAIN only for a node on a path + that neighbour offered. Rejected: taking drain out of the rank, so routes + move only when the session closes; accepting brief request loops. + Recommended wire, for the implementer to confirm or improve in the PR; each departure is a choice to list there. Where the model settled something else, the model wins: @@ -44,7 +65,8 @@ else, the model wins: are safe. The hop limit stays the wire's 32. - One message names a dead hop. A relay sends it once per session it loses, to every neighbour, covering every publisher, and it lists the routes that - moved to another path, as the prototype has it. + moved to another path, as the prototype has it. The same message carries DRAIN + entries, each naming a node that sent GOAWAY, covering every publisher. - The answer that ends a hold rides the publisher half: the stream that carries the answering side's routes to the relay that waits. It is then ordered behind every route its sender sent first, on today's streams. A @@ -61,10 +83,18 @@ Work: - `route/node.rs` drives the new router. `home` serves a request through the route the rule picks, in place of every feasible route whose neighbour `carries` the claim. + Honor `path::Router::new`'s fresh-hop-per-process contract, including when + `Config::hop` is configured: the old seqno cannot recover reused path-hop names. + The driver also implements `watch`/`synced`'s single-prefix scope contract, + completing cancelled streams locally when no replacement can answer. This + stage supports one scope region per neighbour. Multi-region permits need + [the interest set](/quest/m2/cluster-routes/interest-set.md), including its + per-prefix SYNCED; requesting each region under the same neighbour key would + replace the previous answer. Do not claim that coverage before it lands. - Delete today's router with what only it used: the feasibility distance and its drain, seqno requests and their pending state, the echo, and their fixtures. A fixture that states a claim rule moves to the new router if - [the route per scope](/quest/m2/cluster-routes/scope.md) did not + [the route per scope](https://github.com/moq-dev/moq/pull/8622) did not move it already. - The Rust codec and sessions (`rs/moq-net/src/lite/`), and `js/net`'s route layer (`js/net/src/lite/publisher.ts`, `js/net/src/lite/subscriber.ts`, @@ -122,10 +152,6 @@ rewritten codec and the path router keep it; if not, it follows this switch. Public API: none expected. Wire: lite-07-wip only. -## Required - -- [The router runs the path rule with a route per scope](/quest/m2/cluster-routes/scope.md) - the router these sessions drive - ## Related - [Each side says everything about routing on one stream](/quest/m2/cluster-routes/route-stream.md) - moves these messages onto one stream per side afterwards diff --git a/quest/m3/route-keep-next-hop.md b/quest/m3/route-keep-next-hop.md index cb30776d56b..ed51fc632e1 100644 --- a/quest/m3/route-keep-next-hop.md +++ b/quest/m3/route-keep-next-hop.md @@ -41,8 +41,8 @@ selection. means what it says, rather than editing expectations to match. - How a route is selected is being replaced ([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27). - Apply this to the rule that stands when this starts. The prototype in - `route/tests/paths.rs` already stays on its next hop among equals. + Apply this to the rule that stands when this starts. The scoped path router in + `rs/moq-net/src/route/path.rs` already stays on its next hop among equals. - Measure with `Sim`'s `hop_updates` count and, if it has landed, `mesh_sweep.rs`'s benchmark, before and after. diff --git a/rs/moq-net/benches/routes.rs b/rs/moq-net/benches/routes.rs index 44efbc4ca68..b3c35f9f339 100644 --- a/rs/moq-net/benches/routes.rs +++ b/rs/moq-net/benches/routes.rs @@ -23,6 +23,16 @@ //! - refollow: the same node closes one prefix's stream and opens it again. The claims //! axis is expected to grow, since the stream hears its prefix's claims again; the //! prefixes axis no faster than the prefixes held. +//! - path/watched, path/unwatched, path/received: the same fan-out grids on the path rule. +//! - path/follow, path/teardown, path/refollow: the prefix-by-claim grid on the path +//! router, including initial following and timer work with a virtual clock. +//! - path/drain: a relay hears a node it reaches over many sessions send GOAWAY, with +//! a route per session sent to each stream it answers. Swept over the leaving +//! node's sessions and the streams: both axes are expected to grow, since each +//! stream is re-sent each route ranked draining, and no faster than their product. +//! - path/drained: after the drain, an RTT change on a session with no routes. Swept +//! over the same grid: the streams axis is expected to grow, since each stream's +//! DRAIN notice is kept; the sessions axis is not. //! //! The route layer is crate-private, so this reaches it through the `fuzz` module. //! @@ -43,6 +53,10 @@ const HELD: [(usize, usize); 4] = [(10, 10), (10_000, 10), (10, 100), (10_000, 1 /// `(prefixes, claims)` shapes for a node following one origin per prefix. const FOLLOWED: [(usize, usize); 4] = [(16, 1), (1_024, 1), (16, 16), (1_024, 16)]; +/// `(sessions, streams)` shapes for a relay hearing a node it reaches over many +/// sessions drain. +const DRAINING: [(usize, usize); 4] = [(4, 4), (128, 4), (4, 128), (128, 128)]; + /// `(claims, lag)` shapes for a relay hearing a stream that lags another. const LAGGING: [(usize, usize); 4] = [(100, 1), (10_000, 1), (100, 100), (10_000, 10_000)]; @@ -150,6 +164,125 @@ fn bench_refollow(c: &mut Criterion) { group.finish(); } +fn bench_path_teardown(c: &mut Criterion) { + let mut group = c.benchmark_group("routes/path/teardown"); + for (prefixes, claims) in FOLLOWED { + let id = BenchmarkId::from_parameter(format!("{prefixes}p_{claims}c")); + group.bench_function(id, |b| { + b.iter_batched( + || fuzz::PathRouteFollower::new(prefixes, claims), + fuzz::PathRouteFollower::teardown, + BatchSize::PerIteration, + ); + }); + } + group.finish(); +} + +fn bench_path_follow(c: &mut Criterion) { + let mut group = c.benchmark_group("routes/path/follow"); + for (prefixes, claims) in FOLLOWED { + let id = BenchmarkId::from_parameter(format!("{prefixes}p_{claims}c")); + group.bench_function(id, |b| { + b.iter_batched( + || (), + |()| fuzz::PathRouteFollower::new(prefixes, claims), + BatchSize::PerIteration, + ); + }); + } + group.finish(); +} + +fn bench_path(c: &mut Criterion, name: &str, interest: fn(usize) -> String, sent: usize) { + let mut group = c.benchmark_group(format!("routes/path/{name}")); + for (claims, streams) in SHAPES { + let id = BenchmarkId::from_parameter(format!("{claims}c_{streams}s")); + let mut fixture = None; + group.bench_function(id, |b| { + let node = fixture.get_or_insert_with(|| { + let interests: Vec = (0..streams).map(interest).collect(); + let interests: Vec<&str> = interests.iter().map(String::as_str).collect(); + fuzz::PathRouteNode::new(claims, &interests) + }); + b.iter(|| { + node.own("live/extra", true); + assert_eq!(node.drain(), sent * streams); + node.own("live/extra", false); + assert_eq!(node.drain(), sent * streams); + }); + }); + } + group.finish(); +} + +fn bench_path_watched(c: &mut Criterion) { + bench_path(c, "watched", |_| String::new(), 1); +} + +fn bench_path_unwatched(c: &mut Criterion) { + bench_path(c, "unwatched", |i| format!("other/{i}"), 0); +} + +fn bench_path_received(c: &mut Criterion) { + let mut group = c.benchmark_group("routes/path/received"); + for (claims, changes) in RECEIVED { + let id = BenchmarkId::from_parameter(format!("{claims}c_{changes}d")); + let mut fixture = None; + group.bench_function(id, |b| { + let relay = fixture.get_or_insert_with(|| fuzz::PathRouteRelay::new(claims)); + b.iter(|| assert_eq!(relay.change(changes), (1, changes))); + }); + } + group.finish(); +} + +fn bench_path_refollow(c: &mut Criterion) { + let mut group = c.benchmark_group("routes/path/refollow"); + for (prefixes, claims) in FOLLOWED { + let id = BenchmarkId::from_parameter(format!("{prefixes}p_{claims}c")); + let mut fixture = None; + group.bench_function(id, |b| { + let follower = fixture.get_or_insert_with(|| fuzz::PathRouteFollower::new(prefixes, claims)); + b.iter(|| follower.refollow()); + }); + } + group.finish(); +} + +fn bench_path_drain(c: &mut Criterion) { + let mut group = c.benchmark_group("routes/path/drain"); + for (sessions, streams) in DRAINING { + let id = BenchmarkId::from_parameter(format!("{sessions}n_{streams}s")); + group.bench_function(id, |b| { + // Borrowed, so dropping the fixture stays out of the sample. + b.iter_batched_ref( + || fuzz::PathRouteDrain::new(sessions, streams), + |node| node.drain(), + BatchSize::PerIteration, + ); + }); + } + group.finish(); +} + +fn bench_path_drained(c: &mut Criterion) { + let mut group = c.benchmark_group("routes/path/drained"); + for (sessions, streams) in DRAINING { + let id = BenchmarkId::from_parameter(format!("{sessions}n_{streams}s")); + let mut fixture = None; + group.bench_function(id, |b| { + let node = fixture.get_or_insert_with(|| { + let mut node = fuzz::PathRouteDrain::new(sessions, streams); + node.drain(); + node + }); + b.iter(|| node.idle()); + }); + } + group.finish(); +} + criterion_group!( benches, bench_watched, @@ -158,6 +291,14 @@ criterion_group!( bench_lagging, bench_confirmed, bench_teardown, - bench_refollow + bench_refollow, + bench_path_follow, + bench_path_watched, + bench_path_unwatched, + bench_path_received, + bench_path_teardown, + bench_path_refollow, + bench_path_drain, + bench_path_drained ); criterion_main!(benches); diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index e8b0532d015..4b8b66df06e 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -866,6 +866,292 @@ impl RouteFollower { } } +/// The path router originating claims and answering streams, for the routes bench. +pub struct PathRouteNode(crate::route::path::Router); + +impl PathRouteNode { + /// An origin with `claims` claims, answering the given interests. + pub fn new(claims: usize, interests: &[&str]) -> Self { + let mut node = Self(crate::route::path::Router::new(1, 1, Default::default())); + node.0.originate(1); + for i in 0..claims { + node.own(&format!("live/{i}"), true); + } + for (peer, prefix) in interests.iter().enumerate() { + node.0.connect(peer, peer as u64 + 2, Default::default()).unwrap(); + node.0.peer_interest(peer, Some(Path::new(prefix).unwrap())).unwrap(); + } + node.drain(); + node + } + + /// Start or withdraw a claim. + pub fn own(&mut self, claim: &str, start: bool) { + let claim = Path::new(claim).unwrap(); + if start { + self.0.announce(claim, ()).unwrap(); + } else { + self.0.unannounce(claim).unwrap(); + } + } + + /// Consume messages and reader notifications, returning the messages sent. + pub fn drain(&mut self) -> usize { + let sent = self.0.drain().len(); + self.0.events(); + self.0.changed(); + sent + } +} + +/// A path router receiving one publisher and forwarding its claims to a reader. +pub struct PathRouteRelay { + router: crate::route::path::Router, + revision: u64, +} + +impl PathRouteRelay { + /// Hear `claims` claims and answer one stream watching the root. + pub fn new(claims: usize) -> Self { + use crate::route::path::{Hop, Offer, Reach}; + let mut relay = Self { + router: crate::route::path::Router::new(2, 2, Default::default()), + revision: 1, + }; + relay.router.connect(0, 1, Default::default()).unwrap(); + relay.router.route( + 0, + 1, + Path::empty(), + Some(Reach::Finite(Offer { + path: [Hop { node: 1, session: 0 }].into(), + draining: false, + priority: i8::MAX, + metric: 0, + down_only: false, + rtt: None, + })), + 0, + ); + let snapshot = (0..claims) + .map(|i| (Path::new(&format!("live/{i}")).unwrap(), Some(1))) + .collect(); + relay.router.batch(0, 1, Path::empty(), snapshot, Some(1), 0); + relay.router.connect(1, 3, Default::default()).unwrap(); + relay.router.peer_interest(1, Some(Path::empty())).unwrap(); + relay.drain(); + relay + } + + /// Receive one batch and return its forwarded messages and reader changes. + pub fn change(&mut self, claims: usize) -> (usize, usize) { + self.revision += 1; + let changes = (0..claims) + .map(|i| (Path::new(&format!("live/{i}")).unwrap(), Some(self.revision))) + .collect(); + self.router.batch(0, 1, Path::empty(), changes, Some(self.revision), 0); + self.drain() + } + + fn drain(&mut self) -> (usize, usize) { + let sent = self.router.drain().len(); + let events = self.router.events().len(); + self.router.changed(); + (sent, events) + } +} + +/// The path router following one publisher on a session per prefix, for the routes bench. +pub struct PathRouteFollower { + router: crate::route::path::Router, + prefixes: Vec, + claims: usize, + next: usize, + now: kio::clock::Instant, +} + +impl PathRouteFollower { + /// A route for each `live/{k}`, each carrying `claims` claims. + pub fn new(prefixes: usize, claims: usize) -> Self { + let mut follower = Self { + router: crate::route::path::Router::new(2, 2, crate::route::path::Doubt::default()), + prefixes: (0..prefixes) + .map(|k| Path::new(&format!("live/{k}")).unwrap()) + .collect(), + claims, + next: 0, + now: kio::clock::Instant::now(), + }; + for k in 0..prefixes { + follower.follow(k); + } + follower.drain(); + assert_eq!(follower.router.shown().len(), prefixes * claims); + follower + } + + fn follow(&mut self, k: usize) { + use crate::route::path::{Hop, Offer, Reach}; + let prefix = self.prefixes[k].clone(); + self.router.connect(k, 1, crate::route::Link::default()).unwrap(); + self.router.route( + k, + 1, + prefix.clone(), + Some(Reach::Finite(Offer { + path: [Hop { node: 1, session: 0 }].into(), + draining: false, + priority: i8::MAX, + metric: 0, + down_only: false, + rtt: None, + })), + 0, + ); + let changes = (0..self.claims) + .map(|j| (Path::new(&format!("live/{k}/{j}")).unwrap(), Some(()))) + .collect(); + self.router.batch(k, 1, prefix, changes, Some(1), 0); + } + + /// Close every session in one call, including the resulting timer work. + pub fn teardown(mut self) { + self.router.disconnect(0..self.prefixes.len(), self.now); + self.expire(); + assert!(self.router.shown().is_empty()); + self.drain(); + } + + /// Replace one prefix's session and snapshot, including the resulting timer work. + pub fn refollow(&mut self) { + let k = self.next % self.prefixes.len(); + self.next += 1; + self.router.disconnect([k], self.now); + self.follow(k); + self.expire(); + assert_eq!(self.router.shown().len(), self.prefixes.len() * self.claims); + self.drain(); + } + + fn expire(&mut self) { + // Advance the mocked input clock rather than waiting a real doubt interval. + self.now = self.now.saturating_add(crate::route::path::Doubt::default().timer); + self.router.timeout(self.now); + } + + fn drain(&mut self) { + self.router.drain(); + self.router.events(); + self.router.changed(); + } +} + +/// A path router hearing one publisher through `sessions` sessions of a node that +/// then sends GOAWAY, a route per session, and answering `recipients` streams that +/// watch the root, for the routes bench. +pub struct PathRouteDrain { + router: crate::route::path::Router, + recipients: usize, + idle: usize, + rtt: u64, +} + +impl PathRouteDrain { + const LEAVING: u64 = 500; + + /// Hear a route over each of `sessions` sessions of the leaving node, and answer + /// `recipients` streams watching the root. + pub fn new(sessions: usize, recipients: usize) -> Self { + use crate::route::path::{Hop, Offer, Reach}; + let mut router = crate::route::path::Router::new(2, 2, crate::route::path::Doubt::default()); + for i in 0..sessions { + let prefix = Path::new(&format!("live/{i}")).unwrap(); + let via = 1_000 + i as u64; + router.connect(i, via, crate::route::Link::default()).unwrap(); + let path = [ + Hop { node: 1, session: 0 }, + Hop { + node: Self::LEAVING, + session: i as u64 + 1, + }, + Hop { node: via, session: 1 }, + ]; + router.route( + i, + 1, + prefix.clone(), + Some(Reach::Finite(Offer { + path: path.into(), + draining: false, + priority: i8::MAX, + metric: 1, + down_only: false, + rtt: None, + })), + 0, + ); + let claim = Path::new(&format!("live/{i}/x")).unwrap(); + router.batch(i, 1, prefix, vec![(claim, Some(()))], Some(1), 0); + } + for r in 0..recipients { + let peer = sessions + r; + router + .connect(peer, 5_000 + r as u64, crate::route::Link::default()) + .unwrap(); + router.peer_interest(peer, Some(Path::empty())).unwrap(); + } + let idle = sessions + recipients; + router.connect(idle, 9_000, crate::route::Link::default()).unwrap(); + let mut node = Self { + router, + recipients, + idle, + rtt: 0, + }; + node.flush(); + node + } + + /// Hear the leaving node drain, returning the messages sent: each recipient's + /// DRAIN, then each route it was sent, now ranked draining. + pub fn drain(&mut self) -> usize { + use crate::route::path::{Dead, Out}; + let now = kio::clock::Instant::now(); + self.router.dead(0, Dead::Draining { node: Self::LEAVING }, now); + let out = self.router.drain(); + let told = (out.iter()) + .filter(|out| { + matches!( + out, + Out::Dead { + dead: Dead::Draining { .. }, + .. + } + ) + }) + .count(); + assert_eq!(told, self.recipients, "each recipient hears the drain once"); + self.router.events(); + self.router.changed(); + out.len() + } + + /// An RTT change on a session with no routes, which changes nothing it was sent. + pub fn idle(&mut self) { + self.rtt = self.rtt % 8 + 1; + self.router + .rtt(self.idle, Some(std::time::Duration::from_millis(self.rtt))); + assert_eq!(self.flush(), 0); + } + + fn flush(&mut self) -> usize { + let sent = self.router.drain().len(); + self.router.events(); + self.router.changed(); + sent + } +} + /// What a client session fed into an origin hears it should ask its peer about, for /// the interests bench. pub struct Interests(crate::model::Interests); diff --git a/rs/moq-net/src/route/mod.rs b/rs/moq-net/src/route/mod.rs index c1b36c31417..8e17adef175 100644 --- a/rs/moq-net/src/route/mod.rs +++ b/rs/moq-net/src/route/mod.rs @@ -17,7 +17,9 @@ )] pub(crate) mod node; +pub(crate) mod path; mod proof; +mod reader; mod rtt; mod view; @@ -40,10 +42,11 @@ pub(crate) use node::Node; pub(crate) use proof::EXPORTER_LABEL; pub use proof::RouteKey; pub(crate) use proof::{Proof, Transcript}; +use reader::Reader; #[cfg(feature = "async")] pub(crate) use rtt::LinkRtt; pub(crate) use view::{View, within}; -use view::{above, ancestors, below, covering, narrow, within_any}; +use view::{above, below, covering, narrow, within_any}; /// An opaque routing identity, declared by an origin or derived by [`Scope`]. pub(crate) type Id = u64; @@ -452,24 +455,17 @@ pub(crate) struct Router { current: BTreeMap>, heard: BTreeMap<(Id, N), Heard>, sent_views: BTreeMap<(Id, N), View>, - shown: BTreeMap<(Id, Path), Shown>, + reader: Reader, /// The claims changed per identity since it was last recomputed. dirty: BTreeMap, /// Each identity's context as last recomputed: a change marks every claim dirty. ctxs: BTreeMap>, - /// The identities recomputed since [`Self::changed`] last took them, each with - /// the claims whose shown state changed. - changed: BTreeMap>, /// Mints a fresh identity on rotation. rng: u64, - /// Per claim this node has shown, how many times it forgot that claim. - #[cfg(test)] - forgot: BTreeMap<(Id, Path), u32>, /// Identities recomputed, so fixtures can bound the work per message. #[cfg(test)] touched: usize, out: Vec>, - events: Vec>, } /// A refused input. @@ -504,24 +500,20 @@ impl Router { current: BTreeMap::new(), heard: BTreeMap::new(), sent_views: BTreeMap::new(), - shown: BTreeMap::new(), + reader: Reader::default(), dirty: BTreeMap::new(), ctxs: BTreeMap::new(), - changed: BTreeMap::new(), rng: seed | 1, #[cfg(test)] - forgot: BTreeMap::new(), - #[cfg(test)] touched: 0, out: Vec::new(), - events: Vec::new(), } } /// Take the identities recomputed since the last call, whose selection may have /// changed, each with the claims whose shown state did. pub fn changed(&mut self) -> BTreeMap> { - std::mem::take(&mut self.changed) + self.reader.changed() } /// Take the messages queued for neighbors, in order. @@ -531,7 +523,7 @@ impl Router { /// Take the reader events, in order. pub fn events(&mut self) -> Vec> { - std::mem::take(&mut self.events) + self.reader.events() } // --- Sessions --- @@ -1223,13 +1215,13 @@ impl Router { /// What this node's reader sees. pub fn shown(&self) -> &BTreeMap<(Id, Path), Shown> { - &self.shown + self.reader.shown() } #[cfg(test)] /// Whether this node shows `claim` from `id`, and whether it is available. pub fn shows(&self, id: Id, claim: &Path) -> Option { - self.shown.get(&(id, claim.clone())).map(|shown| shown.available) + self.reader.get(id, claim).map(|shown| shown.available) } #[cfg(test)] @@ -1443,14 +1435,8 @@ impl Router { } /// Count a forget against each shown claim of `id` overlapping none of `kept`. - #[cfg_attr(not(test), allow(unused_variables, reason = "only fixtures count forgets"))] fn forgot_outside(&mut self, id: Id, kept: &BTreeSet) { - #[cfg(test)] - for ((key, claim), count) in self.forgot.iter_mut() { - if *key == id && !within_any(claim, kept) { - *count += 1; - } - } + self.reader.forget(id, |claim| within_any(claim, kept)); } /// The view this node shows and forwards for `id`, having selected `route`: what @@ -1522,7 +1508,7 @@ impl Router { ids.extend(self.sent.keys().map(|(id, _)| *id)); ids.extend(self.unacked.keys().map(|(id, _)| *id)); ids.extend(self.heard.keys().map(|(id, _)| *id)); - ids.extend(self.shown.keys().map(|(id, _)| *id)); + ids.extend(self.reader.shown().keys().map(|(id, _)| *id)); ids } @@ -1572,7 +1558,7 @@ impl Router { self.touched += ids.len(); } for &id in &ids { - self.changed.entry(id).or_default(); + self.reader.touch(id); } for id in ids { if prune { @@ -1930,10 +1916,7 @@ impl Router { fn show(&mut self, id: Id, ctx: &Ctx, dirty: &Dirty) { let claims: BTreeSet = match dirty.all { true => { - let mut claims: BTreeSet = (self.shown.range((id, Path::empty())..)) - .take_while(|((key, _), _)| *key == id) - .map(|((_, claim), _)| claim.clone()) - .collect(); + let mut claims: BTreeSet = self.reader.claims(id).cloned().collect(); if let Some(view) = self.adopted(id, self.selected(id)) { claims.extend(view.items.keys().cloned()); } @@ -1945,92 +1928,23 @@ impl Router { #[cfg(test)] if !dirty.all { let route = self.selected(id); - let mut all: BTreeSet = (self.shown.range((id, Path::empty())..)) - .take_while(|((key, _), _)| *key == id) - .map(|((_, claim), _)| claim.clone()) - .collect(); + let mut all: BTreeSet = self.reader.claims(id).cloned().collect(); if let Some(view) = self.adopted(id, route) { all.extend(view.items.keys().cloned()); } - for (claim, full) in self.shown_now(id, ctx, all.iter()) { - let got = match now.get(&claim) { - Some(got) => got.as_ref(), - None => self.shown.get(&(id, claim.clone())), - }; - assert_eq!(got, full.as_ref(), "incremental reader view of {claim:?}"); - } + self.reader.check(id, &now, self.shown_now(id, ctx, all.iter())); } // A claim no routed neighbor's region covers was let go, not ended, and so // was one the view never certified: only its stream left. let regions = self.certified(id, true); - let shown = |claim: &Path| match now.get(claim) { - Some(now) => now.as_ref(), - None => self.shown.get(&(id, claim.clone())), + let views = &self.views; + let uncertified = |claim: &Path| { + regions.as_ref().is_some_and(|regions| !within_any(claim, regions)) + || !ctx.own && views.get(&id).is_none_or(|view| view.as_of(claim).is_none()) }; - let mut events = Vec::new(); - for (claim, now) in &now { - let Some(was) = self.shown.get(&(id, claim.clone())) else { - continue; - }; - match now { - Some(now) => { - if was.item != now.item { - events.push((claim, Kind::Update, &now.item)); - } - if was.available != now.available { - let kind = match now.available { - true => Kind::Available, - false => Kind::Unavailable, - }; - events.push((claim, kind, &now.item)); - } - } - None => { - let uncertified = regions.as_ref().is_some_and(|regions| !within_any(claim, regions)) - || !ctx.own && self.views.get(&id).is_none_or(|view| view.as_of(claim).is_none()); - // A claim presented at a narrower region moved up to the claim it - // stood for, which is still shown. That claim may have changed - // meanwhile, so a presented claim needs only something shown above it. - let moved = ancestors(claim) - .filter(|above| above != claim) - .any(|above| shown(&above).is_some_and(|now| was.presented || now.item == was.item)); - let kind = if !within(claim, &self.watch) { - Kind::Unwatched - } else if ctx.live && !uncertified && !moved { - Kind::End - } else { - Kind::Offline - }; - events.push((claim, kind, &was.item)); - } - } - } - for (claim, now) in &now { - if let Some(now) = now - && !self.shown.contains_key(&(id, claim.clone())) - { - events.push((claim, Kind::Start, &now.item)); - } - } - let events: Vec<(Path, Kind, A)> = (events.into_iter()) - .map(|(claim, kind, item)| (claim.clone(), kind, item.clone())) - .collect(); - for (claim, kind, item) in events { - self.log(id, &claim, kind, &item); - } - let changed = self.changed.entry(id).or_default(); - for (claim, now) in now { - let key = (id, claim); - if self.shown.get(&key) == now.as_ref() { - continue; - } - changed.insert(key.1.clone()); - match now { - Some(now) => self.shown.insert(key, now), - None => self.shown.remove(&key), - }; - } + self.reader + .apply(id, &self.watch, now, |claim| ctx.live && !uncertified(claim)); } /// How the reader now sees each of `claims` of `id`, in `ctx`. @@ -2080,17 +1994,4 @@ impl Router { }; claims.map(|claim| (claim.clone(), at(claim))).collect() } - - fn log(&mut self, id: Id, claim: &Path, kind: Kind, item: &A) { - #[cfg(test)] - let forgot = *self.forgot.entry((id, claim.clone())).or_insert(0); - self.events.push(Event { - id, - claim: claim.clone(), - kind, - item: item.clone(), - #[cfg(test)] - forgot, - }); - } } diff --git a/rs/moq-net/src/route/path.rs b/rs/moq-net/src/route/path.rs new file mode 100644 index 00000000000..0c318bfa13c --- /dev/null +++ b/rs/moq-net/src/route/path.rs @@ -0,0 +1,2810 @@ +//! The route rule the routes epic's Plan item 27 picked, as one node's state +//! machine: a route per publisher and prefix, believed by its own path. +//! +//! A hop names a relay and the session it learned a route on. A node uses the +//! best route whose path does not name it, at once: one crossing no draining +//! node, then the one whose lowest link priority is highest, then the cheapest. +//! Each is a property of the whole path, so no relay prefers a path its +//! neighbour ranked lower, and selection converges. A draining path is the worst +//! rank but stays usable: it is never removed. A node that hears GOAWAY names the +//! leaving node in a [`Dead::Draining`] to each neighbour it sent a route through +//! it, which passes it on the same way, and each node doubts the leaving node as +//! a closed session's far end is doubted, so no node moves onto a stale offer +//! through it and a request cannot loop through offers that predate the drain. When a session closes, +//! the node that lost it names that hop in one [`Dead::Closed`] per neighbour, which +//! lists the routes that moved to another path, and every node drops every route +//! through the hop, whoever offered it. A node with no route left holds its +//! claims, unavailable, until each neighbour that learned the route from it has +//! answered. A hold follows only the path it was built on: a node holds under its +//! next hop while that hop holds the path this node's route extended, and a stream +//! holds only the path it last sent, so holds never close a cycle. A closed +//! session casts doubt on the node at its far end, since a dead relay is reported +//! one neighbour at a time: no node moves to a route through it until a timer +//! lifts the doubt. There is no distance, seqno, route request, or echo. +//! +//! A route covers a prefix: the one its receiver asked for, or a narrower one a +//! node on its path asked for. A stream answering a prefix carries the sender's +//! routes inside it under their own prefixes, and one for the prefix itself from +//! the best usable route covering it. A node keeps one view of a +//! publisher's claims per prefix: the newest floor any believed route covering +//! that prefix gave it, kept whole while it holds a route covering the prefix. It +//! shows each claim from the narrowest view above it, available while a live +//! route covers it. +//! +//! `rs/moq-net/verify/route/RouteHandoff.tla` model-checks the rule. Nothing drives +//! this router on the wire yet: sessions run [`super::Router`] until the switch +//! (`quest/m2/cluster-routes/switch.md`). + +#![cfg_attr(not(test), allow(dead_code, reason = "only the fixtures drive it until the switch"))] + +use std::{ + cmp::Reverse, + collections::{BTreeMap, BTreeSet}, + sync::Arc, + time::Duration, +}; + +use kio::clock::Instant; + +use crate::Path; + +use super::reader::Reader; +use super::view::{ancestors, below, covering, within_set}; +use super::{Error, Event, GENERATED, Id, Key, Link, MAX, MAX_HOPS, Metric, QUALIFIED, Rev, Shown, View, within}; + +/// One node on a route's path, and the session it learned the route on. A +/// publisher names itself with session 0. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub(crate) struct Hop { + pub node: Id, + pub session: u64, +} + +/// A route's identity: its publisher, and the prefix it covers. +pub(crate) type RouteId = (Id, Path); + +/// What a route says. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct Offer { + /// Every node it crossed, the publisher first and the sender last. + pub path: Arc<[Hop]>, + /// A node on its path sent GOAWAY, which ranks it below every route that avoids + /// one, though it stays usable. + pub draining: bool, + /// The lowest local priority of a link on its path, ranked before cost; the + /// publisher's own route has no link to lower it. + pub priority: i8, + /// The cost ranked after drain and priority: the sum of configured link costs. + pub metric: Metric, + /// An upstream link is on its path, so it is never sent to one. + pub down_only: bool, + /// The round trip to the publisher: informational, never read to rank a route. + pub rtt: Option, +} + +impl Offer { + /// A received path must leave room for this node's hop. + fn extendable(&self) -> bool { + !self.path.is_empty() && self.path.len() < MAX_HOPS + } + + /// How it ranks, lower first. Extending a path never improves drain or priority + /// and strictly raises the cost, so every relay ranks it no better than the + /// relay it came from did, and selection converges. Ranking a relay's own drain + /// or link priority alone lets two relays prefer each other's paths forever. + fn rank(&self) -> (bool, Reverse, Metric) { + (self.draining, Reverse(self.priority), self.metric) + } +} + +/// A route's state on a stream. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) enum Reach { + Finite(Offer), + /// The sender has no route left but keeps the claims while one may return. + Held, +} + +/// A route that moved to another path when the session a [`Dead`] names closed. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct Moved { + pub id: Id, + pub prefix: Path, + pub offer: Offer, +} + +/// News about a node or session on routes this node offered, sent to each +/// neighbour once per hop or node while a route sent to it names that hop or node. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) enum Dead { + /// The session `hop` names closed, so every route through it is dead, whoever + /// offered it. Of the sender's own routes through it, each one `moved` lists + /// continues on its new path; the rest are ended if `gone`, and held if not. + Closed { + hop: Hop, + /// The node at the other end of that session, which the news casts doubt on. + far: Id, + moved: Vec, + gone: bool, + }, + /// `node` sent GOAWAY: every path through it ranks as draining, the worst rank, + /// and stays usable. The news casts doubt on the node, so no relay moves onto a + /// stale offer through it until the timer fires, and a request cannot loop through + /// offers that predate the drain. + Draining { node: Id }, +} + +/// A message for one neighbour. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) enum Out { + /// ROUTE_START or ROUTE_UPDATE, ROUTE_HOLD for a held reach, or ROUTE_END for `None`. + Route { + to: N, + id: Id, + prefix: Path, + reach: Option, + }, + /// Changes to a route's claims, as presented at its prefix, closed by the floor + /// they are complete at. + Batch { + to: N, + id: Id, + prefix: Path, + changes: Vec<(Path, Option)>, + floor: Option, + }, + Dead { + to: N, + dead: Dead, + }, + /// The stream answering the neighbour's interest sent its initial set. + Synced { + to: N, + }, +} + +/// When a doubt about a node lifts. +#[cfg(test)] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum Lift { + /// The rejected comparison rule: lift sooner once no route names the node. + Unnamed, + /// Only when its timer fires. + Timer, +} + +/// How a closed session's doubt works. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct Doubt { + /// How long the doubt lasts. + pub timer: Duration, + #[cfg(test)] + pub lift: Lift, +} + +impl Doubt { + /// A second, kept until the timer fires. + pub const DEFAULT: Self = Self { + timer: Duration::from_secs(1), + #[cfg(test)] + lift: Lift::Timer, + }; +} + +impl Default for Doubt { + fn default() -> Self { + Self::DEFAULT + } +} + +/// Where a request for a claim goes from a node. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) enum Forward { + /// The node publishes it. + Here, + /// To `next`, naming the route as `next` offered it. + Next { next: N, named: Id, prefix: Path }, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum State { + Live, + /// Its path holds a hop this node knows is dead, and the neighbour has not said + /// what became of it yet. + Dead, + /// The neighbour lost the route and holds its claims. + Held, +} + +/// A route a neighbour offered. +#[derive(Clone, Debug)] +struct Heard { + offer: Offer, + state: State, + /// The interest generation of the stream it came on. + interest: u32, + /// Its claims as presented at its prefix, with one floor or none yet. + view: View, +} + +/// A dead hop keeps its own deadline; another failure of the far node cannot extend it. +#[derive(Clone, Copy, Debug)] +enum Death { + Waiting { far: Id, until: Instant }, + Expired { far: Id }, +} + +impl Death { + fn far(self) -> Id { + match self { + Self::Waiting { far, .. } | Self::Expired { far } => far, + } + } + + fn deadline(self) -> Option { + match self { + Self::Waiting { until, .. } => Some(until), + Self::Expired { .. } => None, + } + } +} + +/// A node's timed doubt. Tests also represent early collection before the +/// timer retires. +#[derive(Clone, Copy, Debug)] +enum Suspicion { + Doubted(Instant), + #[cfg(test)] + Lifted(Instant), +} + +impl Suspicion { + fn deadline(self) -> Instant { + match self { + Self::Doubted(at) => at, + #[cfg(test)] + Self::Lifted(at) => at, + } + } +} + +/// What this node last sent a neighbour for a route. `src` remains after END +/// while a child may still answer, but an ended route holds no offer or claims. +#[derive(Clone, Debug)] +struct Sent { + src: Path, + state: SentState, +} + +#[derive(Clone, Debug)] +enum SentState { + Live(SentOffer), + Held(SentOffer), + Ended, +} + +#[derive(Clone, Debug)] +struct SentOffer { + offer: Offer, + /// The prefix of the view its claims were presented from. + from: Path, + view: View, +} + +impl Sent { + fn active(&self) -> Option<&SentOffer> { + match &self.state { + SentState::Live(sent) | SentState::Held(sent) => Some(sent), + SentState::Ended => None, + } + } + + fn offer(&self) -> Option<&Offer> { + self.active().map(|sent| &sent.offer) + } + + fn held(&self) -> bool { + matches!(self.state, SentState::Held(_)) + } + + fn hold(&mut self) { + if matches!(self.state, SentState::Live(_)) { + let SentState::Live(sent) = std::mem::replace(&mut self.state, SentState::Ended) else { + unreachable!() + }; + self.state = SentState::Held(sent); + } + } + + fn end(&mut self) { + self.state = SentState::Ended; + } + + fn reoffer(&mut self, offer: Offer) -> &mut SentOffer { + let sent = match std::mem::replace(&mut self.state, SentState::Ended) { + SentState::Live(mut sent) | SentState::Held(mut sent) => { + sent.offer = offer; + sent + } + SentState::Ended => SentOffer { + offer, + from: Path::empty(), + view: View::default(), + }, + }; + self.state = SentState::Live(sent); + let SentState::Live(sent) = &mut self.state else { + unreachable!() + }; + sent + } +} + +/// The route a node uses for one [`RouteId`]. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) enum Route { + Origin(Offer), + Live { + next: N, + offer: Offer, + }, + /// Live choices are installed before reflected sources move and holds resolve. + Planning { + under: Option, + /// The path of the last live route. + path: Arc<[Hop]>, + }, + /// Unavailable while `under` may still get the route back or a child has yet to + /// answer. `path` is the last live route's, so a hold follows only the path it + /// was built on: a hold under a next hop that has since sent another path could + /// close a cycle of holds that never resolves. + Held { + under: Option, + path: Arc<[Hop]>, + }, +} + +impl Route { + pub fn next(&self) -> Option { + match self { + Self::Origin(_) => None, + Self::Live { next, .. } => Some(*next), + Self::Planning { under, .. } | Self::Held { under, .. } => *under, + } + } + + pub fn offer(&self) -> Option<&Offer> { + match self { + Self::Origin(offer) | Self::Live { offer, .. } => Some(offer), + Self::Planning { .. } | Self::Held { .. } => None, + } + } + + fn live(&self) -> bool { + self.offer().is_some() + } + + /// The path of the route, or of the last live one while it plans or holds. + fn path(&self) -> &Arc<[Hop]> { + match self { + Self::Origin(offer) | Self::Live { offer, .. } => &offer.path, + Self::Planning { path, .. } | Self::Held { path, .. } => path, + } + } +} + +struct Session { + /// How this node names the session in the hops of the routes it learns there. + number: u64, + /// How the neighbour names itself in a path. + hop: Id, + link: Link, + rtt: Option, + draining: bool, + /// The newest receive stream whose initial set has arrived (or was cancelled). + /// Older streams cannot recreate a route after that boundary. + synced: u32, + /// The routes it offered, so closing it costs what it held. + heard: BTreeSet, +} + +/// Presence in `Router::answering` is the answering phase; absence is idle. +struct Answer { + watch: Path, + tx: BTreeMap>, + /// The dead hops it was told of, while a route sent to it names one. + told: BTreeSet, + /// The draining nodes it was told of, while a route sent to it names one. + told_draining: BTreeSet, +} + +struct Origin { + view: View, +} + +#[derive(Default)] +struct Dirty { + all: bool, + claims: BTreeSet, + /// Regions whose presence, availability, or source view changed. + regions: BTreeSet, +} + +/// The batch an update follows, so a view tracking its route moves by the same changes. +struct Delta { + route: RouteId, + from: N, + /// The route's floor before the batch. + was: Option, + claims: Vec, +} + +/// The changes that bring what a neighbour was sent for a route to what a node +/// presents there now, with the prefix of the view they come from and its floor. +type Presented = (Path, Vec<(Path, Option)>, Option); + +/// What a node sends one neighbour for one route. +enum Want { + Path(Offer, Path), + Hold, + End, + /// No claim meets the stream any more: what was sent there is withdrawn, at the + /// floor that says so, and then the route ends. Another route the neighbour holds + /// for the prefix may lag, and must not bring the withdrawn claims back. + Empty, + /// Whatever it sent last: the neighbour is the next hop this node holds under, + /// and waits for this node's answer. + Keep, +} + +/// News an update tells the neighbours that were sent a route naming it. +#[derive(Clone, Copy)] +enum Report { + Closed(Hop), + Draining(Id), +} + +/// Retained state, summed over the node, for budgets and leak checks. +#[cfg(test)] +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) struct Retained { + /// Claims in views of other publishers. + pub items: usize, + pub views: usize, + /// Claims on routes heard. + pub heard: usize, + /// Claims on routes sent. + pub sent: usize, + pub routes: usize, + pub dead: usize, + pub doubt: usize, + pub draining: usize, +} + +pub(crate) struct Router { + hop: Id, + doubt: Doubt, + opened: u64, + sessions: BTreeMap, + /// The sessions currently answering an interest. + answering: BTreeMap>, + origins: BTreeMap>, + rx: BTreeMap>>, + /// The heard routes whose path holds each hop. + rx_hops: BTreeMap>>, + /// The sent routes whose path holds each hop. + tx_hops: BTreeMap>, + /// For each node, the neighbours sent a route naming it, with how many of the + /// sent paths' hops name it: what DRAIN news is owed, without a scan of its hops. + tx_nodes: BTreeMap>, + /// For each of this node's own routes, the streams and prefixes it was sent under. + reflects: BTreeMap>, + /// Child replies still owed to a stream that the peer cancelled or replaced. + waiting: BTreeMap>, + routes: BTreeMap>, + views: BTreeMap>, + /// Prefixes with heard routes, selected routes, or retained views, per publisher. + scopes: BTreeMap>, + /// For a key this node qualified, the identity the neighbour named. + filed: BTreeMap<(Id, N), Id>, + /// Dead hops, each with the node at the far end of its session. + dead: BTreeMap, + /// Each node's doubt and deadline. + doubted: BTreeMap, + /// Nodes that sent GOAWAY, kept while a heard route names one or it is in doubt. + draining: BTreeSet, + watch: Path, + interest: u32, + reader: Reader, + /// Claim paths indexed by publisher, only for this rule's scoped reader updates. + shown_claims: BTreeMap>, + rng: u64, + #[cfg(test)] + touched: usize, + /// Routes, prefixes, streams, and reader claims visited, for per-change work checks. + #[cfg(test)] + work: std::cell::Cell, + out: Vec>, +} + +/// The hop-index range for every session of one node. +fn hop_range(node: Id) -> std::ops::RangeInclusive { + Hop { node, session: 0 }..=Hop { + node, + session: u64::MAX, + } +} + +/// Whether `path` names `node`. +fn names(path: &[Hop], node: Id) -> bool { + path.iter().any(|hop| hop.node == node) +} + +/// Whether the node that offered `path` learned it from `node`. +fn child(path: &[Hop], node: Id) -> bool { + path.len() >= 2 && path[path.len() - 2].node == node +} + +/// The narrower of two nested prefixes, or `None` when they are disjoint. +pub(super) fn meet(a: &Path, b: &Path) -> Option { + match (within(a, b), within(b, a)) { + (true, _) => Some(a.clone()), + (_, true) => Some(b.clone()), + _ => None, + } +} + +/// The entries of `map` under identity `id`. +fn of(map: &BTreeMap, id: Id) -> impl Iterator { + map.range((id, Path::empty())..) + .take_while(move |((key, _), _)| *key == id) +} + +impl Router { + /// A node with no sessions, naming itself `hop` in the routes it forwards and + /// minting rotated identities from `seed`. `hop` must be fresh per process: + /// a restart starts a new session namespace and must not reuse old hop names. + pub fn new(hop: Id, seed: u64, doubt: Doubt) -> Self { + Self { + hop, + doubt, + opened: 0, + sessions: BTreeMap::new(), + answering: BTreeMap::new(), + origins: BTreeMap::new(), + rx: BTreeMap::new(), + rx_hops: BTreeMap::new(), + tx_hops: BTreeMap::new(), + tx_nodes: BTreeMap::new(), + reflects: BTreeMap::new(), + waiting: BTreeMap::new(), + routes: BTreeMap::new(), + views: BTreeMap::new(), + scopes: BTreeMap::new(), + filed: BTreeMap::new(), + dead: BTreeMap::new(), + doubted: BTreeMap::new(), + draining: BTreeSet::new(), + watch: Path::empty(), + interest: 0, + reader: Reader::default(), + shown_claims: BTreeMap::new(), + rng: seed | 1, + #[cfg(test)] + touched: 0, + #[cfg(test)] + work: std::cell::Cell::new(0), + out: Vec::new(), + } + } + + /// Take the messages queued for neighbours, in order. + pub fn drain(&mut self) -> Vec> { + std::mem::take(&mut self.out) + } + + /// Take the reader events, in order. + pub fn events(&mut self) -> Vec> { + self.reader.events() + } + + /// Take the identities recomputed since the last call, each with the claims whose + /// shown state changed. + pub fn changed(&mut self) -> BTreeMap> { + self.reader.changed() + } + + // --- Sessions --- + + /// A session to `neighbor`, which names itself `hop` in a path, opened. + pub fn connect(&mut self, neighbor: N, hop: Id, link: Link) -> Result<(), Error> { + if link.cost == 0 { + return Err(Error::ZeroCost); + } + self.opened += 1; + let session = Session { + number: self.opened, + hop, + link, + rtt: None, + draining: false, + synced: 0, + heard: BTreeSet::new(), + }; + self.sessions.insert(neighbor, session); + // Nothing is sent until the neighbour asks. + Ok(()) + } + + /// The sessions to `neighbors` closed at `now`: everything learned or sent on them + /// goes, and each names its hop to the others. + pub fn disconnect(&mut self, neighbors: impl IntoIterator, now: Instant) { + let mut ids = BTreeSet::new(); + let mut hops = Vec::new(); + for neighbor in neighbors { + let Some(session) = self.sessions.remove(&neighbor) else { + continue; + }; + let answer = self.answering.remove(&neighbor); + for route in &session.heard { + ids.insert(route.0); + self.forget_heard(route, neighbor); + self.filed.remove(&(route.0, neighbor)); + self.unwait(neighbor, route); + } + for (route, sent) in answer.iter().flat_map(|answer| &answer.tx) { + ids.insert(route.0); + if let Some(offer) = sent.offer() { + self.unindex_sent(neighbor, route, &offer.path); + } + self.unreflect(neighbor, route, &sent.src); + } + let hop = Hop { + node: self.hop, + session: session.number, + }; + ids.extend(self.fail(hop, session.hop, None, Vec::new(), false, now)); + hops.push(Report::Closed(hop)); + } + self.update(ids, hops, None); + } + + /// `neighbor` asked this node for announcements under `prefix`, or for none: its + /// stream starts over, a complete snapshot closed by SYNCED. Each neighbour has + /// one interest prefix; another call replaces it rather than adding a region. + pub fn peer_interest(&mut self, neighbor: N, prefix: Option) -> Result<(), Error> { + if !self.sessions.contains_key(&neighbor) { + return Err(Error::UnknownNeighbor); + } + if let Some(old) = self.answering.remove(&neighbor) { + for (key, sent) in old.tx { + if let Some(offer) = sent.offer() { + self.unindex_sent(neighbor, &key, &offer.path); + } + if self.child_pending(key.0, neighbor, &key.1) { + self.waiting + .entry((key.0, sent.src.clone())) + .or_default() + .insert((neighbor, key.1.clone())); + } + self.unreflect(neighbor, &key, &sent.src); + } + } + let has = prefix.is_some(); + if let Some(watch) = prefix { + self.answering.insert( + neighbor, + Answer { + watch, + tx: BTreeMap::new(), + told: BTreeSet::new(), + told_draining: BTreeSet::new(), + }, + ); + } + let ids = self.ids(); + self.update(ids, Vec::new(), None); + if has { + self.out.push(Out::Synced { to: neighbor }); + } + Ok(()) + } + + /// `neighbor` sent GOAWAY at `now`. Every path through it ranks below every path + /// that avoids it, and stays usable while its session stays up. This node moves at + /// once only onto a path that avoids it, and tells the neighbours it sent a route + /// through it. + pub fn goaway(&mut self, neighbor: N, now: Instant) { + let Some(session) = self.sessions.get_mut(&neighbor) else { + return; + }; + session.draining = true; + let node = session.hop; + let mut ids = self.through(neighbor); + ids.extend(self.drains(node, now)); + self.update(ids, vec![Report::Draining(node)], None); + } + + /// The link to `neighbor` has `rtt`, or `None` unmeasured. + pub fn rtt(&mut self, neighbor: N, rtt: Option) { + let Some(session) = self.sessions.get_mut(&neighbor) else { + return; + }; + if session.rtt == rtt { + return; + } + session.rtt = rtt; + let ids = self.through(neighbor); + self.update(ids, Vec::new(), None); + } + + /// The identities `neighbor` offered a route for. + fn through(&self, neighbor: N) -> BTreeSet { + (self.sessions.get(&neighbor).into_iter()) + .flat_map(|session| session.heard.iter().map(|(id, _)| *id)) + .collect() + } + + /// Change the local interest to `prefix`, returning its new generation. + /// The driver replaces each neighbour's stream with the intersection of + /// `prefix` and its single scope region (the whole prefix if unscoped), then + /// calls `synced` after the replacement's initial set arrives. This interface + /// supports one prefix per neighbour: disjoint regions on the same neighbour + /// need the interest-set extension, including its per-prefix SYNCED. + /// Until then, old routes stay but only new claims are + /// believed. With no intersection, the driver cancels the old streams and + /// calls `synced` locally for this generation: no replacement can answer. + /// What this node knows outside `prefix` is forgotten. + pub fn watch(&mut self, prefix: Path) -> u32 { + self.interest += 1; + self.watch = prefix.clone(); + // The narrowest covering view carries the knowledge the reader already used. + let keys: Vec = self.views.keys().rev().cloned().collect(); + for key in keys { + if within(&key.1, &prefix) { + continue; + } + let view = self.views.remove(&key).expect("a view"); + self.forget(key.0, &key.1, Some(&prefix)); + if within(&prefix, &key.1) { + let narrow = (key.0, prefix.clone()); + self.scopes.entry(key.0).or_default().insert(prefix.clone()); + self.views.entry(narrow).or_insert_with(|| view.present(&prefix)); + } + } + let ids = self.ids(); + self.refresh(ids); + self.interest + } + + /// The current interest generation. + pub fn interest(&self) -> u32 { + self.interest + } + + // --- The origin's own identity --- + + /// Start originating `id`, with no claims. + pub fn originate(&mut self, id: Id) { + let mut view = View::default(); + view.floors.insert(Path::empty(), 0); + self.originate_view(id, view); + self.refresh([id].into()); + } + + /// Install an origin before updating it, so rotation retires the old identity + /// and publishes the new one's claims in one update. + fn originate_view(&mut self, id: Id, view: View) { + debug_assert_eq!(id & QUALIFIED, 0, "a declared identity is never qualified"); + self.origins.insert(id, Origin { view }); + let path: Arc<[Hop]> = Arc::new([Hop { + node: self.hop, + session: 0, + }]); + let route = Route::Origin(Offer { + path, + draining: false, + priority: i8::MAX, + metric: 0, + down_only: false, + rtt: Some(Duration::ZERO), + }); + self.routes.insert((id, Path::empty()), route); + self.scopes.entry(id).or_default().insert(Path::empty()); + } + + /// The identity this node declares. + pub fn declared(&self) -> Option { + self.origins.keys().next().copied() + } + + /// The declared identity's claim revision. + pub fn revision(&self) -> Option { + let origin = self.origins.values().next()?; + origin.view.floors.get(&Path::empty()).copied() + } + + /// Start or update a claim of the declared identity. + pub fn announce(&mut self, claim: Path, item: A) -> Result<(), Error> { + self.change(claim, Some(item)) + } + + /// Withdraw a claim of the declared identity. + pub fn unannounce(&mut self, claim: Path) -> Result<(), Error> { + self.change(claim, None) + } + + fn change(&mut self, claim: Path, item: Option) -> Result<(), Error> { + let id = self.declared().ok_or(Error::NoIdentity)?; + let rev = self.origins[&id].view.floors[&Path::empty()] + 1; + if rev == MAX { + self.rotate(id); + return self.change(claim, item); + } + let view = &mut self.origins.get_mut(&id).expect("declared").view; + view.floors.insert(Path::empty(), rev); + match item { + Some(item) => view.items.insert(claim.clone(), item), + None => view.items.remove(&claim), + }; + let dirty = Dirty { + all: false, + claims: BTreeSet::from([claim]), + ..Default::default() + }; + self.update([id].into(), Vec::new(), Some((id, dirty))); + Ok(()) + } + + /// Jump the declared identity's revision to `rev`, as a long-lived origin's claim + /// changes reach it. + #[cfg(test)] + pub fn age(&mut self, rev: Rev) { + let id = self.declared().expect("declared"); + let view = &mut self.origins.get_mut(&id).expect("declared").view; + view.floors.insert(Path::empty(), rev); + } + + /// Retire the declared identity for a fresh one, announcing its claims again there. + fn rotate(&mut self, old: Id) { + let origin = self.origins.remove(&old).expect("declared"); + self.routes.remove(&(old, Path::empty())); + let id = loop { + // xorshift64: a fresh identity within a generated hop id's range. + self.rng ^= self.rng << 13; + self.rng ^= self.rng >> 7; + self.rng ^= self.rng << 17; + let id = self.rng & GENERATED; + if id != 0 && id != old && !self.origins.contains_key(&id) { + break id; + } + }; + let rev = origin.view.items.len() as Rev; + let mut view = origin.view; + view.floors = BTreeMap::from([(Path::empty(), rev)]); + self.originate_view(id, view); + self.refresh([old, id].into()); + } + + // --- Received messages --- + + /// A route `from` sent for `named` under `prefix`, on the stream answering interest + /// generation `interest`: ROUTE_START or ROUTE_UPDATE, ROUTE_HOLD, or ROUTE_END for `None`. + pub fn route(&mut self, from: N, named: Id, prefix: Path, reach: Option, interest: u32) { + if self.sessions.get(&from).is_none_or(|session| interest < session.synced) { + return; + } + let Some(id) = self.admit(from, named, &prefix) else { + return; + }; + let key = (id, prefix); + let entry = self.rx.get(&key).and_then(|routes| routes.get(&from)); + // A message from a stream older than the route it names. + if entry.is_some_and(|entry| entry.interest > interest) { + return; + } + let same = entry.is_some_and(|entry| entry.interest == interest); + match reach { + // A path a node could not extend is refused, as if ended. + Some(Reach::Finite(offer)) if offer.extendable() => { + let view = match same { + true => self.rx[&key][&from].view.clone(), + false => View::default(), + }; + if entry.is_some() { + self.forget_heard(&key, from); + } + let state = match self.dead_in(&offer.path) { + true => State::Dead, + false => State::Live, + }; + let heard = Heard { + offer, + state, + interest, + view, + }; + self.hear(key.clone(), from, heard); + if id != named { + self.filed.insert((id, from), named); + } + } + Some(Reach::Held) if same => { + let heard = self.rx.get_mut(&key).and_then(|routes| routes.get_mut(&from)); + if let Some(heard) = heard { + heard.state = State::Held; + } + } + Some(Reach::Held) => return, + Some(Reach::Finite(_)) | None => { + if entry.is_none() || !same && reach.is_none() { + return; + } + self.forget_heard(&key, from); + } + } + self.update_route(key); + } + + /// A batch of claim changes for the route `from` sent for `named` under `prefix`, + /// closed by its floor, on the stream answering interest generation `interest`. + pub fn batch( + &mut self, + from: N, + named: Id, + prefix: Path, + changes: Vec<(Path, Option)>, + floor: Option, + interest: u32, + ) { + let Some(id) = self.admit(from, named, &prefix) else { + return; + }; + let key = (id, prefix.clone()); + let Some(heard) = self.rx.get_mut(&key).and_then(|routes| routes.get_mut(&from)) else { + return; + }; + if heard.interest != interest { + return; + } + let was = heard.view.floors.get(&prefix).copied(); + let claims: Vec = changes.iter().map(|(claim, _)| claim.clone()).collect(); + let floors = floor.map(|rev| BTreeMap::from([(prefix, rev)])).unwrap_or_default(); + heard.view.apply(changes, floors); + let delta = Delta { + route: key.clone(), + from, + was, + claims, + }; + self.run([id].into(), Vec::new(), Some(delta), BTreeMap::new(), Some(key)); + } + + /// A [`Dead`] from `from`, at `now`. + pub fn dead(&mut self, from: N, dead: Dead, now: Instant) { + if !self.sessions.contains_key(&from) { + return; + } + let (hop, far, moved, gone) = match dead { + Dead::Closed { hop, far, moved, gone } => (hop, far, moved, gone), + Dead::Draining { node } => { + // As with a closed hop, only a node on a path this peer offered counts. + let credited = (self.rx_hops.range(hop_range(node))).any(|(_, peers)| peers.contains_key(&from)); + if node == self.hop || !credited { + return; + } + // A node already draining ranks and is doubted as before: the news only + // reaches a neighbour not yet told. Each upstream relays it, so a repeat + // must not revisit every route sent through the node. + if self.draining.contains(&node) { + self.report_draining(node); + return; + } + let ids = self.drains(node, now); + self.update(ids, vec![Report::Draining(node)], None); + return; + } + }; + // A peer only reports hops it previously advertised to us. Also bind the far + // node to that path: unrelated DEADs must not allocate hop or doubt state. + let credited = self.rx_hops.get(&hop).and_then(|peers| peers.get(&from)); + if !credited.is_some_and(|keys| { + keys.iter().any(|key| { + self.rx[key][&from] + .offer + .path + .windows(2) + .any(|pair| pair[1] == hop && pair[0].node == far) + }) + }) { + return; + } + let mut filed = Vec::new(); + for moved in moved { + if let Some(id) = self.admit(from, moved.id, &moved.prefix) { + filed.push(((id, moved.prefix), moved.offer)); + } + } + let ids = self.fail(hop, far, Some(from), filed, gone, now); + self.update(ids, vec![Report::Closed(hop)], None); + } + + /// `from`'s stream answering interest generation `interest` sent its initial set: + /// what an older stream left is gone. The driver also calls this after cancelling + /// every stream to a scoped peer whose scope no longer intersects the interest. + pub fn synced(&mut self, from: N, interest: u32) { + let Some(session) = self.sessions.get_mut(&from) else { + return; + }; + session.synced = session.synced.max(interest); + let stale: Vec = (session.heard.iter()) + .filter(|key| self.rx[*key][&from].interest < interest) + .cloned() + .collect(); + let ids: BTreeSet = stale.iter().map(|(id, _)| *id).collect(); + for key in &stale { + self.forget_heard(key, from); + } + self.update(ids, Vec::new(), None); + } + + /// Fire every doubt and dead-hop timer due by `now`. + pub fn timeout(&mut self, now: Instant) { + let due: Vec = self + .doubted + .iter() + .filter_map(|(node, state)| (state.deadline() <= now).then_some(*node)) + .collect(); + let expired: Vec = self + .dead + .iter() + .filter_map(|(hop, state)| state.deadline().filter(|at| *at <= now).map(|_| *hop)) + .collect(); + if due.is_empty() && expired.is_empty() { + return; + } + let mut ids = BTreeSet::new(); + for node in due { + self.doubted.remove(&node); + ids.extend(self.naming(node)); + } + for hop in expired { + let far = self.dead[&hop].far(); + self.dead.insert(hop, Death::Expired { far }); + } + self.update(ids, Vec::new(), None); + } + + /// When the next doubt or dead-hop timer fires. + pub fn deadline(&self) -> Option { + self.doubted + .values() + .map(|state| state.deadline()) + .chain(self.dead.values().filter_map(|state| state.deadline())) + .min() + } + + /// Forget `id`, once nothing holds it. + pub fn collect(&mut self, id: Id) { + if self.holds(id) || self.origins.contains_key(&id) { + return; + } + let views: Vec = of(&self.views, id).map(|(key, _)| key.clone()).collect(); + for key in views { + self.views.remove(&key); + self.forget(id, &key.1, None); + } + self.refresh([id].into()); + } + + // --- Queries --- + + /// The route this node uses for `id` under `prefix`. + pub fn route_for(&self, id: Id, prefix: &Path) -> Option<&Route> { + self.routes.get(&(id, prefix.clone())) + } + + /// Everything this node holds, for a failure report. + #[cfg(test)] + pub fn dump(&self) -> String { + let mut out = String::new(); + out += &format!( + " dead {:?} doubt {:?} filed {:?}\n", + self.dead, self.doubted, self.filed + ); + for (peer, answer) in &self.answering { + out += &format!(" to {peer:?} watch {:?} sent {:?}\n", answer.watch, answer.tx); + } + for (key, routes) in &self.rx { + for (peer, heard) in routes { + out += &format!( + " heard {key:?} from {peer:?}: {:?} {:?} gen {} view {:?}\n", + heard.state, heard.offer.path, heard.interest, heard.view + ); + } + } + for (key, route) in &self.routes { + out += &format!(" route {key:?}: {route:?}\n"); + } + for (key, view) in &self.views { + out += &format!(" view {key:?}: {view:?}\n"); + } + out += &format!( + " interest {} watch {:?} shown {:?}", + self.interest, + self.watch, + self.reader.shown() + ); + out + } + + /// What `from` last offered for `id` under `prefix`, as this node files it. + #[cfg(test)] + pub fn heard_offer(&self, id: Id, prefix: &Path, from: N) -> Option<&Offer> { + let heard = self.rx.get(&(id, prefix.clone()))?.get(&from)?; + (heard.state == State::Live).then_some(&heard.offer) + } + + /// Every route this node uses for `id`, by prefix. + pub fn routes(&self, id: Id) -> impl Iterator)> { + of(&self.routes, id).map(|((_, prefix), route)| (prefix, route)) + } + + /// Where a request for `claim` of `id` goes: from this node's own reader when + /// `from` is `None`, else from the neighbour that sent it, naming the route this + /// node offered it under that prefix. `None` when it stops here. + /// + /// A reader's request starts on the narrowest live route covering the claim. A + /// forwarded one follows the route this node offered the neighbour it came from, + /// and never one whose path names that neighbour. + pub fn forward(&self, id: Id, claim: &Path, from: Option<(N, Path)>) -> Option> { + if self.origins.contains_key(&id) { + return Some(Forward::Here); + } + let (prefix, prev) = match from { + None => { + let prefix = ancestors(claim) + .rev() + .find(|prefix| self.routes.get(&(id, prefix.clone())).is_some_and(Route::live))?; + (prefix, None) + } + Some((prev, prefix)) => { + let watch = self.answering.get(&prev)?.watch.clone(); + (self.src(id, &prefix, prev, &watch), Some(prev)) + } + }; + let route = self.routes.get(&(id, prefix.clone()))?; + let offer = route.offer()?; + if let Some(prev) = prev + && names(&offer.path, self.sessions[&prev].hop) + { + return None; + } + let next = route.next()?; + Some(Forward::Next { + next, + named: self.name(next, id), + prefix, + }) + } + + /// The identity this node names for `id` to `next`: the one `next` sent, if this + /// node qualified it. + pub fn name(&self, next: N, id: Id) -> Id { + self.filed.get(&(id, next)).copied().unwrap_or(id) + } + + /// Whether this node holds any route for `id`, heard or its own. + pub fn holds(&self, id: Id) -> bool { + of(&self.rx, id).next().is_some() || of(&self.routes, id).next().is_some() + } + + /// What this node's reader sees. + pub fn shown(&self) -> &BTreeMap<(Id, Path), Shown> { + self.reader.shown() + } + + /// Every view this node keeps of another publisher, by route. + #[cfg(test)] + pub fn views(&self) -> impl Iterator)> { + self.views.iter() + } + + /// Every view a route heard carries. + #[cfg(test)] + pub fn heard(&self) -> impl Iterator)> { + self.rx + .iter() + .flat_map(|(key, routes)| routes.values().map(move |heard| (key, &heard.view))) + } + + /// The view of `id` this node keeps for `prefix`, or its own. + pub fn view(&self, id: Id, prefix: &Path) -> Option<&View> { + match self.origins.get(&id) { + Some(origin) => Some(&origin.view), + None => self.views.get(&(id, prefix.clone())), + } + } + + /// How many identities this node has recomputed. + #[cfg(test)] + pub fn touched(&self) -> usize { + self.touched + } + + /// How many routes, prefixes, streams, and reader claims this node has visited. + #[cfg(test)] + pub fn work(&self) -> usize { + self.work.get() + } + + /// Count one visit. + fn step(&self) { + #[cfg(test)] + self.work.set(self.work.get() + 1); + } + + #[cfg(test)] + pub fn retained(&self) -> Retained { + Retained { + items: self.views.values().map(|view| view.items.len()).sum(), + views: self.views.len(), + heard: self.heard().map(|(_, view)| view.items.len()).sum(), + sent: (self.answering.values()) + .flat_map(|answer| answer.tx.values()) + .map(|sent| sent.active().map_or(0, |sent| sent.view.items.len())) + .sum(), + routes: self + .routes + .keys() + .filter(|(id, _)| !self.origins.contains_key(id)) + .count(), + dead: self.dead.len(), + doubt: self.doubted.len(), + draining: self.draining.len(), + } + } + + // --- Internals: what is heard --- + + /// The key this node files `named` under when `from` sends it a route covering + /// `prefix`, or `None` when refused: a scoped neighbour's identities are all + /// qualified, one already qualified is refused, and so is a route outside its scope. + /// A node also ignores routes to its own identities. + fn admit(&self, from: N, named: Id, prefix: &Path) -> Option { + let session = self.sessions.get(&from)?; + let id = match &session.link.scope { + Some(scope) => { + if named & QUALIFIED != 0 || !scope.regions.iter().any(|region| within(prefix, region)) { + return None; + } + (scope.qualify)(named) | QUALIFIED + } + None => named, + }; + (!self.origins.contains_key(&id)).then_some(id) + } + + /// Whether `path` holds a hop this node knows is dead. + fn dead_in(&self, path: &[Hop]) -> bool { + path.iter().any(|hop| self.dead.contains_key(hop)) + } + + /// Whether `path` crosses a node known to have sent GOAWAY. + fn drains_path(&self, path: &[Hop]) -> bool { + path.iter().any(|hop| self.draining.contains(&hop.node)) + } + + /// Whether `path` crosses a node in doubt. + fn doubts_path(&self, path: &[Hop]) -> bool { + path.iter() + .any(|hop| matches!(self.doubted.get(&hop.node), Some(Suspicion::Doubted(_)))) + } + + fn hear(&mut self, key: RouteId, from: N, heard: Heard) { + self.scopes.entry(key.0).or_default().insert(key.1.clone()); + for hop in heard.offer.path.iter() { + self.rx_hops + .entry(*hop) + .or_default() + .entry(from) + .or_default() + .insert(key.clone()); + } + self.sessions + .get_mut(&from) + .expect("a session") + .heard + .insert(key.clone()); + self.rx.entry(key).or_default().insert(from, heard); + } + + fn forget_heard(&mut self, key: &RouteId, from: N) { + let Some(routes) = self.rx.get_mut(key) else { + return; + }; + let Some(heard) = routes.remove(&from) else { + return; + }; + if routes.is_empty() { + self.rx.remove(key); + } + for hop in heard.offer.path.iter() { + if let Some(peers) = self.rx_hops.get_mut(hop) { + if let Some(keys) = peers.get_mut(&from) { + keys.remove(key); + if keys.is_empty() { + peers.remove(&from); + } + } + if peers.is_empty() { + self.rx_hops.remove(hop); + } + } + } + if let Some(session) = self.sessions.get_mut(&from) { + session.heard.remove(key); + if !session + .heard + .range((key.0, Path::empty())..) + .next() + .is_some_and(|(id, _)| *id == key.0) + { + self.filed.remove(&(key.0, from)); + } + } + } + + /// Re-path a heard route in place, keeping its claims. + fn repath(&mut self, key: &RouteId, from: N, offer: Offer) { + let named = self.name(from, key.0); + let mut heard = self.rx[key][&from].clone(); + self.forget_heard(key, from); + heard.state = match self.dead_in(&offer.path) { + true => State::Dead, + false => State::Live, + }; + heard.offer = offer; + self.hear(key.clone(), from, heard); + if named != key.0 { + self.filed.insert((key.0, from), named); + } + } + + /// The identities with a heard route naming `node`. + fn naming(&self, node: Id) -> BTreeSet { + (self.rx_hops.range(hop_range(node))) + .flat_map(|(_, peers)| peers.values().flatten().map(|(id, _)| *id)) + .collect() + } + + /// Whether any heard route names `node`. + fn named(&self, node: Id) -> bool { + self.rx_hops.range(hop_range(node)).next().is_some() + } + + /// `node` sent GOAWAY, as of `now`. The first word of it casts doubt on the node, + /// so no stale offer through it is taken before the timer fires. Returns the + /// identities whose routes rank anew. + fn drains(&mut self, node: Id, now: Instant) -> BTreeSet { + if !self.draining.insert(node) { + return BTreeSet::new(); + } + let at = now.saturating_add(self.doubt.timer); + let at = self.doubted.get(&node).map_or(at, |state| state.deadline().max(at)); + self.doubted.insert(node, Suspicion::Doubted(at)); + self.naming(node) + } + + /// The session `hop` names closed. `from` reported it, with what became of its own + /// routes through it, or this node saw it close. It casts doubt on `far`. Returns + /// the identities touched. + fn fail( + &mut self, + hop: Hop, + far: Id, + from: Option, + moved: Vec<(RouteId, Offer)>, + gone: bool, + now: Instant, + ) -> BTreeSet { + let mut ids = BTreeSet::new(); + let moved: BTreeMap = moved.into_iter().collect(); + let through: Vec<(RouteId, N)> = self + .rx_hops + .get(&hop) + .into_iter() + .flatten() + .flat_map(|(peer, keys)| keys.iter().map(|key| (key.clone(), *peer))) + .collect(); + // The hop is dead before any route moves, so a moved path through it stays dead. + self.dead.entry(hop).or_insert(Death::Waiting { + far, + until: now.saturating_add(self.doubt.timer), + }); + for (key, peer) in through { + self.step(); + ids.insert(key.0); + let reported = from == Some(peer); + if let Some(offer) = moved.get(&key).filter(|_| reported) { + if offer.extendable() { + self.repath(&key, peer, offer.clone()); + } else { + self.forget_heard(&key, peer); + } + continue; + } + if reported { + if gone { + self.forget_heard(&key, peer); + } else { + let is_child = self.is_child(key.0, peer, &self.rx[&key][&peer].offer.path); + let heard = self + .rx + .get_mut(&key) + .and_then(|routes| routes.get_mut(&peer)) + .expect("heard"); + // A neighbour that holds under this node answers once those behind it have. + if !is_child { + heard.state = State::Held; + } + } + continue; + } + let heard = self + .rx + .get_mut(&key) + .and_then(|routes| routes.get_mut(&peer)) + .expect("heard"); + if heard.state == State::Live { + heard.state = State::Dead; + } + } + if far != self.hop { + let at = now.saturating_add(self.doubt.timer); + let at = self.doubted.get(&far).map_or(at, |state| state.deadline().max(at)); + self.doubted.insert(far, Suspicion::Doubted(at)); + // Any route through that node may have become unusable. + ids.extend(self.naming(far)); + } + ids + } + + // --- Internals: selection --- + + /// Every identity this node keeps any state for. + fn ids(&self) -> BTreeSet { + let mut ids: BTreeSet = self.rx.keys().map(|(id, _)| *id).collect(); + ids.extend(self.routes.keys().map(|(id, _)| *id)); + ids.extend(self.views.keys().map(|(id, _)| *id)); + ids.extend(self.shown_claims.keys().copied()); + for answer in self.answering.values() { + ids.extend(answer.tx.keys().map(|(id, _)| *id)); + } + ids + } + + /// Retained prefixes of `id`, intersecting `region` when supplied. + fn prefixes(&self, id: Id, region: Option<&Path>) -> BTreeSet { + let Some(scopes) = self.scopes.get(&id) else { + return BTreeSet::new(); + }; + let Some(region) = region else { return scopes.clone() }; + ancestors(region) + .filter(|prefix| scopes.contains(prefix)) + .chain(within_set(scopes, region).cloned()) + .collect() + } + + /// Pick the best-ranked usable live route ([`Offer::rank`]), staying on the next + /// hop among equals. + /// Doubt keeps this node from moving to a route, not on the one it uses. + fn select(&self, key: &RouteId) -> Option> { + let was = self.routes.get(key); + let current = was.and_then(Route::next); + let using = current.filter(|_| was.is_some_and(Route::live)); + type Rank = ((bool, Reverse, Metric), bool, N); + let mut best: Option<(Rank, Offer)> = None; + for (&peer, heard) in self.rx.get(key).into_iter().flatten() { + self.step(); + if heard.state != State::Live || names(&heard.offer.path, self.hop) { + continue; + } + let session = &self.sessions[&peer]; + let Some(metric) = heard.offer.metric.checked_add(session.link.cost) else { + continue; + }; + if Some(peer) != using && self.doubts_path(&heard.offer.path) { + // It may come back once the doubt lifts. + continue; + } + let draining = session.draining || heard.offer.draining || self.drains_path(&heard.offer.path); + let priority = session.link.priority.min(heard.offer.priority); + let rank = ((draining, Reverse(priority), metric), Some(peer) != current, peer); + if best.as_ref().is_some_and(|(best, _)| *best <= rank) { + continue; + } + let path: Arc<[Hop]> = (heard.offer.path.iter().copied()) + .chain([Hop { + node: self.hop, + session: session.number, + }]) + .collect(); + let offer = Offer { + path, + draining, + priority, + metric, + down_only: session.link.upstream || heard.offer.down_only, + rtt: heard.offer.rtt.zip(session.rtt).and_then(|(a, b)| a.checked_add(b)), + }; + best = Some((rank, offer)); + } + best.map(|((.., next), offer)| Route::Live { next, offer }) + } + + /// With no live route, keep what an upstream or a child has yet to answer. + fn retain(&self, key: &RouteId) -> Option> { + // Only a node that had the route holds it. + let was = self.routes.get(key)?; + let path = was.path().clone(); + // Its next hop holds, or has yet to say what became of a dead route, on the + // path this node's route was built on. A next hop that has since sent another + // path, perhaps one through this node, may wait on this node instead: were + // every hold in a cycle on the path it was built on, each path would extend + // itself. + let under = was.next().filter(|next| { + (self.rx.get(key).and_then(|routes| routes.get(next))).is_some_and(|heard| { + heard.state != State::Live && path.split_last().is_some_and(|(_, on)| *on == *heard.offer.path) + }) + }); + if under.is_none() && !self.hopes(key) && !self.pending(key) { + return None; + } + Some(Route::Held { under, path }) + } + + /// Whether a neighbour that learned this node's route for `key` has yet to answer + /// its loss. + fn pending(&self, key: &RouteId) -> bool { + self.reflects + .get(key) + .into_iter() + .chain(self.waiting.get(key)) + .flatten() + .any(|(peer, prefix)| { + self.step(); + self.child_pending(key.0, *peer, prefix) + }) + } + + fn child_pending(&self, id: Id, peer: N, prefix: &Path) -> bool { + self.rx + .get(&(id, prefix.clone())) + .and_then(|routes| routes.get(&peer)) + .is_some_and(|heard| heard.state != State::Held && self.is_child(id, peer, &heard.offer.path)) + } + + /// A new answer under this key replaces what the cancelled stream reflected. + fn unwait(&mut self, peer: N, key: &RouteId) { + for src in ancestors(&key.1) { + let own = (key.0, src); + if let Some(waiting) = self.waiting.get_mut(&own) { + waiting.remove(&(peer, key.1.clone())); + if waiting.is_empty() { + self.waiting.remove(&own); + } + } + } + } + + fn collect_waiting(&mut self, ids: &BTreeSet) { + let answered: Vec<_> = ids + .iter() + .flat_map(|id| of(&self.waiting, *id)) + .flat_map(|(key, waiting)| waiting.iter().map(move |(peer, prefix)| (key, peer, prefix))) + .filter(|(key, peer, prefix)| { + self.step(); + !self.child_pending(key.0, **peer, prefix) + }) + .map(|(key, peer, prefix)| (key.clone(), *peer, prefix.clone())) + .collect(); + for (key, peer, prefix) in answered { + let waiting = self.waiting.get_mut(&key).expect("a wait"); + waiting.remove(&(peer, prefix)); + if waiting.is_empty() { + self.waiting.remove(&key); + } + } + } + + /// A returned raw key that we qualify is not an answer about the qualified key + /// we sent: the peer may have learned that raw route from us under another key. + fn is_child(&self, id: Id, peer: N, path: &[Hop]) -> bool { + self.name(peer, id) == id && child(path, self.hop) + } + + /// Whether a neighbour offers a route for `key` that only doubt keeps this node off. + fn hopes(&self, key: &RouteId) -> bool { + (self.rx.get(key).into_iter().flatten()).any(|(peer, heard)| { + heard.state == State::Live + && !names(&heard.offer.path, self.hop) + && heard.offer.metric.checked_add(self.sessions[peer].link.cost).is_some() + && self.doubts_path(&heard.offer.path) + }) + } + + /// The prefix of this node's own route that what it offers a stream answering + /// `watch` under `prefix` reflects: the route itself inside the interest, and at the + /// interest the best-ranked live route covering it that does not name the + /// neighbour. Prefer the narrower scope among equal ranks. Without a usable + /// path, reflect a live or held route so a child can answer its loss. Preferring + /// a narrower route regardless of cost can make covering paths oscillate. + fn src(&self, id: Id, prefix: &Path, peer: N, watch: &Path) -> Path { + if prefix != watch { + return prefix.clone(); + } + let covering = || ancestors(prefix).rev().map(|above| (id, above)); + let hop = self.sessions[&peer].hop; + let best = |usable| { + covering() + .filter_map(|key| { + let route = self.routes.get(&key)?; + let offer = route.offer()?; + if self.sessions[&peer].link.upstream && offer.down_only || usable && names(&offer.path, hop) { + return None; + } + Some((offer.rank(), key)) + }) + .min_by_key(|(rank, _)| *rank) + .map(|(_, key)| key) + }; + best(true) + .or_else(|| best(false)) + .or_else(|| covering().find(|key| self.routes.get(key).is_some_and(|route| !route.live()))) + .or_else(|| covering().find(|key| self.routes.contains_key(key))) + .map_or_else(|| prefix.clone(), |(_, prefix)| prefix) + } + + // --- Internals: views --- + + /// Whether this node holds a route of `id` covering `prefix`. + fn presence(&self, id: Id, prefix: &Path) -> bool { + ancestors(prefix).any(|above| self.routes.contains_key(&(id, above))) + } + + /// The narrowest view of `id` strictly above `prefix` that knows something: one a + /// route has given a floor. A view with none yet stands for nothing. + fn above(&self, id: Id, prefix: &Path) -> Option<(&Path, &View)> { + ancestors(prefix) + .rev() + .filter(|above| above != prefix) + .find_map(|above| { + (self.views.get_key_value(&(id, above))).filter(|((_, above), view)| view.floors.contains_key(above)) + }) + .map(|((_, prefix), view)| (prefix, view)) + } + + /// The narrowest view of `id` covering `prefix`, or the publisher's own. + fn covering_view(&self, id: Id, prefix: &Path) -> Option<(Path, &View)> { + if let Some(origin) = self.origins.get(&id) { + return Some((Path::empty(), &origin.view)); + } + ancestors(prefix) + .rev() + .find_map(|above| self.views.get_key_value(&(id, above))) + .map(|((_, prefix), view)| (prefix.clone(), view)) + } + + /// Whether a route this node heard is believed for its claims: live, its path does + /// not name this node, and its stream answers the current interest. + fn believed(&self, heard: &Heard) -> bool { + heard.state == State::Live && !names(&heard.offer.path, self.hop) && heard.interest == self.interest + } + + /// Bring every view of `id` up to date: each prefix within the interest that a route + /// covers keeps the newest floor a believed route covering it gave, kept whole while + /// a route covers it. Returns whether the views changed shape, and marks changed + /// claims in `dirty`. + fn know(&mut self, id: Id, keys: &BTreeSet, delta: Option<&Delta>, dirty: &mut Dirty) { + if self.origins.contains_key(&id) { + return; + } + let was: Vec<(Path, Option)> = keys + .iter() + .filter_map(|prefix| { + let view = self.views.get(&(id, prefix.clone()))?; + Some((prefix.clone(), view.floors.get(prefix).copied())) + }) + .collect(); + for prefix in keys { + let key = (id, prefix.clone()); + if !self.presence(id, prefix) { + if self.views.remove(&key).is_some() { + self.forget(id, prefix, None); + dirty.regions.insert(prefix.clone()); + } + continue; + } + // Taken out while it is brought up to date, and always put back. + let base = match self.views.remove(&key) { + Some(view) => view, + None => { + dirty.regions.insert(prefix.clone()); + self.above(id, prefix) + .map(|(_, view)| view.present(prefix)) + .unwrap_or_default() + } + }; + self.scopes.entry(id).or_default().insert(prefix.clone()); + let floor = base.floors.get(prefix).copied(); + // The newest floor any believed route covering this prefix gave. + let mut best: Option<(Rev, RouteId, N)> = None; + for above in ancestors(prefix) { + let route = (id, above); + for (&peer, heard) in self.rx.get(&route).into_iter().flatten() { + self.step(); + let Some(&rev) = heard.view.floors.get(&route.1) else { + continue; + }; + if !self.believed(heard) || Some(rev) <= floor { + continue; + } + if best.as_ref().is_none_or(|(best, ..)| rev > *best) { + best = Some((rev, route.clone(), peer)); + } + } + } + let Some((_, route, peer)) = best else { + self.views.insert(key, base); + continue; + }; + let heard = &self.rx[&route][&peer].view; + let follows = delta.filter(|delta| delta.route == route && delta.from == peer && delta.was == floor); + let view = match follows { + Some(delta) => { + let mut view = base; + view.represent(heard, std::slice::from_ref(prefix), &delta.claims); + #[cfg(test)] + assert_eq!(view, heard.present(prefix), "incremental view {key:?}"); + dirty + .claims + .extend(delta.claims.iter().filter(|claim| within(claim, prefix)).cloned()); + dirty.claims.insert(prefix.clone()); + view + } + None => { + let view = heard.present(prefix); + for (claim, _) in view.diff(&base) { + dirty.claims.insert(claim); + } + view + } + }; + if floor.is_some() != view.floors.contains_key(prefix) { + dirty.regions.insert(prefix.clone()); + } + self.views.insert(key, view); + } + // Which view the reader sees a claim at a view's own prefix from turns on floors. + let floors: Vec<(Path, Option)> = keys + .iter() + .filter_map(|prefix| { + let view = self.views.get(&(id, prefix.clone()))?; + Some((prefix.clone(), view.floors.get(prefix).copied())) + }) + .collect(); + if floors != was { + dirty.claims.extend(keys.iter().cloned()); + dirty.claims.extend(floors.into_iter().map(|(prefix, _)| prefix)); + } + // A view no route keys exactly that says what the one above it says is not kept. + for prefix in keys.iter().rev() { + let key = (id, prefix.clone()); + if self.routes.contains_key(&key) { + continue; + } + let Some(view) = self.views.get(&key) else { + continue; + }; + let floor = view.floors.get(prefix); + if self + .above(id, prefix) + .is_some_and(|(above, wider)| wider.floors.get(above) == floor) + { + self.views.remove(&key); + dirty.regions.insert(prefix.clone()); + } + } + } + + /// Count a forget against each claim of `id` this node has shown within `prefix`, + /// outside `kept`. + fn forget(&mut self, id: Id, prefix: &Path, kept: Option<&Path>) { + let views = &self.views; + (self.reader).forget(id, |claim| { + !within(claim, prefix) + || kept.is_some_and(|kept| within(claim, kept)) + || of(views, id).any(|((_, prefix), view)| within(claim, prefix) && view.floors.contains_key(prefix)) + }); + } + + // --- Internals: sending --- + + /// What this node sends `peer`, whose stream answers `watch`, for `id` under `prefix`. + fn want(&self, id: Id, prefix: &Path, peer: N, watch: &Path) -> Want { + let session = &self.sessions[&peer]; + if !session.link.proxy && !self.origins.contains_key(&id) { + return Want::End; + } + let src = self.src(id, prefix, peer, watch); + let Some(route) = self.routes.get(&(id, src.clone())) else { + return Want::End; + }; + // A route qualified here is a different key from the one this peer + // offered. Returning its own path under the qualified key would look like + // a child of another route. The peer cannot use that path anyway. + if self.name(peer, id) != id + && (route.next() == Some(peer) + || route + .offer() + .is_some_and(|offer| names(&offer.path, self.sessions[&peer].hop))) + { + return Want::End; + } + let want = match route.offer() { + // A route goes to the node that published it no more than to an upstream link + // when down-only: neither would use it. + Some(offer) if session.link.upstream && offer.down_only => Want::End, + Some(offer) if offer.path[0].node == session.hop => Want::End, + Some(offer) => Want::Path(offer.clone(), src.clone()), + // The next hop it holds under hears nothing until those behind this node answered. + None if route.next() == Some(peer) => { + let key = (id, src.clone()); + match self.pending(&key) || self.hopes(&key) { + true => return Want::Keep, + false => Want::End, + } + } + // A hold names no path, so a stream holds only the path it last sent: one now + // answered by a held route on another path ends, or the neighbour would hold + // under a path this node no longer has, and holds could close a cycle. + None => { + let sent = (self.answering.get(&peer)).and_then(|answer| answer.tx.get(&(id, prefix.clone()))); + match sent.and_then(Sent::offer) { + Some(sent) if sent.path != *route.path() => Want::End, + _ => Want::Hold, + } + } + }; + // A route goes to a stream only while a claim meets it. + match (self.meets(id, prefix), want) { + (true, want) => want, + (false, Want::Path(..)) => Want::Empty, + (false, _) => Want::End, + } + } + + /// Whether the claims this node presents for `id` at `prefix` hold anything. + fn meets(&self, id: Id, prefix: &Path) -> bool { + let Some((_, view)) = self.covering_view(id, prefix) else { + return false; + }; + below(&view.items, prefix).next().is_some() + || covering(&view.floors, prefix, |claim| view.items.get(claim)).is_some() + } + + /// The prefixes a stream answering `watch` carries for `id` to `peer`: the interest + /// itself, each route inside it, and each it was sent and may need to end. + fn stream(&self, id: Id, peer: N, watch: &Path, region: Option<&Path>) -> BTreeSet { + let session = &self.sessions[&peer]; + let answer = &self.answering[&peer]; + let sent_only = !session.link.proxy && !self.origins.contains_key(&id); + if sent_only && region.is_none() { + // A non-proxy peer is sent only what it was already sent, so skip scanning every route. + return of(&answer.tx, id).map(|((_, prefix), _)| prefix.clone()).collect(); + } + let mut out = BTreeSet::new(); + if region.is_none_or(|region| meet(watch, region).is_some()) { + out.insert(watch.clone()); + } + if let Some(region) = region { + for prefix in self.prefixes(id, Some(region)) { + let key = (id, prefix.clone()); + if self.routes.contains_key(&key) && within(&prefix, watch) || answer.tx.contains_key(&key) { + out.insert(prefix); + } + } + } else { + out.extend( + of(&self.routes, id) + .map(|((_, prefix), _)| prefix) + .filter(|prefix| within(prefix, watch)) + .cloned(), + ); + out.extend(of(&answer.tx, id).map(|((_, prefix), _)| prefix.clone())); + } + if sent_only { + out.retain(|prefix| answer.tx.contains_key(&(id, prefix.clone()))); + } + out + } + + /// Tell each neighbour that was sent a route through `hop` what became of those + /// routes, in one [`Dead`] where it is the first word on that hop. + fn report(&mut self, hop: Hop) { + let peers: Vec = self.sessions.keys().copied().collect(); + for peer in peers { + let lo = (peer, (0, Path::empty())); + let through: Vec = (self.tx_hops.get(&hop).into_iter()) + .flat_map(|set| set.range(lo.clone()..)) + .take_while(|(to, _)| *to == peer) + .map(|(_, key)| key.clone()) + .collect(); + if through.is_empty() { + continue; + } + let Some(answer) = self.answering.get(&peer) else { + continue; + }; + let watch = answer.watch.clone(); + let first = !self.answering[&peer].told.contains(&hop); + let (mut moved, mut holds, mut parents, mut ends, mut rest) = + (Vec::new(), Vec::new(), Vec::new(), Vec::new(), 0); + for key in &through { + self.step(); + let want = self.want(key.0, &key.1, peer, &watch); + let sent = &self.answering[&peer].tx[key]; + match want { + Want::Path(offer, src) => { + let old = sent.offer().cloned().expect("sent"); + let was = sent.src.clone(); + self.unindex_sent(peer, key, &old.path); + self.index_sent(peer, key, &offer.path); + self.unreflect(peer, key, &was); + self.reflect(peer, key, &src); + let sent = self + .answering + .get_mut(&peer) + .expect("answering") + .tx + .get_mut(key) + .expect("sent"); + sent.reoffer(offer.clone()); + sent.src = src; + moved.push(Moved { + id: key.0, + prefix: key.1.clone(), + offer, + }); + } + Want::Hold => { + if !sent.held() { + // The neighbour it learned the route from takes silence to mean + // this node holds under it, so it is told outright. + let parent = child(&sent.offer().expect("sent").path, self.sessions[&peer].hop); + holds.push(key.clone()); + if parent { + parents.push(key.clone()); + } + let sent = self + .answering + .get_mut(&peer) + .expect("answering") + .tx + .get_mut(key) + .expect("sent"); + sent.hold(); + } + rest += 1; + } + Want::Keep => rest += 1, + end @ (Want::End | Want::Empty) => { + let batch = + (matches!(end, Want::Empty) && self.covering_view(key.0, &key.1).is_some()).then(|| { + self.present( + key.0, + peer, + &key.1, + &Dirty { + all: true, + ..Default::default() + }, + ) + }); + self.end_sent(peer, key, batch); + ends.push(key.clone()); + } + } + } + let gone = rest == 0; + let route = |key: &RouteId, reach| Out::Route { + to: peer, + id: key.0, + prefix: key.1.clone(), + reach, + }; + if first || (gone && !ends.is_empty()) { + let far = self.dead.get(&hop).expect("a dead hop").far(); + self.out.push(Out::Dead { + to: peer, + dead: Dead::Closed { hop, far, moved, gone }, + }); + if !gone { + for key in &parents { + let out = route(key, Some(Reach::Held)); + self.out.push(out); + } + for key in &ends { + let out = route(key, None); + self.out.push(out); + } + } + } else { + for moved in moved { + let key = (moved.id, moved.prefix); + let out = route(&key, Some(Reach::Finite(moved.offer))); + self.out.push(out); + } + for key in &holds { + let out = route(key, Some(Reach::Held)); + self.out.push(out); + } + for key in &ends { + let out = route(key, None); + self.out.push(out); + } + } + self.answering.get_mut(&peer).expect("answering").told.insert(hop); + } + } + + /// Tell each neighbour that was sent a route through draining `node`, once while + /// such a route stays sent. It goes before the routes that re-rank, so the + /// neighbour doubts the node before it hears them. + fn report_draining(&mut self, node: Id) { + let peers: Vec = self + .tx_nodes + .get(&node) + .into_iter() + .flatten() + .map(|(peer, _)| *peer) + .collect(); + for peer in peers { + self.step(); + let answer = self.answering.get_mut(&peer).expect("answering"); + if answer.told_draining.insert(node) { + self.out.push(Out::Dead { + to: peer, + dead: Dead::Draining { node }, + }); + } + } + } + + /// What this node sent `peer` under `key` reflects its own route under `src`. + fn reflect(&mut self, peer: N, key: &RouteId, src: &Path) { + let own = (key.0, src.clone()); + self.reflects.entry(own).or_default().insert((peer, key.1.clone())); + } + + fn unreflect(&mut self, peer: N, key: &RouteId, src: &Path) { + let own = (key.0, src.clone()); + if let Some(set) = self.reflects.get_mut(&own) { + set.remove(&(peer, key.1.clone())); + if set.is_empty() { + self.reflects.remove(&own); + } + } + } + + fn index_sent(&mut self, peer: N, key: &RouteId, path: &[Hop]) { + for hop in path { + self.tx_hops.entry(*hop).or_default().insert((peer, key.clone())); + *self.tx_nodes.entry(hop.node).or_default().entry(peer).or_default() += 1; + } + } + + fn unindex_sent(&mut self, peer: N, key: &RouteId, path: &[Hop]) { + for hop in path { + if let Some(set) = self.tx_hops.get_mut(hop) { + set.remove(&(peer, key.clone())); + if set.is_empty() { + self.tx_hops.remove(hop); + } + } + if let Some(peers) = self.tx_nodes.get_mut(&hop.node) { + if let Some(count) = peers.get_mut(&peer) { + *count -= 1; + if *count == 0 { + peers.remove(&peer); + } + } + if peers.is_empty() { + self.tx_nodes.remove(&hop.node); + } + } + } + } + + /// Bring what each watching neighbour was sent for `id` up to date, with the `dirty` + /// claims changed since. + fn advertise(&mut self, id: Id, dirty: &Dirty, region: Option<&Path>) { + let peers: Vec<(N, Path)> = self + .answering + .iter() + .map(|(peer, answer)| (*peer, answer.watch.clone())) + .collect(); + // Plan every stream against the state as it stands, then send. + let mut plans = Vec::new(); + let mut planned: BTreeMap> = BTreeMap::new(); + for (peer, watch) in &peers { + for prefix in self.stream(id, *peer, watch, region) { + planned.entry(*peer).or_default().insert(prefix.clone()); + self.step(); + let want = self.want(id, &prefix, *peer, watch); + let on = + (self.answering[peer].tx.get(&(id, prefix.clone()))).is_some_and(|sent| sent.offer().is_some()); + let batch = match &want { + Want::Path(..) => Some(self.present(id, *peer, &prefix, dirty)), + Want::Empty if on && self.covering_view(id, &prefix).is_some() => { + Some(self.present(id, *peer, &prefix, dirty)) + } + _ => None, + }; + plans.push((*peer, prefix, want, batch)); + } + } + for (peer, prefix, want, batch) in plans { + let key = (id, prefix.clone()); + let sent = self.answering[&peer].tx.get(&key); + let on = sent.is_some_and(|sent| sent.offer().is_some()); + match want { + Want::Keep => {} + Want::Hold => { + if on && !sent.expect("sent").held() { + self.answering + .get_mut(&peer) + .expect("answering") + .tx + .get_mut(&key) + .expect("sent") + .hold(); + self.out.push(Out::Route { + to: peer, + id, + prefix, + reach: Some(Reach::Held), + }); + } + } + Want::End | Want::Empty => { + if on { + self.end_sent(peer, &key, batch); + self.out.push(Out::Route { + to: peer, + id, + prefix, + reach: None, + }); + } + } + Want::Path(offer, src) => { + let (from, changes, floor) = batch.expect("a batch"); + let reflected = sent.map(|sent| sent.src.clone()); + let reoffer = sent.is_none_or(|sent| sent.held() || sent.offer() != Some(&offer)); + if reoffer { + if let Some(old) = sent.and_then(|sent| sent.offer().cloned()) { + self.unindex_sent(peer, &key, &old.path); + } + self.index_sent(peer, &key, &offer.path); + self.out.push(Out::Route { + to: peer, + id, + prefix: prefix.clone(), + reach: Some(Reach::Finite(offer.clone())), + }); + } + if let Some(was) = reflected { + self.unreflect(peer, &key, &was); + } + self.reflect(peer, &key, &src); + self.unwait(peer, &key); + let answer = self.answering.get_mut(&peer).expect("answering"); + let sent = answer.tx.entry(key).or_insert_with(|| Sent { + src: src.clone(), + state: SentState::Ended, + }); + sent.src = src; + let sent = sent.reoffer(offer); + sent.from = from; + // Even unchanged claims need their newer floor forwarded: a neighbour + // may still hold an intervening revision learned on another path. + let advanced = floor.is_some() && sent.view.floors.get(&prefix).copied() != floor; + if !changes.is_empty() || advanced { + let floors = floor + .map(|rev| BTreeMap::from([(prefix.clone(), rev)])) + .unwrap_or_default(); + sent.view.apply(changes.clone(), floors); + self.out.push(Out::Batch { + to: peer, + id, + prefix, + changes, + floor, + }); + } + } + } + } + // Only entries visited above can have ended in this update. + for (peer, prefixes) in planned { + for prefix in prefixes { + let key = (id, prefix); + let Some(sent) = self.answering[&peer].tx.get(&key) else { + continue; + }; + if sent.offer().is_some() + || self + .rx + .get(&key) + .and_then(|routes| routes.get(&peer)) + .is_some_and(|heard| heard.state != State::Held && self.is_child(id, peer, &heard.offer.path)) + { + continue; + } + let src = sent.src.clone(); + self.unreflect(peer, &key, &src); + self.answering.get_mut(&peer).expect("answering").tx.remove(&key); + } + } + } + + /// Send a final withdrawing batch before retiring a sent offer. + fn end_sent(&mut self, peer: N, key: &RouteId, batch: Option>) { + if let Some((_, changes, floor)) = batch + && (!changes.is_empty() || floor.is_some()) + { + self.out.push(Out::Batch { + to: peer, + id: key.0, + prefix: key.1.clone(), + changes, + floor, + }); + } + let old = self.answering[&peer].tx[key].offer().cloned().expect("sent"); + self.unindex_sent(peer, key, &old.path); + self.answering + .get_mut(&peer) + .expect("answering") + .tx + .get_mut(key) + .expect("sent") + .end(); + } + + /// The changes that bring what `peer` was sent for `id` under `prefix` to what this + /// node presents there now, with the view they come from and its floor there. + fn present(&self, id: Id, peer: N, prefix: &Path, dirty: &Dirty) -> Presented { + let (from, view) = self.covering_view(id, prefix).expect("a route that meets"); + let floor = view.floors.get(&from).copied(); + let sent = self.answering[&peer].tx.get(&(id, prefix.clone())); + let fresh = sent.is_none_or(|sent| sent.active().is_none_or(|sent| sent.from != from)); + let changes: Vec<(Path, Option)> = match (sent, dirty.all || fresh) { + (Some(sent), false) => { + let old = &sent.active().expect("offered").view; + let mut claims: BTreeSet<&Path> = dirty.claims.iter().collect(); + // A held stream may have missed withdrawals the local reader already + // consumed. When its region changes, include its own previous claims. + for region in &dirty.regions { + if let Some(region) = meet(prefix, region) { + claims.extend(below(&old.items, ®ion).map(|(claim, _)| claim)); + } + } + let mut changes: Vec<(&Path, Option<&A>)> = claims + .into_iter() + .filter(|claim| within(claim, prefix) && *claim != prefix) + .map(|claim| (claim, view.items.get(claim))) + .collect(); + changes.push((prefix, covering(&view.floors, prefix, |claim| view.items.get(claim)))); + changes.retain(|(claim, now)| old.items.get(*claim) != *now); + // Starts and changes, then ends, as a whole snapshot's diff orders them. + changes.sort_by_key(|(_, now)| now.is_none()); + (changes.into_iter()) + .map(|(claim, now)| (claim.clone(), now.cloned())) + .collect() + } + (sent, _) => { + let empty = View::default(); + let old = match sent { + Some(sent) if sent.offer().is_some() => &sent.active().expect("offered").view, + _ => &empty, + }; + view.present(prefix).diff(old) + } + }; + #[cfg(test)] + { + let mut check = (sent.filter(|sent| sent.offer().is_some())) + .map(|sent| sent.active().expect("offered").view.clone()) + .unwrap_or_default(); + let floors = floor + .map(|rev| BTreeMap::from([(prefix.clone(), rev)])) + .unwrap_or_default(); + check.apply(changes.clone(), floors); + assert_eq!( + check, + view.present(prefix), + "incremental presentation of {id} {prefix:?} to {peer:?}" + ); + } + (from, changes, floor) + } + + // --- Internals: the update --- + + /// Recompute each of `ids`. + fn update(&mut self, ids: BTreeSet, reports: Vec, claims: Option<(Id, Dirty)>) { + let mut dirty = BTreeMap::new(); + if let Some((id, claims)) = claims { + dirty.insert(id, claims); + } + self.run(ids, reports, None, dirty, None); + } + + fn update_route(&mut self, key: RouteId) { + self.run([key.0].into(), Vec::new(), None, BTreeMap::new(), Some(key)); + } + + /// A new interest or an explicit collection can affect every shown claim. + fn refresh(&mut self, ids: BTreeSet) { + let dirty = ids + .iter() + .map(|id| { + ( + *id, + Dirty { + all: true, + ..Default::default() + }, + ) + }) + .collect(); + self.run(ids, Vec::new(), None, dirty, None); + } + + /// Select, learn, report each dead hop, advertise, collect, and show: the model's + /// `Update` for the identities `ids`. + fn run( + &mut self, + mut ids: BTreeSet, + reports: Vec, + delta: Option>, + mut dirty: BTreeMap, + scope: Option, + ) { + for report in &reports { + match report { + Report::Closed(hop) => { + ids.extend(self.tx_hops.get(hop).into_iter().flatten().map(|(_, (id, _))| *id)); + } + Report::Draining(node) => ids.extend( + (self.tx_hops.range(hop_range(*node))) + .flat_map(|(_, set)| set) + .map(|(_, (id, _))| *id), + ), + } + } + #[cfg(test)] + { + self.touched += ids.len(); + } + let mut affected: BTreeMap> = ids + .iter() + .map(|id| { + let region = scope.as_ref().filter(|(key, _)| key == id).map(|(_, prefix)| prefix); + (*id, self.prefixes(*id, region)) + }) + .collect(); + // A claim batch changes no heard path or state, so it answers no wait. + if delta.is_none() { + self.collect_waiting(&ids); + } + let mut choices = Vec::new(); + let mut before = BTreeMap::new(); + for (&id, prefixes) in &affected { + if self.origins.contains_key(&id) { + continue; + } + for prefix in prefixes { + let key = (id, prefix.clone()); + before.insert(key.clone(), self.routes.get(&key).map(Route::live)); + choices.push((key.clone(), self.select(&key))); + } + } + let mut held = Vec::new(); + let mut route_changed = false; + for (key, live) in choices { + route_changed |= self.routes.get(&key) != live.as_ref(); + match live { + Some(route) => { + self.routes.insert(key, route); + } + None => { + if let Some(was) = self.routes.get_mut(&key) { + *was = Route::Planning { + under: was.next(), + path: was.path().clone(), + }; + held.push(key); + } + } + } + } + // Reflected sources must move before old holds ask whether a child answered. + let mut sources = Vec::new(); + for (&peer, answer) in self.answering.iter().filter(|_| route_changed || !held.is_empty()) { + let watch = &answer.watch; + for &id in &ids { + let region = scope.as_ref().filter(|(key, _)| *key == id).map(|(_, prefix)| prefix); + for prefix in self.stream(id, peer, watch, region) { + let key = (id, prefix); + if let Some(sent) = answer.tx.get(&key).filter(|sent| sent.offer().is_some()) { + sources.push((peer, key.clone(), sent.src.clone(), self.src(id, &key.1, peer, watch))); + } + } + } + } + for (peer, key, was, src) in sources { + self.step(); + if was != src { + self.unreflect(peer, &key, &was); + self.reflect(peer, &key, &src); + self.answering + .get_mut(&peer) + .expect("answering") + .tx + .get_mut(&key) + .expect("sent") + .src = src; + } + } + for key in held { + match self.retain(&key) { + Some(route) => { + self.routes.insert(key, route); + } + None => { + self.routes.remove(&key); + } + } + } + for (key, was) in before { + if was != self.routes.get(&key).map(Route::live) { + dirty.entry(key.0).or_default().regions.insert(key.1); + } + } + for &id in &ids { + let dirty = dirty.entry(id).or_default(); + let prefixes = affected.get_mut(&id).expect("affected"); + for region in &dirty.regions { + prefixes.extend(self.prefixes(id, Some(region))); + } + let keys: BTreeSet = prefixes + .iter() + .flat_map(|prefix| { + let key = (id, prefix.clone()); + self.views + .contains_key(&key) + .then_some(prefix.clone()) + .into_iter() + .chain( + self.routes + .contains_key(&key) + .then(|| meet(prefix, &self.watch)) + .flatten(), + ) + }) + .collect(); + prefixes.extend(keys.iter().cloned()); + self.know(id, &keys, delta.as_ref(), dirty); + } + for report in reports { + match report { + Report::Closed(hop) => self.report(hop), + Report::Draining(node) => self.report_draining(node), + } + } + for &id in &ids { + let mut dirty = dirty.remove(&id).unwrap_or_default(); + self.expand_dirty(id, &mut dirty); + // Losing an ancestor can affect siblings of the original changed prefix. + let region = scope.as_ref().filter(|(key, _)| *key == id).map(|(_, prefix)| { + ancestors(prefix) + .find(|above| dirty.regions.contains(above)) + .unwrap_or_else(|| prefix.clone()) + }); + self.advertise(id, &dirty, region.as_ref()); + self.show(id, &dirty); + } + for (id, prefixes) in affected { + for prefix in prefixes { + let key = (id, prefix.clone()); + if !self.rx.contains_key(&key) + && !self.routes.contains_key(&key) + && !self.views.contains_key(&key) + && let Some(scopes) = self.scopes.get_mut(&id) + { + scopes.remove(&prefix); + } + } + if self.scopes.get(&id).is_some_and(BTreeSet::is_empty) { + self.scopes.remove(&id); + } + } + self.collect_doubt(); + #[cfg(test)] + self.check_indexes(); + } + + /// Expand only regions whose shape changed. A sibling prefix's claims are not + /// visited; each range uses the prefix and reader indexes. + fn expand_dirty(&self, id: Id, dirty: &mut Dirty) { + if dirty.all { + return; + } + for region in dirty + .regions + .iter() + .filter(|region| !ancestors(region).any(|above| above != **region && dirty.regions.contains(&above))) + { + dirty.claims.insert(region.clone()); + dirty.claims.extend( + self.shown_claims + .get(&id) + .into_iter() + .flat_map(|claims| within_set(claims, region)) + .cloned(), + ); + if let Some(origin) = self.origins.get(&id) { + dirty + .claims + .extend(below(&origin.view.items, region).map(|(claim, _)| claim.clone())); + } + for prefix in self.prefixes(id, Some(region)) { + self.step(); + if let Some(view) = self.views.get(&(id, prefix)) { + dirty + .claims + .extend(below(&view.items, region).map(|(claim, _)| claim.clone())); + } + } + } + } + + /// Forget each dead hop no heard route names once its timer fired, each draining + /// node no heard route names once its doubt lifted, and each told hop or draining + /// node no sent route names. Tests also compare early doubt collection. + fn collect_doubt(&mut self) { + let named: BTreeSet = self + .dead + .keys() + .filter(|hop| self.rx_hops.contains_key(hop)) + .copied() + .collect(); + self.dead + .retain(|hop, state| named.contains(hop) || matches!(state, Death::Waiting { .. })); + #[cfg(test)] + if self.doubt.lift == Lift::Unnamed { + let lifted: Vec = self + .doubted + .iter() + .filter_map(|(node, state)| { + (matches!(state, Suspicion::Doubted(_)) && !self.named(*node)).then_some(*node) + }) + .collect(); + for node in lifted { + let at = self.doubted[&node].deadline(); + self.doubted.insert(node, Suspicion::Lifted(at)); + } + } + let draining: Vec = self.draining.iter().copied().collect(); + for node in draining { + if !self.named(node) && !self.doubted.contains_key(&node) { + self.draining.remove(&node); + } + } + let tx_hops = &self.tx_hops; + let sent = |peer: &N, hop: &Hop| { + tx_hops.get(hop).is_some_and(|set| { + set.range((*peer, (0, Path::empty()))..) + .next() + .is_some_and(|(to, _)| to == peer) + }) + }; + let tx_nodes = &self.tx_nodes; + #[cfg(test)] + let work = &self.work; + for (peer, answer) in &mut self.answering { + answer.told.retain(|hop| sent(peer, hop)); + answer.told_draining.retain(|node| { + #[cfg(test)] + work.set(work.get() + 1); + tx_nodes.get(node).is_some_and(|peers| peers.contains_key(peer)) + }); + } + } + + #[cfg(test)] + fn check_indexes(&self) { + let mut scopes: BTreeMap> = BTreeMap::new(); + for (id, prefix) in self.rx.keys().chain(self.routes.keys()).chain(self.views.keys()) { + scopes.entry(*id).or_default().insert(prefix.clone()); + } + assert_eq!(self.scopes, scopes, "prefix index"); + assert!( + self.answering.keys().all(|peer| self.sessions.contains_key(peer)), + "answering sessions" + ); + assert!( + self.routes + .values() + .all(|route| !matches!(route, Route::Planning { .. })), + "unresolved route plan" + ); + let mut rx_hops: BTreeMap>> = BTreeMap::new(); + for (key, routes) in &self.rx { + for (peer, heard) in routes { + for hop in heard.offer.path.iter() { + rx_hops + .entry(*hop) + .or_default() + .entry(*peer) + .or_default() + .insert(key.clone()); + } + assert!(self.sessions[peer].heard.contains(key), "heard index"); + } + } + assert_eq!(rx_hops, self.rx_hops, "rx hop index"); + let mut tx_hops: BTreeMap> = BTreeMap::new(); + for (peer, answer) in &self.answering { + for (key, sent) in &answer.tx { + for hop in sent.offer().into_iter().flat_map(|offer| offer.path.iter()) { + tx_hops.entry(*hop).or_default().insert((*peer, key.clone())); + } + } + } + assert_eq!(tx_hops, self.tx_hops, "tx hop index"); + let mut tx_nodes: BTreeMap> = BTreeMap::new(); + for (peer, answer) in &self.answering { + for sent in answer.tx.values() { + for hop in sent.offer().into_iter().flat_map(|offer| offer.path.iter()) { + *tx_nodes.entry(hop.node).or_default().entry(*peer).or_default() += 1; + } + } + } + assert_eq!(tx_nodes, self.tx_nodes, "tx node index"); + let mut reflects: BTreeMap> = BTreeMap::new(); + for (peer, answer) in &self.answering { + for (key, sent) in &answer.tx { + reflects + .entry((key.0, sent.src.clone())) + .or_default() + .insert((*peer, key.1.clone())); + } + } + assert_eq!(reflects, self.reflects, "reflection index"); + } + + // --- Internals: the reader --- + + /// The view the reader sees `claim` of `id` from: the narrowest view above it that + /// knows something, where the claim sits at its own path. The view at the claim + /// itself presents there whichever claim covers it, so it is the source only when + /// none above knows as much: with none above, or a newer floor than theirs. + fn source(&self, id: Id, claim: &Path) -> Option<(Path, &View)> { + let at = (self.views.get(&(id, claim.clone()))).filter(|view| view.floors.contains_key(claim)); + match (self.above(id, claim), at) { + (Some((above, wider)), Some(at)) if at.floors.get(claim) > wider.floors.get(above) => { + Some((claim.clone(), at)) + } + (Some((above, wider)), _) => Some((above.clone(), wider)), + (None, at) => Some((claim.clone(), at.or_else(|| self.views.get(&(id, claim.clone())))?)), + } + } + + /// How the reader sees `claim` of `id` now, from its [`Self::source`], available + /// while a live route covers it. + fn shown_now(&self, id: Id, claim: &Path) -> Option> { + if !within(claim, &self.watch) { + return None; + } + let available = || ancestors(claim).any(|above| self.routes.get(&(id, above)).is_some_and(Route::live)); + if let Some(origin) = self.origins.get(&id) { + let item = origin.view.items.get(claim)?; + return Some(Shown { + item: item.clone(), + available: true, + presented: false, + }); + } + let (from, view) = self.source(id, claim)?; + let item = view.items.get(claim)?; + Some(Shown { + item: item.clone(), + available: available(), + presented: from == *claim, + }) + } + + /// Recompute what this node's reader sees of `id` and log each change: only the + /// `dirty` claims, unless any could have changed. + fn show(&mut self, id: Id, dirty: &Dirty) { + let claims = match dirty.all { + true => self.claims(id), + false => dirty.claims.clone(), + }; + #[cfg(test)] + self.work.set(self.work.get() + claims.len()); + let now: BTreeMap>> = claims + .iter() + .map(|claim| (claim.clone(), self.shown_now(id, claim))) + .collect(); + #[cfg(test)] + if !dirty.all { + let full = (self.claims(id).into_iter()) + .map(|claim| { + let shown = self.shown_now(id, &claim); + (claim, shown) + }) + .collect(); + self.reader.check(id, &now, full); + } + // A claim no longer shown under a route that still covers it was withdrawn. + let routes = &self.routes; + let ended = |claim: &Path| ancestors(claim).any(|above| routes.contains_key(&(id, above))); + for (claim, shown) in &now { + match shown { + Some(_) => { + self.shown_claims.entry(id).or_default().insert(claim.clone()); + } + None => { + if let Some(claims) = self.shown_claims.get_mut(&id) { + claims.remove(claim); + if claims.is_empty() { + self.shown_claims.remove(&id); + } + } + } + } + } + self.reader.apply(id, &self.watch, now, ended); + #[cfg(test)] + assert_eq!( + self.shown_claims + .get(&id) + .into_iter() + .flatten() + .cloned() + .collect::>(), + self.reader.claims(id).cloned().collect(), + "reader claim index" + ); + } + + /// Every claim of `id` the reader sees or could. + fn claims(&self, id: Id) -> BTreeSet { + let mut claims: BTreeSet = self.reader.claims(id).cloned().collect(); + match self.origins.get(&id) { + Some(origin) => claims.extend(origin.view.items.keys().cloned()), + None => { + for (_, view) in of(&self.views, id) { + claims.extend(view.items.keys().cloned()); + } + } + } + claims + } +} diff --git a/rs/moq-net/src/route/reader.rs b/rs/moq-net/src/route/reader.rs new file mode 100644 index 00000000000..41453589082 --- /dev/null +++ b/rs/moq-net/src/route/reader.rs @@ -0,0 +1,186 @@ +//! What a node's announcement reader sees, and the events it logs as that changes: +//! one implementation for every route rule. + +use std::collections::{BTreeMap, BTreeSet}; + +use crate::Path; + +use super::view::ancestors; +use super::{Event, Id, Kind, Shown, within}; + +/// How the reader sees each claim, and the events not yet taken. +pub(crate) struct Reader { + shown: BTreeMap<(Id, Path), Shown>, + /// The identities recomputed since [`Self::changed`] last took them, each with + /// the claims whose shown state changed. + changed: BTreeMap>, + events: Vec>, + /// Per claim this node has shown, how many times it forgot that claim. + #[cfg(test)] + forgot: BTreeMap<(Id, Path), u32>, +} + +impl Default for Reader { + fn default() -> Self { + Self { + shown: BTreeMap::new(), + changed: BTreeMap::new(), + events: Vec::new(), + #[cfg(test)] + forgot: BTreeMap::new(), + } + } +} + +impl Reader { + pub fn shown(&self) -> &BTreeMap<(Id, Path), Shown> { + &self.shown + } + + /// How the reader sees `claim` of `id`. + pub fn get(&self, id: Id, claim: &Path) -> Option<&Shown> { + self.shown.get(&(id, claim.clone())) + } + + /// The claims of `id` the reader sees. + pub fn claims(&self, id: Id) -> impl Iterator { + self.shown + .range((id, Path::empty())..) + .take_while(move |((key, _), _)| *key == id) + .map(|((_, claim), _)| claim) + } + + /// Take the identities recomputed since the last call, each with the claims whose + /// shown state changed. + pub fn changed(&mut self) -> BTreeMap> { + std::mem::take(&mut self.changed) + } + + /// Mark `id` recomputed, whether or not anything it shows changed. + pub fn touch(&mut self, id: Id) { + self.changed.entry(id).or_default(); + } + + /// Take the events, in order. + pub fn events(&mut self) -> Vec> { + std::mem::take(&mut self.events) + } + + /// Count a forget against each claim of `id` this node has shown that `kept` does not keep. + #[cfg_attr(not(test), allow(unused_variables, reason = "only fixtures count forgets"))] + pub fn forget(&mut self, id: Id, kept: impl Fn(&Path) -> bool) { + #[cfg(test)] + for ((key, claim), count) in self.forgot.iter_mut() { + if *key == id && !kept(claim) { + *count += 1; + } + } + } + + /// Assert that `now`, recomputed for some claims of `id`, agrees with `full`, + /// recomputed for all of them. + #[cfg(test)] + pub fn check(&self, id: Id, now: &BTreeMap>>, full: BTreeMap>>) { + for (claim, full) in full { + let got = match now.get(&claim) { + Some(got) => got.as_ref(), + None => self.get(id, &claim), + }; + assert_eq!(got, full.as_ref(), "incremental reader view of {id} {claim:?}"); + } + } + + /// The reader now sees each claim of `id` in `now` as it says, inside `watch`. Log + /// each change. A claim no longer shown ended if `ended` says the origin withdrew it + /// as far as this node knows, and went offline if the node let go of it instead. + pub fn apply( + &mut self, + id: Id, + watch: &Path, + now: BTreeMap>>, + ended: impl Fn(&Path) -> bool, + ) { + let shown = |claim: &Path| match now.get(claim) { + Some(now) => now.as_ref(), + None => self.get(id, claim), + }; + let mut events = Vec::new(); + for (claim, now) in &now { + let Some(was) = self.get(id, claim) else { + continue; + }; + match now { + Some(now) => { + if was.item != now.item { + events.push((claim, Kind::Update, &now.item)); + } + if was.available != now.available { + let kind = match now.available { + true => Kind::Available, + false => Kind::Unavailable, + }; + events.push((claim, kind, &now.item)); + } + } + None => { + // A claim presented at a narrower region moved up to the claim it + // stood for, which is still shown. That claim may have changed + // meanwhile, so a presented claim needs only something shown above it. + let moved = ancestors(claim) + .filter(|above| above != claim) + .any(|above| shown(&above).is_some_and(|now| was.presented || now.item == was.item)); + let kind = if !within(claim, watch) { + Kind::Unwatched + } else if !moved && ended(claim) { + Kind::End + } else { + Kind::Offline + }; + events.push((claim, kind, &was.item)); + } + } + } + for (claim, now) in &now { + if let Some(now) = now + && self.get(id, claim).is_none() + { + events.push((claim, Kind::Start, &now.item)); + } + } + let events: Vec<(Path, Kind, A)> = (events.into_iter()) + .map(|(claim, kind, item)| (claim.clone(), kind, item.clone())) + .collect(); + for (claim, kind, item) in events { + self.log(id, claim, kind, item); + } + let changed = self.changed.entry(id).or_default(); + for (claim, now) in now { + let key = (id, claim); + if self.shown.get(&key) == now.as_ref() { + continue; + } + changed.insert(key.1.clone()); + match now { + Some(now) => { + self.shown.insert(key, now); + } + None => { + self.shown.remove(&key); + } + }; + } + } + + fn log(&mut self, id: Id, claim: Path, kind: Kind, item: A) { + #[cfg(test)] + let forgot = *self.forgot.entry((id, claim.clone())).or_insert(0); + self.events.push(Event { + id, + claim, + kind, + item, + #[cfg(test)] + forgot, + }); + } +} diff --git a/rs/moq-net/src/route/tests/compare.rs b/rs/moq-net/src/route/tests/compare.rs index bfd764464bd..400997f50a1 100644 --- a/rs/moq-net/src/route/tests/compare.rs +++ b/rs/moq-net/src/route/tests/compare.rs @@ -1,7 +1,8 @@ -//! One script on two routers: [`Router`], which keeps Babel's distance, and the -//! prototype in `paths.rs`, which keeps a route loop-free by its path alone. +//! One script on both rules: [`Router`], which keeps Babel's distance, and +//! [`path::Router`], which keeps its doubt until the timer fires. The ignored +//! benchmarks also compare the rejected early-collection policy. //! -//! A [`Lab`] builds the same mesh on either, applies one change at a time, and +//! A [`Lab`] builds the same mesh on each, applies one change at a time, and //! delivers in rounds: every message queued when a round starts crosses its link //! in that round, so a round is one hop of delay. After each delivery it follows //! every relay's next hops, and after each round it counts the relays whose route @@ -10,9 +11,14 @@ use std::collections::BTreeSet; -use super::paths::{PathSim, Rules}; use super::*; +/// The production router and its accepted replacement, checked in ordinary CI. +const RULES: [Rule; 2] = [Rule::Distance, Rule::PATH]; + +/// The rules compared, in the report's order. +pub(super) const COMPARED: [Rule; 3] = [Rule::Distance, Rule::EARLY, Rule::PATH]; + /// A queue of messages, or of echoed ROUTE_ENDs, from one node to another. #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] pub(super) enum Chan { @@ -75,10 +81,6 @@ pub(super) trait Net { fn tick(&mut self) -> bool { false } - /// What `node` holds for `id`, for a failure report. - fn dump(&self, _node: Node, _id: Id) -> String { - String::new() - } /// The neighbour `node` routes `id` through: `Some(None)` at the publisher. fn next(&self, node: Node, id: Id) -> Option>; fn showing(&self, node: Node, id: Id, claim: &str) -> Option; @@ -123,14 +125,7 @@ impl Net for Sim { } fn sever(&mut self, node: Node, peer: Node) { - self.up.remove(&Self::pair(node, peer)); - for key in [(node, peer), (peer, node)] { - self.queues.remove(&key); - self.acks.remove(&key); - self.held.remove(&key); - } - self.nodes[peer].disconnect([node]); - self.flush(); + Sim::sever(self, node, peer); self.closed(peer); } @@ -165,8 +160,13 @@ impl Net for Sim { }; } + fn tick(&mut self) -> bool { + self.wait(HOP); + self.deadline().is_some() + } + fn next(&self, node: Node, id: Id) -> Option> { - self.route(node, id).map(|route| route.next) + self.toward(node, id) } fn showing(&self, node: Node, id: Id, claim: &str) -> Option { @@ -177,7 +177,7 @@ impl Net for Sim { let (offline, unavailable, starts) = tally(self.events.iter().map(|event| event.kind)); Sent { msgs: self.stats.route_msgs + self.stats.requests, - items: self.stats.route_msgs + self.stats.requests, + items: self.stats.route_items + self.stats.requests, claims: self.stats.claim_items, offline, unavailable, @@ -189,150 +189,14 @@ impl Net for Sim { impl Sim { /// A publisher advances its seqno when a session its route went out on closes, /// as `route::Node` has it do. A new session lengthens no path, so it does not. + /// The path rule has no seqno. fn closed(&mut self, node: Node) { - if self.declared[node].is_some() { + if self.declared[node].is_some() && !self.paths() { self.bump(node); } } } -impl Net for PathSim { - fn client(&mut self, node: Node) { - self.leaf(node); - } - - fn join(&mut self, a: Node, b: Node, cost: Metric) { - PathSim::join(self, a, b, cost); - } - - fn open(&mut self, a: Node, b: Node) { - self.connect(a, b); - } - - fn close(&mut self, a: Node, b: Node) { - self.disconnect(a, b); - } - - fn mute(&mut self, node: Node, peer: Node) { - PathSim::mute(self, node, peer); - } - - fn sever(&mut self, node: Node, peer: Node) { - PathSim::sever(self, node, peer); - } - - fn reset(&mut self, node: Node) { - self.kill(node); - } - - fn publish(&mut self, node: Node, id: Id) { - self.originate(node, id); - } - - fn claim(&mut self, node: Node, claim: &str, live: bool) { - match live { - true => self.announce(node, claim), - false => self.unannounce(node, claim), - } - } - - fn ready(&self) -> Vec<(Chan, usize)> { - (self.busy().into_iter()) - .map(|(from, to)| (Chan::Msg(from, to), self.queued(from, to))) - .collect() - } - - fn dump(&self, node: Node, id: Id) -> String { - self.nodes[node].dump(id) - } - - fn step(&mut self, chan: Chan) { - let Chan::Msg(from, to) = chan else { - unreachable!("the prototype echoes nothing"); - }; - self.deliver(from, to); - } - - fn tick(&mut self) -> bool { - PathSim::tick(self) - } - - fn next(&self, node: Node, id: Id) -> Option> { - PathSim::next(self, node, id) - } - - fn showing(&self, node: Node, id: Id, claim: &str) -> Option { - self.shows(node, id, claim) - } - - fn sent(&self) -> Sent { - let (offline, unavailable, starts) = tally(self.events.iter().map(|(.., kind)| *kind)); - Sent { - msgs: self.stats.route_msgs, - items: self.stats.route_items, - claims: self.stats.claim_items, - offline, - unavailable, - starts, - } - } -} - -/// The routers compared, by the name the report gives each. -#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] -pub(super) enum Rule { - /// Today's router: Babel's distance, seqno requests, holds, and echoes. - Distance, - /// Paths and nothing else. - Paths, - /// Paths, and a withdrawal that names the hop that died. - Causes, - /// Causes, one message per cause and session, and holds. - Holds, - /// Holds, and no move to a route through a node in doubt for 4 hops of delay. - Wait4, - /// The same for 32 hops of delay. - Wait32, - /// Holds, and no move to a route through a node in doubt until it advances its epoch. - Epochs, -} - -impl Rule { - pub const ALL: [Self; 7] = [ - Self::Distance, - Self::Paths, - Self::Causes, - Self::Holds, - Self::Wait4, - Self::Wait32, - Self::Epochs, - ]; - - pub fn as_str(self) -> &'static str { - match self { - Self::Distance => "distance", - Self::Paths => "paths", - Self::Causes => "paths+cause", - Self::Holds => "paths+cause+hold", - Self::Wait4 => "paths+cause+hold+wait4", - Self::Wait32 => "paths+cause+hold+wait32", - Self::Epochs => "paths+cause+hold+epoch", - } - } - - pub fn net(self, nodes: usize) -> Box { - match self { - Self::Distance => Box::new(Sim::new(nodes)), - Self::Paths => Box::new(PathSim::new(nodes, Rules::BARE)), - Self::Causes => Box::new(PathSim::new(nodes, Rules::CAUSES)), - Self::Holds => Box::new(PathSim::new(nodes, Rules::HOLDS)), - Self::Wait4 => Box::new(PathSim::new(nodes, Rules::waits(4))), - Self::Wait32 => Box::new(PathSim::new(nodes, Rules::waits(32))), - Self::Epochs => Box::new(PathSim::new(nodes, Rules::EPOCHS)), - } - } -} - /// A mesh of relays, each publisher attached to one of them. #[derive(Clone, Debug)] pub(super) struct Mesh { @@ -345,6 +209,36 @@ pub(super) struct Mesh { } impl Mesh { + /// A deterministic cheapest-path tree, shared by every rule when choosing + /// which link or transit relay to fail. Ties use the next relay's number. + fn tree(&self) -> Vec> { + let home = self.homes[0]; + let mut best = vec![None; self.relays]; + best[home] = Some((0, home)); + let mut pending = BTreeSet::from([(0, home)]); + while let Some((cost, at)) = pending.pop_first() { + if best[at].is_none_or(|(best, _)| cost != best) { + continue; + } + for &(a, b, link) in &self.links { + let peer = match at { + _ if at == a => b, + _ if at == b => a, + _ => continue, + }; + let candidate = (cost + link, at); + if best[peer].is_none_or(|best| candidate < best) { + best[peer] = Some(candidate); + pending.insert((candidate.0, peer)); + } + } + } + best.into_iter() + .enumerate() + .map(|(node, best)| best.and_then(|(_, next)| (node != home).then_some(next))) + .collect() + } + /// A ring, the publisher on relay 0. pub fn ring(relays: usize) -> Self { let links = (0..relays).map(|a| (a, (a + 1) % relays, 1)).collect(); @@ -423,6 +317,8 @@ pub(super) struct Cost { pub stale: usize, /// Relays, summed over rounds, with no route while a path to the publisher existed. pub dark: usize, + /// Hops of delay until every relay's route led to the publisher for good. + pub right: usize, /// Whether it settled at all. pub settled: bool, } @@ -432,6 +328,8 @@ pub(super) struct Lab { pub mesh: Mesh, /// Open links, relay to relay and publisher to relay. up: BTreeSet<(Node, Node)>, + /// Links opened beyond the mesh's own. + linked: BTreeSet<(Node, Node)>, down: BTreeSet, order: XorShift, cost: Cost, @@ -446,9 +344,10 @@ impl Lab { pub fn new(rule: Rule, mesh: &Mesh) -> Self { let nodes = mesh.relays + mesh.homes.len(); let mut lab = Self { - net: rule.net(nodes), + net: Box::new(Sim::with(rule, nodes)), mesh: mesh.clone(), up: BTreeSet::new(), + linked: BTreeSet::new(), down: BTreeSet::new(), order: XorShift(0x5eed_0001), cost: Cost::default(), @@ -585,10 +484,18 @@ impl Lab { self.net.tick(); } self.cost.rounds += 1; + self.note(); + true + } + + /// Count the relays left wrong as the latest round ends. + fn note(&mut self) { let (_, stale, dark) = self.wrong(); self.cost.stale += stale; self.cost.dark += dark; - true + if stale + dark > 0 { + self.cost.right = self.cost.rounds + 1; + } } /// Deliver in rounds until nothing is left. False if that never happens. @@ -607,9 +514,7 @@ impl Lab { self.cost = Cost::default(); change(self); // What the change left wrong before any message moved. - let (_, stale, dark) = self.wrong(); - self.cost.stale += stale; - self.cost.dark += dark; + self.note(); self.cost.settled = self.settle(); self.cost.sent = self.net.sent() - before; self.cost @@ -625,6 +530,15 @@ impl Lab { self.up.insert(pair(a, b)); } + /// Open a link the mesh does not have, configuring it the first time. + pub fn attach(&mut self, a: Node, b: Node) { + match self.linked.insert(pair(a, b)) { + true => self.net.join(a, b, 1), + false => self.net.open(a, b), + } + self.up.insert(pair(a, b)); + } + /// The process at `relay` exits. Each neighbour notices in turn, a hop of delay apart. pub fn crash(&mut self, relay: Node) { let mut peers = self.peers(relay); @@ -708,15 +622,12 @@ impl Lab { /// The link of a publisher's cheapest-path tree that the most relays route /// over, among those whose loss leaves every relay a path (`detour`) or cuts /// some off. - pub fn tree_link(&self, index: usize, detour: bool) -> Option<(Node, Node)> { - let (_, id) = self.mesh.publisher(index); + fn tree_link(&self, index: usize, detour: bool, tree: &[Option]) -> Option<(Node, Node)> { let mut uses: BTreeMap<(Node, Node), usize> = BTreeMap::new(); for relay in 0..self.mesh.relays { let mut at = relay; - while let Some(Some(next)) = self.net.next(at, id) { - if next < self.mesh.relays { - *uses.entry(pair(at, next)).or_default() += 1; - } + while let Some(next) = tree[at] { + *uses.entry(pair(at, next)).or_default() += 1; at = next; } } @@ -752,6 +663,10 @@ pub(super) enum Change { Home, /// The publisher's session closes. Leave, + /// The publisher's session closes and it reconnects to the same relay at once. + Rejoin, + /// The publisher's session closes and it reconnects at once through another relay. + Move, /// The publisher withdraws one claim. Withdraw, } @@ -768,6 +683,8 @@ impl Change { Self::Restart => "relay_restarts", Self::Home => "home_relay_dies", Self::Leave => "publisher_leaves", + Self::Rejoin => "publisher_rejoins", + Self::Move => "publisher_moves", Self::Withdraw => "claim_withdrawn", } } @@ -778,6 +695,7 @@ impl Change { pub(super) fn run(rule: Rule, mesh: &Mesh) -> Vec<(Change, Cost)> { let mut out = Vec::new(); let mut lab = Lab::new(rule, mesh); + let tree = mesh.tree(); let mut apply = |lab: &mut Lab, change: Change, apply: &dyn Fn(&mut Lab)| { let cost = lab.measure(apply); assert!(cost.settled, "{}: {change:?} never settles on {mesh:?}", rule.as_str()); @@ -792,26 +710,44 @@ pub(super) fn run(rule: Rule, mesh: &Mesh) -> Vec<(Change, Cost)> { let home = mesh.homes[last]; apply(&mut lab, Change::Leave, &|lab| lab.cut(node, home)); apply(&mut lab, Change::Arrive, &|lab| lab.mend(node, home)); + apply(&mut lab, Change::Rejoin, &|lab| { + lab.cut(node, home); + lab.mend(node, home); + }); + // A relay next to its home, so the publisher's new route takes a detour no longer + // than the old one did. + let next = (mesh.links.iter()).find_map(|&(a, b, _)| match home { + _ if a == home => Some(b), + _ if b == home => Some(a), + _ => None, + }); + if let Some(next) = next { + apply(&mut lab, Change::Move, &|lab| { + lab.cut(node, home); + lab.attach(node, next); + }); + lab.cut(node, next); + lab.mend(node, home); + assert!(lab.settle()); + } for (detour, down, up) in [ (true, Change::Detour, Change::Shortcut), (false, Change::Cut, Change::Mend), ] { - if let Some((a, b)) = lab.tree_link(0, detour) { + if let Some((a, b)) = lab.tree_link(0, detour, &tree) { apply(&mut lab, down, &|lab| lab.cut(a, b)); apply(&mut lab, up, &|lab| lab.mend(a, b)); } } // The relay the most others route through, the publisher's own aside. - let (node, id) = mesh.publisher(0); + let (node, _) = mesh.publisher(0); let mut through = vec![0; mesh.relays]; for relay in 0..mesh.relays { let mut at = relay; - while let Some(Some(next)) = lab.net.next(at, id) { - if next < mesh.relays { - through[next] += 1; - } + while let Some(next) = tree[at] { + through[next] += 1; at = next; } } @@ -907,11 +843,7 @@ pub(super) fn churn(rule: Rule, cases: std::ops::Range) -> Churn { } assert!(lab.settle(), "{}: case {case} never settles on {mesh:?}", rule.as_str()); if let Err(err) = lab.check() { - let (_, id) = mesh.publisher(0); - let dump: Vec = (0..=relays) - .map(|node| format!("{node}: {}", lab.net.dump(node, id))) - .collect(); - panic!("{}: case {case} on {mesh:?}: {err}\n{}", rule.as_str(), dump.join("\n")); + panic!("{}: case {case} on {mesh:?}: {err}", rule.as_str()); } let sent = lab.net.sent() - before; total.sent = Sent { @@ -930,10 +862,10 @@ pub(super) fn churn(rule: Rule, cases: std::ops::Range) -> Churn { /// never loops, even for a step. #[test] fn every_rule_settles_after_churn() { - for rule in Rule::ALL { - let churn = churn(rule, 0..60); + for rule in RULES { + let cost = churn(rule, 0..60); if rule == Rule::Distance { - assert_eq!(churn.loops, 0); + assert_eq!(cost.loops, 0); } } } @@ -947,7 +879,7 @@ fn benchmark_route_churn() { // message on the wire. println!("rule,route_messages,claim_messages"); eprintln!("rule,steps,loops,stale,dark,msgs,items,claims,offline,unavailable"); - for rule in Rule::ALL { + for rule in COMPARED { let churn = churn(rule, 0..400); let (rule, steps, sent) = (rule.as_str(), churn.steps, churn.sent); println!("{rule},{},{}", sent.msgs, sent.claims); @@ -966,57 +898,29 @@ fn every_rule_converges_on_every_shape() { let mut meshes = vec![Mesh::ring(6), Mesh::clique(6, 1), Mesh::clique(4, 3)]; meshes.extend((0..8).map(|_| Mesh::sparse(6, &mut rng))); for mesh in &meshes { - for rule in Rule::ALL { + for rule in RULES { run(rule, mesh); } } } -/// What the report's table shows, pinned on six relays. +/// Path changes carry no claim changes; a claim withdrawal changes no route. +/// Counts and recovery times are measurements in the benchmarks below. #[test] -fn the_rules_differ_as_reported() { - let cost = |rule: Rule, mesh: &Mesh, change: Change| { - let costs = run(rule, mesh); - costs.into_iter().find(|(found, _)| *found == change).unwrap().1 - }; - - // The publisher leaves a clique. A bare path hunts through every stale path; - // naming the hop that died ends them all at once, in fewer messages and hops of - // delay than the distance takes. - let clique = Mesh::clique(6, 1); - let bare = cost(Rule::Paths, &clique, Change::Leave); - let cause = cost(Rule::Causes, &clique, Change::Leave); - let distance = cost(Rule::Distance, &clique, Change::Leave); - assert!(bare.loops > 0 && bare.sent.msgs > 5 * cause.sent.msgs, "{bare:?}"); - assert_eq!(cause.loops, 0); - assert!(cause.sent.msgs < distance.sent.msgs && cause.rounds < distance.rounds); - - // A link of a ring closes, and the way round runs through relays that routed - // through the one that lost it. The distance waits for the publisher's next - // seqno. Without a hold the relay lets its claims go and they are sent again. - let ring = Mesh::ring(6); - let distance = cost(Rule::Distance, &ring, Change::Detour); - let holds = cost(Rule::Holds, &ring, Change::Detour); - assert!(holds.rounds < distance.rounds && holds.dark < distance.dark); - assert_eq!((holds.sent.claims, holds.sent.offline), (0, 0)); - assert!(cost(Rule::Causes, &ring, Change::Detour).sent.claims > 0); - - // The publisher's relay dies, and each neighbour reports its own session. A relay - // told of one takes a route through another for a moment, unless the dead relay - // itself is in doubt: until it advances its epoch, or for a while by the clock, - // which costs no message. - assert!(cost(Rule::Holds, &ring, Change::Home).loops > 0); - assert_eq!(cost(Rule::Epochs, &ring, Change::Home).loops, 0); - let wait = cost(Rule::Wait32, &ring, Change::Home); - assert_eq!(wait.loops, 0); - assert!(wait.sent.msgs <= cost(Rule::Epochs, &ring, Change::Home).sent.msgs); - // The clock's price: a detour through the far end of the closed link waits it out. - let (wait, epochs) = ( - cost(Rule::Wait32, &clique, Change::Detour), - cost(Rule::Epochs, &clique, Change::Detour), - ); - assert!(wait.rounds > 32 && epochs.rounds < 8 && wait.sent.msgs < epochs.sent.msgs); - assert_eq!(cost(Rule::Distance, &ring, Change::Home).loops, 0); +fn path_changes_keep_claims_and_withdrawals_keep_routes() { + for mesh in [Mesh::clique(6, 1), Mesh::ring(6)] { + let costs = RULES.map(|rule| run(rule, &mesh).into_iter().collect::>()); + for costs in &costs { + assert!(costs.values().all(|cost| cost.settled && cost.loops == 0)); + } + for path in &costs[1..] { + for change in [Change::Leave, Change::Detour, Change::Home] { + assert_eq!(path[&change].sent.claims, 0); + } + assert_eq!(path[&Change::Detour].sent.offline, 0, "a detour keeps claims"); + assert_eq!(path[&Change::Withdraw].sent.msgs, 0, "claims alone change no route"); + } + } } /// Per change, mesh size, and rule, over fixed seeds: the mean of what converging cost. @@ -1025,8 +929,8 @@ fn the_rules_differ_as_reported() { fn benchmark_route_rules() { // As above: message counts to stdout for the report, which keeps the largest mesh // alone, and the whole table to stderr. - println!("shape,relays,change,rule,route_messages,claim_messages,loop_messages"); - eprintln!("shape,relays,change,rule,msgs,items,claims,rounds,stale,dark,loops,offline,unavailable"); + println!("shape,relays,change,rule,route_messages,claim_messages,loop_messages,reroute_ms"); + eprintln!("shape,relays,change,rule,msgs,items,claims,rounds,right,stale,dark,loops,offline,unavailable"); for relays in [3, 6, 12, 24] { let mut rng = XorShift(0x6d6f_7100 + relays as u64); let mut shapes = vec![ @@ -1036,12 +940,7 @@ fn benchmark_route_rules() { shapes.push(("clique_8_publishers", vec![Mesh::clique(relays, 8)])); shapes.push(("sparse", (0..16).map(|_| Mesh::sparse(relays, &mut rng)).collect())); for (shape, meshes) in &shapes { - for rule in Rule::ALL { - // A bare path hunts through every stale path of a clique: 65,219 messages - // for one publisher leaving twelve relays, and it grows as the factorial. - if rule == Rule::Paths && shape.starts_with("clique") && (relays > 12 || meshes[0].homes.len() > 1) { - continue; - } + for rule in COMPARED { let mut all: BTreeMap> = BTreeMap::new(); for mesh in meshes { for (change, cost) in run(rule, mesh) { @@ -1054,18 +953,20 @@ fn benchmark_route_rules() { let (change, rule) = (change.as_str(), rule.as_str()); if relays == 24 { println!( - "{shape},{relays},{change},{rule},{:.1},{:.1},{:.1}", + "{shape},{relays},{change},{rule},{:.1},{:.1},{:.1},{:.1}", mean(|cost| cost.sent.msgs), mean(|cost| cost.sent.claims), mean(|cost| cost.loops), + mean(|cost| cost.right) * HOP.as_secs_f64() * 1000.0, ); } eprintln!( - "{shape},{relays},{change},{rule},{:.1},{:.1},{:.1},{:.1},{:.1},{:.1},{:.1},{:.1},{:.1}", + "{shape},{relays},{change},{rule},{:.1},{:.1},{:.1},{:.1},{:.1},{:.1},{:.1},{:.1},{:.1},{:.1}", mean(|cost| cost.sent.msgs), mean(|cost| cost.sent.items), mean(|cost| cost.sent.claims), mean(|cost| cost.rounds), + mean(|cost| cost.right), mean(|cost| cost.stale), mean(|cost| cost.dark), mean(|cost| cost.loops), diff --git a/rs/moq-net/src/route/tests/handoff.rs b/rs/moq-net/src/route/tests/handoff.rs index 248910b1f83..1d1ddcebebf 100644 --- a/rs/moq-net/src/route/tests/handoff.rs +++ b/rs/moq-net/src/route/tests/handoff.rs @@ -11,11 +11,49 @@ use std::collections::BTreeSet; use super::*; +both!( + an_older_complete_snapshot_revives_no_known_withdrawal, + trailing_announcements_report_no_false_absence, + a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it, + widening_interest_needs_floors_scoped_to_what_they_cover, + cancelling_interest_retires_its_region, + widening_hides_what_was_frozen_until_a_covering_floor_arrives, + a_snapshot_queued_before_widening_keeps_the_frozen_rest_hidden, + widening_behind_a_scoped_grant_shows_the_rest_of_the_scope, + widening_behind_two_grants_waits_for_each_scope, + a_closed_grant_lets_its_scope_go, + a_disjoint_interest_is_answered_with_nothing, + an_interest_matches_whole_segments, + a_lost_scope_keeps_newer_knowledge_where_still_certified, + a_relay_lets_go_of_a_scope_its_uplinks_lost, + a_claim_above_an_interest_is_presented_at_it, + widening_past_a_changed_claim_moves_it_up, + widening_while_the_relay_lost_the_origin_moves_the_claim_up, + a_claim_withdrawn_under_a_live_one_ends, + wildcard_claims_end_per_origin_and_are_certified_below, + origin_restart_takes_a_fresh_identity_for_its_claims, + stale_replay_after_collection_lasts_only_until_the_withdrawal_behind_it, + holds_retire_once_nothing_can_recover, + exhausted_counters_rotate_the_identity, + a_quiet_origin_is_reached_by_a_kept_request, + sweep_requests_reach_a_quiet_origin, + announcement_cost_follows_sessions_and_live_claims, + sweep_scoped_handoff, + sweep_handoff, + a_link_flap_costs_routes_not_broadcasts, + a_restarted_route_sends_a_complete_snapshot, + a_covering_claim_moves_with_a_disjoint_interest, + a_restarted_floorless_route_drops_what_it_no_longer_carries, + a_change_of_hops_alone_lets_go_of_no_claim, + a_publishers_goaway_settles_between_its_neighbours, + mutually_preferred_links_settle, +); + /// An origin reaching a node through two relays, the first preferred and the /// second a feasible successor. -fn two_relays() -> Sim { +fn two_relays(rule: Rule) -> Sim { let (origin, a, b, node) = (0, 1, 2, 3); - let mut sim = Sim::new(4); + let mut sim = Sim::with(rule, 4); sim.leaf(origin); sim.join(origin, a, 1); sim.join(origin, b, 1); @@ -29,10 +67,9 @@ fn two_relays() -> Sim { /// A withdrawal reaches the node through its next hop while the alternative's /// copy is still in flight. When the next hop goes, the alternative's complete /// snapshot is older than what the node already knows, and is refused. -#[test] -fn an_older_complete_snapshot_revives_no_known_withdrawal() { +fn an_older_complete_snapshot_revives_no_known_withdrawal(rule: Rule) { let (origin, a, b, node) = (0, 1, 2, 3); - let mut sim = two_relays(); + let mut sim = two_relays(rule); sim.announce(origin, "live/x", 0, None); // Keeps the routes up once `live/x` goes. sim.announce(origin, "live/y", 0, None); @@ -48,7 +85,7 @@ fn an_older_complete_snapshot_revives_no_known_withdrawal() { sim.disconnect(node, a); assert!(sim.run()); - assert_eq!(sim.route(node, 7).and_then(|route| route.next), Some(b)); + assert_eq!(sim.next_hop(node, 7), Some(b)); assert_eq!(sim.shows(node, 7, "live/x"), None); sim.release(origin, b); @@ -61,10 +98,9 @@ fn an_older_complete_snapshot_revives_no_known_withdrawal() { /// A new next hop's ROUTE arrives before its announcements. Merged views keep /// what the node knows, and the reader sees nothing change. -#[test] -fn trailing_announcements_report_no_false_absence() { +fn trailing_announcements_report_no_false_absence(rule: Rule) { let (origin, a, b, node) = (0, 1, 2, 3); - let mut sim = two_relays(); + let mut sim = two_relays(rule); sim.disconnect(node, b); sim.announce(origin, "live/x", 0, None); assert!(sim.run()); @@ -77,7 +113,7 @@ fn trailing_announcements_report_no_false_absence() { assert!(sim.deliver(b, node), "only the ROUTE"); sim.disconnect(node, a); assert!(sim.run()); - assert_eq!(sim.route(node, 7).and_then(|route| route.next), Some(b)); + assert_eq!(sim.next_hop(node, 7), Some(b)); assert_eq!(sim.shows(node, 7, "live/x"), Some(true)); sim.release(b, node); @@ -90,11 +126,12 @@ fn trailing_announcements_report_no_false_absence() { /// The partitioned local mesh, with a client behind the starved relay. The /// CDN detour is infeasible until the origin advances its seqno, and through /// that window the relay and its client keep the claim unavailable rather -/// than ending it. The relay's seqno request recovers it through the CDN. -#[test] -fn a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it() { +/// than ending it. The relay's seqno request recovers it through the CDN. Under +/// the path rule the detour crosses the far end of the closed session, so the +/// relay holds until that doubt lifts, and needs no request. +fn a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it(rule: Rule) { let (c1, c2, l1, l2, origin, client) = (0, 1, 2, 3, 4, 5); - let mut sim = Sim::new(6); + let mut sim = Sim::with(rule, 6); sim.join(c1, c2, 10); sim.join(l1, l2, 1); sim.link(l1, c1, SimLink::cost(1).upstream(), SimLink::cost(1)); @@ -112,9 +149,17 @@ fn a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it() { let before = sim.events.len(); sim.hold(l1, c1); sim.disconnect(l1, l2); - assert!(sim.run()); - assert!(sim.starved(l1, 7)); - assert!(sim.grounded(client, 7)); + match sim.paths() { + true => { + assert!(sim.flow()); + assert!(sim.holds(l1, 7) && !sim.routed(l1, 7), "held while l2 is in doubt"); + } + false => { + assert!(sim.run()); + assert!(sim.starved(l1, 7)); + assert!(sim.grounded(client, 7)); + } + } for node in [l1, client] { assert_eq!(sim.shows(node, 7, "local/x"), Some(false), "node {node}"); } @@ -122,8 +167,10 @@ fn a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it() { sim.release(l1, c1); assert!(sim.run()); - assert_eq!(sim.seqno(origin), 1, "the request reached the origin"); - assert_eq!(sim.route(l1, 7).and_then(|route| route.next), Some(c1)); + if !sim.paths() { + assert_eq!(sim.seqno(origin), 1, "the request reached the origin"); + } + assert_eq!(sim.next_hop(l1, 7), Some(c1)); let events: Vec = sim.events[before..] .iter() .filter(|e| e.node == client) @@ -133,9 +180,9 @@ fn a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it() { } /// A client of two relays, one lagging behind its origin. -fn lagging(watch: &str) -> Sim { +fn lagging(rule: Rule, watch: &str) -> Sim { let (origin, r1, r2, client) = (0, 1, 2, 3); - let mut sim = Sim::new(4); + let mut sim = Sim::with(rule, 4); sim.leaf(origin); sim.leaf(client); sim.join(origin, r1, 1); @@ -151,10 +198,9 @@ fn lagging(watch: &str) -> Sim { /// everything. The lagging relay's wider snapshot must not revive `live/a/x`, /// yet is the first to report `live/b/y`: floors scoped to the region they /// cover do both. -#[test] -fn widening_interest_needs_floors_scoped_to_what_they_cover() { +fn widening_interest_needs_floors_scoped_to_what_they_cover(rule: Rule) { let (origin, r1, r2, client) = (0, 1, 2, 3); - let mut sim = lagging("live/a/"); + let mut sim = lagging(rule, "live/a/"); sim.announce(origin, "live/a/x", 0, None); sim.announce(origin, "live/b/y", 0, None); assert!(sim.run()); @@ -178,10 +224,9 @@ fn widening_interest_needs_floors_scoped_to_what_they_cover() { /// the client lets go of the origin and collects it. On rejoin a lagging relay's /// snapshot replays a withdrawal the client saw before it narrowed, as after any /// collection, and only until the newer snapshot lands. -#[test] -fn cancelling_interest_retires_its_region() { +fn cancelling_interest_retires_its_region(rule: Rule) { let (origin, r1, r2, client) = (0, 1, 2, 3); - let mut sim = lagging(""); + let mut sim = lagging(rule, ""); sim.announce(origin, "live/a/x", 0, None); assert!(sim.run()); sim.hold(origin, r2); @@ -213,17 +258,17 @@ fn cancelling_interest_retires_its_region() { [Kind::Start, Kind::End, Kind::Start, Kind::End] ); assert_eq!(revivals(&sim.events), 0, "a retired region, like a collection"); - assert_eq!(sim.floors(client, 7), 1); + // The path rule keeps a view only while a route covers it, and none does. + assert_eq!(sim.floors(client, 7), usize::from(!sim.paths())); sim.check().unwrap(); } /// A claim outside a narrowed interest ends unseen. On widening, what was /// kept there stays hidden until a floor covering it arrives, rather than /// showing a stale Start the next snapshot ends. -#[test] -fn widening_hides_what_was_frozen_until_a_covering_floor_arrives() { +fn widening_hides_what_was_frozen_until_a_covering_floor_arrives(rule: Rule) { let (origin, r1, r2, client) = (0, 1, 2, 3); - let mut sim = lagging(""); + let mut sim = lagging(rule, ""); sim.announce(origin, "live/a/x", 0, None); sim.announce(origin, "live/b/y", 0, None); assert!(sim.run()); @@ -248,10 +293,9 @@ fn widening_hides_what_was_frozen_until_a_covering_floor_arrives() { /// A snapshot sent before the client narrowed still waits when it widens /// again. It answers the old interest, so it must not lift the guard on what /// stayed frozen. -#[test] -fn a_snapshot_queued_before_widening_keeps_the_frozen_rest_hidden() { +fn a_snapshot_queued_before_widening_keeps_the_frozen_rest_hidden(rule: Rule) { let (origin, r1, r2, client) = (0, 1, 2, 3); - let mut sim = lagging(""); + let mut sim = lagging(rule, ""); sim.announce(origin, "live/a/x", 0, None); sim.announce(origin, "live/b/y", 0, None); assert!(sim.run()); @@ -280,10 +324,9 @@ fn a_snapshot_queued_before_widening_keeps_the_frozen_rest_hidden() { /// cut to the grant's scope, so no floor covers the widened interest, yet the /// batch answering it shows the newly covered claims, and claim churn keeps /// the certified regions bounded. -#[test] -fn widening_behind_a_scoped_grant_shows_the_rest_of_the_scope() { +fn widening_behind_a_scoped_grant_shows_the_rest_of_the_scope(rule: Rule) { let (origin, relay, client) = (0, 1, 2); - let mut sim = Sim::new(3); + let mut sim = Sim::with(rule, 3); sim.leaf(origin); sim.leaf(client); sim.join(origin, relay, 1); @@ -304,13 +347,17 @@ fn widening_behind_a_scoped_grant_shows_the_rest_of_the_scope() { sim.watch(client, ""); assert!(sim.run()); assert_eq!(sim.shows(client, id, "live/a/y"), Some(true)); - let certified = sim.retained().certified; + let guard = |sim: &Sim| match sim.paths() { + true => sim.kept().views, + false => sim.retained().certified, + }; + let certified = guard(&sim); for _ in 0..8 { sim.announce(origin, "live/a/z", 0, None); sim.unannounce(origin, "live/a/z"); assert!(sim.run()); } - assert_eq!(sim.retained().certified, certified, "claim churn adds no state"); + assert_eq!(guard(&sim), certified, "claim churn adds no state"); assert!( sim.check().unwrap().contains(&(client, id)), "the qualified view is checked" @@ -321,10 +368,9 @@ fn widening_behind_a_scoped_grant_shows_the_rest_of_the_scope() { /// batch through the first grant certifies only its own scope: a claim in the /// second scope that ended while unwatched stays hidden until the second /// grant answers too. -#[test] -fn widening_behind_two_grants_waits_for_each_scope() { +fn widening_behind_two_grants_waits_for_each_scope(rule: Rule) { let (origin, r1, r2, client) = (0, 1, 2, 3); - let mut sim = scoped_pair(["live/a/", "live/b/"]); + let mut sim = scoped_pair(rule, ["live/a/", "live/b/"]); sim.announce(origin, "live/a/x/1", 0, None); sim.announce(origin, "live/b/z", 0, None); assert!(sim.run()); @@ -350,9 +396,9 @@ fn widening_behind_two_grants_waits_for_each_scope() { /// An origin reaching a client through two relays, each admitting the client /// under a grant scoped to one of `scopes` with one subject. -fn scoped_pair(scopes: [&'static str; 2]) -> Sim { +fn scoped_pair(rule: Rule, scopes: [&'static str; 2]) -> Sim { let (origin, r1, r2, client) = (0, 1, 2, 3); - let mut sim = Sim::new(4); + let mut sim = Sim::with(rule, 4); sim.leaf(origin); sim.leaf(client); sim.join(origin, r1, 1); @@ -371,10 +417,9 @@ fn scoped_pair(scopes: [&'static str; 2]) -> Sim { /// One key arrives through two grants. When one grant's session closes, /// nothing certifies its scope, so its claims go offline rather than staying /// shown through the other grant, which could never deliver their withdrawal. -#[test] -fn a_closed_grant_lets_its_scope_go() { +fn a_closed_grant_lets_its_scope_go(rule: Rule) { let (origin, r1, client) = (0, 1, 3); - let mut sim = scoped_pair(["live/a/", "live/b/"]); + let mut sim = scoped_pair(rule, ["live/a/", "live/b/"]); sim.announce(origin, "live/a/x", 0, None); sim.announce(origin, "live/b/z", 0, None); assert!(sim.run()); @@ -399,9 +444,9 @@ fn a_closed_grant_lets_its_scope_go() { /// Two relays each admit an uplink scoped to a different prefix and forward /// the key to a third, whose view carries one floor per scope. -fn two_scoped_uplinks() -> Sim { +fn two_scoped_uplinks(rule: Rule) -> Sim { let (origin, local, x, y, z) = (0, 1, 2, 3, 4); - let mut sim = Sim::new(5); + let mut sim = Sim::with(rule, 5); sim.leaf(origin); sim.join(origin, local, 1); for (relay, scope) in [(x, "live/a/"), (y, "live/b/")] { @@ -421,10 +466,9 @@ fn two_scoped_uplinks() -> Sim { /// peer's whole interest, so the peer shows both presentations uncertified. /// When the peer moves to an interest no scope meets, each relay ends the route /// on it, so the peer holds no route and waits on no certifier for the origin. -#[test] -fn a_disjoint_interest_is_answered_with_nothing() { +fn a_disjoint_interest_is_answered_with_nothing(rule: Rule) { let (origin, z) = (0, 4); - let mut sim = two_scoped_uplinks(); + let mut sim = two_scoped_uplinks(rule); sim.announce(origin, "live/", 0, None); assert!(sim.run()); let id = qualified("local", 7); @@ -432,7 +476,7 @@ fn a_disjoint_interest_is_answered_with_nothing() { assert_eq!(sim.shows(z, id, "live/a/"), Some(true)); assert_eq!(sim.shows(z, id, "live/b/"), Some(true)); assert!( - sim.nodes[z].views().all(|(_, view)| view.items.is_empty()), + sim.paths() || sim.babel(z).views().all(|(_, view)| view.items.is_empty()), "nothing certified" ); @@ -446,10 +490,9 @@ fn a_disjoint_interest_is_answered_with_nothing() { /// An interest matches whole path segments: watching `live/a` sees what lies /// below it, never a sibling `live/ab` that merely shares its bytes. -#[test] -fn an_interest_matches_whole_segments() { +fn an_interest_matches_whole_segments(rule: Rule) { let (origin, client) = (0, 1); - let mut sim = Sim::new(2); + let mut sim = Sim::with(rule, 2); sim.leaf(origin); sim.join(origin, client, 1); sim.originate(origin, 7, 0); @@ -466,10 +509,9 @@ fn an_interest_matches_whole_segments() { /// lags behind a withdrawal the wide one delivered; when the wide session /// closes, only what lies outside the narrow scope goes offline, and the newer /// knowledge inside it stays rather than reviving the withdrawn claim. -#[test] -fn a_lost_scope_keeps_newer_knowledge_where_still_certified() { +fn a_lost_scope_keeps_newer_knowledge_where_still_certified(rule: Rule) { let (origin, r1, r2, client) = (0, 1, 2, 3); - let mut sim = scoped_pair(["live/", "live/b/"]); + let mut sim = scoped_pair(rule, ["live/", "live/b/"]); sim.announce(origin, "live/a/x", 0, None); sim.announce(origin, "live/b/z", 0, None); assert!(sim.run()); @@ -502,12 +544,16 @@ fn a_lost_scope_keeps_newer_knowledge_where_still_certified() { /// relay. Once the other relay's newer seqno makes the starved one infeasible, /// nothing carries the lost scope and the third lets it go, whether or not the /// starved relay's own requests get through. Certifiers are feasible neighbors -/// only, so the two never keep the scope alive through each other. -#[test] -fn a_relay_lets_go_of_a_scope_its_uplinks_lost() { +/// only, so the two never keep the scope alive through each other. The path rule +/// sends no request, and its relay holds until the third answers. +fn a_relay_lets_go_of_a_scope_its_uplinks_lost(rule: Rule) { let (origin, local, x, z) = (0, 1, 2, 4); - for request in [false, true] { - let mut sim = two_scoped_uplinks(); + let requests: &[bool] = match rule { + Rule::Distance => &[false, true], + Rule::Path(_) => &[true], + }; + for &request in requests { + let mut sim = two_scoped_uplinks(rule); sim.announce(origin, "live/a/x", 0, None); sim.announce(origin, "live/b/z", 0, None); assert!(sim.run()); @@ -545,10 +591,9 @@ fn a_relay_lets_go_of_a_scope_its_uplinks_lost() { /// interest covers it, and a stream answering the interest presents it there. /// Withdrawing it moves the presentation to the next claim up rather than /// ending it. -#[test] -fn a_claim_above_an_interest_is_presented_at_it() { +fn a_claim_above_an_interest_is_presented_at_it(rule: Rule) { let (origin, client) = (0, 3); - let mut sim = lagging("live/a/x"); + let mut sim = lagging(rule, "live/a/x"); sim.announce(origin, "live", 0, None); sim.announce(origin, "live/a", 0, None); sim.announce(origin, "live/b", 0, None); @@ -569,10 +614,9 @@ fn a_claim_above_an_interest_is_presented_at_it() { /// A client presented a claim at a narrow interest widens it after the claim /// changed, while cut off from the change. Its stale presentation may come /// back first, and moves up to the changed claim without ending. -#[test] -fn widening_past_a_changed_claim_moves_it_up() { +fn widening_past_a_changed_claim_moves_it_up(rule: Rule) { let (origin, relay, client) = (0, 1, 2); - let mut sim = Sim::new(3); + let mut sim = Sim::with(rule, 3); sim.leaf(origin); sim.leaf(client); sim.join(origin, relay, 1); @@ -597,10 +641,9 @@ fn widening_past_a_changed_claim_moves_it_up() { /// A client widens past a claim that changed while its relay had lost the /// origin. When the relay gets the origin back, the client's narrow /// presentation moves up to the changed claim without ending. -#[test] -fn widening_while_the_relay_lost_the_origin_moves_the_claim_up() { +fn widening_while_the_relay_lost_the_origin_moves_the_claim_up(rule: Rule) { let (origin, relay, client) = (0, 1, 2); - let mut sim = Sim::new(3); + let mut sim = Sim::with(rule, 3); sim.leaf(origin); sim.leaf(client); sim.join(origin, relay, 1); @@ -609,13 +652,23 @@ fn widening_while_the_relay_lost_the_origin_moves_the_claim_up() { assert!(sim.run()); sim.announce(origin, "live/", 2, None); sim.watch(client, "live/b"); - for (from, to) in [(origin, relay), (origin, relay), (relay, client), (relay, client)] { - assert!(sim.deliver(from, to)); + // The rules send different numbers of route/control messages before the claim. + for (from, to) in [(origin, relay), (relay, client)] { + for _ in 0..sim.queued(from, to) { + if sim.shows(client, 7, "live/b") == Some(true) { + break; + } + assert!(sim.deliver(from, to)); + } } assert_eq!(sim.shows(client, 7, "live/b"), Some(true)); // The relay's answer to the old interest is still queued when the client widens. sim.disconnect(origin, relay); + assert!( + sim.queued(relay, client) > 0, + "the old interest's answer is still queued" + ); sim.announce(origin, "live/", 0, None); sim.watch(client, ""); sim.connect(origin, relay); @@ -627,10 +680,9 @@ fn widening_while_the_relay_lost_the_origin_moves_the_claim_up() { /// A claim withdrawn below one its origin still has ends: the claim above /// covers its path but is not the claim it was. -#[test] -fn a_claim_withdrawn_under_a_live_one_ends() { +fn a_claim_withdrawn_under_a_live_one_ends(rule: Rule) { let (origin, relay, client) = (0, 1, 2); - let mut sim = Sim::new(3); + let mut sim = Sim::with(rule, 3); sim.leaf(origin); sim.leaf(client); sim.join(origin, relay, 1); @@ -649,10 +701,9 @@ fn a_claim_withdrawn_under_a_live_one_ends() { /// watches a region below it. Withdrawing one origin's wildcard ends only that /// origin's claim, and a lagging relay's older copy of it is refused after the /// client's next hop moves there. -#[test] -fn wildcard_claims_end_per_origin_and_are_certified_below() { +fn wildcard_claims_end_per_origin_and_are_certified_below(rule: Rule) { let (o1, o2, r1, r2, client) = (0, 1, 2, 3, 4); - let mut sim = Sim::new(5); + let mut sim = Sim::with(rule, 5); for origin in [o1, o2] { sim.leaf(origin); sim.join(origin, r1, 1); @@ -697,12 +748,12 @@ fn wildcard_claims_end_per_origin_and_are_certified_below() { /// revision history and the old identity's go offline once nothing can /// recover them. Reusing the identity would restart its revision below the /// floors other nodes hold: they keep the dead process's claims and refuse -/// the new ones. -#[test] -fn origin_restart_takes_a_fresh_identity_for_its_claims() { +/// the new ones. The path rule keeps no view once nothing covers it, so a +/// reused identity starts over everywhere. +fn origin_restart_takes_a_fresh_identity_for_its_claims(rule: Rule) { let (origin, a, b) = (0, 1, 2); for fresh in [false, true] { - let mut sim = Sim::new(3); + let mut sim = Sim::with(rule, 3); sim.leaf(origin); sim.join(origin, a, 1); sim.join(a, b, 1); @@ -731,6 +782,9 @@ fn origin_restart_takes_a_fresh_identity_for_its_claims() { if fresh { assert_eq!(sim.shows(node, 8, "live/x"), Some(true)); assert_eq!(sim.shown(node).len(), 1); + } else if sim.paths() { + assert_eq!(sim.shows(node, 7, "live/x"), Some(true)); + assert_eq!(sim.shown(node).len(), 1); } else { assert_eq!(sim.shows(node, 7, "live/y"), Some(true), "the dead process's claim"); assert_eq!(sim.shows(node, 7, "live/x"), None); @@ -742,12 +796,12 @@ fn origin_restart_takes_a_fresh_identity_for_its_claims() { /// Node b learned that `live/x` ended, then lost every route and collected the /// origin. A session still delivering older messages brings the claim back, /// but only until the withdrawal queued behind it lands. Without collection, -/// b's retained floor refuses the replay outright. -#[test] -fn stale_replay_after_collection_lasts_only_until_the_withdrawal_behind_it() { +/// b's retained floor refuses the replay outright. The path rule forgets the view +/// with the last route, so it replays either way. +fn stale_replay_after_collection_lasts_only_until_the_withdrawal_behind_it(rule: Rule) { let (origin, a, b, c) = (0, 1, 2, 3); for collect in [false, true] { - let mut sim = Sim::new(4); + let mut sim = Sim::with(rule, 4); sim.leaf(origin); sim.join(origin, a, 1); sim.join(a, b, 1); @@ -767,16 +821,18 @@ fn stale_replay_after_collection_lasts_only_until_the_withdrawal_behind_it() { if collect { sim.collect(b, 7); } - // The held ROUTE, then the snapshot carrying `live/x`. - for _ in 0..2 { + // The held ROUTE, then the snapshot carrying `live/x`: after SYNCED under the + // path rule, whose stream opens with it. + for _ in 0..2 + usize::from(sim.paths()) { assert!(sim.deliver(a, b)); } - assert_eq!(sim.shows(b, 7, "live/x").is_some(), collect); + let forgot = collect || sim.paths(); + assert_eq!(sim.shows(b, 7, "live/x").is_some(), forgot); assert!(sim.deliver(a, b), "the withdrawal"); assert_eq!(sim.shows(b, 7, "live/x"), None); sim.release(a, b); assert!(sim.run()); - let replayed: &[Kind] = if collect { + let replayed: &[Kind] = if forgot { &[Kind::Start, Kind::End, Kind::Start, Kind::End] } else { &[Kind::Start, Kind::End] @@ -789,10 +845,9 @@ fn stale_replay_after_collection_lasts_only_until_the_withdrawal_behind_it() { /// Holds follow feasible advertisements only, so they cannot prop each other /// up. When the origin dies, every node in a ring goes offline and keeps no /// route state, with no timer, and none reports an END the origin did not send. -#[test] -fn holds_retire_once_nothing_can_recover() { +fn holds_retire_once_nothing_can_recover(rule: Rule) { let origin = 4; - let mut sim = Sim::new(5); + let mut sim = Sim::with(rule, 5); for relay in 0..4 { sim.join(relay, (relay + 1) % 4, 1); } @@ -807,18 +862,20 @@ fn holds_retire_once_nothing_can_recover() { for relay in 0..4 { assert!(sim.shown(relay).is_empty()); assert!(!sim.holds(relay, 7), "relay {relay}"); - assert!(!sim.remembers(relay, 7), "relay {relay} drained its distance"); + assert!( + sim.paths() || !sim.remembers(relay, 7), + "relay {relay} drained its distance" + ); } assert_eq!(false_ends(&sim.events), 0); - assert_eq!(sim.retained().pending, 0); + assert_eq!(sim.pending(), 0); } /// Seqnos and revisions are varints. An origin never advertises the last /// value: a bump, a claim change, or a request that would reach it rotates the /// identity instead, so every node lets the old one go and sees the claims /// again under the new one. Nothing wraps. -#[test] -fn exhausted_counters_rotate_the_identity() { +fn exhausted_counters_rotate_the_identity(rule: Rule) { let (origin, a, b, c, client) = (0, 1, 2, 3, 4); let rotated = |sim: &Sim| { let fresh = sim.proven(origin); @@ -826,11 +883,11 @@ fn exhausted_counters_rotate_the_identity() { assert!(fresh <= crate::route::GENERATED, "a lite-06 hop carries it as itself"); assert_eq!(sim.shows(client, 7, "live/x"), None); assert_eq!(sim.shows(client, fresh, "live/x"), Some(true)); - assert!(sim.route(a, 7).is_none()); + assert!(!sim.routed(a, 7)); sim.check().unwrap(); }; let build = |seqno| { - let mut sim = Sim::new(5); + let mut sim = Sim::with(rule, 5); sim.leaf(origin); sim.leaf(client); sim.join(origin, a, 1); @@ -845,13 +902,16 @@ fn exhausted_counters_rotate_the_identity() { sim }; - let mut sim = build(MAX - 2); - sim.bump(origin); - assert!(sim.run()); - assert_eq!(sim.proven(origin), 7); - sim.bump(origin); - assert!(sim.run()); - rotated(&sim); + // The path rule has no seqno. + if rule == Rule::Distance { + let mut sim = build(MAX - 2); + sim.bump(origin); + assert!(sim.run()); + assert_eq!(sim.proven(origin), 7); + sim.bump(origin); + assert!(sim.run()); + rotated(&sim); + } let mut sim = build(0); sim.age(origin, MAX - 2); @@ -861,6 +921,9 @@ fn exhausted_counters_rotate_the_identity() { sim.unannounce(origin, "live/y"); assert!(sim.run()); rotated(&sim); + if sim.paths() { + return; + } // The origin moves to a farther attachment, and the starved node asks for // the last seqno. @@ -869,7 +932,7 @@ fn exhausted_counters_rotate_the_identity() { sim.connect(origin, c); assert!(sim.run()); rotated(&sim); - assert!(sim.route(a, sim.proven(origin)).is_some()); + assert!(sim.routed(a, sim.proven(origin))); } /// Node b forwards a request along its route through x, whose own route just @@ -877,11 +940,11 @@ fn exhausted_counters_rotate_the_identity() { /// the retraction lands, b moves to an equal-metric route through y and /// advertises nothing new. Keeping the request at every hop that forwarded /// it, and resending it when that hop's next hop changes, reaches the origin -/// with no timer. -#[test] -fn a_quiet_origin_is_reached_by_a_kept_request() { +/// with no timer. The path rule needs no request: n's route through b never +/// named z, the far end of the session it lost. +fn a_quiet_origin_is_reached_by_a_kept_request(rule: Rule) { let (origin, w, x, y2, y, b, n, z, client) = (0, 1, 2, 3, 4, 5, 6, 7, 8); - let mut sim = Sim::new(9); + let mut sim = Sim::with(rule, 9); sim.leaf(origin); sim.leaf(client); sim.join(origin, w, 1); @@ -897,33 +960,32 @@ fn a_quiet_origin_is_reached_by_a_kept_request() { sim.originate(origin, 7, 0); sim.announce(origin, "live/x", 0, None); assert!(sim.run()); - assert_eq!(sim.route(n, 7).and_then(|route| route.next), Some(z)); - assert_eq!(sim.route(b, 7).and_then(|route| route.next), Some(x)); + assert_eq!(sim.next_hop(n, 7), Some(z)); + assert_eq!(sim.next_hop(b, 7), Some(x)); sim.disconnect(n, z); sim.hold(x, b); sim.disconnect(w, x); assert!(sim.run()); - assert!(sim.starved(n, 7)); + assert!(sim.paths() || sim.starved(n, 7)); sim.release(x, b); assert!(sim.run()); - assert_eq!(sim.route(b, 7).and_then(|route| route.next), Some(y)); + assert_eq!(sim.next_hop(b, 7), Some(y)); assert_eq!(sim.shows(client, 7, "live/x"), Some(true)); - assert_eq!(sim.seqno(origin), 1); - assert!(sim.route(n, 7).is_some()); + assert!(sim.paths() || sim.seqno(origin) == 1); + assert!(sim.routed(n, 7)); } /// Random meshes churn sessions and hold and deliver messages out of step, /// then release every hold. Every relay that can still reach a quiet origin /// ends with a route to it, which only a seqno request can bring. -#[test] -fn sweep_requests_reach_a_quiet_origin() { +fn sweep_requests_reach_a_quiet_origin(rule: Rule) { const RELAYS: usize = 5; let mut stranded = 0; for seed in 0..300u64 { let mut rng = XorShift(0x9e37_79b9 ^ (seed * 0x1234_5678_9abc)); let nodes = RELAYS + 2; - let mut sim = Sim::new(nodes); + let mut sim = Sim::with(rule, nodes); let mut links = Vec::new(); for a in 0..RELAYS { for b in a + 1..RELAYS { @@ -999,7 +1061,7 @@ fn sweep_requests_reach_a_quiet_origin() { } stranded += reach .iter() - .filter(|&&node| node < RELAYS && sim.route(node, origin as Id).is_none()) + .filter(|&&node| node < RELAYS && !sim.routed(node, origin as Id)) .count(); } } @@ -1010,11 +1072,10 @@ fn sweep_requests_reach_a_quiet_origin() { /// route and one event per node watching it; retained state follows sessions, /// live claims, and current interest, and neither withdrawal history nor /// interest history leaves anything behind. -#[test] -fn announcement_cost_follows_sessions_and_live_claims() { +fn announcement_cost_follows_sessions_and_live_claims(rule: Rule) { const PREFIXES: [&str; 4] = ["live/a", "live/b", "live/c", "live/d"]; for (relays, claims) in [(4, 2), (8, 2), (4, 4)] { - let mut sim = Sim::new(relays + 2); + let mut sim = Sim::with(rule, relays + 2); for relay in 0..relays { sim.join(relay, (relay + 1) % relays, 1); } @@ -1034,8 +1095,9 @@ fn announcement_cost_follows_sessions_and_live_claims() { // Each relay sends to both ring neighbors, and the origin and its relay to // each other: without split horizon the relay echoes the origin's claims. - let sessions = 2 * relays + 2; - let retained = sim.retained(); + // The path rule sends no route to its own publisher. + let sessions = 2 * relays + 2 - usize::from(sim.paths()); + let retained = sim.state(); assert_eq!(retained.items, relays * claims); // The client's interest meets no claim, so it holds no route and no floor. assert_eq!(retained.floors, relays); @@ -1055,10 +1117,10 @@ fn announcement_cost_follows_sessions_and_live_claims() { sim.unannounce(origin, PREFIXES[0]); } assert!(sim.run()); - let after = sim.retained(); + let after = sim.state(); assert_eq!(after.items, relays * (claims - 1)); assert_eq!(after.floors, retained.floors, "no tombstones"); - assert_eq!(after.echoes, 0, "every ROUTE_END echo drained"); + assert_eq!(after.pending, 0, "every ROUTE_END echo drained"); sim.check().unwrap(); // The client cycles through disjoint regions, one with a live claim. @@ -1075,7 +1137,7 @@ fn announcement_cost_follows_sessions_and_live_claims() { // No relay meets the interest, so the client lets the origin go. assert!(!sim.holds(client, 7)); sim.collect(client, 7); - assert_eq!(sim.retained(), after, "interest history after {cycles} cycles"); + assert_eq!(sim.state(), after, "interest history after {cycles} cycles"); } sim.check().unwrap(); } @@ -1086,8 +1148,7 @@ fn announcement_cost_follows_sessions_and_live_claims() { /// view stays some revision of the origin's truth, the widening guard stays /// bounded, no claim ends while the origin still has it, and once churn stops /// the client shows exactly the claims its interest and the scopes admit. -#[test] -fn sweep_scoped_handoff() { +fn sweep_scoped_handoff(rule: Rule) { const CLAIMS: [&str; 5] = ["live/a/x", "live/a/y/", "live/b/z", "live/c", "live/"]; const WATCHES: [&str; 5] = ["", "live/", "live/a/", "live/b/z", "live/c"]; const SCOPES: [&str; 2] = ["live/a/", "live/b/"]; @@ -1095,7 +1156,7 @@ fn sweep_scoped_handoff() { let mut rng = XorShift(0x5c0b_ed00); let mut live_starts = 0; for scenario in 0..scenarios() { - let mut sim = Sim::new(4); + let mut sim = Sim::with(rule, 4); sim.leaf(origin); sim.leaf(client); sim.join(origin, r1, 1); @@ -1150,7 +1211,7 @@ fn sweep_scoped_handoff() { sim.settle(1); sim.check().unwrap_or_else(|err| panic!("scenario {scenario}: {err}")); assert!( - sim.retained().certified <= SCOPES.len() + 1, + sim.paths() || sim.retained().certified <= SCOPES.len() + 1, "scenario {scenario}: the widening guard grows" ); } @@ -1202,17 +1263,15 @@ const RELAYS: usize = 5; const CLAIMS: [&str; 4] = ["live/", "live/a/", "live/a/x", "live/b"]; /// Random meshes with two origins and a client under churn: claims change, -/// sessions drop and return, messages are held and replayed out of step, the -/// client changes interest, nodes collect route-less identities, and an -/// origin restarts with a fresh identity. Selected routes never loop, every -/// view is always some revision of the origin's truth, nothing revives and -/// nothing ends falsely, and once churn stops every node shows every live -/// origin's current claims and holds nothing for a dead one. -#[test] -fn sweep_handoff() { - let mut rng = XorShift(0x4a4d_0ff5); +/// sessions drop and return, relays send GOAWAY, messages are held and replayed +/// out of step, the client changes interest, nodes collect route-less +/// identities, and an origin restarts with a fresh identity. Selected routes +/// never loop, every view is always some revision of the origin's truth, +/// nothing revives and nothing ends falsely, and once churn stops every node +/// shows every live origin's current claims and holds nothing for a dead one. +fn sweep_handoff(rule: Rule) { for scenario in 0..scenarios() { - let mut sim = churn(&mut rng); + let mut sim = churn(rule, scenario); let label = format!("scenario {scenario}"); assert_eq!(revivals(&sim.events), 0, "{label}"); assert_eq!(false_ends(&sim.events), 0, "{label}"); @@ -1220,10 +1279,65 @@ fn sweep_handoff() { } } -/// Origins sit at `RELAYS` and `RELAYS + 1`, the client at `RELAYS + 2`. -fn churn(rng: &mut XorShift) -> Sim { +/// A neighbour's unchanged claims at a newer floor must displace what another +/// path taught us in between, even when that other path now comes through us. +#[test] +fn an_unchanged_snapshot_advances_the_floor_others_need() { + let (origin, primary, other) = (0, 1, 2); + let mut sim = Sim::with(Rule::PATH, 3); + sim.join(origin, primary, 1); + sim.join(primary, other, 1); + sim.join(origin, other, 3); + sim.originate(origin, 7, 0); + sim.announce(origin, "live/a", 0, None); + assert!(sim.run()); + + sim.hold(origin, primary); + sim.announce(origin, "live/b", 0, None); + assert!(sim.run()); + assert_eq!(sim.shows(other, 7, "live/b"), Some(true)); + assert_eq!(sim.shows(primary, 7, "live/b"), None); + sim.hold(other, primary); + sim.disconnect(origin, other); + sim.disconnect(origin, primary); + sim.unannounce(origin, "live/b"); + sim.release(origin, primary); + sim.connect(origin, primary); + assert!(sim.run()); + sim.release(other, primary); + assert!(sim.run()); + assert_eq!(sim.shows(other, 7, "live/b"), None, "{}", sim.path(other).dump()); + assert_eq!(sim.reaches(other, 7, &p("live/a")), Some(origin)); +} + +/// Both rules run the same external script and settle to the same claims. Their +/// different message graphs interleave snapshots differently, so reader histories +/// need not match, but neither may revive a known withdrawal or report a false END. +#[test] +fn the_rules_agree_on_every_script() { + for scenario in 0..scenarios() { + let label = format!("scenario {scenario}"); + let sims = [Rule::Distance, Rule::PATH].map(|rule| { + let mut sim = churn(rule, scenario); + assert_eq!(revivals(&sim.events), 0, "{label}: {rule:?}"); + assert_eq!(false_ends(&sim.events), 0, "{label}: {rule:?}"); + settled(&mut sim, &label); + sim + }); + for node in 0..RELAYS + 3 { + assert_eq!(sims[0].shown(node), sims[1].shown(node), "{label}: node {node}"); + } + } +} + +/// Origins sit at `RELAYS` and `RELAYS + 1`, the client at `RELAYS + 2`. The script, +/// every change made from outside, depends on `scenario` alone, so both rules run +/// the same one; which queued message arrives next is drawn apart from it. +fn churn(rule: Rule, scenario: usize) -> Sim { + let rng = &mut XorShift(0x4a4d_0ff5 ^ (scenario as u64 + 1).wrapping_mul(0x9e37_79b9_7f4a_7c15) | 1); + let mut pick = XorShift(0x5eed_0003 + scenario as u64); let (first, client) = (RELAYS, RELAYS + 2); - let mut sim = Sim::new(RELAYS + 3); + let mut sim = Sim::with(rule, RELAYS + 3); let mut links = Vec::new(); for relay in 0..RELAYS { links.push((relay, (relay + 1) % RELAYS)); @@ -1258,6 +1372,7 @@ fn churn(rng: &mut XorShift) -> Sim { sim.unannounce(first + origin, claim); } } + 4 if rng.chance(20) => sim.goaway(rng.next(RELAYS as u64) as usize), 4 => { let (a, b) = links[rng.next(links.len() as u64) as usize]; if sim.connected(a, b) { @@ -1302,7 +1417,7 @@ fn churn(rng: &mut XorShift) -> Sim { if busy.is_empty() { break; } - match busy[rng.next(busy.len() as u64) as usize] { + match busy[pick.next(busy.len() as u64) as usize] { (false, from, to) => sim.deliver(from, to), (true, from, to) => sim.deliver_ack(from, to), }; @@ -1338,7 +1453,7 @@ fn settled(sim: &mut Sim, label: &str) { let presented = sim.presented(id, sim.watching(node), sim.revision(id)); // A route reaches a node only while a claim meets its interest. assert_eq!( - sim.route(node, id).is_some(), + sim.routed(node, id), node == origin || !presented.is_empty(), "{label}: node {node}'s route to {id}" ); @@ -1347,7 +1462,16 @@ fn settled(sim: &mut Sim, label: &str) { } } let shown: BTreeSet<&(Id, Path)> = sim.shown(node).keys().collect(); - assert_eq!(shown, expected.keys().collect(), "{label}: node {node}"); + assert_eq!( + shown, + expected.keys().collect(), + "{label}: node {node}\n{}", + if sim.paths() { + sim.path(node).dump() + } else { + String::new() + } + ); for ((id, claim), shown) in sim.shown(node) { assert!(shown.available, "{label}: node {node} {id} {claim}"); assert_eq!( @@ -1359,23 +1483,22 @@ fn settled(sim: &mut Sim, label: &str) { for id in (100..110).chain(200..210).filter(|id| !live.contains(id)) { assert!(!sim.holds(node, id), "{label}: node {node} holds dead {id}"); assert!( - !sim.remembers(node, id), + sim.paths() || !sim.remembers(node, id), "{label}: node {node} keeps dead {id}'s distance" ); } } - assert_eq!(sim.retained().pending, 0, "{label}"); + assert_eq!(sim.pending(), 0, "{label}"); } /// A link flap with many broadcasts behind one origin costs ROUTE changes /// only: while the link is down no session re-sends a claim and the client /// sees no claim end or restart, and reopening the link costs one snapshot per /// direction of that session. -#[test] -fn a_link_flap_costs_routes_not_broadcasts() { +fn a_link_flap_costs_routes_not_broadcasts(rule: Rule) { const CLAIMS: usize = 100; let (origin, a, b, c, client) = (0, 1, 2, 3, 4); - let mut sim = Sim::new(5); + let mut sim = Sim::with(rule, 5); sim.leaf(origin); sim.leaf(client); sim.join(origin, a, 1); @@ -1414,16 +1537,15 @@ fn a_link_flap_costs_routes_not_broadcasts() { } fn next_hop(sim: &Sim, node: Node) -> Option { - sim.route(node, 7).and_then(|route| route.next) + sim.next_hop(node, 7) } /// A neighbor ends a route, then advertises it again on the same stream. Its /// new snapshot is complete, so a claim withdrawn in between must not survive /// from the view it sent before the end. -#[test] -fn a_restarted_route_sends_a_complete_snapshot() { +fn a_restarted_route_sends_a_complete_snapshot(rule: Rule) { let (origin, relay, node) = (0, 1, 2); - let mut sim = Sim::new(3); + let mut sim = Sim::with(rule, 3); sim.leaf(origin); sim.join(origin, relay, 1); sim.join(relay, node, 1); @@ -1442,7 +1564,7 @@ fn a_restarted_route_sends_a_complete_snapshot() { sim.unannounce(origin, "live/x"); sim.connect(origin, relay); assert!(sim.run()); - assert!(sim.route(node, 7).is_some()); + assert!(sim.routed(node, 7)); assert_eq!(sim.shows(node, 7, "live/x"), None); assert_eq!(revivals(&sim.events), 0); sim.check().unwrap(); @@ -1453,10 +1575,9 @@ fn a_restarted_route_sends_a_complete_snapshot() { /// at once. A withdrawal still queued on the old stream is discarded with it, /// and the new stream's snapshot never shows the withdrawn claim: nothing /// revives and no END is false. -#[test] -fn a_covering_claim_moves_with_a_disjoint_interest() { +fn a_covering_claim_moves_with_a_disjoint_interest(rule: Rule) { let (origin, relay, client) = (0, 1, 2); - let mut sim = Sim::new(3); + let mut sim = Sim::with(rule, 3); sim.leaf(origin); sim.leaf(client); sim.join(origin, relay, 1); @@ -1487,10 +1608,9 @@ fn a_covering_claim_moves_with_a_disjoint_interest() { /// that relay holds its claims uncertified. When the scoped relay ends the route /// and restarts it, its new snapshot is complete: a claim it carried before the /// end and dropped while the route was down must not survive at the receiver. -#[test] -fn a_restarted_floorless_route_drops_what_it_no_longer_carries() { +fn a_restarted_floorless_route_drops_what_it_no_longer_carries(rule: Rule) { let (origin, local, x, z) = (0, 1, 2, 4); - let mut sim = two_scoped_uplinks(); + let mut sim = two_scoped_uplinks(rule); sim.announce(origin, "live/a/x", 0, None); // Keeps the route up once `live/a/x` goes. sim.announce(origin, "live/a/y", 0, None); @@ -1523,10 +1643,9 @@ fn a_restarted_floorless_route_drops_what_it_no_longer_carries() { /// relay 3, which asked only for `b`. While 4's own distance sets 0 aside, 0 moves /// between two equal routes and says so. That change of hops moves no certifier, so /// 4 lets go of nothing, where pruning on it lost the root claim for good. -#[test] -fn a_change_of_hops_alone_lets_go_of_no_claim() { +fn a_change_of_hops_alone_lets_go_of_no_claim(rule: Rule) { let (origin, other) = (6, 7); - let mut sim = Sim::new(8); + let mut sim = Sim::with(rule, 8); sim.join(other, 4, 1); sim.join(other, 5, 1); sim.join(origin, 3, 1); @@ -1550,9 +1669,141 @@ fn a_change_of_hops_alone_lets_go_of_no_claim() { let hops = sim.stats.hop_updates; sim.disconnect(origin, 3); assert!(sim.run()); - assert!(sim.stats.hop_updates > hops, "a route changed its hops alone"); + // The path rule names the session too, so the route through 3 ends instead. + assert!( + sim.paths() || sim.stats.hop_updates > hops, + "a route changed its hops alone" + ); assert_eq!(sim.shows(4, 7, ""), Some(true)); sim.disconnect(2, 3); assert!(sim.run()); assert_eq!(sim.shows(4, 7, ""), Some(true)); } + +/// A publisher joined to two relays that are joined to each other. +fn triangle(rule: Rule, ab: SimLink) -> Sim { + let (origin, a, b) = (0, 1, 2); + let mut sim = Sim::with(rule, 3); + sim.join(origin, a, 1); + sim.join(origin, b, 1); + sim.link(a, b, ab, ab); + sim.originate(origin, 7, 0); + sim.announce(origin, "x", 0, None); + assert!(sim.run()); + sim +} + +/// Both relays hear the publisher's GOAWAY before either hears the other move. +/// Ranking only its own drain, each would take the other's path, find itself on +/// the reply, and go back, forever. Every path drains, so both stay direct. +fn a_publishers_goaway_settles_between_its_neighbours(rule: Rule) { + let (origin, a, b) = (0, 1, 2); + let mut sim = triangle(rule, SimLink::cost(1)); + sim.goaway(origin); + assert!(sim.settle(100)); + for relay in [a, b] { + assert_eq!(sim.toward(relay, 7), Some(Some(origin)), "relay {relay}"); + assert_eq!(sim.shows(relay, 7, "x"), Some(true), "relay {relay}"); + } +} + +/// Each relay prefers the link to the other. Ranking only its own link's priority, +/// each would take the other's path and flip back forever. A path ranks by the +/// lowest priority on it, so the preferred link lifts neither and both stay direct. +fn mutually_preferred_links_settle(rule: Rule) { + let (origin, a, b) = (0, 1, 2); + let sim = triangle(rule, SimLink::cost(1).priority(1)); + for relay in [a, b] { + assert_eq!(sim.toward(relay, 7), Some(Some(origin)), "relay {relay}"); + } +} + +/// A relay leaves a draining neighbour for a costlier path that does not drain, +/// and that path, not draining itself, is what it offers on. The distance rule +/// keeps the edge on the draining neighbour until a seqno refresh instead. +#[test] +fn goaway_moves_a_relay_to_a_path_that_does_not_drain() { + let (origin, near, far, relay, edge) = (0, 1, 2, 3, 4); + let mut sim = Sim::with(Rule::PATH, 5); + sim.join(origin, near, 1); + sim.join(origin, far, 1); + sim.join(relay, near, 1); + sim.join(relay, far, 2); + sim.join(edge, relay, 1); + sim.join(edge, near, 3); + sim.originate(origin, 7, 0); + sim.announce(origin, "x", 0, None); + assert!(sim.run()); + assert_eq!(sim.toward(relay, 7), Some(Some(near))); + assert_eq!(sim.toward(edge, 7), Some(Some(relay))); + sim.goaway(near); + assert!(sim.run()); + assert_eq!(sim.toward(relay, 7), Some(Some(far))); + assert_eq!( + sim.toward(edge, 7), + Some(Some(relay)), + "the relay's new path does not drain" + ); + assert_eq!(sim.shows(edge, 7, "x"), Some(true)); +} + +/// Three relays in a ring that is cheap one way round and dear the other, each +/// joined to `hub`, which reaches the publisher. A GOAWAY from `leaving` reaches +/// the hub's side before any ring relay hears the others re-rank. Ranking drain +/// alone, each moved onto the next relay's old, undrained offer, and a request went +/// round the ring with no session closed. The drain news casts doubt on the leaving +/// node, so no relay moves onto a stale offer through it, and every request reaches +/// the publisher after each delivery and timer. +fn goaway_around_a_ring(sim: &mut Sim, hub: Node, leaving: Node) { + let (origin, ring) = (0, [1, 2, 3]); + for relay in ring { + sim.join(hub, relay, 1); + } + for (i, &relay) in ring.iter().enumerate() { + sim.link(relay, ring[(i + 1) % 3], SimLink::cost(1), SimLink::cost(2)); + } + sim.originate(origin, 7, 0); + sim.announce(origin, "x", 0, None); + assert!(sim.run()); + sim.goaway(leaving); + for _ in 0..10_000 { + for relay in ring { + assert_eq!(sim.reaches(relay, 7, &p("x")), Some(origin), "relay {relay}"); + } + // News from outside the ring first: every ring relay re-ranks before it hears + // the others do. + let busy = sim.busy(); + let next = busy.iter().find(|(from, _)| !ring.contains(from)).or(busy.first()); + if let Some(&(from, to)) = next { + sim.deliver(from, to); + } else if sim.deadline().is_some() { + sim.wait(HOP); + } else { + break; + } + } + assert_eq!(sim.pending(), 0); + for relay in ring { + assert_eq!(sim.toward(relay, 7), Some(Some(hub)), "relay {relay}"); + assert_eq!(sim.shows(relay, 7, "x"), Some(true), "relay {relay}"); + } +} + +/// The publisher itself leaves, and each ring relay hears it directly. +#[test] +fn a_goaway_in_a_ring_never_loops_a_request() { + let mut sim = Sim::with(Rule::PATH, 4); + goaway_around_a_ring(&mut sim, 0, 0); + // Each relay remembers the publisher drains while a route it hears names it. + assert_eq!(sim.kept().draining, 3); +} + +/// A relay two hops from the ring leaves: the ring hears of it only from the hub. +#[test] +fn a_goaway_two_hops_from_a_ring_never_loops_a_request() { + let (origin, leaving, hub) = (0, 4, 5); + let mut sim = Sim::with(Rule::PATH, 6); + sim.join(origin, leaving, 1); + sim.join(leaving, hub, 1); + goaway_around_a_ring(&mut sim, hub, leaving); +} diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index 6c47635252b..85f7f5d9667 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -1,9 +1,13 @@ -//! The routing contract's fixtures, run against [`Router`] in a simulated network. +//! The routing contract's fixtures, run in a simulated network against either +//! rule: [`Router`], which sessions run today, or [`path::Router`], the rule the +//! routes epic's Plan item 27 picked. //! //! [`Sim`] wires one router per node through ordered, holdable message queues, -//! so a fixture can deliver messages out of step and replay them late. Claims -//! are written with a trailing `/` where the contract means a wildcard; every -//! lite-07 claim already covers the paths below it, so the slash is dropped. +//! so a fixture can deliver messages out of step and replay them late. Each round +//! of deliveries takes [`HOP`] on a mocked clock, and a quiet network jumps to the +//! next doubt timer. Claims are written with a trailing `/` where the contract +//! means a wildcard; every lite-07 claim already covers the paths below it, so +//! the slash is dropped. mod cache; mod compare; @@ -12,16 +16,162 @@ mod paths; mod residual; mod routing; mod rtt; +mod scope; use std::{ collections::{BTreeMap, BTreeSet, VecDeque}, sync::Arc, + time::Duration, }; +use kio::clock::Instant; + +pub(super) use super::path::{Doubt, Lift}; use super::*; pub(super) type Node = usize; +/// How long one round of deliveries takes on the mocked clock. +pub(super) const HOP: Duration = Duration::from_millis(25); + +/// Which route rule every node of a [`Sim`] runs. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) enum Rule { + /// [`Router`]: Babel's distance and seqno, which sessions run today. + Distance, + /// [`path::Router`], with its doubt. + Path(Doubt), +} + +impl Rule { + /// The path rule as the model checks it. + pub const PATH: Self = Self::Path(Doubt::DEFAULT); + /// The comparison rule lifting doubt once no heard route names the node. + pub const EARLY: Self = Self::Path(Doubt { + lift: Lift::Unnamed, + ..Doubt::DEFAULT + }); + + pub fn as_str(self) -> &'static str { + match self { + Self::Distance => "distance", + Self::Path(Doubt { + lift: Lift::Unnamed, .. + }) => "path+early", + Self::Path(Doubt { lift: Lift::Timer, .. }) => "path", + } + } +} + +/// Run each fixture named once per rule, as a test per rule: the rule sessions run +/// today, and the one that replaces it. +macro_rules! both { + ($($name:ident),* $(,)?) => { + mod distance { + $(#[test] fn $name() { super::$name(super::Rule::Distance) })* + } + mod path { + $(#[test] fn $name() { super::$name(super::Rule::PATH) })* + } + }; +} +pub(super) use both; + +/// One node's router, under either rule. +enum Relay { + Distance(Router), + Path(path::Router), +} + +impl Relay { + fn new(rule: Rule, hop: Id, seed: u64) -> Self { + match rule { + Rule::Distance => Self::Distance(Router::new(hop, seed)), + Rule::Path(doubt) => Self::Path(path::Router::new(hop, seed, doubt)), + } + } + + fn interest(&self) -> u32 { + match self { + Self::Distance(router) => router.interest(), + Self::Path(router) => router.interest(), + } + } + + fn declared(&self) -> Option { + match self { + Self::Distance(router) => router.declared(), + Self::Path(router) => router.declared(), + } + } + + fn revision(&self) -> Option { + match self { + Self::Distance(router) => router.revision(), + Self::Path(router) => router.revision(), + } + } + + /// The declared identity's own claims. + fn own(&self, id: Id) -> Option<&View> { + match self { + Self::Distance(router) => router.view(id), + Self::Path(router) => router.view(id, &Path::empty()), + } + } + + fn events(&mut self) -> Vec> { + // The simulator reads full views directly. Consume change notifications + // alongside events, as a driver does, so they retain no script history. + match self { + Self::Distance(router) => { + drop(router.changed()); + router.events() + } + Self::Path(router) => { + drop(router.changed()); + router.events() + } + } + } + + fn holds(&self, id: Id) -> bool { + match self { + Self::Distance(router) => router.holds(id), + Self::Path(router) => router.holds(id), + } + } + + fn shown(&self) -> &BTreeMap<(Id, Path), Shown> { + match self { + Self::Distance(router) => router.shown(), + Self::Path(router) => router.shown(), + } + } + + fn touched(&self) -> usize { + match self { + Self::Distance(router) => router.touched(), + Self::Path(router) => router.touched(), + } + } + + fn deadline(&self) -> Option { + match self { + Self::Distance(_) => None, + Self::Path(router) => router.deadline(), + } + } + + /// Every view of another publisher's claims, by the identity it files them under. + fn views(&self) -> Vec<(Id, &View)> { + match self { + Self::Distance(router) => router.views().collect(), + Self::Path(router) => router.views().map(|((id, _), view)| (*id, view)).collect(), + } + } +} + /// A claim's attributes, plus the revision its life began at so fixtures can /// tell a revival from a fresh announcement. #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -100,6 +250,7 @@ impl SimLink { #[derive(Clone, Debug)] enum Msg { + /// A [`Router`] route. Route { id: Id, reach: Option, @@ -114,6 +265,25 @@ enum Msg { id: Id, seqno: Seqno, }, + /// A [`path::Router`] route, on the stream answering interest generation `interest`. + Path { + id: Id, + prefix: Path, + reach: Option, + interest: u32, + }, + /// A [`path::Router`] route's claims. + Claims { + id: Id, + prefix: Path, + changes: Vec<(Path, Option)>, + floor: Option, + interest: u32, + }, + Dead(path::Dead), + Synced { + interest: u32, + }, } #[derive(Clone, Debug, PartialEq, Eq)] @@ -129,10 +299,28 @@ pub(super) struct SimEvent { pub withdrawn: bool, } +/// What either rule retains, summed over every node. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(super) struct State { + /// Claims in views of other publishers. + pub items: usize, + /// One per view under the path rule. + pub floors: usize, + /// Claims on routes heard. + pub heard: usize, + /// Claims on routes sent. + pub sent: usize, + /// What waits on another node: requests and echoes, or dead hops and doubts. + pub pending: usize, +} + /// Messages sent, as protocol counts. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] pub(super) struct Stats { + /// Route messages: ROUTE, DEAD, and echoed ROUTE_ENDs. pub route_msgs: usize, + /// Routes those messages name: a DEAD counts itself and each route it moved. + pub route_items: usize, /// Route messages that changed nothing but the hops: what naming them costs. pub hop_updates: usize, pub claim_msgs: usize, @@ -151,7 +339,9 @@ struct Change { } pub(super) struct Sim { - nodes: Vec>, + rule: Rule, + nodes: Vec, + now: Instant, leaf: Vec, watch: Vec, restarts: Vec, @@ -169,16 +359,28 @@ pub(super) struct Sim { requested: BTreeSet<(Node, Node, Id, Seqno)>, /// The last finite route each node sent each neighbor, to spot a change of hops alone. routed: BTreeMap<(Node, Node, Id), Advert>, + /// The same for [`path::Router`], by prefix. + offered: BTreeMap<(Node, Node, Id, Path), path::Offer>, pub stats: Stats, + /// The work of routers a restart replaced. + spent: usize, pub events: Vec, } impl Sim { + /// `nodes` nodes running the rule sessions run today. pub fn new(nodes: usize) -> Self { + Self::with(Rule::Distance, nodes) + } + + /// `nodes` nodes running `rule`. + pub fn with(rule: Rule, nodes: usize) -> Self { Self { + rule, nodes: (0..nodes) - .map(|node| Router::new(hop(node, 0), node as u64 + 1)) + .map(|node| Relay::new(rule, hop(node, 0), node as u64 + 1)) .collect(), + now: Instant::ZERO, leaf: vec![false; nodes], watch: vec![Path::empty(); nodes], restarts: vec![0; nodes], @@ -192,11 +394,45 @@ impl Sim { history: BTreeMap::new(), requested: BTreeSet::new(), routed: BTreeMap::new(), + offered: BTreeMap::new(), stats: Stats::default(), + spent: 0, events: Vec::new(), } } + pub fn rule(&self) -> Rule { + self.rule + } + + /// Whether the nodes run [`path::Router`]. + pub fn paths(&self) -> bool { + matches!(self.rule, Rule::Path(_)) + } + + /// `node`'s [`Router`]. + pub fn babel(&self, node: Node) -> &Router { + match &self.nodes[node] { + Relay::Distance(router) => router, + Relay::Path(_) => panic!("node {node} runs the path rule"), + } + } + + fn babel_mut(&mut self, node: Node) -> &mut Router { + match &mut self.nodes[node] { + Relay::Distance(router) => router, + Relay::Path(_) => panic!("node {node} runs the path rule"), + } + } + + /// `node`'s [`path::Router`]. + pub fn path(&self, node: Node) -> &path::Router { + match &self.nodes[node] { + Relay::Path(router) => router, + Relay::Distance(_) => panic!("node {node} runs the distance rule"), + } + } + /// A node that sends only identities it serves, like a browser. pub fn leaf(&mut self, node: Node) { self.leaf[node] = true; @@ -241,15 +477,46 @@ impl Sim { self.up.insert(Self::pair(a, b)); for (x, y) in [(a, b), (b, a)] { let link = self.config(x, y); - self.nodes[x].connect(y, link).expect("a positive cost"); + let named = hop(y, self.restarts[y]); + match &mut self.nodes[x] { + Relay::Distance(router) => router.connect(y, link), + Relay::Path(router) => router.connect(y, named, link), + } + .expect("a positive cost"); } for (x, y) in [(a, b), (b, a)] { - let watch = self.watch[y].clone(); - self.nodes[x].peer_interest(y, watch).unwrap(); + self.ask(y, x); } self.flush(); } + /// `node` asks `peer` for announcements under its interest. Under the path rule a + /// node asks a neighbour publishing under a scoped grant for only what the scope + /// covers of it, and for nothing where they are disjoint. + fn ask(&mut self, node: Node, peer: Node) { + let watch = self.watch[node].clone(); + let scope = self.links[node][peer] + .and_then(|side| side.grant) + .map(|grant| p(grant.scope)); + let ask = match scope { + None => Some(watch.clone()), + Some(scope) if within(&scope, &watch) => Some(scope), + Some(scope) if within(&watch, &scope) => Some(watch.clone()), + Some(_) => None, + }; + let absent = ask.is_none(); + match &mut self.nodes[peer] { + Relay::Distance(router) => router.peer_interest(node, watch), + Relay::Path(router) => router.peer_interest(node, ask), + } + .unwrap(); + // No replacement stream will send SYNCED when the grant is disjoint. + // The receiver already knows it cancelled this part of its interest. + if absent && let Relay::Path(router) = &mut self.nodes[node] { + router.synced(peer, router.interest()); + } + } + pub fn connected(&self, a: Node, b: Node) -> bool { self.up.contains(&Self::pair(a, b)) } @@ -261,7 +528,32 @@ impl Sim { self.queues.remove(&(x, y)); self.acks.remove(&(x, y)); self.held.remove(&(x, y)); - self.nodes[y].disconnect([x]); + match &mut self.nodes[y] { + Relay::Distance(router) => router.disconnect([x]), + Relay::Path(router) => router.disconnect([x], self.now), + } + } + self.flush(); + } + + /// Every session of `node` closes at once, as a node shutting down closes them. + pub fn isolate(&mut self, node: Node) { + let peers: Vec = self.peers(node).collect(); + for &peer in &peers { + self.up.remove(&Self::pair(node, peer)); + for key in [(node, peer), (peer, node)] { + self.queues.remove(&key); + self.acks.remove(&key); + self.held.remove(&key); + } + match &mut self.nodes[peer] { + Relay::Distance(router) => router.disconnect([node]), + Relay::Path(router) => router.disconnect([node], self.now), + } + } + match &mut self.nodes[node] { + Relay::Distance(router) => router.disconnect(peers), + Relay::Path(router) => router.disconnect(peers, self.now), } self.flush(); } @@ -276,9 +568,15 @@ impl Sim { self.declared[node] = None; let restarts = self.restarts[node]; let seed = fnv("restart", &[node as u64, u64::from(restarts)]); - self.nodes[node] = Router::new(hop(node, restarts), seed); + if let Relay::Path(router) = &self.nodes[node] { + self.spent += router.work(); + } + self.nodes[node] = Relay::new(self.rule, hop(node, restarts), seed); let watch = self.watch[node].clone(); - self.nodes[node].watch(watch); + match &mut self.nodes[node] { + Relay::Distance(router) => router.watch(watch), + Relay::Path(router) => router.watch(watch), + }; self.flush(); } @@ -296,8 +594,12 @@ impl Sim { (0..self.nodes.len()).filter(move |&peer| self.links[node][peer].is_some() && self.connected(node, peer)) } + /// `node` starts originating `id`, at `seqno` under the distance rule. pub fn originate(&mut self, node: Node, id: Id, seqno: Seqno) { - self.nodes[node].originate(id, seqno); + match &mut self.nodes[node] { + Relay::Distance(router) => router.originate(id, seqno), + Relay::Path(router) => router.originate(id), + } self.declared[node] = Some(id); self.history.entry(id).or_default(); self.flush(); @@ -310,17 +612,20 @@ impl Sim { epoch: None, since: 0, }; - self.nodes[node].forge(id, seqno, Path::empty(), item); + self.babel_mut(node).forge(id, seqno, Path::empty(), item); self.flush(); } pub fn bump(&mut self, node: Node) { - self.nodes[node].bump().unwrap(); + self.babel_mut(node).bump().unwrap(); self.flush(); } pub fn age(&mut self, node: Node, rev: Rev) { - self.nodes[node].age(rev); + match &mut self.nodes[node] { + Relay::Distance(router) => router.age(rev), + Relay::Path(router) => router.age(rev), + } self.flush(); } @@ -330,7 +635,7 @@ impl Sim { } pub fn seqno(&self, node: Node) -> Seqno { - self.nodes[node].seqno().expect("not an origin") + self.babel(node).seqno().expect("not an origin") } pub fn announce(&mut self, node: Node, claim: &str, seed: Metric, epoch: Option) { @@ -339,17 +644,25 @@ impl Sim { let id = router.declared().expect("not an origin"); let rev = router.revision().unwrap() + 1; let since = router - .view(id) + .own(id) .and_then(|view| view.items.get(&claim)) .map_or(rev, |item| item.since); let item = Item { seed, epoch, since }; - self.nodes[node].announce(claim.clone(), item).unwrap(); + match &mut self.nodes[node] { + Relay::Distance(router) => router.announce(claim.clone(), item), + Relay::Path(router) => router.announce(claim.clone(), item), + } + .unwrap(); self.record(node, claim, Some(item)); } pub fn unannounce(&mut self, node: Node, claim: &str) { let claim = p(claim); - self.nodes[node].unannounce(claim.clone()).unwrap(); + match &mut self.nodes[node] { + Relay::Distance(router) => router.unannounce(claim.clone()), + Relay::Path(router) => router.unannounce(claim.clone()), + } + .unwrap(); self.record(node, claim, None); } @@ -368,10 +681,13 @@ impl Sim { pub fn watch(&mut self, node: Node, prefix: &str) { let prefix = p(prefix); self.watch[node] = prefix.clone(); - self.nodes[node].watch(prefix.clone()); + match &mut self.nodes[node] { + Relay::Distance(router) => router.watch(prefix), + Relay::Path(router) => router.watch(prefix), + }; let peers: Vec = self.peers(node).collect(); for peer in peers { - self.nodes[peer].peer_interest(node, prefix.clone()).unwrap(); + self.ask(node, peer); } self.flush(); } @@ -381,32 +697,43 @@ impl Sim { } pub fn goaway(&mut self, node: Node) { + let now = self.now; let peers: Vec = self.peers(node).collect(); for peer in peers { - self.nodes[peer].goaway(node); + match &mut self.nodes[peer] { + Relay::Distance(router) => router.goaway(node), + Relay::Path(router) => router.goaway(node, now), + } } self.flush(); } pub fn collect(&mut self, node: Node, id: Id) { - self.nodes[node].collect(id); + match &mut self.nodes[node] { + Relay::Distance(router) => router.collect(id), + Relay::Path(router) => router.collect(id), + } self.flush(); } /// `node` measures its link to `peer` at `ms`, or nothing. pub fn rtt(&mut self, node: Node, peer: Node, ms: Option) { - self.nodes[node].rtt(peer, ms.map(std::time::Duration::from_millis)); + let rtt = ms.map(Duration::from_millis); + match &mut self.nodes[node] { + Relay::Distance(router) => router.rtt(peer, rtt), + Relay::Path(router) => router.rtt(peer, rtt), + } self.flush(); } /// What `node` last sent `peer` for `id`. pub fn sent(&self, node: Node, peer: Node, id: Id) -> Option { - self.nodes[node].sent.get(&(id, peer)).cloned() + self.babel(node).sent.get(&(id, peer)).cloned() } /// The streams `node` sends a route for `id` on. pub fn sending(&self, node: Node, id: Id) -> usize { - Router::::of(&self.nodes[node].sent, id).count() + Router::::of(&self.babel(node).sent, id).count() } pub fn hold(&mut self, from: Node, to: Node) { @@ -422,16 +749,41 @@ impl Sim { let Some(msg) = self.queues.get_mut(&(from, to)).and_then(VecDeque::pop_front) else { return false; }; - let router = &mut self.nodes[to]; - match msg { - Msg::Route { id, reach } => router.route(from, id, reach), - Msg::Batch { - id, - changes, - floor, - interest, - } => router.batch(from, id, changes, floor, interest), - Msg::Request { id, seqno } => router.request(from, id, seqno), + let now = self.now; + match (&mut self.nodes[to], msg) { + (Relay::Distance(router), Msg::Route { id, reach }) => router.route(from, id, reach), + ( + Relay::Distance(router), + Msg::Batch { + id, + changes, + floor, + interest, + }, + ) => router.batch(from, id, changes, floor, interest), + (Relay::Distance(router), Msg::Request { id, seqno }) => router.request(from, id, seqno), + ( + Relay::Path(router), + Msg::Path { + id, + prefix, + reach, + interest, + }, + ) => router.route(from, id, prefix, reach, interest), + ( + Relay::Path(router), + Msg::Claims { + id, + prefix, + changes, + floor, + interest, + }, + ) => router.batch(from, id, prefix, changes, floor, interest), + (Relay::Path(router), Msg::Dead(dead)) => router.dead(from, dead, now), + (Relay::Path(router), Msg::Synced { interest }) => router.synced(from, interest), + (_, msg) => panic!("node {to} cannot read {msg:?}"), } self.stats.delivered += 1; self.flush(); @@ -443,7 +795,7 @@ impl Sim { let Some(id) = self.acks.get_mut(&(from, to)).and_then(VecDeque::pop_front) else { return false; }; - self.nodes[to].echo(from, id); + self.babel_mut(to).echo(from, id); self.stats.delivered += 1; self.flush(); true @@ -473,13 +825,27 @@ impl Sim { .collect() } - /// Deliver round-robin until every unheld queue is empty. False if it never - /// settles within a generous budget, which is how oscillation shows up. + /// Deliver round-robin until every unheld queue is empty and no doubt timer is + /// left. False if it never settles within a generous budget, which is how + /// oscillation shows up. pub fn run(&mut self) -> bool { self.settle(10_000) } + /// Up to `rounds` rounds of [`Self::run`]. pub fn settle(&mut self, rounds: usize) -> bool { + self.rounds(rounds, true) + } + + /// Deliver round-robin until every unheld queue is empty, firing only the doubt + /// timers that come due meanwhile. + pub fn flow(&mut self) -> bool { + self.rounds(10_000, false) + } + + /// Each round delivers the head of every unheld queue and takes [`HOP`]. A quiet + /// round jumps to the next doubt timer if `wait`, and ends otherwise. + fn rounds(&mut self, rounds: usize, wait: bool) -> bool { for _ in 0..rounds { let ready: Vec<(Node, Node)> = self.busy().into_iter().filter(|key| !self.held.contains(key)).collect(); let acks: Vec<(Node, Node)> = self @@ -488,7 +854,14 @@ impl Sim { .filter(|key| !self.held.contains(key)) .collect(); if ready.is_empty() && acks.is_empty() { - return true; + match self.deadline().filter(|_| wait) { + Some(at) => { + self.now = self.now.max(at); + self.tick(); + continue; + } + None => return true, + } } for (from, to) in ready { self.deliver(from, to); @@ -496,14 +869,55 @@ impl Sim { for (from, to) in acks { self.deliver_ack(from, to); } + self.now = self.now.saturating_add(HOP); + self.tick(); } false } + /// The next doubt timer on any node. + pub fn deadline(&self) -> Option { + self.nodes.iter().filter_map(Relay::deadline).min() + } + + /// Let `elapsed` pass, firing each doubt timer due by then. + pub fn wait(&mut self, elapsed: Duration) { + self.now = self.now.saturating_add(elapsed); + self.tick(); + } + + /// Fire every doubt timer due by now. + fn tick(&mut self) { + let now = self.now; + for relay in &mut self.nodes { + if let Relay::Path(router) = relay + && router.deadline().is_some_and(|at| at <= now) + { + router.timeout(now); + } + } + self.flush(); + } + /// Move every router's output onto the queues and log its reader events. fn flush(&mut self) { for node in 0..self.nodes.len() { - for out in self.nodes[node].drain() { + if self.paths() { + self.flush_paths(node); + } else { + self.flush_routes(node); + } + self.flush_events(node); + } + } + + fn flush_routes(&mut self, node: Node) { + { + let outs = match &mut self.nodes[node] { + Relay::Distance(router) => router.drain(), + Relay::Path(_) => unreachable!(), + }; + for out in outs { let (to, msg) = match out { Out::Route { to, id, reach } => { // A route names every node it crossed once, the sender last. @@ -524,6 +938,7 @@ impl Sim { self.routed.remove(&(node, to, id)); } self.stats.route_msgs += 1; + self.stats.route_items += 1; (to, Msg::Route { id, reach }) } Out::Batch { to, id, changes, floor } => { @@ -549,12 +964,99 @@ impl Sim { } Out::Echo { to, id } => { self.stats.route_msgs += 1; + self.stats.route_items += 1; self.acks.entry((node, to)).or_default().push_back(id); continue; } }; self.queues.entry((node, to)).or_default().push_back(msg); } + } + } + + fn flush_paths(&mut self, node: Node) { + let outs = match &mut self.nodes[node] { + Relay::Path(router) => router.drain(), + Relay::Distance(_) => unreachable!(), + }; + for out in outs { + let (to, msg) = match out { + path::Out::Route { to, id, prefix, reach } => { + // A path names every node it crossed once, the sender last. + let key = (node, to, id, prefix.clone()); + if let Some(path::Reach::Finite(offer)) = &reach { + let me = hop(node, self.restarts[node]); + assert_eq!( + offer.path.last().map(|hop| hop.node), + Some(me), + "node {node}'s path for {id}" + ); + let named: BTreeSet = offer.path.iter().map(|hop| hop.node).collect(); + assert_eq!(named.len(), offer.path.len(), "node {node} sent {id} round a loop"); + let was = self.offered.insert(key, offer.clone()); + let same = |was: path::Offer| { + path::Offer { + path: offer.path.clone(), + ..was + } == *offer + }; + self.stats.hop_updates += usize::from(was.is_some_and(same)); + } else { + self.offered.remove(&key); + } + self.stats.route_msgs += 1; + self.stats.route_items += 1; + let interest = self.nodes[to].interest(); + ( + to, + Msg::Path { + id, + prefix, + reach, + interest, + }, + ) + } + path::Out::Batch { + to, + id, + prefix, + changes, + floor, + } => { + self.stats.claim_msgs += 1; + self.stats.claim_items += changes.len(); + let interest = self.nodes[to].interest(); + ( + to, + Msg::Claims { + id, + prefix, + changes, + floor, + interest, + }, + ) + } + path::Out::Dead { to, dead } => { + self.stats.route_msgs += 1; + self.stats.route_items += 1 + match &dead { + path::Dead::Closed { moved, .. } => moved.len(), + path::Dead::Draining { .. } => 0, + }; + (to, Msg::Dead(dead)) + } + path::Out::Synced { to } => { + let interest = self.nodes[to].interest(); + (to, Msg::Synced { interest }) + } + }; + self.queues.entry((node, to)).or_default().push_back(msg); + } + } + + fn flush_events(&mut self, node: Node) { + { // A rotated origin starts a fresh history from the claims it carried. let declared = self.nodes[node].declared(); if self.declared[node].is_some() && declared != self.declared[node] { @@ -562,7 +1064,7 @@ impl Sim { let router = &self.nodes[node]; let rev = router.revision().unwrap(); let changes = router - .view(id) + .own(id) .unwrap() .items .iter() @@ -594,25 +1096,89 @@ impl Sim { } } + /// Routes, prefixes, and streams visited across every node, under the path rule. + pub fn work(&self) -> usize { + self.spent + (0..self.nodes.len()).map(|node| self.path(node).work()).sum::() + } + /// Identities recomputed across every node. pub fn touched(&self) -> usize { - self.nodes.iter().map(Router::touched).sum() + self.nodes.iter().map(Relay::touched).sum() } pub fn route(&self, node: Node, id: Id) -> Option> { - self.nodes[node].selected(id) + self.babel(node).selected(id) } pub fn export_route(&self, node: Node, id: Id) -> Option> { - self.nodes[node].export(id) + self.babel(node).export(id) + } + + /// The neighbour `node` routes `id` through: under the path rule, the widest + /// live route's. + pub fn next_hop(&self, node: Node, id: Id) -> Option { + match &self.nodes[node] { + Relay::Distance(router) => router.selected(id)?.next, + Relay::Path(router) => (router.routes(id).find(|(_, route)| route.offer().is_some()))?.1.next(), + } + } + + /// The neighbour `node` routes `id` through, `Some(None)` at its publisher, or + /// `None` with no route: under the path rule, no live one. + pub fn toward(&self, node: Node, id: Id) -> Option> { + match &self.nodes[node] { + Relay::Distance(router) => router.selected(id).map(|route| route.next), + Relay::Path(router) => { + (router.routes(id).find(|(_, route)| route.offer().is_some())).map(|(_, route)| route.next()) + } + } + } + + /// `peer` notices its session to `node` closed, which `node` has yet to. + pub fn sever(&mut self, node: Node, peer: Node) { + self.up.remove(&Self::pair(node, peer)); + for key in [(node, peer), (peer, node)] { + self.queues.remove(&key); + self.acks.remove(&key); + self.held.remove(&key); + } + match &mut self.nodes[peer] { + Relay::Distance(router) => router.disconnect([node]), + Relay::Path(router) => router.disconnect([node], self.now), + } + self.flush(); + } + + /// Whether `node` has a route for `id`: under the path rule, a live one. + pub fn routed(&self, node: Node, id: Id) -> bool { + match &self.nodes[node] { + Relay::Distance(router) => router.selected(id).is_some(), + Relay::Path(router) => router.routes(id).any(|(_, route)| route.offer().is_some()), + } + } + + /// The node a request from `node`'s reader for `claim` of `id` ends at, following + /// [`path::Router::forward`] hop by hop: `None` where it stops short of a publisher. + pub fn reaches(&self, node: Node, id: Id, claim: &Path) -> Option { + let (mut at, mut id, mut from) = (node, id, None); + for _ in 0..=self.nodes.len() { + match self.path(at).forward(id, claim, from.take())? { + path::Forward::Here => return Some(at), + path::Forward::Next { next, named, prefix } => { + from = Some((at, prefix)); + (at, id) = (next, named); + } + } + } + panic!("a request for {id} {claim:?} from node {node} loops"); } pub fn starved(&self, node: Node, id: Id) -> bool { - self.nodes[node].starved(id) + self.babel(node).starved(id) } pub fn grounded(&self, node: Node, id: Id) -> bool { - self.nodes[node].grounded(id) + self.babel(node).grounded(id) } pub fn holds(&self, node: Node, id: Id) -> bool { @@ -620,19 +1186,23 @@ impl Sim { } pub fn remembers(&self, node: Node, id: Id) -> bool { - self.nodes[node].remembers(id) + self.babel(node).remembers(id) } + /// The floors `node` keeps for `id`: one per view under the path rule. pub fn floors(&self, node: Node, id: Id) -> usize { - self.nodes[node].floors(id) + match &self.nodes[node] { + Relay::Distance(router) => router.floors(id), + Relay::Path(router) => router.views().filter(|((key, _), _)| *key == id).count(), + } } pub fn identities(&self, node: Node) -> usize { - self.nodes[node].identities() + self.babel(node).identities() } pub fn aliases(&self, node: Node) -> usize { - self.nodes[node].aliases() + self.babel(node).aliases() } pub fn shown(&self, node: Node) -> &BTreeMap<(Id, Path), Shown> { @@ -640,14 +1210,64 @@ impl Sim { } pub fn shows(&self, node: Node, id: Id, claim: &str) -> Option { - self.nodes[node].shows(id, &p(claim)) + let claim = p(claim); + self.shown(node).get(&(id, claim)).map(|shown| shown.available) + } + + /// What either rule retains, summed over every node, in the terms they share. + pub fn state(&self) -> State { + match self.rule { + Rule::Distance => { + let r = self.retained(); + State { + items: r.items, + floors: r.floors, + heard: r.heard, + sent: r.sent, + pending: r.echoes + r.pending, + } + } + Rule::Path(_) => { + let r = self.kept(); + State { + items: r.items, + floors: r.views, + heard: r.heard, + sent: r.sent, + pending: r.dead + r.doubt, + } + } + } + } + + /// What waits on another node: requests and echoes under the distance rule, dead + /// hops and doubts under the path rule. + pub fn pending(&self) -> usize { + self.state().pending + } + + /// [`path::Router`]'s retained state summed over every node. + pub fn kept(&self) -> path::Retained { + let mut out = path::Retained::default(); + for node in 0..self.nodes.len() { + let r = self.path(node).retained(); + out.items += r.items; + out.views += r.views; + out.heard += r.heard; + out.sent += r.sent; + out.routes += r.routes; + out.dead += r.dead; + out.doubt += r.doubt; + out.draining += r.draining; + } + out } /// Retained state summed over every node. pub fn retained(&self) -> Retained { let mut out = Retained::default(); - for router in &self.nodes { - let r = router.retained(); + for node in 0..self.nodes.len() { + let r = self.babel(node).retained(); out.items += r.items; out.floors += r.floors; out.heard += r.heard; @@ -830,16 +1450,66 @@ pub(super) fn false_ends(events: &[SimEvent]) -> usize { .count() } -/// Every node's next-hop chain toward `id` ends without revisiting a node. +/// Every node's next-hop chain toward `id` ends without revisiting a node. Under the +/// path rule a chain may come round only through a stale route, one whose next hop +/// has moved since offering it: the model's `LoopsAreStale`. pub(super) fn assert_loop_free(sim: &Sim, nodes: usize, id: Id) { + if sim.paths() { + let claims: BTreeSet = (0..nodes) + .flat_map(|node| { + sim.path(node).routes(id).map(|(prefix, _)| prefix.clone()).chain( + sim.path(node) + .shown() + .keys() + .filter_map(move |(key, claim)| (*key == id).then_some(claim.clone())), + ) + }) + .collect(); + for node in 0..nodes { + for claim in &claims { + assert_scoped_chain(sim, node, id, claim); + } + } + return; + } for node in 0..nodes { - let mut at = node; - for hop in 0.. { - let Some(next) = sim.route(at, id).and_then(|selected| selected.next) else { - break; - }; - assert!(hop <= nodes, "{id} loops from node {node}"); - at = next; + let mut chain = vec![node]; + while let Some(next) = sim.next_hop(*chain.last().expect("a node"), id) { + if let Some(start) = chain.iter().position(|&seen| seen == next) { + let cycle = &chain[start..]; + panic!("{id} loops from node {node} round {cycle:?}"); + } + chain.push(next); + } + } +} + +/// Follow an actual request, keeping the route prefix and translated identity at +/// each hop. A transient cycle must contain an offer its next hop no longer uses. +fn assert_scoped_chain(sim: &Sim, node: Node, id: Id, claim: &Path) { + let (mut at, mut key, mut from) = (node, id, None); + let mut chain: Vec<(Node, bool)> = Vec::new(); + while let Some(path::Forward::Next { next, named, prefix }) = sim.path(at).forward(key, claim, from.take()) { + let heard = sim.path(at).heard_offer(key, &prefix, next).map(|offer| &offer.path); + let current_prefix = match sim.path(next).forward(named, claim, Some((at, prefix.clone()))) { + Some(path::Forward::Next { prefix, .. }) => Some(prefix), + Some(path::Forward::Here) => Some(Path::empty()), + None => None, + }; + let current = current_prefix + .as_ref() + .and_then(|prefix| sim.path(next).route_for(named, prefix)) + .and_then(|route| route.offer()) + .map(|offer| &offer.path); + chain.push((at, heard.is_none() || heard != current)); + if let Some(start) = chain.iter().position(|(seen, _)| *seen == next) { + assert!( + chain[start..].iter().any(|(_, stale)| *stale), + "{id} {claim:?} loops from node {node} round {chain:?}" + ); + break; } + from = Some((at, prefix)); + (at, key) = (next, named); } } diff --git a/rs/moq-net/src/route/tests/paths.rs b/rs/moq-net/src/route/tests/paths.rs index 4a13ac303b9..042848be12b 100644 --- a/rs/moq-net/src/route/tests/paths.rs +++ b/rs/moq-net/src/route/tests/paths.rs @@ -1,8 +1,9 @@ -//! A prototype router that keeps a route free of loops by its path alone. +//! A prototype router that keeps a route free of loops by its path alone, carrying +//! the warm copies `cache.rs` proves. //! -//! It exists to be measured against [`Router`], which keeps Babel's distance: the -//! comparison in `compare.rs` runs one script on both. Nothing outside the tests -//! uses it. +//! The router comparisons in `compare.rs` run on [`Sim`] and the real +//! `route::path::Router`, which has no warm copies yet; this prototype exists only +//! for the copies. Nothing outside the tests uses it. //! //! A route is one per publisher and names every relay it crossed, each with the //! session that relay learned the route on. A relay uses the cheapest route whose @@ -22,7 +23,7 @@ //! and pulls each broadcast it has demand for from the cheapest of its publisher's //! route and the copies offered, which `cache.rs` proves. //! -//! Left out, since the comparison does not need them: scoped grants, qualified +//! Left out, since the copies do not need them: scoped grants, qualified //! keys, interests, priorities, upstream marks, and RTT. GOAWAY only moves //! pulls off a neighbour's copy. @@ -444,21 +445,10 @@ impl Relay { } pub fn announce(&mut self, claim: Path) { - self.change(claim, true); - } - - pub fn unannounce(&mut self, claim: Path) { - self.change(claim, false); - } - - fn change(&mut self, claim: Path, live: bool) { let id = self.own.expect("not a publisher"); let route = self.routes.get_mut(&id).unwrap(); route.rev += 1; - match live { - true => route.claims.insert(claim, route.rev), - false => route.claims.remove(&claim), - }; + route.claims.insert(claim, route.rev); self.show(id); self.advertise(id); } @@ -1429,8 +1419,6 @@ pub(super) struct PathStats { /// [`Sim`] for the prototype: one [`Relay`] per node, joined by ordered queues. pub(super) struct PathSim { pub nodes: Vec, - rules: Rules, - restarts: Vec, links: Vec>>, up: BTreeSet<(Node, Node)>, queues: BTreeMap<(Node, Node), VecDeque>, @@ -1438,8 +1426,6 @@ pub(super) struct PathSim { /// each other: true pulls, false cancels. subs: BTreeMap<(Node, Node, Cast, u64), VecDeque>, held: BTreeSet<(Node, Node)>, - /// Nodes whose process is gone and has yet to restart. - stopped: BTreeSet, pub stats: PathStats, /// Copy offers and their ends sent per broadcast. pub warm: BTreeMap, @@ -1450,14 +1436,11 @@ impl PathSim { pub fn new(nodes: usize, rules: Rules) -> Self { Self { nodes: (0..nodes).map(|node| Relay::new(hop(node, 0), rules)).collect(), - rules, - restarts: vec![0; nodes], links: vec![vec![None; nodes]; nodes], up: BTreeSet::new(), queues: BTreeMap::new(), subs: BTreeMap::new(), held: BTreeSet::new(), - stopped: BTreeSet::new(), stats: PathStats::default(), warm: BTreeMap::new(), events: Vec::new(), @@ -1488,7 +1471,7 @@ impl PathSim { self.up.insert(Self::pair(a, b)); for (x, y) in [(a, b), (b, a)] { let cost = self.links[x][y].expect("a configured link"); - let peer = hop(y, self.restarts[y]); + let peer = hop(y, 0); let forwards = !self.nodes[y].leaf; let reads = forwards && !self.nodes[y].legacy; self.nodes[x].connect(y, peer, forwards, reads, cost); @@ -1511,46 +1494,12 @@ impl PathSim { self.flush(); } - /// `peer` notices its session to `node` closed. `node`, whose process is gone, - /// does nothing. - pub fn sever(&mut self, node: Node, peer: Node) { - self.up.remove(&Self::pair(node, peer)); - for key in [(node, peer), (peer, node)] { - self.queues.remove(&key); - self.subs.retain(|(from, to, ..), _| (*from, *to) != key); - self.held.remove(&key); - } - self.nodes[peer].disconnect(node); - self.flush(); - } - fn peers(&self, node: Node) -> Vec { (0..self.nodes.len()) .filter(|&peer| self.links[node][peer].is_some() && self.connected(node, peer)) .collect() } - /// The process at `node` stops: nothing passes between it and `peer` from now on, - /// though `peer` has yet to notice, and no timer of its runs until `kill`. - pub fn mute(&mut self, node: Node, peer: Node) { - self.held.insert((node, peer)); - self.held.insert((peer, node)); - self.stopped.insert(node); - } - - /// The process at `node` exits: its sessions close and its state is lost. It says - /// nothing on the way out, so each neighbour notices and it does not. - pub fn kill(&mut self, node: Node) { - for peer in self.peers(node) { - self.sever(node, peer); - } - self.stopped.remove(&node); - self.restarts[node] += 1; - let leaf = self.nodes[node].leaf; - self.nodes[node] = Relay::new(hop(node, self.restarts[node]), self.rules); - self.nodes[node].leaf = leaf; - } - pub fn originate(&mut self, node: Node, id: Id) { self.nodes[node].originate(id); self.flush(); @@ -1561,11 +1510,6 @@ impl PathSim { self.flush(); } - pub fn unannounce(&mut self, node: Node, claim: &str) { - self.nodes[node].unannounce(p(claim)); - self.flush(); - } - /// A reader for `cast` arrives at `node`. pub fn carry(&mut self, node: Node, cast: &Cast) { self.nodes[node].carry(cast.clone()); @@ -1637,22 +1581,16 @@ impl PathSim { true } - /// One hop of delay passes at every running node. Whether any still waits on a timer. + /// One hop of delay passes at every node. Whether any still waits on a timer. pub fn tick(&mut self) -> bool { let mut waiting = false; - for (node, relay) in self.nodes.iter_mut().enumerate() { - if !self.stopped.contains(&node) { - waiting |= relay.tick(); - } + for relay in &mut self.nodes { + waiting |= relay.tick(); } self.flush(); waiting } - pub fn queued(&self, from: Node, to: Node) -> usize { - self.queues.get(&(from, to)).map_or(0, VecDeque::len) - } - /// The queues with a message to deliver that are not held. pub fn busy(&self) -> Vec<(Node, Node)> { (self.queues.iter()) @@ -1779,31 +1717,3 @@ fn a_ring_takes_the_other_way_round() { } } } - -impl Relay { - pub fn dump(&self, id: Id) -> String { - let mut out = format!( - "hop {} route {:?} dead {:?} epochs {:?} doubt {:?}", - self.hop, - self.routes.get(&id).map(|r| (r.next, r.path.is_some())), - self.dead, - self.epochs, - self.doubt - ); - for (peer, session) in &self.sessions { - out += &format!( - "\n {peer}: rx {:?} tx {:?} told {:?}", - session - .rx - .get(&id) - .map(|h| (h.state, h.path.iter().map(|x| (x.node, x.session)).collect::>())), - session - .tx - .get(&id) - .map(|s| (s.held, s.path.iter().map(|x| (x.node, x.session)).collect::>())), - session.told - ); - } - out - } -} diff --git a/rs/moq-net/src/route/tests/residual.rs b/rs/moq-net/src/route/tests/residual.rs index d6ea9c57841..a47cf104add 100644 --- a/rs/moq-net/src/route/tests/residual.rs +++ b/rs/moq-net/src/route/tests/residual.rs @@ -1,19 +1,40 @@ //! What the reviews of the hop list found on meshes where neighbours know different -//! parts of a publisher's claims, pinned as [`Router`] behaves today. +//! parts of a publisher's claims, run on both rules. //! -//! A `residual_` test asserts a bug. Each goes when the router keeps a route per -//! scope (`quest/m2/cluster-routes/scope.md`): flip its assertion there and -//! drop the prefix. The others here are rules any replacement must keep. +//! The first three were bugs of [`Router`], which the route per scope fixes: each +//! asserts the fix under the path rule and still pins the bug under the distance +//! rule, which sessions run until the switch (`quest/m2/cluster-routes/switch.md`) +//! deletes it. The others are rules either must keep. use super::*; +both!( + a_seqno_advance_storms_between_two_grants, + a_relay_behind_a_scoped_neighbour_is_blind, + a_relay_forgets_a_claim_a_fresh_relay_would_show, + a_certifier_that_returns_brings_back_what_it_certifies, + a_seqno_advance_leaves_a_relays_claims_alone, + two_narrowed_interests_settle, + two_relays_behind_a_scoped_relay_let_a_dead_claim_go, +); + +/// The cost of `node`'s route for `id` under `prefix`. +fn metric(sim: &Sim, node: Node, id: Id, prefix: &str) -> Option { + match sim.rule() { + Rule::Distance => sim.route(node, id).map(|route| route.metric), + Rule::Path(_) => (sim.path(node).route_for(id, &p(prefix))) + .and_then(|route| route.offer()) + .map(|offer| offer.metric), + } +} + /// One advance of the origin's seqno and this mesh never goes quiet again under /// round-robin delivery. Relay 5 files the origin under a qualified key for its /// scoped peers 4 and 7, whose claims there come and go with whichever copy is /// feasible, so 5 starts and ends the route to 4 on each advance and 4 asks for the -/// next seqno every time. -#[test] -fn residual_a_seqno_advance_storms_between_two_grants() { +/// next seqno every time. The path rule has no seqno: the session the advance +/// stands for closes and opens again, and the mesh goes quiet. +fn a_seqno_advance_storms_between_two_grants(rule: Rule) { let ga = Grant { subject: "ga", scope: "", @@ -23,7 +44,7 @@ fn residual_a_seqno_advance_storms_between_two_grants() { scope: "", }; let origin = 9; - let mut sim = Sim::new(10); + let mut sim = Sim::with(rule, 10); sim.link(origin, 4, SimLink::cost(1), SimLink::cost(2)); sim.join(origin, 7, 1); sim.join(1, 4, 1); @@ -41,20 +62,28 @@ fn residual_a_seqno_advance_storms_between_two_grants() { sim.watch(4, "a"); sim.watch(7, "b"); assert!(sim.settle(3000)); - assert_eq!(sim.seqno(origin), 0, "it converges with no request"); - - sim.bump(origin); - assert!(!sim.settle(500), "the storm is fixed: drop the residual"); - assert!(sim.seqno(origin) > 20, "each lap asks the origin for another seqno"); + if rule == Rule::Distance { + assert_eq!(sim.seqno(origin), 0, "it converges with no request"); + sim.bump(origin); + assert!(!sim.settle(500), "the storm is fixed: drop the residual"); + assert!(sim.seqno(origin) > 20, "each lap asks the origin for another seqno"); + return; + } + let shown = |sim: &Sim| [4, 7].map(|node| sim.shown(node).clone()); + let before = shown(&sim); + sim.disconnect(origin, 4); + sim.connect(origin, 4); + assert!(sim.settle(500)); + assert_eq!(shown(&sim), before); } /// The relay's closest neighbour asked for `a` only. A neighbour as far from the /// origin as the relay itself knows everything, and is not feasible, so the relay -/// neither shows `b/x` nor has a route to ask for it on. -#[test] -fn residual_a_relay_behind_a_scoped_neighbour_is_blind() { +/// neither shows `b/x` nor has a route to ask for it on. The path rule keeps the +/// wide neighbour's route beside the narrow one's, and a request for `b/x` takes it. +fn a_relay_behind_a_scoped_neighbour_is_blind(rule: Rule) { let (origin, narrow, wide, far, relay, tail) = (0, 1, 2, 3, 4, 5); - let mut sim = Sim::new(6); + let mut sim = Sim::with(rule, 6); sim.watch(narrow, "a"); sim.join(origin, narrow, 1); sim.join(narrow, relay, 1); @@ -69,22 +98,33 @@ fn residual_a_relay_behind_a_scoped_neighbour_is_blind() { assert_eq!(sim.shows(wide, 7, "b/x"), Some(true)); for node in [relay, tail] { assert_eq!(sim.shows(node, 7, "a/x"), Some(true), "node {node}"); - assert_eq!( - sim.shows(node, 7, "b/x"), - None, - "node {node} sees it: drop the residual" - ); + let blind = rule == Rule::Distance; + assert_eq!(sim.shows(node, 7, "b/x"), (!blind).then_some(true), "node {node}"); + } + if rule == Rule::Distance { + let routes: Vec = sim.babel(relay).routes(7).into_iter().map(|(next, ..)| next).collect(); + assert_eq!(routes, [narrow], "the wide neighbour is no route either"); + return; } - let routes: Vec = sim.nodes[relay].routes(7).into_iter().map(|(next, ..)| next).collect(); - assert_eq!(routes, [narrow], "the wide neighbour is no route either"); + assert_eq!(sim.reaches(relay, 7, &p("b/x")), Some(origin)); + let first = sim.path(relay).forward(7, &p("b/x"), None); + assert!( + matches!(first, Some(crate::route::path::Forward::Next { next, .. }) if next == wide), + "{first:?}" + ); + let first = sim.path(relay).forward(7, &p("a/x"), None); + assert!( + matches!(first, Some(crate::route::path::Forward::Next { next, .. }) if next == narrow), + "{first:?}" + ); } /// The mesh the reviews found the forgotten claim on: two origins, and six relays /// with a link to relay 0 from each of the others. `cut` leaves out the link from /// relay 0 to relay 3, and `narrow` has relay 2 ask for `b` alone from the start. -fn forgetful(cut: bool, narrow: bool) -> Sim { +fn forgetful(rule: Rule, cut: bool, narrow: bool) -> Sim { let (origin, other) = (6, 7); - let mut sim = Sim::new(8); + let mut sim = Sim::with(rule, 8); if narrow { sim.watch(2, "b"); } @@ -114,29 +154,29 @@ fn forgetful(cut: bool, narrow: bool) -> Sim { /// link to relay 3, and relay 2 then asks for `b` alone. Relay 0 is left routing /// through relay 2 and shows nothing of the origin, though its neighbour relay 4 /// shows the claim, and a relay that came up in the mesh as it now stands shows it -/// too. Nothing will bring the claim back. -#[test] -fn residual_a_relay_forgets_a_claim_a_fresh_relay_would_show() { - let mut sim = forgetful(false, false); +/// too. Nothing will bring the claim back. Under the path rule relay 0 shows what a +/// fresh relay would. +fn a_relay_forgets_a_claim_a_fresh_relay_would_show(rule: Rule) { + let mut sim = forgetful(rule, false, false); assert_eq!(sim.shows(0, 7, ""), Some(true)); sim.disconnect(0, 3); assert!(sim.run()); sim.watch(2, "b"); assert!(sim.run()); - let fresh = forgetful(true, true); + let fresh = forgetful(rule, true, true); for sim in [&sim, &fresh] { assert_eq!(sim.shows(4, 7, ""), Some(true)); - assert_eq!(sim.route(0, 7).map(|route| route.metric), Some(4)); + assert_eq!(metric(sim, 0, 7, ""), Some(4)); } assert_eq!(fresh.shows(0, 7, ""), Some(true)); - assert_eq!(sim.shows(0, 7, ""), None, "it shows the claim: drop the residual"); + let forgot = rule == Rule::Distance; + assert_eq!(sim.shows(0, 7, ""), (!forgot).then_some(true)); } /// A neighbour that stops certifying and returns brings back what it certified. /// The fix first tried for the forgotten claim broke the two tests below. -#[test] -fn a_certifier_that_returns_brings_back_what_it_certifies() { - let mut sim = forgetful(false, false); +fn a_certifier_that_returns_brings_back_what_it_certifies(rule: Rule) { + let mut sim = forgetful(rule, false, false); sim.watch(3, "b"); assert!(sim.run()); assert_eq!(sim.shows(4, 7, ""), Some(true)); @@ -151,11 +191,11 @@ fn a_certifier_that_returns_brings_back_what_it_certifies() { /// A neighbour at the relay's own distance is feasible for one message during a /// seqno advance. A relay that takes its view back then flaps its claims on every -/// advance. -#[test] -fn a_seqno_advance_leaves_a_relays_claims_alone() { +/// advance. The path rule has no seqno and shows the whole claim through the wide +/// neighbour, and a flap of the narrow neighbour's uplink leaves it alone. +fn a_seqno_advance_leaves_a_relays_claims_alone(rule: Rule) { let (origin, wide, narrow, relay) = (0, 1, 2, 3); - let mut sim = Sim::new(4); + let mut sim = Sim::with(rule, 4); sim.watch(narrow, "a"); sim.link(origin, wide, SimLink::cost(1), SimLink::cost(2)); sim.join(origin, narrow, 1); @@ -164,22 +204,31 @@ fn a_seqno_advance_leaves_a_relays_claims_alone() { sim.originate(origin, 7, 0); sim.announce(origin, "", 0, None); assert!(sim.run()); - assert_eq!(sim.shows(relay, 7, "a"), Some(true)); + let claim = match rule { + Rule::Distance => "a", + Rule::Path(_) => "", + }; + assert_eq!(sim.shows(relay, 7, claim), Some(true)); let before = sim.events.len(); - sim.bump(origin); - assert!(sim.run()); + match rule { + Rule::Distance => sim.bump(origin), + Rule::Path(_) => { + sim.disconnect(origin, narrow); + sim.connect(origin, narrow); + } + } + assert!(sim.settle(500)); assert!(sim.events[before..].iter().all(|event| event.node != relay)); - assert_eq!(sim.shows(relay, 7, "a"), Some(true)); - assert_eq!(sim.shows(relay, 7, ""), None); + assert_eq!(sim.shows(relay, 7, claim), Some(true)); + assert_eq!(sim.shown(relay).len(), 1); } /// The same flap makes a relay start and end its route on a neighbour's stream on /// every advance, and that neighbour asks for the next seqno each time. -#[test] -fn two_narrowed_interests_settle() { +fn two_narrowed_interests_settle(rule: Rule) { let (relay, wide, a, b, tail, origin) = (0, 1, 2, 3, 4, 5); - let mut sim = Sim::new(6); + let mut sim = Sim::with(rule, 6); sim.watch(a, "a"); sim.watch(b, "b"); sim.join(origin, a, 1); @@ -191,18 +240,17 @@ fn two_narrowed_interests_settle() { sim.link(tail, a, SimLink::cost(4), SimLink::cost(1)); sim.originate(origin, 7, 0); sim.announce(origin, "", 0, None); - assert!(sim.settle(3000), "never settles: seqno {}", sim.seqno(origin)); - assert!(sim.seqno(origin) <= 1); + assert!(sim.settle(3000), "never settles"); + assert!(sim.paths() || sim.seqno(origin) <= 1); } /// Two relays behind a relay whose only way to the origin asked for `a`. One of /// them knew everything through a dear link of its own, which closes, and the /// origin then withdraws `b/x`. Neither may keep it alive through the other: a rule /// that believes any neighbour whose route did not cross the relay does. -#[test] -fn two_relays_behind_a_scoped_relay_let_a_dead_claim_go() { +fn two_relays_behind_a_scoped_relay_let_a_dead_claim_go(rule: Rule) { let (origin, narrow, relay, left, right) = (0, 1, 2, 3, 4); - let mut sim = Sim::new(5); + let mut sim = Sim::with(rule, 5); sim.watch(narrow, "a"); sim.join(origin, narrow, 1); sim.join(narrow, relay, 1); diff --git a/rs/moq-net/src/route/tests/routing.rs b/rs/moq-net/src/route/tests/routing.rs index 397c2dc3679..05d0ea4ae74 100644 --- a/rs/moq-net/src/route/tests/routing.rs +++ b/rs/moq-net/src/route/tests/routing.rs @@ -1416,7 +1416,7 @@ fn sweep_no_route_outgrows_a_loop_free_path() { } cut += usize::from(reached.len() <= relays); for node in 0..relays { - for &id in sim.nodes[node].rx.keys() { + for &id in sim.babel(node).rx.keys() { assert!(!sim.starved(node, id), "case {case}: node {node} waits on {id}"); } // A scoped link files the origin under another key past it. diff --git a/rs/moq-net/src/route/tests/rtt.rs b/rs/moq-net/src/route/tests/rtt.rs index 0abdd554600..07259b61ca4 100644 --- a/rs/moq-net/src/route/tests/rtt.rs +++ b/rs/moq-net/src/route/tests/rtt.rs @@ -44,7 +44,7 @@ fn each_relay_adds_its_link_to_the_sum() { assert_eq!(advertised(&sim, origin, a, 7), ms(0), "the origin's own route"); assert_eq!(advertised(&sim, a, b, 7), ms(10)); assert_eq!(advertised(&sim, b, c, 7), ms(30)); - let routes = sim.nodes[c].routes(7); + let routes = sim.babel(c).routes(7); assert_eq!(routes, [(b, 3, ms(60))]); } @@ -77,7 +77,7 @@ fn an_unknown_link_ends_the_sum() { assert!(sim.run()); assert_eq!(advertised(&sim, a, b, 7), ms(10)); assert_eq!(advertised(&sim, b, c, 7), None); - assert_eq!(sim.nodes[c].routes(7)[0].2, None); + assert_eq!(sim.babel(c).routes(7)[0].2, None); } /// The metric selects; a far lower RTT through a costlier link changes nothing. diff --git a/rs/moq-net/src/route/tests/scope.rs b/rs/moq-net/src/route/tests/scope.rs new file mode 100644 index 00000000000..a1244e81352 --- /dev/null +++ b/rs/moq-net/src/route/tests/scope.rs @@ -0,0 +1,1139 @@ +//! What a route per scope keeps: [`path::Router`] on meshes where neighbours ask for +//! different prefixes, and how it compares with [`Router`] on the same scripts. + +use super::path::meet; +use super::*; + +fn offered(via: Id, session: u64) -> path::Reach { + path::Reach::Finite(path::Offer { + path: [path::Hop { node: 7, session }, path::Hop { node: via, session: 1 }].into(), + draining: false, + priority: i8::MAX, + metric: 1, + down_only: false, + rtt: None, + }) +} + +/// An interest edit keeps the newer withdrawal a narrow view already learned, +/// even while a wider neighbour has not heard it and the new streams have not arrived. +#[test] +fn narrowing_keeps_the_newest_covering_withdrawal() { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + for peer in [1, 2] { + router.connect(peer, peer as Id, Link::default()).unwrap(); + } + router.route(1, 7, p(""), Some(offered(1, 0)), 0); + router.batch(1, 7, p(""), vec![(p("a/b/x"), Some(1))], Some(1), 0); + router.route(2, 7, p("a"), Some(offered(2, 0)), 0); + router.batch(2, 7, p("a"), Vec::new(), Some(3), 0); + assert!(router.events().iter().any(|event| event.kind == Kind::End)); + assert!(!router.shown().contains_key(&(7, p("a/b/x")))); + + router.watch(p("a/b")); + assert!(!router.shown().contains_key(&(7, p("a/b/x")))); + assert!(router.events().iter().all(|event| event.kind != Kind::Start)); +} + +/// Another report for an expired hop retires its last heard copy without +/// restarting the hop's timer. +#[test] +fn repeating_an_expired_dead_hop_does_not_restart_its_timer() { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + let hop = path::Hop { node: 4, session: 1 }; + for peer in [1, 2] { + router.connect(peer, peer as Id, Link::default()).unwrap(); + let path::Reach::Finite(mut offer) = offered(peer as Id, 0) else { + unreachable!() + }; + offer.path = [ + path::Hop { node: 7, session: 0 }, + path::Hop { node: 6, session: 1 }, + hop, + path::Hop { + node: peer as Id, + session: 1, + }, + ] + .into(); + router.route(peer, 7, p(""), Some(path::Reach::Finite(offer)), 0); + } + let dead = path::Dead::Closed { + hop, + far: 6, + moved: Vec::new(), + gone: true, + }; + router.dead(1, dead.clone(), Instant::ZERO); + let expired = Instant::ZERO.saturating_add(Doubt::DEFAULT.timer); + router.timeout(expired); + assert_eq!(router.retained().dead, 1, "the second peer still names the hop"); + router.dead(2, dead, expired); + assert_eq!(router.retained().dead, 0, "the expired hop is no longer named"); +} + +#[test] +fn losing_a_wide_route_does_not_forget_a_kept_narrow_view() { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + for peer in [1, 2] { + router.connect(peer, peer as Id, Link::default()).unwrap(); + } + router.route(1, 7, p(""), Some(offered(1, 0)), 0); + router.batch(1, 7, p(""), vec![(p("a/x"), Some(1))], Some(1), 0); + router.route(2, 7, p("a"), Some(offered(2, 0)), 0); + router.batch(2, 7, p("a"), Vec::new(), Some(3), 0); + let ended = router + .events() + .into_iter() + .find(|event| event.kind == Kind::End) + .unwrap(); + router.route(1, 7, p(""), None, 0); + router.batch(2, 7, p("a"), vec![(p("a/x"), Some(2))], Some(4), 0); + let started = router + .events() + .into_iter() + .find(|event| event.kind == Kind::Start) + .unwrap(); + assert_eq!(started.forgot, ended.forgot, "the narrow view kept the withdrawal"); +} + +#[test] +fn flapping_a_far_node_does_not_extend_older_dead_hops() { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + router.connect(1, 1, Link::default()).unwrap(); + let mut now = Instant::ZERO; + let retained = (Doubt::DEFAULT.timer.as_nanos() / HOP.as_nanos()) as usize + 1; + for session in 0..3 * retained { + let path::Reach::Finite(mut offer) = offered(1, 0) else { + unreachable!() + }; + offer.path = [ + path::Hop { node: 7, session: 0 }, + path::Hop { + node: 2, + session: session as u64, + }, + path::Hop { node: 1, session: 1 }, + ] + .into(); + router.route(1, 7, p(""), Some(path::Reach::Finite(offer)), 0); + router.dead( + 1, + path::Dead::Closed { + hop: path::Hop { + node: 2, + session: session as u64, + }, + far: 7, + moved: Vec::new(), + gone: true, + }, + now, + ); + router.timeout(now); + assert!( + router.retained().dead <= retained, + "dead hop {session} extended older timers" + ); + now = now.saturating_add(HOP); + } +} + +#[test] +fn a_dead_repath_withdraws_its_claims_before_ending_the_sent_route() { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + for peer in [1, 2, 3] { + router.connect(peer, peer as Id, Link::default()).unwrap(); + } + let path::Reach::Finite(mut old) = offered(1, 0) else { + unreachable!() + }; + old.path = [ + path::Hop { node: 7, session: 0 }, + path::Hop { node: 6, session: 1 }, + path::Hop { node: 4, session: 1 }, + path::Hop { node: 1, session: 1 }, + ] + .into(); + for prefix in ["", "a"] { + router.route(1, 7, p(prefix), Some(path::Reach::Finite(old.clone())), 0); + router.batch(1, 7, p(prefix), vec![(p("a/x"), Some(1))], Some(1), 0); + } + router.peer_interest(3, Some(p("a"))).unwrap(); + // A child still names this relay, so its newer empty view is not believed yet. + let mut child = old; + child.path = [ + path::Hop { node: 7, session: 0 }, + path::Hop { node: 6, session: 1 }, + path::Hop { node: 4, session: 1 }, + path::Hop { node: 99, session: 1 }, + path::Hop { node: 2, session: 1 }, + ] + .into(); + router.route(2, 7, p("a"), Some(path::Reach::Finite(child)), 0); + router.batch(2, 7, p("a"), Vec::new(), Some(3), 0); + router.drain(); + let path::Reach::Finite(offer) = offered(2, 1) else { + unreachable!() + }; + router.dead( + 2, + path::Dead::Closed { + hop: path::Hop { node: 4, session: 1 }, + far: 6, + moved: vec![path::Moved { + id: 7, + prefix: p("a"), + offer, + }], + gone: false, + }, + Instant::ZERO, + ); + let messages = router.drain(); + let withdrawal = messages.iter().position(|message| { + matches!(message, + path::Out::Batch { to: 3, prefix, changes, floor: Some(3), .. } + if *prefix == p("a") && changes.contains(&(p("a/x"), None)) + ) + }); + assert!( + withdrawal.is_some(), + "the withdrawing floor must precede END: {messages:?}" + ); + assert!(messages[..withdrawal.unwrap()].iter().all(|message| !matches!( + message, + path::Out::Dead { to: 3, .. } | path::Out::Route { to: 3, reach: None, .. } + ))); +} + +/// A follower holds a route per prefix for one key, each through its own relay, +/// as a viewer session holding many announce prefixes does: one origin's claims +/// reach `prefixes` relays, each admitted under a grant scoped to its own prefix. +fn follower(prefixes: usize) -> (Sim, Node) { + let (origin, hub, follower) = (0, 1, 2 + prefixes); + let mut sim = Sim::with(Rule::PATH, prefixes + 3); + sim.leaf(origin); + sim.leaf(follower); + sim.join(origin, hub, 1); + for spoke in 0..prefixes { + let scope: &'static str = Box::leak(format!("live/{spoke}").into_boxed_str()); + let grant = Grant { + subject: "spokes", + scope, + }; + sim.link(2 + spoke, hub, SimLink::cost(1).grant(grant), SimLink::cost(1)); + sim.join(2 + spoke, follower, 1); + } + sim.originate(origin, 7, 0); + for spoke in 0..prefixes { + sim.announce(origin, &format!("live/{spoke}/x"), 0, None); + } + assert!(sim.run()); + let id = qualified("spokes", 7); + assert_eq!(sim.path(follower).routes(id).count(), prefixes, "a route per prefix"); + assert_eq!(sim.shown(follower).len(), prefixes); + (sim, follower) +} + +/// The routes bench's `teardown` and `refollow` cells on the path rule: a follower +/// closing every session at once, and closing one prefix's session and opening it +/// again, cost about what they held. Twice the prefixes visit about twice the routes, +/// prefixes, and streams. +#[test] +fn teardown_is_linear_in_prefixes_held() { + let teardown = |prefixes: usize| { + let (mut sim, follower) = follower(prefixes); + let before = sim.work(); + sim.isolate(follower); + assert!(sim.run()); + assert!(sim.shown(follower).is_empty()); + sim.work() - before + }; + let refollow = |prefixes: usize| { + let (mut sim, follower) = follower(prefixes); + let before = sim.work(); + sim.disconnect(2, follower); + sim.connect(2, follower); + assert!(sim.run()); + assert_eq!(sim.shown(follower).len(), prefixes); + sim.work() - before + }; + for (name, cost) in [ + ("teardown", &teardown as &dyn Fn(usize) -> usize), + ("refollow", &refollow), + ] { + let costs: Vec = [8, 16, 32].into_iter().map(cost).collect(); + for pair in costs.windows(2) { + // Quadratic work would quadruple. + assert!(pair[1] <= 5 * pair[0] / 2, "{name}: {costs:?}"); + } + } +} + +/// A relay sends each of its downstream neighbours one route through a leaving +/// node, each over its own session of that node. Hearing the node drain, hearing it +/// again from another upstream, and any later update that changes no route each cost +/// work linear in the neighbours: a scan of the node's sessions for each neighbour, +/// or a repeat that revisits every route sent through the node, would be quadratic. +/// The `cost`s sweep both axes at once, since there is a session per neighbour. +#[test] +fn drain_news_is_linear_in_neighbours() { + let cost = |peers: usize| { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + let leaving: Id = 500; + let idle = 2 * peers; + router.connect(idle, 3000, Link::default()).unwrap(); + for i in 0..peers { + let (up, down) = (i, peers + i); + router.connect(up, 1000 + i as Id, Link::default()).unwrap(); + router.connect(down, 2000 + i as Id, Link::default()).unwrap(); + let prefix = p(&format!("live/{i}")); + router.peer_interest(down, Some(prefix.clone())).unwrap(); + let path = [ + path::Hop { node: 7, session: 0 }, + path::Hop { + node: leaving, + session: i as u64 + 1, + }, + path::Hop { + node: 1000 + i as Id, + session: 1, + }, + ]; + let offer = path::Offer { + path: path.into(), + draining: false, + priority: i8::MAX, + metric: 1, + down_only: false, + rtt: None, + }; + router.route(up, 7, prefix.clone(), Some(path::Reach::Finite(offer)), 0); + router.batch(up, 7, prefix, vec![(p(&format!("live/{i}/x")), Some(1))], Some(1), 0); + } + router.drain(); + let before = router.work(); + router.dead(0, path::Dead::Draining { node: leaving }, Instant::ZERO); + let drained = router.work() - before; + let told = (router.drain().iter()) + .filter(|out| { + matches!( + out, + path::Out::Dead { + dead: path::Dead::Draining { .. }, + .. + } + ) + }) + .count(); + assert_eq!(told, peers, "each neighbour hears the drain once"); + // Every upstream relays the news: a repeat tells no one and revisits no route. + let before = router.work(); + router.dead(1, path::Dead::Draining { node: leaving }, Instant::ZERO); + let repeated = router.work() - before; + assert!(router.drain().is_empty(), "a repeated drain sends nothing"); + assert!( + repeated <= peers, + "a repeat visits each neighbour at most once: {repeated}" + ); + let before = router.work(); + router.rtt(idle, Some(Duration::from_millis(5))); + (drained, repeated, router.work() - before) + }; + let costs: Vec<(usize, usize, usize)> = [8, 16, 32].into_iter().map(cost).collect(); + for pair in costs.windows(2) { + // Quadratic work would quadruple. + assert!(pair[1].0 <= 5 * pair[0].0 / 2, "hearing the drain: {costs:?}"); + assert!(pair[1].1 <= 5 * pair[0].1 / 2, "hearing it again: {costs:?}"); + assert!(pair[1].2 <= 5 * pair[0].2 / 2, "an idle update: {costs:?}"); + } +} + +/// Adding an unrelated scope visits the same number of routes and reader claims, +/// regardless of how many sibling scopes this publisher already has. +#[test] +fn following_a_prefix_does_not_revisit_its_siblings() { + for claims in [1, 16] { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + let mut first = None; + for scope in 0..64 { + let before = router.work(); + let prefix = p(&format!("live/{scope}")); + router.connect(scope, 1, Link::default()).unwrap(); + router.route(scope, 7, prefix.clone(), Some(offered(1, 0)), 0); + let changes = (0..claims) + .map(|claim| (p(&format!("live/{scope}/{claim}")), Some(1))) + .collect(); + router.batch(scope, 7, prefix, changes, Some(1), 0); + let work = router.work() - before; + let expected = *first.get_or_insert(work); + assert_eq!( + work, expected, + "scope {scope}, {claims} claims: sibling scopes were revisited" + ); + router.events(); + router.changed(); + router.drain(); + } + } +} + +const CLAIMS: [&str; 6] = ["a/x", "a/y/", "a/", "b/x", "b/y/z", ""]; +const WATCHES: [&str; 5] = ["", "", "a/", "b/", "a/x"]; +const SCOPES: [&str; 2] = ["a/", "b/"]; +const SUBJECTS: [&str; 2] = ["ga", "gb"]; + +/// Two grants qualify the same raw key. A raw route learned from us through +/// either grant is not a child of the qualified route we advertised separately. +#[test] +fn another_grant_does_not_turn_a_raw_route_into_a_qualified_child() { + let (relay, first, second, origin) = (0, 1, 2, 3); + let mut sim = Sim::with(Rule::PATH, 4); + for peer in [first, second] { + sim.link( + relay, + peer, + SimLink::cost(2).grant(Grant { + subject: "ga", + scope: "b", + }), + SimLink::cost(2), + ); + } + sim.join(origin, relay, 2); + sim.join(origin, second, 1); + sim.originate(origin, 7, 0); + sim.announce(origin, "b/x", 0, None); + assert!(sim.run()); + let key = qualified("ga", 7); + assert_eq!(sim.reaches(relay, key, &p("b/x")), Some(origin)); + sim.disconnect(origin, second); + assert!(sim.run()); + assert_eq!(sim.pending(), 0, "{}", mesh(&sim)); + assert_eq!(sim.shows(relay, key, "b/x"), None); + assert_eq!(sim.shows(first, key, "b/x"), None); + assert_eq!(sim.reaches(relay, 7, &p("b/x")), Some(origin)); +} + +#[test] +fn widening_past_a_childs_narrower_interest_releases_the_old_scope() { + let (origin, primary, relay, child) = (0, 1, 2, 3); + let mut sim = Sim::with(Rule::PATH, 4); + sim.join(origin, primary, 1); + sim.join(primary, relay, 1); + sim.join(relay, child, 1); + sim.watch(relay, "a"); + sim.watch(child, "a/x"); + sim.originate(origin, 7, 0); + sim.announce(origin, "a/x", 0, None); + sim.announce(origin, "a/y", 0, None); + assert!(sim.run()); + sim.watch(relay, ""); + assert!(sim.run()); + assert!( + sim.path(relay).routes(7).all(|(_, route)| route.offer().is_some()), + "{}", + mesh(&sim) + ); + assert_eq!(sim.reaches(child, 7, &p("a/x")), Some(origin)); +} + +#[test] +fn overlapping_scopes_do_not_oscillate_between_covering_paths() { + let mut sim = Sim::with(Rule::PATH, 6); + for (a, b, cost) in [ + (5, 0, 1), + (0, 1, 3), + (0, 2, 2), + (1, 3, 3), + (1, 4, 1), + (2, 3, 2), + (2, 4, 3), + (3, 4, 2), + ] { + sim.join(a, b, cost); + } + for node in [2, 3, 4] { + sim.watch(node, "b"); + } + sim.originate(5, 7, 0); + sim.announce(5, "b/x", 0, None); + assert!(sim.run(), "{}", mesh(&sim)); + for node in 0..5 { + assert_eq!(sim.reaches(node, 7, &p("b/x")), Some(5)); + } +} + +/// Retiring a narrow stream must not leave its former route circulating while +/// another neighbour still asks for that prefix. +#[test] +fn widening_a_relay_does_not_oscillate_its_former_prefix() { + let mut sim = Sim::with(Rule::PATH, 5); + sim.join(0, 1, 2); + sim.join(0, 3, 2); + sim.join(1, 3, 3); + sim.join(1, 2, 3); + sim.join(2, 3, 3); + sim.leaf(4); + sim.join(4, 0, 1); + sim.join(4, 1, 1); + sim.watch(0, "a"); + sim.watch(2, "a"); + sim.originate(4, 7, 0); + sim.announce(4, "a/x", 0, None); + assert!(sim.run()); + + sim.watch(0, ""); + assert!(sim.run(), "{}", mesh(&sim)); + quiet(&sim, 4, "widened"); +} + +/// A DEAD that repaths a scoped route keeps the publisher's original key for +/// forwarding: the neighbour admitted under the grant never hears the qualified key. +#[test] +fn a_repathed_scoped_route_keeps_its_original_key() { + let (origin, primary, backup, receiver, near) = (0, 1, 2, 3, 4); + let mut sim = Sim::with(Rule::PATH, 5); + sim.leaf(origin); + sim.join(origin, near, 1); + sim.join(near, primary, 1); + sim.join(origin, backup, 3); + sim.join(primary, backup, 1); + sim.link( + receiver, + primary, + SimLink::cost(1).grant(Grant { + subject: "ga", + scope: "a", + }), + SimLink::cost(1), + ); + sim.originate(origin, 7, 0); + sim.announce(origin, "a/x", 0, None); + assert!(sim.run()); + let key = qualified("ga", 7); + assert_eq!(sim.reaches(receiver, key, &p("a/x")), Some(origin)); + sim.disconnect(near, primary); + assert!(sim.run()); + assert_eq!(sim.shows(receiver, key, "a/x"), Some(true)); + assert_eq!(sim.reaches(receiver, key, &p("a/x")), Some(origin)); +} + +/// A qualified route switches onto a raw route from its former child. The raw +/// route is a different key: the child cannot answer a hold for the qualified one. +#[test] +fn qualification_does_not_turn_another_key_into_a_child() { + let (origin, raw, relay, alternate) = (0, 1, 2, 3); + let grant = Grant { + subject: "ga", + scope: "a", + }; + let mut sim = Sim::with(Rule::PATH, 4); + sim.leaf(origin); + sim.join(origin, raw, 1); + sim.link(relay, raw, SimLink::cost(4).grant(grant), SimLink::cost(1)); + sim.link(alternate, origin, SimLink::cost(1).grant(grant), SimLink::cost(1)); + sim.join(alternate, relay, 1); + sim.originate(origin, 7, 0); + sim.announce(origin, "a/x", 0, None); + assert!(sim.run()); + let key = qualified("ga", 7); + assert_eq!(sim.reaches(raw, key, &p("a/x")), Some(origin)); + sim.disconnect(alternate, relay); + assert!(sim.run()); + assert_eq!(sim.shows(raw, key, "a/x"), None); + assert!(!sim.holds(raw, key)); + assert_eq!(sim.reaches(relay, key, &p("a/x")), Some(origin)); +} + +/// Cancelling the only stream for a grant has no new stream whose SYNCED could +/// retire its old routes. The local cancellation retires them instead. +#[test] +fn an_unwatched_grant_retires_its_previous_stream() { + let mut sim = Sim::with(Rule::PATH, 2); + sim.link( + 1, + 0, + SimLink::cost(1).grant(Grant { + subject: "ga", + scope: "b", + }), + SimLink::cost(1), + ); + sim.originate(0, 7, 0); + sim.announce(0, "b/x", 0, None); + assert!(sim.run()); + let key = qualified("ga", 7); + assert_eq!(sim.shows(1, key, "b/x"), Some(true)); + sim.watch(1, "a"); + assert!(sim.run()); + assert!(!sim.holds(1, key)); + assert!(sim.shown(1).is_empty()); + sim.watch(1, "b"); + assert!(sim.run()); + assert_eq!(sim.reaches(1, key, &p("b/x")), Some(0)); +} + +/// A snapshot already in flight must not restore a cancelled grant's routes. +#[test] +fn a_cancelled_stream_cannot_restore_a_route() { + let mut sim = Sim::with(Rule::PATH, 2); + sim.link( + 1, + 0, + SimLink::cost(1).grant(Grant { + subject: "ga", + scope: "b", + }), + SimLink::cost(1), + ); + sim.originate(0, 7, 0); + sim.announce(0, "b/x", 0, None); + // The original snapshot is still in flight when the grant is cancelled. + sim.watch(1, "a"); + assert!(sim.run()); + assert!(!sim.holds(1, qualified("ga", 7)), "{}", sim.path(1).dump()); + assert!(sim.shown(1).is_empty()); + sim.watch(1, "b"); + assert!(sim.run()); + assert_eq!(sim.reaches(1, qualified("ga", 7), &p("b/x")), Some(0)); +} + +/// The regions of each key that reach each node from `origin` along a path of +/// sessions that are up and visit no node twice: each session passes the meet of +/// what the receiver asks for and the scope its grant gives the sender, and files +/// it under the key that grant qualifies, refusing one already qualified. +fn reachable(sim: &Sim, origin: Node, id: Id) -> Vec> { + fn visit(sim: &Sim, at: Node, key: Id, region: Path, seen: &mut Vec, out: &mut [BTreeSet<(Id, Path)>]) { + out[at].insert((key, region.clone())); + seen.push(at); + for to in 0..out.len() { + let Some(side) = sim.links[to][at] else { + continue; + }; + if seen.contains(&to) || !sim.connected(at, to) { + continue; + } + let ask = match side.grant { + None => Some(sim.watching(to).clone()), + Some(grant) => meet(sim.watching(to), &p(grant.scope)), + }; + let Some(region) = ask.and_then(|ask| meet(®ion, &ask)) else { + continue; + }; + let key = match side.grant { + None => key, + Some(_) if key & QUALIFIED != 0 => continue, + Some(grant) => qualified(grant.subject, key), + }; + visit(sim, to, key, region, seen, out); + } + seen.pop(); + } + let mut out = vec![BTreeSet::new(); sim.nodes.len()]; + visit(sim, origin, id, Path::empty(), &mut Vec::new(), &mut out); + out +} + +/// The sessions up, each side's grant, and each node's interest, for a failure report. +fn mesh(sim: &Sim) -> String { + let mut out = String::new(); + for ((from, to), queue) in &sim.queues { + if !queue.is_empty() { + out += &format!(" {from} -> {to}: {} queued, next {:?}\n", queue.len(), queue.front()); + } + } + for a in 0..sim.nodes.len() { + out += &format!(" node {a} watches {:?}\n", sim.watching(a)); + out += &sim.path(a).dump(); + for b in 0..sim.nodes.len() { + if let Some(side) = sim.links[a][b] { + let up = if sim.connected(a, b) { "up" } else { "down" }; + out += &format!(" from {b}: {up} cost {} grant {:?}\n", side.cost, side.grant); + } + } + } + out +} + +/// The claims of `id` presented at each of `regions` and not inside a wider one. +fn presenting(sim: &Sim, id: Id, regions: impl IntoIterator) -> BTreeMap { + let regions: BTreeSet = regions.into_iter().collect(); + let rev = sim.revision(id); + (regions.iter()) + .filter(|region| !regions.iter().any(|wider| wider != *region && within(region, wider))) + .flat_map(|region| sim.presented(id, region, rev)) + .collect() +} + +/// At a quiet point each relay shows exactly what its live routes cover, as the +/// publisher now has it. Each route is the end of a real loop-free path, and a relay +/// shows every claim some such path brings it, under one key or another. A request +/// for each claim reaches the publisher, nothing is held or waits, and a relay keeps +/// nothing of a key no path brings it. +fn quiet(sim: &Sim, origin: Node, label: &str) { + sim.check().unwrap_or_else(|err| panic!("{label}: {err}")); + assert_eq!(sim.pending(), 0, "{label}: something waits\n{}", mesh(sim)); + let reach = reachable(sim, origin, 7); + let keys: Vec = std::iter::once(7) + .chain(SUBJECTS.iter().map(|subject| qualified(subject, 7))) + .collect(); + for node in (0..sim.nodes.len()).filter(|&node| node != origin) { + let router = sim.path(node); + let mut expected = BTreeMap::new(); + for &key in &keys { + let routes: Vec<(&Path, &path::Route)> = router.routes(key).collect(); + assert!( + routes.iter().all(|(_, route)| route.offer().is_some()), + "{label}: node {node} holds {key}\n{}{}", + router.dump(), + mesh(sim) + ); + for (prefix, _) in &routes { + assert!( + (reach[node].iter()).any(|(reached, region)| *reached == key && within(prefix, region)), + "{label}: node {node} routes {key} {prefix:?}, which no path brings it\n{}{}reach {:?}", + router.dump(), + mesh(sim), + reach[node] + ); + } + let kept = reach[node].iter().any(|(reached, _)| *reached == key); + // A neighbour may offer our own path back under a qualified key. It is + // retained to answer holds but cannot feed a route or a claim view here. + assert!( + kept || routes.is_empty(), + "{label}: node {node} routes {key}\n{}", + router.dump() + ); + assert!( + kept || router.views().all(|((id, _), _)| *id != key), + "{label}: node {node} keeps a view of {key}\n{}", + router.dump() + ); + for (claim, item) in presenting(sim, 7, routes.into_iter().map(|(prefix, _)| prefix.clone())) { + expected.insert((key, claim), item); + } + } + let shown: BTreeMap<(Id, Path), Item> = (sim.shown(node).iter()) + .map(|(key, shown)| { + assert!(shown.available, "{label}: node {node} {key:?} unavailable"); + (key.clone(), shown.item) + }) + .collect(); + assert_eq!(shown, expected, "{label}: node {node}\n{}", router.dump()); + let claims: BTreeSet<&Path> = shown.keys().map(|(_, claim)| claim).collect(); + let paths = presenting(sim, 7, reach[node].iter().map(|(_, region)| region.clone())); + for claim in paths.keys() { + assert!( + claims.contains(claim), + "{label}: node {node} shows no {claim:?} though a path brings it\n{}", + router.dump() + ); + } + for (key, claim) in shown.keys() { + assert_eq!( + sim.reaches(node, *key, claim), + Some(origin), + "{label}: node {node} {key} {claim:?}" + ); + } + } +} + +/// The review harness's scoped and two-prefix families on the path rule, beside +/// `sweep_no_route_outgrows_a_loop_free_path`: random meshes of three to five relays +/// and a publisher whose claims sit under two prefixes. In one family some sessions +/// publish under a grant scoped to one prefix, in another some relays ask for one +/// prefix, and the third has both. Links flap, the publisher's most, relays restart, +/// claims and interests change, doubt timers fire, and every message arrives in a +/// random order. Loops are only ever stale and every view is some revision of the +/// publisher's truth; at every quiet point each relay shows what the rule says and +/// nothing waits. +#[test] +fn sweep_scoped_meshes_show_what_the_rule_says() { + let mut rng = XorShift(0x5c0b_e2ed); + let (mut quiets, mut qualified_shown, mut narrowed_shown, mut cut) = (0, 0, 0, 0); + for scenario in 0..scenarios() { + let relays = 3 + rng.next(3) as usize; + let origin = relays; + let scoped = scenario % 3 != 1; + let narrowed = scenario % 3 != 0; + let mut sim = Sim::with(Rule::PATH, relays + 1); + let mut links = Vec::new(); + for a in 0..relays { + for b in a + 1..relays { + if b == a + 1 || rng.chance(50) { + let side = SimLink::cost(1 + rng.next(3)); + let grant = |rng: &mut XorShift| Grant { + subject: SUBJECTS[rng.next(2) as usize], + scope: SCOPES[rng.next(2) as usize], + }; + match (scoped && rng.chance(30), scoped && rng.chance(30)) { + (false, false) => sim.link(a, b, side, side), + (true, false) => sim.link(a, b, side.grant(grant(&mut rng)), side), + (false, true) => sim.link(a, b, side, side.grant(grant(&mut rng))), + (true, true) => { + let (ab, ba) = (grant(&mut rng), grant(&mut rng)); + sim.link(a, b, side.grant(ab), side.grant(ba)) + } + } + links.push((a, b)); + } + } + } + for _ in 0..1 + rng.next(2) { + let relay = rng.next(relays as u64) as usize; + if !sim.connected(origin, relay) { + sim.join(origin, relay, 1 + rng.next(2)); + links.insert(0, (origin, relay)); + } + } + if narrowed { + for relay in 0..relays { + if rng.chance(40) { + sim.watch(relay, WATCHES[rng.next(WATCHES.len() as u64) as usize]); + } + } + } + sim.originate(origin, 7, 0); + for claim in CLAIMS { + if rng.chance(40) { + sim.announce(origin, claim, rng.next(3), None); + } + } + + let mut quiet_at = |sim: &mut Sim, step: usize| { + assert!( + sim.run(), + "scenario {scenario} step {step} never settles\n{}", + mesh(sim) + ); + quiet(sim, origin, &format!("scenario {scenario} step {step}")); + quiets += 1; + for node in 0..relays { + let shown = sim.shown(node); + qualified_shown += usize::from(shown.keys().any(|(key, _)| key & QUALIFIED != 0)); + narrowed_shown += usize::from(!sim.watching(node).is_empty() && !shown.is_empty()); + cut += usize::from(shown.is_empty()); + } + }; + for step in 0..120 { + match rng.next(24) { + 0..=2 => { + let claim = CLAIMS[rng.next(CLAIMS.len() as u64) as usize]; + match rng.chance(60) { + true => sim.announce(origin, claim, rng.next(3), None), + false => sim.unannounce(origin, claim), + } + } + 3 if narrowed => { + let relay = rng.next(relays as u64) as usize; + sim.watch(relay, WATCHES[rng.next(WATCHES.len() as u64) as usize]); + } + 4 | 5 => { + let pick = match rng.chance(50) { + true => 0, + false => rng.next(links.len() as u64) as usize, + }; + let (a, b) = links[pick]; + match sim.connected(a, b) { + true => sim.disconnect(a, b), + false => sim.connect(a, b), + } + } + 6 if rng.chance(25) => { + let relay = rng.next(relays as u64) as usize; + sim.kill(relay); + sim.reconnect(relay); + } + 7 => sim.wait(HOP), + 8 => quiet_at(&mut sim, step), + _ => { + let busy = sim.busy(); + if !busy.is_empty() { + let (from, to) = busy[rng.next(busy.len() as u64) as usize]; + sim.deliver(from, to); + } + } + } + sim.check() + .unwrap_or_else(|err| panic!("scenario {scenario} step {step}: {err}")); + assert_loop_free(&sim, relays + 1, 7); + } + quiet_at(&mut sim, usize::MAX); + } + assert!( + quiets > 0 && qualified_shown > 0 && narrowed_shown > 0 && cut > 0, + "{quiets} quiet points, {qualified_shown} qualified, {narrowed_shown} narrowed, {cut} cut" + ); +} + +/// Relays 0 and 1 ask for "a/" alone, each nearer the other's choice of open relay. +/// Relay 3 takes its narrow route from 0, which answers it from 4's covering route, +/// and 4 takes its narrow route from 1, which answers it from 3's. When the +/// publisher's session closes and the covering routes end, each held stream would +/// fall back to the narrow route, on another path, and the holds would wait on +/// each other for good. A stream holds only the path it last sent, so they end. +#[test] +fn holds_never_close_a_cycle() { + let (origin, hub) = (5, 2); + let mut sim = Sim::with(Rule::PATH, 6); + for (a, b, cost) in [(hub, 3, 1), (hub, 4, 1), (0, 3, 2), (0, 4, 1), (1, 3, 1), (1, 4, 2)] { + sim.join(a, b, cost); + } + sim.join(origin, hub, 1); + sim.watch(0, "a/"); + sim.watch(1, "a/"); + sim.originate(origin, 7, 0); + sim.announce(origin, "a/x", 0, None); + assert!(sim.run()); + quiet(&sim, origin, "connected"); + for (relay, next) in [(3, 0), (4, 1)] { + let narrow = sim.path(relay).routes(7).find(|(prefix, _)| **prefix == p("a/")); + assert_eq!(narrow.and_then(|(_, route)| route.next()), Some(next), "relay {relay}"); + } + sim.disconnect(origin, hub); + assert!(sim.run()); + quiet(&sim, origin, "cut off"); + for node in 0..origin { + assert_eq!(sim.path(node).routes(7).count(), 0, "node {node}\n{}", mesh(&sim)); + } +} + +/// A DEAD needs a path from this peer naming both the hop and its far end. +#[test] +fn unrelated_dead_messages_do_not_allocate_or_poison_routes() { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + for peer in [1, 2] { + router.connect(peer, peer as Id, Link::default()).unwrap(); + } + let now = Instant::ZERO; + for node in 100..1100 { + router.dead( + 1, + path::Dead::Closed { + hop: path::Hop { node, session: 1 }, + far: node + 1, + moved: Vec::new(), + gone: true, + }, + now, + ); + } + assert_eq!((router.retained().dead, router.retained().doubt), (0, 0)); + router.route(1, 7, p(""), Some(offered(1, 0)), 0); + router.batch(1, 7, p(""), vec![(p("a"), Some(1))], Some(1), 0); + let hop = path::Hop { node: 1, session: 1 }; + // Another peer has no credit for this hop, even though this node heard it. + router.dead( + 2, + path::Dead::Closed { + hop, + far: 7, + moved: Vec::new(), + gone: true, + }, + now, + ); + // Nor can the advertising peer attach arbitrary far nodes to a known hop. + for far in 100..1100 { + router.dead( + 1, + path::Dead::Closed { + hop, + far, + moved: Vec::new(), + gone: true, + }, + now, + ); + } + assert_eq!((router.retained().dead, router.retained().doubt), (0, 0)); + assert!(router.shown()[&(7, p("a"))].available); + router.dead( + 1, + path::Dead::Closed { + hop, + far: 7, + moved: Vec::new(), + gone: true, + }, + now, + ); + assert_eq!(router.retained().dead, 1); + assert!(router.routes(7).next().is_none()); +} + +#[test] +fn dead_replacement_paths_obey_the_same_limit_as_route_paths() { + for length in [0, MAX_HOPS - 1, MAX_HOPS] { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + router.connect(1, 1, Link::default()).unwrap(); + router.route(1, 7, p(""), Some(offered(1, 0)), 0); + router.batch(1, 7, p(""), vec![(p("a"), Some(1))], Some(1), 0); + let offer = path::Offer { + path: (0..length) + .map(|i| path::Hop { + node: 100 + i as Id, + session: 1, + }) + .collect(), + draining: false, + priority: i8::MAX, + metric: 1, + down_only: false, + rtt: None, + }; + router.dead( + 1, + path::Dead::Closed { + hop: path::Hop { node: 1, session: 1 }, + far: 7, + moved: vec![path::Moved { + id: 7, + prefix: p(""), + offer, + }], + gone: true, + }, + Instant::ZERO, + ); + assert_eq!( + router.routes(7).any(|(_, route)| route.offer().is_some()), + length == MAX_HOPS - 1, + "replacement length {length}" + ); + if length == MAX_HOPS - 1 { + assert_eq!(router.routes(7).next().unwrap().1.offer().unwrap().path.len(), MAX_HOPS); + } + } +} + +#[test] +fn an_upstream_stream_uses_a_narrower_exportable_source() { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + for peer in [1, 2, 3] { + router + .connect( + peer, + peer as Id, + Link { + upstream: peer != 2, + ..Link::default() + }, + ) + .unwrap(); + } + router.route(1, 7, p(""), Some(offered(1, 0)), 0); + router.batch(1, 7, p(""), vec![(p("a/x"), Some(1))], Some(1), 0); + let path::Reach::Finite(mut narrow) = offered(2, 0) else { + unreachable!() + }; + narrow.metric = 2; + router.route(2, 7, p("a"), Some(path::Reach::Finite(narrow)), 0); + router.peer_interest(3, Some(p("a"))).unwrap(); + assert!(router.drain().iter().any(|msg| matches!(msg, path::Out::Route { to: 3, prefix, reach: Some(path::Reach::Finite(offer)), .. } if *prefix == p("a") && !offer.down_only && offer.path[1].node == 2))); +} + +#[test] +fn replacing_a_childs_interest_keeps_the_hold_until_its_answer() { + for interest in [None, Some(p("a"))] { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + for peer in [1, 2] { + router.connect(peer, peer as Id, Link::default()).unwrap(); + } + router.route(1, 7, p(""), Some(offered(1, 0)), 0); + router.batch(1, 7, p(""), vec![(p("a/x"), Some(1))], Some(1), 0); + router.peer_interest(2, Some(p(""))).unwrap(); + let path::Reach::Finite(mut child) = offered(2, 0) else { + unreachable!() + }; + child.path = [ + path::Hop { node: 7, session: 0 }, + path::Hop { node: 1, session: 1 }, + path::Hop { node: 99, session: 1 }, + path::Hop { node: 2, session: 1 }, + ] + .into(); + router.route(2, 7, p(""), Some(path::Reach::Finite(child)), 0); + router.disconnect([1], Instant::ZERO); + assert!(!router.shown()[&(7, p("a/x"))].available); + router.events(); + router.peer_interest(2, interest).unwrap(); + assert!( + !router.shown()[&(7, p("a/x"))].available, + "the child still owes an answer" + ); + assert!(router.events().is_empty()); + router.route(2, 7, p(""), Some(offered(2, 0)), 0); + assert!(router.shown()[&(7, p("a/x"))].available); + assert!(router.events().iter().all(|event| event.kind == Kind::Available)); + } +} + +/// An unusable covering path must not replace a narrow hold on an upstream +/// stream: the child still owes an answer about the path it learned here. +#[test] +fn an_upstream_stream_keeps_a_hold_over_an_unexportable_live_source() { + for (live, held) in [("", "a"), ("a", "")] { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + for peer in [1, 2, 3] { + router + .connect( + peer, + peer as Id, + Link { + upstream: peer != 2, + ..Link::default() + }, + ) + .unwrap(); + } + router.route(1, 7, p(live), Some(offered(1, 0)), 0); + router.batch(1, 7, p(live), vec![(p("a/x"), Some(1))], Some(1), 0); + let path::Reach::Finite(mut offer) = offered(2, 0) else { + unreachable!() + }; + offer.metric = 2; + router.route(2, 7, p(held), Some(path::Reach::Finite(offer)), 0); + router.batch(2, 7, p(held), vec![(p("a/x"), Some(1))], Some(1), 0); + router.peer_interest(3, Some(p("a"))).unwrap(); + let path::Reach::Finite(mut child) = offered(3, 0) else { + unreachable!() + }; + child.path = [ + path::Hop { node: 7, session: 0 }, + path::Hop { node: 2, session: 1 }, + path::Hop { node: 99, session: 2 }, + path::Hop { node: 3, session: 1 }, + ] + .into(); + router.route(3, 7, p("a"), Some(path::Reach::Finite(child)), 0); + router.drain(); + router.disconnect([2], Instant::ZERO); + let messages = router.drain(); + assert!( + router + .routes(7) + .any(|(prefix, route)| *prefix == p(held) && route.offer().is_none()), + "keep {held:?} while its child answers: {messages:?}" + ); + assert!( + messages.iter().any(|msg| matches!( + msg, + path::Out::Dead { + to: 3, + dead: path::Dead::Closed { gone: false, .. } + } + )), + "the child must hear a hold, not an end: {messages:?}" + ); + assert!( + !messages + .iter() + .any(|msg| matches!(msg, path::Out::Route { to: 3, reach: None, .. })) + ); + } +} diff --git a/rs/moq-net/src/route/view.rs b/rs/moq-net/src/route/view.rs index fd33433beb4..a6fc6fc2d9d 100644 --- a/rs/moq-net/src/route/view.rs +++ b/rs/moq-net/src/route/view.rs @@ -2,7 +2,7 @@ use std::{ collections::{BTreeMap, BTreeSet}, - ops::Bound, + ops::{Bound, RangeBounds}, }; use crate::Path; @@ -247,19 +247,52 @@ impl View { } } -/// The entries of `map` strictly below `prefix`, as one range: a path lies below a -/// non-empty prefix exactly when it starts with the prefix and a `/`. +/// All descendants of a prefix. Whole segments lie in `prefix/`..`prefix0`; +/// the root excludes only itself. Both map and set indexes use these bounds. +struct Descendants { + lo: String, + hi: Option, + root: bool, +} + +impl Descendants { + fn new(prefix: &Path) -> Self { + Self { + lo: if prefix.is_empty() { + String::new() + } else { + format!("{prefix}/") + }, + hi: (!prefix.is_empty()).then(|| format!("{prefix}0")), + root: prefix.is_empty(), + } + } +} + +impl RangeBounds for Descendants { + fn start_bound(&self) -> Bound<&str> { + if self.root { + Bound::Excluded(&self.lo) + } else { + Bound::Included(&self.lo) + } + } + + fn end_bound(&self) -> Bound<&str> { + self.hi.as_deref().map_or(Bound::Unbounded, Bound::Excluded) + } +} + +/// The entries of `map` strictly below `prefix`, as one range. pub(crate) fn below<'a, V>(map: &'a BTreeMap, prefix: &Path) -> impl Iterator { - let (lo, hi) = match prefix.is_empty() { - true => (String::new(), None), - false => (format!("{prefix}/"), Some(format!("{prefix}0"))), - }; - let lo = match prefix.is_empty() { - true => Bound::Excluded(lo.as_str()), - false => Bound::Included(lo.as_str()), - }; - let hi = hi.as_deref().map_or(Bound::Unbounded, Bound::Excluded); - map.range::((lo, hi)) + map.range::(Descendants::new(prefix)) +} + +/// The paths in `set` at or below `prefix`, without walking its siblings. +pub(crate) fn within_set<'a>(set: &'a BTreeSet, prefix: &Path) -> impl Iterator { + set.get(prefix) + .into_iter() + .chain(set.range::(Descendants::new(prefix))) } /// `path` and every prefix above it, the root first. diff --git a/rs/moq-net/verify/route/Blind.cfg b/rs/moq-net/verify/route/Blind.cfg index 750ac81f68d..aa709100bdb 100644 --- a/rs/moq-net/verify/route/Blind.cfg +++ b/rs/moq-net/verify/route/Blind.cfg @@ -1,6 +1,6 @@ \* Too large to check every ordering, so TLC samples it at random. \* Simulate: -simulate num=10000 -depth 120 -\* The blind relay (`residual_a_relay_behind_a_scoped_neighbour_is_blind`): +\* The blind relay (`residual::{distance,path}::a_relay_behind_a_scoped_neighbour_is_blind`): \* relay 4's nearest neighbour, relay 1, asks for "a" alone, and relay 2, \* farther from the publisher (0) through relay 3, knows everything. Relay 4, \* and relay 5 behind it, must show "bx" and reach it through 2. @@ -9,6 +9,9 @@ CONSTANTS Origin = 0 Links = {{0, 1}, {1, 4}, {0, 3}, {3, 2}, {2, 4}, {4, 5}} Dear = {} + Ring = {} + Preferred = {} + Leaving = {} Scoped = {1} Claims = {"ax", "bx"} Narrow = {"ax"} @@ -28,6 +31,7 @@ INVARIANTS SimplePaths LoopsAreStale StaleIsChased + RequestsLoopFree NoStaleReturn QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/Dead.cfg b/rs/moq-net/verify/route/Dead.cfg index a6c0d9ac55c..1e1fc4126c8 100644 --- a/rs/moq-net/verify/route/Dead.cfg +++ b/rs/moq-net/verify/route/Dead.cfg @@ -10,6 +10,9 @@ CONSTANTS Origin = 0 Links = {{0, 1}, {1, 2}, {2, 3}, {2, 4}, {3, 4}, {0, 4}} Dear = {{0, 4}} + Ring = {} + Preferred = {} + Leaving = {} Scoped = {1} Claims = {"ax", "bx"} Narrow = {"ax"} @@ -29,6 +32,7 @@ INVARIANTS SimplePaths LoopsAreStale StaleIsChased + RequestsLoopFree NoStaleReturn QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/Drain.cfg b/rs/moq-net/verify/route/Drain.cfg new file mode 100644 index 00000000000..f3da411c468 --- /dev/null +++ b/rs/moq-net/verify/route/Drain.cfg @@ -0,0 +1,37 @@ +\* The publisher (0) joined to relays 1 and 2, which are joined to each other. +\* Both relays may hear the publisher's GOAWAY before either hears the other +\* move. Ranking only its own drain, each took the other's path, found itself on +\* the reply, and went back forever, exceeding Budget. An offer carries its +\* drain, so both stay direct. One session may also close and reopen. +\* The Rust fixture is a_publishers_goaway_settles_between_its_neighbours. +CONSTANTS + Nodes = {0, 1, 2} + Origin = 0 + Links = {{0, 1}, {0, 2}, {1, 2}} + Dear = {} + Ring = {} + Preferred = {} + Leaving = {0} + Scoped = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 1 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/DrainRing.cfg b/rs/moq-net/verify/route/DrainRing.cfg new file mode 100644 index 00000000000..2698259d236 --- /dev/null +++ b/rs/moq-net/verify/route/DrainRing.cfg @@ -0,0 +1,40 @@ +\* Simulate: -simulate num=1000 -depth 150 -seed 1 +\* The publisher (0) joined to relays 1, 2, and 3, which form a ring that is +\* cheap one way round (1 to 2, 2 to 3, 3 to 1) and dear the other. The +\* publisher's GOAWAY may reach every relay before any hears the others re-rank. +\* Ranking drain alone, each moved onto the next relay's old, undrained offer, +\* and a request went round the ring with no session closed, breaking +\* RequestsLoopFree and LoopsAreStale. The DRAIN news casts doubt on the +\* publisher, so no relay takes a stale offer through it. +\* The Rust fixture is a_goaway_in_a_ring_never_loops_a_request. +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {0, 3}, {1, 2}, {2, 3}, {1, 3}} + Dear = {} + Ring = {1, 2, 3} + Preferred = {} + Leaving = {0} + Scoped = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 0 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Early.cfg b/rs/moq-net/verify/route/Early.cfg index c5f1c3f9f4a..96b9e2e89f4 100644 --- a/rs/moq-net/verify/route/Early.cfg +++ b/rs/moq-net/verify/route/Early.cfg @@ -6,6 +6,9 @@ CONSTANTS Origin = 0 Links = {{0, 1}, {1, 2}, {1, 3}, {2, 3}} Dear = {} + Ring = {} + Preferred = {} + Leaving = {} Scoped = {} Claims = {"x"} Narrow = {} @@ -25,5 +28,6 @@ INVARIANTS SimplePaths LoopsAreStale StaleIsChased + RequestsLoopFree QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/HoldCycle.cfg b/rs/moq-net/verify/route/HoldCycle.cfg new file mode 100644 index 00000000000..6e26a5171ae --- /dev/null +++ b/rs/moq-net/verify/route/HoldCycle.cfg @@ -0,0 +1,39 @@ +\* Simulate: -simulate num=2000 -depth 200 -seed 1 +\* Holds that would close a cycle (`holds_never_close_a_cycle`): the publisher +\* (0) on hub 1, open relays 2 and 3 on the hub, and relays 4 and 5 asking for +\* "a" alone, each joined to both open relays and nearer the other's choice. +\* Relay 2 takes its "a" route from 4, which answers it from 3's root route, +\* and 3 takes its "a" route from 5, which answers it from 2's. When the +\* publisher's session closes, each held stream would fall back to the "a" +\* route on another path, and the holds would wait on each other for good. +CONSTANTS + Nodes = {0, 1, 2, 3, 4, 5} + Origin = 0 + Links = {{0, 1}, {1, 2}, {1, 3}, {2, 4}, {3, 4}, {2, 5}, {3, 5}} + Dear = {{2, 4}, {3, 5}} + Ring = {} + Preferred = {} + Leaving = {} + Scoped = {4, 5} + Claims = {"ax"} + Narrow = {"ax"} + MaxRev = 1 + MaxFlaps = 1 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Lag.cfg b/rs/moq-net/verify/route/Lag.cfg index f11ccb79398..1b13967a558 100644 --- a/rs/moq-net/verify/route/Lag.cfg +++ b/rs/moq-net/verify/route/Lag.cfg @@ -7,6 +7,9 @@ CONSTANTS Origin = 0 Links = {{0, 1}, {0, 2}, {1, 3}, {2, 3}} Dear = {} + Ring = {} + Preferred = {} + Leaving = {} Scoped = {} Claims = {"ax"} Narrow = {"ax"} @@ -26,6 +29,7 @@ INVARIANTS SimplePaths LoopsAreStale StaleIsChased + RequestsLoopFree NoStaleReturn QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/LagWide.cfg b/rs/moq-net/verify/route/LagWide.cfg index 38cbc263585..4ebc496667d 100644 --- a/rs/moq-net/verify/route/LagWide.cfg +++ b/rs/moq-net/verify/route/LagWide.cfg @@ -5,6 +5,9 @@ CONSTANTS Origin = 0 Links = {{0, 1}, {0, 2}, {1, 3}, {2, 3}} Dear = {} + Ring = {} + Preferred = {} + Leaving = {} Scoped = {1} Claims = {"ax"} Narrow = {"ax"} @@ -24,6 +27,7 @@ INVARIANTS SimplePaths LoopsAreStale StaleIsChased + RequestsLoopFree NoStaleReturn QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/Loop.cfg b/rs/moq-net/verify/route/Loop.cfg index 059a32e769d..5dc279c44d1 100644 --- a/rs/moq-net/verify/route/Loop.cfg +++ b/rs/moq-net/verify/route/Loop.cfg @@ -1,15 +1,15 @@ -\* Open finding: RequestsLoopFree. The publisher (0) has a session to each of -\* relays 1, 2, and 3, which form a triangle, and all three close in turn. Each -\* relay drops its doubt on the publisher once no route names it, so it takes a -\* neighbour's route whose news has not reached that neighbour yet, and a -\* request goes round 1, 3, 2 until it does. LoopsAreStale and StaleIsChased -\* still hold. A relay that kept its doubt until the timer fires never takes -\* such a route: promote the rule here when the router does. +\* Regression: the mesh whose early-collection rule admitted a request loop +\* through relays 1, 3, and 2. Sample timer-only doubt at the original bounds; +\* promoting the old expected failure to an exhaustive pass is too costly. +\* Simulate: -simulate num=1000 -depth 150 -seed 1 CONSTANTS Nodes = {0, 1, 2, 3} Origin = 0 Links = {{0, 1}, {0, 2}, {0, 3}, {1, 2}, {2, 3}, {1, 3}} Dear = {{1, 2}} + Ring = {} + Preferred = {} + Leaving = {} Scoped = {} Claims = {"x"} Narrow = {} diff --git a/rs/moq-net/verify/route/Overlap.cfg b/rs/moq-net/verify/route/Overlap.cfg new file mode 100644 index 00000000000..aeb6101abc3 --- /dev/null +++ b/rs/moq-net/verify/route/Overlap.cfg @@ -0,0 +1,36 @@ +\* Simulate: -simulate num=1000 -depth 500 +\* Check that cheapest-covering source selection settles with every session up. +\* The former narrowest-first rule oscillated between a scoped path through +\* 2 and 3 and a cheaper covering path through 1, exceeding Budget. +\* The Rust fixture is overlapping_scopes_do_not_oscillate_between_covering_paths. +CONSTANTS + Nodes = {0, 1, 2, 3, 4, 5} + Origin = 5 + Links = {{5, 0}, {0, 1}, {0, 2}, {1, 3}, {1, 4}, {2, 3}, {2, 4}, {3, 4}} + Dear = {{0, 1}, {0, 2}, {1, 3}, {2, 3}, {2, 4}, {3, 4}} + Ring = {} + Preferred = {} + Leaving = {} + Scoped = {2, 3, 4} + Claims = {"x"} + Narrow = {"x"} + MaxRev = 1 + MaxFlaps = 0 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Prefer.cfg b/rs/moq-net/verify/route/Prefer.cfg new file mode 100644 index 00000000000..d2047f90906 --- /dev/null +++ b/rs/moq-net/verify/route/Prefer.cfg @@ -0,0 +1,36 @@ +\* The publisher (0) joined to relays 1 and 2, which each give their link to the +\* other a higher priority. Ranking only its own link's priority, each took the +\* other's path and flipped back forever, exceeding Budget. A path ranks by the +\* lowest priority on it, so both stay direct. One session may also close. The +\* Rust fixture is mutually_preferred_links_settle. +CONSTANTS + Nodes = {0, 1, 2} + Origin = 0 + Links = {{0, 1}, {0, 2}, {1, 2}} + Dear = {} + Ring = {} + Preferred = {{1, 2}} + Leaving = {} + Scoped = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 1 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Ring.cfg b/rs/moq-net/verify/route/Ring.cfg index 6eb3e4b0ba5..c8234dd7ae1 100644 --- a/rs/moq-net/verify/route/Ring.cfg +++ b/rs/moq-net/verify/route/Ring.cfg @@ -7,6 +7,9 @@ CONSTANTS Origin = 0 Links = {{0, 1}, {1, 2}, {2, 3}, {3, 0}} Dear = {} + Ring = {} + Preferred = {} + Leaving = {} Scoped = {} Claims = {"x"} Narrow = {} @@ -26,5 +29,6 @@ INVARIANTS SimplePaths LoopsAreStale StaleIsChased + RequestsLoopFree QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/RouteHandoff.cfg b/rs/moq-net/verify/route/RouteHandoff.cfg index 9f59549e023..0c5dcdd120d 100644 --- a/rs/moq-net/verify/route/RouteHandoff.cfg +++ b/rs/moq-net/verify/route/RouteHandoff.cfg @@ -7,6 +7,9 @@ CONSTANTS Origin = 0 Links = {{0, 1}, {1, 2}, {1, 3}, {2, 3}} Dear = {} + Ring = {} + Preferred = {} + Leaving = {} Scoped = {} Claims = {"x"} Narrow = {} @@ -26,6 +29,7 @@ INVARIANTS SimplePaths LoopsAreStale StaleIsChased + RequestsLoopFree NoStaleReturn QuietIsSettled Budget diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index 2034b914402..e5f2ec9ae35 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -1,16 +1,17 @@ ---------------------------- MODULE RouteHandoff ---------------------------- (***************************************************************************) (* The lite-07 route rule the routes epic's Plan item 27 picked, for one *) -(* publisher and two prefixes, as the prototype in *) -(* rs/moq-net/src/route/tests/paths.rs (`Relay` under `Rules::waits`) runs *) -(* it, extended with a route per scope. *) +(* publisher and two prefixes, as Router in *) +(* rs/moq-net/src/route/path.rs runs it, with a route per scope. *) (* *) (* A route is keyed by the prefix it covers: "root", or "a" inside it. A *) (* relay in `Scoped` asks every neighbour for "a" alone, so a stream to it *) (* carries one route, keyed "a". A route names every relay it crossed, *) (* each with the session that relay learned it on, and a relay uses the *) -(* cheapest live route whose path does not name it. There is no distance, *) -(* seqno, ROUTE_REQUEST, or echo. Instead: *) +(* best live route whose path does not name it: one crossing no draining *) +(* session, then the highest lowest link priority, then the cheapest. An *) +(* offer carries its drain; its priority and cost follow from its path. *) +(* There is no distance, seqno, ROUTE_REQUEST, or echo. Instead: *) (* *) (* - A relay that loses a session names that hop in one DEAD message per *) (* neighbour, listing which of its routes through it moved to another *) @@ -22,7 +23,7 @@ (* is the step `Lift`, which may fire at any moment. *) (* *) (* Each node's every entry point is one atomic step ending in `Update`, *) -(* as in the prototype. Messages travel on one FIFO per direction. A *) +(* as in the router. Messages travel on one FIFO per direction. A *) (* session opens and closes atomically, dropping what was in flight, and *) (* each opening is a new session with a new hop. A claim's attributes are *) (* the revision that last set them. *) @@ -34,16 +35,28 @@ CONSTANTS Origin, \* the publisher Links, \* sessions, as 2-element sets of nodes Dear, \* the links that cost 2; the rest cost 1 + Ring, \* relays round a ring in number order: each one's link to the + \* next costs 1 there and 2 at the next, overriding `Dear` + Preferred, \* the links both ends give priority 1; the rest have 0 + Leaving, \* nodes that may send GOAWAY, once on each session Scoped, \* relays that ask every neighbour for prefix "a" alone Claims, \* claim paths Narrow, \* the claims inside prefix "a" MaxRev, \* the publisher's claim changes MaxFlaps, \* session opens and closes - Patient \* TRUE: a doubt lifts only once no DEAD about its node is in flight + Patient \* TRUE: a doubt lifts only once no news about its node is in flight Relays == Nodes \ {Origin} Cost == [l \in Links |-> IF l \in Dear THEN 2 ELSE 1] +\* The relay after n round `Ring`, the lowest after the highest. +Succ(n) == LET after == {m \in Ring : m > n} + from == IF after # {} THEN after ELSE Ring + IN CHOOSE m \in from : \A k \in from : m <= k +\* What node n's link to m costs n: links round the ring cost by direction. +CostAt(n, m) == IF n \in Ring /\ m \in Ring /\ m = Succ(n) THEN 1 + ELSE IF n \in Ring /\ m \in Ring /\ n = Succ(m) THEN 2 + ELSE Cost[{n, m}] \* A claim's attributes: the revision that last set it, or None when absent. None == -1 @@ -75,24 +88,26 @@ OHop == <> \* A route a neighbour offered (`Heard`). `state` is "live", "dead" (its path \* holds a hop this node knows is dead, and the neighbour has not said what \* became of it), or "held". -NoHeard == [on |-> FALSE, path |-> <<>>, state |-> "live", rev |-> None, items |-> NoItems] -\* The route a node uses for a key (`Route`). Held: no path, and `next` is the -\* neighbour it holds under, or None on its own account. -NoRoute == [on |-> FALSE, held |-> FALSE, next |-> None, path |-> <<>>] +NoHeard == [on |-> FALSE, path |-> <<>>, drain |-> FALSE, state |-> "live", rev |-> None, items |-> NoItems] +\* The route a node uses for a key (`Route`). Held: `path` is the last live +\* route's, and `next` is the neighbour it holds under, or None on its own account. +NoRoute == [on |-> FALSE, held |-> FALSE, next |-> None, path |-> <<>>, drain |-> FALSE] \* What a node last sent a neighbour for a key (`Sent`). `src` is the key of \* the node's own route the offer reflects, kept after an end. -NoSent(k) == [on |-> FALSE, held |-> FALSE, path |-> <<>>, rev |-> None, items |-> NoItems, src |-> k] +NoSent(k) == [on |-> FALSE, held |-> FALSE, path |-> <<>>, drain |-> FALSE, rev |-> None, items |-> NoItems, src |-> k] NotShown == [item |-> None, avail |-> FALSE] \* Messages: one record shape so TLC can compare them. -Blank == [t |-> "", k |-> "root", path |-> <<>>, rev |-> None, items |-> NoItems, - hop |-> OHop, moved |-> {}, gone |-> FALSE] -OfferMsg(k, path) == [Blank EXCEPT !.t = "offer", !.k = k, !.path = path] +Blank == [t |-> "", k |-> "root", path |-> <<>>, drain |-> FALSE, rev |-> None, items |-> NoItems, + hop |-> OHop, moved |-> {}, gone |-> FALSE, node |-> None] +OfferMsg(k, path, drain) == [Blank EXCEPT !.t = "offer", !.k = k, !.path = path, !.drain = drain] HoldMsg(k) == [Blank EXCEPT !.t = "hold", !.k = k] EndMsg(k) == [Blank EXCEPT !.t = "end", !.k = k] ClaimsMsg(k, v) == [Blank EXCEPT !.t = "claims", !.k = k, !.rev = v.floor, !.items = v.items] DeadMsg(hop, moved, gone) == [Blank EXCEPT !.t = "dead", !.hop = hop, !.moved = moved, !.gone = gone] +\* The DEAD message's DRAIN entry: `node` sent GOAWAY. +DrainMsg(node) == [Blank EXCEPT !.t = "drain", !.node = node] VARIABLES up, \* open sessions @@ -129,15 +144,27 @@ Child(path, n) == Len(path) >= 2 /\ path[Len(path) - 1][1] = n \* What the route costs the last node on its path. PathCost(path) == LET F[i \in 0..Len(path)] == - IF i <= 1 THEN 0 ELSE F[i - 1] + Cost[{path[i - 1][1], path[i][1]}] + IF i <= 1 THEN 0 ELSE F[i - 1] + CostAt(path[i][1], path[i - 1][1]) IN F[Len(path)] +\* The lowest priority of a link on the path: each hop's node and the peer it +\* learned the route from. The publisher's own hop crosses no link. +PathPrio(path) == IF \E i \in DOMAIN path : i > 1 /\ {path[i][1], path[i][2]} \notin Preferred + THEN 0 ELSE 1 +\* How a route ranks, lower first: drain, then priority, then cost. Extending a +\* path never improves the first two and strictly raises the cost. +Rank(path, drain) == <> +Below(a, b) == \/ a[1] < b[1] + \/ a[1] = b[1] /\ a[2] < b[2] + \/ a[1] = b[1] /\ a[2] = b[2] /\ a[3] < b[3] Live(r) == r.on /\ ~r.held ----------------------------------------------------------------------------- -(* Selection on node n's state s (paths.rs: `select`, `pending`, `hopes`). *) +(* Selection on node n's state s (path.rs: `select`, `pending`, `hopes`). *) Doubted(s, path) == \E i \in DOMAIN path : path[i][1] \in s.doubt +\* The path crosses a node this node knows sent GOAWAY. +Drains(s, path) == \E i \in DOMAIN path : path[i][1] \in s.draining \* A route a neighbour offered that this node may use: live, and its path does \* not name this node. @@ -146,12 +173,22 @@ Usable(n, s, m, k) == \* The key of this node's own route an offer of key k to p reflects. A stream \* to a scoped relay carries one route, "a": the node's route for "a" where it -\* has a live one, else its root route, which covers "a". +\* has a live one not naming the receiver, else its root route, which covers "a". +\* A narrower path cannot outrank a better covering path: that preference can +\* make neighbours oscillate between scopes. Equal ranks prefer the narrow key. +Cheapest(s, ks) == + LET R(k) == Rank(s.rt[k].path, s.rt[k].drain) + IN CHOOSE k \in ks : \A j \in ks : + \/ Below(R(k), R(j)) + \/ R(k) = R(j) /\ (k = "a" \/ j = "root") + SrcKey(s, p, k) == IF p \notin Scoped THEN k - ELSE IF Live(s.rt["a"]) THEN "a" - ELSE IF Live(s.rt["root"]) THEN "root" - ELSE IF s.rt["a"].on THEN "a" ELSE "root" + ELSE LET live == {j \in Keys : Live(s.rt[j])} + usable == {j \in live : ~Names(s.rt[j].path, p)} + IN IF usable # {} THEN Cheapest(s, usable) + ELSE IF live # {} THEN Cheapest(s, live) + ELSE IF s.rt["a"].on THEN "a" ELSE "root" \* A neighbour that learned this node's route for k has yet to answer its loss. Pending(n, s, k) == @@ -164,7 +201,7 @@ Pending(n, s, k) == \* A neighbour offers a route that only doubt keeps this node off. Hopes(n, s, k) == \E m \in Nodes : Usable(n, s, m, k) /\ Doubted(s, s.rx[m][k].path) -\* `select` for one key: the cheapest usable route, staying on the next hop +\* `select` for one key: the best-ranked usable route, staying on the next hop \* among equals, then the lowest key. Doubt keeps a node from moving to a route, \* not on the one it uses. With none left, hold or let go. SelectKey(n, s, k, gens) == @@ -172,29 +209,47 @@ SelectKey(n, s, k, gens) == current == IF was.on THEN was.next ELSE None using == IF Live(was) THEN was.next ELSE None C == {m \in Nodes : Usable(n, s, m, k) /\ (m = using \/ ~Doubted(s, s.rx[m][k].path))} - Via(m) == PathCost(s.rx[m][k].path) + Cost[{n, m}] + Path(m) == Append(s.rx[m][k].path, <>) + Drain(m) == s.rx[m][k].drain \/ m \in s.drain \/ Drains(s, s.rx[m][k].path) + R(m) == Rank(Path(m), Drain(m)) best == CHOOSE m \in C : \A x \in C : - \/ Via(m) < Via(x) - \/ Via(m) = Via(x) /\ (m = current \/ (x # current /\ m <= x)) + \/ Below(R(m), R(x)) + \/ R(m) = R(x) /\ (m = current \/ (x # current /\ m <= x)) + \* The next hop holds on the path this route was built on. One that has + \* since sent another path may wait on this node instead: were every hold + \* in a cycle on the path it was built on, each path would extend itself. under == /\ current # None /\ s.rx[current][k].on /\ s.rx[current][k].state # "live" - /\ ~Names(s.rx[current][k].path, n) + /\ Len(was.path) > 0 + /\ s.rx[current][k].path = SubSeq(was.path, 1, Len(was.path) - 1) IN IF C # {} - THEN [on |-> TRUE, held |-> FALSE, next |-> best, - path |-> Append(s.rx[best][k].path, <>)] + THEN [on |-> TRUE, held |-> FALSE, next |-> best, path |-> Path(best), drain |-> Drain(best)] ELSE IF ~was.on THEN NoRoute ELSE IF ~under /\ ~Hopes(n, s, k) /\ ~Pending(n, s, k) THEN NoRoute - ELSE [on |-> TRUE, held |-> TRUE, next |-> IF under THEN current ELSE None, path |-> <<>>] + ELSE [on |-> TRUE, held |-> TRUE, next |-> IF under THEN current ELSE None, path |-> was.path, drain |-> FALSE] \* The publisher's own route covers everything. OwnRoutes == [k \in Keys |-> IF k = "root" - THEN [on |-> TRUE, held |-> FALSE, next |-> None, path |-> <>] + THEN [on |-> TRUE, held |-> FALSE, next |-> None, path |-> <>, drain |-> FALSE] ELSE NoRoute] +\* Plan every live route before assigning children to the routes they reflect. +\* Keep the old keys as held placeholders until those assignments decide holds. +PlanKey(n, s, k, gens) == + LET live == SelectKey(n, s, k, gens) + IN IF Live(live) THEN live + ELSE IF s.rt[k].on THEN [s.rt[k] EXCEPT !.held = TRUE, !.drain = FALSE] + ELSE NoRoute + Select(n, s, gens) == IF n = Origin THEN s - ELSE [s EXCEPT !.rt = [k \in Keys |-> SelectKey(n, s, k, gens)]] + ELSE LET planned == [s EXCEPT !.rt = [k \in Keys |-> PlanKey(n, s, k, gens)]] + sourced == [planned EXCEPT !.tx = [p \in Nodes |-> [k \in Keys |-> + IF planned.tx[p][k].on + THEN [planned.tx[p][k] EXCEPT !.src = SrcKey(planned, p, k)] + ELSE planned.tx[p][k]]]] + IN [sourced EXCEPT !.rt = [k \in Keys |-> SelectKey(n, sourced, k, gens)]] ----------------------------------------------------------------------------- (* What a node knows of the claims: one view per key, the newest floor any *) @@ -233,7 +288,7 @@ ShownNow(s) == avail |-> Live(s.rt["root"]) \/ (c \in Narrow /\ Live(s.rt["a"]))]] ----------------------------------------------------------------------------- -(* What a node sends (paths.rs: `want`, `advertise`, `report`). *) +(* What a node sends (path.rs: `want`, `advertise`, `report`). *) \* What this node wants a neighbour p to hold for key k: a path, a hold, an \* end, or to keep what it was sent while the next hop this node holds under @@ -241,12 +296,15 @@ ShownNow(s) == Want(n, s, k, p) == LET src == SrcKey(s, p, k) r == s.rt[src] - IN IF ~r.on THEN [kind |-> "end", path |-> <<>>, src |-> src] - ELSE IF ~r.held THEN [kind |-> "path", path |-> r.path, src |-> src] + IN IF ~r.on THEN [kind |-> "end", path |-> <<>>, drain |-> FALSE, src |-> src] + ELSE IF ~r.held THEN [kind |-> "path", path |-> r.path, drain |-> r.drain, src |-> src] ELSE IF r.next = p THEN [kind |-> IF Pending(n, s, src) \/ Hopes(n, s, src) THEN "keep" ELSE "end", - path |-> <<>>, src |-> src] - ELSE [kind |-> "hold", path |-> <<>>, src |-> src] + path |-> <<>>, drain |-> FALSE, src |-> src] + \* A hold names no path, so a stream holds only the path it last sent. + ELSE IF s.tx[p][k].on /\ s.tx[p][k].path # r.path + THEN [kind |-> "end", path |-> <<>>, drain |-> FALSE, src |-> src] + ELSE [kind |-> "hold", path |-> <<>>, drain |-> FALSE, src |-> src] \* `advertise` for one key on the stream to p. Advertise1(n, s, p, k) == @@ -261,11 +319,11 @@ Advertise1(n, s, p, k) == IF t.on THEN [tx |-> NoSent(t.src), out |-> <>] ELSE [tx |-> t, out |-> <<>>] [] OTHER -> - LET offer == ~t.on \/ t.held \/ t.path # w.path + LET offer == ~t.on \/ t.held \/ t.path # w.path \/ t.drain # w.drain claims == t.rev # v.floor \/ t.items # v.items - IN [tx |-> [on |-> TRUE, held |-> FALSE, path |-> w.path, + IN [tx |-> [on |-> TRUE, held |-> FALSE, path |-> w.path, drain |-> w.drain, rev |-> v.floor, items |-> v.items, src |-> w.src], - out |-> (IF offer THEN <> ELSE <<>>) + out |-> (IF offer THEN <> ELSE <<>>) \o (IF claims THEN <> ELSE <<>>)] Advertise(n, s, ups) == @@ -288,7 +346,7 @@ Report1(n, s, p, hop) == LET t == s.tx[p] through == {k \in Keys : t[k].on /\ HasHop(t[k].path, hop)} w == [k \in Keys |-> Want(n, s, k, p)] - moved == {<> : k \in {j \in through : w[j].kind = "path"}} + moved == {<> : k \in {j \in through : w[j].kind = "path"}} holds == {k \in through : w[k].kind = "hold" /\ ~t[k].held} \* The neighbour this node learned the route from takes silence to mean \* it holds under it, so it is told outright. @@ -298,7 +356,8 @@ Report1(n, s, p, hop) == first == hop \notin s.told[p] tx1 == [k \in Keys |-> IF k \notin through THEN t[k] - ELSE CASE w[k].kind = "path" -> [t[k] EXCEPT !.path = w[k].path, !.held = FALSE, !.src = w[k].src] + ELSE CASE w[k].kind = "path" -> [t[k] EXCEPT !.path = w[k].path, !.drain = w[k].drain, + !.held = FALSE, !.src = w[k].src] [] w[k].kind = "hold" -> [t[k] EXCEPT !.held = TRUE] [] w[k].kind = "end" -> NoSent(t[k].src) [] OTHER -> t[k]] @@ -306,7 +365,7 @@ Report1(n, s, p, hop) == THEN <> \o (IF gone THEN <<>> ELSE Each(parents, LAMBDA k : <>) \o Each(ends, LAMBDA k : <>)) - ELSE Each({k \in through : w[k].kind = "path"}, LAMBDA k : <>) + ELSE Each({k \in through : w[k].kind = "path"}, LAMBDA k : <>) \o Each(holds, LAMBDA k : <>) \o Each(ends, LAMBDA k : <>) IN IF through = {} THEN [tx |-> t, told |-> s.told[p], out |-> <<>>] @@ -318,23 +377,36 @@ Report(n, s, ups, hop) == !.told = [p \in Nodes |-> IF p \in Peers(n, ups) THEN r[p].told ELSE s.told[p]]], out |-> [p \in Nodes |-> IF p \in Peers(n, ups) THEN r[p].out ELSE <<>>]] -\* `collect`: forget each doubt no route names any more, each dead hop no route -\* names once the timer it started has fired, and each told hop no route sent -\* names. +\* `report_draining`: tell each neighbour p that was sent a route through draining +\* node x, once while such a route stays sent. +ReportDrain(n, s, ups, x) == + LET tell == {p \in Peers(n, ups) : + /\ x \notin s.toldDrain[p] + /\ \E k \in Keys : s.tx[p][k].on /\ Names(s.tx[p][k].path, x)} + IN [s |-> [s EXCEPT !.toldDrain = [p \in Nodes |-> + IF p \in tell THEN s.toldDrain[p] \cup {x} ELSE s.toldDrain[p]]], + out |-> [p \in Nodes |-> IF p \in tell THEN <> ELSE <<>>]] + +\* `collect`: forget each dead hop no route names once its timer has fired, +\* and each told hop no sent route names. Doubt lasts until Lift fires. Collect(s) == LET heard == {s.rx[x[1]][x[2]].path : x \in {y \in Nodes \X Keys : s.rx[y[1]][y[2]].on}} IN [s EXCEPT - !.dead = {h \in s.dead : h[2] \in s.wait \/ \E p \in heard : HasHop(p, h)}, - !.doubt = {x \in s.doubt : \E p \in heard : Names(p, x)}, + !.dead = {h \in s.dead : h \in s.expiry \/ \E p \in heard : HasHop(p, h)}, + !.draining = {x \in s.draining : x \in s.doubt \/ \E p \in heard : Names(p, x)}, !.told = [m \in Nodes |-> {h \in s.told[m] : - \E k \in Keys : s.tx[m][k].on /\ HasHop(s.tx[m][k].path, h)}]] + \E k \in Keys : s.tx[m][k].on /\ HasHop(s.tx[m][k].path, h)}], + !.toldDrain = [m \in Nodes |-> {x \in s.toldDrain[m] : + \E k \in Keys : s.tx[m][k].on /\ Names(s.tx[m][k].path, x)}]] ----------------------------------------------------------------------------- (* `update`: select, learn, report a dead hop if any, advertise, collect, *) (* show. Returns the node's new state, its messages per neighbour, and its *) (* reader's events. *) -NoReport == [on |-> FALSE, hop |-> OHop] +NoReport == [on |-> FALSE, hop |-> OHop, node |-> None] +DeadReport(hop) == [NoReport EXCEPT !.on = TRUE, !.hop = hop] +DrainReport(x) == [NoReport EXCEPT !.node = x] \* The reader's events: changes to `ended` and the counters. A claim leaving \* while the node still holds a covering route is an END, else it went offline. @@ -363,6 +435,7 @@ Update(n, s0, ov, ups, gens, eb, rep, quiet) == LET s1 == Select(n, s0, gens) s2 == Know(n, s1, ov) r == IF rep.on THEN Report(n, s2, ups, rep.hop) + ELSE IF rep.node # None THEN ReportDrain(n, s2, ups, rep.node) ELSE [s |-> s2, out |-> [p \in Nodes |-> <<>>]] a == Advertise(n, r.s, ups) s3 == Collect(a.s) @@ -399,7 +472,7 @@ Commit(rs, q0) == /\ learnt' = [n \in Nodes |-> IF n \in ns THEN learnt[n] \cup rs[n].dead ELSE learnt[n]] Quiet == \A d \in Dirs : q[d] = <<>> -NoDoubt == \A n \in Nodes : st[n].wait = {} +NoDoubt == \A n \in Nodes : st[n].wait = {} /\ st[n].expiry = {} \* The messages the updates `rs` send, counted before `Heeded` drops any. Sent(rs) == SumOver(DOMAIN rs, LAMBDA n : SumOver(Nodes, LAMBDA m : Len(rs[n].out[m]))) @@ -426,8 +499,12 @@ InitNode == [ dead |-> {}, doubt |-> {}, wait |-> {}, + expiry |-> {}, \* Pending timers belong to individual dead hops. + drain |-> {}, \* the neighbours that sent GOAWAY on the session open to them + draining |-> {}, \* nodes known to have sent GOAWAY, while named or doubted tx |-> [m \in Nodes |-> [k \in Keys |-> NoSent(k)]], told |-> [m \in Nodes |-> {}], + toldDrain |-> [m \in Nodes |-> {}], shown |-> [c \in Claims |-> NotShown] ] @@ -474,7 +551,7 @@ Announce(c) == rev < MaxRev /\ ChangeClaim(c, rev + 1) Unannounce(c) == rev < MaxRev /\ own[c] # None /\ ChangeClaim(c, None) ----------------------------------------------------------------------------- -(* A message arrives (paths.rs: `recv`, `fail`). *) +(* A message arrives (path.rs: `recv`, `fail`). *) \* `fail`: the session `hop` names closed. `from` reported it with what became \* of its own routes through it, or this node saw it close (`from` is None). @@ -485,8 +562,9 @@ Fail(n, s, hop, from, moved, gone) == LET h == s.rx[p][k] IN IF ~h.on \/ ~HasHop(h.path, hop) THEN h ELSE IF p = from /\ Moved(k) # {} - THEN LET path == (CHOOSE mv \in Moved(k) : TRUE)[2] - IN [h EXCEPT !.path = path, + THEN LET mv == CHOOSE mv \in Moved(k) : TRUE + path == mv[2] + IN [h EXCEPT !.path = path, !.drain = mv[3], !.state = IF DeadIn(path, s.dead) THEN "dead" ELSE "live"] ELSE IF p = from THEN IF gone THEN NoHeard @@ -494,14 +572,24 @@ Fail(n, s, hop, from, moved, gone) == ELSE IF ~Child(h.path, n) THEN [h EXCEPT !.state = "held"] ELSE h ELSE IF h.state = "live" THEN [h EXCEPT !.state = "dead"] ELSE h]] IN [s EXCEPT !.rx = rx1, !.dead = @ \cup {hop}, + !.expiry = IF hop \in s.dead THEN @ ELSE @ \cup {hop}, !.doubt = @ \cup ({hop[2]} \ {n}), !.wait = @ \cup ({hop[2]} \ {n})] +\* `drains`: x sent GOAWAY. The first word of it casts doubt on x, so no relay +\* takes a stale offer through it until the timer fires. +MarkDrain(n, s, x) == + IF x \in s.draining \/ x = n THEN s + ELSE [s EXCEPT !.draining = @ \cup {x}, !.doubt = @ \cup {x}, !.wait = @ \cup {x}] + +\* A DRAIN counts only for a node on a path `from` offered. +Credited(s, from, x) == \E k \in Keys : s.rx[from][k].on /\ Names(s.rx[from][k].path, x) + Recv(n, s, from, msg) == LET k == msg.k h == s.rx[from][k] IN CASE msg.t = "offer" -> [s EXCEPT !.rx[from][k] = - [on |-> TRUE, path |-> msg.path, + [on |-> TRUE, path |-> msg.path, drain |-> msg.drain, state |-> IF DeadIn(msg.path, s.dead) THEN "dead" ELSE "live", rev |-> IF h.on THEN h.rev ELSE None, items |-> IF h.on THEN h.items ELSE NoItems]] @@ -509,6 +597,7 @@ Recv(n, s, from, msg) == [] msg.t = "end" -> [s EXCEPT !.rx[from][k] = NoHeard] [] msg.t = "claims" -> IF h.on THEN [s EXCEPT !.rx[from][k].rev = msg.rev, !.rx[from][k].items = msg.items] ELSE s + [] msg.t = "drain" -> IF Credited(s, from, msg.node) THEN MarkDrain(n, s, msg.node) ELSE s [] OTHER -> Fail(n, s, msg.hop, from, msg.moved, msg.gone) Deliver(d) == @@ -516,9 +605,12 @@ Deliver(d) == n == d[2] msg == Head(q[d]) q0 == [q EXCEPT ![d] = Tail(@)] - rep == IF msg.t = "dead" THEN [on |-> TRUE, hop |-> msg.hop] ELSE NoReport - \* Only the route's path changed. - quiet == msg.t = "offer" /\ st[n].rx[m][msg.k].on + rep == CASE msg.t = "dead" -> DeadReport(msg.hop) + [] msg.t = "drain" /\ msg.node # n /\ Credited(st[n], m, msg.node) -> DrainReport(msg.node) + [] OTHER -> NoReport + \* Only the route's path, or how it ranks, changed. + quiet == \/ msg.t = "offer" /\ st[n].rx[m][msg.k].on + \/ msg.t = "drain" rs == [x \in {n} |-> Update(n, Recv(n, st[n], m, msg), OV, up, gen, ended[n], rep, quiet)] IN /\ q[d] # <<>> /\ Commit(rs, q0) @@ -527,25 +619,53 @@ Deliver(d) == \* The timer node n started on x fires: the doubt on x lifts, and the dead hops \* whose session ended at x may be forgotten. With `Patient`, only once no DEAD -\* about such a hop is in flight: a timer long enough for the news to cross -\* the mesh. +\* about such a hop, and no DRAIN of x, is in flight: a timer long enough for +\* the news to cross the mesh. Lift(n, x) == - LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ {x}, !.wait = @ \ {x}], + LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ {x}, !.wait = @ \ {x}, + !.expiry = {h \in @ : h[2] # x}], OV, up, gen, ended[n], NoReport, TRUE)] IN /\ x \in st[n].wait - /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : ~(q[d][i].t = "dead" /\ q[d][i].hop[2] = x) + /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : + ~(q[d][i].t = "dead" /\ q[d][i].hop[2] = x) /\ ~(q[d][i].t = "drain" /\ q[d][i].node = x) + /\ Commit(rs, q) + /\ Internal(Sent(rs)) + /\ UNCHANGED <> + +\* A hop's own timer can fire before a later failure's timer on the same far +\* node. A new DEAD cannot extend every old hop's retention indefinitely. +Expire(n, hop) == + LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.expiry = @ \ {hop}], + OV, up, gen, ended[n], NoReport, TRUE)] + IN /\ hop \in st[n].expiry + /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : ~(q[d][i].t = "dead" /\ q[d][i].hop = hop) /\ Commit(rs, q) /\ Internal(Sent(rs)) /\ UNCHANGED <> ----------------------------------------------------------------------------- -(* Sessions open and close. *) +(* Sessions open, drain, and close. *) + +\* `goaway`: n hears GOAWAY from m. Every path through m ranks below every path +\* that avoids it and stays usable; n's own offers through it say so. n doubts +\* m, so it moves at once only onto a path that avoids m, and tells each +\* neighbour it sent a route through m. +GoAway(n, m) == + LET rs == [x \in {n} |-> Update(n, MarkDrain(n, [st[n] EXCEPT !.drain = @ \cup {m}], m), + OV, up, gen, ended[n], DrainReport(m), TRUE)] + IN /\ {n, m} \in up + /\ m \notin st[n].drain + /\ Commit(rs, q) + /\ External(Sent(rs)) + /\ UNCHANGED <> \* A closed session: everything learned or sent on it goes. Closed(s, y) == [s EXCEPT !.rx[y] = [k \in Keys |-> NoHeard], !.tx[y] = [k \in Keys |-> NoSent(k)], - !.told[y] = {}] + !.told[y] = {}, + !.toldDrain[y] = {}, + !.drain = @ \ {y}] Connect(l) == LET up1 == up \cup {l} @@ -573,7 +693,7 @@ Disconnect(l) == IN IF x = Origin THEN Update(x, s, OV, up1, gen, ended[x], NoReport, FALSE) ELSE Update(x, Fail(x, s, hop, None, {}, FALSE), OV, up1, gen, ended[x], - [on |-> TRUE, hop |-> hop], FALSE)] + DeadReport(hop), FALSE)] IN /\ l \in up /\ flaps < MaxFlaps /\ up' = up1 @@ -592,7 +712,9 @@ Next == \/ \E c \in Claims : Announce(c) \/ Unannounce(c) \/ \E d \in Dirs : Deliver(d) \/ \E n \in Relays, x \in Nodes : Lift(n, x) + \/ \E n \in Relays : \E hop \in st[n].expiry : Expire(n, hop) \/ \E l \in Links : Connect(l) \/ Disconnect(l) + \/ \E n \in Relays, m \in Leaving : GoAway(n, m) \/ Done Spec == Init /\ [][Next]_vars diff --git a/rs/moq-net/verify/route/Twice.cfg b/rs/moq-net/verify/route/Twice.cfg index e4f3a2ef766..2b277a48b58 100644 --- a/rs/moq-net/verify/route/Twice.cfg +++ b/rs/moq-net/verify/route/Twice.cfg @@ -8,6 +8,9 @@ CONSTANTS Origin = 0 Links = {{0, 1}, {1, 2}, {1, 3}, {2, 3}} Dear = {} + Ring = {} + Preferred = {} + Leaving = {} Scoped = {} Claims = {"x"} Narrow = {} @@ -27,5 +30,6 @@ INVARIANTS SimplePaths LoopsAreStale StaleIsChased + RequestsLoopFree QuietIsSettled Budget diff --git a/test/route-sweep.sh b/test/route-sweep.sh index e946d8dec8a..c9c1d72a7a2 100755 --- a/test/route-sweep.sh +++ b/test/route-sweep.sh @@ -11,4 +11,4 @@ echo "Route sweep: MOQ_ROUTE_SEED=$MOQ_ROUTE_SEED MOQ_ROUTE_CASES=$MOQ_ROUTE_CAS export CARGO_PROFILE_RELEASE_DEBUG_ASSERTIONS=true CARGO_PROFILE_RELEASE_OVERFLOW_CHECKS=true just rs test -p moq-net --release --features async --test integration -E 'test(=mesh_sweep::seeded_meshes_hold_the_route_rules)' # The router's handoff sweeps replay their fixed scenarios past the gate's 150. -MOQ_ROUTE_SCENARIOS=${MOQ_ROUTE_SCENARIOS:-20000} just rs test -P sweep -p moq-net --release -E 'test(/^route::tests::handoff::sweep_/)' +MOQ_ROUTE_SCENARIOS=${MOQ_ROUTE_SCENARIOS:-20000} just rs test -P sweep -p moq-net --release -E 'test(/^route::tests::handoff::(distance|path)::sweep_/) | test(/^route::tests::scope::sweep_scoped_meshes_show_what_the_rule_says$/)'