diff --git a/.cargo/mutants.toml b/.cargo/mutants.toml index 9e4483fe765..d6daad556ed 100644 --- a/.cargo/mutants.toml +++ b/.cargo/mutants.toml @@ -17,28 +17,16 @@ exclude_re = [ # The rotation's xorshift step only picks which fresh identity comes next, # and the guard after it enforces freshness and range. 'replace (\^= with \|=|<< with >>|>> with <<) in Router::rotate', - # An index of a neighbor's interest: `carries()` filters each entry again. - 'replace Router::asked -> Path with Default::default\(\)', # `full(old) == full(new)` is the same with both sides negated. 'replace >= with < in Router::route$', - # A larger `differs` set only costs work: it is compared again before use. - 'replace && with \|\| in Heard::compare$', # The test-only `Proof::LEN`. 'proof\.rs:\d+:\d+: replace \+ with \*$', - # The stale-view shortcut only saves work: without it, or at equal revisions, - # where both views hold the same claims, the full merge changes nothing. - 'replace (> with (==|>=) in View::outdates|View::outdates -> bool with false)$', - # Dropping the cached context more often makes the next pass full, which - # reaches the same result. - ':\d+:(25|38): replace && with \|\| in Router::recompute$', # A claim an answering certifier certifies is always in the merged view. ':\d+:35: replace && with \|\| in Router::meets$', # A new stream's first batch either flips its certifier's answer, which marks # every claim, or reaches the reader only through the merge, which marks its own # changes; marking all only reorders independent changes in a forwarded batch. ':\d+:32: replace != with == in Router::batch$', - # At an equal metric both arms assign the same distance. - 'replace < with <= in Router::record$', # The full presentation of a stream the interest misses is empty, and a full # presentation equals the incremental one. 'replace match guard (!self\.meets\(id, ctx, watch\)|!dirty\.all) with false in Router::advertise$', diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index e1d957aeac7..20f307cce39 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -1,11 +1,11 @@ --- name: verify-route -description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteHandoff.tla`, the rule the routes epic's Plan item 27 picked, as `route::path::Router` in `rs/moq-net/src/route/path.rs` runs it) with TLA+. Use when a change touches route paths, the DEAD message for a closed session, holds, the doubt and its timer, forgetting dead hops, routes per prefix and scoped streams, the newest-floor views, what a reader is shown, or where a request goes, or when a routing bug needs every message ordering checked rather than a random sweep. +description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteHandoff.tla`, the rule the routes epic's Plan item 27 picked, as `route::path::Router` in `rs/moq-net/src/route/path.rs` runs it) with TLA+. Use when a change touches route paths, the DEAD message for a closed session, holds, the doubt and its timer, forgetting dead hops, routes per prefix and scoped streams, the down-only flag on upstream links, the newest-floor views, what a reader is shown, or where a request goes, or when a routing bug needs every message ordering checked rather than a random sweep. --- # Verify the route rule -`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` start out watching alone), and keeps dead hops until their timer fires. A relay asks each neighbour for the set of prefixes it watches, none inside another; a relay in `Editors` may change that set up to `MaxEdits` times. Each change is an edit, sent as an ask on the FIFO and answered by a marker on the neighbour's FIFO, where both sides rekey the routes between them alike (`Rekey`): a key the new set covers stays, the route at a removed `"a"` moves to an added `"root"` without its floor, and the route at a removed `"root"` carries its floor, presented at `"a"`, to an added `"a"`. A heard route is believed for a prefix only while the relay asked for it at the last marker and in every edit still in flight. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. +`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. A route learned on a link its learner marks upstream, or from a down-only route, is down-only, and no relay sends one over a link it marks upstream (`Upstream`, the links whose higher-numbered end marks them), so no relay carries traffic between two upstreams. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` start out watching alone), and keeps dead hops until their timer fires. A relay asks each neighbour for the set of prefixes it watches, none inside another; a relay in `Editors` may change that set up to `MaxEdits` times. Each change is an edit, sent as an ask on the FIFO and answered by a marker on the neighbour's FIFO, where both sides rekey the routes between them alike (`Rekey`): a key the new set covers stays, the route at a removed `"a"` moves to an added `"root"` without its floor, and the route at a removed `"root"` carries its floor, presented at `"a"`, to an added `"a"`. A heard route is believed for a prefix only while the relay asked for it at the last marker and in every edit still in flight. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Every lite-07 session drives the path router through `route::node::Node` since [the switch](https://github.com/moq-dev/moq/pull/8731), and `route::tests::Sim` drives it under `Rule::PATH` (timer-only doubt). ## Rules @@ -21,10 +21,11 @@ Each rule is an INVARIANT; the harm says why it matters. | `NoRegression` | A reader never sees an UPDATE to an older item. | A stale announcement replaces a newer one. | | `Stable` | A step that only changes a route's path, lifts a doubt, or applies an interest edit (its ask, or its marker for the claims both of the edit's sets cover) changes a shown claim only to a newer floor, or lets go of one it held unavailable. | A detour flaps claims, as a seqno advance did under the distance, or an edit that swaps prefixes ends or re-sends a claim both sets cover. | | `SimplePaths` | Every route held or in flight names a loop-free path from the publisher to its sender. | A relay took back a route that crossed it. | +| `DownOnly` | A route is down-only exactly when its path came down a link its learner marks upstream, no path climbs a link its sender marks upstream after that, and nothing down-only is sent, in flight, or heard over such a link. | A relay carries CDN traffic between two upstreams, or a route loses its down-only flag one LAN hop on. | | `LoopsAreStale` | A request that loops crosses a relay whose route names a closed session. | Routing loops with every relay up to date. | | `StaleIsChased` | Every stale route a relay may use has the news that ends it on its way: queued toward the relay, or toward the neighbour it came from, and so on back. | A stale route, and any loop through it, lasts for good. | | `NoStaleReturn` | No relay may use a route through a hop it once learned had died. | A dead path comes back after a relay forgot it. | -| `QuietIsSettled` | Once nothing is in flight and every timer fired: each relay shows exactly the claims a path of open sessions through relays watching them carries to it, available, and a request for each reaches the publisher; no relay holds, keeps a stale route, or remembers a dead hop; and both ends of each session agree on what it asks for. | A hold never ends, a claim is shown that no path carries, a relay behind a scoped neighbour never reaches what a farther one offers, or an edit never takes effect. | +| `QuietIsSettled` | Once nothing is in flight and every timer fired: each relay shows exactly the claims a path of open sessions through relays watching them carries to it, never up a link its sender marks upstream after one its learner does, available, and a request for each reaches the publisher; no relay holds, keeps a stale route, or remembers a dead hop; and both ends of each session agree on what it asks for. | A hold never ends, a claim is shown that no path carries, a relay behind a scoped neighbour never reaches what a farther one offers, or an edit never takes effect. | | `Budget` | From one quiet point to the next, the messages sent stay within 4 each way over each configured session per prefix and per claim, for each external change. | One change sets off announce spam, or routing never goes quiet. | | `RequestsLoopFree` | A request never loops. | A request cycles through relays instead of reaching the publisher. | @@ -67,19 +68,21 @@ change; faster checkers are separate work. | Config | Mesh | Bounds | Previous distinct states | Previous run | | --- | --- | --- | --- | --- | -| `RouteHandoff.cfg` | The publisher behind relay 1, in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 476,515 | a minute or two | -| `Early.cfg` | The same, with a timer that fires at any moment. | 1 claim change, 1 session change | 14,220,298 | half an hour | +| `RouteHandoff.cfg` | The publisher behind relay 1, in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 476,434 | a minute or two | +| `Early.cfg` | The same, with a timer that fires at any moment. | 1 claim change, 1 session change | 14,220,217 | half an hour | | `Twice.cfg` | The same, where two sessions close, as when relay 1 dies. Simulated: `num` counts behaviours per worker. | 1 claim change, 2 session changes | 40,000 behaviours, 4.0M states | a few minutes | -| `Ring.cfg` | The publisher and three relays in a ring, with a timer that fires at any moment. | 1 claim change, 1 session change | 890,827 | a minute or two | -| `Lag.cfg` | Relay 3 hears the publisher through relays 1 and 2: the lagging neighbour. | 2 claim changes, 1 session change | 1,926,671 | a few minutes | -| `LagWide.cfg` | The same, with relay 1 asking for `"a"` alone. | 2 claim changes, 1 session change | 4,675,558 | a quarter of an hour | +| `Ring.cfg` | The publisher and three relays in a ring, with a timer that fires at any moment. | 1 claim change, 1 session change | 813,075 | a minute or two | +| `Lag.cfg` | Relay 3 hears the publisher through relays 1 and 2: the lagging neighbour. | 2 claim changes, 1 session change | 1,702,001 | a few minutes | +| `LagWide.cfg` | The same, with relay 1 asking for `"a"` alone. | 2 claim changes, 1 session change | 3,846,369 | a quarter of an hour | | `Dead.cfg` | Two relays behind a relay whose only uplink asks for `"a"` alone, one with a dear session of its own to the publisher. Simulated. | 3 claim changes, 1 session change | 40,000 behaviours, 4.8M states | ten minutes | | `Blind.cfg` | A relay whose nearest neighbour asks for `"a"` alone, and a farther one knows everything. Simulated. | 2 claim changes, 1 session change | 40,000 behaviours, 4.8M states | ten minutes | | `Overlap.cfg` | Five relays, three asking for `"a"`, with overlapping cheap broad and dear narrow paths. Simulated past the message budget. | 1 claim change, no session change | 1,000 behaviours per worker (4,000 on four workers), depth 500 | simulation | | `Loop.cfg` | The publisher with a session to each of three relays in a triangle, all closing. Samples `RequestsLoopFree` with timer-only doubt. | 3 session changes | 1,000 behaviours per worker, depth 150, seed 1 | simulation | -| `Drain.cfg` | The publisher joined to two relays joined to each other; both hear its GOAWAY. Ranking only a relay's own drain exceeded `Budget`. | 1 claim change, 1 session change, GOAWAY on each session | 648,218 | two minutes | +| `Drain.cfg` | The publisher joined to two relays joined to each other; both hear its GOAWAY. Ranking only a relay's own drain exceeded `Budget`. | 1 claim change, 1 session change, GOAWAY on each session | 511,147 | two minutes | | `DrainRing.cfg` | The publisher joined to three relays in a ring that is cheap one way round (`Ring`); all hear its GOAWAY. Without the DRAIN news and its doubt, a request looped round the ring with no session closed, breaking `LoopsAreStale`. Simulated. | 1 claim change, GOAWAY on each session | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | -| `Prefer.cfg` | The same triangle, where both relays prefer the link between them. Ranking only a relay's own link priority exceeded `Budget`. | 1 claim change, 1 session change | 14,630 | seconds | +| `Prefer.cfg` | The same triangle, where both relays prefer the link between them. Ranking only a relay's own link priority exceeded `Budget`. | 1 claim change, 1 session change | 12,654 | seconds | +| `Upstream.cfg` | Two uplinks: the publisher on a CDN relay joined by a dear link to another, and a local mesh of two relays, each with an upstream link to one CDN relay. With the CDN link down, the second CDN relay has no route rather than one through the local mesh. The Rust fixture is `two_uplinks_never_carry_cdn_transit`. Without the export check, or with a flag that a LAN hop clears, `QuietIsSettled` and `DownOnly` fail within seconds. | 1 claim change, 1 session change | 68,717 | seconds | +| `UpstreamScoped.cfg` | The same, where the second CDN relay asks for `"a"` alone, so the stream to it over an upstream link picks among covering routes that are all down-only and ends. | 1 claim change, 1 session change | 69,576 | seconds | | `EditNarrow.cfg` | The publisher with sessions to relays 1 and 2, the one to relay 2 dear; relay 2 watches everything and narrows to `"a"` or drops it once. | 1 claim change, 1 session change, 1 edit | 768,483 | three minutes | | `EditWiden.cfg` | The same, where relay 2 watches `"a"` and widens to everything or drops it once. | 1 claim change, 1 session change, 1 edit | 3,655,573 | a quarter of an hour | | `Edit.cfg` | The same with two edits, such as widening then narrowing back, or dropping everything then asking again. Before the rekey kept the route the sender would reflect, a widening then a narrowing crossing the close of relay 2's own session kept a route relay 1 learned through it and took a claim offline, breaking `Stable`. Simulated; an exhaustive pass covered every behaviour to depth 18 (10.6M states) before it was stopped. | 1 claim change, 1 session change, 2 edits | 40,000 behaviours, 4.8M states | four minutes | @@ -101,7 +104,7 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on th ## Coverage -Sync commit: `cc17f589737dbedf156b86a8ba327593f3905074`, the base of the interest set in [moq-dev/moq#8728](https://github.com/moq-dev/moq/pull/8728). Run `git log cc17f5897.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `857f2eea4ac4cc7a3fb821f7d250637eb464b248`, where [the switch](https://github.com/moq-dev/moq/pull/8731) took in [the interest set](https://github.com/moq-dev/moq/pull/8728). Run `git log 857f2eea4.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SelectKey` sets a route's down-only flag as `Router::usable_offer` does, and `Want0` ends a live down-only route toward an upstream link. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, leaving out down-only routes toward an upstream link and falling back to a held one, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | @@ -109,6 +112,7 @@ Sync commit: `cc17f589737dbedf156b86a8ba327593f3905074`, the base of the interes | `rs/moq-net/src/route/path.rs` `Offer::rank`, `Router::goaway`, `Router::drains`, `Router::report_draining`, `Router::dead` (`Dead::Draining`) | `Rank`, `Below`, `PathPrio`, `CostAt`, `GoAway`, `MarkDrain`, `Credited`, `ReportDrain`, `Recv` | | `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted` | | `rs/moq-net/src/route/path.rs` `Router::src` | `SrcKey` | +| `rs/moq-net/src/route/path.rs` `Router::usable_offer`, `Offer` | `SelectKey`'s `Down`, `Want0`'s export check, and `DownOnly`: the down-only flag is set on a route learned over an upstream link or from a down-only one, and never sent over an upstream link. | | `rs/moq-net/src/route/path.rs` `Router::want`, `Router::stream`, `Router::advertise`, `Router::report` | `Want`, `Want0`, `Streams`, `Advertise`, `Report` | | `rs/moq-net/src/route/path.rs` `Router::route`, `Router::batch`, `Router::dead`, `Router::fail`, `Router::disconnect`, `Router::connect`, `Router::change` | `Recv`, `Fail`, `Disconnect`, `Connect`, `ChangeClaim` | | `rs/moq-net/src/route/path.rs` `Router::collect_doubt`, `Router::timeout` | `Collect`, `Lift`, `Expire`: each dead hop's own timer retires it independently of later failures of the same node; reporting an already retained hop again does not restart its expiry. | @@ -121,4 +125,4 @@ Sync commit: `cc17f589737dbedf156b86a8ba327593f3905074`, the base of the interes SYNCED is left out: the router only records it per prefix (`Router::synced`, `Router::is_synced`), and nothing it decides reads it. So are scope regions: a relay asks for what it watches, and `interest.rs`'s `a_scope_with_several_regions_asks_for_each` and the sweeps cover a scope met with the interest. -One publisher, so routes for several publishers are left out, and so are the copies and copy rank of cache-aware routing, which `rs/moq-net/src/route/tests/cache.rs` sweeps; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, so an edit adds, removes, widens, or narrows one prefix at a time: no wildcards, swaps of several prefixes for one, stream cancellation, grants, qualified keys, or hidden claims. Leaves out restarts, priorities other than 0 and 1, a priority that differs between a link's two ends, costs other than 1 and 2, RTT, upstream/down-only links, and the hop limit. Messages come only from honest routers: validation of an unrelated DEAD or its far node, and empty or overlong replacement paths, is outside the model's input domain. Rust fixtures cover those guards and exportable covering-source selection, including keeping a covering hold when every live source is unexportable. Interest-edit fixtures also cover a cancelled stream retaining its pending child replies; the model has no stream cancellation. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. A route is sent on every stream whether or not the publisher has a claim there. The publisher ignores what it hears. `Overlap.cfg` also simulates five relays with every session up. No exhaustive pass has more than four relays or two session changes; `Sim`'s sweeps on the path rule (`sweep_handoff`, `sweep_scoped_handoff`, `sweep_scoped_meshes_show_what_the_rule_says`, and `compare.rs`'s churn) cover larger meshes, grants, qualified keys, interest edits, restarts, and churn. +One publisher, so routes for several publishers are left out, and so are the copies and copy rank of cache-aware routing, which `rs/moq-net/src/route/tests/cache.rs` sweeps; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, so an edit adds, removes, widens, or narrows one prefix at a time: no wildcards, swaps of several prefixes for one, stream cancellation, grants, qualified keys, or hidden claims. Leaves out two sessions between one pair of nodes at once, as a GOAWAY migration has: the model keeps one per pair, and among equal routes the router stays on the next hop's node and takes its latest session there (`a_later_session_to_the_next_hop_takes_over_among_equals`). Leaves out restarts, priorities other than 0 and 1, a priority that differs between a link's two ends, costs other than 1 and 2, RTT, and the hop limit. Down-only routes appear only where every relay's best route is also its only exportable one, so the model never shows a down-only best route hiding an exportable alternative, a cost the routes epic's contract accepts, and a relay that accepted a request from an upstream link never ends it when its route turns down-only. The hold deadline (`HOLD`), the stalled-writer close, and a scoped session's DEAD (`fail_heard`, which doubts and remembers nothing) are left out: a hold that ends early only lets go sooner, and the model's sessions are all unscoped. So are the node's per-stream keys: lite-07 files each announce stream as its own router key, whose one edit is the prefix the stream opened with (`route/node.rs`), where the model has one neighbour per session and edits it in place; [the route stream](/quest/m2/cluster-routes/route-stream.md) joins them. Messages come only from honest routers: validation of an unrelated DEAD or its far node, and empty or overlong replacement paths, is outside the model's input domain. Rust fixtures cover those guards and exportable covering-source selection, including keeping a covering hold when every live source is unexportable. Interest-edit fixtures also cover a cancelled stream retaining its pending child replies; the model has no stream cancellation. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. A route is sent on every stream whether or not the publisher has a claim there. The publisher ignores what it hears. `Overlap.cfg` also simulates five relays with every session up. No exhaustive pass has more than four relays or two session changes; `Sim`'s sweeps on the path rule (`sweep_handoff`, `sweep_scoped_handoff`, `sweep_scoped_meshes_show_what_the_rule_says`, and `compare.rs`'s churn) cover larger meshes, grants, qualified keys, interest edits, restarts, and churn. diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index 3731435500d..e0d3d31f617 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -18,9 +18,12 @@ carry it: natively on moq-lite up to moq-lite-06, and via the moq-lite-07-wip carries the list once per publishing node instead of once per broadcast. Each publishing node declares a random routing identity, and relays -run a distance-vector route to it, loop-free by Babel's feasibility condition, -so every claim of one origin shares that route. The route names the relays it -crossed, and a relay never uses one that names itself. +keep a route to it per prefix, which every claim of that origin under the prefix +shares. The route names each relay it crossed and the session it crossed on, and +a relay never uses one that names itself. A relay that loses a session names it +to its neighbours, so every route through it is dropped at once, and a relay left +without a route holds its claims, unavailable, for up to ten seconds while one may +return. The first relay a request reaches picks the origin node, by a hash of the path and the node's identity that every relay computes alike, and names it on the request so later hops follow their own route to that node. A relay running both diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 44e6ecfaad2..37e6584c5bb 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -39,7 +39,6 @@ normative: informative: I-D.lcurley-moq-cluster: RFC7627: - RFC8966: moqflate: title: "DEFLATE Compressed Tracks for MoQ" target: https://datatracker.ietf.org/doc/draft-lcurley-moq-flate/ @@ -402,15 +401,16 @@ There's a 1-byte STREAM_TYPE at the beginning of each stream. ### Announce A subscriber can open an Announce Stream to discover claims matching a prefix. A claim is an announcement that paths under its prefix can be served by one origin node; it claims capability, not inventory, so it never asserts that any specific broadcast exists. -A route is how the publisher reaches an origin node, and every claim on the stream travels under the route to its origin (see [Routing](#routing)). +A route is how the publisher reaches an origin node for the paths under a prefix, and every claim on the stream travels under a route to its origin covering it (see [Routing](#routing)). The subscriber creates the stream with an ANNOUNCE_REQUEST message. The publisher replies with a single ANNOUNCE_OK message followed by the matching routes and claims and any future changes: -- ROUTE_START: the publisher reaches an origin node. -- ROUTE_UPDATE: a route's seqno, metric, down-only bit, RTT, or hops changed. +- ROUTE_START: the publisher reaches an origin node for a prefix. +- ROUTE_UPDATE: a route's path, drain, priority, metric, down-only bit, or RTT changed. - ROUTE_HOLD: the publisher has no usable route but holds it while it recovers. - ROUTE_END: the publisher let go of a route and every claim under it. +- DEAD: a session on the path of routes the publisher sent closed, or a node on it sent a GOAWAY. - ANNOUNCE_START: a matching claim of a route's origin node. - ANNOUNCE_END: the origin withdrew a claim. - ANNOUNCE_UPDATE: a claim's cost changed. @@ -421,7 +421,7 @@ The publisher replies with a single ANNOUNCE_OK message followed by the matching - WARM_END: the publisher withdrew a warm route. - SYNCED: the initial set is complete. -In the other direction, the subscriber sends ROUTE_REQUEST and echoes every ROUTE_END (see [Routing](#routing)). +The subscriber sends nothing on the stream after its ANNOUNCE_REQUEST. The publisher MUST send [SYNCED](#synced) exactly once, after the initial set. The subscriber MAY batch messages until SYNCED; subsequent ones are live updates. @@ -430,15 +430,15 @@ The publisher MAY coalesce changes to a claim it has not yet sent, and sends no A retraction of a claim or route already sent MAY precede SYNCED. Each ROUTE_START implicitly assigns the next Route ID on the stream, each ANNOUNCE_START the next Announce ID, and each WARM_START the next Warm ID: three counters starting at 0 that increment by 1 per assignment. -No id appears on the wire where it is assigned; both endpoints derive it from the message order on the (reliable, ordered) stream, and other messages reference it instead of repeating the origin node, the claim's prefix, or the warm route's path. +No id appears on the wire where it is assigned; both endpoints derive it from the message order on the (reliable, ordered) stream, and other messages reference it instead of repeating the origin node and the route's prefix, the claim's prefix, or the warm route's path. Ids are never reused within a stream. A ROUTE_END retires its Route ID, the Announce ID of every claim under the route, and the Warm ID of every warm route under it; an ANNOUNCE_END retires one Announce ID, and a WARM_END one Warm ID. -A stream has at most one live route per origin node, and each route at most one live claim per prefix. +A stream has at most one live route per origin node and prefix, and each route at most one live claim per prefix, at or below the route's prefix. Another origin claiming the same prefix is another claim, under its own route. An ANNOUNCE_UPDATE or ANNOUNCE_RESTART atomically replaces a claim while keeping its id live; a claim never moves to another route. -The subscriber MUST close the session with a PROTOCOL_VIOLATION if it receives a message referencing a Route ID or Announce ID that was never assigned or already retired, a ROUTE_START for an origin node that already has a live route on the stream, an ANNOUNCE_START for a prefix its route already claims, or any message before ANNOUNCE_OK. +The subscriber MUST close the session with a PROTOCOL_VIOLATION if it receives a message referencing a Route ID or Announce ID that was never assigned or already retired, a ROUTE_START for an origin node and prefix that already have a live route on the stream, an ANNOUNCE_START for a prefix its route already claims or outside the route's prefix, or any message before ANNOUNCE_OK. When the stream is closed, the subscriber MUST assume that every route and claim on it is gone. A broadcast path has at most 32 segments and at most 4,096 bytes; a receiver MUST close the session with a PROTOCOL_VIOLATION on a longer one, except in a claim, which it refuses instead (see [Refused Content](#refused-content)), including one resolved by [Compression](#announce-compression). @@ -452,16 +452,18 @@ Each origin node numbers the changes to its own claims with a Revision, starting A stream carries a route's claims in batches: the ANNOUNCE messages under the route since its last ANNOUNCE_FLOOR, closed by the next one. The publisher sends a ROUTE_START before any claim under it, and SYNCED only once every route it sent has a floor. -A floor names a Revision and certifies that the stream's claims for the route, across the whole requested prefix, are the origin's claims at that Revision as the request presents them (see [Announce](#announce)). +A floor names a Revision and certifies that the stream's claims for the route, across the route's whole prefix, are the origin's claims at that Revision as the request presents them (see [Announce](#announce)). It certifies only what its own request shows: a stream whose ANNOUNCE_REQUEST did not set `Hidden` certifies nothing about hidden claims. -A publisher that does not know the route's claims across the whole requested prefix at one Revision, such as a relay that knows them only beneath part of it, still closes each batch, with a floor that certifies nothing. +A publisher that does not know the route's claims across its whole prefix at one Revision, such as a relay that knows them only beneath part of it, still closes each batch, with a floor that certifies nothing. A subscriber that may receive claims only beneath one prefix, because its [scope](#auth-stream) or the publisher's confines the stream there, SHOULD request that prefix rather than one above it, so the publisher can certify the request. When that prefix changes, it SHOULD open a request for the new one and end the old one once the new one is SYNCED. The subscriber applies a batch whole, once its floor arrives. -A receiver that hears one origin's claims on several streams merges them claim by claim, each at the prefix its stream presents it: of the streams whose floor certifies the claim, the one with the highest Revision decides whether it exists, and a claim no floor certifies exists while some stream carries it. +A receiver keeps one view of an origin's claims for each prefix it holds a route to the origin at: the claims of the floor with the highest Revision that a route it believes, at that prefix or one covering it, gave it. +It believes a route while the route is usable, its Path does not name the receiver, and its stream answers the receiver's current request; a held route, or one the receiver refused, certifies nothing. An older floor never revives a claim whose withdrawal a newer one delivered. -A floor that certifies nothing proves no claim absent: the stream's claims stay until its ANNOUNCE_END or ROUTE_END. -While a receiver has a usable route to the origin and every peer whose last advertised route to it is feasible here has answered the current request, it lets go of a claim none of those peers certifies or carries: it goes offline, which is not an ANNOUNCE_END. +While no route it believes has given a floor that certifies the prefix, the view holds the claims its selected route carries there, uncertified: a floor that certifies nothing proves no claim absent. +The view stays whole while the receiver holds a route covering the prefix, usable or held, and the receiver shows each claim from the view at the narrowest prefix covering it, available while a usable route covers it. +Once the receiver holds no route covering the prefix, the view's claims go offline, which is not an ANNOUNCE_END. #### Compression {#announce-compression} An ANNOUNCE_START MAY copy the head of its path from a live claim on the same stream, its base. @@ -474,16 +476,16 @@ The subscriber MUST close the session with a PROTOCOL_VIOLATION if a non-zero `P A resolved path that breaks its own rules, or a base whose path the subscriber refused, refuses the claim (see [Refused Content](#refused-content)). #### Refused Content {#refused-content} -Route messages carry content the publisher may only have forwarded: a route's Node ID, Seqno, Flags, and Hops, a claim's path, Instance, and Lite-06 Chain, and a warm route's path, Instance, Flags, and Hops. +Route messages carry content the publisher may only have forwarded: a route's Node ID, Route Suffix, and [Route Offer](#route-offer), a claim's path, Instance, and Lite-06 Chain, and a warm route's path, Instance, Flags, and Hops. A newer version or release may send such content that an older receiver cannot use, so a receiver that refuses it withdraws only the route or claim it describes and keeps the session, as BGP treats a malformed attribute as a withdrawal ({{?RFC7606}}). This specification marks each such value "refused"; the receiver skips the rest of the message's body, and SHOULD log and count each refusal so a refused route is never silently missing. -- A refused ROUTE_START still assigns its Route ID, and a refused ROUTE_UPDATE keeps it; either leaves the receiver no route, so its claims go offline as on ROUTE_END. The messages naming the Route ID stay valid, and a later ROUTE_UPDATE the receiver can use makes the route usable again, with the claims its last floor certified. A route to Node ID 0 stays refused. +- A refused ROUTE_START still assigns its Route ID, and a refused ROUTE_UPDATE, or a refused Route Offer among a [DEAD](#dead)'s moved routes, keeps it; each leaves the receiver no route, so its claims go offline as on ROUTE_END. The messages naming the Route ID stay valid, and a later ROUTE_UPDATE the receiver can use makes the route usable again, with the claims its last floor certified. A route to Node ID 0, or whose Route Suffix was refused, stays refused. A refused Route Offer's Path still counts, so a later DEAD naming a hop on it applies to the route as to any other. - A refused ANNOUNCE_START still assigns its Announce ID, and a refused ANNOUNCE_UPDATE or ANNOUNCE_RESTART keeps it; the claim goes offline at its route's next floor, which is not an ANNOUNCE_END. A claim whose path was refused stays refused; any other comes back with a later ANNOUNCE_RESTART the receiver can use. Its origin withdrew nothing, so while a route has a live claim the receiver refused, the receiver takes the route's floors as certifying nothing, and a relay forwards no withdrawal for the claim. - A refused WARM_START still assigns its Warm ID. Only the publisher's own framing and stream state close the session: a message that does not decode within its Message Length, such as an RTT flag without its RTT, and a message naming an id that was never assigned or already retired, or breaking the stream's order, as above. -A relay never forwards content a peer would refuse for the relay's own part in it: a route that arrives with 32 Hops is unusable, so the relay's Hop never takes one past 32. +A relay never forwards content a peer would refuse for the relay's own part in it: a route that arrives with 32 hops is unusable, so the relay's own hop never takes one past 32. #### Hidden Paths {#hidden} A claim is hidden from a request when a segment of its path below the requested prefix starts with `.` (0x2E). @@ -495,7 +497,9 @@ Hiding is a convenience for discovery, not access control: a publisher MAY treat SUBSCRIBE, FETCH, and TRACK resolve a hidden path exactly as any other. #### Routing {#routing} -Routing follows the Babel routing protocol [RFC8966], keyed by origin node: each node selects one route per origin node and advertises it to its neighbours. +Each node selects, per origin node and prefix, one route by its path: the nodes it crossed, each with the session it was learned on. +A node never uses a route whose path names it, and a node that loses a session names that hop to its neighbours, so every route through it is dropped wherever it went. +There is no distance, sequence number, route request, or echo. An origin node names itself with a Node ID: a non-zero 64-bit value, fresh for each process restart and shared by every session of that process. An origin MUST NOT declare a Node ID with the most significant bit set. @@ -504,20 +508,25 @@ The derivation is keyed by the key the peer proved with the [Route Key Parameter A relay MUST ignore a qualified Node ID received from such a peer, and claims under its qualified Node IDs outside that peer's grant. A Node ID is otherwise declared, not proven. -An origin advertises its own route with a Metric of 0 and a Seqno that only it advances. +A route covers a prefix: the one its receiver requested, or a narrower one that a node on its path requested. +A stream answering a request carries the sender's route to each origin node at every prefix inside the request, under that prefix, and at the requested prefix itself the best-ranked usable route covering it. +A node keeps, ranks, and selects its routes per origin node and prefix. + +An origin advertises its own route with a Metric of 0. A receiver adds the session's link cost (see [Cost Parameter](#cost-parameter)) to the Metric of every route the peer sends; the sum is its metric through that peer. Each link adds at least 1, so a route's metric strictly increases away from its origin. A metric that overflows 64 bits makes the route unusable. -A route carries its Hops: the Hop ID of every node it crossed, the origin node first and the sender last. -A Hop ID is a non-zero 64-bit value a node names itself with for as long as its process runs, such as the Node ID it started with; a rotation does not change it. -A receiver tolerates a zero or repeated Hop ID instead of closing the session: such a route harms only itself. -A node sending its selected route appends its own Hop ID to the Hops it received, and an origin's own route starts with its Hop ID alone. -A relay forwards the other entries unchanged and never qualifies them. -A route that arrives with 32 Hops is unusable. +A route carries its Path: one hop per node it crossed, the origin node first and the sender last. +A hop names a node by its Hop ID, and the session the node learned the route on by its Session Number. +A Hop ID is a non-zero 64-bit value a node names itself with for as long as its process runs, such as the Node ID it started with; a rotation of its Node ID does not change it. +A node numbers each session it opens or accepts with a Session Number of its own, unique among its sessions for as long as its process runs, so a restarted process MUST take a new Hop ID. +An origin's own route has a Path of its own hop alone, with Session Number 0, and a node sending a route it learned appends its own hop, naming the session it learned the route on. +A relay forwards the other hops unchanged and never qualifies them. +A route that arrives with 32 hops is unusable, since the receiver's own hop would take it past 32. A route may also carry an RTT: the round trip to its origin node in milliseconds, summed link by link. -It is informational: nothing ranks or selects a route, or judges it feasible, by its RTT. +It is informational: nothing ranks or selects a route by its RTT. An origin advertises its own route with an RTT of 0. A receiver adds its link min RTT for the session to the RTT of every route the peer sends; the sum is its RTT through that peer, which it advertises on the route it selects. A link min RTT is the lowest smoothed RTT [RFC9002] the connection measured over the last 30 seconds, read once a second, in milliseconds rounded up. @@ -528,38 +537,23 @@ A sum too large to encode is sent without one too. An endpoint learns its round trip to an origin node by adding its own link min RTT to the RTT of the route it selected; without its own, the RTT alone covers every hop past its first. Like the Metric, an RTT is unverified: a peer can misstate only what crosses it. -A node keeps a feasibility distance per Node ID: the highest Seqno of any route it advertised, and the lowest metric it advertised at that Seqno, as in [RFC8966] Section 3.5.1. -A route is feasible when no feasibility distance is kept for its Node ID yet, when its Seqno is greater than the distance's, or when its Seqno is equal and its advertised Metric is lower. -A node selects, per Node ID, among its feasible routes that are not held: a session that has not received a GOAWAY, then the highest local priority, then the lowest metric. -It MUST NOT select an infeasible route, and a distance only ever falls within a Seqno, so advertising a longer route never raises it. -It MUST NOT select a route whose Hops name it either, feasible or not: that is a copy of a route the node sent, which no distance refuses once the node dropped it. -Such a route counts below as an infeasible route where the feasibility condition refuses it, and as no route where it does not: its sender routes through the node, and moves once it hears the node has no route. -Under a qualified Node ID it counts only if its Hops also begin with those of the route the node last sent under that Node ID: otherwise a relay past the node derived the Node ID from the node's route under another, and that copy stands whatever the node does. +A route is draining when a node on its Path sent a GOAWAY, as the route's draining flag or a [DEAD](#dead) says, or when its session received one. +Its priority is the lowest local priority of a session on its Path: a receiver takes the lower of the priority the route carries and its own local priority for the session, and an origin's own route carries the highest. +A node ranks its usable routes to an origin node at a prefix: one that is not draining first, then the highest priority, then the lowest metric. +Extending a route never clears its drain, never raises its priority, and always raises its metric, so no node ranks a route above how the neighbour it came from ranked it, and selection converges. +A node selects its best-ranked usable route at once. +A route is unusable while its Path names the node, it is held, a hop on its Path is dead, a node on its Path is in doubt (see [Dead Hops](#dead-hops)) unless it is the route the node already uses, it arrived with 32 hops, or its metric overflows. A node MAY mark a session upstream. A route is down-only when it arrived on an upstream session or with its down-only flag set; a node sends a down-only route with the flag set, and never on an upstream session. -A node sends its selected route to an origin node on an Announce Stream it answers only while the route has a claim the stream's request presents (see [Announce](#announce)): it sends ROUTE_START before the first such claim, and once the last one is withdrawn, the batch carrying that withdrawal, closed by its floor (which certifies the withdrawal where the node can certify the request), then ROUTE_END. It starts that route on the stream again only once the subscriber echoed the ROUTE_END. A subscriber therefore holds no route for an origin node its request does not meet. -A node that does not forward, such as a client, sends only its own route, and holds no feasibility distance for any other. -An origin advances its Seqno whenever its own set of sessions changes. -It never advertises a Seqno or Revision of 2^64-1: a change that would reach it instead rotates its Node ID, ending the old route everywhere and announcing its claims again under the new one. - -A node with no usable route to an origin node holds it while it can still recover: while every route it has to the node is infeasible, or a peer holds a route whose last advertisement is feasible here. -A held route travels as a ROUTE_HOLD, and its claims stay shown but unavailable until a ROUTE_UPDATE makes it usable again. -Otherwise the node lets go, sending ROUTE_END wherever it advertised the route, and its claims go offline. -Neither proves that the origin withdrew its claims. - -A node whose routes to an origin node are all infeasible sends a ROUTE_REQUEST with its distance's Seqno plus 1 to the peer whose route it would select were it feasible. -A node holding a selected route at least that new answers by advertising it, and the origin answers by raising its Seqno to at least the requested one; a request for 2^64-1 rotates the origin's Node ID. -Any other node forwards the request to the peer whose route it would select, even the requester. -Every node keeps one pending request per origin node, its own or forwarded, at the highest Seqno requested, dropping requests no newer, until it selects a route at least that new or has no route left, and sends it again whenever its Seqno rises or that peer changes. - -The receiver of a ROUTE_END echoes it back on the same stream once it dropped the route. -A node drops a feasibility distance once no peer's route to the node is left open, it advertises none, and every ROUTE_END it sent for it was echoed; a closed session counts as an echo. -No peer then routes through the node, and the Hops refuse any copy of its own route that still returns. +A node sends its route to an origin node at a prefix on an Announce Stream it answers only while the route has a claim the stream's request presents (see [Announce](#announce)): it sends ROUTE_START before the first such claim, and once the last one is withdrawn, the batch carrying that withdrawal, closed by its floor (which certifies the withdrawal where the node can certify the request), then ROUTE_END. +A subscriber therefore holds no route for an origin node its request does not meet. +A node that does not forward, such as a client, sends only its own route. +An origin never advertises a Revision of 2^64-1: a change that would reach it instead rotates its Node ID, ending the old route everywhere and announcing its claims again under the new one. -A SUBSCRIBE, FETCH, TRACK, TRACKS, or DATAGRAMS request names the origin node that serves it. -A subscriber that names 0 lets the publisher select one (see [Resolution](#resolution)), and a relay forwarding a request names the origin node it selected or was given. -A relay forwards a request naming an origin node only on its selected route to that node, or a SUBSCRIBE through the neighbor [Warm Routes](#warm-routes) chooses for its broadcast, and resets it with UNROUTABLE when it has no usable route to it, so the request never reaches another origin. +A SUBSCRIBE, FETCH, TRACK, TRACKS, or DATAGRAMS request names the origin node that serves it, and the Route Depth of the route it follows: how many leading segments of its Broadcast Path make up the route's prefix. +A subscriber that names 0 lets the publisher select one (see [Resolution](#resolution)), and a relay forwarding a request names the origin node and Route Depth of the route it selected or was given. +A relay forwards a request naming an origin node only along the route it sent the requester to that node at that prefix, or a SUBSCRIBE through the neighbor [Warm Routes](#warm-routes) chooses for its broadcast, and resets it with UNROUTABLE when that route is no longer usable or its Path names the requester, so the request never reaches another origin or loops. Claims with the same Instance serve the same Broadcast, so a relay MAY move a live subscription between them, including onto a higher Epoch, continuing from the first frame the subscriber lacks, and a route change (a reconnect, a cheaper path, a draining session) is invisible to the subscriber. A relay MUST NOT move a subscription to a claim with a different Instance: it stays on its Instance's claims and ends with the last of them. @@ -568,6 +562,47 @@ A receiver of an ANNOUNCE_RESTART (or of the ANNOUNCE_END and ANNOUNCE_START sta A version before this one carries no Instance, so a receiver of such a session mints one for each ANNOUNCE_START it receives, with Epoch 0, and forwards it as the claim's Instance with `Minted` set, so a reader can tell it names an announcement rather than the publisher's stream. A minted Instance is new for every announcement, so a publisher reconnecting or re-announcing through such a hop is another stream. +#### Dead Hops {#dead-hops} +A node that loses a session names the hop it lost: its own Hop ID and that session's Session Number, with the Hop ID of the node at the far end. +It sends one [DEAD](#dead) for it to each neighbour it sent a route whose Path holds the hop, covering every origin node, and each neighbour passes the hop on in the same way, once per neighbour while it has sent that neighbour a route through it. +The DEAD lists the routes on the stream that moved to another Path, each with its new Route Offer. +A sender whose moved routes do not fit one message lists them over consecutive DEADs naming the same hop, each but the last with `More` set, and the receiver applies them as one DEAD listing all of them once the last arrives. +The sender's other routes on the stream through the hop end if `Gone` is set. +Otherwise each is held, except one the sender ends with a ROUTE_END after the DEAD, and one the sender learned from the receiver itself, which the sender holds only by a ROUTE_HOLD after the DEAD: until then the receiver waits for its answer (see [Holds](#holds)). + +A receiver applies a DEAD only when a route that peer sent it holds the named hop right after a hop of the far node; otherwise it changes nothing. +Then every route through the hop is dead, whoever sent it: the sender's routes carry on as the DEAD says, and every other route through the hop is unusable until its own sender says what became of it. +The receiver remembers the dead hop, so a route through it that arrives late stays unusable. + +A closed session also casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time: a neighbour that has not heard yet may still offer a stale route through it. +A receiver doubts the far node for one second, during which it moves onto no route through that node; the route it already uses stays usable. +Nothing on the wire carries the doubt. +A node that sees a session close doubts its far end in the same way. + +On a session whose peer holds a scoped publish grant (see [AUTH_OK](#auth-ok)), a DEAD applies only to the routes heard on that session, and the receiver neither remembers the hop nor doubts the far node: a peer withdraws only what it sent, and cannot drop routes through a hop it only read in a path. +Such a peer also names itself as it likes, so its GOAWAY ranks only the routes heard on that session draining and is not passed on, and its session closing doubts no node and sends no DEAD: the routes it sent end one by one. + +A node that receives a GOAWAY ranks every route through its peer draining, and doubts the peer as a closed session's far end, so it moves at once only onto routes that avoid it. +It sends a DEAD naming the draining node to each neighbour it sent a route through that node, and each neighbour passes it on in the same way, once per neighbour while it has sent that neighbour such a route. +A receiver applies it only for a node on a route that peer sent it, ranks every route through that node draining, and doubts the node in the same way. +A draining route stays usable. + +#### Holds {#holds} +A node with no usable route to an origin node at a prefix holds the route, unavailable, while it may recover, and sends ROUTE_HOLD wherever it sent the route: + +- while the neighbour it learned the route from holds it, on the Path this node's route extended, or has yet to say what became of it after a DEAD; +- while a neighbour offers a route that only the doubt keeps this node from using; or +- until each neighbour that learned the route from this node, whose route back names this node just before itself, has answered: with another route, a ROUTE_HOLD, or a ROUTE_END. + +A neighbour that holds under this node answers with a ROUTE_HOLD rather than waiting on it, so holds never wait on each other in a cycle. +A node holds a route for at most 10 seconds: a neighbour that has not answered by then counts as answering with nothing. +A hold protects a claim's presence, not loop freedom, so ending it early is safe. +A held route's claims stay shown but unavailable until a ROUTE_UPDATE makes it usable again. +Otherwise the node lets go, sending ROUTE_END wherever it sent the route, and its claims go offline. +Neither proves that the origin withdrew its claims. +A node closes the session when an Announce Stream it answers has made no progress for 10 seconds with messages left to send, so a peer that stops reading never holds a route. +It MAY also close it when the stream falls behind repeated failures of the routes it carries, such as when DEADs it has yet to send outnumber the hops on those routes, so a peer that reads too slowly never holds every failure. + #### Resolution {#resolution} A SUBSCRIBE, FETCH, or TRACK request naming no origin node names a path, and the receiver resolves it against the claims covering that path whose route it selected and is usable. @@ -577,7 +612,7 @@ The winning tier is the whole answer: a refusal from it never falls through to a Nor is a refusal retried at another claim of the same prefix. Within the tier, a claim is ineligible while another claim of the same prefix carries its Instance at a higher Epoch, however cheap it is. -Among the eligible claims, the receiver SHOULD prefer a route whose session has not received a GOAWAY, then the highest local priority of the route's session, then the lowest cost, the claim's Announce Cost plus its route's metric, then the lowest Rendezvous Hash, and then the most recently received, so a reconnecting publisher is not outranked by the stale session it replaced. +Among the eligible claims, the receiver SHOULD prefer a route that is not draining, then the route with the highest priority, then the lowest cost, the claim's Announce Cost plus its route's metric, then the lowest Rendezvous Hash, and then the most recently received, so a reconnecting publisher is not outranked by the stale session it replaced. The Rendezvous Hash is the 64-bit FNV-1a hash, with offset basis `0x420C0DECB00B` and the standard FNV-64 prime, of the requested path's UTF-8 bytes followed by the origin's Node ID as 8 little-endian bytes. Every node that knows the same claims ranks them the same, so equal-cost origins of one prefix share its paths and every relay sends one path to the same origin. When choosing which claim to advertise for a prefix, the requested path is the prefix itself. @@ -1101,7 +1136,7 @@ The Cost Parameter declares what reaching this endpoint costs: a receiver adds t The Parameter Value is a variable-length integer in deployment-chosen units, the same units as the Metric and the Announce Cost. An absent parameter means the default cost of 1, under which a route's metric is its hop count. -A value of 0 is a PROTOCOL_VIOLATION: a free link would leave a route as close as the one it came from, which the feasibility condition relies on never happening. +A value of 0 is a PROTOCOL_VIOLATION: a free link would leave a route as cheap as the one it came from, and selection converges only because extending a route raises its metric (see [Routing](#routing)). Both endpoints send it and the two values need not match: the parameter prices the sender's own egress. A relay MUST NOT forward it. @@ -1183,7 +1218,7 @@ Indicate interest for any broadcasts with a path that starts with this prefix. Any other value is a PROTOCOL_VIOLATION. The publisher MUST respond with an ANNOUNCE_OK message followed by the matching routes and claims and any future changes, subject to [Routing](#routing). -The subscriber then sends only [ROUTE_REQUEST](#route-request) and [ROUTE_END](#route-end) echoes on the stream. +The subscriber then sends nothing more on the stream; a publisher MUST close the session with a PROTOCOL_VIOLATION if it receives more, and treats the subscriber closing its side as the end of its interest. Implementations SHOULD consider reasonable limits on the number of matching claims and routes to prevent resource exhaustion. @@ -1214,20 +1249,16 @@ An Announce Stream ending before SYNCED is a PROTOCOL_VIOLATION; a Tracks Stream ## ROUTE_START {#route-start} -A publisher sends a ROUTE_START message when it reaches an origin node (see [Routing](#routing)). -Each ROUTE_START implicitly assigns the next Route ID on the stream, later referenced by ROUTE_UPDATE, ROUTE_HOLD, ROUTE_END, ANNOUNCE_FLOOR, ANNOUNCE_START, and WARM_START (see [Announce](#announce)). +A publisher sends a ROUTE_START message when it reaches an origin node for a prefix (see [Routing](#routing)). +Each ROUTE_START implicitly assigns the next Route ID on the stream, later referenced by ROUTE_UPDATE, ROUTE_HOLD, ROUTE_END, DEAD, ANNOUNCE_FLOOR, ANNOUNCE_START, and WARM_START (see [Announce](#announce)). ~~~ ROUTE_START Message { Type (i) = 0x6 Message Length (i) Node ID (i), - Seqno (i), - Metric (i), - Flags (8), - [RTT (i),] - Hop Count (i), - Hop ID (i) ..., + Route Suffix (s), + Route Offer (..), } ~~~ @@ -1238,28 +1269,56 @@ Set to 0x6 to indicate a ROUTE_START message. The origin node, as the publisher names it on this session. 0 is refused (see [Refused Content](#refused-content)). -**Seqno**: -The origin's sequence number of the route, which only the origin advances. -2^64-1 is refused. +**Route Suffix**: +The route's prefix, relative to the requested prefix: the route covers the requested prefix joined with it, and is empty for a route at the requested prefix itself. +A suffix that is not a valid path, or that joins the requested prefix past the limits of a broadcast path, is refused. -**Metric**: -The publisher's metric to the origin node, 0 when it is the origin. +**Route Offer**: +The route, as below. + +### Route Offer {#route-offer} +What a route says: its Path, then its attributes. +ROUTE_START, ROUTE_UPDATE, and a DEAD's moved routes carry it. + +~~~ +Route Offer { + Hop Count (i), + Hop (..) ..., + Flags (8), + Priority (8), + Metric (i), + [RTT (i),] +} + +Hop { + Hop ID (i), + Session Number (i), +} +~~~ + +**Hop Count**: +The number of Hops that follow. +0 is a PROTOCOL_VIOLATION, since the publisher names itself last; more than 32 is refused. + +**Hop**: +The route's Path (see [Routing](#routing)), in order, the origin node's first and the publisher's own last: each node's Hop ID and the Session Number it learned the route on, 0 for the origin's own. +A Hop ID of 0 is refused. +The Path comes first, so a receiver that refuses the rest of the offer still learns the hops a later [DEAD](#dead) may name. **Flags**: -A bit field: 0x1 when the route is down-only (see [Routing](#routing)), and 0x2 when an RTT follows. -Any other bit set is refused, rather than ignored: a later version may give a bit a meaning that changes the route, or a field ahead of the Hops. +A bit field: 0x1 when the route is down-only, 0x2 when an RTT follows, and 0x4 when the route is draining (see [Routing](#routing)). +Any other bit set is refused, rather than ignored: a later version may give a bit a meaning that changes the route, or a field ahead of the ones defined here. + +**Priority**: +The route's priority, the lowest local priority of a session on its Path, as a u8 with 128 as the midpoint; higher ranks first. + +**Metric**: +The publisher's metric to the origin node, 0 when it is the origin. **RTT**: The publisher's RTT to the origin node in milliseconds (see [Routing](#routing)), 0 when it is the origin. Present only when Flags has 0x2 set, and absent when the publisher has none. -**Hop Count**: -The number of Hop IDs that follow. -More than 32 is refused. - -**Hop ID**: -The route's Hops (see [Routing](#routing)), in order, the publisher's own last. - ## ROUTE_UPDATE {#route-update} A publisher sends a ROUTE_UPDATE message when a route it sent changes, referencing its Route ID. @@ -1269,12 +1328,7 @@ ROUTE_UPDATE Message { Type (i) = 0x7 Message Length (i) Route ID (i), - Seqno (i), - Metric (i), - Flags (8), - [RTT (i),] - Hop Count (i), - Hop ID (i) ..., + Route Offer (..), } ~~~ @@ -1284,14 +1338,14 @@ Set to 0x7 to indicate a ROUTE_UPDATE message. **Route ID**: The ordinal implicitly assigned by a prior ROUTE_START on this stream. -**Seqno**, **Metric**, **Flags**, **RTT**, **Hop Count**, and **Hop ID**: +**Route Offer**: As defined for [ROUTE_START](#route-start), replacing the route's. A ROUTE_UPDATE for a held route makes it usable again. ## ROUTE_HOLD {#route-hold} -A publisher sends a ROUTE_HOLD message when it has no usable route to the origin node but still holds the route and its claims while it recovers (see [Routing](#routing)), referencing its Route ID. -The receiver keeps the route's last Seqno, Metric, Flags, RTT, and Hops, and does not select it until a ROUTE_UPDATE. +A publisher sends a ROUTE_HOLD message when it has no usable route to the origin node but still holds the route and its claims while it recovers (see [Holds](#holds)), referencing its Route ID. +The receiver keeps the route's last Route Offer, and does not use the route until a ROUTE_UPDATE. ~~~ ROUTE_HOLD Message { @@ -1313,9 +1367,6 @@ A publisher sends a ROUTE_END message when it lets go of a route, referencing it It retires the Route ID and every claim under the route, whose ANNOUNCE_END the publisher does not send. It claims nothing about the origin's claims, which may still be served elsewhere. -The subscriber sends the same message back on the stream once it dropped the route, its echo (see [Routing](#routing)). -A subscriber MUST send exactly one echo for each ROUTE_END, and a publisher MUST close the session with a PROTOCOL_VIOLATION on an echo naming a Route ID it has not ended or already saw echoed. - ~~~ ROUTE_END Message { Type (i) = 0x8 @@ -1331,31 +1382,73 @@ Set to 0x8 to indicate a ROUTE_END message. The ordinal implicitly assigned by a prior ROUTE_START on this stream. -## ROUTE_REQUEST {#route-request} -A subscriber sends a ROUTE_REQUEST message to ask for a route to an origin node at least as new as a Seqno (see [Routing](#routing)). -It names the origin node rather than a Route ID, so it stays valid while a ROUTE_END for that route is in flight. +## DEAD {#dead} +A publisher sends a DEAD message when a session on the Path of routes it sent closed, or a node on it sent a GOAWAY (see [Dead Hops](#dead-hops)). +It covers every origin node, and assigns no id. ~~~ -ROUTE_REQUEST Message { - Type (i) = 0xA +DEAD Message { + Type (i) = 0xF Message Length (i) - Node ID (i), - Seqno (i), + Kind (i), + [Closed (..),] + [Drain (..),] +} + +Closed { + Hop (..), + Far Hop ID (i), + Gone (8), + More (8), + Moved Count (i), + Moved (..) ..., +} + +Moved { + Route ID (i), + Offer Length (i), + Route Offer (..), +} + +Drain { + Hop ID (i), } ~~~ **Type**: -Set to 0xA to indicate a ROUTE_REQUEST message. +Set to 0xF to indicate a DEAD message. -**Node ID**: -The origin node, as the publisher names it on this session. +**Kind**: +0 when a session closed, followed by Closed, and 1 when a node sent a GOAWAY, followed by Drain. +Any other value is a PROTOCOL_VIOLATION. + +**Hop**: +The hop that died: the Hop ID of the node that lost the session, and that session's Session Number, as a [Route Offer](#route-offer)'s Path names it. +A Hop ID of 0 is a PROTOCOL_VIOLATION. + +**Far Hop ID**: +The Hop ID of the node at the session's other end, which the receiver doubts. 0 is a PROTOCOL_VIOLATION. -**Seqno**: -The Seqno the subscriber needs. -A publisher with no route to the origin node drops the request. +**Gone**: +1 when every route on the stream through the hop that Moved does not list has ended, 0 when the rest are held (see [Dead Hops](#dead-hops)). +Any other value is a PROTOCOL_VIOLATION. + +**More**: +1 when the next message on the stream is a DEAD continuing this one's Moved list, which the receiver applies with it, and 0 for the last or only part. +A part with More set followed by any other message, or by a DEAD whose Kind, Hop, Far Hop ID, or Gone differs, is a PROTOCOL_VIOLATION. +Any other value is a PROTOCOL_VIOLATION. + +**Moved Count**: +The number of Moved entries that follow. -A publisher MUST close the session with a PROTOCOL_VIOLATION if it receives any message type other than ROUTE_REQUEST or ROUTE_END after the ANNOUNCE_REQUEST. +**Moved**: +A route on the stream through the hop that moved to another Path: its Route ID, which MUST name a live route, and its new [Route Offer](#route-offer), preceded by the offer's length in bytes. +A refused offer leaves the receiver no route, as a refused ROUTE_UPDATE does, and the length lets it read the next entry; an offer that does not fill its length exactly is a PROTOCOL_VIOLATION. + +**Hop ID** (Drain): +The Hop ID of the node that sent the GOAWAY. +0 is a PROTOCOL_VIOLATION. ## ANNOUNCE_START {#announce-start} @@ -1633,6 +1726,7 @@ SUBSCRIBE Message { Instance (b) Minted (8) Node ID (i) + Route Depth (i) Track Name (s) Subscriber Priority (8) Group Order (8) @@ -1662,7 +1756,7 @@ Anchor { A unique identifier chosen by the subscriber. A Subscribe ID MUST NOT be reused within the same session, even if the prior subscription has been closed. -**Instance**, **Minted**, and **Node ID**: +**Instance**, **Minted**, **Node ID**, and **Route Depth**: As in [TRACK](#track). **Subscriber Priority**: @@ -1784,6 +1878,7 @@ TRACK Message { Instance (b) Minted (8) Node ID (i) + Route Depth (i) Track Name (s) } ~~~ @@ -1800,6 +1895,11 @@ A relay fills in the Instance and Minted of the claim it resolved when forwardin The origin node that serves the request, as the publisher names it on this session, or 0 for whichever the publisher selects (see [Routing](#routing)). TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS carry it the same way. +**Route Depth**: +How many leading segments of the Broadcast Path make up the prefix of the route to the origin node the request follows (see [Routing](#routing)), 0 for a route at the empty prefix. +It MUST be 0 when the Node ID is 0, and at most the Broadcast Path's segments; any other value is a PROTOCOL_VIOLATION. +TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS carry it the same way. + **Track Name**: The name of the track. @@ -1867,13 +1967,14 @@ TRACKS Message { Instance (b) Minted (8) Node ID (i) + Route Depth (i) } ~~~ **Broadcast Path**: The path of the Broadcast to list. -**Instance**, **Minted**, and **Node ID**: +**Instance**, **Minted**, **Node ID**, and **Route Depth**: As in [TRACK](#track). The listing is pinned to the stream that answers it and ends with that stream. @@ -1935,6 +2036,7 @@ DATAGRAMS Message { Instance (b) Minted (8) Node ID (i) + Route Depth (i) Track Name (s) Subscriber Priority (8) } @@ -2145,6 +2247,7 @@ FETCH Message { Instance (b) Minted (8) Node ID (i) + Route Depth (i) Track Name (s) Subscriber Priority (8) Subscriber Max Age (i) @@ -2158,7 +2261,7 @@ FETCH Message { **Broadcast Path**: The broadcast path of the track to fetch from. -**Instance**, **Minted**, and **Node ID**: +**Instance**, **Minted**, **Node ID**, and **Route Depth**: As in [TRACK](#track). **Track Name**: @@ -2445,19 +2548,20 @@ An application-specific payload. - An advertiser MAY announce a path it answers through a claim under the claim's Instance, so subscriptions resolved through the claim continue, and a claim still serving that Instance when the announcement ends restarts under a new one. - A receiver mints an Instance for each announcement a session of an earlier version delivers, and a `Minted` field beside every Instance (ANNOUNCE_START, ANNOUNCE_RESTART, TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS) says whether a receiver minted it, so no Instance text is reserved for minting. - Added announce compression: ANNOUNCE_START gains `Path Base` and `Path Keep` to copy the head of a live claim's suffix. -- Split routing into two layers on the Announce Stream. ROUTE_START (0x6), ROUTE_UPDATE (0x7), and ROUTE_END (0x8) advertise how the publisher reaches an origin node, named by a Node ID, with a seqno, an additive metric, and a down-only bit, and loops are prevented by Babel's feasibility condition. Every ANNOUNCE_START names its route's Route ID in place of the Hop ID list, so a link change sends one ROUTE_UPDATE per origin rather than an update per claim. Removed the Hop ID list and ANNOUNCE_OK's Hop ID, and replaced the SETUP Hop parameter (0x5) with a [Node ID parameter](#node-id-parameter), so an endpoint that reaches itself detects it at setup. +- Split routing into two layers on the Announce Stream. ROUTE_START (0x6), ROUTE_UPDATE (0x7), and ROUTE_END (0x8) advertise how the publisher reaches an origin node, named by a Node ID, for the paths under a prefix, with an additive metric and a down-only bit. Every ANNOUNCE_START names its route's Route ID in place of the Hop ID list, so a link change sends one ROUTE_UPDATE per origin and prefix rather than an update per claim. Removed the Hop ID list and ANNOUNCE_OK's Hop ID, and replaced the SETUP Hop parameter (0x5) with a [Node ID parameter](#node-id-parameter), so an endpoint that reaches itself detects it at setup. - ROUTE_START and ROUTE_UPDATE carry Flags, whose 0x2 bit adds an optional RTT: the informational sum of each link's lowest smoothed RTT over the last 30 seconds to the origin node, which never selects a route. -- Added Hops to ROUTE_START and ROUTE_UPDATE: the Hop ID of every node the route crossed, at most 32. A node never selects a route whose Hops name it, so one that dropped its feasibility distance cannot take back a copy of its own route. A zero or repeated Hop ID in a route's Hops is tolerated, where a repeated non-zero Hop ID in moq-lite-06's Hop ID list is a PROTOCOL_VIOLATION. -- Added ROUTE_HOLD (0xB), which holds a route while it recovers; ROUTE_END lets it go. Neither withdraws the origin's claims, and the subscriber echoes every ROUTE_END back. -- Added ROUTE_REQUEST (0xA), sent by the subscriber on the Announce Stream to ask for a route with a newer seqno when every route it has is infeasible. -- Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the requested prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows. +- ROUTE_START and ROUTE_UPDATE carry a Route Offer that leads with the route's Path: one hop per node the route crossed, each a Hop ID and the Session Number that node learned the route on, at most 32. A node never uses a route whose Path names it. ROUTE_START also carries a Route Suffix naming the prefix the route covers, and the offer carries the route's drain (Flags 0x4) and priority, which rank before its metric. +- Added ROUTE_HOLD (0xB), which holds a route while it may recover, for at most 10 seconds; ROUTE_END lets it go. Neither withdraws the origin's claims. +- Added DEAD (0xF): a node that loses a session names that hop once to each neighbour it sent a route through it, listing the routes that moved, and every node drops every route through the hop and doubts the node at the far end for a second; a DRAIN names a node that sent a GOAWAY, whose routes rank draining. The path, the dead hop, holds, and the doubt keep routes free of loops and stale copies, with no distance, sequence number, route request, or echo. On a scoped session a DEAD applies only to the routes heard there. +- A node closes a session whose peer leaves an Announce Stream undrained for 10 seconds, and the subscriber sends nothing on the stream after ANNOUNCE_REQUEST. +- Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the route's prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows. - A route presents only its most specific claim covering the requested prefix, moved to the next covering claim when it is withdrawn rather than ended. - A subscriber whose scope confines a stream to one prefix SHOULD request that prefix, and replace the request when the prefix changes. -- A route goes on a stream only while it has a claim the request presents, and ends after the batch withdrawing its last claim. It starts on that stream again only once the subscriber echoed the ROUTE_END. -- Added `Node ID` to TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS: the origin node the first hop selected, followed by every later hop. +- A route goes on a stream only while it has a claim the request presents, and ends after the batch withdrawing its last claim. +- Added `Node ID` and `Route Depth` to TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS: the origin node the first hop selected and the prefix of the route it follows, which every later hop follows. - Replaced Warm and Cold Route Cost with the origin's Announce Cost, unchanged in transit. ANNOUNCE_UPDATE carries only the Announce Cost. - The Cost parameter must be at least 1. -- Replaced the shortest-path, Hop ID, Instance Hash, and Chain Hash tie-breaks with the local priority of the route's session, the Announce Cost plus the route's metric, and a Rendezvous Hash of the requested path and the origin's Node ID. +- Replaced the shortest-path, Hop ID, Instance Hash, and Chain Hash tie-breaks with the route's drain and priority, the Announce Cost plus the route's metric, and a Rendezvous Hash of the requested path and the origin's Node ID. - Removed the standby-ordering deployment requirements on Route Cost seeds and bounds. - Added warm routes: WARM_START (0xC), WARM_UPDATE (0xD), and WARM_END (0xE) on the Announce Stream, and `Warm` on SUBSCRIBE and SUBSCRIBE_UPDATE. A relay MAY pull each broadcast through a neighbor that offers a copy of it, carrying the copy's Hops from its carrier, its Warm Cost, its drain, and its priority. A carrier offers its copy whatever it pulls and joins only a lower-ranked carrier's; a relay with no reader offers each neighbor its best source that avoids it, pulls it only while that is what it offered each relay it serves, and follows its route when a neighbor following its own pulls through it. The offers settle as a path vector does, and the pulls are then free of loops. - Added the temporary Lite-06 Chain Setup Parameter (0x9) and a trailing Lite-06 Chain on ANNOUNCE_START, ANNOUNCE_UPDATE, and ANNOUNCE_RESTART: the moq-lite-06 Hop ID list of a claim learned over moq-lite-06, which a node drops when it names that node. @@ -2514,7 +2618,7 @@ An application-specific payload. - FRAME leads with `Payload Length`, the payload size plus one, in place of the trailing `Message Length`; 0 is malformed. - A publisher SHOULD drop a datagram rather than block when the transport has no room for it, where it previously sent one only if the congestion controller could transmit it immediately. - Added an optional `Subscriber Max Age` to FETCH, encoded as milliseconds plus one with zero meaning absent. It can tighten, but never relax, `Publisher Max Age`. -- A receiver refuses forwarded route content it cannot use, such as an unknown route flag, more than 32 Hops, or an invalid claim path or Instance, withdrawing only that route or claim rather than closing the session ([Refused Content](#refused-content)). A refused claim goes offline, and its route's floors certify nothing while it holds one. +- A receiver refuses forwarded route content it cannot use, such as an unknown route flag, more than 32 hops on a route's Path, or an invalid claim path or Instance, withdrawing only that route or claim rather than closing the session ([Refused Content](#refused-content)). A refused claim goes offline, and its route's floors certify nothing while it holds one. ## moq-lite-06 diff --git a/js/net/src/connection/forward.test.ts b/js/net/src/connection/forward.test.ts index fded70ba6f0..2c13e3d9156 100644 --- a/js/net/src/connection/forward.test.ts +++ b/js/net/src/connection/forward.test.ts @@ -69,6 +69,9 @@ class FakeSession { /** Called as the forwarder closes a request, before its stream can wind down. */ onClose?: () => void; + /** Whether a lite-07 claim of `node` at `at` serves `path` beneath it: the session's floors. */ + covering?: (node: bigint, at: Path.Valid, path: Path.Valid) => boolean; + closed: Promise; #die!: (reason: Error | null) => void; @@ -76,6 +79,7 @@ class FakeSession { constructor({ exclusive = false } = {}) { registerWire(this, { consume: (path, instance) => this.consume(path, instance), + covers: (node, at, path) => this.covering?.(node, at, path) ?? true, exclusive, leave: () => {}, auth: new AuthSession({ peerGrant: { publish: new Path.Patterns(), subscribe: new Path.Patterns() } }), @@ -212,6 +216,43 @@ test("the origin holds each origin node's claim at a prefix", async () => { origin.close(); }); +// Two streams carry a lite-07 node's claims, and the newer floor, on the narrower one, presents +// nothing at live/x: the node's claim at live is older knowledge there. A request for live/x skips +// it, per request, for another node's claim at live of another instance. Mirrors `covering` in +// rs/moq-net. +test("the origin skips a claim above a newer floor's region", async () => { + const origin = new OriginProducer(); + const session = new FakeSession(); + forwardAnnounced(session.session, origin); + const a = Instance.parse("a"); + const b = Instance.parse("b"); + const NODE_A = 0x7an; + const NODE_B = 0x8bn; + const live = origin.routes(Path.Pattern.subtree(Path.from("live"))); + await settle(); + start(session.stream("live/**"), "live", { ...WireRoute.default, instance: a, node: NODE_A }); + start(session.stream("live/**"), "live", { ...WireRoute.default, instance: b, node: NODE_B, cost: 5n }); + session.stream("live/**").synced(); + await settle(); + const x = Path.from("live/x"); + session.covering = (node, at, path) => !(node === NODE_A && at === Path.from("live") && Path.hasPrefix(x, path)); + + const request = origin.request(x); + await settle(); + expect(request.active.peek()).toBeDefined(); + expect(session.pins).toEqual([b]); + // Elsewhere under live the cheaper claim still serves. + const y = origin.request(Path.from("live/y")); + await settle(); + expect(session.pins).toEqual([b, a]); + + request.close(); + y.close(); + live.close(); + session.die(); + origin.close(); +}); + // A route handed to another request's report of another instance restarts, so its requests // consume that instance rather than the one the first report named. test("a route handed over to another instance consumes that instance", async () => { diff --git a/js/net/src/connection/forward.ts b/js/net/src/connection/forward.ts index 07032b3e8da..7afe4762801 100644 --- a/js/net/src/connection/forward.ts +++ b/js/net/src/connection/forward.ts @@ -88,7 +88,15 @@ interface Reported { */ export function forwardAnnounced(conn: Session, origin: OriginProducer): void { const originWire = wireOf(origin); - const exclusive = wireOf(conn).exclusive; + const session = wireOf(conn); + const exclusive = session.exclusive; + // A lite-07 claim covers only what its origin node's newest floor on the session still + // certifies beneath it, which the origin asks per request. Mirrors `covering` in rs/moq-net. + const covering = (prefix: Path.Valid, route: WireRoute) => { + const node = route.node; + if (node === undefined || !session.covers) return undefined; + return (path: Path.Valid) => session.covers?.(node, prefix, path) ?? true; + }; // Reassigned on a downgrade, so the origin stops counting this session as discovering. // Called through a closure so the session's death always detaches the current one. let detach = originWire.attach(true); @@ -111,7 +119,7 @@ export function forwardAnnounced(conn: Session, origin: OriginProducer): void { const existing = routes.get(key); if (!existing) { - const handle = originWire.receive(prefix, route); + const handle = originWire.receive(prefix, route, covering(prefix, route)); routes.set(key, { prefix, handle, reporters: new Map([[interest, route]]) }); void drive(handle, conn); return; @@ -135,7 +143,7 @@ export function forwardAnnounced(conn: Session, origin: OriginProducer): void { if (restart || route.instance !== current?.instance || route.epoch !== current?.epoch) { // Inserted before the old one leaves, so local readers see one restart rather than // an end and a start. - const handle = originWire.receive(existing.prefix, route); + const handle = originWire.receive(existing.prefix, route, covering(existing.prefix, route)); existing.handle.close(); existing.handle = handle; void drive(handle, conn); diff --git a/js/net/src/consume.test.ts b/js/net/src/consume.test.ts index 751273173ac..c486093c684 100644 --- a/js/net/src/consume.test.ts +++ b/js/net/src/consume.test.ts @@ -206,11 +206,11 @@ test("the node serving an instance follows its routes even when another instance const c = {}; cache.announce(path, a, { ...WireRoute.default, instance: x, node: 1n }); cache.announce(path, b, { ...WireRoute.default, instance: x, node: 2n, cost: 5n }); - expect(cache.node(path, x)).toBe(1n); + expect(cache.node(path, x)).toEqual({ node: 1n, depth: 0 }); cache.withdraw(path, a); cache.announce(path, c, { ...WireRoute.default, instance: Instance.random(), node: 3n }); expect(cache.instance(path).route.node).toBe(3n); - expect(cache.node(path, x)).toBe(2n); + expect(cache.node(path, x)).toEqual({ node: 2n, depth: 0 }); cache.withdraw(path, b); expect(cache.node(path, x)).toBeUndefined(); }); diff --git a/js/net/src/consume.ts b/js/net/src/consume.ts index 8d01fb0c43e..59714c148ff 100644 --- a/js/net/src/consume.ts +++ b/js/net/src/consume.ts @@ -39,7 +39,10 @@ class Validity { } // The route a consume selected: its instance, and the origin node its requests name (lite-07). -type Selection = Served & { readonly route: Pick; readonly validity?: Validity }; +type Selection = Served & { + readonly route: Pick; + readonly validity?: Validity; +}; class Blind implements Selection { readonly identity = {}; @@ -148,11 +151,13 @@ export class BroadcastCache { } /** - * The origin node of the best route now serving `instance` at `path` (lite-07), whichever - * instance wins the path, or undefined when no route names one. + * The publisher node of the best route now serving `instance` at `path` (lite-07), and the + * depth of that route's prefix, whichever instance wins the path, or undefined when no route + * names one. */ - node(path: Path.Valid, instance: Instance.Valid): bigint | undefined { - return this.pinned(path, instance).route.node; + node(path: Path.Valid, instance: Instance.Valid): { node: bigint; depth: number } | undefined { + const { node, depth } = this.pinned(path, instance).route; + return node === undefined ? undefined : { node, depth: depth ?? 0 }; } /** Clone the live cached consumer only when it serves the selected instance. */ diff --git a/js/net/src/hop.ts b/js/net/src/hop.ts index ab69cc998e9..02aacf8340e 100644 --- a/js/net/src/hop.ts +++ b/js/net/src/hop.ts @@ -186,6 +186,17 @@ export interface WireRoute { * see it. */ readonly node?: bigint; + /** + * How many leading segments of a requested path make up the prefix of the lite-07 route to + * {@link node}, which every request through it names. Local, as `node` is. + */ + readonly depth?: number; + /** + * The lowest priority of a link on the path of the lite-07 route to {@link node}: higher + * ranks first, after drain and before cost, as routes to one node rank. Unset ranks as 0, the + * priority of a link no one set. Local, as `node` is. + */ + readonly priority?: number; /** See {@link RouteInfo.rtt}. */ readonly rtt?: Duration; } @@ -210,6 +221,8 @@ export const WireRoute = { draining: route.draining, available: route.available, node: route.node, + depth: route.depth, + priority: route.priority, rtt: route.rtt, }; }, @@ -229,7 +242,7 @@ export function isAnonymous(route: WireRoute): boolean { /** * Whether two routes name the same instance, epoch, hop chain, cost, auth, drain state, - * availability, origin node, and RTT. + * availability, origin node, route depth and priority, and RTT. * * @internal */ @@ -244,6 +257,8 @@ export function routesEqual(a: WireRoute | undefined, b: WireRoute | undefined): (a.draining ?? false) === (b.draining ?? false) && (a.available ?? true) === (b.available ?? true) && a.node === b.node && + a.depth === b.depth && + (a.priority ?? 0) === (b.priority ?? 0) && (a.rtt === b.rtt || (!!a.rtt && !!b.rtt && a.rtt.eq(b.rtt))) && a.hops.length === b.hops.length && a.hops.every((hop, i) => hop === b.hops[i]) diff --git a/js/net/src/integers.test.ts b/js/net/src/integers.test.ts index d6b4c6052c9..bc7ba1a35fe 100644 --- a/js/net/src/integers.test.ts +++ b/js/net/src/integers.test.ts @@ -165,10 +165,11 @@ for (const version of [Lite.DRAFT_06, Lite.DRAFT_07]) { const body = await Writer.collect(version, async (w) => { await w.string("room"); if (version === Lite.DRAFT_07) { - // No pinned instance: empty and unminted, and no origin node. + // No pinned instance: empty and unminted, and no origin node nor route depth. await w.u53(0); await w.bool(false); await w.u62(0n); + await w.u53(0); } await w.string("video"); await w.u8(0); diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index b232445af2e..164c671d701 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -1502,12 +1502,14 @@ export class Epochs { /** * Prefer a route that is neither draining (a peer that sent GOAWAY) nor unavailable (held), - * which rank below every other, then identified publishers, then lower static cost. + * which rank below every other, then the highest priority on its path, then identified + * publishers, then lower static cost. */ export function compareRoutes(a: WireRoute, b: WireRoute): number { const last = (route: WireRoute) => Number((route.draining ?? false) || route.available === false); return ( last(a) - last(b) || + (b.priority ?? 0) - (a.priority ?? 0) || Number(a.node === undefined && isAnonymous(a)) - Number(b.node === undefined && isAnonymous(b)) || (a.cost < b.cost ? -1 : a.cost > b.cost ? 1 : 0) ); diff --git a/js/net/src/lite/announce.test.ts b/js/net/src/lite/announce.test.ts index d6828203adc..1374a58283f 100644 --- a/js/net/src/lite/announce.test.ts +++ b/js/net/src/lite/announce.test.ts @@ -8,13 +8,11 @@ import { type AnnounceBroadcast, AnnounceHistory, AnnounceOk, - type AnnounceReply, AnnounceRequest, decodeAnnounceBroadcast, decodeAnnounceBroadcastMaybe, - decodeAnnounceReplyMaybe, encodeAnnounceBroadcast, - encodeAnnounceReply, + type Offer, ROUTE_MAX, ROUTE_MAX_HOPS, } from "./announce.ts"; @@ -376,31 +374,91 @@ test("the literal draft-07 stream matches what Rust decodes", async () => { // `rs/moq-net/fixtures/lite.json` pins the same bytes on both sides. test("the route layer round-trips on draft-07", async () => { const v = Version.DRAFT_07; - const advert = { seqno: 3n, metric: 5n, downOnly: true, rtt: undefined, hops: [7n, 9n] }; + const offer: Offer = { + path: [ + { node: 7n, session: 0n }, + { node: 9n, session: 2n }, + ], + draining: false, + priority: 0, + metric: 5n, + downOnly: true, + rtt: undefined, + }; + const one: Offer = { ...offer, path: [{ node: 7n, session: 0n }], downOnly: false, metric: 0n }; const cases: [AnnounceBroadcast, number[]][] = [ - [{ status: "routeStart", node: 0x1234n, advert }, [6, 8, 0x92, 0x34, 3, 5, 1, 2, 7, 9]], - [{ status: "routeUpdate", id: 1n, advert: { ...advert, hops: [] } }, [7, 5, 1, 3, 5, 1, 0]], - // Flag 0x2: an RTT follows, 0 at the origin, whose Hops are its own alone. + // The path first, the publisher first, then flags, priority (offset by 128), and metric. + [ + { status: "routeStart", node: 0x1234n, suffix: Path.empty(), offer }, + [6, 11, 0x92, 0x34, 0, 2, 7, 0, 9, 2, 1, 128, 5], + ], + // Flag 0x2: an RTT follows. Flag 0x4: a node on the path drains. [ - { status: "routeStart", node: 0x1234n, advert: { ...advert, downOnly: false, rtt: 0n, hops: [0x1234n] } }, - [6, 9, 0x92, 0x34, 3, 5, 2, 0, 1, 0x92, 0x34], + { + status: "routeUpdate", + id: 1n, + offer: { + ...offer, + path: [{ node: 7n, session: 0n }], + draining: true, + priority: -3, + downOnly: false, + rtt: 25n, + }, + }, + [7, 8, 1, 1, 7, 0, 6, 125, 5, 25], ], - [{ status: "routeUpdate", id: 1n, advert: { ...advert, rtt: 25n, hops: [] } }, [7, 6, 1, 3, 5, 3, 25, 0]], [ - { status: "routeUpdate", id: 1n, advert: { ...advert, rtt: ROUTE_MAX, hops: [ROUTE_MAX] } }, - [7, 23, 1, 3, 5, 3, ...Array(9).fill(0xff), 1, ...Array(9).fill(0xff)], + { status: "routeUpdate", id: 1n, offer: { ...offer, rtt: ROUTE_MAX } }, + [7, 18, 1, 2, 7, 0, 9, 2, 3, 128, 5, ...Array(9).fill(0xff)], ], // As many hops as a route may carry. [ { status: "routeUpdate", id: 1n, - advert: { ...advert, hops: Array.from({ length: ROUTE_MAX_HOPS }, (_, i) => BigInt(i + 1)) }, + offer: { + ...one, + path: Array.from({ length: ROUTE_MAX_HOPS }, (_, i) => ({ node: BigInt(i + 1), session: 0n })), + }, }, - [7, 37, 1, 3, 5, 1, ROUTE_MAX_HOPS, ...Array.from({ length: ROUTE_MAX_HOPS }, (_, i) => i + 1)], + [ + 7, + 5 + 2 * ROUTE_MAX_HOPS, + 1, + ROUTE_MAX_HOPS, + ...Array.from({ length: ROUTE_MAX_HOPS }, (_, i) => [i + 1, 0]).flat(), + 0, + 128, + 0, + ], ], [{ status: "routeHold", id: 1n }, [0xb, 1, 1]], [{ status: "routeEnd", id: 1n }, [8, 1, 1]], + [{ status: "dead", dead: { kind: "drain", node: 7n } }, [15, 2, 1, 7]], + // Each moved offer is sized, so a receiver refusing one still reads the next. + [ + { + status: "dead", + dead: { + kind: "closed", + hop: { node: 7n, session: 3n }, + far: 9n, + gone: false, + more: false, + moved: [{ id: 1n, offer: one }], + }, + }, + [15, 15, 0, 7, 3, 9, 0, 0, 1, 1, 6, 1, 7, 0, 0, 128, 0], + ], + // More: the next DEAD continues this one's moved routes. + [ + { + status: "dead", + dead: { kind: "closed", hop: { node: 7n, session: 3n }, far: 9n, gone: true, more: true, moved: [] }, + }, + [15, 7, 0, 7, 3, 9, 1, 1, 0], + ], // Revision 4 travels as 5: 0 certifies nothing. [{ status: "floor", id: 1n, revision: 4n }, [9, 2, 1, 5]], [{ status: "floor", id: 1n, revision: undefined }, [9, 2, 1, 0]], @@ -422,96 +480,128 @@ test("the route layer round-trips on draft-07", async () => { await expect(bytes((w) => encodeAnnounceBroadcast(w, { ...literal, hops: [toHop(1n)] }, v), v)).rejects.toThrow(); }); -test("the route layer refuses the last seqno, Node ID 0, unknown flags, and too many hops", async () => { +test("the route layer refuses Node ID 0, unknown flags, and too many hops", async () => { const v = Version.DRAFT_07; - const last = { seqno: ROUTE_MAX, metric: 0n, downOnly: false, rtt: undefined, hops: [] }; - await expect( - bytes((w) => encodeAnnounceBroadcast(w, { status: "routeStart", node: 1n, advert: last }, v), v), - ).rejects.toThrow(); - const crowded = { ...last, seqno: 0n, hops: Array.from({ length: ROUTE_MAX_HOPS + 1 }, () => 1n) }; + const offer: Offer = { + path: [{ node: 1n, session: 0n }], + draining: false, + priority: 0, + metric: 0n, + downOnly: false, + rtt: undefined, + }; + const encode = (msg: AnnounceBroadcast) => bytes((w) => encodeAnnounceBroadcast(w, msg, v), v); + // No node sends an offer it would refuse. + await expect(encode({ status: "routeStart", node: 0n, suffix: Path.empty(), offer })).rejects.toThrow(); + await expect(encode({ status: "routeUpdate", id: 0n, offer: { ...offer, path: [] } })).rejects.toThrow(); await expect( - bytes((w) => encodeAnnounceBroadcast(w, { status: "routeStart", node: 1n, advert: crowded }, v), v), + encode({ status: "routeUpdate", id: 0n, offer: { ...offer, path: [{ node: 0n, session: 0n }] } }), ).rejects.toThrow(); + const crowded = Array.from({ length: ROUTE_MAX_HOPS + 1 }, () => ({ node: 1n, session: 0n })); + await expect(encode({ status: "routeUpdate", id: 0n, offer: { ...offer, path: crowded } })).rejects.toThrow(); await expect( - bytes((w) => encodeAnnounceReply(w, { kind: "request", node: 0n, seqno: 1n }, v), v), + encode({ + status: "dead", + dead: { + kind: "closed", + hop: { node: 1n, session: 0n }, + far: 2n, + gone: false, + more: false, + moved: [{ id: 0n, offer: { refused: "flags", path: offer.path } }], + }, + }), ).rejects.toThrow(); // Built by hand, as only a nonconforming peer sends them. - const forged = async (body: (w: Writer) => Promise) => { + const forged = async (type: number, body: (w: Writer) => Promise) => { const inner = await bytes(body, v); return bytes(async (w) => { - await w.u53(6); + await w.u53(type); await w.u53(inner.byteLength); await w.write(inner); }, v); }; - const zero = await forged(async (w) => { - await w.u62(0n); - await w.u62(0n); - await w.u62(0n); - await w.u8(0); - }); + // A ROUTE_START to `node` along `path`, its flags `flags`, and nothing after them. + const start = (node: bigint, path: bigint[], flags: number) => + forged(6, async (w) => { + await w.u62(node); + await w.u53(0); + await w.u62(BigInt(path.length)); + for (const hop of path) { + await w.u62(hop); + await w.u62(0n); + } + await w.u8(flags); + await w.u8(128); + await w.u62(0n); + }); // What the sender only forwarded is refused, the message decoded all the same, so the receiver - // withdraws the route and keeps the stream. + // withdraws the route and keeps the stream, with the path a later DEAD may name. const decode = (wire: Uint8Array) => decodeAnnounceBroadcast(new Reader(undefined, wire, v), v); - expect(await decode(zero)).toEqual({ + const path = [{ node: 1n, session: 0n }]; + expect(await decode(await start(0n, [1n], 0))).toEqual({ status: "refused", - message: { kind: "routeStart", node: 0n }, + message: { kind: "routeStart", node: 0n, suffix: Path.empty(), path }, reason: "node", }); - const maxed = await forged(async (w) => { + expect(await decode(await start(1n, [0n], 0))).toMatchObject({ status: "refused", reason: "node" }); + // A flag no version defines is refused, and the RTT flag without its field is malformed. + expect(await decode(await start(1n, [1n], 8))).toEqual({ + status: "refused", + message: { kind: "routeStart", node: 1n, suffix: Path.empty(), path }, + reason: "flags", + }); + await expect(decode(await start(1n, [1n], 2))).rejects.toThrow(); + // More hops than a route may carry. + const over = Array.from({ length: ROUTE_MAX_HOPS + 1 }, () => 1n); + expect(await decode(await start(1n, over, 0))).toMatchObject({ status: "refused", reason: "hops" }); + // Far past the limit, only the hops up to one past it are kept. + const far = Array.from({ length: 4 * (ROUTE_MAX_HOPS + 1) }, () => 1n); + const kept = await decode(await start(1n, far, 0)); + expect(kept).toMatchObject({ status: "refused", reason: "hops" }); + if (kept.status !== "refused" || kept.message.kind !== "routeStart") throw new Error("a refused ROUTE_START"); + expect(kept.message.path).toHaveLength(ROUTE_MAX_HOPS + 1); + // An empty path is the sender's own error: it names itself last. + await expect(decode(await start(1n, [], 0))).rejects.toThrow(ProtocolViolation); + + // A DEAD refuses each moved offer on its own, and still reads the next. + const dead = await forged(15, async (w) => { + await w.u62(0n); await w.u62(1n); - await w.u62(ROUTE_MAX); await w.u62(0n); + await w.u62(2n); await w.u8(0); - }); - expect(await decode(maxed)).toEqual({ - status: "refused", - message: { kind: "routeStart", node: 1n }, - reason: "seqno", - }); - // A flag no version defines is refused, and the RTT flag without its field is malformed. - const flagged = (flags: number) => - forged(async (w) => { + await w.u8(0); + await w.u62(2n); + for (const flags of [8, 0]) { + await w.u62(flags === 8 ? 3n : 4n); + await w.u53(6); + await w.u62(1n); await w.u62(1n); - await w.u62(0n); await w.u62(0n); await w.u8(flags); - }); - expect(await decode(await flagged(4))).toMatchObject({ status: "refused", reason: "flags" }); - await expect(decode(await flagged(2))).rejects.toThrow(); - // More hops than a route may carry, refused before reading them. - const overHops = await forged(async (w) => { - await w.u62(1n); - await w.u62(0n); - await w.u62(0n); - await w.u8(0); - await w.u53(ROUTE_MAX_HOPS + 1); + await w.u8(128); + await w.u62(0n); + } + }); + expect(await decode(dead)).toEqual({ + status: "dead", + dead: { + kind: "closed", + hop: { node: 1n, session: 0n }, + far: 2n, + gone: false, + more: false, + moved: [ + { id: 3n, offer: { refused: "flags", path } }, + { + id: 4n, + offer: { path, draining: false, priority: 0, metric: 0n, downOnly: false, rtt: undefined }, + }, + ], + }, }); - expect(await decode(overHops)).toMatchObject({ status: "refused", reason: "hops" }); -}); - -test("the subscriber's replies round-trip on draft-07", async () => { - const v = Version.DRAFT_07; - for (const [msg, wire] of [ - [{ kind: "request", node: 7n, seqno: 2n }, [0xa, 2, 7, 2]], - [{ kind: "echo", id: 3n }, [8, 1, 3]], - ] as [AnnounceReply, number[]][]) { - const encoded = await bytes((w) => encodeAnnounceReply(w, msg, v), v); - expect([...encoded]).toEqual(wire); - expect(await decodeAnnounceReplyMaybe(new Reader(undefined, encoded, v), v)).toEqual(msg); - } - // Anything else is a violation. - const other = await bytes(async (w) => { - await w.u53(0); - await w.u53(0); - }, v); - await expect(decodeAnnounceReplyMaybe(new Reader(undefined, other, v), v)).rejects.toThrow(ProtocolViolation); - // A finished stream is no reply. - expect(await decodeAnnounceReplyMaybe(new Reader(undefined, new Uint8Array(), v), v)).toBeUndefined(); - await expect( - bytes((w) => encodeAnnounceReply(w, { kind: "echo", id: 1n }, Version.DRAFT_06), Version.DRAFT_06), - ).rejects.toThrow(); }); test("AnnounceHistory rejects a base that is not live", () => { diff --git a/js/net/src/lite/announce.ts b/js/net/src/lite/announce.ts index 807ab5f18bb..6decfbed045 100644 --- a/js/net/src/lite/announce.ts +++ b/js/net/src/lite/announce.ts @@ -6,6 +6,7 @@ import type { Reader, Writer } from "../stream.ts"; import { SEGMENT_BYTES } from "../util/limits.ts"; import { decodeForwarded, decodeRoute, encodeRoute, utf8 } from "./instance.ts"; import * as Message from "./message.ts"; +import { fromWire, toWire } from "./priority.ts"; import { hasAnnounceCompression, hasAnnounceId, @@ -34,18 +35,20 @@ const ANNOUNCE_UPDATE = 2; // lite-07: another publisher instance replaces the advertisement. const ANNOUNCE_RESTART = 3; const SYNCED = 4; -// lite-07: the route layer. ROUTE_END also travels back as the echo. +// lite-07: the route layer. const ROUTE_START = 6; const ROUTE_UPDATE = 7; const ROUTE_END = 8; const ANNOUNCE_FLOOR = 9; -const ROUTE_REQUEST = 10; const ROUTE_HOLD = 11; +const DEAD = 15; -/** - * The largest lite-07 varint. No origin advertises it as a Seqno or Revision: it leaves room for - * the successor a ROUTE_REQUEST names and for the floor's offset. - */ +// DEAD kinds. +const DEAD_CLOSED = 0n; +const DEAD_DRAIN = 1n; + +/** The largest lite-07 varint. No origin advertises it as a Revision: it leaves room for the + * floor's offset. */ export const ROUTE_MAX = 2n ** 64n - 1n; /** Lite-07: copy the first `keep` path segments of the live announcement `distance` back from @@ -53,23 +56,57 @@ export const ROUTE_MAX = 2n ** 64n - 1n; * {@link AnnounceHistory}. */ export type Base = { distance: bigint; keep: number }; -/** How a lite-07 publisher reaches an origin node: the origin's seqno, the publisher's metric to it, - * whether it may only travel away from an upstream, its informational RTT to it in milliseconds, - * `undefined` when unmeasured, and the Hop ID of every node it crossed, the origin first. */ -export interface Advert { - seqno: bigint; +/** One node a lite-07 route crossed: its Node ID, and the session it learned the route on, an + * opaque per-node counter that is 0 for the publisher's own hop. */ +export interface RouteHop { + node: bigint; + session: bigint; +} + +/** How a lite-07 publisher reaches a publisher node: the `path` it crossed, the publisher first and + * the sender last, whether a node on it sent GOAWAY, the lowest link priority on it, the sender's + * metric to the publisher, whether it may only travel away from an upstream, and its informational + * RTT in milliseconds, `undefined` when unmeasured. Mirrors `route::path::Offer` in rs/moq-net. */ +export interface Offer { + path: RouteHop[]; + draining: boolean; + priority: number; metric: bigint; downOnly: boolean; rtt: bigint | undefined; - hops: bigint[]; } +/** A Route Offer this node refuses: the path it still names, which a later DEAD may match, and why. */ +export interface Unusable { + refused: Refusal; + path: RouteHop[]; +} + +/** News on DEAD about the routes the sender offered. `closed`: the session `hop` names closed, at + * the node `far` at its other end. Each route on the stream through that hop that `moved` names by + * Route ID continues on its offer there; the rest end if `gone`, and are held if not, unless the + * receiver is the node just before the sender on the route's path. With `more`, the next message is + * a DEAD naming the same hop, far node, and `gone` whose `moved` continues this one's, and the + * receiver applies them as one. `drain`: `node` sent GOAWAY, so every path through it ranks as + * draining. */ +export type Dead = + | { + kind: "closed"; + hop: RouteHop; + far: bigint; + gone: boolean; + more: boolean; + moved: { id: bigint; offer: Offer | Unusable }[]; + } + | { kind: "drain"; node: bigint }; + /** The most Hops a route may carry. One that arrives with this many is unusable. */ export const ROUTE_MAX_HOPS = 32; -// ROUTE_START and ROUTE_UPDATE flags. +// Route Offer flags. const DOWN_ONLY = 0x1; const RTT = 0x2; +const DRAINING = 0x4; /** * A message on the publisher's side of the Announce Stream. @@ -115,17 +152,18 @@ export type AnnounceBroadcast = | { status: "skipped" } /** Lite07+: the initial set has been delivered. */ | { status: "synced" } - /** Lite07+ ROUTE_START: the publisher reaches the origin node `node`, and assigns the route - * the next Route ID. */ - | { status: "routeStart"; node: bigint; advert: Advert } + /** Lite07+ ROUTE_START: the publisher reaches the publisher node `node` along `offer`, covering + * the requested prefix joined with `suffix`, and assigns the route the next Route ID. */ + | { status: "routeStart"; node: bigint; suffix: Path.Valid; offer: Offer } /** Lite07+ ROUTE_UPDATE: the route with this id changed, and is usable. */ - | { status: "routeUpdate"; id: bigint; advert: Advert } + | { status: "routeUpdate"; id: bigint; offer: Offer } /** Lite07+ ROUTE_HOLD: the publisher has no usable route with this id, but holds it and its * claims while it recovers. */ | { status: "routeHold"; id: bigint } - /** Lite07+ ROUTE_END: the publisher let go of the route with this id and every claim under it. - * The subscriber echoes it back. */ + /** Lite07+ ROUTE_END: the publisher let go of the route with this id and every claim under it. */ | { status: "routeEnd"; id: bigint } + /** Lite07+ DEAD: a session or node on the routes the publisher offered is gone or going. */ + | { status: "dead"; dead: Dead } /** Lite07+ ANNOUNCE_FLOOR: closes the batch of claims under the route with this id, certifying * the stream's claims for it across the requested prefix as the origin's at `revision`, or * nothing when `revision` is undefined. */ @@ -137,10 +175,11 @@ export type AnnounceBroadcast = /** What a refused route message still tells the receiver, so the ids that follow it resolve. */ export type Refusable = - /** ROUTE_START: assigns the next Route ID to the origin node `node`. */ - | { kind: "routeStart"; node: bigint } - /** ROUTE_UPDATE of the route with this Route ID. */ - | { kind: "routeUpdate"; id: bigint } + /** ROUTE_START: assigns the next Route ID to the publisher `node`, covering the requested prefix + * joined with `suffix`, or no prefix when the suffix itself was refused, along `path`. */ + | { kind: "routeStart"; node: bigint; suffix?: Path.Valid; path: RouteHop[] } + /** ROUTE_UPDATE of the route with this Route ID, now along `path`. */ + | { kind: "routeUpdate"; id: bigint; path: RouteHop[] } /** ANNOUNCE_START under Route ID `route`: assigns the next Announce ID, to `path`, or to no path * when the path itself was refused. */ | { kind: "start"; route: bigint; path?: { suffix: Path.Valid; pathBase?: Base } } @@ -148,16 +187,7 @@ export type Refusable = | { kind: "change"; id: bigint }; /** Why a node refuses content a peer forwarded: well framed, but nothing it can use. */ -export type Refusal = "node" | "seqno" | "flags" | "hops" | "path" | "instance"; - -/** - * A message the subscriber sends on a lite-07 Announce Stream after its ANNOUNCE_REQUEST. - * - * `request` (ROUTE_REQUEST) asks for a route to `node`, as the publisher named it, at least as new - * as `seqno`. `echo` is the ROUTE_END that retired Route ID `id`, sent back once the subscriber - * dropped the route. - */ -export type AnnounceReply = { kind: "request"; node: bigint; seqno: bigint } | { kind: "echo"; id: bigint }; +export type Refusal = "node" | "flags" | "hops" | "path" | "instance"; // Both wire rules on a hop chain, applied to what we send and to what we receive: a // chain that revisits a hop looped, so neither forwarding it nor subscribing through it @@ -277,44 +307,95 @@ async function decodeRouteCost(r: Reader, version: Version): Promise= ROUTE_MAX) throw new Error("a route never advertises the last seqno"); - if (advert.hops.length > ROUTE_MAX_HOPS) throw new Error("a route crossed too many nodes"); - await w.u62(advert.seqno); - await w.u62(advert.metric); - await w.u8((advert.downOnly ? DOWN_ONLY : 0) | (advert.rtt === undefined ? 0 : RTT)); - if (advert.rtt !== undefined) await w.u62(advert.rtt); - await w.u62(BigInt(advert.hops.length)); - for (const hop of advert.hops) await w.u62(hop); +/** Encode a Route Offer: the path, then flags, priority, metric, and RTT. No node offers an empty + * path or one past {@link ROUTE_MAX_HOPS}, nor names Node ID 0 in one. */ +async function encodeOffer(w: Writer, offer: Offer) { + if (offer.path.length === 0 || offer.path.length > ROUTE_MAX_HOPS || offer.path.some((hop) => hop.node === 0n)) + throw new Error("no node offers that path"); + await w.u62(BigInt(offer.path.length)); + for (const hop of offer.path) { + await w.u62(hop.node); + await w.u62(hop.session); + } + await w.u8( + (offer.downOnly ? DOWN_ONLY : 0) | (offer.rtt === undefined ? 0 : RTT) | (offer.draining ? DRAINING : 0), + ); + await w.u8(toWire(offer.priority)); + await w.u62(offer.metric); + if (offer.rtt !== undefined) await w.u62(offer.rtt); } -/** Decode an advertisement, refusing one whose seqno no origin advertises, whose flags this node - * does not know, or that crossed too many hops. The rest of the body is skipped, since an unknown - * flag may announce a field ahead of the hops. Mirrors rs/moq-net. */ -async function decodeAdvert(r: Reader): Promise { - const refuse = async (reason: Refusal) => { +/** Decode a Route Offer. Its path comes first, so a node refusing the rest still learns the path a + * later DEAD may name. An empty path is the sender's own error, since it names itself last. Hops + * past the limit, a hop naming Node ID 0, and a flag this node does not know refuse the offer: a + * newer flag may add a field after the ones this node reads, so a refusal skips the rest of the + * body. Every hop a path counts is read, but of a path past the limit only those up to one past + * it are kept: a sender within the limit never sends one, and the whole would hold a body's + * worth for each route. Mirrors rs/moq-net. */ +async function decodeOffer(r: Reader): Promise { + const count = await r.u62(); + if (count === 0n) throw new ProtocolViolation("a route offer with no hops"); + const path: RouteHop[] = []; + for (let i = 0n; i < count; i++) { + const hop = { node: await r.u62(), session: await r.u62() }; + if (path.length <= ROUTE_MAX_HOPS) path.push(hop); + } + const refuse = async (refused: Refusal): Promise => { await r.readAll(); - return reason; + return { refused, path }; }; - const seqno = await r.u62(); - if (seqno === ROUTE_MAX) return refuse("seqno"); - const metric = await r.u62(); + if (count > BigInt(ROUTE_MAX_HOPS)) return refuse("hops"); + if (path.some((hop) => hop.node === 0n)) return refuse("node"); const flags = await r.u8(); - if (flags & ~(DOWN_ONLY | RTT)) return refuse("flags"); + if (flags & ~(DOWN_ONLY | RTT | DRAINING)) return refuse("flags"); + const priority = fromWire(await r.u8()); + const metric = await r.u62(); const rtt = flags & RTT ? await r.u62() : undefined; - const count = await r.u62(); - if (count > BigInt(ROUTE_MAX_HOPS)) return refuse("hops"); - const hops: bigint[] = []; - for (let i = 0n; i < count; i++) hops.push(await r.u62()); - return { seqno, metric, downOnly: (flags & DOWN_ONLY) !== 0, rtt, hops }; + return { path, draining: (flags & DRAINING) !== 0, priority, metric, downOnly: (flags & DOWN_ONLY) !== 0, rtt }; } -/** Refuse a route to the reserved Node ID 0, skipping the rest of its body. */ -async function refuseNode(r: Reader): Promise { - await r.readAll(); - return "node"; +async function encodeDead(w: Writer, dead: Dead) { + if (dead.kind === "drain") { + if (dead.node === 0n) throw new Error("Node ID 0 names no node"); + await w.u62(DEAD_DRAIN); + await w.u62(dead.node); + return; + } + if (dead.hop.node === 0n || dead.far === 0n) throw new Error("Node ID 0 names no node"); + await w.u62(DEAD_CLOSED); + await w.u62(dead.hop.node); + await w.u62(dead.hop.session); + await w.u62(dead.far); + await w.u8(dead.gone ? 1 : 0); + await w.u8(dead.more ? 1 : 0); + await w.u62(BigInt(dead.moved.length)); + for (const { id, offer } of dead.moved) { + if ("refused" in offer) throw new Error("a refused offer is never sent"); + await w.u62(id); + // Sized, so a receiver refusing one offer still reads the next. + await Message.encode(w, (w) => encodeOffer(w, offer)); + } +} + +async function decodeDead(r: Reader): Promise { + const nonzero = (id: bigint) => { + if (id === 0n) throw new ProtocolViolation("Node ID 0 names no node"); + return id; + }; + const kind = await r.u62(); + if (kind === DEAD_DRAIN) return { kind: "drain", node: nonzero(await r.u62()) }; + if (kind !== DEAD_CLOSED) throw new ProtocolViolation(`unknown DEAD kind ${kind}`); + const hop = { node: nonzero(await r.u62()), session: await r.u62() }; + const far = nonzero(await r.u62()); + const gone = await r.u8(); + if (gone > 1) throw new ProtocolViolation(`DEAD Gone ${gone}`); + const more = await r.u8(); + if (more > 1) throw new ProtocolViolation(`DEAD More ${more}`); + const count = await r.u62(); + const moved: { id: bigint; offer: Offer | Unusable }[] = []; + // Each entry takes bytes, so the body bounds the count before this loop runs out. + for (let i = 0n; i < count; i++) moved.push({ id: await r.u62(), offer: await Message.decode(r, decodeOffer) }); + return { kind: "closed", hop, far, gone: gone === 1, more: more === 1, moved }; } /** Decode a lite-07 suffix: its base, which the sender picked, then the path it forwarded, or @@ -386,11 +467,15 @@ async function encodeAnnounce06Body(w: Writer, msg: AnnounceBroadcast, version: case "routeStart": if (msg.node === 0n) throw new Error("Node ID 0 names no origin"); await w.u62(msg.node); - await encodeAdvert(w, msg.advert); + await w.string(Path.encode(msg.suffix)); + await encodeOffer(w, msg.offer); break; case "routeUpdate": await w.u62(msg.id); - await encodeAdvert(w, msg.advert); + await encodeOffer(w, msg.offer); + break; + case "dead": + await encodeDead(w, msg.dead); break; case "routeHold": case "routeEnd": @@ -433,6 +518,7 @@ function announce06Type(msg: AnnounceBroadcast, version: Version): number { case "routeUpdate": case "routeHold": case "routeEnd": + case "dead": case "floor": if (!hasRoutes(version)) throw new Error("routes not supported for this version"); return { @@ -440,6 +526,7 @@ function announce06Type(msg: AnnounceBroadcast, version: Version): number { routeUpdate: ROUTE_UPDATE, routeHold: ROUTE_HOLD, routeEnd: ROUTE_END, + dead: DEAD, floor: ANNOUNCE_FLOOR, }[msg.status]; case "ended": @@ -482,19 +569,30 @@ async function decodeAnnounce06Body(r: Reader, typ: number, version: Version): P case ROUTE_START: { if (!hasRoutes(version)) break; const node = await r.u62(); - const advert = node === 0n ? await refuseNode(r) : await decodeAdvert(r); - if (typeof advert === "string") - return { status: "refused", message: { kind: "routeStart", node }, reason: advert }; - return { status: "routeStart", node, advert }; + // The suffix is sized, so the offer's path reads even when it is refused. + const text = utf8(await r.read(await r.u53())); + const suffix = text === undefined ? undefined : Path.tryDecode(text); + const offer = await decodeOffer(r); + if (node !== 0n && suffix !== undefined && !("refused" in offer)) + return { status: "routeStart", node, suffix, offer }; + const reason = node === 0n ? "node" : suffix === undefined ? "path" : (offer as Unusable).refused; + return { status: "refused", message: { kind: "routeStart", node, suffix, path: offer.path }, reason }; } case ROUTE_UPDATE: { if (!hasRoutes(version)) break; const id = await r.u62(); - const advert = await decodeAdvert(r); - if (typeof advert === "string") - return { status: "refused", message: { kind: "routeUpdate", id }, reason: advert }; - return { status: "routeUpdate", id, advert }; + const offer = await decodeOffer(r); + if ("refused" in offer) + return { + status: "refused", + message: { kind: "routeUpdate", id, path: offer.path }, + reason: offer.refused, + }; + return { status: "routeUpdate", id, offer }; } + case DEAD: + if (!hasRoutes(version)) break; + return { status: "dead", dead: await decodeDead(r) }; case ROUTE_HOLD: if (!hasRoutes(version)) break; return { status: "routeHold", id: await r.u62() }; @@ -579,41 +677,6 @@ export async function decodeAnnounceBroadcastMaybe( return Message.decodeMaybe(r, (r) => decodeLegacyBody(r, version)); } -/** Encode one subscriber reply on a lite-07 Announce Stream. */ -export async function encodeAnnounceReply(w: Writer, msg: AnnounceReply, version: Version): Promise { - if (!hasRoutes(version)) throw new Error("routes not supported for this version"); - if (msg.kind === "echo") return Message.encode(w, (w) => w.u62(msg.id), { id: ROUTE_END }); - if (msg.node === 0n) throw new Error("Node ID 0 names no origin"); - return Message.encode( - w, - async (w) => { - await w.u62(msg.node); - await w.u62(msg.seqno); - }, - { id: ROUTE_REQUEST }, - ); -} - -/** Decode one subscriber reply, or `undefined` on a clean FIN. Any other type is a violation. */ -export async function decodeAnnounceReplyMaybe(r: Reader, version: Version): Promise { - if (!hasRoutes(version)) throw new Error("routes not supported for this version"); - if (await r.done()) return undefined; - const typ = await r.u53(); - return Message.decode(r, async (r): Promise => { - switch (typ) { - case ROUTE_REQUEST: { - const node = await r.u62(); - if (node === 0n) throw new ProtocolViolation("Node ID 0 names no origin"); - return { kind: "request", node, seqno: await r.u62() }; - } - case ROUTE_END: - return { kind: "echo", id: await r.u62() }; - default: - throw new ProtocolViolation(`unexpected announce reply type ${typ}`); - } - }); -} - /** * The live announcements on one lite-06+ announce stream, by Announce ID: what ANNOUNCE_END, * ANNOUNCE_UPDATE, and ANNOUNCE_RESTART reference, and what a lite-07 base copies from. Every diff --git a/js/net/src/lite/connection.ts b/js/net/src/lite/connection.ts index 05b102f939b..3e3ed288aed 100644 --- a/js/net/src/lite/connection.ts +++ b/js/net/src/lite/connection.ts @@ -264,6 +264,7 @@ export class Connection implements Session { if (!hasSetupStream(version)) this.#peerImplementation.resolve(undefined); registerWire(this, { consume: (path, instance) => this.#subscriber.consume(path, instance), + covers: (node, at, path) => this.#subscriber.covers(node, at, path), exclusive: false, leave: () => this.#leave(), auth: this.#auth, diff --git a/js/net/src/lite/corpus.test.ts b/js/net/src/lite/corpus.test.ts index 70803e30476..2abb51b6abb 100644 --- a/js/net/src/lite/corpus.test.ts +++ b/js/net/src/lite/corpus.test.ts @@ -6,13 +6,7 @@ import { expect, test } from "bun:test"; import { select } from "../connection/accept.ts"; import { Reader, UnexpectedEnd, Writer } from "../stream.ts"; import { isDecodeError } from "../util/decode_error.ts"; -import { - AnnounceInit, - decodeAnnounceBroadcast, - decodeAnnounceReplyMaybe, - encodeAnnounceBroadcast, - encodeAnnounceReply, -} from "./announce.ts"; +import { AnnounceInit, decodeAnnounceBroadcast, encodeAnnounceBroadcast } from "./announce.ts"; import { ClientSetup } from "./compat.ts"; import { Group } from "./group.ts"; import { Version, versionName } from "./version.ts"; @@ -20,7 +14,7 @@ import { Version, versionName } from "./version.ts"; interface Case { name: string; version: string; - message: "announce" | "announce_reply" | "announce_init" | "group" | "client_setup"; + message: "announce" | "announce_init" | "group" | "client_setup"; hex: string; /** `announce_init`: the announce slots left, unbounded when absent. */ budget?: number; @@ -43,13 +37,15 @@ async function decode(c: Case, r: Reader, v: Version): Promise<((w: Writer) => P case "announce": { const msg = await decodeAnnounceBroadcast(r, v); if (msg.status === "refused") return "withdrawn"; + // A DEAD withdraws each route a refused moved offer names. + if ( + msg.status === "dead" && + msg.dead.kind === "closed" && + msg.dead.moved.some(({ offer }) => "refused" in offer) + ) + return "withdrawn"; return (w) => encodeAnnounceBroadcast(w, msg, v); } - case "announce_reply": { - const msg = await decodeAnnounceReplyMaybe(r, v); - if (!msg) throw new Error("no reply"); - return (w) => encodeAnnounceReply(w, msg, v); - } case "announce_init": { const msg = await AnnounceInit.decode(r, v, c.budget ?? Number.POSITIVE_INFINITY); return (w) => msg.encode(w, v); diff --git a/js/net/src/lite/datagram.test.ts b/js/net/src/lite/datagram.test.ts index bfaafe7d3c2..b0eaad6156d 100644 --- a/js/net/src/lite/datagram.test.ts +++ b/js/net/src/lite/datagram.test.ts @@ -95,8 +95,8 @@ test("lite-07 datagram requests match Rust's size-prefixed fields", async () => const { Timescale } = await import("../time.ts"); const request = new Datagrams({ id: 7n, broadcast: Path.from("room/alice"), track: "audio", priority: -3 }); const body = await Writer.collect(Version.DRAFT_07, (writer) => request.encode(writer, Version.DRAFT_07)); - // Id, broadcast, an empty unminted pin, Node ID 0, track, priority. - expect([...body]).toEqual([22, 7, 10, ...enc.encode("room/alice"), 0, 0, 0, 5, ...enc.encode("audio"), 125]); + // Id, broadcast, an empty unminted pin, Node ID 0, Route Depth 0, track, priority. + expect([...body]).toEqual([23, 7, 10, ...enc.encode("room/alice"), 0, 0, 0, 0, 5, ...enc.encode("audio"), 125]); const decoded = await Datagrams.decode(new Reader(undefined, body, Version.DRAFT_07), Version.DRAFT_07); expect(decoded).toEqual(request); for (const timescale of [undefined, Timescale.MILLI]) { diff --git a/js/net/src/lite/datagram.ts b/js/net/src/lite/datagram.ts index d14757ebd88..4e7baf6366e 100644 --- a/js/net/src/lite/datagram.ts +++ b/js/net/src/lite/datagram.ts @@ -113,8 +113,10 @@ export class Datagrams { broadcast: Path.Valid; /** The publisher instance the subscriber expects. */ instance?: Instance.Valid; - /** The origin node that serves the request, 0 for whichever the publisher selects. Lite-07+. */ + /** The publisher node that serves the request, 0 for whichever the publisher selects. Lite-07+. */ node: bigint; + /** How many leading segments of `broadcast` make up the prefix of the route it follows. Lite-07+. */ + depth: number; /** The datagram track's name. */ track: string; /** The subscriber priority, higher first. */ @@ -125,11 +127,13 @@ export class Datagrams { broadcast: Path.Valid; instance?: Instance.Valid; node?: bigint; + depth?: number; track: string; priority: number; }) { this.instance = properties.instance; this.node = properties.node ?? 0n; + this.depth = properties.depth ?? 0; this.id = properties.id; this.broadcast = properties.broadcast; this.track = properties.track; @@ -143,7 +147,7 @@ export class Datagrams { await w.u62(this.id); await w.string(Path.encode(this.broadcast)); await encodePin(w, version, this.instance); - await encodeNode(w, version, this.node); + await encodeNode(w, version, this.node, this.depth); await w.string(this.track); await w.u8(toWire(this.priority)); }); @@ -152,18 +156,21 @@ export class Datagrams { /** Decode the size-prefixed request. */ static async decode(reader: Reader, version: Version): Promise { guard(version); - return Message.decode( - reader, - async (r) => - new Datagrams({ - id: await r.u62(), - broadcast: Path.decode(await r.string()), - instance: await decodePin(r, version), - node: await decodeNode(r, version), - track: await r.string(), - priority: fromWire(await r.u8()), - }), - ); + return Message.decode(reader, async (r) => { + const id = await r.u62(); + const broadcast = Path.decode(await r.string()); + const instance = await decodePin(r, version); + const { node, depth } = await decodeNode(r, version, broadcast); + return new Datagrams({ + id, + broadcast, + instance, + node, + depth, + track: await r.string(), + priority: fromWire(await r.u8()), + }); + }); } } diff --git a/js/net/src/lite/fetch.test.ts b/js/net/src/lite/fetch.test.ts index 9a6dbc8516d..63d2655facd 100644 --- a/js/net/src/lite/fetch.test.ts +++ b/js/net/src/lite/fetch.test.ts @@ -83,6 +83,7 @@ test("Fetch decodes the optional delay offset before converting to milliseconds" await w.u53(0); // no instance pin: empty and unminted await w.bool(false); await w.u62(0n); // no origin node + await w.u53(0); // nor route depth await w.string("video"); await w.u8(0); await w.u62(delay + 1n); diff --git a/js/net/src/lite/fetch.ts b/js/net/src/lite/fetch.ts index 92f71dfa8c1..b4f9ee49012 100644 --- a/js/net/src/lite/fetch.ts +++ b/js/net/src/lite/fetch.ts @@ -21,8 +21,10 @@ export class Fetch { broadcast: Path.Valid; /** The publisher instance the subscriber expects. Lite-07+. */ instance?: Instance.Valid; - /** The origin node that serves the request, 0 for whichever the publisher selects. Lite-07+. */ + /** The publisher node that serves the request, 0 for whichever the publisher selects. Lite-07+. */ node: bigint; + /** How many leading segments of `broadcast` make up the prefix of the route it follows. Lite-07+. */ + depth: number; track: string; priority: number; /** Reader freshness budget in whole milliseconds. Lite-07+ only; omission leaves the publisher's max age as the only bound. */ @@ -48,6 +50,7 @@ export class Fetch { broadcast, instance, node = 0n, + depth = 0, track, priority, maxDelay, @@ -59,6 +62,7 @@ export class Fetch { broadcast: Path.Valid; instance?: Instance.Valid; node?: bigint; + depth?: number; track: string; priority: number; maxDelay?: number; @@ -70,6 +74,7 @@ export class Fetch { this.broadcast = broadcast; this.instance = instance; this.node = node; + this.depth = depth; this.track = track; this.priority = priority; this.maxDelay = maxDelay; @@ -82,7 +87,7 @@ export class Fetch { async #encode(w: Writer, version: Version) { await w.string(Path.encode(this.broadcast)); await encodePin(w, version, this.instance); - await encodeNode(w, version, this.node); + await encodeNode(w, version, this.node, this.depth); await w.string(this.track); await w.u8(toWire(this.priority)); if (hasFetchMaxDelay(version)) await Duration.encodeOptional(w, this.maxDelay); @@ -105,14 +110,14 @@ export class Fetch { static async #decode(r: Reader, version: Version): Promise { const broadcast = Path.decode(await r.string()); const instance = await decodePin(r, version); - const node = await decodeNode(r, version); + const { node, depth } = await decodeNode(r, version, broadcast); const track = await r.string(); const priority = fromWire(await r.u8()); const maxDelay = hasFetchMaxDelay(version) ? await Duration.decodeOptional(r) : undefined; const group = await r.u62(); if (!hasFrameBounds(version)) { - return new Fetch({ broadcast, instance, node, track, priority, maxDelay, group }); + return new Fetch({ broadcast, instance, node, depth, track, priority, maxDelay, group }); } const startFrame = await r.u53(); @@ -126,6 +131,7 @@ export class Fetch { broadcast, instance, node, + depth, track, priority, maxDelay, diff --git a/js/net/src/lite/instance.ts b/js/net/src/lite/instance.ts index 1caeec38ab9..f64b03fdcd9 100644 --- a/js/net/src/lite/instance.ts +++ b/js/net/src/lite/instance.ts @@ -1,5 +1,7 @@ +import { ProtocolViolation } from "../error.ts"; import * as Instance from "../instance.ts"; import * as Mint from "../mint.ts"; +import * as Path from "../path.ts"; import type { Reader, Writer } from "../stream.ts"; import { hasInstance, hasRoutes, type Version } from "./version.ts"; @@ -27,15 +29,31 @@ export async function decodePin(r: Reader, version: Version): Promise { - if (hasRoutes(version)) await w.u62(node); +/** Encode the publisher node a request names, 0 letting the publisher select one, then how many + * leading segments of the request's path make up the prefix of the route it follows, 0 with no + * node. Only lite-07 carries them, so an older peer always selects, and a node set here is + * dropped. Mirrors rs/moq-net. */ +export async function encodeNode(w: Writer, version: Version, node: bigint, depth = 0): Promise { + if (!hasRoutes(version)) return; + // A route needs its publisher. + if (node === 0n && depth !== 0) throw new Error("a route depth without its node"); + await w.u62(node); + await w.u53(depth); } -/** Decode the origin node a request names; see {@link encodeNode}. */ -export async function decodeNode(r: Reader, version: Version): Promise { - return hasRoutes(version) ? await r.u62() : 0n; +/** Decode the publisher node a request for `broadcast` names, and its route's depth; see + * {@link encodeNode}. A depth past the path's segments names no prefix of it. */ +export async function decodeNode( + r: Reader, + version: Version, + broadcast: Path.Valid, +): Promise<{ node: bigint; depth: number }> { + if (!hasRoutes(version)) return { node: 0n, depth: 0 }; + const node = await r.u62(); + const depth = await r.u53(); + if ((node === 0n && depth !== 0) || depth > Path.parts(broadcast).length) + throw new ProtocolViolation(`route depth ${depth} names no prefix of ${broadcast}`); + return { node, depth }; } /** Encode the instance, its minted flag, and the epoch an announcement serves. Every route diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index 20ae1929221..ef86760488d 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -18,14 +18,7 @@ import { Reader, Stream, Writer, writeBudget } from "../stream.ts"; import { Timescale, Timestamp } from "../time.ts"; import { Subscriber as TrackSubscriber } from "../track.ts"; import { wireOf } from "../wire.ts"; -import { - type AnnounceBroadcast, - AnnounceOk, - type AnnounceReply, - AnnounceRequest, - decodeAnnounceBroadcast, - encodeAnnounceReply, -} from "./announce.ts"; +import { type AnnounceBroadcast, AnnounceOk, AnnounceRequest, decodeAnnounceBroadcast } from "./announce.ts"; import { Fetch } from "./fetch.ts"; import { Group as GroupMessage } from "./group.ts"; import { Probe as ProbeMessage } from "./probe.ts"; @@ -3990,10 +3983,11 @@ function routedAnswer(origin: OriginProducer, grant?: Signal, const running = publisher.runAnnounce(new AnnounceRequest(prefix), stream); return { node: wireOf(origin.consume()).own.node, + hop: wireOf(origin.consume()).own.hop, running, next: () => decodeAnnounceBroadcast(reader, version), - answer: async (msg: AnnounceReply) => - replies.enqueue(await Writer.collect(version, (w) => encodeAnnounceReply(w, msg, version))), + // The peer sends bytes after its request, which no message is. + send: (bytes: Uint8Array) => replies.enqueue(bytes), ok: () => AnnounceOk.decode(reader, version), // The peer finishes its replies, which ends the stream. finish: () => replies.close(), @@ -4008,15 +4002,19 @@ function routedAnswer(origin: OriginProducer, grant?: Signal, test("a lite-07 answer carries this origin's claims under its own route", async () => { const origin = new OriginProducer(); const cam = publish(origin, Path.from("cam")); - const { node, running, next, answer, ok, close } = routedAnswer(origin); + const { node, hop, running, next, ok, close } = routedAnswer(origin); + // The origin's own route covers every path, along its own hop alone, at the top priority. + const offer = { + path: [{ node: hop, session: 0n }], + draining: false, + priority: 127, + metric: 0n, + downOnly: false, + rtt: 0n, + }; try { await ok(); - // Answering a stream lengthens no path, so the seqno stays. - expect(await next()).toEqual({ - status: "routeStart", - node, - advert: { seqno: 0n, metric: 0n, downOnly: false, rtt: 0n, hops: [node] }, - }); + expect(await next()).toEqual({ status: "routeStart", node, suffix: Path.empty(), offer }); expect(await next()).toMatchObject({ status: "active", suffix: "cam", route: 0n }); expect(await next()).toEqual({ status: "floor", id: 0n, revision: 0n }); expect(await next()).toEqual({ status: "synced" }); @@ -4035,36 +4033,22 @@ test("a lite-07 answer carries this origin's claims under its own route", async expect(await next()).toEqual({ status: "floor", id: 0n, revision: 3n }); expect(await next()).toEqual({ status: "routeEnd", id: 0n }); - // The route starts again only once the end is echoed, under the next Route ID. + // The next claim starts the route again at once, under the next Route ID: no echo waits. const again = publish(origin, Path.from("again")); - await yieldNow(); - await answer({ kind: "echo", id: 0n }); - expect(await next()).toMatchObject({ status: "routeStart", node }); + expect(await next()).toEqual({ status: "routeStart", node, suffix: Path.empty(), offer }); expect(await next()).toMatchObject({ status: "active", suffix: "again", route: 1n }); expect(await next()).toMatchObject({ status: "floor", id: 1n }); - - // A ROUTE_REQUEST for this origin raises its seqno; one for another node changes nothing. - await answer({ kind: "request", node: node + 1n, seqno: 20n }); - await answer({ kind: "request", node, seqno: 10n }); - expect(await next()).toEqual({ - status: "routeUpdate", - id: 1n, - advert: { seqno: 10n, metric: 0n, downOnly: false, rtt: 0n, hops: [node] }, - }); - - // An echo of a route that is not ending is a violation. - await answer({ kind: "echo", id: 0n }); - await expect(running).rejects.toThrow(ProtocolViolation); again.close(); } finally { close(); origin.close(); + await running.catch(() => {}); } }); -// A browser forwards nothing, so every session its route went out on is a path to it: closing one -// advances the seqno. A session that never carried the route changes no path. -test("a lite-07 seqno advances only when a session carrying the route closes", async () => { +// The subscriber sends nothing after its request: its FIN ends the answer cleanly, and any byte, +// such as a lite-07-wip ROUTE_REQUEST from an older build, is a violation. +test("a lite-07 answer ends on the subscriber's FIN and closes on any reply", async () => { const origin = new OriginProducer(); const cam = publish(origin, Path.from("cam")); const first = routedAnswer(origin); @@ -4072,7 +4056,7 @@ test("a lite-07 seqno advances only when a session carrying the route closes", a try { for (const { ok, next } of [first, second]) { await ok(); - expect(await next()).toMatchObject({ status: "routeStart", advert: { seqno: 0n } }); + expect(await next()).toMatchObject({ status: "routeStart" }); expect(await next()).toMatchObject({ status: "active", suffix: "cam" }); expect(await next()).toMatchObject({ status: "floor" }); expect(await next()).toEqual({ status: "synced" }); @@ -4080,17 +4064,13 @@ test("a lite-07 seqno advances only when a session carrying the route closes", a first.finish(); await first.running; - expect(await second.next()).toMatchObject({ status: "routeUpdate", id: 0n, advert: { seqno: 1n } }); - - // The unmatched stream shows no claim, so no route goes out on it, and its close is no bump. - const unmatched = routedAnswer(origin, undefined, Path.from("other")); - await unmatched.ok(); - expect(await unmatched.next()).toEqual({ status: "synced" }); - unmatched.finish(); - await unmatched.running; + // The other stream goes on as it was: no session's close changes the origin's own route. const mic = publish(origin, Path.from("mic")); expect(await second.next()).toMatchObject({ status: "active", suffix: "mic" }); mic.close(); + + second.send(Uint8Array.of(0xa, 2, 7, 2)); + await expect(second.running).rejects.toThrow(ProtocolViolation); } finally { cam.close(); for (const answer of [first, second]) answer.close(); @@ -4098,14 +4078,13 @@ test("a lite-07 seqno advances only when a session carrying the route closes", a } }); -// A request for the last seqno rotates this origin's identity, which ends its route. A peer -// that keeps rotating it and never echoes holds one ended route here, not one per rotation. -test("a lite-07 route starts again only once every end is echoed", async () => { +// A rotated identity, which only an exhausted revision space forces, ends the route with every +// claim under it, and starts the new one at once. +test("a lite-07 rotated identity ends the route and starts the next", async () => { const origin = new OriginProducer(); const cam = publish(origin, Path.from("cam")); - const { node, running, next, answer, ok, close } = routedAnswer(origin); + const { node, running, next, ok, close } = routedAnswer(origin); const own = wireOf(origin.consume()).own; - const last = 2n ** 64n - 1n; try { await ok(); expect(await next()).toMatchObject({ status: "routeStart", node }); @@ -4113,15 +4092,9 @@ test("a lite-07 route starts again only once every end is echoed", async () => { expect(await next()).toMatchObject({ status: "floor", id: 0n }); expect(await next()).toEqual({ status: "synced" }); - await answer({ kind: "request", node, seqno: last }); - expect(await next()).toEqual({ status: "routeEnd", id: 0n }); - await yieldNow(); + own.rotate(); expect(own.node).not.toBe(node); - await answer({ kind: "request", node: own.node, seqno: last }); - await yieldNow(); - - // Nothing under the newer identities until the end is echoed, then only the latest. - await answer({ kind: "echo", id: 0n }); + expect(await next()).toEqual({ status: "routeEnd", id: 0n }); expect(await next()).toMatchObject({ status: "routeStart", node: own.node }); expect(await next()).toMatchObject({ status: "active", suffix: "cam", route: 1n }); expect(await next()).toMatchObject({ status: "floor", id: 1n }); diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 847219ad977..51783294295 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -51,14 +51,12 @@ import { Budget } from "../util/yield.ts"; import type { Resolved } from "../window.ts"; import { type Advertised, type Advertisements, sameInstance, wireOf } from "../wire.ts"; import { - type Advert, type AnnounceBroadcast, AnnounceInit, AnnounceOk, - type AnnounceReply, type AnnounceRequest, - decodeAnnounceReplyMaybe, encodeAnnounceBroadcast, + type Offer, } from "./announce.ts"; import { Datagram as DatagramMessage, type Datagrams, DatagramsInfo, DatagramsUpdate } from "./datagram.ts"; import { closeViolation } from "./error.ts"; @@ -828,18 +826,9 @@ export class Publisher { // The route while the peer holds it: from the first claim shown until the last one goes. // `certified` is whether its last floor certified, so a grant change alone re-floors it. - let route: { id: bigint; node: bigint; seqno: bigint; certified?: boolean } | undefined; + let route: { id: bigint; node: bigint; certified?: boolean } | undefined; let nextRoute = 0n; let nextAnnounce = 0n; - // Route IDs ended and not yet echoed, and the origin node each reached. No route starts - // until every end is echoed, so a peer that rotates our identity and withholds its echoes - // holds at most one here. - const ending = new Map(); - // Whether our route went out on this stream, making it a path to this origin: its close - // then advances the seqno, so a node remembering this shorter path accepts a longer one - // without a ROUTE_REQUEST. A new stream lengthens no path. Mirrors `links` in - // rs/moq-net's `route/node.rs` for a node that does not forward. - let link = false; // The claims the peer holds, by suffix. const live = new Map(); @@ -869,10 +858,9 @@ export class Publisher { // End the route on the wire: it retires every claim under it. const end = async () => { if (!route) return; - const { id, node } = route; + const { id } = route; route = undefined; live.clear(); - ending.set(id, node); await write({ status: "routeEnd", id }); }; @@ -889,8 +877,6 @@ export class Publisher { // A rotated identity ends the old route with every claim under it. if (route && route.node !== identity?.node) await end(); if (!identity) return; - // Shown once the peer echoes the last route's end. - if (ending.size > 0) return; let changed = false; for (const [suffix, { id }] of [...live]) { @@ -900,19 +886,15 @@ export class Publisher { console.debug(`announce: broadcast=${suffix} active=false`); await write({ status: "endedId", id }); } - if (route && route.seqno !== identity.seqno) { - route.seqno = identity.seqno; - await write({ status: "routeUpdate", id: route.id, advert: advert(identity.seqno) }); - } for (const [suffix, snap] of latest) { const prev = live.get(suffix); if (prev && !restarted(prev.snap, snap) && routesEqual(onWire(prev.snap.route), onWire(snap.route))) continue; changed = true; if (!route) { - route = { id: nextRoute++, node: identity.node, seqno: identity.seqno }; - link = true; - await write({ status: "routeStart", node: identity.node, advert: advert(identity.seqno) }); + route = { id: nextRoute++, node: identity.node }; + // The origin's own route covers every path, so the requested prefix too. + await write({ status: "routeStart", node: identity.node, suffix: Path.empty(), offer: offer() }); } const { instance, epoch, cost } = snap.route; if (!prev) { @@ -935,13 +917,15 @@ export class Publisher { if (live.size === 0) await end(); }; - // The origin's own route: it starts its Hops, alone, and forwards no other. - const advert = (seqno: bigint): Advert => ({ - seqno, + // The origin's own route: a path of its own hop alone, learned on no session, at the top + // priority and no cost. It forwards no other. Mirrors the origin's offer in rs/moq-net. + const offer = (): Offer => ({ + path: origin ? [{ node: origin.own.hop, session: 0n }] : [], + draining: false, + priority: 127, metric: 0n, downOnly: false, rtt: 0n, - hops: origin ? [origin.own.hop] : [], }); // What the peer decodes for a claim: lite-07 carries its instance, epoch, and cost. const onWire = (route: WireRoute): WireRoute => ({ ...route, auth: undefined }); @@ -957,25 +941,12 @@ export class Publisher { notify?.(); }; - // Replies arrive on their own task, each applied as it lands, so a peer that keeps sending - // while our writes stall queues nothing: an echo frees a route to start again, and a request - // raises the identity, whose change marks the stream dirty. + // The subscriber sends nothing after its ANNOUNCE_REQUEST: a clean FIN ends the stream, and + // anything else is a violation. Mirrors rs/moq-net. let departed: Error | null | undefined; - const reply = (msg: AnnounceReply) => { - if (msg.kind === "request") { - origin?.own.request(msg.node, msg.seqno); - return; - } - if (!ending.delete(msg.id)) throw new ProtocolViolation(`echo of route ${msg.id}, which is not ending`); - poke(); - }; void (async () => { try { - for (;;) { - const msg = await decodeAnnounceReplyMaybe(stream.reader, this.version); - if (!msg) break; - reply(msg); - } + if (!(await stream.reader.done())) throw new ProtocolViolation("a message after ANNOUNCE_REQUEST"); departed = null; } catch (err) { departed = error(err); @@ -1027,7 +998,6 @@ export class Publisher { } } finally { for (const stop of stops) stop?.(); - if (link) origin?.own.bump(); } } diff --git a/js/net/src/lite/subscribe.ts b/js/net/src/lite/subscribe.ts index 6c3bba03a76..6f14cfdb044 100644 --- a/js/net/src/lite/subscribe.ts +++ b/js/net/src/lite/subscribe.ts @@ -768,8 +768,10 @@ export class Subscribe { broadcast: Path.Valid; /** The publisher instance the subscriber expects. Lite-07+. */ instance?: Instance.Valid; - /** The origin node that serves the request, 0 for whichever the publisher selects. Lite-07+. */ + /** The publisher node that serves the request, 0 for whichever the publisher selects. Lite-07+. */ node: bigint; + /** How many leading segments of `broadcast` make up the prefix of the route it follows. Lite-07+. */ + depth: number; track: string; priority: number; /** @@ -819,6 +821,7 @@ export class Subscribe { broadcast: Path.Valid; instance?: Instance.Valid; node?: bigint; + depth?: number; track: string; priority: number; order?: Order; @@ -835,6 +838,7 @@ export class Subscribe { this.broadcast = props.broadcast; this.instance = props.instance; this.node = props.node ?? 0n; + this.depth = props.depth ?? 0; this.track = props.track; this.priority = props.priority; this.order = props.order ?? "descending"; @@ -852,7 +856,7 @@ export class Subscribe { await w.u62(this.id); await w.string(Path.encode(this.broadcast)); await encodePin(w, version, this.instance); - await encodeNode(w, version, this.node); + await encodeNode(w, version, this.node, this.depth); await w.string(this.track); await w.u8(toWire(this.priority)); @@ -875,7 +879,7 @@ export class Subscribe { const id = await r.u62(); const broadcast = Path.decode(await r.string()); const instance = await decodePin(r, version); - const node = await decodeNode(r, version); + const { node, depth } = await decodeNode(r, version, broadcast); const track = await r.string(); const priority = fromWire(await r.u8()); @@ -895,6 +899,7 @@ export class Subscribe { broadcast, instance, node, + depth, track, priority, order, diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index 632bdfb7e2e..5a8ed3a9954 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -39,8 +39,8 @@ import { AnnounceInit, AnnounceOk, AnnounceRequest, - decodeAnnounceReplyMaybe, encodeAnnounceBroadcast, + type Offer, ROUTE_MAX_HOPS, } from "./announce.ts"; import { Group } from "./group.ts"; @@ -191,7 +191,24 @@ const PEER = toHop(2n); // Lite-07 origin nodes, and the route a peer advertises to each. const NODE_A = 0x7an; const NODE_B = 0x8bn; -const ADVERT = { seqno: 1n, metric: 0n, downOnly: false, rtt: 20n, hops: [] as bigint[] }; +// The peer's hop on a route's path, which it learned on a session of its own per publisher node. +const PEER_NODE = 2n; + +/** The Route Offer a peer sends for its route to `node`: the publisher, then the peer, at 20ms. */ +function offer(node: bigint, extra: Partial = {}): Offer { + return { + path: [ + { node, session: 0n }, + { node: PEER_NODE, session: node }, + ], + draining: false, + priority: 0, + metric: 0n, + downOnly: false, + rtt: 20n, + ...extra, + }; +} /** * Write a claim the way `version` carries it: before lite-07 with its hop chain, and on lite-07 @@ -210,7 +227,11 @@ function claimer(version: Version) { return; } const route = routes++; - await encodeAnnounceBroadcast(w, { status: "routeStart", node, advert: ADVERT }, version); + await encodeAnnounceBroadcast( + w, + { status: "routeStart", node, suffix: Path.empty(), offer: offer(node) }, + version, + ); await encodeAnnounceBroadcast(w, { status: "active", suffix, route, instance }, version); await encodeAnnounceBroadcast(w, { status: "floor", id: route }, version); }; @@ -4339,8 +4360,8 @@ test("a lite-07 refused route or claim withdraws only itself", async () => { const [a, b, c, d] = ["live/a", "live/b", "live/c", "live/d"].map((path) => Path.from(path)); const instance = Instance.parse("i"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); - await send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await send({ status: "active", suffix: a, route: 0n, instance }); await send({ status: "active", suffix: b, route: 1n, instance }); await send({ status: "floor", id: 0n, revision: 0n }); @@ -4349,7 +4370,7 @@ test("a lite-07 refused route or claim withdraws only itself", async () => { expect(await announced.next()).toMatchObject({ prefix: b, kind: "start" }); // ROUTE_UPDATE of Route ID 0 with a flag no version defines, and the field it might announce. - raw([7, 5, 0, 1, 0, 0x80, 42]); + raw([7, 8, 0, 1, 5, 0, 0x80, 128, 0, 42]); expect(await announced.next()).toMatchObject({ prefix: a, kind: "offline" }); // Announce ID 2: an ANNOUNCE_START under Route ID 1 whose path has an empty segment. @@ -4360,11 +4381,11 @@ test("a lite-07 refused route or claim withdraws only itself", async () => { expect(await announced.next()).toMatchObject({ prefix: c, kind: "start" }); // Forwarded usably again, the route's claims come back. - await send({ status: "routeUpdate", id: 0n, advert: ADVERT }); + await send({ status: "routeUpdate", id: 0n, offer: offer(NODE_A) }); expect(await announced.next()).toMatchObject({ prefix: a, kind: "start" }); // ROUTE_START to the reserved node takes Route ID 2, and its claims never show. - raw([6, 5, 0, 1, 0, 0, 0]); + raw([6, 8, 0, 0, 1, 5, 0, 0, 128, 0]); await send({ status: "active", suffix: d, route: 2n, instance }); await send({ status: "floor", id: 2n, revision: 0n }); await send({ status: "endedId", id: 3n }); @@ -4383,7 +4404,7 @@ test("a lite-07 refused claim goes offline, its route certifying nothing meanwhi const [a, b, e] = ["live/a", "live/b", "live/e"].map((path) => Path.from(path)); const instance = Instance.parse("i"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: a, route: 0n, instance }); await send({ status: "active", suffix: b, route: 0n, instance }); await send({ status: "floor", id: 0n, revision: 0n }); @@ -4429,11 +4450,11 @@ test("a lite-07 refused claim another stream still certifies goes offline", asyn const x = Path.from("live/x"); const instance = Instance.parse("i"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: x, route: 0n, instance }); await send({ status: "floor", id: 0n, revision: 0n }); expect(await announced.next()).toMatchObject({ prefix: x, kind: "start" }); - await live.send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await live.send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await live.send({ status: "active", suffix: Path.from("x"), route: 0n, instance }); await live.send({ status: "floor", id: 0n, revision: 0n }); expect(await live.announced.next()).toMatchObject({ prefix: x, kind: "start" }); @@ -4456,16 +4477,16 @@ test("a lite-07 refused claim a lagging stream never had goes offline", async () const x = Path.from("live/x"); const instance = Instance.parse("i"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: x, route: 0n, instance }); await send({ status: "floor", id: 0n, revision: 7n }); expect(await announced.next()).toMatchObject({ prefix: x, kind: "start" }); // The second stream lags at revision 5, before live/x existed. - await live.send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await live.send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await live.send({ status: "active", suffix: Path.from("y"), route: 0n, instance }); await live.send({ status: "floor", id: 0n, revision: 5n }); // A claim of another node on that stream shows once it read all of the above. - await live.send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await live.send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await live.send({ status: "active", suffix: Path.from("z"), route: 1n, instance }); await live.send({ status: "floor", id: 1n, revision: 0n }); expect(await live.announced.next()).toMatchObject({ prefix: Path.from("live/z"), kind: "start" }); @@ -4486,7 +4507,7 @@ test("a lite-07 claim at the root updates and ends", async () => { const { subscriber, announced, send } = await routedStream(); const root = Path.empty(); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: root, route: 0n, instance: Instance.parse("i") }); await send({ status: "floor", id: 0n, revision: 0n }); expect(await announced.next()).toMatchObject({ prefix: root, kind: "start" }); @@ -4517,16 +4538,16 @@ test("a lite-07 refused route's floors hide no other stream's claims", async () // One varint byte, so the raw message below spells it. const node = 7n; - await send({ status: "routeStart", node, advert: ADVERT }); + await send({ status: "routeStart", node, suffix: Path.empty(), offer: offer(node) }); await send({ status: "active", suffix: Path.from("live/x"), route: 0n, instance }); await send({ status: "floor", id: 0n, revision: 0n }); expect(await announced.next()).toMatchObject({ prefix: Path.from("live/x"), kind: "start" }); // ROUTE_START to the node with a flag no version defines, then a newer floor without live/x. - live.raw([6, 5, 7, 1, 0, 0x80, 42]); + live.raw([6, 9, 7, 0, 1, 7, 0, 0x80, 128, 0, 42]); await live.send({ status: "floor", id: 0n, revision: 5n }); // A claim of another node on that stream shows once it read all of the above. - await live.send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await live.send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await live.send({ status: "active", suffix: Path.from("z"), route: 1n, instance }); await live.send({ status: "floor", id: 1n, revision: 0n }); expect(await live.announced.next()).toMatchObject({ prefix: Path.from("live/z"), kind: "start" }); @@ -4551,17 +4572,17 @@ test("a lite-07 route refused once live leaves its last floor as it was", async const x = Path.from("live/x"); // Both streams carry live/x, the first at revision 5 and the second lagging at 3. - await send({ status: "routeStart", node, advert: ADVERT }); + await send({ status: "routeStart", node, suffix: Path.empty(), offer: offer(node) }); await send({ status: "active", suffix: x, route: 0n, instance }); await send({ status: "floor", id: 0n, revision: 5n }); expect(await announced.next()).toMatchObject({ prefix: x, kind: "start" }); - await live.send({ status: "routeStart", node, advert: ADVERT }); + await live.send({ status: "routeStart", node, suffix: Path.empty(), offer: offer(node) }); await live.send({ status: "active", suffix: Path.from("x"), route: 0n, instance }); await live.send({ status: "floor", id: 0n, revision: 3n }); expect(await live.announced.next()).toMatchObject({ prefix: x, kind: "start" }); // The first stream's route is refused, then ends live/x at a floor no view takes. - raw([7, 5, 0, 1, 0, 0x80, 42]); + raw([7, 8, 0, 1, 5, 0, 0x80, 128, 0, 42]); expect(await announced.next()).toMatchObject({ prefix: x, kind: "offline" }); await send({ status: "endedId", id: 0n }); await send({ status: "floor", id: 0n, revision: 6n }); @@ -4569,7 +4590,7 @@ test("a lite-07 route refused once live leaves its last floor as it was", async // The second stream's next floor still has live/x, as revision 5 did: what it shows next is // another node's claim, not live/x withdrawn. await live.send({ status: "floor", id: 0n, revision: 4n }); - await live.send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await live.send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await live.send({ status: "active", suffix: Path.from("z"), route: 1n, instance }); await live.send({ status: "floor", id: 1n, revision: 0n }); expect(await live.announced.next()).toMatchObject({ prefix: Path.from("live/z"), kind: "start" }); @@ -4591,18 +4612,18 @@ test("a lite-07 claim another stream's newer floor withdrew is not selected", as const y = Instance.parse("y"); // NODE_A claims live/x and live/y, cheaply, on both streams at revision 0. - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: x, route: 0n, instance: a }); await send({ status: "active", suffix: Path.from("live/y"), route: 0n, instance: y }); await send({ status: "floor", id: 0n, revision: 0n }); - await live.send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await live.send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await live.send({ status: "active", suffix: Path.from("x"), route: 0n, instance: a }); await live.send({ status: "active", suffix: Path.from("y"), route: 0n, instance: y }); await live.send({ status: "floor", id: 0n, revision: 0n }); expect(await live.announced.next()).toMatchObject({ prefix: x, kind: "start", route: { node: NODE_A } }); expect(await live.announced.next()).toMatchObject({ prefix: Path.from("live/y"), kind: "start" }); // A dearer NODE_B claims live/x on the first stream only. - await send({ status: "routeStart", node: NODE_B, advert: { ...ADVERT, metric: 5n } }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B, { metric: 5n }) }); await send({ status: "active", suffix: x, route: 1n, instance: b }); await send({ status: "floor", id: 1n, revision: 0n }); @@ -4619,10 +4640,15 @@ test("a lite-07 claim another stream's newer floor withdrew is not selected", as // still passes over the cheaper copy. const exact = await another(3, Path.Pattern.subtree(x)); await send({ status: "routeEnd", id: 0n }); - await exact.send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await exact.send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await exact.send({ status: "active", suffix: Path.empty(), route: 0n, instance: a }); await exact.send({ status: "floor", id: 0n, revision: 0n }); - await exact.send({ status: "routeStart", node: NODE_B, advert: { ...ADVERT, metric: 5n } }); + await exact.send({ + status: "routeStart", + node: NODE_B, + suffix: Path.empty(), + offer: offer(NODE_B, { metric: 5n }), + }); await exact.send({ status: "active", suffix: Path.empty(), route: 1n, instance: b }); await exact.send({ status: "floor", id: 1n, revision: 0n }); expect(await exact.announced.next()).toMatchObject({ prefix: x, kind: "start", route: { node: NODE_B } }); @@ -4655,15 +4681,15 @@ test("a lite-07 floor that certifies nothing keeps the withdrawal its route cert const b = Instance.parse("b"); // NODE_A claims live/x cheaply on both streams at revision 0; a dearer NODE_B on the first. - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: x, route: 0n, instance: a }); await send({ status: "floor", id: 0n, revision: 0n }); expect(await announced.next()).toMatchObject({ prefix: x, kind: "start", route: { node: NODE_A } }); - await live.send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await live.send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await live.send({ status: "active", suffix: Path.from("x"), route: 0n, instance: a }); await live.send({ status: "floor", id: 0n, revision: 0n }); expect(await live.announced.next()).toMatchObject({ prefix: x, kind: "start", route: { node: NODE_A } }); - await send({ status: "routeStart", node: NODE_B, advert: { ...ADVERT, metric: 5n } }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B, { metric: 5n }) }); await send({ status: "active", suffix: x, route: 1n, instance: b }); await send({ status: "floor", id: 1n, revision: 0n }); @@ -4699,12 +4725,12 @@ test("a lite-07 uncertified claim shows once its last certifier ends", async () const a = Instance.parse("a"); // The first stream certifies NODE_A's claims at revision 1, without live/x. - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: y, route: 0n, instance: a }); await send({ status: "floor", id: 0n, revision: 1n }); expect(await announced.next()).toMatchObject({ prefix: y, kind: "start" }); // The second carries live/x under a floor that certifies nothing: held back. - await live.send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await live.send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await live.send({ status: "active", suffix: Path.from("x"), route: 0n, instance: a }); await live.send({ status: "active", suffix: Path.from("y"), route: 0n, instance: a }); await live.send({ status: "floor", id: 0n }); @@ -4727,7 +4753,7 @@ test("a lite-07 claim a pruned floor held back shows without a batch", async () const w = Path.from("live/w"); const a = Instance.parse("a"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: y, route: 0n, instance: a }); await send({ status: "floor", id: 0n, revision: 1n }); expect(await announced.next()).toMatchObject({ prefix: y, kind: "start" }); @@ -4737,7 +4763,7 @@ test("a lite-07 claim a pruned floor held back shows without a batch", async () // A floor that certifies nothing leaves no floor to decide live/x, so it shows. await send({ status: "floor", id: 0n }); // Another node's claim marks the floors as applied. - await send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await send({ status: "active", suffix: w, route: 1n, instance: a }); await send({ status: "floor", id: 1n, revision: 0n }); expect(await announced.next()).toMatchObject({ prefix: x, kind: "start", route: { node: NODE_A } }); @@ -4755,15 +4781,15 @@ test("a lite-07 consume skips a claim above a newer floor's region", async () => const a = Instance.parse("a"); // NODE_A claims live at revision 0; a dearer NODE_B claims it too. - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: root, route: 0n, instance: a }); await send({ status: "floor", id: 0n, revision: 0n }); expect(await announced.next()).toMatchObject({ prefix: root, kind: "start", route: { node: NODE_A } }); - await send({ status: "routeStart", node: NODE_B, advert: { ...ADVERT, metric: 5n } }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B, { metric: 5n }) }); await send({ status: "active", suffix: root, route: 1n, instance: a }); await send({ status: "floor", id: 1n, revision: 0n }); // NODE_A's revision 1 presents only live/x/m under live/x: it withdrew live. - await x.send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await x.send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await x.send({ status: "active", suffix: Path.from("m"), route: 0n, instance: a }); await x.send({ status: "floor", id: 0n, revision: 1n }); expect(await x.announced.next()).toMatchObject({ prefix: Path.from("live/x/m"), kind: "start" }); @@ -4787,7 +4813,7 @@ test("a lite-07 route refused once live gives a held consume back its broader cl const minted = Mint.fresh(); // NODE_A claims live under a floor that certifies nothing, and a consume of live/x/y takes it. - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: live, route: 0n, instance: minted }); await send({ status: "floor", id: 0n }); expect(await announced.next()).toMatchObject({ prefix: live, kind: "start" }); @@ -4796,14 +4822,14 @@ test("a lite-07 route refused once live gives a held consume back its broader cl expect(held.request).toMatchObject({ node: NODE_A }); // NODE_A's revision 1 presents only live/x/m under live/x, so live no longer covers live/x/y. - await x.send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await x.send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await x.send({ status: "active", suffix: Path.from("m"), route: 0n, instance: minted }); await x.send({ status: "floor", id: 0n, revision: 1n }); expect(await x.announced.next()).toMatchObject({ prefix: Path.from("live/x/m"), kind: "start" }); await held.info; // That route is refused: nothing certifies live/x/y any more, so live covers it again. - x.raw([7, 5, 0, 1, 0, 0x80, 42]); + x.raw([7, 8, 0, 1, 5, 0, 0x80, 128, 0, 42]); expect(await x.announced.next()).toMatchObject({ prefix: Path.from("live/x/m"), kind: "offline" }); const info = held.consumed .track("audio") @@ -4828,17 +4854,22 @@ test("a lite-07 tie keeps its winner across a route's unusable spell", async () const x = Path.from("live/x"); const a = Instance.parse("a"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: x, route: 0n, instance: a }); await send({ status: "floor", id: 0n, revision: 0n }); expect(await announced.next()).toMatchObject({ prefix: x, kind: "start", route: { node: NODE_A } }); - await send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await send({ status: "active", suffix: x, route: 1n, instance: a }); await send({ status: "floor", id: 1n, revision: 0n }); expect(await announced.next()).toMatchObject({ prefix: x, kind: "update", route: { node: NODE_B } }); // NODE_A's route crosses this session, then recovers. - await send({ status: "routeUpdate", id: 0n, advert: { ...ADVERT, hops: [subscriber.hop] } }); - await send({ status: "routeUpdate", id: 0n, advert: ADVERT }); + const crossing = [ + { node: NODE_A, session: 0n }, + { node: subscriber.hop, session: 1n }, + ...offer(NODE_A).path.slice(1), + ]; + await send({ status: "routeUpdate", id: 0n, offer: offer(NODE_A, { path: crossing }) }); + await send({ status: "routeUpdate", id: 0n, offer: offer(NODE_A) }); // Another claim marks the updates as applied. const w = Path.from("live/w"); await send({ status: "active", suffix: w, route: 1n, instance: a }); @@ -4862,12 +4893,12 @@ test("a lite-07 consume pinned to a minted instance asks for it", async () => { const a = Instance.parse("a"); const minted = Mint.fresh(); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: x, route: 0n, instance: a }); await send({ status: "floor", id: 0n, revision: 0n }); expect(await announced.next()).toMatchObject({ prefix: x, kind: "start", route: { node: NODE_A } }); // A dearer route carries the minted instance. - await send({ status: "routeStart", node: NODE_B, advert: { ...ADVERT, metric: 5n } }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B, { metric: 5n }) }); await send({ status: "active", suffix: x, route: 1n, instance: minted }); await send({ status: "floor", id: 1n, revision: 0n }); // Another claim marks the floor as applied. @@ -4894,15 +4925,15 @@ test("a lite-07 lagging copy carries the cost a newer floor certified", async () const a = Instance.parse("a"); const b = Instance.parse("b"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: x, route: 0n, instance: a }); await send({ status: "floor", id: 0n, revision: 0n }); expect(await announced.next()).toMatchObject({ prefix: x, kind: "start", route: { node: NODE_A } }); - await live.send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await live.send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await live.send({ status: "active", suffix: Path.from("x"), route: 0n, instance: a }); await live.send({ status: "floor", id: 0n, revision: 0n }); expect(await live.announced.next()).toMatchObject({ prefix: x, kind: "start", route: { cost: 1n } }); - await send({ status: "routeStart", node: NODE_B, advert: { ...ADVERT, metric: 5n } }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B, { metric: 5n }) }); await send({ status: "active", suffix: x, route: 1n, instance: b }); await send({ status: "floor", id: 1n, revision: 0n }); @@ -4928,11 +4959,11 @@ test("a lite-07 consume names the node now serving its instance", async () => { const a = Instance.parse("a"); // Two origin nodes reach one instance; the cheaper serves. - await send({ status: "routeStart", node: NODE_A, advert: { ...ADVERT, metric: 5n } }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A, { metric: 5n }) }); await send({ status: "active", suffix: room, route: 0n, instance: a }); await send({ status: "floor", id: 0n }); await announced.next(); - await send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await send({ status: "active", suffix: room, route: 1n, instance: a }); await send({ status: "floor", id: 1n }); await announced.next(); @@ -4952,17 +4983,48 @@ test("a lite-07 consume names the node now serving its instance", async () => { await Promise.all([first.info, second.info]); }); +// A route that ends unindexes itself before its floors reshow another route to the node, which +// refreshes every route at the paths it shows: the ended one must not be among them. +test("a lite-07 certified route ending shows another route's copy of its claim", async () => { + const { subscriber, announced, send } = await routedStream(); + const x = Path.from("live/x"); + const next = async () => { + const event = await announced.next(); + return `${event?.kind} ${event?.prefix} ${String(event?.route.instance)}`; + }; + + // A narrow route's floor certifies nothing, so it shows x at b while it carries it. + await send({ status: "routeStart", node: NODE_A, suffix: Path.from("live"), offer: offer(NODE_A) }); + await send({ status: "active", suffix: x, route: 0n, instance: Instance.parse("b") }); + await send({ status: "floor", id: 0n }); + expect(await next()).toBe("start live/x b"); + + // A wide route's floor certifies x at a, which then decides it. + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); + await send({ status: "active", suffix: x, route: 1n, instance: Instance.parse("a") }); + await send({ status: "floor", id: 1n, revision: 1n }); + let seen = await next(); + while (seen !== "start live/x a") seen = await next(); + + // With no live floor left, the narrow route's copy shows again. + await send({ status: "routeEnd", id: 1n }); + expect(await next()).toBe("restart live/x b"); + + announced.close(); + subscriber.close(); +}); + test("a lite-07 path ranks only an instance's newest epoch", async () => { const { subscriber, announced, send } = await routedStream(); const room = Path.from("room"); const a = Instance.parse("a"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: room, route: 0n, instance: a, epoch: 0n }); await send({ status: "floor", id: 0n }); expect(await announced.next()).toMatchObject({ prefix: room, route: { epoch: 0n } }); // Dearer, but newer: it takes over. - await send({ status: "routeStart", node: NODE_B, advert: { ...ADVERT, metric: 9n } }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B, { metric: 9n }) }); await send({ status: "active", suffix: room, route: 1n, instance: a, epoch: 1n }); await send({ status: "floor", id: 1n }); expect(await announced.next()).toMatchObject({ prefix: room, kind: "restart", route: { epoch: 1n, node: NODE_B } }); @@ -4981,7 +5043,7 @@ test("lite-07 children of a prefix two origins claim spread by node", async () = const { subscriber, announced, send, trackRequest } = await routedStream(); const pool = Path.from("pool"); for (const [route, node] of order.entries()) { - await send({ status: "routeStart", node, advert: ADVERT }); + await send({ status: "routeStart", node, suffix: Path.empty(), offer: offer(node) }); await send({ status: "active", suffix: pool, route: BigInt(route), instance: Instance.parse(`i${node}`) }); await send({ status: "floor", id: BigInt(route) }); await announced.next(); @@ -5009,11 +5071,11 @@ test("a lite-07 tie at a claim's path goes to the newest claim, a re-price keepi const b = Instance.parse("b"); const c = Instance.parse("c"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: room, route: 0n, instance: a, cost: 4n }); await send({ status: "floor", id: 0n }); expect(await announced.next()).toMatchObject({ prefix: room, kind: "start", route: { node: NODE_A } }); - await send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await send({ status: "active", suffix: room, route: 1n, instance: b }); await send({ status: "floor", id: 1n }); expect(await announced.next()).toMatchObject({ prefix: room, kind: "restart", route: { node: NODE_B } }); @@ -5050,11 +5112,11 @@ test("a lite-07 consume pinned to an instance names its route's node", async () const { subscriber, announced, send, trackRequest } = await routedStream(); const pool = Path.from("pool"); const a = Instance.parse("a"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: pool, route: 0n, instance: a }); await send({ status: "floor", id: 0n }); await announced.next(); - await send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await send({ status: "active", suffix: pool, route: 1n, instance: Instance.parse("b") }); await send({ status: "floor", id: 1n }); await announced.next(); @@ -5074,7 +5136,7 @@ test("a lite-07 route reports its rtt as the relay's sum plus this link's", asyn const stats = Promise.withResolvers<{ smoothedRtt: number }>(); const { subscriber, announced, send } = await routedStream(undefined, () => stats.promise); const x = Path.from("x"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: x, route: 0n, instance: Instance.parse("a") }); await send({ status: "floor", id: 0n, revision: 0n }); const first = announced.next(); @@ -5086,7 +5148,7 @@ test("a lite-07 route reports its rtt as the relay's sum plus this link's", asyn expect(next.route.rtt?.asMillis()).toBe(31); // The relay could not measure a link on the path: the sum is unknown. - await send({ status: "routeUpdate", id: 0n, advert: { ...ADVERT, seqno: 2n, rtt: undefined } }); + await send({ status: "routeUpdate", id: 0n, offer: offer(NODE_A, { rtt: undefined }) }); next = (await announced.next()) as { kind: string; route: WireRoute }; expect(next).toMatchObject({ kind: "update", route: { node: NODE_A } }); expect(next.route.rtt).toBeUndefined(); @@ -5099,11 +5161,11 @@ test("a lite-07 route reports its rtt as the relay's sum plus this link's", asyn test("a lite-07 stream reports each node's claim with claims", async () => { const { subscriber, announced, send } = await routedStream({ claims: true }); const pool = Path.from("pool"); - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: pool, route: 0n, instance: Instance.parse("a") }); await send({ status: "floor", id: 0n }); expect(await announced.next()).toMatchObject({ prefix: pool, kind: "start", route: { node: NODE_A } }); - await send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await send({ status: "active", suffix: pool, route: 1n, instance: Instance.parse("b") }); await send({ status: "floor", id: 1n }); expect(await announced.next()).toMatchObject({ prefix: pool, kind: "start", route: { node: NODE_B } }); @@ -5120,15 +5182,15 @@ test("a lite-07 stream reports each node's claim with claims", async () => { }); test("a lite-07 batch applies at its floor, and a held route turns unavailable", async () => { - const { subscriber, announced, send, replies, version } = await routedStream(); + const { subscriber, announced, send } = await routedStream(); const room = Path.from("room"); const other = Path.from("other"); const a = Instance.parse("a"); // Nothing under a route shows before its floor, even while another route's batch lands. - await send({ status: "routeStart", node: NODE_A, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); await send({ status: "active", suffix: room, route: 0n, instance: a, cost: 2n }); - await send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await send({ status: "active", suffix: other, route: 1n, instance: a }); await send({ status: "floor", id: 1n, revision: 0n }); expect(await announced.next()).toMatchObject({ prefix: other, kind: "start" }); @@ -5140,29 +5202,34 @@ test("a lite-07 batch applies at its floor, and a held route turns unavailable", // A held route keeps its claims, unavailable, until an update makes it usable again. await send({ status: "routeHold", id: 0n }); expect(await announced.next()).toMatchObject({ prefix: room, kind: "update", route: { available: false } }); - await send({ status: "routeUpdate", id: 0n, advert: { ...ADVERT, metric: 5n } }); + await send({ status: "routeUpdate", id: 0n, offer: offer(NODE_A, { metric: 5n }) }); expect(await announced.next()).toMatchObject({ prefix: room, kind: "update", route: { available: true, cost: 8n }, }); // Nothing ranks by RTT, so a change to it alone reports nothing. - await send({ status: "routeUpdate", id: 0n, advert: { ...ADVERT, metric: 5n, rtt: 90n } }); - // Nor by hops, until the route crossed as many nodes as one may, or crossed us: then it is no - // route at all, until it is usable again. - await send({ status: "routeUpdate", id: 0n, advert: { ...ADVERT, metric: 5n, hops: [NODE_A, 3n] } }); - const full = Array.from({ length: ROUTE_MAX_HOPS }, (_, i) => BigInt(i + 1)); - for (const hops of [full, [NODE_A, subscriber.hop]]) { - await send({ status: "routeUpdate", id: 0n, advert: { ...ADVERT, metric: 5n, hops } }); + await send({ status: "routeUpdate", id: 0n, offer: offer(NODE_A, { metric: 5n, rtt: 90n }) }); + // Nor by its path, until the route crossed as many nodes as one may, or crossed us: then it is + // no route at all, until it is usable again. + const longer = [{ node: NODE_A, session: 0n }, { node: 3n, session: 1n }, ...offer(NODE_A).path.slice(1)]; + await send({ status: "routeUpdate", id: 0n, offer: offer(NODE_A, { metric: 5n, path: longer }) }); + const full = Array.from({ length: ROUTE_MAX_HOPS }, (_, i) => ({ node: BigInt(i + 1), session: 0n })); + const crossing = [ + { node: NODE_A, session: 0n }, + { node: subscriber.hop, session: 1n }, + ...offer(NODE_A).path.slice(1), + ]; + for (const path of [full, crossing]) { + await send({ status: "routeUpdate", id: 0n, offer: offer(NODE_A, { metric: 5n, path }) }); expect(await announced.next()).toMatchObject({ prefix: room, kind: "offline" }); - await send({ status: "routeUpdate", id: 0n, advert: { ...ADVERT, metric: 5n } }); + await send({ status: "routeUpdate", id: 0n, offer: offer(NODE_A, { metric: 5n }) }); expect(await announced.next()).toMatchObject({ prefix: room, kind: "start", route: { cost: 8n } }); } - // A route the peer lets go takes its claims offline, and is echoed back. + // A route the peer lets go takes its claims offline. await send({ status: "routeEnd", id: 0n }); expect(await announced.next()).toMatchObject({ prefix: room, kind: "offline" }); - expect(await decodeAnnounceReplyMaybe(replies, version)).toEqual({ kind: "echo", id: 0n }); // Its Announce IDs went with it. await send({ status: "endedId", id: 0n }); await expect(announced.next()).rejects.toThrow(ProtocolViolation); @@ -5171,18 +5238,18 @@ test("a lite-07 batch applies at its floor, and a held route turns unavailable", }); test("a lite-07 path served by two routes fails over when the cheaper one ends", async () => { - const { subscriber, announced, send, replies, version } = await routedStream(); + const { subscriber, announced, send } = await routedStream(); const room = Path.from("room"); const a = Instance.parse("a"); const b = Instance.parse("b"); - await send({ status: "routeStart", node: NODE_A, advert: { ...ADVERT, metric: 5n } }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A, { metric: 5n }) }); await send({ status: "active", suffix: room, route: 0n, instance: a }); await send({ status: "floor", id: 0n }); expect(await announced.next()).toMatchObject({ prefix: room, kind: "start", route: { instance: a, node: NODE_A } }); // Another origin claims the path more cheaply: it takes over, as another instance. - await send({ status: "routeStart", node: NODE_B, advert: ADVERT }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B) }); await send({ status: "active", suffix: room, route: 1n, instance: b }); await send({ status: "floor", id: 1n }); expect(await announced.next()).toMatchObject({ @@ -5199,7 +5266,6 @@ test("a lite-07 path served by two routes fails over when the cheaper one ends", kind: "restart", route: { instance: a, node: NODE_A }, }); - expect(await decodeAnnounceReplyMaybe(replies, version)).toEqual({ kind: "echo", id: 1n }); expect(subscriber.consume(room).closed).not.toBe(consumed.closed); consumed.close(); @@ -5207,18 +5273,267 @@ test("a lite-07 path served by two routes fails over when the cheaper one ends", subscriber.close(); }); +// A DEAD names a closed session on the peer's paths: each route through it moves to the offer it +// names, ends when the far end is gone, and is held otherwise. Routes off that hop, and a hop no +// route crosses, change nothing. Mirrors rs/moq-net. +test("a lite-07 DEAD moves, ends, or holds the routes through its hop", async () => { + const { subscriber, announced, send } = await routedStream(); + const [a, b, c] = ["a", "b", "c"].map((path) => Path.from(path)); + const instance = Instance.parse("i"); + const NODE_C = 0x9cn; + for (const [id, node, path] of [ + [0n, NODE_A, a], + [1n, NODE_B, b], + [2n, NODE_C, c], + ] as const) { + await send({ status: "routeStart", node, suffix: Path.empty(), offer: offer(node) }); + await send({ status: "active", suffix: path, route: id, instance }); + await send({ status: "floor", id, revision: 0n }); + expect(await announced.next()).toMatchObject({ prefix: path, kind: "start", route: { cost: 1n } }); + } + // The peer's session to a publisher, as each route's path names it. + const closed = (node: bigint, gone: boolean, moved: { id: bigint; offer: Offer }[] = []): AnnounceBroadcast => ({ + status: "dead", + dead: { kind: "closed", hop: { node: PEER_NODE, session: node }, far: node, gone, more: false, moved }, + }); + + // A hop no route crosses changes nothing: the next thing shown is the move below. + await send({ + status: "dead", + dead: { kind: "closed", hop: { node: 0x55n, session: 4n }, far: 0x56n, gone: true, more: false, moved: [] }, + }); + // Route 0 moved to a costlier path through another of the peer's sessions. + const detour = [ + { node: NODE_A, session: 0n }, + { node: 3n, session: 1n }, + { node: PEER_NODE, session: 3n }, + ]; + await send(closed(NODE_A, false, [{ id: 0n, offer: offer(NODE_A, { metric: 4n, path: detour }) }])); + expect(await announced.next()).toMatchObject({ prefix: a, kind: "update", route: { cost: 5n, available: true } }); + // Its old hop names it no more: a DEAD of that hop leaves it alone. + await send(closed(NODE_A, true)); + // Route 1's far end is gone: it ends with its claims. + await send(closed(NODE_B, true)); + expect(await announced.next()).toMatchObject({ prefix: b, kind: "offline" }); + await send({ status: "endedId", id: 1n }); + await expect(announced.next()).rejects.toThrow(ProtocolViolation); + subscriber.close(); + + // Held, a route stays unavailable until its next offer. + const held = await routedStream(); + await held.send({ status: "routeStart", node: NODE_C, suffix: Path.empty(), offer: offer(NODE_C) }); + await held.send({ status: "active", suffix: c, route: 0n, instance }); + await held.send({ status: "floor", id: 0n, revision: 0n }); + expect(await held.announced.next()).toMatchObject({ prefix: c, kind: "start" }); + await held.send(closed(NODE_C, false)); + expect(await held.announced.next()).toMatchObject({ prefix: c, kind: "update", route: { available: false } }); + await held.send({ status: "routeUpdate", id: 0n, offer: offer(NODE_C) }); + expect(await held.announced.next()).toMatchObject({ prefix: c, kind: "update", route: { available: true } }); + // Moved to an offer this node refuses, it goes offline, and a later usable offer brings it back. + // DEAD: the session NODE_C (two varint bytes) closed, route 0 moved to an offer with an unknown flag. + await held.raw([15, 17, 0, 2, 0x80, 0x9c, 0x80, 0x9c, 0, 0, 1, 0, 6, 1, 7, 0, 0x80, 128, 0]); + expect(await held.announced.next()).toMatchObject({ prefix: c, kind: "offline" }); + await held.send({ status: "routeUpdate", id: 0n, offer: offer(NODE_C) }); + expect(await held.announced.next()).toMatchObject({ prefix: c, kind: "start" }); + held.announced.close(); + held.subscriber.close(); +}); + +// A DEAD in parts applies with its last: a route an earlier part moved is not ended by the gone +// hop, and nothing shows before the last part. Mirrors rs/moq-net. +test("a lite-07 DEAD in parts applies with its last", async () => { + const { subscriber, announced, send } = await routedStream(); + const [a, b] = ["a", "b"].map((path) => Path.from(path)); + const instance = Instance.parse("i"); + // Both routes reach the peer over its session to NODE_A. + const hop = { node: PEER_NODE, session: NODE_A }; + const through = (node: bigint) => [ + { node, session: 0n }, + ...(node === NODE_A ? [] : [{ node: NODE_A, session: 1n }]), + hop, + ]; + for (const [id, node, path] of [ + [0n, NODE_A, a], + [1n, NODE_B, b], + ] as const) { + await send({ status: "routeStart", node, suffix: Path.empty(), offer: offer(node, { path: through(node) }) }); + await send({ status: "active", suffix: path, route: id, instance }); + await send({ status: "floor", id, revision: 0n }); + expect(await announced.next()).toMatchObject({ prefix: path, kind: "start" }); + } + const detour = (node: bigint) => ({ + id: node === NODE_A ? 0n : 1n, + offer: offer(node, { metric: 4n, path: [...through(node).slice(0, -1), { node: PEER_NODE, session: 9n }] }), + }); + const part = (more: boolean, moved: { id: bigint; offer: Offer }[]): AnnounceBroadcast => ({ + status: "dead", + dead: { kind: "closed", hop, far: NODE_A, gone: true, more, moved }, + }); + await send(part(true, [detour(NODE_A)])); + await send(part(false, [detour(NODE_B)])); + for (const path of [a, b]) { + expect(await announced.next()).toMatchObject({ prefix: path, kind: "update", route: { cost: 5n } }); + } + + announced.close(); + subscriber.close(); +}); + +// DRAIN: a node on a route's path sent GOAWAY, so it ranks below every route without one, until +// its next offer. +test("a lite-07 DRAIN ranks the routes through its node last", async () => { + const { subscriber, announced, send } = await routedStream(); + const room = Path.from("room"); + const instance = Instance.parse("i"); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); + await send({ status: "active", suffix: room, route: 0n, instance }); + await send({ status: "floor", id: 0n, revision: 0n }); + expect(await announced.next()).toMatchObject({ prefix: room, kind: "start", route: { node: NODE_A } }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B, { metric: 5n }) }); + await send({ status: "active", suffix: room, route: 1n, instance }); + await send({ status: "floor", id: 1n, revision: 0n }); + + await send({ status: "dead", dead: { kind: "drain", node: NODE_A } }); + expect(await announced.next()).toMatchObject({ prefix: room, kind: "update", route: { node: NODE_B } }); + // Its next offer says whether it still drains. + await send({ status: "routeUpdate", id: 0n, offer: offer(NODE_A) }); + expect(await announced.next()).toMatchObject({ prefix: room, kind: "update", route: { node: NODE_A } }); + + announced.close(); + subscriber.close(); +}); + +// A claim ranks by the lowest priority of a link on its route's path before its cost, as routes to +// one node do, so a cheaper claim behind a demoted link loses to a dearer one. Mirrors rs/moq-net. +test("a lite-07 claim ranks by its path's priority before its cost", async () => { + const { subscriber, announced, send } = await routedStream(); + const room = Path.from("room"); + const instance = Instance.parse("i"); + await send({ + status: "routeStart", + node: NODE_A, + suffix: Path.empty(), + offer: offer(NODE_A, { priority: -10 }), + }); + await send({ status: "active", suffix: room, route: 0n, instance }); + await send({ status: "floor", id: 0n, revision: 0n }); + expect(await announced.next()).toMatchObject({ prefix: room, kind: "start", route: { node: NODE_A } }); + await send({ status: "routeStart", node: NODE_B, suffix: Path.empty(), offer: offer(NODE_B, { metric: 5n }) }); + await send({ status: "active", suffix: room, route: 1n, instance }); + await send({ status: "floor", id: 1n, revision: 0n }); + expect(await announced.next()).toMatchObject({ prefix: room, kind: "update", route: { node: NODE_B } }); + + announced.close(); + subscriber.close(); +}); + +// A route covers the requested prefix joined with its suffix: one origin node may have a route +// per prefix, and a request through one names how many segments of its path that prefix spans. +test("a lite-07 request names its route's prefix depth", async () => { + const { subscriber, announced, send, trackRequest } = await routedStream(); + const x = Path.from("live/room/x"); + const y = Path.from("y"); + const instance = Instance.parse("i"); + await send({ status: "routeStart", node: NODE_A, suffix: Path.from("live/room"), offer: offer(NODE_A) }); + await send({ status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }); + await send({ status: "active", suffix: x, route: 0n, instance }); + await send({ status: "floor", id: 0n, revision: 0n }); + await send({ status: "active", suffix: y, route: 1n, instance }); + await send({ status: "floor", id: 1n, revision: 0n }); + expect(await announced.next()).toMatchObject({ prefix: x, kind: "start" }); + expect(await announced.next()).toMatchObject({ prefix: y, kind: "start" }); + + const deep = await trackRequest(x, 1); + expect(deep.request).toMatchObject({ node: NODE_A, depth: 2 }); + const shallow = await trackRequest(y, 2); + expect(shallow.request).toMatchObject({ node: NODE_A, depth: 0 }); + + for (const pick of [deep, shallow]) pick.consumed.close(); + announced.close(); + subscriber.close(); + for (const pick of [deep, shallow]) await pick.info; +}); + test.each([ [ - "a second route to one origin node", + "a second route to one origin node at one prefix", + [ + { status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }, + { status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }, + ], + ], + [ + "a claim outside its route's prefix", [ - { status: "routeStart", node: NODE_A, advert: ADVERT }, - { status: "routeStart", node: NODE_A, advert: ADVERT }, + { status: "routeStart", node: NODE_A, suffix: Path.from("live"), offer: offer(NODE_A) }, + { status: "active", suffix: Path.from("other"), route: 0n, instance: Instance.parse("a") }, + ], + ], + [ + "a DEAD moving an unknown route", + [ + { + status: "dead", + dead: { + kind: "closed", + hop: { node: PEER_NODE, session: 1n }, + far: 3n, + gone: false, + more: false, + moved: [{ id: 0n, offer: offer(NODE_A) }], + }, + }, + ], + ], + [ + "a DEAD part followed by another message", + [ + { status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }, + { + status: "dead", + dead: { + kind: "closed", + hop: { node: PEER_NODE, session: NODE_A }, + far: NODE_A, + gone: true, + more: true, + moved: [], + }, + }, + { status: "floor", id: 0n }, + ], + ], + [ + "a DEAD part continued by one naming another hop", + [ + { + status: "dead", + dead: { + kind: "closed", + hop: { node: PEER_NODE, session: 1n }, + far: 3n, + gone: true, + more: true, + moved: [], + }, + }, + { + status: "dead", + dead: { + kind: "closed", + hop: { node: PEER_NODE, session: 2n }, + far: 3n, + gone: true, + more: false, + moved: [], + }, + }, ], ], [ "a route claiming one path twice", [ - { status: "routeStart", node: NODE_A, advert: ADVERT }, + { status: "routeStart", node: NODE_A, suffix: Path.empty(), offer: offer(NODE_A) }, { status: "active", suffix: Path.from("room"), route: 0n, instance: Instance.parse("a") }, { status: "active", suffix: Path.from("room"), route: 0n, instance: Instance.parse("b") }, ], diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index a0a71b5a85a..9e2e6a8251f 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -63,17 +63,19 @@ import { Budget } from "../util/yield.ts"; import { hasTime, rangesStart, sameRanges, Window, wireRanges, wireSpan } from "../window.ts"; import { overrideBroadcastWire, wireOf } from "../wire.ts"; import { - type Advert, AnnounceHistory, AnnounceInit, AnnounceOk, type Refusal as AnnounceRefusal, AnnounceRequest, type Base, + type Dead, decodeAnnounceBroadcastMaybe, - encodeAnnounceReply, + type Offer, type Refusable, ROUTE_MAX_HOPS, + type RouteHop, + type Unusable, } from "./announce.ts"; import { Datagram as DatagramMessage, Datagrams, DatagramsInfo, DatagramsUpdate } from "./datagram.ts"; import { closeViolation } from "./error.ts"; @@ -158,9 +160,10 @@ interface SubscribeSetup { */ interface Consumed { readonly instance?: Instance.Valid; - // The origin node every request names (lite-07): the one the route selected at consume - // reaches, or 0 to leave the choice to the publisher. + // The publisher node every request names (lite-07): the one the route selected at consume + // reaches, or 0 to leave the choice to the publisher, and the depth of that route's prefix. readonly node?: bigint; + readonly depth?: number; readonly id?: number; readonly valid?: AbortSignal; // Aborted once another stream takes the path, which may answer a request without an instance @@ -168,10 +171,12 @@ interface Consumed { readonly answering?: AbortSignal; } -// What a request names to pin it: the serving instance, and on lite-07 its origin node. +// What a request names to pin it: the serving instance, and on lite-07 its publisher node and the +// depth of the route's prefix. interface Pin { readonly instance?: Instance.Valid; readonly node?: bigint; + readonly depth?: number; } interface DatagramEntry { @@ -1021,7 +1026,7 @@ export class Subscriber { // origin ranks routes, and a consume pins its requests to that claim's origin node. A claim // another stream's newer floor withdrew or replaced is held back until this stream catches up // (see `Floors`). Mirrors `subscriber/routed.rs` in rs/moq-net for a node that forwards - // nothing: it holds no feasibility distance, so it never sends a ROUTE_REQUEST. + // nothing: an endpoint applies the peer's withdrawals and holds, and casts no doubt. async #receiveRouted( stream: Stream, announced: announce.Producer, @@ -1037,10 +1042,17 @@ export class Subscriber { type Route = { id: bigint; node: bigint; + // The prefix it covers, the requested one joined with the peer's suffix, or undefined when + // this node refused that suffix, which leaves the route refused for good. + prefix: Path.Valid | undefined; + // The path the peer last named, kept while this node refuses the rest, so a DEAD finds it. + path: RouteHop[]; // Undefined while this node refuses what the peer forwarded: no route at all, its claims - // offline, until a ROUTE_UPDATE it can use. Mirrors rs/moq-net. - advert: Advert | undefined; - // ROUTE_HOLD: no usable path to the origin, until the next ROUTE_UPDATE. + // offline, until an offer it can use. Mirrors rs/moq-net. + offer: Offer | undefined; + // A DRAIN named a node on its path since its last offer: it ranks last. + drain: boolean; + // ROUTE_HOLD, or a DEAD that held it: no usable path to the publisher, until its next offer. held: boolean; // The Announce ID of each live claim, by path: one per path, checked by lookup. claims: Map; @@ -1068,8 +1080,10 @@ export class Subscriber { // A claim refused has no `claim` until a restart names all of it, and one refused before its // path has no `path` either, and never will. const claims = new SlotMap(this.#announces); - // The origin nodes with a live route here: at most one route each. - const nodes = new Set(); + // The live route to each publisher node and prefix: at most one each. + const named = new Map(); + const key = (node: bigint, at: Path.Valid | undefined) => + at === undefined || node === 0n ? undefined : `${node} ${at}`; // The routes whose committed claims include each path. const byPath = new Map>(); // What each visible path reports; see `refresh`. @@ -1145,27 +1159,38 @@ export class Subscriber { else wire.pending.set(path, claim); }; + // Whether this node takes what the peer offers for the route at all: not a refused offer or + // suffix, nor a route to the reserved node. Only such a route joins the node's floors. + const joined = (wire: Route): wire is Route & { offer: Offer; prefix: Path.Valid } => + wire.offer !== undefined && wire.prefix !== undefined && wire.node !== 0n; // A route is no route at all once it crossed as many nodes as one may, or crossed us: a copy // of our own route, come back. - const usable = (wire: Route): wire is Route & { advert: Advert } => - wire.advert !== undefined && - wire.advert.hops.length < ROUTE_MAX_HOPS && - !wire.advert.hops.includes(this.hop); + // The same, without narrowing, to compare a route before and after it changes. + const filed = (wire: Route): boolean => joined(wire); + const usable = (wire: Route): wire is Route & { offer: Offer; prefix: Path.Valid } => + joined(wire) && + wire.offer.path.length < ROUTE_MAX_HOPS && + !wire.offer.path.some((hop) => hop.node === this.hop); // A claim through this session: its cost plus the route's metric, past this session's link - // cost of 1 as Rust's default, and unavailable while the route is held. - const candidate = (wire: Route & { advert: Advert }, claim: Claim): WireRoute => + // cost of 1 as Rust's default, ranked last while a node on its path drains, then by the + // lowest priority on its path, past this link's 0 as Rust's default, and unavailable while + // the route is held. Requests through it name its publisher and its prefix's depth. + const candidate = (wire: Route & { offer: Offer; prefix: Path.Valid }, claim: Claim): WireRoute => this.#drained( WireRoute.normalize({ instance: claim.instance, epoch: claim.epoch, - cost: claim.cost + wire.advert.metric + 1n, + cost: claim.cost + wire.offer.metric + 1n, node: wire.node, + depth: Path.parts(wire.prefix).length, + priority: Math.min(0, wire.offer.priority), + draining: wire.offer.draining || wire.drain || undefined, available: !wire.held, // The relay's sum plus this link's own, unknown while either is. rtt: - wire.advert.rtt !== undefined && linkRtt !== undefined - ? linkRtt.checkedAdd(Duration.millis(Number(wire.advert.rtt))) + wire.offer.rtt !== undefined && linkRtt !== undefined + ? linkRtt.checkedAdd(Duration.millis(Number(wire.offer.rtt))) : undefined, }), ); @@ -1259,17 +1284,30 @@ export class Subscriber { } }; - // The peer started a route to `node` with the next Route ID, reaching it by `advert`, or - // undefined when this node refuses what the peer forwarded. - const startRoute = async (node: bigint, advert: Advert | undefined) => { - // The peer keeps one route per origin node; a refused reserved node is none. - if (node !== 0n && nodes.has(node)) throw new ProtocolViolation(`second route to node ${node}`); + // The peer started a route to `node` with the next Route ID, covering the requested prefix + // joined with `suffix`, along `offer`, or along `path` when this node refuses the offer. + const startRoute = async ( + node: bigint, + suffix: Path.Valid | undefined, + offer: Offer | undefined, + path: RouteHop[], + ) => { + // The interest prefix can push a joined path past the limits: refused too. + const joinedPrefix = suffix === undefined ? undefined : Path.join(prefix, suffix); + const at = joinedPrefix !== undefined && Path.check(joinedPrefix) === undefined ? joinedPrefix : undefined; + const named_ = key(node, at); + // The peer keeps one route per publisher and prefix. + if (named_ !== undefined && named.has(named_)) + throw new ProtocolViolation(`second route to node ${node} at ${at}`); const id = nextRoute++; - const view: Route["view"] = { prefix, hidden, revision: undefined, claims: new Map() }; + const view: Route["view"] = { prefix: at ?? prefix, hidden, revision: undefined, claims: new Map() }; const wire: Route = { id, node, - advert, + prefix: at, + path, + offer, + drain: false, held: false, claims: new Map(), unnamed: new Set(), @@ -1288,9 +1326,9 @@ export class Subscriber { }; routes.set(id, wire); const started = sample(); - if (node !== 0n) nodes.add(node); + if (named_ !== undefined) named.set(named_, id); // A refused route's floors stay its own, as Rust's node never holds the route. - if (advert !== undefined) this.#floors.join(node, wire.carrier); + if (joined(wire)) this.#floors.join(node, wire.carrier); // The link's first read lands before the route's first claims show, as Rust // reads the link before it serves them: a listing reports a path's route when // it turns present and not again for an RTT, so a read landing after would @@ -1298,6 +1336,27 @@ export class Subscriber { if (started) await race([firstRead, announced.closed]); }; + // The peer let go of the route and every claim under it: offline, not withdrawn. + const endRoute = (wire: Route) => { + for (const id of [...wire.claims.values(), ...wire.unnamed]) { + history.end(id); + claims.delete(id); + } + routes.delete(wire.id); + sample(); + const named_ = key(wire.node, wire.prefix); + if (named_ !== undefined && named.get(named_) === wire.id) named.delete(named_); + // Unindexed first: leaving its floors reshows another carrier at once, which + // refreshes these paths and so looks up every route indexed there. + for (const path of wire.view.claims.keys()) byPath.get(path)?.delete(wire.id); + // What its last floor certified stays, so no other stream's lagging claim revives. + this.#floors.leave(wire.node, wire.carrier); + wire.shown.clear(); + refreshRoute(wire, true); + this.#consumes.revalidate(); + for (const path of wire.view.claims.keys()) if (byPath.get(path)?.size === 0) byPath.delete(path); + }; + // A claim the peer started under `routeId` with the next Announce ID, at `path` unless this // node refused it, and `claim` unless it refuses what the peer forwarded. const startClaim = ( @@ -1318,30 +1377,98 @@ export class Subscriber { return; } if (wire.claims.has(named)) throw new ProtocolViolation(`route ${routeId} claims ${named} twice`); + // A route covers only the claims at or below its prefix. + if (wire.prefix !== undefined && !Path.hasPrefix(wire.prefix, named)) + throw new ProtocolViolation(`route ${routeId} at ${wire.prefix} claims ${named}`); claims.set(id, { route: routeId, path: named, claim }); wire.claims.set(named, id); if (claim) change(routeId, named, claim); else wire.withheld.add(id); }; - // The route now reaches its node by `advert`, or undefined while this node refuses what the - // peer forwarded. A refused route leaves the node's floors, as an ended one does, and joins - // again once usable, with what its last floor certified, as rs/moq-net files it again. - const reach = (wire: Route, advert: Advert | undefined) => { - const joined = wire.advert !== undefined; - // The reserved node names no route, whatever the peer forwards for it. - wire.advert = wire.node === 0n ? undefined : advert; - if (joined && wire.advert === undefined) { + // The route now reaches its node by `offer`, or along `path` while this node refuses what the + // peer forwarded. Either way the peer no longer holds it, nor drains a node on it but as the + // offer says. A refused route leaves the node's floors, as an ended one does, and joins again + // once usable, with what its last floor certified, as rs/moq-net files it again. + const reach = (wire: Route, offer: Offer | undefined, path: RouteHop[]) => { + const was = filed(wire); + wire.offer = offer; + wire.path = path; + wire.held = false; + wire.drain = false; + const now = filed(wire); + if (was && !now) { this.#floors.leave(wire.node, wire.carrier); // What its floor certified stays as it was, while the route goes on reading claims. wire.view.claims = new Map(wire.view.claims); } - if (joined || wire.advert === undefined) return; + if (was || !now) return; this.#floors.join(wire.node, wire.carrier); const others = this.#floors.floor(wire.node, wire.carrier); show(wire, new Map(), true); if (others) this.#consumes.revalidate(); }; + // The route moved to `offer`, which this node refuses with its path when it is `Unusable`. + const move = (wire: Route, offer: Offer | Unusable) => { + const was = filed(wire); + if ("refused" in offer) { + console.warn(`refusing a forwarded route: node=${wire.node} route=${wire.id} reason=${offer.refused}`); + reach(wire, undefined, offer.path); + } else { + reach(wire, offer, offer.path); + } + refreshRoute(wire, !usable(wire)); + // Its floors no longer cover anything, as on ROUTE_END. + if (was && !filed(wire)) this.#consumes.revalidate(); + }; + + // The parts of a DEAD read so far, which apply as one once its last arrives. + let continued: { hop: RouteHop; far: bigint; gone: boolean; offers: Map } | undefined; + + // A DEAD: the stream's routes through the hop it names carry on as it says. A route this + // node refuses still follows it, so both ends agree on which routes live. An endpoint casts + // no doubt: it never forwards a route, so no detour waits on one. + const dead = (news: Dead) => { + if (news.kind === "drain") { + for (const wire of routes.values()) { + if (!wire.path.some((hop) => hop.node === news.node)) continue; + wire.drain = true; + refreshRoute(wire, false); + } + return; + } + // A DEAD in parts applies with its last, each part naming the same hop, far node, and gone. + const parts = continued ?? { hop: news.hop, far: news.far, gone: news.gone, offers: new Map() }; + continued = undefined; + const same = + parts.hop.node === news.hop.node && + parts.hop.session === news.hop.session && + parts.far === news.far && + parts.gone === news.gone; + if (!same) throw new ProtocolViolation("DEAD part names another hop"); + // Each part names live routes, so what the parts hold stays within the stream's. + const offers = parts.offers; + for (const { id, offer } of news.moved) offers.set(route(id).id, offer); + if (news.more) { + continued = parts; + return; + } + const through = [...routes.values()].filter((wire) => + wire.path.some((hop) => hop.node === news.hop.node && hop.session === news.hop.session), + ); + for (const wire of through) { + const moved = offers.get(wire.id); + const before = wire.path.at(-2); + if (moved !== undefined) move(wire, moved); + else if (news.gone) endRoute(wire); + // The peer learned it from this node, which it holds under: unchanged. + else if (before?.node === this.hop) continue; + else { + wire.held = true; + refreshRoute(wire, false); + } + } + }; // Apply a message whose content, which the peer only forwarded, this node refuses: withdraw // the route or claim it names, and keep the stream. Mirrors rs/moq-net. @@ -1349,17 +1476,9 @@ export class Subscriber { switch (message.kind) { case "routeStart": console.warn(`refusing a forwarded route: node=${message.node} reason=${reason}`); - return startRoute(message.node, undefined); - case "routeUpdate": { - const wire = route(message.id); - console.warn(`refusing a forwarded route: node=${wire.node} route=${message.id} reason=${reason}`); - const joined = wire.advert !== undefined; - reach(wire, undefined); - refreshRoute(wire, true); - // Its floors no longer cover anything, as on ROUTE_END. - if (joined) this.#consumes.revalidate(); - return; - } + return startRoute(message.node, message.suffix, undefined, message.path); + case "routeUpdate": + return move(route(message.id), { refused: reason, path: message.path }); case "start": console.warn( `refusing a forwarded claim: route=${message.route} claim=${history.next} reason=${reason}`, @@ -1392,45 +1511,29 @@ export class Subscriber { // undefined: the stream ended. null: the consumer closed cleanly. if (!msg) break; if (msg instanceof Error) throw msg; + // The parts of a DEAD arrive back to back. + if (continued && !(msg.status === "dead" && msg.dead.kind === "closed")) + throw new ProtocolViolation("DEAD part followed by another message"); switch (msg.status) { case "routeStart": - await startRoute(msg.node, msg.advert); + await startRoute(msg.node, msg.suffix, msg.offer, msg.offer.path); break; - case "routeUpdate": { - const wire = route(msg.id); - wire.held = false; - reach(wire, msg.advert); - refreshRoute(wire, !usable(wire)); + case "routeUpdate": + move(route(msg.id), msg.offer); break; - } case "routeHold": { const wire = route(msg.id); wire.held = true; refreshRoute(wire, false); break; } - case "routeEnd": { - // Every claim under the route goes with it: offline, not withdrawn. - const wire = route(msg.id); - for (const id of [...wire.claims.values(), ...wire.unnamed]) { - history.end(id); - claims.delete(id); - } - routes.delete(msg.id); - sample(); - nodes.delete(wire.node); - // What its last floor certified stays, so no other stream's lagging claim revives. - this.#floors.leave(wire.node, wire.carrier); - for (const path of wire.view.claims.keys()) byPath.get(path)?.delete(msg.id); - wire.shown.clear(); - refreshRoute(wire, true); - this.#consumes.revalidate(); - for (const path of wire.view.claims.keys()) - if (byPath.get(path)?.size === 0) byPath.delete(path); - await encodeAnnounceReply(stream.writer, { kind: "echo", id: msg.id }, this.version); + case "routeEnd": + endRoute(route(msg.id)); + break; + case "dead": + dead(msg.dead); break; - } case "floor": { // The batch applies whole, at the revision its floor certifies: none while this // node refuses one of the route's claims, which its origin never withdrew. @@ -1438,8 +1541,8 @@ export class Subscriber { const revision = wire.unnamed.size === 0 && wire.withheld.size === 0 ? msg.revision : undefined; // A floor that certifies less keeps what the route's last one certified. const certified = wire.view.revision; - const joined = wire.advert !== undefined; - if (joined && certified !== undefined && (revision === undefined || revision < certified)) + const filed = joined(wire); + if (filed && certified !== undefined && (revision === undefined || revision < certified)) this.#floors.regress(wire.node, wire.carrier); const touched = new Map(); for (const [path, claim] of wire.pending) { @@ -1467,7 +1570,7 @@ export class Subscriber { wire.view.revision = revision; // Every other route to the node is shown anew first; with one, this route's // unchanged claims may change state too. - const others = joined && this.#floors.floor(wire.node, wire.carrier); + const others = filed && this.#floors.floor(wire.node, wire.carrier); show(wire, touched, others); // Another view of the node may now cover a consumed path differently. if (others) this.#consumes.revalidate(); @@ -1544,6 +1647,16 @@ export class Subscriber { } } + /** + * Whether the session's claim of origin `node` at `at` may serve a request for `path` beneath + * it: what the node's newest floor certifies there. See `Floors.covers`. + * + * @internal + */ + covers(node: bigint, at: Path.Valid, path: Path.Valid): boolean { + return this.#floors.covers(node, at, path); + } + /** * Consumes a broadcast from the connection. * @@ -1566,13 +1679,21 @@ export class Subscriber { #createConsume(path: Path.Valid, instance: ReturnType): broadcast.Consumer { const consumes = this.#consumes; - const pinned = instance.route.node; + const pinned = + instance.route.node === undefined + ? undefined + : { node: instance.route.node, depth: instance.route.depth ?? 0 }; + // The best route now serving the instance, which moves with failover even once another + // instance wins the path; the one it first resolved through once none does. + const target = (instance: Instance.Valid | undefined) => + (instance !== undefined ? consumes.node(path, instance) : undefined) ?? pinned; const consumed = { instance: instance.route.instance, - // The node of the best route now serving the instance, which moves with failover even - // once another instance wins the path; the one it first resolved through once none does. get node() { - return (this.instance !== undefined ? consumes.node(path, this.instance) : undefined) ?? pinned; + return target(this.instance)?.node; + }, + get depth() { + return target(this.instance)?.depth; }, id: this.#consumeNext++, get valid() { @@ -1671,6 +1792,7 @@ export class Subscriber { broadcast: path, instance: consumed.instance, node: consumed.node, + depth: consumed.depth, track: request.name, priority, }).encode(opened.writer, this.version); @@ -1829,7 +1951,10 @@ export class Subscriber { // Gave up, perhaps handed a stream as it did: the catch below hands back any stream. if (!stream || cut.aborted) throw cut.reason; await stream.writer.u53(StreamId.Tracks); - await new TracksMessage(broadcast, consumed.instance, consumed.node).encode(stream.writer, this.version); + await new TracksMessage(broadcast, consumed.instance, consumed.node, consumed.depth).encode( + stream.writer, + this.version, + ); let synced = false; const answering = consumed.answering; let unanswered = answering ? AbortSignal.any([stop, answering]) : stop; @@ -2039,6 +2164,7 @@ export class Subscriber { broadcast, instance: consumed.instance, node: consumed.node, + depth: consumed.depth, track: name, priority: subscription.priority ?? 0, order: subscription.order, @@ -2372,7 +2498,7 @@ export class Subscriber { // stays open, as interest in the track, until the caller closes it. async #trackInfo( broadcast: Path.Valid, - { instance, node }: Pin, + { instance, node, depth }: Pin, track: string, signal: AbortSignal | undefined, deadline: Deadline, @@ -2381,7 +2507,7 @@ export class Subscriber { { version: this.version }, async (stream) => { await stream.writer.u53(StreamId.Track); - await new TrackMessage(broadcast, track, instance, node).encode(stream.writer, this.version); + await new TrackMessage(broadcast, track, instance, node, depth).encode(stream.writer, this.version); const info = await TrackInfo.decode(stream.reader, this.version); return { info, stream }; }, @@ -2589,7 +2715,7 @@ export class Subscriber { deadline: Deadline, maxDelay?: Duration, ): Promise<{ stream: Stream; info: TrackInfo }> { - const { instance, node, valid } = consumed; + const { instance, node, depth, valid } = consumed; const cancel = new AbortController(); void group.closed.then((err) => cancel.abort(err ?? new StreamError(StreamCode.Cancel))); const signal = valid ? AbortSignal.any([valid, cancel.signal]) : cancel.signal; @@ -2609,6 +2735,7 @@ export class Subscriber { broadcast, instance, node, + depth, track, priority, maxDelay: @@ -3067,6 +3194,7 @@ export class Subscriber { broadcast, instance: consumed.instance, node: consumed.node, + depth: consumed.depth, track: name, priority, maxDelay, diff --git a/js/net/src/lite/track.ts b/js/net/src/lite/track.ts index dc6508347b2..980695c4b27 100644 --- a/js/net/src/lite/track.ts +++ b/js/net/src/lite/track.ts @@ -37,29 +37,32 @@ export class Track { track: string; /** The publisher instance the subscriber expects. Lite-07+. */ instance?: Instance.Valid; - /** The origin node that serves the request, 0 for whichever the publisher selects. Lite-07+. */ + /** The publisher node that serves the request, 0 for whichever the publisher selects. Lite-07+. */ node: bigint; + /** How many leading segments of `broadcast` make up the prefix of the route it follows. Lite-07+. */ + depth: number; - constructor(broadcast: Path.Valid, track: string, instance?: Instance.Valid, node = 0n) { + constructor(broadcast: Path.Valid, track: string, instance?: Instance.Valid, node = 0n, depth = 0) { this.broadcast = broadcast; this.track = track; this.instance = instance; this.node = node; + this.depth = depth; } async #encode(w: Writer, version: Version) { await w.string(Path.encode(this.broadcast)); await encodePin(w, version, this.instance); - await encodeNode(w, version, this.node); + await encodeNode(w, version, this.node, this.depth); await w.string(this.track); } static async #decode(r: Reader, version: Version): Promise { const broadcast = Path.decode(await r.string()); const instance = await decodePin(r, version); - const node = await decodeNode(r, version); + const { node, depth } = await decodeNode(r, version, broadcast); const track = await r.string(); - return new Track(broadcast, track, instance, node); + return new Track(broadcast, track, instance, node, depth); } async encode(w: Writer, version: Version): Promise { diff --git a/js/net/src/lite/tracks.ts b/js/net/src/lite/tracks.ts index 0c6a4f62fe9..3c25a629fb1 100644 --- a/js/net/src/lite/tracks.ts +++ b/js/net/src/lite/tracks.ts @@ -26,13 +26,16 @@ export class Tracks { broadcast: Path.Valid; /** The publisher instance the subscriber expects. */ instance?: Instance.Valid; - /** The origin node that serves the request, 0 for whichever the publisher selects. Lite-07+. */ + /** The publisher node that serves the request, 0 for whichever the publisher selects. Lite-07+. */ node: bigint; + /** How many leading segments of `broadcast` make up the prefix of the route it follows. Lite-07+. */ + depth: number; - constructor(broadcast: Path.Valid, instance?: Instance.Valid, node = 0n) { + constructor(broadcast: Path.Valid, instance?: Instance.Valid, node = 0n, depth = 0) { this.broadcast = broadcast; this.instance = instance; this.node = node; + this.depth = depth; } async encode(w: Writer, version: Version): Promise { @@ -40,7 +43,7 @@ export class Tracks { return Message.encode(w, async (w) => { await w.string(Path.encode(this.broadcast)); await encodePin(w, version, this.instance); - await encodeNode(w, version, this.node); + await encodeNode(w, version, this.node, this.depth); }); } @@ -49,7 +52,8 @@ export class Tracks { return Message.decode(r, async (r) => { const broadcast = Path.decode(await r.string()); const instance = await decodePin(r, version); - return new Tracks(broadcast, instance, await decodeNode(r, version)); + const { node, depth } = await decodeNode(r, version, broadcast); + return new Tracks(broadcast, instance, node, depth); }); } } diff --git a/js/net/src/origin.ts b/js/net/src/origin.ts index f4ffa56d944..bded4b83e16 100644 --- a/js/net/src/origin.ts +++ b/js/net/src/origin.ts @@ -299,6 +299,12 @@ interface RouteEntry { readonly route: Signal; readonly originated: boolean; readonly server?: ServeState; + /** + * Whether the entry may serve a request for `path` strictly beneath its prefix, asked per + * request; unset when it serves its whole subtree. A lite-07 claim covers only what its origin + * node's newest floor still certifies there, which its announcement cannot say. + */ + readonly covers?: (path: Path.Valid) => boolean; /** * Set as a received route leaves because its session lost it, rather than its announcer * ending it: readers then see it go offline, not end. @@ -306,6 +312,11 @@ interface RouteEntry { lost: boolean; } +/** Whether `entry`, at `prefix`, may serve a request for `path`. */ +function admits(entry: RouteEntry, prefix: Path.Valid, path: Path.Valid): boolean { + return entry.scope.matches(path) && (prefix === path || entry.covers === undefined || entry.covers(path)); +} + /** One advertisement at a prefix. `exact` marks an announced local broadcast, which is only its own path. */ interface Candidate extends Advertised { readonly exact: boolean; @@ -965,7 +976,7 @@ class OriginState { for (const [prefix, entries] of this.routes.peek()) { if (!Path.hasPrefix(prefix, path)) continue; for (const entry of entries) { - if (!entry.server || !entry.scope.matches(path) || entry.route.peek().instance !== instance) continue; + if (!entry.server || !admits(entry, prefix, path) || entry.route.peek().instance !== instance) continue; if (matches(entry.identity)) return true; } } @@ -1138,7 +1149,7 @@ class OriginState { path, prefix, entries, - (candidate) => !candidate.scope.matches(path) || (skip?.(candidate) ?? false), + (candidate) => !admits(candidate, prefix, path) || (skip?.(candidate) ?? false), prefix === path && local ? [local] : [], ); if (entry) return entry; @@ -1355,7 +1366,7 @@ export class Producer implements Table { constructor() { const thisProducer = this; registerWire(this, { - receive: (prefix, route) => this.#receive(prefix, route), + receive: (prefix, route, covers) => this.#receive(prefix, route, covers), interests: (listener) => { if (this.#borrowed?.closed) return () => {}; // What the listener currently wants, so a derived handle's close can retract it @@ -1613,12 +1624,30 @@ export class Producer implements Table { return this.#serving(prefix, info, true, true, !options.hidden); } - /** Land a route a peer announced, taken as given. Not originated, so a session never announces it back. */ - #receive(prefix: Path.Valid, route?: Partial): Claim { - return this.#serving(prefix, WireRoute.normalize(route), false, false, true); + /** + * Land a route a peer announced, taken as given. Not originated, so a session never announces it + * back. `covers` asks, per request, whether it serves a path beneath `prefix`, both as the + * session names them. + */ + #receive(prefix: Path.Valid, route?: Partial, covers?: (path: Path.Valid) => boolean): Claim { + const at = this.#scope.prefix(prefix); + const below = + covers && + ((path: Path.Valid) => { + const suffix = Path.stripPrefix(at, path); + return suffix === null || covers(Path.join(prefix, suffix)); + }); + return this.#serving(prefix, WireRoute.normalize(route), false, false, true, below); } - #serving(prefix: Path.Valid, route: WireRoute, originated: boolean, publishes: boolean, visible: boolean): Claim { + #serving( + prefix: Path.Valid, + route: WireRoute, + originated: boolean, + publishes: boolean, + visible: boolean, + covers?: (path: Path.Valid) => boolean, + ): Claim { prefix = this.#scope.prefix(prefix); // A route that names no instance serves one of its own, and a claim hands that one to its jobs. const minted = route.instance === undefined; @@ -1633,6 +1662,7 @@ export class Producer implements Table { route: new Signal(route), originated, server, + covers, lost: false, }; server.onReject = (path, err) => this.#state.refuse(path, entry, err); @@ -2101,7 +2131,7 @@ export class Consumer implements Reader { local: (path, instance) => this.#local(scope.path(path), instance), demand: (path, instance, signal) => this.#demand(scope.path(path), instance, signal), own: state.own, - covers: (prefix) => scope.covers(prefix), + covers: (prefix: Path.Valid) => scope.covers(prefix), }); } diff --git a/js/net/src/route.ts b/js/net/src/route.ts index 6ac6ba781fe..16724c938cf 100644 --- a/js/net/src/route.ts +++ b/js/net/src/route.ts @@ -5,7 +5,7 @@ */ import { Signal } from "@moq/signals"; -/** The largest lite-07 varint, which no origin advertises as a Seqno or Revision. */ +/** The largest lite-07 varint, which no origin advertises as a Revision. */ const MAX = 2n ** 64n - 1n; /** A fresh Node ID: random, non-zero, and with the most significant bit clear, which marks a qualified one. */ @@ -16,23 +16,22 @@ function fresh(): bigint { return id === 0n ? 1n : id; } -/** The origin node an origin's claims name, and the seqno only it advances. */ +/** The publisher node an origin's claims name. */ export interface Identity { readonly node: bigint; - readonly seqno: bigint; } /** - * One origin's own route: the Node ID every session publishing it declares, the seqno only it - * advances, and the revision numbering the changes to its claims. A browser forwards nothing, so - * this is the only route it originates. Mirrors the declared identity of `route::Node` in - * rs/moq-net. + * One origin's own route: the Node ID every session publishing it declares, and the revision + * numbering the changes to its claims. A browser forwards nothing, so this is the only route it + * originates, along a path of its own hop alone. Mirrors the declared identity of + * `route::path::Router` in rs/moq-net. * * @internal */ export class Own { - /** The current identity; a rotation replaces the node, and a seqno advance the seqno. */ - readonly identity = new Signal({ node: fresh(), seqno: 0n }); + /** The current identity; a rotation replaces the node. */ + readonly identity = new Signal({ node: fresh() }); /** How the origin names itself in a route's Hops: the Node ID it started with, which a rotation keeps. */ readonly hop = this.identity.peek().node; @@ -68,28 +67,9 @@ export class Own { return this.#revision; } - /** A session that carried this origin's route closed, changing a path to it: advance the seqno. */ - bump(): void { - const { node, seqno } = this.identity.peek(); - if (seqno + 1n >= MAX) this.rotate(); - else this.identity.set({ node, seqno: seqno + 1n }); - } - - /** - * A peer asked for a route to `node` at least as new as `seqno`. Only the origin answers, by - * raising its seqno; a request for the last one rotates the identity. A request for another - * node names nothing this origin reaches, and is dropped. - */ - request(node: bigint, seqno: bigint): void { - const current = this.identity.peek(); - if (node !== current.node || seqno <= current.seqno) return; - if (seqno >= MAX) this.rotate(); - else this.identity.set({ node, seqno }); - } - /** Take a fresh identity, with a fresh revision space, ending the old one everywhere. */ rotate(): void { - this.identity.set({ node: fresh(), seqno: 0n }); + this.identity.set({ node: fresh() }); this.#revision = 0n; this.#seen = this.#table(); } diff --git a/js/net/src/wire.ts b/js/net/src/wire.ts index 9bfae3e9912..cc41b68d359 100644 --- a/js/net/src/wire.ts +++ b/js/net/src/wire.ts @@ -62,8 +62,11 @@ export interface OriginProducer { interests(listener: InterestListener): Dispose; /** Whether an advertised prefix overlaps the handle's allowed paths. */ accepts(prefix: Path.Valid): boolean; - /** Land a route a peer announced, taken as given and never forwarded back to a peer. */ - receive(prefix: Path.Valid, route?: Partial): origin.Claim; + /** + * Land a route a peer announced, taken as given and never forwarded back to a peer. `covers`, + * when set, says per request whether it serves a path strictly beneath `prefix`. + */ + receive(prefix: Path.Valid, route?: Partial, covers?: (path: Path.Valid) => boolean): origin.Claim; attach(discovery: boolean): Dispose; /** * Hold the origin's announcement streams overlapping `prefix` unsynced until @@ -164,6 +167,11 @@ export interface Established { * keeps serving. */ leave(): void; + /** + * Whether the session's claim of origin `node` at `at` may serve a request for `path` beneath + * it: lite-07's floors. Unset where every claim covers its subtree. + */ + covers?(node: bigint, at: Path.Valid, path: Path.Valid): boolean; // private: plumbing /** * This session's side of its auth handle. diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index fb6cabff2c4..01ff7e29b8e 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -15,11 +15,11 @@ claim whose withdrawal is already known. 1. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **The draft describes the target before the code switches.** A hop-less ANNOUNCE cannot run before the router drives sessions; the quest's own Plan says so. So the draft now specifies the whole lite-07 route wire, while main's lite-07 session keeps sending hop-based ANNOUNCEs until route session integration ([moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023)) lands. lite-07 is unpublished `-wip`, off by default, and the epic lands in one release before the cut. The other option was to put only the new message sections in the draft and leave ANNOUNCE hop-based. I rejected it because ANNOUNCE_FLOOR's "for that route" and the Route ID would then be undefined in the draft. 2. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **The request field is `Node ID (i)`, with 0 meaning "the publisher selects".** It is added to every request that resolves a path: TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS. Node ID 0 is reserved everywhere. -3. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ROUTE_REQUEST names a Node ID, not a Route ID.** A request can then never race a ROUTE_END into a protocol violation. +3. ~~✅ Decided 2026-11-05 (maintainer, /quest-plan): **ROUTE_REQUEST names a Node ID, not a Route ID.** A request can then never race a ROUTE_END into a protocol violation.~~ Superseded by item 27: ROUTE_REQUEST is gone ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)). 4. ~~A held ROUTE_UPDATE carries a seqno and down-only bit of 0.~~ Superseded by item 14: a held route is its own message and carries no seqno or bit. 5. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Floor regions have no order, and a repeated region is a violation.** A sorted order would need Rust's `Path` ordering reproduced exactly in JS. Decoding into a map rejects duplicates at the same cost. Superseded the same day by item 10, which removes regions. 6. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **The Hop setup parameter and ANNOUNCE_OK's Hop ID are dropped from lite-07.** Their only roles were path-vector exclusion and the implicit trailing hop. ANNOUNCE_OK reserves no body for a route-key proof: [moq-dev/moq#7937](https://github.com/moq-dev/moq/pull/7937), accepted the same day, proves the route key once per session in SETUP. The live hop-based codec still switches in session integration, as choice 1 requires. -7. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **An echo naming a Route ID that was never ended, or was already echoed, is a violation at the publisher.** The publisher can check this from its live and unacknowledged ids alone, with no growing state. +7. ~~✅ Decided 2026-11-05 (maintainer, /quest-plan): **An echo naming a Route ID that was never ended, or was already echoed, is a violation at the publisher.** The publisher can check this from its live and unacknowledged ids alone, with no growing state.~~ Superseded by item 27: the ROUTE_END echo is gone, and a subscriber sends nothing after its ANNOUNCE_REQUEST ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)). 8. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ANNOUNCE_TAKEOVER keeps its shape, with a Route ID in place of hops, and its predecessor must be under the same route.** [moq-dev/moq#8063](https://github.com/moq-dev/moq/pull/8063) deletes the message, so this keeps the change minimal. Decided 2026-11-06 (maintainer, /quest-complete on [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023)), asked whether session integration ports the message or lets lite-07 answer a claim as a restart or start until announce-takeover lands: ✅ **port it**, so a job answering a claim stays a takeover across relays. Rejected: accepting the restart and start. 9. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **Unknown announce types stay skipped by length on lite-06+**, as before. An unknown reply type is refused. 10. ✅ Decided 2026-11-05 (maintainer, /quest-plan): **ANNOUNCE_FLOOR is a marker, not a region list:** `{Route ID, Revision}`. It closes a batch and certifies the stream's claims for that route across the whole requested prefix as the origin's at that Revision. A publisher that cannot vouch for the whole requested prefix at one Revision still closes the batch, with a floor that certifies nothing, encoded per item 13. Regions, the replacing list, and the 65,535-byte overflow go away. Accepted cost: a relay with partial knowledge of an origin certifies nothing for that route, so a receiver lets its stale claims go only on ROUTE_END. Rejected: one region per message; a multipart atomic list. @@ -35,17 +35,17 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 18. ✅ Asked who is an origin node: **every publisher stays an origin node.** Its identity is for signing, for one node across several links (redundant publishers), and because browsers may forward (peer-to-peer routing, outside this repository since 2026-11-14). Rejected: a relay originating, under its own Node ID, the claims of any peer that sends no Node ID or holds a scoped publish grant, as it already does for lite-06 and moq-transport publishers. That would make route state follow relays rather than publishers and remove qualified Node IDs, the Route Key, and the echo, but a publisher would have no identity of its own. Accepted cost: routes, holds, and seqno requests scale with publishers, so a node must still forget dead identities, and the echo that makes forgetting safe has to be sound. 19. ✅ Asked whether to keep them: **ROUTE_HOLD stays, and the newest floor wins per identity.** Rejected: dropping holds, where each stall takes every claim behind the node offline and re-sends them all; and claims that follow only the selected next hop, which re-pulls a node's whole claim table on each next-hop change and showed revivals and false ENDs in the sweeps below. -20. ✅ Asked when an origin advances its seqno unprompted: **only when a route link changes**, implemented as a bump when a route link closes. Rejected: only to answer a ROUTE_REQUEST; keeping the per-stream bump. +20. ~~✅ Asked when an origin advances its seqno unprompted: **only when a route link changes**, implemented as a bump when a route link closes. Rejected: only to answer a ROUTE_REQUEST; keeping the per-stream bump.~~ Superseded by item 27: there is no seqno; a closed route link is reported by the DEAD naming its hop ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)). 21. ✅ Asked how one side's routing messages travel so none overtakes another: **one unidirectional Announce Stream per side**, carrying its requests, its answers, and its echoes in order ([the route stream](/quest/m2/cluster-routes/route-stream.md)). It replaces "There MAY be multiple Announce Streams". Rejected: one bidirectional Announce Stream per direction with the echo moved to the answering half; one bidirectional stream per session. 22. ✅ **A side's interest is a set of prefixes edited in place.** The live set never nests, and one edit may swap prefixes for a wider or narrower one without re-sending what both cover ([the interest set](https://github.com/moq-dev/moq/pull/8728)). A token-scope change is such an edit, with one prefix per region of the scope. This supersedes items 16 and 17, and the handoff contract's "An interest change opens a new stream" below, once the route stream lands. 23. ✅ Asked what a floor covers with several prefixes on one stream: **one prefix, named by its id.** This amends item 10: the floor is still a marker and never a region list. Rejected: a floor over the whole set. 24. ✅ Asked how a side learns a newly added prefix's initial set is complete: **SYNCED per prefix, naming its id**, plus one marker per edit where it takes effect for both sides. Rejected: one SYNCED per edit, counted in order. -25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region, unless it changes whether the node may use the route: by crossing it, or by reaching the hop limit. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes. +25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region, unless it changes whether the node may use the route: by crossing it, or by reaching the hop limit. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes. Its echo and distance drain are superseded by item 27, which keeps the self-naming rule and drops the distance, the seqno, the request, and the echo ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)). 26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **a route is shared per publisher and per scope, and a relay believes a route by that route's own path** ([the prototype](https://github.com/moq-dev/moq/pull/8533), the model (`rs/moq-net/verify/route/RouteHandoff.tla`), then [the router](https://github.com/moq-dev/moq/pull/8622)). A scope is the prefix a relay asked a neighbour for when its grant there is narrower than its interest. Today only a feasible neighbour counts, one strictly cheaper than the distance the relay remembers, so a relay's claims move whenever a cost or seqno does: it can forget a claim for good, a mesh with two grants never settles after one seqno advance, and behind a scoped neighbour it never shows what a farther one knows. Only a broadcast some relay carries gets its own route, and that route propagates ([cache-aware routing](/quest/m3/cache-routing/README.md) Plan item 8). What keeps a route free of loops and stale copies is item 27. The contract's bullet on which neighbours certify now states the rule the model settled. First decided the same day as any neighbour whose path to the publisher does not pass through the relay; tried on the router, two relays behind a scoped relay then keep a dead claim alive through each other, and the relay behind a scoped neighbour shows a claim it has no route to request. Rejected: that rule; believing a neighbour that is cheaper than the relay is now, which fixed the storm and the forgotten claim with no wire change but rests belief on distance; a path per broadcast as lite-06 has; link-state among relays; routes to relays. -27. ✅ Decided 2026-11-11 (maintainer, on [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533), which measured it), asked what keeps a route free of loops and stale copies: **the route's path, a withdrawal that names the hop that died, holds, and a timed doubt. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo go.** The model (`rs/moq-net/verify/route/RouteHandoff.tla`) checks it, with dead hops kept until their timer fires, and [the router](https://github.com/moq-dev/moq/pull/8622) builds it as `route::path::Router` beside today's, which only the tests drive. [The switch](/quest/m2/cluster-routes/switch.md) puts it on the wire (item 28), and the contract below describes today's router until it does. The rule, as `rs/moq-net/src/route/path.rs` runs it: +27. ✅ Decided 2026-11-11 (maintainer, on [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533), which measured it), asked what keeps a route free of loops and stale copies: **the route's path, a withdrawal that names the hop that died, holds, and a timed doubt. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo go.** The model (`rs/moq-net/verify/route/RouteHandoff.tla`) checks it, with dead hops kept until their timer fires, and [the router](https://github.com/moq-dev/moq/pull/8622) builds it as `route::path::Router` beside today's, which only the tests drive. [The switch](https://github.com/moq-dev/moq/pull/8731) put it on the wire (item 28) and deleted Babel's router, and the contract below describes it as built. The rule, as `rs/moq-net/src/route/path.rs` runs it: - A hop names a relay and the session it learned the route on. A relay uses the cheapest route whose path does not name it, at once. - A relay that loses a session names that hop, and every relay drops every route through it, whoever offered it. One message per session covers every publisher and lists the routes that moved to another path. - A relay with no route left holds its claims, unavailable, until each neighbour that learned the route from it has answered: with another way, or with nothing once those behind it have answered. Then it lets go. @@ -74,22 +74,27 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout Neither rule looped, and churn's stale observations stayed near 3,300 under both. -28. ✅ Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel), asked how item 27's removal is split and ordered: **the router first with no wire change, then a switch of the sessions, and the switch lands before the route stream.** [The router](https://github.com/moq-dev/moq/pull/8622) runs the path rule with a route per scope, sans-IO, beside today's router. [The switch](/quest/m2/cluster-routes/switch.md) puts it on the lite-07-wip wire, the draft, the Rust sessions, and `js/net`, and deletes Babel's router. [The interest set](https://github.com/moq-dev/moq/pull/8728) then edits the path router only, and [the route stream](/quest/m2/cluster-routes/route-stream.md) follows the switch with no echo or route request to carry. The switch's recommended wire puts the answer that ends a hold on the publisher half, behind the routes its sender sent first, so the path rule needs no order between streams that today's sessions lack for one prefix. [Acceptance](/quest/m2/cluster-routes/acceptance.md) is of the switched sessions, and the inductive invariant stays in m1, rewritten for the path, without `LoopFree` as an invariant. Rejected: the router quest in m1; one quest for router, wire, draft, sessions, and `js/net`; the route stream first, which builds the echo and ROUTE_REQUEST on the new stream and then deletes them; one wire change for the path rule and the one stream together; moving the inductive invariant to m4; deleting it. Reversed 2026-11-13 (maintainer, /quest-complete): ✅ the inductive invariant is deleted, since Apalache cannot step even one transition of the route model ([moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564)), no Rust TLA+ checker qualifies ([moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600), [moq-dev/moq#8669](https://github.com/moq-dev/moq/pull/8669)), and no more JVM model checking is wanted here (rejected: keeping it paused). +28. ✅ Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel), asked how item 27's removal is split and ordered: **the router first with no wire change, then a switch of the sessions, and the switch lands before the route stream.** [The router](https://github.com/moq-dev/moq/pull/8622) runs the path rule with a route per scope, sans-IO, beside today's router. [The switch](https://github.com/moq-dev/moq/pull/8731) puts it on the lite-07-wip wire, the draft, the Rust sessions, and `js/net`, and deletes Babel's router. [The interest set](https://github.com/moq-dev/moq/pull/8728) then edits the path router only, and [the route stream](/quest/m2/cluster-routes/route-stream.md) follows the switch with no echo or route request to carry. The switch's recommended wire puts the answer that ends a hold on the publisher half, behind the routes its sender sent first, so the path rule needs no order between streams that today's sessions lack for one prefix. [Acceptance](/quest/m2/cluster-routes/acceptance.md) is of the switched sessions, and the inductive invariant stays in m1, rewritten for the path, without `LoopFree` as an invariant. Rejected: the router quest in m1; one quest for router, wire, draft, sessions, and `js/net`; the route stream first, which builds the echo and ROUTE_REQUEST on the new stream and then deletes them; one wire change for the path rule and the one stream together; moving the inductive invariant to m4; deleting it. Reversed 2026-11-13 (maintainer, /quest-complete): ✅ the inductive invariant is deleted, since Apalache cannot step even one transition of the route model ([moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564)), no Rust TLA+ checker qualifies ([moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600), [moq-dev/moq#8669](https://github.com/moq-dev/moq/pull/8669)), and no more JVM model checking is wanted here (rejected: keeping it paused). -29. ✅ Decided 2026-11-12 (maintainer, /quest-audit before the cut), asked how the path rule keeps the `upstream` guarantee, which the prototype left out and [the switch](/quest/m2/cluster-routes/switch.md) had listed for removal with Babel's machinery: **a route keeps one sticky down-only flag beside its path.** A relay sets it on a route it learns over a link marked `upstream`, it is never cleared on a later hop, and a relay never sends a flagged route over an upstream link. The flag is export policy, not loop avoidance, so the path does not replace it: a relay that hears a CDN route from a LAN peer learned it on a link that is not upstream, and only the flag tells it the route came down from an upstream. That case is a local cluster with more than one uplink ([outbound federation](/quest/m3/cluster-routing/federation.md)), where a relay whose own uplink died reaches the CDN through a peer; an edge that dials only its cores needs just the local rule. This is the Only-To-Customer attribute of BGP roles (RFC 9234). The router (`rs/moq-net/src/route/path.rs`, built in [#8622](https://github.com/moq-dev/moq/pull/8622)) carries the flag and its export check, [the switch](/quest/m2/cluster-routes/switch.md) adds them to `RouteHandoff.tla` and keeps the flag on ROUTE_START and ROUTE_UPDATE, and [acceptance](/quest/m2/cluster-routes/acceptance.md)'s two-uplink mesh proves it. Sender side only, as today. Rejected: a mark on each hop of the path, which gives the same answer from more bytes and model state; the local rule with no flag, which either leaks through a LAN peer or gives up the dead-uplink fallback; also refusing a flagged route that arrives from a link whose grant marks the sender as a customer, which can be added later without a wire change. +29. ✅ Decided 2026-11-12 (maintainer, /quest-audit before the cut), asked how the path rule keeps the `upstream` guarantee, which the prototype left out and [the switch](https://github.com/moq-dev/moq/pull/8731) had listed for removal with Babel's machinery: **a route keeps one sticky down-only flag beside its path.** A relay sets it on a route it learns over a link marked `upstream`, it is never cleared on a later hop, and a relay never sends a flagged route over an upstream link. The flag is export policy, not loop avoidance, so the path does not replace it: a relay that hears a CDN route from a LAN peer learned it on a link that is not upstream, and only the flag tells it the route came down from an upstream. That case is a local cluster with more than one uplink ([outbound federation](/quest/m3/cluster-routing/federation.md)), where a relay whose own uplink died reaches the CDN through a peer; an edge that dials only its cores needs just the local rule. This is the Only-To-Customer attribute of BGP roles (RFC 9234). The router (`rs/moq-net/src/route/path.rs`, built in [#8622](https://github.com/moq-dev/moq/pull/8622)) carries the flag and its export check, [the switch](https://github.com/moq-dev/moq/pull/8731) adds them to `RouteHandoff.tla` and keeps the flag on ROUTE_START and ROUTE_UPDATE, and [acceptance](/quest/m2/cluster-routes/acceptance.md)'s two-uplink mesh proves it. Sender side only, as today. Rejected: a mark on each hop of the path, which gives the same answer from more bytes and model state; the local rule with no flag, which either leaks through a LAN peer or gives up the dead-uplink fallback; also refusing a flagged route that arrives from a link whose grant marks the sender as a customer, which can be added later without a wire change. -30. ✅ Decided 2026-11-12 (maintainer, /quest-audit before the cut), after a survey of BGP, EIGRP, and their successors against item 27: most of the path rule already matches that prior art (AS_PATH loop detection, root-cause withdrawal, graceful restart, End-of-RIB, outbound route filtering, and item 29's Only-To-Customer flag). Three gaps are taken, all in [the switch](/quest/m2/cluster-routes/switch.md) and [acceptance](/quest/m2/cluster-routes/acceptance.md): **a dead-hop message on a scoped session applies only to the routes heard on that session and casts no doubt**, since in BGP a neighbour withdraws only what it sent, and a scoped peer (a federated relay, later a browser) could otherwise name a core hop it only read in a path and drop healthy routes outside its scope; **holds are bounded in time**, a neighbour's missing answer counting as nothing after a deadline, and a session whose announce stream has stopped draining is closed, which avoids EIGRP's stuck-in-active and BGP's zombie routes (RFC 9687), and is safe because a hold protects presence, not loop freedom; and **three tests before any design change**: neighbours noticing a dead relay up to the 3 s keep-alive apart (to size the doubt timer, measured in hops of delay today), a three-relay priority dispute wheel (RFC 4264's wedgies, since priority ranks above cost), and a primary that fails and returns. No wire bytes change; the first defines what the dead-hop message means. Rejected: declaring the link relation in SETUP and refusing leaked routes on receipt (RFC 9234), since the accepting side already knows the relation from its grant and it adds wire for a misconfiguration; MRAI and route-flap damping, which exist because BGP delays announcements; link state; per-hop signatures. +30. ✅ Decided 2026-11-12 (maintainer, /quest-audit before the cut), after a survey of BGP, EIGRP, and their successors against item 27: most of the path rule already matches that prior art (AS_PATH loop detection, root-cause withdrawal, graceful restart, End-of-RIB, outbound route filtering, and item 29's Only-To-Customer flag). Three gaps are taken, all in [the switch](https://github.com/moq-dev/moq/pull/8731) and [acceptance](/quest/m2/cluster-routes/acceptance.md): **a dead-hop message on a scoped session applies only to the routes heard on that session and casts no doubt**, since in BGP a neighbour withdraws only what it sent, and a scoped peer (a federated relay, later a browser) could otherwise name a core hop it only read in a path and drop healthy routes outside its scope; **holds are bounded in time**, a neighbour's missing answer counting as nothing after a deadline, and a session whose announce stream has stopped draining is closed, which avoids EIGRP's stuck-in-active and BGP's zombie routes (RFC 9687), and is safe because a hold protects presence, not loop freedom; and **three tests before any design change**: neighbours noticing a dead relay up to the 3 s keep-alive apart (to size the doubt timer, measured in hops of delay today), a three-relay priority dispute wheel (RFC 4264's wedgies, since priority ranks above cost), and a primary that fails and returns. No wire bytes change; the first defines what the dead-hop message means. Rejected: declaring the link relation in SETUP and refusing leaked routes on receipt (RFC 9234), since the accepting side already knows the relation from its grant and it adds wire for a misconfiguration; MRAI and route-flap damping, which exist because BGP delays announcements; link state; per-hop signatures. The route layer's core landed in [moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676): a crate-private, sans-IO `Router` in `rs/moq-net/src/route/`, generic over the neighbor key and the claim attributes, with every holding rule of the -contract below. Each message recomputes only the identity it names. The -handoff fixtures, both sweeps, and the route-layer fixtures of the contract -run against it in `route/tests/`; the forwarding and first-hop selection -fixtures still run only on the reference model in -`rs/moq-net/tests/integration/routing_contracts/`. Nothing drives the router -on the wire yet. The children below carry the rest, in order, and land in +contract below. Each message recomputes only the identity it names. Since +[the switch](https://github.com/moq-dev/moq/pull/8731), every +lite-07 session in Rust and `js/net` drives the path router +(`route::path::Router`, through `route::node::Node`), and Babel's router is +deleted with its distance, seqno requests, and echo. The handoff fixtures, +both sweeps, and the route-layer fixtures of the contract run against it in +`route/tests/`, and real sessions with mocked time in +`tests/integration/path_routes.rs`; the forwarding and first-hop selection +fixtures still run only on the Babel-era reference model in +`rs/moq-net/tests/integration/routing_contracts/`, which +[acceptance](/quest/m2/cluster-routes/acceptance.md) deletes. The children below carry the rest, in order, and land in one release: lite-07 carries the route layer as a whole ([lite-07 finalize](/quest/m2/lite07-finalize.md) waits on this epic). This README holds the routing contract every child implements. @@ -129,38 +134,32 @@ the bullets below are the contract it carries. - The announce cost is the origin's per-prefix seed (`transcode/**` at 10 and `transcode/foobar` at 1 from one node, decided 2026-10-01) and nobody changes it in transit; a path's cost through a route is that seed plus the - route metric. Link changes therefore touch only ROUTE, and feasibility runs - on the route metric alone. This is today's single route cost, which stops + route metric. Link changes therefore touch only ROUTE, and routes rank on + the route metric alone, after drain and priority. This is today's single route cost, which stops accumulating per hop once this lands. - Decided 2026-10-13: the metric is configured additive cost with strictly positive link increments, defaulting to one per link. Refuse zero-cost links; origin prefix seeds may still be zero. Overflow is unreachable, never - saturated, using checked arithmetic. No hop-count component is needed - because each link strictly increases the metric, as Babel's feasibility - requires (RFC 8966 s3.5.2); a zero-cost or saturated link would leave a - route equal to its own feasibility distance (`zero_cost_links_are_refused`, + saturated, using checked arithmetic. No hop-count component is needed: + each link strictly increases the metric, so extending a route never ranks + it better and selection converges, where a zero-cost link would let two + relays prefer each other's paths (`zero_cost_links_are_refused`, `overflow_is_unreachable`). Update `?cost=0` examples with the implementation and report the breaking configuration change. - Decided 2026-10-13: GOAWAY gives the peer lowest preference, below every eligible non-draining peer, and triggers failover. Retain it as a last resort - while connected; do not encode drain by saturating the metric. Feasibility - still gates alternatives, so an infeasible alternative waits for a sequence - refresh while the draining peer keeps serving - (`goaway_fails_over_and_keeps_the_draining_peer_last`). Decided 2026-10-14: - the refresh reaches the node through both peers, and if the draining peer's - arrives first the node selects it at the new seqno, so it stays on the - still-serving draining peer until that session closes and then waits for one - more refresh, which the accepted recovery pause covers - (`goaway_keeps_serving_until_a_refresh_makes_the_alternative_feasible` pins - both orders). Using a draining peer's routes locally without advertising - them was rejected: downstream nodes would lose their only route during the - drain. The path rule carries drain in each offer instead (decided - 2026-11-13, with priority below), and, decided the same day, a relay that - hears GOAWAY sends a DRAIN naming the leaving node in the DEAD message, so - every relay doubts it and moves at once only onto paths that avoid it: a - draining path ranks worst but stays usable, and no request loops through - offers that predate the drain. Existing readers resume only - across a matching explicit epoch; epochless changes preserve Restart. + while connected; do not encode drain by saturating the metric. Using a + draining peer's routes locally without advertising them was rejected: + downstream nodes would lose their only route during the drain. As built + (decided 2026-11-13): a route drains when any session on its path drains, + carried in each offer and ranked before priority and cost, and a relay that + hears GOAWAY sends a DRAIN naming the leaving node in a DEAD to each + neighbour it sent a route through it. Every relay doubts that node and + moves at once only onto paths that avoid it: a draining path ranks worst + but stays usable, and no request loops through offers that predate the + drain (`goaway_moves_a_relay_to_a_path_that_does_not_drain`). Existing + readers resume only across a matching explicit epoch; epochless changes + preserve Restart. - Fresh routing identity per process/tab restart, declared by the origin and shared across its live sessions, decided 2026-10-13. It is unverified. Decided 2026-11-02 (maintainer, /quest-unblock): peer grants are dropped and no key proof is @@ -178,23 +177,17 @@ the bullets below are the contract it carries. one, and a declared identity kept beside the stamp as a loop hint would stay unverified. Switching between paths resumes only across a matching explicit epoch. - Keep administrative identity and broadcast epoch separate. The origin - advances its seqno unprompted when a route link closes: a session whose - peer forwarded it a route, or any session carrying the route of a node - that forwards nothing. Feasibility distances held elsewhere otherwise - starve the old neighbors of an origin that moved to a farther attachment - until a ROUTE_REQUEST reaches it - (`reconnect_without_restart_needs_a_seqno_advance`). A new link lengthens - no path, and a client session that forwards nothing changes none. A random identity alone - does not keep a dead incarnation away. What bounds it is ordered sessions, - state scoped to the session that delivered it, never advertising an - unreachable route, keeping the feasibility distance until every - ROUTE_END it sent is echoed back, and the hops each route carries, which - refuse a copy of a route this node sent once the distance is gone: a - collected incarnation then reappears only from a message already in flight - on a live session that never crossed this node, which delivers the - retraction next - (`collected_incarnation_returns_only_while_its_retraction_is_in_flight`). + Keep administrative identity and broadcast epoch separate. A node names + itself in paths by a hop fresh per process, since its session numbers + restart with it (`Router::new`); the origin's `Config::hop` is both that + hop and its declared identity, so one set by hand must be fresh per + process too. A random identity alone does not keep a dead incarnation + away. What bounds it is ordered sessions, state scoped to the session that + delivered it, never advertising an unreachable route, the DEAD that drops + every route through a closed session's hop, the dead hop each node keeps + while a heard route names it, and the doubt on the far end: a dead + incarnation then reappears only from a message already in flight, which + delivers its withdrawal next. `cluster.id` stops naming a routing identity. - Decided 2026-10-14: every identity a scoped connection sends is qualified: filed under a key the receiver derives, rather than its bare @@ -210,8 +203,7 @@ the bullets below are the contract it carries. forwarding a request into that connection. Claims under it are refused outside the grant's scope. Routes are shared by all of an identity's claims, so otherwise a scoped peer forging an identity, or declaring a - victim's id as its own, captures claims outside its namespace and leaves a - seqno the real origin cannot pass after the forger leaves + victim's id as its own, captures claims outside its namespace (`forged_identity_hijacks_claims_outside_the_forgers_scope`); with it, a forger or a misconfigured uplink only affects its own scope. The `peer` claim no longer decides transit trust. Legitimate scoped delivery survives @@ -233,71 +225,51 @@ the bullets below are the contract it carries. keying a forwarded identity by its bare id and the narrowest scope it crossed, which needs the scope in ROUTE, and accepting growth bounded only by metric overflow. -- Loop freedom is Babel's feasibility condition (RFC 8966) keyed by node: - accept a route if its seqno is newer, or equal with a metric below the - feasibility distance. Only the origin advances its seqno. Retraction is - ROUTE_HOLD or ROUTE_END. The route model found a count to infinity - (2026-11-10): an echo can overtake a route derived from the echoed one, - and even in order a per-neighbour echo misses a derived copy held by a - third relay. Each route therefore names the nodes it crossed, and a node - never selects one that names itself (Plan item 25, - `sweep_no_route_outgrows_a_loop_free_path`). -- Starvation recovery is Babel's seqno request (RFC 8966 s3.8, decided - 2026-10-12). A node whose routes to an origin are all infeasible sends - ROUTE_REQUEST for that origin, with its feasibility seqno plus one, to the - neighbour it would use. A node holding a route at least that new answers - with it, and only the origin advances its seqno to answer. Otherwise it - forwards the request to the neighbour it would use for its own: a - feasible route's next hop, else an infeasible route's, even when that - neighbour asked it. This departs from RFC 8966 s3.8.1.2, which avoids the - requester: nothing cancels a request, so a requester that stopped waiting - stays excluded, and a hop whose only route runs through it strands the - request (`sweep_requests_reach_a_quiet_origin`). Next hops form no loop, - and one pending request per hop stops a bounce. Every node - holding a request, its own or forwarded, keeps one per origin pending, - suppressing duplicates, until it selects a route at least that new or has - no usable route left, and sends it again whenever the neighbour it would - send it to changes. No retry timer and no hop count: a looping copy meets - its own pending entry and stops. Decided 2026-10-16 from the handoff - proof: the earlier sketch (a few timed resends, then wait for the next - infeasible update) leaves a node starved for good behind a quiet origin - when every attempt is dropped by a forwarder whose retraction is still in - flight, and the previous hop then moves to an equal-metric route that - changes nothing it advertises (`a_quiet_origin_stays_starved_after_bounded_retries`). - Recovery is eventual without timers: a drop happens only at a node with no - usable route, whose retraction then changes the previous hop's next hop - and resends the request, and once the topology is quiet the selected next - hops, loop-free under feasibility, carry it to the origin. Feasibility is - never reset to accept a longer route after a retraction or a session replay. -- Feasibility makes a detour longer than the lost route wait for that request - to reach the origin, where today's path vector fails over at once. Redundant - cores are feasible successors and switch immediately - (`redundant_cores_fail_over_without_a_seqno_request`); a local mesh reaching - its other half through the CDN, or a routing node falling back from P2P to - the CDN, starves until the origin answers - (`partitioned_local_mesh_reaches_itself_through_the_cdn`, - `p2p_fallback_is_immediate_for_leaves_and_needs_a_seqno_for_transit`). - Decided 2026-10-13: accept this recovery pause; instant failover is not a - requirement. Measure transport failure detection, additional sequence-refresh - recovery, and total interruption separately. The fixtures' manual origin - bump proves feasibility recovery; the handoff fixtures also run the request - exchange, though not its latency. +- Loop freedom is the path rule (Plan item 27). A route carries its path, + each hop a node and the session it learned the route on, and a node never + uses a route whose path names it. A node that loses a session sends one + DEAD per neighbour naming that hop and listing the routes that moved, and + every node drops every route through the hop, whoever offered it, and keeps + the hop while a heard route names it. The far end of a closed session is + doubted for a second, since a dead relay is reported one neighbour at a + time: no node moves onto a route through it until the timer fires, and the + route a node already uses is exempt. On a scoped session a DEAD applies + only to the routes heard there and casts no doubt (Plan item 30). Selected + routes can loop only while the news that ends a stale route is on its way + (`sweep_no_route_outgrows_a_loop_free_path`). +- Holds replace starvation recovery. A node with no usable route holds its + claims, unavailable, and sends ROUTE_HOLD while the neighbour it learned + the route from holds it on the path this node's route extended, while a + neighbour offers a route that only the doubt keeps it off, or until each + neighbour that learned the route from it has answered, with another route, + a ROUTE_HOLD, or a ROUTE_END. A neighbour that holds under the node answers + with ROUTE_HOLD, so holds never wait on each other in a cycle. A hold lasts + 10 s at most, and a session whose announce stream stops draining for 10 s + is closed (Plan item 30): a hold protects presence, not loop freedom + (`holds_retire_once_nothing_can_recover`, and on real sessions + `a_hold_ends_behind_an_answering_neighbour`, + `a_hold_ends_behind_a_closed_session`, + `a_hold_ends_at_its_deadline_behind_a_silent_neighbour`, and + `a_peer_that_stops_reading_is_closed`). +- A detour that avoids the far end of a closed link is taken at once, and one + through it waits out the doubt + (`a_detour_avoiding_the_far_end_is_taken_at_once`, + `a_detour_through_the_far_end_waits_out_the_doubt`); a relay that dies + reaches every relay as withdrawals, and none moves onto a route through it + early (`a_dead_relay_reaches_every_neighbour_as_withdrawals`). Decided + 2026-10-13: instant failover is not a requirement; measure transport + failure detection, the doubt, and total interruption separately. - Claim freshness follows the handoff contract below, independently of the requested path's forwarding choice. -- Decided 2026-10-13: preserve application priority among feasible transit - routes, then compare additive cost. Feasibility always takes precedence: - a preferred longer route can remain ineligible after a shorter route was - advertised, so session arrival order can affect the eventual selection - (`transit_priority_depends_on_arrival_order`). This limitation is accepted; - priority remains local policy, not a propagated metric tier. Endpoints - that advertise no transit routes hold no transit feasibility distance. - Reversed for the path rule on 2026-11-13 (maintainer, - [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)): with no - feasibility, a local preference lets two relays prefer each other's paths - forever. A route's priority is the lowest link priority on its path, and a - route drains if any session on it drains; drain, then priority, rank before - cost. The switch carries both on the wire. -- Decided 2026-10-14: one selection per identity, keeping the `upstream` +- Decided 2026-10-13: preserve application priority among transit routes, + then compare additive cost. Revised for the path rule on 2026-11-13 + (maintainer, [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)): + a local preference alone lets two relays prefer each other's paths forever, + so a route's priority is the lowest link priority on its path, and a route + drains if any session on it drains; drain, then priority, rank before cost, + and ROUTE_START and ROUTE_UPDATE carry both + (`mutually_preferred_links_settle`). +- Decided 2026-10-14: one selection per identity and prefix, keeping the `upstream` link mark and its down-only bit. A node advertises its best route whether or not anything subscribes. A route learned on an upstream link, or through a neighbor's down-only route, is down-only, and a down-only route is never @@ -325,134 +297,76 @@ the bullets below are the contract it carries. - A plain client (`proxy` off) advertises its own origin and publications only, under the one identity it declares on every live session. Clients receive only authorized, interested claims; opaque ids expose no backbone addresses, but are not themselves authorization. A client - advertises nothing it receives, so it holds no feasibility distance and - falls back from P2P to the CDN at once. It still echoes every ROUTE_END. + advertises nothing it receives. Handoff contract, proven 2026-10-15 by the fixtures in `handoff.rs` against next-hop authority and a snapshot barrier alone. A next-hop change adopts the newest knowledge any session has, never an older complete snapshot, and route loss never reports an END: -- Each origin numbers the changes to its own claims with a revision that - starts at zero for each routing identity, separate from the route seqno. - Both are varints. An origin never advertises the last value, 2^64-1 (the - largest lite-07 varint): a - seqno advance, claim change, or seqno request that would reach it rotates - the identity instead, ending the old one everywhere and announcing the - claims again under a fresh one, so nothing wraps and every advertised - seqno leaves room for the successor a request names - (`exhausted_counters_rotate_the_identity`). +- Each origin numbers the changes to its own claims with a revision, a + varint that starts at zero for each routing identity. An origin never + advertises the last value, 2^64-1 (the largest lite-07 varint): a claim + change that would reach it rotates the identity instead, ending the old one + everywhere and announcing the claims again under a fresh one, so nothing + wraps (`exhausted_counters_rotate_the_identity`). A session sends an origin's announcements in batches, each closed by its floor: the revision at which the stream's announcements for that route, - across the whole interest, are the origin's, or nothing when the sender - does not know them across the interest at one revision (Plan item 10). A - qualified key is known only within the grant scopes it arrived through, - and one key can arrive through grants whose pattern unions differ, so the - view inside a node carries one floor per scope, and a stream whose - interest is wider than those scopes certifies nothing. -- A receiver keeps each session's last view, to apply the next batch, and - merges every session's view into its own claim by claim: whichever view - covers the claim with the newer floor decides. A floor certifies only the - claims overlapping its prefix: those inside it and wildcards above it, not - an exact claim above it - (`only_a_wildcard_claim_overlaps_an_interest_below_it`). A node - keeps floors only within its current interest, dropping any that a wider, - newer floor covers. An older complete snapshot is therefore ignored, where - next-hop authority and a barrier both revive the withdrawal - (`an_older_complete_snapshot_revives_a_known_withdrawal`); a new next hop - whose ROUTE precedes its announcements changes nothing the reader sees - (`next_hop_authority_reports_false_absence_while_announcements_trail`); - widening keeps a narrower, newer floor yet still accepts an older, wider - snapshot for what it newly covers, where one floor per origin either - refuses that snapshot or revives under it - (`widening_interest_needs_floors_scoped_to_what_they_cover`). An interest - change opens a new stream: batches still in flight on the old one are - discarded, and each neighbour starts the new one with a complete - snapshot. The first batch answering the new interest retires whatever - lies outside it, so state follows the current interest, not every prefix - ever watched; a rejoin then replays a lagging relay's view, as after a - collection, only until the newer snapshot behind it lands - (`cancelling_interest_retires_its_region`). That batch re-certifies any - wildcard above the interest, so retiring never ends a claim still - watched. Until it arrives, a widened node hides what it has not retired - yet, rather than showing a claim that ended unwatched - (`widening_hides_what_was_frozen_until_a_covering_floor_arrives`); a root - snapshot still queued on the old stream does not count - (`a_snapshot_queued_before_widening_keeps_the_frozen_rest_hidden`), and - a batch cut to a scoped grant does - (`widening_behind_a_scoped_grant_shows_the_rest_of_the_scope`). A batch - certifies the interest only within its session's grant scope, so a view - merged from several grants stays hidden in each scope until that grant - answers (`widening_behind_two_grants_waits_for_each_scope`). A floor - outside the interest stays while it is newer than every floor inside, - since it may certify a wildcard above both at a later revision; and - withdrawing one origin's wildcard leaves another origin's copy - (`wildcard_claims_end_per_origin_and_are_certified_below`). -- Not yet on the wire ([the router](https://github.com/moq-dev/moq/pull/8622) - built it and [the switch](/quest/m2/cluster-routes/switch.md) puts - it on the wire; the rest of this contract describes today's router until - then), - settled by `rs/moq-net/verify/route/RouteHandoff.tla` with Plan item 27's - rule: a route is keyed by its publisher and the prefix it covers, and - carries its own path. A node that asks a neighbour for a narrower prefix + across the route's prefix, are the origin's, or nothing when the sender + does not know them across that prefix at one revision (Plan item 10). +- As built, settled by `rs/moq-net/verify/route/RouteHandoff.tla` with Plan + item 27's rule: a route is keyed by its publisher and the prefix it covers, + and carries its own path. A node that asks a neighbour for a narrower prefix gets one route there, covering that prefix: the neighbour's live route for that prefix if it has one, else its route for a prefix above it. A node - believes a route whose path does not name it, and keeps one view per - prefix: the newest floor any route covering that prefix gave it, kept - whole while it holds a route covering the prefix, so a lagging route - cannot revive a withdrawal another delivered, and a route that ends takes - only what no other route covers. A claim shows from the narrowest prefix's - view and is available while a live route covers it. A request starts on - the narrowest live route covering its claim, follows the route each node + believes a route that is live, whose path does not name it, and whose + stream answers its current interest, and keeps one view per prefix: the + newest floor any believed route covering that prefix gave it, kept whole + while it holds a route covering the prefix, so a lagging route cannot + revive a withdrawal another delivered, and a route that ends takes only + what no other route covers. While no believed route certifies the prefix, + the view shows what the selected route carries there, uncertified. A claim + shows from the narrowest prefix's view and is available while a live route + covers it. A request starts on the narrowest live route covering its claim, + names its publisher and that route's depth, follows the route each node offered the one it came from, and is never forwarded along a route whose path names the node it came from. A node remembers a dead hop while a neighbour's route names it and until the timer the hop's close started fires, so no route through a hop it knows is dead returns. Selected routes can loop only while the news that ends a stale route is on its way, and - only through at least one stale route. What its timer costs fired early, and the three-failure loop regression `Loop.cfg` pins, are in the `verify-route` skill. - A sender sends a route on a stream only while the route meets the - stream's interest (Plan item 15), and a node presents each stream only the - claims that changed. -- Per identity, a node is usable (a feasible selected route: announcements - shown available), held, or let go. It is held while starved, or while a - neighbor's held route has a last advertisement feasible here; it - keeps showing the announcements as unavailable and sends ROUTE_HOLD. Holds therefore follow strictly decreasing metrics - down to a starved node and cannot prop each other up. Otherwise the node - lets go: ROUTE_END to every neighbor it advertised to, and its reader sees - the announcements go offline, a distinct event from an END the origin - sent; while held, the reader sees them unavailable. The - detour window keeps presence at the starved relay and behind it - (`a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it`); - a dead origin or a total partition goes offline everywhere with no timer - (`holds_retire_once_nothing_can_recover`). No promise is made of learning a - withdrawal during a partition. + only through at least one stale route. What its timer costs fired early, + and the three-failure loop regression `Loop.cfg` pins, are in the + `verify-route` skill. A sender sends a route on a stream only while the + route meets the stream's interest (Plan item 15), and a node presents each + stream only the claims that changed. An interest change opens a new + stream, and the first batch answering it retires whatever lies outside it + (`cancelling_interest_retires_its_region`), until + [the route stream](/quest/m2/cluster-routes/route-stream.md) carries the + router's edits in place (Plan item 22). +- Per identity and prefix, a node is usable (a live selected route: + announcements shown available), held, or let go. It holds as the path rule + says (above), showing the announcements as unavailable and sending + ROUTE_HOLD. Otherwise the node lets go: ROUTE_END to every neighbour it sent + the route to, and its reader sees the announcements go offline, a distinct + event from an END the origin sent; while held, the reader sees them + unavailable. A dead origin or a total partition goes offline everywhere + within the hold's bound (`holds_retire_once_nothing_can_recover`). No + promise is made of learning a withdrawal during a partition. - A node may collect an identity's views and floors once it holds no route - entry for it; correctness does not depend on when. It drops the - feasibility distance once it also advertises the identity nowhere and - every neighbour has echoed each ROUTE_END it sent, or closed. Until then - an advertisement derived from its own can still wait on some session, and - accepting it loops; once all echoes are back no neighbour routes through - it (`collection_keeps_the_feasibility_distance_against_a_queued_echo`), - and a copy a third node still holds names this node in its hops, so it is - refused when it returns - (`a_route_handed_round_a_triangle_is_refused_where_it_began`). The - echo is per neighbour, not a diffusing computation: every other node - keeps its own distance. A stale route accepted after the drain dead-ends at - a node that let go, like any collected incarnation. - After collection, or a process restart, a stale view returns only from a - sender that has not yet received the withdrawal, and only until it does - (`stale_replay_after_collection_lasts_only_until_the_withdrawal_behind_it`). - Retained state is each session's last view, one merged view per held - identity with floors only within the current interest (and any newer one - certifying a wildcard above it), one pending - request per origin, and a feasibility distance only while an identity is - routed, advertised, or draining; neither withdrawal history nor interest - history leaves anything. One claim change costs one + for it; correctness does not depend on when. A dead hop stays while a heard + route names it and until its timer fires, and a doubt until its timer + fires; nothing else outlives the routes. After collection, or a process + restart, a stale view returns only from a sender that has not yet received + the withdrawal, and only until it does. Retained state is each session's + routes and their last views, one view per held identity and prefix within + the current interest, and those dead hops and doubts; neither withdrawal + history nor interest history leaves anything. One claim change costs one batch per session carrying the route and one event per watching node (`announcement_cost_follows_sessions_and_live_claims`). Without split horizon a relay echoes an origin's claims back to it. - A fresh identity per restart also starts a fresh revision space; reusing - an identity would need its revision persisted along with its seqno + an identity would need its revision persisted, and its hop's sessions + numbered past the old process's (`origin_restart_takes_a_fresh_identity_for_its_claims`). - Presence and availability are separate. A forwarded subscription follows only a usable route, so a held route ends it as unreachability does while @@ -468,7 +382,8 @@ loss never reports an END: and once churn stops every node shows each live origin's current claims and holds nothing for a dead one. `sweep_scoped_handoff` churns a client behind two scoped grants across interests inside, above, and outside both - scopes, with no false END. Over `sweep_handoff`'s scenarios: + scopes, with no false END. Over `sweep_handoff`'s scenarios, measured on + Babel's router on 2026-10-16: | Rule | Revivals | False ENDs | Re-announcements | Reader events | Claim KiB | | --- | --- | --- | --- | --- | --- | @@ -498,7 +413,7 @@ Decided with the maintainer 2026-10-16, from the handoff proof: when a held route recovers; let-go reported as END, as today; and availability with let-go kept as END, which hides whether the origin ended or the network lost it. -4. Collection: ✅ collect routes and announcements once no neighbor's route +4. ~~Collection: ✅ collect routes and announcements once no neighbor's route entry remains, held entries included. `sweep_handoff` checks selected routes stay loop-free at every step; collecting while a neighbor's entry remains loops at once. Rejected: Babel's source GC timer. @@ -506,14 +421,16 @@ Decided with the maintainer 2026-10-16, from the handoff proof: the node's own advertisement has drained from every session, and accepting that echo loops (`collection_keeps_the_feasibility_distance_against_a_queued_echo`), so - decision 6 drains the feasibility distance instead. -5. Seqno requests: ✅ one pending request per origin at every hop, resent on + decision 6 drains the feasibility distance instead.~~ + Superseded by Plan item 27 ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)): with no distance, a node collects an identity once it holds no route for it, and keeps only dead hops and doubts past their routes. +5. ~~Seqno requests: ✅ one pending request per origin at every hop, resent on a next-hop change, with no timer and no hop count. Rejected: the bounded-retry sketch, which strands a quiet origin's node. Revised 2026-10-16 after review: a hop forwards to its next hop even when that neighbour asked it, since excluding requesters strands requests - (`sweep_requests_reach_a_quiet_origin`). -6. Feasibility-distance retention: ✅ drain it. A receiver echoes every + (`sweep_requests_reach_a_quiet_origin`).~~ + Superseded by Plan item 27 ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)): there is no seqno or request; holds and the doubt cover recovery. +6. ~~Feasibility-distance retention: ✅ drain it. A receiver echoes every ROUTE_END back on the same stream once applied, and a node drops the distance once it routes and advertises the identity nowhere and every echo is back, a closed session counting as one. Reusing ROUTE_END adds no @@ -536,8 +453,9 @@ Decided with the maintainer 2026-10-16, from the handoff proof: (`sweep_no_route_outgrows_a_loop_free_path`). Rejected: keeping the distance forever, which grows with publisher churn times uptime on every relay that saw an identity, about 16 bytes each, until a restart; and Babel's source GC - timer, which assumes stale echoes are gone by then rather than proving it. -7. Scope coverage under one key: ✅ per-region floors (superseded on the + timer, which assumes stale echoes are gone by then rather than proving it.~~ + Superseded by Plan item 27 ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)): there is no distance to retain and no echo; the path's self-naming rule from item 25 stays. +7. ~~Scope coverage under one key: ✅ per-region floors (superseded on the wire 2026-11-05 by Plan item 10; the core keeps regions internally). ANNOUNCE_FLOOR names the region it certifies, a suffix of the stream's interest prefix (the scope met with the interest), each node tracks which regions each @@ -549,8 +467,8 @@ Decided with the maintainer 2026-10-16, from the handoff proof: union per key, which refuses or re-files a refresh that changes the union and cannot be checked across relays; and deferring to the implementation, which leaves the proof's freshness claim false for scoped - delivery. - + delivery.~~ + Superseded by Plan item 27 ([moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)): a route covers a prefix, and a node keeps one view per prefix from the newest floor any believed route covering it gave. 8. Region-set framing (superseded 2026-11-05 by Plan item 10, a whole-interest marker): ✅ one ANNOUNCE_FLOOR per batch lists every region the sender certifies for the route: (Route ID, count, then a region suffix @@ -585,7 +503,9 @@ benchmarks. - Starvation: a five-node ring loses its preferred origin link while a working four-hop backup remains. The backup's same-seqno route stays infeasible for good, and restarting the backup's session does not help; - only an origin seqno advance does. Hence the seqno request above. + only an origin seqno advance does. Hence Babel's seqno request, which + Plan item 27 removed with the distance: the path rule takes the backup once + the doubt on the origin, the closed link's far end, lifts. - Ending a broadcast on the 45-node topology costs 124.8 KiB against path vector's 2817.0 KiB (1304.0 KiB with #4399's withdrawal hiding), but still 2000 client updates against a 900 once-only minimum (up to 11 per node diff --git a/quest/m2/cluster-routes/acceptance.md b/quest/m2/cluster-routes/acceptance.md index 9460a5e4c5c..5923f82007d 100644 --- a/quest/m2/cluster-routes/acceptance.md +++ b/quest/m2/cluster-routes/acceptance.md @@ -22,7 +22,7 @@ lite-07-wip before it freezes. Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel): acceptance is of the path rule ([epic](/quest/m2/cluster-routes/README.md) Plan item 27), on the -sessions [the switch](/quest/m2/cluster-routes/switch.md) leaves. +sessions [the switch](https://github.com/moq-dev/moq/pull/8731) left. The seqno request fixtures are replaced below, since the rule has no seqno and no request. @@ -113,10 +113,6 @@ keep-alives must shorten. Three relays whose local priorities form a dispute wheel settle, or the test pins how they fail (RFC 4264). A primary route fails and returns, and every relay ends back on it. -## Required - -- [lite-07 sessions run the path router](/quest/m2/cluster-routes/switch.md) - the sessions and the route rule these fixtures accept - ## Related - [Routes and announces](/quest/m2/cluster-routes/README.md) - the contract and its fixtures diff --git a/quest/m2/cluster-routes/flapping-publisher.md b/quest/m2/cluster-routes/flapping-publisher.md index 960b68df961..43a34e0127c 100644 --- a/quest/m2/cluster-routes/flapping-publisher.md +++ b/quest/m2/cluster-routes/flapping-publisher.md @@ -58,8 +58,7 @@ Work: - Re-run #8622's reconnect and churn measurements under the new rule and report reachable-but-unrouted observations, stale routes, and loops beside timer-only doubt. -- Update the epic's Plan item 27, the draft's Routing section if - [the switch](/quest/m2/cluster-routes/switch.md) has landed, and the +- Update the epic's Plan item 27, the draft's Routing section, and the `verify-route` skill. ## Related diff --git a/quest/m2/cluster-routes/route-stream.md b/quest/m2/cluster-routes/route-stream.md index 7d2b81e7faf..58c367f6e31 100644 --- a/quest/m2/cluster-routes/route-stream.md +++ b/quest/m2/cluster-routes/route-stream.md @@ -32,7 +32,7 @@ session. This removes the ordering hazard as a class and makes a token-scope change an edit. Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for -Babel): ✅ [the switch](/quest/m2/cluster-routes/switch.md) to the +Babel): ✅ [the switch](https://github.com/moq-dev/moq/pull/8731) to the path rule lands first, on today's streams, and this follows (rejected: this first, which builds the echo and ROUTE_REQUEST on the new stream and then deletes them; one wire change for both, which joins two XL quests). The path @@ -63,6 +63,45 @@ Settled: reopened requests that could hold three times the announce cap ([moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375)'s open review finding) cannot occur. +4. ✅ **A DEAD has a fixed size, and the routes it moves follow it as its own + ROUTE_UPDATEs** (decided 2026-11-14 by the maintainer on + [moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)'s decision + 22, [the comment](https://github.com/moq-dev/moq/pull/8731#issuecomment-6889336680)'s + B; rejected: A, the parts with a More byte that the switch ships, where + each part repeats the header and the sender splits by encoded size; C, a + DEAD that holds every route through the hop with the moves after it as + ordinary messages, which brings churn on every failure that moves routes + and changes the model). A DEAD Closed is `Closed {Hop, Far, Gone, Moved + Count}`, and the next Moved Count messages on the stream are ROUTE_UPDATEs, + one per moved route. The receiver holds the DEAD until they arrive and + then applies everything as one step, the route model's `Fail`, so + `RouteHandoff.tla` and its configs are unchanged. A refused move is a + refused ROUTE_UPDATE: the route ends, and its path still credits the DEAD. + These are PROTOCOL_VIOLATIONs, so a DEAD always completes or ends the + session: + - any other message between the DEAD and its last update, whichever + prefix it answers; + - an update for a route that isn't live or doesn't cross the hop; + - a count above the stream's routes through the hop. + + It replaces the parts in the same PR: + - the DEAD codec (`rs/moq-net/src/lite/announce.rs`, + `js/net/src/lite/announce.ts`): Moved Count in place of More and the + moved list, and `Dead::split` goes; + - `buffer_dead` (`rs/moq-net/src/lite/publisher/routed.rs`) writes the + DEAD and then its updates, with nothing between them and no size split; + - both receivers' reassembly (`Continued` in + `rs/moq-net/src/lite/subscriber/routed.rs`, `continued` in + `js/net/src/lite/subscriber.ts`) counts the updates instead of the + parts; + - the shared corpus (`rs/moq-net/fixtures/lite.json` and the fuzz seeds in + `rs/moq-net/src/fuzz.rs`) drops the More cases and pins a DEAD with its + updates and each violation; + - the draft's DEAD section and Dead Hops describe the count. + + Test that a DEAD and its updates apply as one, in Rust and `js/net`, each + failing when an update applies on its own, and that each violation + closes the session. Recommended wire, for the implementer to confirm or improve in the PR; each departure is a choice to list there: @@ -77,7 +116,7 @@ departure is a choice to list there: sends as a publisher: one ANNOUNCE_OK per edit it applied, the marker where the edit takes effect, then ROUTE_START, UPDATE, HOLD, and END, the withdrawal that names a dead hop and the hold answer as - [the switch](/quest/m2/cluster-routes/switch.md) shaped them, the + [the switch](https://github.com/moq-dev/moq/pull/8731) shaped them, the ANNOUNCE messages, ANNOUNCE_FLOOR with a Prefix ID, and SYNCED with a Prefix ID. ANNOUNCE_START names the Prefix ID its suffix is relative to, and a compression base shares it. @@ -109,11 +148,48 @@ Code this replaces, for lite-07 only: - `Routed` (`lite/subscriber/routed.rs`) and `Answer` (`lite/publisher/routed.rs`) become one per session, holding the prefixes. `Answer` already reads several node streams (one per mount). -- `route::Node` registers one neighbour per session for both directions. Today `receive` and `answer` each take a fresh key (`route/node.rs`), so one session is two neighbours in the router; the interest set makes the router accept one key for both and this makes the sessions use it. +- `Router::admit` (`rs/moq-net/src/route/path.rs`) files a scoped + neighbour's route that is wider than the scope's regions at each region + inside it, because a stream for a scope with several regions asks for a + prefix above them all ([the switch](https://github.com/moq-dev/moq/pull/8731) + added this). Two routes of one node at nested prefixes then both file at + a region inside both, so one overwrites the other and an end of either + drops the route there until the other sends again. With one prefix per + region in the set, the peer answers at the region and nothing narrows, so + the narrowing goes. Test a two-region scope behind a peer that holds a + broad and a narrow route of one node, ending each in turn: the region + inside both stays online through whichever route still lives, and goes + offline only once both have ended. Decided 2026-11-14 (maintainer, on + [moq-dev/moq#8731](https://github.com/moq-dev/moq/pull/8731)'s decision + 12): ✅ deferred here (rejected: tracking per filed slot which wire routes + file into it in #8731, state this quest deletes). +- `route::Node::closing` and the quiet retire of a stream dropped outside + it (`route/node.rs`), for lite-07: with each side's stream living as long + as its session, every drop of one is a session close. Today a session's + receive stream lands under the guard only because `Publisher::poll` + returns the transport error before `poll_protocol` reaches the + subscriber, which [#8731](https://github.com/moq-dev/moq/pull/8731)'s + review flagged as resting on poll order. A guard taken on every poll + would lock the node on each session wake, so the stream's lifetime is + the fix. +- `route::Node` registers one neighbour per session for both directions, + carrying [the interest set](https://github.com/moq-dev/moq/pull/8728)'s + edits on it: `Out::Ask`, `Out::Edited` and `edited`, and SYNCED per + prefix. Today `receive` and `answer` each take a fresh key + (`route/node.rs`), so one session is two neighbours in the router, and + each key's one edit is the prefix its stream opened with: + `State::register` applies it at once and `flush` drops the router's asks + and markers. A scoped peer's route wider than its regions is filed at each + region inside it (`path::Router::admit`) rather than asked for per region. - `js/net`: each `routes()` call opens its own stream today (`js/net/src/lite/subscriber.ts`), so JS gains the covering-prefix logic `Discovery` has. Its route layer keeps per-stream state, so this PR changes `js/net` - too. + too. A floor that changes what a claim covers, with no route change, also + re-runs admission for the open requests it affects: `forwardAnnounced` + (`js/net/src/connection/forward.ts`) asks `session.covers` only when a + request is admitted, so a request can stay on a node that no longer + covers it until something else refreshes it (flagged in + [#8731](https://github.com/moq-dev/moq/pull/8731)'s review). Also touching the same files, to merge around: [moq-dev/moq#8458](https://github.com/moq-dev/moq/pull/8458) (its @@ -141,10 +217,6 @@ earlier lite-07 wire change did. Public API: none expected. Wire: lite-07-wip only. -## Required - -- [lite-07 sessions run the path router](/quest/m2/cluster-routes/switch.md) - the messages this moves onto one stream, with no echo or route request among them - ## Related - [Announce interest streams share one route table](/quest/m4/announce-interest-memory.md) - lite-07 then holds one interest stream per side; the bound still matters for older versions diff --git a/quest/m2/cluster-routes/switch.md b/quest/m2/cluster-routes/switch.md deleted file mode 100644 index 21b98323a90..00000000000 --- a/quest/m2/cluster-routes/switch.md +++ /dev/null @@ -1,173 +0,0 @@ -# [XL] lite-07 sessions run the path router - -## Goal - -Every lite-07 session, in Rust and in `js/net`, routes by the rule of the -[epic](/quest/m2/cluster-routes/README.md)'s Plan item 27, on the -router [the route per scope](https://github.com/moq-dev/moq/pull/8622) -built (`route::path::Router`). A route on the wire carries its path with a session in each hop and -the prefix it covers. A relay that loses a session names that hop in one -message, and a relay answers a neighbour that holds for it. Babel's -distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo are gone from -lite-07-wip, the draft, both codecs, and the tree: today's router is -deleted, and one route rule remains. - -A detour no longer waits on the publisher, a relay that dies costs one -withdrawal per session in place of a seqno round, and the draft and the -epic's contract say what the sessions do. lite-06 and older and -moq-transport are untouched. - -## Plan - -Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for -Babel): - -- ✅ The removal is two quests in this epic: the router first with no wire - change, then this switch (rejected: the router quest in m1; one quest for - everything). -- ✅ The switch lands on today's streams, before - [the route stream](/quest/m2/cluster-routes/route-stream.md) - (rejected: the route stream first, which builds the echo and ROUTE_REQUEST - on the new stream and then deletes them; one wire change for both, which - joins two XL quests and waits on the interest set). The path rule sends no - routing message on the subscriber half, so the ordering the route stream - was planned to give the echo is not needed here. - -Decided 2026-11-13 (maintainer, -[moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)): - -- ✅ ROUTE_START and ROUTE_UPDATE carry the route's drain bit, set when any - session on its path drains after GOAWAY, and its priority, the lowest link - priority on its path. Both rank before the metric, as `path::Offer` - carries them (rejected: drain and priority as tie-breakers below cost; a - drain cost penalty that drops priority). The router sends an update when - either changes, and a DEAD's moved routes carry them too. -- ✅ The message that carries DEAD gains a DRAIN entry naming a node that - sent GOAWAY (decided 2026-11-13, maintainer, /quest-complete). The relay - that hears GOAWAY sends it to each neighbour it sent a route through that - node, and each relay passes it on the same way, once per neighbour while - such a route stays sent. Every relay ranks a path through the node as - draining, the worst rank, but keeps it usable and never removes it. It - doubts the node until the existing timer fires, so it moves at once only - onto paths that avoid the node, and a request never loops through offers - that predate the drain. A relay accepts a DRAIN only for a node on a path - that neighbour offered. Rejected: taking drain out of the rank, so routes - move only when the session closes; accepting brief request loops. - -Recommended wire, for the implementer to confirm or improve in the PR; each -departure is a choice to list there. Where the model settled something -else, the model wins: - -- ROUTE_START and ROUTE_UPDATE carry the path as hops, each a Node ID and - the session that node learned the route on, and the prefix the route - covers as the model settled it. The seqno goes. The down-only flag stays as - a route attribute beside the path (epic Plan item 29, decided 2026-11-12: - set when a route is learned over an `upstream` link, never cleared, never - sent over one), which amends this quest's first draft, where it went. The - metric stays as the cost that ranks routes, and no longer decides which - are safe. The hop limit stays the wire's 32. -- One message names a dead hop. A relay sends it once per session it loses, - to every neighbour, covering every publisher, and it lists the routes that - moved to another path, as the prototype has it. The same message carries DRAIN - entries, each naming a node that sent GOAWAY, covering every publisher. On a - scoped session it applies only to the routes heard on that session and casts - no doubt (epic Plan item 30), so a peer cannot drop routes it only read in a - path. -- The answer that ends a hold rides the publisher half: the stream that - carries the answering side's routes to the relay that waits. It is then - ordered behind every route its sender sent first, on today's streams. A - relay waits only on a neighbour it listens to, since no other can hand a - stale route back. A neighbour that has not answered by a deadline counts - as answering with nothing, and a session whose announce stream has stopped - draining for a set time is closed (epic Plan item 30); a hold protects - presence, not loop freedom, so either is safe. -- ROUTE_HOLD stays as the model has it (epic Plan items 14 and 19). -- `RouteHandoff.tla` gains the down-only flag and its export rule (epic Plan - item 29), which the router already has, so the model checks the rule the - wire carries. -- ROUTE_REQUEST and the echo are removed with their types. The Node ID a - request names (epic Plan item 2) stays. -- The doubt is local: nothing on the wire carries it. Its timer is the - router's, settled by the model and the sweeps from the maintainer's 1 s. - -Work: - -- `route/node.rs` drives the new router. `home` serves a request through - the route the rule picks, in place of every feasible route whose neighbour - `carries` the claim. - Honor `path::Router::new`'s fresh-hop-per-process contract, including when - `Config::hop` is configured: the old seqno cannot recover reused path-hop names. - The driver also implements the path router's interest set - ([the interest set](https://github.com/moq-dev/moq/pull/8728)): `watch` takes a set of prefixes, the router asks - each neighbour for one prefix per scope region it meets (`Out::Ask`), and the - driver carries each edit, its marker (`Out::Edited`, `edited`), and SYNCED - per prefix (`Out::Synced`, `synced`) under one neighbour key, completing - cancelled streams locally when no replacement can answer. Until the route - stream lands, the session layer may still open one announce stream per - prefix. -- Delete today's router with what only it used: the feasibility distance and - its drain, seqno requests and their pending state, the echo, and their - fixtures. A fixture that states a claim rule moves to the new router if - [the route per scope](https://github.com/moq-dev/moq/pull/8622) did not - move it already. -- The Rust codec and sessions (`rs/moq-net/src/lite/`), and `js/net`'s route - layer (`js/net/src/lite/publisher.ts`, `js/net/src/lite/subscriber.ts`, - `js/net/src/lite/floors.ts`, `js/net/src/route.ts`) in the same PR. A - browser is an endpoint: it sends its own route, refuses a route whose path - names it, and applies a withdrawal that names a hop. -- Make js/net's origin apply the covering rule (`covering` in - `route/view.rs`) when it picks among a session's routes, as its consume - cache does through `Floors.covers`. Decided 2026-11-12 (maintainer, on - [moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443)): ✅ leave it to - this rewrite (rejected: port the per-node merged view in #8443). Until then, - when a newer floor withdraws a node's claim above `live/x` while another - node claims `live` with a different instance, `forwardAnnounced` still - offers the withdrawn claim, and an origin request for `live/x` fails - `Unroutable` until the lagging stream's floor lands. Recommended: the - origin asks a session's route, per request, whether it covers the - requested path (the session's `Floors.covers`) and skips one that does not, - since an announcement at `live` cannot say it stops short of `live/x`. - Test it with the two-stream, two-instance case from #8443's review. -- [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)'s warm-route - sweep (`route/tests/cache.rs`) and `Router::eligible` rest on the - feasibility distance. - [moq-dev/moq#8732](https://github.com/moq-dev/moq/pull/8732) rewrites the - sweep on `route::path::Router`; if it has not landed, move the sweep onto the - new router's query here and say what it no longer checks. -- Draft: Routing states the path rule, the withdrawal, holds, and the doubt. - Floors says a floor certifies its route's prefix, and the condition "every - peer whose last advertised route to it is feasible here has answered the - current request" becomes the settled rule. Remove ROUTE_REQUEST and the - echo from the message sections and the stream tables, and add a changelog - bullet. -- Rewrite the epic's contract as built, mark its Plan items 3, 7, 20, and - the echo and drain of item 25 as superseded by item 27, and update the - `verify-route` skill's rules, coverage, and sync commit. -- Tests on real sessions with mocked time: a detour that avoids the far end - of a closed link is taken at once, and one through it after the doubt; a - relay with several neighbours that dies reaches every relay as withdrawals - and no relay moves onto a route through it early; a hold ends behind a - neighbour that answers and behind one whose session closes; an unknown - hop named in a withdrawal changes nothing; a path at the hop limit is - refused; a scoped peer's dead-hop message leaves routes heard on other - sessions alone; a hold ends at its deadline behind a neighbour that never - answers; and a peer that stops reading its announce stream is closed. -- Measure route and claim messages per change with - `rs/moq-net/tests/integration/mesh_sweep.rs`'s benchmark before and after, - and put both in the PR. -- Run `just check`, `just test interop --all`, and `just drafts check`. - -Rollout: lite-07 is `-wip`, so a relay on this wire refuses one on the old -by misparse, not by version. Relays pinned to `main` roll together, as each -earlier lite-07 wire change did. - -A relay refuses a route its neighbour only forwarded by withdrawing that -route, not by closing the session (the draft's Refused Content section). The -rewritten codec and the path router keep that rule. - -Public API: none expected. Wire: lite-07-wip only. - -## Related - -- [Each side says everything about routing on one stream](/quest/m2/cluster-routes/route-stream.md) - moves these messages onto one stream per side afterwards -- [Route layer acceptance](/quest/m2/cluster-routes/acceptance.md) - proves the switched sessions end to end diff --git a/quest/m3/cluster-routing/README.md b/quest/m3/cluster-routing/README.md index aa41bb6423d..1090caaa9f9 100644 --- a/quest/m3/cluster-routing/README.md +++ b/quest/m3/cluster-routing/README.md @@ -56,10 +56,10 @@ Decided with the maintainer, 2026-10-13: kept free of loops by its path, a withdrawal that names the hop that died, holds, and a timed doubt ([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27). A - detour no longer waits on the publisher, and today's router keeps Babel - until [the switch](/quest/m2/cluster-routes/switch.md) lands. From - then, "feasible" in this Plan reads as: the route's path does not name the - relay and no doubt keeps the relay off it. + detour no longer waits on the publisher, and + [the switch](https://github.com/moq-dev/moq/pull/8731) deleted Babel's router. "Feasible" in this Plan + reads as: the route's path does not name the relay and no doubt keeps the + relay off it. - An origin declares its own routing identity, fresh per process or tab restart and shared across its live sessions. It is unverified. Decided 2026-11-02 (maintainer, /quest-unblock): no key proof is planned; if a use case needs a provable identity, moq-lite-07's diff --git a/quest/m3/cluster-routing/request-loop.md b/quest/m3/cluster-routing/request-loop.md index 6a0ffd99d27..b50a15f1fdb 100644 --- a/quest/m3/cluster-routing/request-loop.md +++ b/quest/m3/cluster-routing/request-loop.md @@ -42,10 +42,6 @@ and one still unanswered when it forms. Check which relay state carries the pending request (the front in `rs/moq-net/src/model/front.rs`, and how `route/node.rs` `home`s it) before choosing where the fix goes. -## Required - -- [lite-07 sessions run the path router](/quest/m2/cluster-routes/switch.md) - the sessions whose next hops can loop - ## Related - [A dual-homed publisher stays reachable while one link flaps](/quest/m2/cluster-routes/flapping-publisher.md) - another failure mode from the same BGP comparison diff --git a/quest/m3/route-split-horizon.md b/quest/m3/route-split-horizon.md index b548521e65a..bc2e3fd269b 100644 --- a/quest/m3/route-split-horizon.md +++ b/quest/m3/route-split-horizon.md @@ -6,9 +6,7 @@ A relay never advertises a route back over the session it learned it from. Since [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), each neighbor holds two stream keys, so the router advertises a route back to its source. That is loop-free, since the source refuses a route whose path names -it ([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27; -by the feasibility condition until -[the switch](/quest/m2/cluster-routes/switch.md) lands), but it +it ([routes epic](/quest/m2/cluster-routes/README.md) Plan item 27), but it doubles route traffic between every pair of relays. ## Plan diff --git a/rs/moq-net/fixtures/lite.json b/rs/moq-net/fixtures/lite.json index 6e97f943b2d..70177e3067a 100644 --- a/rs/moq-net/fixtures/lite.json +++ b/rs/moq-net/fixtures/lite.json @@ -42,68 +42,103 @@ "expect": "refused" }, { - "name": "a ROUTE_START names its origin node, seqno, metric, flags, and hops", + "name": "a Route Offer counting more hops than its body holds is malformed, though a count past the hop limit is otherwise refused as content", "version": "moq-lite-07-wip", "message": "announce", - "hex": "06089234030501020709", + "hex": "0649070083e80100020003000400050006000700080009000a000b000c000d000e000f0010001100120013001400150016001700180019001a001b001c001d001e001f0020002100008001", + "expect": "refused" + }, + { + "name": "a ROUTE_START names its publisher, suffix, and offer: a path of node and session hops, then flags, priority, and metric", + "version": "moq-lite-07-wip", + "message": "announce", + "hex": "060b9234000207000903018005", "expect": "decoded" }, { - "name": "a ROUTE_START's 0x2 flag carries an RTT in milliseconds", + "name": "a ROUTE_START's suffix extends the requested prefix, and its 0x2 flag carries an RTT in milliseconds", "version": "moq-lite-07-wip", "message": "announce", - "hex": "060792340305031900", + "hex": "060b9234016101090002ff0019", "expect": "decoded" }, { - "name": "a ROUTE_START's RTT flag without its RTT is refused", + "name": "a Route Offer's 0x4 flag marks a path through a draining node", "version": "moq-lite-07-wip", "message": "announce", - "hex": "06059234030502", + "hex": "0609923400010900047f03", + "expect": "decoded" + }, + { + "name": "a Route Offer's RTT flag without its RTT is refused", + "version": "moq-lite-07-wip", + "message": "announce", + "hex": "0609923400010900028003", "expect": "refused" }, { - "name": "a ROUTE_START flag no version defines is withdrawn", + "name": "a Route Offer flag no version defines is withdrawn, with whatever field it adds", "version": "moq-lite-07-wip", "message": "announce", - "hex": "06059234030504", + "hex": "060a9234000109000880032a", "expect": "withdrawn" }, { - "name": "a ROUTE_START with more than 32 hops is withdrawn", + "name": "a Route Offer with more than 32 hops is withdrawn", "version": "moq-lite-07-wip", "message": "announce", - "hex": "0606923403050121", + "hex": "0649923400210100020003000400050006000700080009000a000b000c000d000e000f0010001100120013001400150016001700180019001a001b001c001d001e001f0020002100008001", "expect": "withdrawn" }, { - "name": "a ROUTE_START missing a hop it counts is refused", + "name": "a Route Offer with no hops is refused", "version": "moq-lite-07-wip", "message": "announce", - "hex": "060792340305010207", + "hex": "060792340000008001", + "expect": "refused" + }, + { + "name": "a Route Offer missing a hop it counts is refused", + "version": "moq-lite-07-wip", + "message": "announce", + "hex": "0609923400030700008003", "expect": "refused" }, { "name": "a ROUTE_START naming Node ID 0 is withdrawn", "version": "moq-lite-07-wip", "message": "announce", - "hex": "06050003050100", + "hex": "06080000010700008001", + "expect": "withdrawn" + }, + { + "name": "a Route Offer hop naming Node ID 0 is withdrawn", + "version": "moq-lite-07-wip", + "message": "announce", + "hex": "0609923400010000008001", "expect": "withdrawn" }, { - "name": "a ROUTE_START advertising the last seqno is withdrawn", + "name": "a ROUTE_START suffix no path spells is withdrawn", "version": "moq-lite-07-wip", "message": "announce", - "hex": "060d01ffffffffffffffffff000000", + "hex": "060c9234032f2f78010700008001", "expect": "withdrawn" }, { "name": "a ROUTE_UPDATE references its Route ID", "version": "moq-lite-07-wip", "message": "announce", - "hex": "07050103050100", + "hex": "070701010500008003", "expect": "decoded" }, + { + "name": "a ROUTE_UPDATE flag no version defines is withdrawn", + "version": "moq-lite-07-wip", + "message": "announce", + "hex": "0708010105008080032a", + "expect": "withdrawn" + }, { "name": "a ROUTE_HOLD references its Route ID", "version": "moq-lite-07-wip", @@ -119,80 +154,108 @@ "expect": "decoded" }, { - "name": "an ANNOUNCE_FLOOR carries its Revision plus one", + "name": "a DEAD CLOSED names the hop, the far node, Gone, More, and the routes that moved, each offer sized", "version": "moq-lite-07-wip", "message": "announce", - "hex": "09020105", + "hex": "0f110007020901000100080207000501008003", "expect": "decoded" }, { - "name": "an ANNOUNCE_FLOOR of 0 certifies nothing", + "name": "a DEAD CLOSED with More set continues in the next DEAD", "version": "moq-lite-07-wip", "message": "announce", - "hex": "09020100", + "hex": "0f0700070209000100", "expect": "decoded" }, { - "name": "a lite-07 ANNOUNCE_START names its Route ID in place of a hop chain", + "name": "a DEAD CLOSED moved offer this node refuses is withdrawn, and the entries after it still read", "version": "moq-lite-07-wip", "message": "announce", - "hex": "000a00000178020161000100", + "hex": "0f1900070209010002000802070005010880030306010700008001", + "expect": "withdrawn" + }, + { + "name": "a DEAD CLOSED moved offer short of its size is refused", + "version": "moq-lite-07-wip", + "message": "announce", + "hex": "0f110007020901000100080107000080030000", + "expect": "refused" + }, + { + "name": "a DEAD DRAIN names the node that sent GOAWAY", + "version": "moq-lite-07-wip", + "message": "announce", + "hex": "0f020107", "expect": "decoded" }, { - "name": "a ROUTE_UPDATE flag no version defines is withdrawn", + "name": "a DEAD of a kind no version defines is refused", "version": "moq-lite-07-wip", "message": "announce", - "hex": "0705010305802a", - "expect": "withdrawn" + "hex": "0f020207", + "expect": "refused" }, { - "name": "a lite-07 ANNOUNCE_START with an invalid path is withdrawn", + "name": "a DEAD whose Gone is neither 0 nor 1 is refused", "version": "moq-lite-07-wip", "message": "announce", - "hex": "000c0000032f2f78020161000100", - "expect": "withdrawn" + "hex": "0f0700070209020000", + "expect": "refused" }, { - "name": "a lite-07 ANNOUNCE_START with an invalid instance is withdrawn", + "name": "a DEAD whose More is neither 0 nor 1 is refused", "version": "moq-lite-07-wip", "message": "announce", - "hex": "0009000001780200000100", - "expect": "withdrawn" + "hex": "0f0700070209000200", + "expect": "refused" }, { - "name": "a lite-07 ANNOUNCE_START cut short is refused", + "name": "a DEAD naming Far Node ID 0 is refused", "version": "moq-lite-07-wip", "message": "announce", - "hex": "0003000003", + "hex": "0f06000702000000", "expect": "refused" }, { - "name": "a ROUTE_REQUEST names an origin node and seqno", + "name": "an ANNOUNCE_FLOOR carries its Revision plus one", "version": "moq-lite-07-wip", - "message": "announce_reply", - "hex": "0a020702", + "message": "announce", + "hex": "09020105", "expect": "decoded" }, { - "name": "a ROUTE_REQUEST naming Node ID 0 is refused", + "name": "an ANNOUNCE_FLOOR of 0 certifies nothing", "version": "moq-lite-07-wip", - "message": "announce_reply", - "hex": "0a020002", - "expect": "refused" + "message": "announce", + "hex": "09020100", + "expect": "decoded" }, { - "name": "a ROUTE_END echo names the Route ID it retired", + "name": "a lite-07 ANNOUNCE_START names its Route ID in place of a hop chain", "version": "moq-lite-07-wip", - "message": "announce_reply", - "hex": "080103", + "message": "announce", + "hex": "000a00000178020161000100", "expect": "decoded" }, { - "name": "an announce reply other than ROUTE_REQUEST or ROUTE_END is refused", + "name": "a lite-07 ANNOUNCE_START with an invalid path is withdrawn", + "version": "moq-lite-07-wip", + "message": "announce", + "hex": "000c0000032f2f78020161000100", + "expect": "withdrawn" + }, + { + "name": "a lite-07 ANNOUNCE_START with an invalid instance is withdrawn", + "version": "moq-lite-07-wip", + "message": "announce", + "hex": "0009000001780200000100", + "expect": "withdrawn" + }, + { + "name": "a lite-07 ANNOUNCE_START cut short is refused", "version": "moq-lite-07-wip", - "message": "announce_reply", - "hex": "0000", + "message": "announce", + "hex": "0003000003", "expect": "refused" }, { diff --git a/rs/moq-net/src/auth.rs b/rs/moq-net/src/auth.rs index 33ca3fe46a6..cf4be4c3de2 100644 --- a/rs/moq-net/src/auth.rs +++ b/rs/moq-net/src/auth.rs @@ -656,6 +656,12 @@ impl Permit { } } + /// The limit this side set on the peer, or `None` while unrestricted. + #[cfg(feature = "async")] + pub(crate) fn limit(&self) -> Option<&Patterns> { + self.limit.as_ref() + } + /// Whether both set the same limit on the peer, whatever either was granted. #[cfg(feature = "async")] pub(crate) fn same_limit(&self, other: &Self) -> bool { diff --git a/rs/moq-net/src/coding/writer.rs b/rs/moq-net/src/coding/writer.rs index 581cc14d4f1..2a9510da592 100644 --- a/rs/moq-net/src/coding/writer.rs +++ b/rs/moq-net/src/coding/writer.rs @@ -193,6 +193,11 @@ impl Writer { Poll::Ready(Ok(())) } + /// How many bytes the stream took through this writer. + pub fn written(&self) -> u64 { + self.written + } + /// Whether every buffered byte has hit the stream, so a FIN would not cut a message short. pub fn is_flushed(&self) -> bool { self.buffer.is_empty() @@ -750,6 +755,7 @@ mod tests { broadcast: crate::Path::from_static("test"), instance: None, node: 0, + depth: 0, track: "video".into(), }) .unwrap(); diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index fa3fc0d6952..b6a50cca9c9 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -258,7 +258,7 @@ const LITE_VERSIONS: &[lite::Version] = &[ const IETF_VERSIONS: &[ietf::Version] = &[ietf::Version::Draft18, ietf::Version::Draft20, ietf::Version::Draft22]; /// How many types [`lite_wire`] dispatches over. -const LITE_KINDS: u8 = 34; +const LITE_KINDS: u8 = 33; /// How many types [`ietf_wire`] dispatches over. const IETF_KINDS: u8 = 27; @@ -426,7 +426,6 @@ pub fn lite_wire(data: &[u8]) -> bool { 30 => roundtrip::, _>(rest, &mut Vec::new(), version), 31 => roundtrip::(rest, &mut Vec::new(), version), 32 => roundtrip::(rest, &mut Vec::new(), version), - 33 => roundtrip::(rest, &mut Vec::new(), version), _ => unreachable!("kind is taken modulo LITE_KINDS"), } } @@ -570,6 +569,28 @@ pub fn snapshot(origin: &crate::origin::Producer, prefix: &str) -> Snapshot { Snapshot(origin.snapshot(prefix)) } +/// A clock for a route node in a bench, which never fires its timers. +fn route_clock() -> kio::clock::Deadlines { + kio::clock::Deadlines::new(kio::clock::Instant::now()) +} + +/// The session `tag` to the node `node`. +fn route_peer(tag: u64, node: u64) -> crate::route::node::Peer { + crate::route::node::Peer { tag, node } +} + +/// The route the publisher `node` sends about itself. +fn route_offer(node: u64) -> crate::route::path::Offer { + crate::route::path::Offer { + path: [crate::route::path::Hop { node, session: 0 }].into(), + draining: false, + priority: i8::MAX, + metric: 0, + down_only: false, + rtt: None, + } +} + /// One origin's lite-07 route node answering announce streams, for the routes bench. pub struct RouteNode { node: crate::route::Node, @@ -581,15 +602,16 @@ impl RouteNode { /// A node holding `claims` claims of its own under `live/`, answering one stream /// per prefix in `streams`, each already sent its snapshot. pub fn new(claims: usize, streams: &[&str]) -> Self { - let node = crate::route::Node::new(1, 1); + let node = crate::route::Node::new(1, 1, route_clock()); let instance = crate::Instance::random(); for i in 0..claims { node.own(Path::new(&format!("live/{i}")).unwrap(), Some(Self::claim(&instance))); } - let streams = (streams.iter()) - .map(|prefix| { + let streams = (streams.iter().enumerate()) + .map(|(i, prefix)| { let link = crate::route::Link::default(); - node.answer(link, Path::new(prefix).unwrap()).unwrap() + let peer = route_peer(i as u64 + 1, i as u64 + 2); + node.answer(peer, link, Path::new(prefix).unwrap()).unwrap() }) .collect(); let node = Self { @@ -650,27 +672,25 @@ impl RouteRelay { /// A relay that already heard the origin's `claims` claims under `live/` on each of /// `streams` streams, and forwarded them on each of `answers` streams. fn new(claims: usize, streams: usize, answers: usize) -> Self { - let node = crate::route::Node::new(2, 2); - let answer = (0..answers) - .map(|_| node.answer(crate::route::Link::default(), Path::empty()).unwrap()) + let node = crate::route::Node::new(2, 2, route_clock()); + let answer = (0..answers as u64) + .map(|i| { + let peer = route_peer(i + 1, i + 3); + node.answer(peer, crate::route::Link::default(), Path::empty()).unwrap() + }) .collect(); let instance = crate::Instance::random(); let rev = claims as u64; - let advert = crate::route::Advert { - seqno: 0, - metric: 0, - down_only: false, - rtt: None, - hops: [].into(), - }; - let receive = (0..streams) - .map(|_| { - let receive = node.receive(crate::route::Link::default(), Path::empty()).unwrap(); - receive.route(Self::ORIGIN, Some(crate::route::Reach::Finite(advert.clone()))); + let receive = (0..streams as u64) + .map(|i| { + let peer = route_peer(answers as u64 + i + 1, Self::ORIGIN); + let receive = node.receive(peer, crate::route::Link::default()).unwrap(); + let reach = crate::route::path::Reach::Finite(route_offer(Self::ORIGIN)); + receive.route(Self::ORIGIN, Path::empty(), Some(reach)); let snapshot = (0..claims) .map(|i| (Self::path(i), Some(Self::claim(&instance, 0)))) .collect(); - receive.batch(Self::ORIGIN, snapshot, Some(rev)); + receive.batch(Self::ORIGIN, Path::empty(), snapshot, Some(rev)); receive }) .collect(); @@ -731,7 +751,7 @@ impl RouteRelay { self.rev += 1; let claim = Self::claim(&self.instance, self.rev); let batch = (0..changes).map(|i| (Self::path(i), Some(claim.clone()))).collect(); - self.receive[0].batch(Self::ORIGIN, batch, Some(self.rev)); + self.receive[0].batch(Self::ORIGIN, Path::empty(), batch, Some(self.rev)); self.drain() } @@ -748,11 +768,19 @@ impl RouteRelay { epoch, ..Self::claim(&self.instance, 0) }; - self.receive[0].batch(Self::ORIGIN, vec![(Self::path(0), Some(claim))], Some(self.rev)); + let batch = vec![(Self::path(0), Some(claim))]; + self.receive[0].batch(Self::ORIGIN, Path::empty(), batch, Some(self.rev)); let serves: Vec<_> = std::iter::from_fn(|| self.receive[0].try_output()).collect(); for serve in &serves { - if let crate::route::node::Output::Serve { id, claim, served, .. } = serve { - self.receive[0].serving(*id, claim, &served.claim); + if let crate::route::node::Output::Serve { + id, + claim, + prefix, + served, + .. + } = serve + { + self.receive[0].serving(*id, claim, prefix, &served.claim); } } serves.len() + self.drain() @@ -763,7 +791,7 @@ impl RouteRelay { self.rev += 1; let i = self.rev as usize % self.claims; let batch = vec![(Self::path(i), Some(Self::claim(&self.instance, self.rev)))]; - self.receive[0].batch(Self::ORIGIN, batch, Some(self.rev)); + self.receive[0].batch(Self::ORIGIN, Path::empty(), batch, Some(self.rev)); self.behind.push_back((self.rev, i)); } @@ -773,7 +801,7 @@ impl RouteRelay { self.advance(); let (rev, i) = self.behind.pop_front().expect("the second stream lags"); let batch = vec![(Self::path(i), Some(Self::claim(&self.instance, rev)))]; - self.receive[1].batch(Self::ORIGIN, batch, Some(rev)); + self.receive[1].batch(Self::ORIGIN, Path::empty(), batch, Some(rev)); self.drain() } @@ -800,11 +828,13 @@ impl RouteFollower { /// The origin's identity and revision as the node hears them. const ORIGIN: u64 = 1; const REV: u64 = 1; + /// The viewer session every stream belongs to. + const TAG: u64 = 1; /// A node following `prefixes` prefixes `live/{k}`, each holding `claims` claims. pub fn new(prefixes: usize, claims: usize) -> Self { let mut follower = Self { - node: crate::route::Node::new(2, 2), + node: crate::route::Node::new(2, 2, route_clock()), streams: Vec::new(), instance: crate::Instance::random(), claims, @@ -818,15 +848,10 @@ impl RouteFollower { /// Open the stream for prefix `k`, which hears the origin's route and snapshot. fn follow(&self, k: usize) -> crate::route::node::Stream { let prefix = Path::new(&format!("live/{k}")).unwrap(); - let stream = self.node.receive(crate::route::Link::default(), prefix).unwrap(); - let advert = crate::route::Advert { - seqno: 0, - metric: 0, - down_only: false, - rtt: None, - hops: [].into(), - }; - stream.route(Self::ORIGIN, Some(crate::route::Reach::Finite(advert))); + let peer = route_peer(Self::TAG, Self::ORIGIN); + let stream = self.node.receive(peer, crate::route::Link::default()).unwrap(); + let reach = crate::route::path::Reach::Finite(route_offer(Self::ORIGIN)); + stream.route(Self::ORIGIN, prefix.clone(), Some(reach)); let claim = crate::route::node::Claim { instance: self.instance.clone(), epoch: 0, @@ -838,7 +863,7 @@ impl RouteFollower { let snapshot = (0..self.claims) .map(|j| (Path::new(&format!("live/{k}/{j}")).unwrap(), Some(claim.clone()))) .collect(); - stream.batch(Self::ORIGIN, snapshot, Some(Self::REV)); + stream.batch(Self::ORIGIN, prefix, snapshot, Some(Self::REV)); stream } @@ -855,7 +880,7 @@ impl RouteFollower { /// Close every stream at once, as a session ending does. pub fn teardown(self) { - let _closing = self.node.closing(); + let _closing = self.node.closing(Self::TAG); drop(self.streams); } @@ -1391,6 +1416,7 @@ impl LiteSample { order: crate::track::Order::Descending, instance: None, node: 0, + depth: 0, id: 3, broadcast: Path::from_static("room/alice"), track: "video".into(), @@ -1591,6 +1617,7 @@ impl Default for Messages { order: crate::track::Order::Descending, instance: None, node: 0, + depth: 0, id: 7, broadcast: Path::from_static("room/alice"), track: "video".into(), @@ -1776,6 +1803,7 @@ pub fn encode_subscribe(broadcast: &Path, track: &str, ietf: bool, out: &mut Vec broadcast: broadcast.clone(), instance: None, node: 0, + depth: 0, track: track.into(), priority: 0, max_delay: std::time::Duration::ZERO, @@ -1995,21 +2023,23 @@ pub fn pattern(data: &[u8]) -> bool { } /// Lite seeds whose envelope bytes the uniform sweep never forms, on the newest -/// version: a typed empty TRACKS_SYNCED, an AUTH_OK reply, every route message -/// (the sweep sets ROUTE_START's type byte but fills its size byte with the body -/// pattern, so none decodes whole), the ROUTE_END echo, ROUTE_REQUEST, and a DATAGRAMS -/// request, whose seven fields outgrow the sweep's longest body. +/// version: a typed empty TRACKS_SYNCED, an AUTH_OK reply, every route message and +/// both kinds of DEAD (the sweep sets ROUTE_START's type byte but fills its size byte +/// with the body pattern, so none decodes whole), and a DATAGRAMS request, whose eight +/// fields outgrow the sweep's longest body. const TYPED_LITE_SEEDS: &[(u8, &[u8])] = &[ (23, &[4, 0]), - (30, &[7, 0, 0, 0, 0, 0, 0, 0]), + (30, &[8, 0, 0, 0, 0, 0, 0, 0, 0]), (27, &[0, 3, 0, 0, 0]), - (5, &[6, 6, 1, 0, 1, 0, 1, 1]), - (5, &[7, 6, 0, 1, 2, 0, 1, 1]), + (5, &[6, 8, 1, 0, 1, 1, 0, 0, 0x80, 1]), + (5, &[7, 7, 0, 1, 1, 0, 0, 0x80, 1]), (5, &[8, 1, 0]), (5, &[9, 2, 0, 1]), (5, &[11, 1, 0]), - (33, &[8, 1, 0]), - (33, &[10, 2, 1, 0]), + (5, &[15, 7, 0, 1, 1, 2, 0, 0, 0]), + (5, &[15, 7, 0, 1, 1, 2, 0, 1, 0]), + (5, &[15, 15, 0, 1, 1, 2, 1, 0, 1, 0, 6, 1, 1, 1, 0, 0x80, 1]), + (5, &[15, 2, 1, 1]), ]; /// The inputs the fuzzer starts from: every (version, type) pair the dispatch knows, diff --git a/rs/moq-net/src/lite/announce.rs b/rs/moq-net/src/lite/announce.rs index c161c89e74a..0d286c82de4 100644 --- a/rs/moq-net/src/lite/announce.rs +++ b/rs/moq-net/src/lite/announce.rs @@ -6,7 +6,7 @@ use crate::{ Hop, Hops, Instance, Path, coding::*, origin::Cost, - route::{self, Advert}, + route::{self, path::Offer}, }; use super::{ @@ -23,17 +23,23 @@ const ANNOUNCE_UPDATE: u64 = 2; // lite-07: the advertisement serves another instance, or a newer epoch of its own. const ANNOUNCE_RESTART: u64 = 3; const SYNCED: u64 = 4; -// lite-07: the route layer. ROUTE_END also travels back as the echo. +// lite-07: the route layer. 10 was ROUTE_REQUEST, which the path rule retired. const ROUTE_START: u64 = 6; const ROUTE_UPDATE: u64 = 7; const ROUTE_END: u64 = 8; const ANNOUNCE_FLOOR: u64 = 9; -const ROUTE_REQUEST: u64 = 10; const ROUTE_HOLD: u64 = 11; // lite-07: warm routes, copies of one broadcast offered toward the subscriber. const WARM_START: u64 = 12; const WARM_UPDATE: u64 = 13; const WARM_END: u64 = 14; +// lite-07: news of a closed session or a draining node on the routes the sender offered. +const DEAD: u64 = 15; + +/// DEAD kind: a session on the routes' paths closed. +const DEAD_CLOSED: u64 = 0; +/// DEAD kind: a node on the routes' paths sent GOAWAY. +const DEAD_DRAIN: u64 = 1; /// Whether the negotiated version can update a live advertisement's metadata. On lite-05 an /// update travels as a duplicate ANNOUNCE (a second `active` for an already-announced path); on @@ -103,17 +109,24 @@ pub enum AnnounceBroadcast { Skipped, /// The initial set has been delivered (lite-07+). Synced, - /// ROUTE_START (lite-07+): the sender has a route to the origin node `node`, and - /// assigns it the next Route ID. - RouteStart { node: route::Id, advert: Advert }, + /// ROUTE_START (lite-07+): the sender has a route to the publisher `node` covering + /// the requested prefix joined with `suffix`, and assigns it the next Route ID. + RouteStart { + node: route::Id, + suffix: Path, + offer: Offer, + }, /// ROUTE_UPDATE (lite-07+): the route with this id changed, and is usable. - RouteUpdate { id: u64, advert: Advert }, + RouteUpdate { id: u64, offer: Offer }, /// ROUTE_HOLD (lite-07+): the sender has no usable route with this id, but still /// holds its advertisements while it recovers. RouteHold { id: u64 }, /// ROUTE_END (lite-07+): the sender let go of the route with this id, and of every - /// advertisement under it. The receiver echoes it back. + /// advertisement under it. RouteEnd { id: u64 }, + /// DEAD (lite-07+): a session or node on the routes the sender offered is gone or + /// going. + Dead(Dead), /// ANNOUNCE_FLOOR (lite-07+): closes the batch of advertisements under the route /// with this id, certifying them across the whole requested prefix at the origin's /// revision, or certifying nothing (`None`). @@ -141,10 +154,16 @@ pub enum AnnounceBroadcast { /// resolve. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) enum Refusable { - /// ROUTE_START: assigns the next Route ID to the origin node `node`. - RouteStart { node: route::Id }, - /// ROUTE_UPDATE of the route with this Route ID. - RouteUpdate { id: u64 }, + /// ROUTE_START: assigns the next Route ID to the publisher `node`, covering the + /// requested prefix joined with `suffix`, or no prefix when the suffix itself was + /// refused, along `path`. + RouteStart { + node: route::Id, + suffix: Option, + path: Arc<[route::path::Hop]>, + }, + /// ROUTE_UPDATE of the route with this Route ID, now along `path`. + RouteUpdate { id: u64, path: Arc<[route::path::Hop]> }, /// ANNOUNCE_START under the route with Route ID `route`: assigns the next Announce /// ID, to `suffix`, or to no path when the path itself was refused. Start { route: u64, suffix: Option }, @@ -159,10 +178,8 @@ pub(crate) enum Refusable { /// Why a node refuses content a peer forwarded: well framed, but nothing it can use. #[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] pub(crate) enum Refusal { - #[error("the reserved origin node")] + #[error("the reserved node")] Node, - #[error("the reserved seqno")] - Seqno, #[error("unknown flags {0:#04x}")] Flags(u8), #[error("{0} hops, past the limit")] @@ -257,73 +274,284 @@ fn decode_warm_start(r: &mut Decoder<'_>) -> Content<(Path, Instance, Warm)> { Ok(decode_warm(r)?.map(|warm| (suffix, instance, warm))) } -/// ROUTE_START and ROUTE_UPDATE flag: the route is down-only. +/// Route Offer flag: an upstream link is on the path, so the route is down-only. const DOWN_ONLY: u8 = 0x1; -/// ROUTE_START and ROUTE_UPDATE flag: an RTT follows. +/// Route Offer flag: an RTT follows. const RTT: u8 = 0x2; +/// Route Offer flag: a node on the path sent GOAWAY. +const DRAINING: u8 = 0x4; + +/// News on DEAD about the routes the sender offered. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) enum Dead { + /// CLOSED: the session `hop` names closed, at the node `far` at its other end, which + /// the receiver doubts. Each route on the stream through that hop that `moved` names + /// by Route ID continues on its offer there; the rest end if `gone`, and are held if + /// not, unless the receiver is the node just before the sender on the route's path. + /// With `more`, the next message is a DEAD naming the same hop, far node, and `gone` + /// whose `moved` continues this one's, and the receiver applies them as one. + Closed { + hop: route::path::Hop, + far: route::Id, + gone: bool, + more: bool, + moved: Vec<(u64, Heard)>, + }, + /// DRAIN: `node` sent GOAWAY, so every path through it ranks as draining. + Drain { node: route::Id }, +} + +/// A Route Offer off the wire: usable, or refused with the path it still names. +pub(crate) type Heard = Result; -/// Encode a seqno, metric, flags, RTT in milliseconds, and the hops the route -/// crossed. No origin advertises the last seqno, which would leave no room for the -/// successor a ROUTE_REQUEST names, and no relay forwards a route that crossed too many. -fn encode_advert(w: &mut Encoder<'_>, advert: &Advert) -> Result<(), EncodeError> { - if advert.seqno == route::MAX || advert.hops.len() > route::MAX_HOPS { +/// A Route Offer this node refuses: the path it names, which a later DEAD may match, and +/// why. The offer is the route's, so refusing it refuses the route, not the session. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct Unusable { + pub path: Arc<[route::path::Hop]>, + pub reason: Refusal, +} + +/// Encode a Route Offer: the path, then flags, priority, metric, and RTT in +/// milliseconds. No node offers an empty path or one past the hop limit, nor names Node +/// ID 0 in one. +fn encode_offer(w: &mut Encoder<'_>, offer: &Offer) -> Result<(), EncodeError> { + if offer.path.is_empty() || offer.path.len() > route::MAX_HOPS || offer.path.iter().any(|hop| hop.node == 0) { return Err(EncodeError::InvalidState); } - w.varint(advert.seqno)?; - w.varint(advert.metric)?; + w.varint(offer.path.len() as u64)?; + for hop in offer.path.iter() { + w.varint(hop.node)?; + w.varint(hop.session)?; + } // A sum past the field's range, which only a peer's absurd value reaches, is unknown. - let rtt = advert.rtt.and_then(|rtt| u64::try_from(rtt.as_millis()).ok()); + let rtt = offer.rtt.and_then(|rtt| u64::try_from(rtt.as_millis()).ok()); let mut flags = 0; - if advert.down_only { + if offer.down_only { flags |= DOWN_ONLY; } if rtt.is_some() { flags |= RTT; } + if offer.draining { + flags |= DRAINING; + } w.u8(flags); + w.u8(super::priority::to_wire(offer.priority)); + w.varint(offer.metric)?; if let Some(rtt) = rtt { w.varint(rtt)?; } - w.varint(advert.hops.len() as u64)?; - for hop in advert.hops.iter() { - w.varint(*hop)?; - } Ok(()) } -/// Decode an advertisement, refusing one whose seqno no origin advertises, whose flags -/// this node does not know, or that crossed too many hops. -/// -/// An unknown flag refuses the route rather than being ignored: a newer flag may change -/// what the route means, or add a field ahead of the hops that would be misread. -fn decode_advert(r: &mut Decoder<'_>) -> Content { - let seqno = r.varint()?; - if seqno == route::MAX { - return refuse(r, Refusal::Seqno); +/// Decode a Route Offer. Its path comes first, so a node refusing the rest still learns +/// the path a later DEAD may name. An empty path is the sender's own error, since it +/// names itself last. Hops past the limit, a hop naming Node ID 0, and a flag this node +/// does not know refuse the offer: a newer flag may change what the route means, or add +/// a field after the ones this node reads, so a refusal skips the rest of `r`. Every hop +/// a path counts is read, but of a path past the limit only those up to one past it are +/// kept: a sender within the limit never sends one, and the whole would hold a body's +/// worth for each route. +fn decode_offer(r: &mut Decoder<'_>) -> Result { + let count = r.varint()?; + if count == 0 { + return Err(DecodeError::InvalidValue); + } + // Each hop takes at least two bytes, so the body bounds the count. + if count > (r.remaining() / 2) as u64 { + return Err(DecodeError::TooMany); + } + let mut path = Vec::with_capacity(count.min(route::MAX_HOPS as u64 + 1) as usize); + for index in 0..count { + let hop = route::path::Hop { + node: r.varint()?, + session: r.varint()?, + }; + if index <= route::MAX_HOPS as u64 { + path.push(hop); + } + } + let path: Arc<[route::path::Hop]> = path.into(); + let refuse = |r: &mut Decoder<'_>, reason| { + r.rest(); + Ok(Err(Unusable { + path: path.clone(), + reason, + })) + }; + if count > route::MAX_HOPS as u64 { + return refuse(r, Refusal::Hops(count)); + } + if path.iter().any(|hop| hop.node == 0) { + return refuse(r, Refusal::Node); } - let metric = r.varint()?; let flags = r.u8()?; - if flags & !(DOWN_ONLY | RTT) != 0 { + if flags & !(DOWN_ONLY | RTT | DRAINING) != 0 { return refuse(r, Refusal::Flags(flags)); } + let priority = super::priority::from_wire(r.u8()?); + let metric = r.varint()?; let rtt = match flags & RTT { 0 => None, _ => Some(std::time::Duration::from_millis(r.varint()?)), }; - let count = r.varint()?; - if count > route::MAX_HOPS as u64 { - return refuse(r, Refusal::Hops(count)); - } - let hops = (0..count).map(|_| r.varint()).collect::>()?; - Ok(Ok(Advert { - seqno, + Ok(Ok(Offer { + path, + draining: flags & DRAINING != 0, + priority, metric, down_only: flags & DOWN_ONLY != 0, rtt, - hops, })) } +fn encode_dead(w: &mut Encoder<'_>, dead: &Dead) -> Result<(), EncodeError> { + match dead { + Dead::Closed { + hop, + far, + gone, + more, + moved, + } => { + if hop.node == 0 || *far == 0 { + return Err(EncodeError::InvalidState); + } + w.varint(DEAD_CLOSED)?; + w.varint(hop.node)?; + w.varint(hop.session)?; + w.varint(*far)?; + w.u8(u8::from(*gone)); + w.u8(u8::from(*more)); + w.varint(moved.len() as u64)?; + for (id, offer) in moved { + encode_moved(w, *id, offer)?; + } + } + Dead::Drain { node } => { + if *node == 0 { + return Err(EncodeError::InvalidState); + } + w.varint(DEAD_DRAIN)?; + w.varint(*node)?; + } + } + Ok(()) +} + +fn encode_moved(w: &mut Encoder<'_>, id: u64, offer: &Heard) -> Result<(), EncodeError> { + // A refused offer is decode-only: this node never forwards one. + let Ok(offer) = offer else { + return Err(EncodeError::Unsupported); + }; + w.varint(id)?; + // Sized, so a receiver refusing one offer still reads the next. + let prefix = w.prefix_varint(); + encode_offer(w, offer)?; + w.fill(prefix)?; + Ok(()) +} + +impl Dead { + /// Split into consecutive DEADs that each fit one message, in order, every one but + /// the last with `more` set, so the receiver applies them as this one. + pub(crate) fn split(self) -> Vec { + let Self::Closed { + hop, + far, + gone, + more, + moved, + } = self + else { + return vec![self]; + }; + let form = Version::Lite07.into(); + let size = |encode: &dyn Fn(&mut Encoder<'_>) -> Result<(), EncodeError>| { + let mut body = Vec::new(); + // What cannot encode fails the DEAD when it is sent instead. + let _ = encode(&mut Encoder::new(&mut body, form)); + body.len() + }; + let part = |more, moved| Self::Closed { + hop, + far, + gone, + more, + moved, + }; + // A DEAD without its moved routes, its Moved Count grown to the widest varint. + let bare = size(&|w| encode_dead(w, &part(true, Vec::new()))) + 7; + let mut parts = Vec::new(); + let mut listed = Vec::new(); + let mut used = bare; + for (id, offer) in moved { + let entry = size(&|w| encode_moved(w, id, &offer)); + if !listed.is_empty() && used + entry > MAX_MESSAGE_SIZE { + parts.push(part(true, std::mem::take(&mut listed))); + used = bare; + } + used += entry; + listed.push((id, offer)); + } + parts.push(part(more, listed)); + parts + } +} + +fn decode_dead(r: &mut Decoder<'_>) -> Result { + let nonzero = |id: u64| match id { + 0 => Err(DecodeError::InvalidValue), + id => Ok(id), + }; + Ok(match r.varint()? { + DEAD_CLOSED => { + let hop = route::path::Hop { + node: nonzero(r.varint()?)?, + session: r.varint()?, + }; + let far = nonzero(r.varint()?)?; + let flag = |byte| match byte { + 0 => Ok(false), + 1 => Ok(true), + _ => Err(DecodeError::InvalidValue), + }; + let gone = flag(r.u8()?)?; + let more = flag(r.u8()?)?; + let count = r.varint()?; + // Each entry takes at least a byte per field, so the body bounds the count. + if count > r.remaining() as u64 { + return Err(DecodeError::TooMany); + } + let moved = (0..count) + .map(|_| { + let id = r.varint()?; + let size = usize::try_from(r.varint()?).map_err(|_| DecodeError::TooMany)?; + let mut offer = r.sub(size)?; + let heard = decode_offer(&mut offer)?; + // A usable offer fills its size; a refused one was skipped to its end. + if !offer.is_empty() { + return Err(DecodeError::Long); + } + Ok((id, heard)) + }) + .collect::>()?; + Dead::Closed { + hop, + far, + gone, + more, + moved, + } + } + DEAD_DRAIN => Dead::Drain { + node: nonzero(r.varint()?)?, + }, + _ => return Err(DecodeError::InvalidValue), + }) +} + /// Decode a path a peer forwarded, refusing one this node cannot hold. fn decode_path(r: &mut Decoder<'_>) -> Content { let bytes = r.bytes()?; @@ -583,6 +811,7 @@ impl Encode for AnnounceBroadcast { Self::RouteUpdate { .. } if version.has_routes() => ROUTE_UPDATE, Self::RouteHold { .. } if version.has_routes() => ROUTE_HOLD, Self::RouteEnd { .. } if version.has_routes() => ROUTE_END, + Self::Dead(_) if version.has_routes() => DEAD, Self::Floor { .. } if version.has_routes() => ANNOUNCE_FLOOR, Self::WarmStart { .. } if version.has_warm() => WARM_START, Self::WarmUpdate { .. } if version.has_warm() => WARM_UPDATE, @@ -591,6 +820,7 @@ impl Encode for AnnounceBroadcast { | Self::RouteUpdate { .. } | Self::RouteHold { .. } | Self::RouteEnd { .. } + | Self::Dead(_) | Self::Floor { .. } | Self::WarmStart { .. } | Self::WarmUpdate { .. } @@ -639,18 +869,20 @@ impl Encode for AnnounceBroadcast { encode_lite06_chain(w, version, hops)?; } Self::Synced => {} - Self::RouteStart { node, advert } => { + Self::RouteStart { node, suffix, offer } => { if *node == 0 { return Err(EncodeError::InvalidState); } w.varint(*node)?; - encode_advert(w, advert)?; + suffix.encode(w, version)?; + encode_offer(w, offer)?; } - Self::RouteUpdate { id, advert } => { + Self::RouteUpdate { id, offer } => { w.varint(*id)?; - encode_advert(w, advert)?; + encode_offer(w, offer)?; } Self::RouteHold { id } | Self::RouteEnd { id } => w.varint(*id)?, + Self::Dead(dead) => encode_dead(w, dead)?, Self::Floor { id, revision } => encode_floor(w, *id, *revision)?, Self::WarmStart { route, @@ -712,6 +944,7 @@ impl Encode for AnnounceBroadcast { | Self::RouteUpdate { .. } | Self::RouteHold { .. } | Self::RouteEnd { .. } + | Self::Dead(_) | Self::Floor { .. } | Self::WarmStart { .. } | Self::WarmUpdate { .. } @@ -765,29 +998,40 @@ impl<'a> Decode<'a, Version> for AnnounceBroadcast { } ROUTE_START if version.has_routes() => { let node = body.varint()?; - let refused = |reason| Self::Refused { - message: Refusable::RouteStart { node }, - reason, - }; - match node { - 0 => { - body.rest(); - refused(Refusal::Node) + // The suffix is sized, so the offer's path reads even when it is refused. + let suffix = decode_path(&mut body)?; + let heard = decode_offer(&mut body)?; + match (node, suffix, heard) { + (1.., Ok(suffix), Ok(offer)) => Self::RouteStart { node, suffix, offer }, + (node, suffix, heard) => { + let reason = match (node, &suffix, &heard) { + (0, _, _) => Refusal::Node, + (_, Err(reason), _) => *reason, + (_, _, Err(unusable)) => unusable.reason, + _ => unreachable!("matched above"), + }; + Self::Refused { + message: Refusable::RouteStart { + node, + suffix: suffix.ok(), + path: heard.map_or_else(|unusable| unusable.path, |offer| offer.path), + }, + reason, + } } - _ => decode_advert(&mut body)? - .map_or_else(refused, |advert| Self::RouteStart { node, advert }), } } ROUTE_UPDATE if version.has_routes() => { let id = body.varint()?; - decode_advert(&mut body)?.map_or_else( - |reason| Self::Refused { - message: Refusable::RouteUpdate { id }, + match decode_offer(&mut body)? { + Ok(offer) => Self::RouteUpdate { id, offer }, + Err(Unusable { path, reason }) => Self::Refused { + message: Refusable::RouteUpdate { id, path }, reason, }, - |advert| Self::RouteUpdate { id, advert }, - ) + } } + DEAD if version.has_routes() => Self::Dead(decode_dead(&mut body)?), ROUTE_HOLD if version.has_routes() => Self::RouteHold { id: body.varint()? }, ROUTE_END if version.has_routes() => Self::RouteEnd { id: body.varint()? }, ANNOUNCE_FLOOR if version.has_routes() => decode_floor(&mut body)?, @@ -952,71 +1196,6 @@ impl<'a> Message<'a> for AnnounceRequest { } } -/// A message the subscriber sends on the Announce Stream after its ANNOUNCE_REQUEST -/// (lite-07+), toward the publisher that sent the routes. -#[derive(Clone, Debug, PartialEq, Eq)] -pub enum AnnounceReply { - /// ROUTE_REQUEST: asks for a route to `node`, as the publisher named it, at least - /// as new as `seqno`. - RouteRequest { node: route::Id, seqno: route::Seqno }, - /// The echo of the ROUTE_END that retired this Route ID, sent once the subscriber - /// dropped the route. - RouteEnd { id: u64 }, -} - -impl Encode for AnnounceReply { - fn encode(&self, w: &mut Encoder<'_>, version: Version) -> Result<(), EncodeError> { - w.control = Some(crate::stats::Message::Announce); - if !version.has_routes() { - return Err(EncodeError::Version); - } - w.varint(match self { - Self::RouteRequest { .. } => ROUTE_REQUEST, - Self::RouteEnd { .. } => ROUTE_END, - })?; - let prefix = w.prefix_varint(); - match self { - Self::RouteRequest { node: 0, .. } => return Err(EncodeError::InvalidState), - Self::RouteRequest { node, seqno } => { - w.varint(*node)?; - w.varint(*seqno)?; - } - Self::RouteEnd { id } => w.varint(*id)?, - } - w.fill(prefix) - } -} - -impl<'a> Decode<'a, Version> for AnnounceReply { - fn decode(buf: &mut Decoder<'a>, version: Version) -> Result { - buf.control = Some(crate::stats::Message::Announce); - if !version.has_routes() { - return Err(DecodeError::Version); - } - let typ = buf.varint()?; - let size = decode_size(buf, MAX_MESSAGE_SIZE)?; - let mut body = buf.sub(size)?; - let msg = (|| { - Ok(match typ { - ROUTE_REQUEST => match body.varint()? { - 0 => return Err(DecodeError::InvalidValue), - node => Self::RouteRequest { - node, - seqno: body.varint()?, - }, - }, - ROUTE_END => Self::RouteEnd { id: body.varint()? }, - typ => return Err(DecodeError::InvalidMessage(typ)), - }) - })() - .map_err(DecodeError::complete)?; - if !body.is_empty() { - return Err(DecodeError::Long); - } - Ok(msg) - } -} - /// Send by the publisher, used to determine the message that follows. #[derive(Clone, Copy, Debug, IntoPrimitive, TryFromPrimitive)] #[repr(u8)] @@ -1964,13 +2143,18 @@ mod tests { } } - fn advert(seqno: u64, metric: u64, down_only: bool) -> Advert { - Advert { - seqno, + /// An offer over `path`, each hop `(node, session)`, the publisher first. + fn offer(path: &[(u64, u64)], metric: u64, down_only: bool) -> Offer { + Offer { + path: path + .iter() + .map(|&(node, session)| route::path::Hop { node, session }) + .collect(), + draining: false, + priority: 0, metric, down_only, rtt: None, - hops: [].into(), } } @@ -1992,44 +2176,64 @@ mod tests { crate::coding::decode_buf(&mut slice, version, |r, v| AnnounceBroadcast::decode(r, v)) } - fn decode_reply(buf: &[u8], version: Version) -> Result { - let mut slice = buf; - crate::coding::decode_buf(&mut slice, version, |r, v| AnnounceReply::decode(r, v)) - } - #[test] fn route_messages_round_trip_on_lite07() { + let long: Vec<(u64, u64)> = (1..=route::MAX_HOPS as u64).map(|node| (node, node * 3)).collect(); for msg in [ AnnounceBroadcast::RouteStart { node: 0x7a3f, - advert: advert(41, 12, false), + suffix: Path::empty(), + offer: offer(&[(0x7a3f, 0)], 0, false), }, AnnounceBroadcast::RouteStart { node: route::QUALIFIED | 9, - advert: advert(route::MAX - 1, route::MAX, true), + suffix: Path::from_static("room/a"), + offer: offer(&[(9, 0), (4, u64::MAX >> 2)], route::MAX, true), }, AnnounceBroadcast::RouteUpdate { id: 0, - advert: advert(42, 0, true), + offer: Offer { + draining: true, + priority: i8::MIN, + ..offer(&[(5, 0), (6, 1)], 2, false) + }, }, AnnounceBroadcast::RouteStart { node: 0x7a40, - advert: Advert { + suffix: Path::empty(), + offer: Offer { rtt: Some(std::time::Duration::ZERO), - hops: [5, route::MAX, 9].into(), - ..advert(1, 0, false) + priority: i8::MAX, + ..offer(&[(5, 0), (route::MAX, 7), (9, 2)], 0, false) }, }, AnnounceBroadcast::RouteUpdate { id: 1, - advert: Advert { + offer: Offer { rtt: Some(std::time::Duration::from_millis(u64::MAX)), - hops: (1..=route::MAX_HOPS as u64).collect(), - ..advert(2, 3, true) + ..offer(&long, 3, true) }, }, AnnounceBroadcast::RouteHold { id: 3 }, AnnounceBroadcast::RouteEnd { id: 0 }, + AnnounceBroadcast::Dead(Dead::Closed { + hop: route::path::Hop { node: 4, session: 2 }, + far: 9, + gone: false, + more: true, + moved: Vec::new(), + }), + AnnounceBroadcast::Dead(Dead::Closed { + hop: route::path::Hop { node: 4, session: 0 }, + far: route::MAX, + gone: true, + more: false, + moved: vec![ + (0, Ok(offer(&[(1, 0), (5, 1)], 3, false))), + (7, Ok(offer(&[(1, 0)], 0, true))), + ], + }), + AnnounceBroadcast::Dead(Dead::Drain { node: 4 }), AnnounceBroadcast::Floor { id: 0, revision: Some(0), @@ -2044,15 +2248,33 @@ mod tests { } } + /// The priority travels as the moq-lite priority byte, ordered as the model's i8. + #[test] + fn an_offer_carries_the_priority_byte() { + let mut buf = Vec::new(); + let msg = AnnounceBroadcast::RouteUpdate { + id: 0, + offer: Offer { + priority: -1, + ..offer(&[(1, 0)], 0, false) + }, + }; + msg.encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07) + .unwrap(); + // Type, length, Route ID, the one hop's count, node, and session, flags, then the + // priority byte. + assert_eq!(buf[7], 0x7f); + } + /// A peer's largest RTT plus a link goes past the field: it is forwarded as /// unknown, never an error that would end the stream. #[test] fn an_rtt_past_the_field_is_unknown() { let msg = |rtt| AnnounceBroadcast::RouteUpdate { id: 1, - advert: Advert { + offer: Offer { rtt, - ..advert(2, 3, false) + ..offer(&[(1, 0)], 3, false) }, }; let past = std::time::Duration::from_millis(u64::MAX).checked_add(std::time::Duration::from_millis(1)); @@ -2081,14 +2303,16 @@ mod tests { for msg in [ AnnounceBroadcast::RouteStart { node: 1, - advert: advert(0, 1, false), + suffix: Path::empty(), + offer: offer(&[(1, 0)], 0, false), }, AnnounceBroadcast::RouteUpdate { id: 0, - advert: advert(0, 1, false), + offer: offer(&[(1, 0)], 0, false), }, AnnounceBroadcast::RouteHold { id: 0 }, AnnounceBroadcast::RouteEnd { id: 0 }, + AnnounceBroadcast::Dead(Dead::Drain { node: 1 }), AnnounceBroadcast::Floor { id: 0, revision: None }, ] { for version in [Version::Lite05, Version::Lite06] { @@ -2103,45 +2327,40 @@ mod tests { decode_broadcast(&wire, Version::Lite06).unwrap(), AnnounceBroadcast::Skipped ); - - for reply in [ - AnnounceReply::RouteRequest { node: 1, seqno: 0 }, - AnnounceReply::RouteEnd { id: 0 }, - ] { - let mut buf = Vec::new(); - let res = reply.encode(&mut Encoder::new(&mut buf, Version::Lite06.into()), Version::Lite06); - assert!(matches!(res, Err(EncodeError::Version))); - } - assert!(matches!( - decode_reply(&wire, Version::Lite06), - Err(DecodeError::Version) - )); } - /// What no node advertises is refused on both ends: node 0, and the last seqno or - /// revision. + /// What no node offers is refused on both ends: node 0, an empty path or one past the + /// hop limit, and the last revision. #[test] - fn route_messages_refuse_what_no_node_advertises() { + fn route_messages_refuse_what_no_node_offers() { + let over: Vec<(u64, u64)> = (1..=route::MAX_HOPS as u64 + 1).map(|node| (node, 0)).collect(); let refused = [ AnnounceBroadcast::RouteStart { node: 0, - advert: advert(0, 1, false), + suffix: Path::empty(), + offer: offer(&[(1, 0)], 1, false), }, AnnounceBroadcast::RouteStart { node: 1, - advert: advert(route::MAX, 1, false), + suffix: Path::empty(), + offer: offer(&[], 1, false), }, AnnounceBroadcast::RouteUpdate { id: 0, - advert: advert(route::MAX, 1, false), + offer: offer(&over, 1, false), }, AnnounceBroadcast::RouteUpdate { id: 0, - advert: Advert { - hops: (0..=route::MAX_HOPS as u64).collect(), - ..advert(0, 1, false) - }, + offer: offer(&[(1, 0), (0, 1)], 1, false), }, + AnnounceBroadcast::Dead(Dead::Drain { node: 0 }), + AnnounceBroadcast::Dead(Dead::Closed { + hop: route::path::Hop { node: 1, session: 1 }, + far: 0, + gone: false, + more: false, + moved: Vec::new(), + }), // The last revision has no room for the offset. AnnounceBroadcast::Floor { id: 0, @@ -2154,52 +2373,98 @@ mod tests { assert!(matches!(res, Err(EncodeError::InvalidState)), "{msg:?}"); } - let advert = |node: u64, seqno: u64, metric: u64, down_only: u8| { + // An offer whose path counts `count` hops and names `named` of them, from Node ID + // `first` up, each on session 0, then these flags, the default priority, and a + // metric of 1. + let path = |flags: u8, count: u64, named: u64, first: u64| { move |w: &mut Encoder<'_>| { - w.varint(node).unwrap(); - w.varint(seqno).unwrap(); - w.varint(metric).unwrap(); - w.u8(down_only); - w.varint(0u64).unwrap(); - } - }; - // A route with `count` hops, naming `named` of them. - let hops = |count: u64, named: u64| { - move |w: &mut Encoder<'_>| { - for field in [1, 0, 1] { - w.varint(field as u64).unwrap(); - } - w.u8(0); w.varint(count).unwrap(); for hop in 0..named { - w.varint(hop + 1).unwrap(); + w.varint(first + hop).unwrap(); + w.varint(0u64).unwrap(); } + w.u8(flags); + w.u8(0x80); + w.varint(1u64).unwrap(); } }; + let start = |node: u64, offer: Box)>| { + forge(ROUTE_START, move |w| { + w.varint(node).unwrap(); + w.varint(0u64).unwrap(); + offer(w); + }) + }; + let hops = |count: u64, first: u64| -> Arc<[route::path::Hop]> { + (first..first + count) + .map(|node| route::path::Hop { node, session: 0 }) + .collect() + }; + let named = |node, path| Refusable::RouteStart { + node, + suffix: Some(Path::empty()), + path, + }; let over = route::MAX_HOPS as u64 + 1; - // What the sender only forwarded is refused, the message decoded all the same, so - // the receiver withdraws the route and keeps the stream. - let start = |node| Refusable::RouteStart { node }; + // What the sender only forwarded is refused, the message decoded all the same and + // its path with it, so the receiver withdraws the route and keeps the stream. let refused = [ - (forge(ROUTE_START, hops(over, over)), start(1), Refusal::Hops(over)), - (forge(ROUTE_START, advert(0, 0, 1, 0)), start(0), Refusal::Node), ( - forge(ROUTE_START, advert(1, route::MAX, 1, 0)), - start(1), - Refusal::Seqno, + start(1, Box::new(path(0, over, over, 1))), + named(1, hops(over, 1)), + Refusal::Hops(over), + ), + // Far past the limit: only the hops up to one past it are kept. + ( + start(1, Box::new(path(0, 4 * over, 4 * over, 1))), + named(1, hops(over, 1)), + Refusal::Hops(4 * over), + ), + ( + start(1, Box::new(path(0, 1, 1, 0))), + named(1, hops(1, 0)), + Refusal::Node, + ), + ( + start(0, Box::new(path(0, 1, 1, 1))), + named(0, hops(1, 1)), + Refusal::Node, ), // A flag no version defines, which may change what follows it. - (forge(ROUTE_START, advert(1, 0, 1, 4)), start(1), Refusal::Flags(4)), ( - forge(ROUTE_START, advert(1, 0, 1, 0x86)), - start(1), + start(1, Box::new(path(8, 1, 1, 1))), + named(1, hops(1, 1)), + Refusal::Flags(8), + ), + ( + start(1, Box::new(path(0x86, 1, 1, 1))), + named(1, hops(1, 1)), Refusal::Flags(0x86), ), - // ROUTE_UPDATE's leading field is the Route ID, so 0 is a valid id here. + // A suffix no path spells. ( - forge(ROUTE_UPDATE, advert(0, route::MAX, 1, 0)), - Refusable::RouteUpdate { id: 0 }, - Refusal::Seqno, + forge(ROUTE_START, |w| { + w.varint(1u64).unwrap(); + w.bytes(b"x//y").unwrap(); + path(0, 1, 1, 1)(w); + }), + Refusable::RouteStart { + node: 1, + suffix: None, + path: hops(1, 1), + }, + Refusal::Path, + ), + ( + forge(ROUTE_UPDATE, |w| { + w.varint(0u64).unwrap(); + path(8, 1, 1, 1)(w); + }), + Refusable::RouteUpdate { + id: 0, + path: hops(1, 1), + }, + Refusal::Flags(8), ), ]; for (wire, message, reason) in refused { @@ -2209,17 +2474,112 @@ mod tests { "{wire:?}" ); } + + // A DEAD's moved offer is sized, so one refused leaves the next readable. + let wire = forge(DEAD, |w| { + for field in [DEAD_CLOSED, 1, 1, 2] { + w.varint(field).unwrap(); + } + w.u8(1); + w.u8(0); + w.varint(2u64).unwrap(); + for (id, flags) in [(4u64, 8u8), (5, 0)] { + w.varint(id).unwrap(); + let prefix = w.prefix_varint(); + path(flags, 1, 1, 3)(w); + w.fill(prefix).unwrap(); + } + }); + let AnnounceBroadcast::Dead(Dead::Closed { moved, .. }) = decode_broadcast(&wire, Version::Lite07).unwrap() + else { + panic!("a DEAD"); + }; + assert_eq!( + moved.iter().map(|(id, heard)| (*id, heard.is_ok())).collect::>(), + [(4, false), (5, true)] + ); + assert_eq!( + moved[0].1, + Err(Unusable { + path: hops(1, 3), + reason: Refusal::Flags(8) + }) + ); + let malformed = [ - // Hops missing, and one too many. - forge(ROUTE_START, hops(2, 1)), - forge(ROUTE_START, hops(1, 2)), - // An RTT flag without its field. - forge(ROUTE_START, advert(1, 0, 1, 2)), + // No hops at all, and fewer than it counts. + start(1, Box::new(path(0, 0, 0, 1))), + start(1, Box::new(path(0, 3, 1, 1))), + // An RTT flag without its field, and a byte past a usable offer. + forge(ROUTE_UPDATE, |w| { + w.varint(0u64).unwrap(); + path(RTT, 1, 1, 1)(w); + }), + forge(ROUTE_UPDATE, |w| { + w.varint(0u64).unwrap(); + path(0, 1, 1, 1)(w); + w.u8(0); + }), + // A moved offer short of its size. + forge(DEAD, |w| { + for field in [DEAD_CLOSED, 1, 1, 2] { + w.varint(field).unwrap(); + } + w.u8(1); + w.varint(1u64).unwrap(); + w.varint(4u64).unwrap(); + let prefix = w.prefix_varint(); + path(0, 1, 1, 3)(w); + w.u8(0); + w.fill(prefix).unwrap(); + }), // A hold carries only its Route ID. forge(ROUTE_HOLD, |w| { w.varint(0u64).unwrap(); w.varint(0u64).unwrap(); }), + // A DEAD of a kind no version defines, naming Node ID 0, or Gone or More past 1. + forge(DEAD, |w| { + w.varint(2u64).unwrap(); + w.varint(1u64).unwrap(); + }), + forge(DEAD, |w| { + w.varint(DEAD_DRAIN).unwrap(); + w.varint(0u64).unwrap(); + }), + forge(DEAD, |w| { + for field in [DEAD_CLOSED, 1, 1, 0] { + w.varint(field).unwrap(); + } + w.u8(0); + w.u8(0); + w.varint(0u64).unwrap(); + }), + forge(DEAD, |w| { + for field in [DEAD_CLOSED, 1, 1, 2] { + w.varint(field).unwrap(); + } + w.u8(2); + w.u8(0); + w.varint(0u64).unwrap(); + }), + forge(DEAD, |w| { + for field in [DEAD_CLOSED, 1, 1, 2] { + w.varint(field).unwrap(); + } + w.u8(0); + w.u8(2); + w.varint(0u64).unwrap(); + }), + // More moved entries than the body could hold. + forge(DEAD, |w| { + for field in [DEAD_CLOSED, 1, 1, 2] { + w.varint(field).unwrap(); + } + w.u8(1); + w.u8(0); + w.varint(1u64 << 40).unwrap(); + }), // A floor without its revision, and one with trailing bytes. forge(ANNOUNCE_FLOOR, |w| w.varint(0u64).unwrap()), forge(ANNOUNCE_FLOOR, |w| { @@ -2237,49 +2597,4 @@ mod tests { assert!(decode_broadcast(&wire, Version::Lite07).is_err(), "{wire:?}"); } } - - #[test] - fn announce_reply_round_trips_on_lite07() { - for reply in [ - AnnounceReply::RouteRequest { - node: 0x7a3f, - seqno: 42, - }, - // A request may name the last seqno: only a rotation answers it. - AnnounceReply::RouteRequest { - node: route::QUALIFIED | 1, - seqno: route::MAX, - }, - AnnounceReply::RouteEnd { id: 0 }, - ] { - let mut buf = Vec::new(); - reply - .encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07) - .unwrap(); - assert_eq!(decode_reply(&buf, Version::Lite07).unwrap(), reply); - } - } - - #[test] - fn announce_reply_refuses_node_zero_and_unknown_types() { - let mut buf = Vec::new(); - let res = AnnounceReply::RouteRequest { node: 0, seqno: 1 } - .encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07); - assert!(matches!(res, Err(EncodeError::InvalidState))); - - let zero = forge(ROUTE_REQUEST, |w| { - w.varint(0u64).unwrap(); - w.varint(1u64).unwrap(); - }); - assert!(matches!( - decode_reply(&zero, Version::Lite07), - Err(DecodeError::InvalidValue) - )); - // Only routes travel back: an announcement type is not a reply. - let start = forge(ROUTE_START, |w| w.varint(1u64).unwrap()); - assert!(matches!( - decode_reply(&start, Version::Lite07), - Err(DecodeError::InvalidMessage(ROUTE_START)) - )); - } } diff --git a/rs/moq-net/src/lite/corpus.rs b/rs/moq-net/src/lite/corpus.rs index 309eff3fab3..539f4cb67a9 100644 --- a/rs/moq-net/src/lite/corpus.rs +++ b/rs/moq-net/src/lite/corpus.rs @@ -7,7 +7,7 @@ //! equal the input, so a case also pins the value, `refused`, where the input is rejected //! without asking for more bytes, or `withdrawn`, where a lite-07 route message decodes but its //! forwarded content is refused, so the receiver withdraws the route or claim and keeps the -//! session. A fuzz crash on a single moq-lite message belongs here +//! session (for a DEAD, any route a moved offer names). A fuzz crash on a single moq-lite message belongs here //! rather than in `fuzz/regressions/`, so JS replays it too. use crate::coding::{Decode, DecodeError, Decoder, Encode, Form}; @@ -31,9 +31,6 @@ struct Case { enum Message { /// One announcement on the announce stream, type and length prefix included. Announce, - /// One lite-07 reply on the announce stream, subscriber to publisher, type and length - /// prefix included. - AnnounceReply, /// A lite-01/02 ANNOUNCE_INIT, length prefix included. AnnounceInit, /// A group stream header, length prefix included. @@ -77,9 +74,13 @@ fn run(case: &Case, data: &[u8]) -> String { match case.message { Message::Announce => match super::AnnounceBroadcast::decode_slice(data, version) { Ok((super::AnnounceBroadcast::Refused { .. }, used)) if used == data.len() => "withdrawn".to_string(), + Ok((super::AnnounceBroadcast::Dead(super::Dead::Closed { moved, .. }), used)) + if used == data.len() && moved.iter().any(|(_, heard)| heard.is_err()) => + { + "withdrawn".to_string() + } decoded => outcome(data, version, decoded), }, - Message::AnnounceReply => outcome(data, version, super::AnnounceReply::decode_slice(data, version)), Message::AnnounceInit => { let mut r = decoder(); let decoded = super::AnnounceInit::decode_within(&mut r, version, case.budget.unwrap_or(usize::MAX)); diff --git a/rs/moq-net/src/lite/datagram.rs b/rs/moq-net/src/lite/datagram.rs index f4ace452d76..d5643497ada 100644 --- a/rs/moq-net/src/lite/datagram.rs +++ b/rs/moq-net/src/lite/datagram.rs @@ -120,6 +120,9 @@ pub struct Datagrams<'a> { /// The Node ID of the origin that serves the request, or 0 for whichever the /// publisher selects. pub node: u64, + /// How many leading segments of the broadcast path make up the prefix of the route + /// the request follows (lite-07); 0 with no node named. + pub depth: u64, pub track: Cow<'a, str>, /// The subscriber's priority. pub priority: i8, @@ -133,6 +136,7 @@ impl Datagrams<'_> { broadcast: self.broadcast, instance: self.instance, node: self.node, + depth: self.depth, track: Cow::Owned(self.track.into_owned()), priority: self.priority, } @@ -149,7 +153,7 @@ impl<'a> Message<'a> for Datagrams<'a> { let id = r.varint()?; let broadcast = Path::decode(r, version)?; let instance = super::instance::decode_pin(r, version)?; - let node = super::instance::decode_node(r, version)?; + let (node, depth) = super::instance::decode_node(r, version, &broadcast)?; let track = Cow::Borrowed(r.str()?); let priority = super::priority::from_wire(r.u8()?); Ok(Self { @@ -157,6 +161,7 @@ impl<'a> Message<'a> for Datagrams<'a> { broadcast, instance, node, + depth, track, priority, }) @@ -169,7 +174,7 @@ impl<'a> Message<'a> for Datagrams<'a> { w.varint(self.id)?; self.broadcast.encode(w, version)?; super::instance::encode_pin(w, version, self.instance.as_ref())?; - super::instance::encode_node(w, version, self.node)?; + super::instance::encode_node(w, version, self.node, self.depth)?; w.string(&self.track)?; w.u8(super::priority::to_wire(self.priority)); Ok(()) @@ -370,6 +375,7 @@ mod test { broadcast: Path::from_static("room/alice"), instance: None, node: 0x7a3f, + depth: 0, track: "audio".into(), priority: -3, }; diff --git a/rs/moq-net/src/lite/fetch.rs b/rs/moq-net/src/lite/fetch.rs index 81024882e4e..f132b702cd6 100644 --- a/rs/moq-net/src/lite/fetch.rs +++ b/rs/moq-net/src/lite/fetch.rs @@ -19,6 +19,9 @@ pub struct Fetch<'a> { /// The Node ID of the origin that serves the request, or 0 for whichever the /// publisher selects. Lite07+ only. pub node: u64, + /// How many leading segments of the broadcast path make up the prefix of the route + /// the request follows (lite-07); 0 with no node named. + pub depth: u64, pub track: Cow<'a, str>, pub priority: i8, /// Reader freshness budget. Lite07+ only; omission uses the publisher's max age alone. @@ -41,6 +44,7 @@ impl Fetch<'_> { broadcast: self.broadcast, instance: self.instance, node: self.node, + depth: self.depth, track: Cow::Owned(self.track.into_owned()), priority: self.priority, max_delay: self.max_delay, @@ -64,7 +68,7 @@ impl<'a> Message<'a> for Fetch<'a> { let broadcast = Path::decode(r, version)?; let instance = super::instance::decode_pin(r, version)?; - let node = super::instance::decode_node(r, version)?; + let (node, depth) = super::instance::decode_node(r, version, &broadcast)?; let track = Cow::Borrowed(r.str()?); let priority = super::priority::from_wire(r.u8()?); let max_delay = match version.has_fetch_max_delay() { @@ -94,6 +98,7 @@ impl<'a> Message<'a> for Fetch<'a> { broadcast, instance, node, + depth, track, priority, max_delay, @@ -114,7 +119,7 @@ impl<'a> Message<'a> for Fetch<'a> { self.broadcast.encode(w, version)?; super::instance::encode_pin(w, version, self.instance.as_ref())?; - super::instance::encode_node(w, version, self.node)?; + super::instance::encode_node(w, version, self.node, self.depth)?; w.string(&self.track)?; w.u8(super::priority::to_wire(self.priority)); match version.has_fetch_max_delay() { @@ -159,6 +164,7 @@ mod test { Fetch { instance: None, node: 0, + depth: 0, broadcast: Path::from_static("room"), track: Cow::Borrowed("video"), priority: 3, diff --git a/rs/moq-net/src/lite/instance.rs b/rs/moq-net/src/lite/instance.rs index f38022cf146..bcc0596a4eb 100644 --- a/rs/moq-net/src/lite/instance.rs +++ b/rs/moq-net/src/lite/instance.rs @@ -1,5 +1,5 @@ use crate::{ - Instance, + Instance, Path, coding::{DecodeError, Decoder, EncodeError, Encoder}, }; @@ -38,21 +38,33 @@ pub(super) fn decode_pin(r: &mut Decoder<'_>, version: Version) -> Result)>, - floor: Option, - }, - /// ROUTE_REQUEST, toward the neighbor whose route this node would use. - Request { to: N, id: Id, seqno: Seqno }, - /// The echo of a ROUTE_END this node applied. - Echo { to: N, id: Id }, -} - -/// The route a node selected for an identity. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub(crate) struct Selected { - /// The neighbor it goes through, or `None` when this node is the identity. - pub next: Option, - pub seqno: Seqno, - /// This node's metric. - pub metric: Metric, - /// Whether an upstream link is on the path. - pub down_only: bool, -} - /// How a node's reader sees one announced claim. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) struct Shown { @@ -279,213 +199,6 @@ pub(crate) struct Event { pub forgot: u32, } -/// A route a node holds from one neighbor. -#[derive(Clone, Debug)] -struct Entry { - /// The last finite advertisement. - advert: Advert, - held: bool, - /// Its hops name this node: a copy of a route this node sent, come back. This - /// node never uses it, which the distance alone cannot ensure once it is gone. - crossed: bool, -} - -/// A seqno request a node keeps until a route at least that new is selected or -/// no route is left. -#[derive(Clone, Debug)] -struct Pending { - seqno: Seqno, - /// Neighbors it forwards the request for; empty when only its own. - from: BTreeSet, - /// The neighbor it last sent the request to. - asked: Option, -} - -/// The last view a neighbor sent for one identity. -#[derive(Clone, Debug)] -struct Heard { - /// As the neighbor sent it, to apply the next batch to. - raw: View, - /// What it certifies here, where that is not `raw`: presented within the - /// neighbor's scope, or what it certified before its route ended. - kept: Option>, - /// The route ended: the next batch starts a fresh view. - ended: bool, - /// The interest generation it answers. - interest: u32, - /// The claims within the view's floors where the merged view holds something - /// else: the only ones merging the view can change. - differs: BTreeSet, -} - -impl Heard { - fn new(interest: u32) -> Self { - Self { - raw: View::default(), - kept: None, - ended: false, - interest, - differs: BTreeSet::new(), - } - } - - /// What the neighbor certifies here. - fn view(&self) -> &View { - self.kept.as_ref().unwrap_or(&self.raw) - } -} - -impl Heard { - /// Note whether `claim` differs from `merged`'s. - fn compare(&mut self, merged: Option<&View>, claim: &Path) { - let view = self.view(); - let mine = view.items.get(claim); - let differs = view.as_of(claim).is_some() && merged.and_then(|merged| merged.items.get(claim)) != mine; - match differs { - true => self.differs.insert(claim.clone()), - false => self.differs.remove(claim), - }; - } - - /// Compare every claim against `merged`. - fn resync(&mut self, merged: Option<&View>) { - let view = self.view(); - let mut claims: BTreeSet = view.items.keys().cloned().collect(); - if let Some(merged) = merged { - for region in view.floors.keys() { - let exact = merged.items.get_key_value(region); - claims.extend( - exact - .into_iter() - .chain(below(&merged.items, region)) - .map(|(claim, _)| claim.clone()), - ); - } - } - self.differs.clear(); - for claim in &claims { - self.compare(merged, claim); - } - } -} - -#[derive(Clone, Debug)] -struct Peer { - link: Link, - /// The link's min RTT, which routes learned here add; `None` while unmeasured. - rtt: Option, - /// The neighbor sent GOAWAY: rank it below every other eligible route. - draining: bool, - /// The prefix the neighbor asked this node for, once it opened a stream. - watch: Option, - /// The prefix this node asked the neighbor for, which its floors certify; the - /// node's interest when unset. - asked: Option, -} - -impl Peer { - /// Whether a request for a claim at `claim` may go through this neighbor: the - /// claim lies within what this node asked it for and within its scope. A route - /// reaches an origin node, not each of its claims. - fn carries(&self, claim: &Path) -> bool { - self.asked.as_ref().is_none_or(|asked| within(claim, asked)) - && (self.link.scope.as_ref()).is_none_or(|scope| scope.regions.iter().any(|region| within(claim, region))) - } -} - -/// The claims of an identity changed since it was last recomputed: the ones named, -/// or any when `all` is set. -#[derive(Clone, Debug, Default)] -struct Dirty { - all: bool, - claims: BTreeSet, -} - -/// What an identity's presentation and reader view depend on besides its claims: -/// while it stays the same, only the claims marked [`Dirty`] can change. -#[derive(Clone, Debug, PartialEq, Eq)] -struct Ctx { - own: bool, - /// Routed, starved, or holding a recoverable route. - live: bool, - /// A usable route is selected. - usable: bool, - /// The certifiers, the selected next hop first, each with whether its route is - /// held and whether its stream answers the current interest. - certifiers: Vec<(N, bool, bool)>, -} - -/// An identity this node serves itself. -#[derive(Clone, Debug)] -struct Origin { - seqno: Seqno, - /// Whether the node declared it, rather than forging it with no claims. - declared: bool, - view: View, -} - -/// Retained state, summed over the node, for budgets and leak checks. -#[cfg(test)] -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] -pub(crate) struct Retained { - pub items: usize, - pub floors: usize, - pub heard: usize, - pub sent: usize, - pub echoes: usize, - pub pending: usize, - pub distances: usize, - pub certified: usize, -} - -#[derive(Clone)] -pub(crate) struct Router { - /// How this node names itself in the hops of a route it forwards. A rotation - /// leaves it alone, so a route that crossed this node always says so. - hop: Id, - neighbors: BTreeMap, - origins: BTreeMap>, - rx: BTreeMap>, - // Per-identity state is keyed identity first, so one identity's state is - // one range and a message costs only the identity it names. - /// For a key this node qualified, the identity the neighbor named. - filed: BTreeMap<(Id, N), Id>, - /// Babel's feasibility distance per identity. - fd: BTreeMap, - /// The hops of the route this node last sent under a qualified key, kept as long - /// as its distance: a route that extends them came through this node's own. - advertised: BTreeMap>, - sent: BTreeMap<(Id, N), Reach>, - /// ROUTE_ENDs sent to each neighbor and not yet echoed back. Until they all - /// return, a neighbor may still hold this node's advertisement, so the - /// feasibility distance stays. - unacked: BTreeMap<(Id, N), u32>, - pending: BTreeMap>, - /// The prefix this node asks its neighbors for announcements under. - watch: Path, - /// Counts this node's interest changes. - interest: u32, - /// Each origin's claims as this node knows them. - views: BTreeMap>, - /// After the interest widened, the regions each view is known current in: - /// the old interest, plus what each batch answering the new one certified. - /// The rest is hidden until certified. - current: BTreeMap>, - heard: BTreeMap<(Id, N), Heard>, - sent_views: BTreeMap<(Id, N), View>, - reader: Reader, - /// The claims changed per identity since it was last recomputed. - dirty: BTreeMap, - /// Each identity's context as last recomputed: a change marks every claim dirty. - ctxs: BTreeMap>, - /// Mints a fresh identity on rotation. - rng: u64, - /// Identities recomputed, so fixtures can bound the work per message. - #[cfg(test)] - touched: usize, - out: Vec>, -} - /// A refused input. #[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] pub(crate) enum Error { @@ -500,1586 +213,3 @@ pub(crate) enum Error { #[error("no such interest prefix")] UnknownPrefix, } - -impl Router { - /// A node with no sessions, naming itself `hop` in the routes it forwards and - /// minting rotated identities from `seed`. - pub fn new(hop: Id, seed: u64) -> Self { - Self { - hop, - neighbors: BTreeMap::new(), - origins: BTreeMap::new(), - rx: BTreeMap::new(), - filed: BTreeMap::new(), - fd: BTreeMap::new(), - advertised: BTreeMap::new(), - sent: BTreeMap::new(), - unacked: BTreeMap::new(), - pending: BTreeMap::new(), - watch: Path::empty(), - interest: 0, - views: BTreeMap::new(), - current: BTreeMap::new(), - heard: BTreeMap::new(), - sent_views: BTreeMap::new(), - reader: Reader::default(), - dirty: BTreeMap::new(), - ctxs: BTreeMap::new(), - rng: seed | 1, - #[cfg(test)] - touched: 0, - out: Vec::new(), - } - } - - /// Take the identities recomputed since the last call, whose selection may have - /// changed, each with the claims whose shown state did. - pub fn changed(&mut self) -> BTreeMap> { - self.reader.changed() - } - - /// Take the messages queued for neighbors, in order. - pub fn drain(&mut self) -> Vec> { - std::mem::take(&mut self.out) - } - - /// Take the reader events, in order. - pub fn events(&mut self) -> Vec> { - self.reader.events() - } - - // --- Sessions --- - - /// A session to `neighbor` opened. - pub fn connect(&mut self, neighbor: N, link: Link) -> Result<(), Error> { - if link.cost == 0 { - return Err(Error::ZeroCost); - } - self.neighbors.insert( - neighbor, - Peer { - link, - rtt: None, - draining: false, - watch: None, - asked: None, - }, - ); - // Nothing is sent or learned until a stream opens. - Ok(()) - } - - /// The sessions to `neighbors` closed: everything learned on them goes, and each - /// counts as the echo of every ROUTE_END sent on it. Closing them together - /// recomputes each identity once, rather than once per neighbor. - pub fn disconnect(&mut self, neighbors: impl IntoIterator) { - let gone: BTreeSet = (neighbors.into_iter()) - .filter(|neighbor| self.neighbors.remove(neighbor).is_some()) - .collect(); - if gone.is_empty() { - return; - } - let mut ids = BTreeSet::new(); - for (id, routes) in self.rx.iter_mut() { - let before = routes.len(); - routes.retain(|next, _| !gone.contains(next)); - if routes.len() != before { - ids.insert(*id); - } - } - self.rx.retain(|_, routes| !routes.is_empty()); - for (id, pending) in self.pending.iter_mut() { - let before = pending.from.len(); - pending.from.retain(|from| !gone.contains(from)); - if pending.from.len() != before { - ids.insert(*id); - } - } - let mut on = |(id, n): &(Id, N)| { - let on = gone.contains(n); - if on { - ids.insert(*id); - } - !on - }; - self.filed.retain(|key, _| on(key)); - self.heard.retain(|key, _| on(key)); - self.unacked.retain(|key, _| on(key)); - self.sent.retain(|key, _| on(key)); - self.sent_views.retain(|key, _| on(key)); - self.update(ids); - } - - /// `neighbor` opened a stream asking for announcements under `prefix`: it - /// gets a complete snapshot. - pub fn peer_interest(&mut self, neighbor: N, prefix: Path) -> Result<(), Error> { - let peer = self.neighbors.get_mut(&neighbor).ok_or(Error::UnknownNeighbor)?; - peer.watch = Some(prefix); - self.sent_views.retain(|(_, to), _| *to != neighbor); - self.update_all(); - Ok(()) - } - - /// This node asked `neighbor` for announcements under `prefix`: the region its - /// floors certify. - pub fn asks(&mut self, neighbor: N, prefix: Path) -> Result<(), Error> { - let peer = self.neighbors.get_mut(&neighbor).ok_or(Error::UnknownNeighbor)?; - peer.asked = Some(prefix); - Ok(()) - } - - /// `neighbor` sent GOAWAY: it ranks below every other eligible route while - /// its session stays up. - pub fn goaway(&mut self, neighbor: N) { - if let Some(peer) = self.neighbors.get_mut(&neighbor) { - peer.draining = true; - } - self.update(self.through(neighbor)); - } - - /// The link to `neighbor` has `rtt`, or `None` unmeasured: every route learned - /// there adds it. - pub fn rtt(&mut self, neighbor: N, rtt: Option) { - let Some(peer) = self.neighbors.get_mut(&neighbor) else { - return; - }; - if peer.rtt == rtt { - return; - } - peer.rtt = rtt; - self.update(self.through(neighbor)); - } - - /// The identities `neighbor` sent a route for. - fn through(&self, neighbor: N) -> Vec { - (self.rx.iter()) - .filter(|(_, routes)| routes.contains_key(&neighbor)) - .map(|(id, _)| *id) - .collect() - } - - #[cfg(test)] - /// Ask every neighbor for announcements under `prefix` instead, returning - /// the new interest generation. Each neighbor answers on a new stream, and a - /// batch from an older one is discarded. - pub fn watch(&mut self, prefix: Path) -> u32 { - if !within(&prefix, &self.watch) { - let old = self.watch.clone(); - for id in self.views.keys().filter(|id| !self.origins.contains_key(id)) { - self.current.entry(*id).or_insert_with(|| vec![old.clone()]); - } - } - self.current - .retain(|_, regions| !regions.iter().any(|region| within(&prefix, region))); - self.watch = prefix; - self.interest += 1; - let ids = self.ids(); - for id in &ids { - self.mark_all(*id); - } - self.update(ids); - self.interest - } - - /// The current interest generation. - pub fn interest(&self) -> u32 { - self.interest - } - - // --- The origin's own identity --- - - /// Start originating `id` at `seqno`, with no claims. - pub fn originate(&mut self, id: Id, seqno: Seqno) { - debug_assert_eq!(id & QUALIFIED, 0, "a declared identity is never qualified"); - let mut view = View::default(); - view.floors.insert(Path::empty(), 0); - self.origins.insert( - id, - Origin { - seqno, - declared: true, - view, - }, - ); - self.mark_all(id); - self.update([id]); - } - - /// Advertise `id` as if serving it, with one uncertified claim: what a forwarder - /// that lies about an identity does. - #[cfg(test)] - pub fn forge(&mut self, id: Id, seqno: Seqno, claim: Path, item: A) { - let mut view = View::default(); - view.items.insert(claim, item); - self.origins.insert( - id, - Origin { - seqno, - declared: false, - view, - }, - ); - self.mark_all(id); - self.update([id]); - } - - /// The identity this node declares. - pub fn declared(&self) -> Option { - self.origins - .iter() - .find(|(_, origin)| origin.declared) - .map(|(id, _)| *id) - } - - #[cfg(test)] - /// The declared identity's seqno. - pub fn seqno(&self) -> Option { - self.origins.values().find(|origin| origin.declared).map(|o| o.seqno) - } - - #[cfg(test)] - /// The declared identity's claim revision. - pub fn revision(&self) -> Option { - let origin = self.origins.values().find(|origin| origin.declared)?; - origin.view.floors.get(&Path::empty()).copied() - } - - /// Advance the declared identity's seqno, rotating it rather than reach [`MAX`]. - pub fn bump(&mut self) -> Result<(), Error> { - let id = self.declared().ok_or(Error::NoIdentity)?; - let seqno = self.origins[&id].seqno; - if seqno + 1 == MAX { - self.rotate(id); - } else { - self.origins.get_mut(&id).expect("declared").seqno = seqno + 1; - self.update([id]); - } - Ok(()) - } - - /// Start or update a claim of the declared identity. - pub fn announce(&mut self, claim: Path, item: A) -> Result<(), Error> { - self.change(claim, Some(item)) - } - - /// Withdraw a claim of the declared identity. - pub fn unannounce(&mut self, claim: Path) -> Result<(), Error> { - self.change(claim, None) - } - - fn change(&mut self, claim: Path, item: Option) -> Result<(), Error> { - let id = self.declared().ok_or(Error::NoIdentity)?; - let rev = self.origins[&id].view.floors[&Path::empty()] + 1; - if rev == MAX { - self.rotate(id); - return self.change(claim, item); - } - let view = &mut self.origins.get_mut(&id).expect("declared").view; - view.floors.insert(Path::empty(), rev); - match item { - Some(item) => view.items.insert(claim.clone(), item), - None => view.items.remove(&claim), - }; - self.mark(id, [claim]); - self.update([id]); - Ok(()) - } - - /// Jump the declared identity's revision to `rev`, as a long-lived origin's - /// claim changes reach it. - #[cfg(test)] - pub fn age(&mut self, rev: Rev) { - let id = self.declared().expect("declared"); - let view = &mut self.origins.get_mut(&id).expect("declared").view; - view.floors.insert(Path::empty(), rev); - } - - /// Retire the declared identity for a fresh one, announcing its claims again - /// there. Every neighbor lets the old one go. - fn rotate(&mut self, old: Id) { - let origin = self.origins.remove(&old).expect("declared"); - let id = loop { - // xorshift64: a fresh identity within a generated hop id's range. - self.rng ^= self.rng << 13; - self.rng ^= self.rng >> 7; - self.rng ^= self.rng << 17; - let id = self.rng & GENERATED; - if id != 0 && id != old && !self.origins.contains_key(&id) { - break id; - } - }; - let rev = origin.view.items.len() as Rev; - let mut view = origin.view; - view.floors = BTreeMap::from([(Path::empty(), rev)]); - self.origins.insert( - id, - Origin { - seqno: 0, - declared: true, - view, - }, - ); - // The views heard for it merge from here on. - self.resync(old); - self.mark_all(old); - self.mark_all(id); - self.update([old, id]); - } - - // --- Received messages --- - - /// ROUTE_START or ROUTE_UPDATE from `from`, or ROUTE_END for `None`. - pub fn route(&mut self, from: N, named: Id, reach: Option) { - if !self.neighbors.contains_key(&from) { - return; - } - // Echo every ROUTE_END once applied, filed or refused. - let ended = reach.is_none(); - if ended { - self.out.push(Out::Echo { to: from, id: named }); - } - let Some(id) = self.key(from, named) else { - return; - }; - if id != named { - match reach { - Some(_) => self.filed.insert((id, from), named), - None => self.filed.remove(&(id, from)), - }; - } - let routes = self.rx.entry(id).or_default(); - // Whether only the route's hops or RTT changed, and not whether this node may - // use it: that moves no route here. - let mut same = false; - match reach { - Some(Reach::Finite(advert)) => { - let crossed = advert.hops.contains(&self.hop); - let full = |advert: &Advert| advert.hops.len() >= MAX_HOPS; - same = routes.get(&from).is_some_and(|was| { - let (old, new) = (&was.advert, &advert); - !was.held - && was.crossed == crossed - && full(old) == full(new) - && (old.seqno, old.metric, old.down_only) == (new.seqno, new.metric, new.down_only) - }); - let entry = Entry { - advert, - held: false, - crossed, - }; - routes.insert(from, entry); - } - Some(Reach::Held) => { - if let Some(entry) = routes.get_mut(&from) { - entry.held = true; - } - } - None => { - routes.remove(&from); - } - } - if routes.is_empty() { - self.rx.remove(&id); - } - // The sender forgets what it sent for a route it ended, so a restart - // starts from an empty view: apply the next batch to that, keeping the - // last view only for what it certified. An uncertified claim lives only - // while the stream carries it, and the restart's batch names only what it - // still carries, so one kept here would never be withdrawn. - if ended && let Some(heard) = self.heard.get_mut(&(id, from)) { - let raw = std::mem::take(&mut heard.raw); - let kept = heard.kept.get_or_insert(raw); - let floors = &kept.floors; - kept.items.retain(|claim, _| above(floors, claim).next().is_some()); - // What it drops lay outside every floor, so nothing it differed in. - heard.ended = true; - } - self.recompute([id], !same); - } - - /// A batch of announcement changes for `named` from `from`, closed by its - /// floor, on the stream answering interest generation `interest`. A floor - /// certifies the whole region this node asked for at that revision; `None` - /// certifies nothing, so the stream's claims stay only while it carries them. - pub fn batch(&mut self, from: N, named: Id, changes: Vec<(Path, Option)>, floor: Option, interest: u32) { - let Some(peer) = self.neighbors.get(&from) else { - return; - }; - // A batch from a stream the node has since replaced. - if interest != self.interest { - return; - } - let Some(id) = self.key(from, named) else { - return; - }; - let scope = peer.link.scope.as_ref().map(|scope| scope.regions.clone()); - let region = peer.asked.clone().unwrap_or_else(|| self.watch.clone()); - let floors = floor.map(|rev| BTreeMap::from([(region, rev)])).unwrap_or_default(); - let heard = self.heard.entry((id, from)).or_insert_with(|| Heard::new(interest)); - // What the batch can change in this node's view: the claims it names, and - // the claims a scope presents at its regions. A new stream, or certified - // regions that moved, can change any claim. - let mut all = heard.interest != interest; - // A new stream, or a restarted route, starts from an empty view. - let fresh = heard.interest != interest || heard.ended; - if fresh { - heard.raw = View::default(); - heard.interest = interest; - heard.ended = false; - } - let mut dirty: Vec = changes.iter().map(|(claim, _)| claim.clone()).collect(); - let regions = heard.view().floors.keys().cloned().collect::>(); - heard.raw.apply(changes, floors); - match &scope { - Some(scope) => { - dirty.extend(scope.iter().cloned()); - let Heard { raw, kept, .. } = heard; - let view = match fresh { - true => kept.insert(View::default()), - false => kept.get_or_insert_default(), - }; - match fresh { - true => view.represent(raw, scope, raw.items.keys()), - false => view.represent(raw, scope, &dirty), - } - } - None => heard.kept = None, - } - #[cfg(test)] - if let Some(scope) = &scope { - assert_eq!(*heard.view(), heard.raw.present_all(scope), "incremental scoped view"); - } - let moved = !heard.view().floors.keys().eq(regions.iter()); - all |= moved; - // This node's own identities merge nothing, so they track no differences. - if !self.origins.contains_key(&id) { - let merged = self.views.get(&id); - match fresh || moved { - true => heard.resync(merged), - false => { - for claim in &dirty { - heard.compare(merged, claim); - } - } - } - } - let certifies: Vec = heard.view().floors.keys().cloned().collect(); - - // The batch certifies the regions its floors name. - for region in &certifies { - let Some(regions) = self.current.get_mut(&id) else { - break; - }; - all = true; - if *region == self.watch { - self.current.remove(&id); - } else if !regions.iter().any(|known| within(region, known)) { - regions.retain(|known| !within(known, region)); - regions.push(region.clone()); - } - } - if !self.origins.contains_key(&id) { - let merged = self.views.entry(id).or_default(); - let heard = &self.heard[&(id, from)]; - let regions = merged.floors.keys().cloned().collect::>(); - #[cfg(test)] - let full = { - let mut full = merged.clone(); - full.merge(heard.view()); - full - }; - let changed = merged.merge_claims(heard.view(), &heard.differs); - #[cfg(test)] - assert_eq!(*merged, full, "incremental merge"); - all |= !merged.floors.keys().eq(regions.iter()); - // The batch certifies the interest, so what lies outside can go. While every - // floor lies within the interest, nothing does. - let outside = !merged.floors.keys().all(|region| within(region, &self.watch)); - #[cfg(test)] - if !outside { - assert_eq!(merged.retire(&self.watch), *merged, "nothing to retire"); - } - let retired = outside && { - let kept = merged.retire(&self.watch); - let retired = kept != *merged; - *merged = kept; - retired - }; - if retired { - all = true; - let watch = BTreeSet::from([self.watch.clone()]); - self.forgot_outside(id, &watch); - self.resync(id); - } else { - self.compare(id, &changed); - } - dirty.extend(changed); - } - match all { - true => self.mark_all(id), - false => self.mark(id, dirty), - } - self.update([id]); - } - - /// ROUTE_REQUEST from `from` for `id`, as this node advertised it. - pub fn request(&mut self, from: N, id: Id, seqno: Seqno) { - if !self.neighbors.contains_key(&from) { - return; - } - if let Some(origin) = self.origins.get_mut(&id) { - // Only the origin advances its seqno to answer. - if seqno == MAX { - if origin.declared { - self.rotate(id); - } - } else { - origin.seqno = origin.seqno.max(seqno); - self.update([id]); - } - return; - } - if self.selected(id).is_some_and(|route| route.seqno >= seqno) { - // Already advertised to the requester. - return; - } - let pending = self.pending.entry(id).or_insert(Pending { - seqno, - from: BTreeSet::new(), - asked: None, - }); - if seqno > pending.seqno { - pending.seqno = seqno; - pending.asked = None; - } - pending.from.insert(from); - self.update([id]); - } - - /// The echo of a ROUTE_END this node sent `from` for `id`. - pub fn echo(&mut self, from: N, id: Id) { - let Some(left) = self.unacked.get_mut(&(id, from)) else { - // Unsolicited: nothing of ours to drain. - return; - }; - *left -= 1; - if *left == 0 { - self.unacked.remove(&(id, from)); - } - self.update([id]); - } - - /// Forget `id`'s routes and announcements. The feasibility distance goes on - /// its own once drained. - pub fn collect(&mut self, id: Id) { - self.rx.remove(&id); - let filed: Vec<_> = Self::of(&self.filed, id).map(|(key, _)| *key).collect(); - for key in filed { - self.filed.remove(&key); - } - self.views.remove(&id); - self.current.remove(&id); - let heard: Vec<_> = Self::of(&self.heard, id).map(|(key, _)| *key).collect(); - for key in heard { - self.heard.remove(&key); - } - self.pending.remove(&id); - self.forgot_outside(id, &BTreeSet::new()); - self.mark_all(id); - self.update([id]); - } - - // --- Queries --- - - /// The route this node selected for `id`. - pub fn selected(&self, id: Id) -> Option> { - if let Some(origin) = self.origins.get(&id) { - return Some(Selected { - next: None, - seqno: origin.seqno, - metric: 0, - down_only: false, - }); - } - type Rank = (bool, Reverse, Metric, N); - let mut best: Option<(Rank, Selected)> = None; - for (&next, entry) in self.rx.get(&id)? { - if entry.held || !self.eligible(id, entry) { - continue; - } - let peer = &self.neighbors[&next]; - let Some(metric) = self.metric(next, entry) else { - continue; - }; - let rank = (peer.draining, Reverse(peer.link.priority), metric, next); - if best.as_ref().is_some_and(|(best, _)| *best <= rank) { - continue; - } - let selected = Selected { - next: Some(next), - seqno: entry.advert.seqno, - metric, - down_only: peer.link.upstream || entry.advert.down_only, - }; - best = Some((rank, selected)); - } - best.map(|(_, selected)| selected) - } - - /// Every usable feasible route to `id`, with this node's metric and RTT through - /// each. Any of them forwards without a loop, if it [carries](Self::carries) the - /// claim. - pub fn routes(&self, id: Id) -> Vec<(N, Metric, Option)> { - if self.origins.contains_key(&id) { - return Vec::new(); - } - (self.rx.get(&id).into_iter().flatten()) - .filter(|(_, entry)| !entry.held && self.eligible(id, entry)) - .filter_map(|(&next, entry)| { - let rtt = through_rtt(&entry.advert, &self.neighbors[&next]); - Some((next, self.metric(next, entry)?, rtt)) - }) - .collect() - } - - /// This node's RTT to `id` along `route`: 0 at the identity's own node. - fn route_rtt(&self, id: Id, route: &Selected) -> Option { - let Some(next) = route.next else { - return Some(Duration::ZERO); - }; - let entry = self.rx.get(&id)?.get(&next)?; - through_rtt(&entry.advert, &self.neighbors[&next]) - } - - /// Whether a request for a claim at `claim` may go through `next`. - pub fn carries(&self, next: N, claim: &Path) -> bool { - self.neighbors.get(&next).is_some_and(|peer| peer.carries(claim)) - } - - /// The prefix this node asked `next` for, outside which it carries no claim: the - /// root when it asked for none. - pub fn asked(&self, next: N) -> Path { - (self.neighbors.get(&next)) - .and_then(|peer| peer.asked.clone()) - .unwrap_or_default() - } - - #[cfg(test)] - /// The route this node would offer an upstream link for `id`. - pub fn export(&self, id: Id) -> Option> { - self.selected(id).filter(|route| !route.down_only) - } - - #[cfg(test)] - /// Whether this node holds a reachable route to `id` but none is feasible. - pub fn starved(&self, id: Id) -> bool { - self.selected(id).is_none() && self.infeasible(id).next().is_some() - } - - #[cfg(test)] - /// Whether this node, with no usable route to `id`, holds a held route it - /// could recover through. - pub fn grounded(&self, id: Id) -> bool { - self.selected(id).is_none() && self.recoverable(id) - } - - /// Whether this node holds any route state for `id`. - pub fn holds(&self, id: Id) -> bool { - self.rx.contains_key(&id) - } - - #[cfg(test)] - /// Whether this node keeps a feasibility distance for `id`. - pub fn remembers(&self, id: Id) -> bool { - self.fd.contains_key(&id) - } - - #[cfg(test)] - /// Whether this node keeps no state at all for `id`, beyond the changes - /// [`Self::changed`] has yet to take. - pub fn keeps_nothing(&self, id: Id) -> bool { - fn none(map: &BTreeMap<(K, N), V>, id: K) -> bool { - !map.keys().any(|(key, _)| *key == id) - } - !self.origins.contains_key(&id) - && !self.rx.contains_key(&id) - && !self.fd.contains_key(&id) - && !self.advertised.contains_key(&id) - && !self.pending.contains_key(&id) - && !self.views.contains_key(&id) - && !self.current.contains_key(&id) - && !self.dirty.contains_key(&id) - && !self.ctxs.contains_key(&id) - && none(&self.filed, id) - && none(&self.sent, id) - && none(&self.unacked, id) - && none(&self.heard, id) - && none(&self.sent_views, id) - && self.reader.claims(id).next().is_none() - } - - #[cfg(test)] - /// How many identities this node holds routes to. - pub fn identities(&self) -> usize { - self.rx.len() - } - - #[cfg(test)] - /// How many qualified keys this node remembers the forwarded name of. - pub fn aliases(&self) -> usize { - self.filed.len() - } - - #[cfg(test)] - /// How many region floors this node keeps for `id`. - pub fn floors(&self, id: Id) -> usize { - self.views.get(&id).map_or(0, |view| view.floors.len()) - } - - /// The view this node keeps for `id`, if any. - pub fn view(&self, id: Id) -> Option<&View> { - self.origins - .get(&id) - .map(|origin| &origin.view) - .or_else(|| self.views.get(&id)) - } - - /// Every other origin's view this node keeps. - #[cfg(test)] - pub fn views(&self) -> impl Iterator)> { - self.views - .iter() - .filter(|(id, _)| !self.origins.contains_key(id)) - .map(|(id, view)| (*id, view)) - } - - /// How many identities this node has recomputed. - #[cfg(test)] - pub fn touched(&self) -> usize { - self.touched - } - - /// Hash what decides this node's future, leaving out the work counter. Its output - /// and reader events must already be taken. - #[cfg(test)] - pub fn fingerprint(&self, hasher: &mut impl std::hash::Hasher) { - let Self { - hop, - neighbors, - origins, - rx, - filed, - fd, - advertised, - sent, - unacked, - pending, - watch, - interest, - views, - current, - heard, - sent_views, - reader, - dirty, - ctxs, - rng, - touched: _, - out, - } = self; - debug_assert!(out.is_empty(), "output not taken"); - digest(hasher, &(hop, neighbors, origins, rx, filed, fd, advertised, sent)); - digest(hasher, &(unacked, pending, watch, interest, views, current, heard)); - digest(hasher, &(sent_views, reader, dirty, ctxs, rng)); - } - - /// What this node's reader sees. - pub fn shown(&self) -> &BTreeMap<(Id, Path), Shown> { - self.reader.shown() - } - - #[cfg(test)] - /// Whether this node shows `claim` from `id`, and whether it is available. - pub fn shows(&self, id: Id, claim: &Path) -> Option { - self.reader.get(id, claim).map(|shown| shown.available) - } - - #[cfg(test)] - pub fn retained(&self) -> Retained { - let mut out = Retained::default(); - for (id, view) in &self.views { - if !self.origins.contains_key(id) { - out.items += view.items.len(); - out.floors += view.floors.len(); - } - } - out.heard = self.heard.values().map(|heard| heard.raw.items.len()).sum(); - out.sent = self.sent_views.values().map(|view| view.items.len()).sum(); - out.echoes = self.unacked.values().map(|&count| count as usize).sum(); - out.pending = self.pending.len(); - out.distances = self.fd.keys().filter(|id| !self.origins.contains_key(id)).count(); - out.certified = self.current.values().map(Vec::len).sum(); - out - } - - // --- Internals --- - - /// The key this node files `named` under when `from` sends it, or `None` - /// when refused: a scoped neighbor's identities are all qualified, and one - /// already qualified is refused rather than qualified again. - fn key(&self, from: N, named: Id) -> Option { - match &self.neighbors.get(&from)?.link.scope { - Some(scope) => { - if named & QUALIFIED != 0 { - return None; - } - let key = (scope.qualify)(named); - debug_assert_ne!(key & QUALIFIED, 0, "a qualified key sets QUALIFIED"); - Some(key | QUALIFIED) - } - None => Some(named), - } - } - - /// The identity this node names for `id` in a request to `next`: the one - /// `next` advertised, if this node qualified it. - pub fn name(&self, next: N, id: Id) -> Id { - self.filed.get(&(id, next)).copied().unwrap_or(id) - } - - /// Babel's feasibility condition against this node's feasibility distance. - fn feasible(&self, id: Id, advert: &Advert) -> bool { - match self.fd.get(&id) { - None => true, - Some(&(seqno, metric)) => advert.seqno > seqno || (advert.seqno == seqno && advert.metric < metric), - } - } - - /// Whether this node may select or certify through `entry`: a feasible route that - /// did not cross it. One that crossed it and is feasible all the same is no route - /// at all, since no seqno would change where it has been. - fn eligible(&self, id: Id, entry: &Entry) -> bool { - !entry.crossed && self.feasible(id, &entry.advert) - } - - /// This node's metric through `entry`, the route `next` sent, or `None` when it is - /// unreachable, never saturated: the metric overflows, or the route already - /// crossed as many nodes as one may. - fn metric(&self, next: N, entry: &Entry) -> Option { - if entry.advert.hops.len() >= MAX_HOPS { - return None; - } - entry.advert.metric.checked_add(self.neighbors[&next].link.cost) - } - - /// The usable but infeasible routes for `id`, with their metrics: this node holds - /// the identity while it has one. One that crossed this node counts like any other: - /// its sender routes through this node, and moves once it hears this node has no - /// route. Under a qualified key that holds only if the route extends the one this - /// node last sent under the key. Otherwise a relay past this node derived the key - /// from this node's route to another identity, which stands whatever happens to - /// this one, and waiting on it would never end. - fn infeasible(&self, id: Id) -> impl Iterator + '_ { - let sent = self.advertised.get(&id); - let derived = move |entry: &Entry| match id & QUALIFIED { - 0 => true, - _ => sent.is_some_and(|sent| entry.advert.hops.starts_with(sent)), - }; - (self.rx.get(&id).into_iter().flatten()) - .filter(move |(_, entry)| !entry.held && !self.feasible(id, &entry.advert)) - .filter(move |(_, entry)| !entry.crossed || derived(entry)) - .filter_map(move |(&next, entry)| Some((self.metric(next, entry)?, next))) - } - - /// Where this node sends a request for `id`: its feasible route's next hop, - /// else its best infeasible route's, even when that neighbor asked. - fn target(&self, id: Id) -> Option { - self.selected(id) - .and_then(|route| route.next) - .or_else(|| self.infeasible(id).min().map(|(_, next)| next)) - } - - /// Whether this node holds a held route to `id` it could recover through. - /// One counts only if its last finite advertisement is feasible and - /// reachable here, so - /// holds follow a strictly decreasing metric toward a starved node and - /// cannot hold each other up. - fn recoverable(&self, id: Id) -> bool { - self.rx.get(&id).is_some_and(|routes| { - routes - .iter() - .any(|(&next, entry)| entry.held && self.reachable(id, next, entry)) - }) - } - - /// Whether this node, having selected `route` for `id`, keeps showing its - /// announcements: routed, starved, or holding a recoverable route. - fn live(&self, id: Id, route: Option>) -> bool { - route.is_some() || self.infeasible(id).next().is_some() || self.recoverable(id) - } - - /// The neighbors whose views can certify `id`: those whose last finite - /// advertisement is feasible and reachable here, held or not. Feasible edges strictly - /// decrease the metric toward the origin, so certification never circulates. - fn certifiers(&self, id: Id) -> Vec { - self.rx.get(&id).map_or_else(Vec::new, |routes| { - routes - .iter() - .filter(|&(&next, entry)| self.reachable(id, next, entry)) - .map(|(&next, _)| next) - .collect() - }) - } - - /// Whether `entry`'s last finite advertisement from `next` is feasible here - /// and reachable across the link, as selection requires of a usable route. - fn reachable(&self, id: Id, next: N, entry: &Entry) -> bool { - self.eligible(id, entry) && self.metric(next, entry).is_some() - } - - /// The regions of `id` that this node's certifiers certify, those whose - /// route is held only when `held`, or `None` while it holds no route or a - /// certifier has not yet answered the current interest. - fn certified(&self, id: Id, held: bool) -> Option> { - let routes = self.rx.get(&id)?; - let mut regions = BTreeSet::new(); - for next in self.certifiers(id) { - let heard = self - .heard - .get(&(id, next)) - .filter(|heard| heard.interest == self.interest)?; - if held || !routes[&next].held { - regions.extend(heard.view().floors.keys().cloned()); - } - } - Some(regions) - } - - /// Let go of every region of `id`'s merged view that no certifier certifies - /// any more, once each has answered the current interest. - fn prune(&mut self, id: Id) { - // A starved or held node keeps presence until it recovers or lets go. - if self.origins.contains_key(&id) || self.selected(id).is_none() { - return; - } - let Some(regions) = self.certified(id, true) else { - return; - }; - let Some(merged) = self.views.get(&id) else { - return; - }; - let lost = (merged.floors.iter()) - .filter(|(floor, _)| !within_any(floor, ®ions)) - .map(|(_, &rev)| rev) - .max(); - let Some(lost) = lost else { - return; - }; - // Keep what the view knows within the regions still certified. That is - // sound only if no lost floor is newer than every kept one: otherwise a - // claim above both was last learned through the lost floor, and is - // unknown at the revision the kept floors certify. - let kept = merged.restrict_all(®ions); - let (rebuilt, remembered) = if kept.floors.values().all(|&floor| floor >= lost) { - (kept, regions) - } else { - let mut rebuilt = View::default(); - for next in self.certifiers(id) { - rebuilt.merge(self.heard[&(id, next)].view()); - } - (rebuilt, BTreeSet::new()) - }; - self.views.insert(id, rebuilt); - self.resync(id); - self.forgot_outside(id, &remembered); - self.mark_all(id); - } - - /// Compare `claims` of every view heard for `id` against the merged view, once - /// they changed there. - fn compare(&mut self, id: Id, claims: &[Path]) { - let merged = self.views.get(&id); - for (_, heard) in self.heard.range_mut((id, N::MIN)..=(id, N::MAX)) { - for claim in claims { - heard.compare(merged, claim); - } - } - } - - /// Compare every view heard for `id` against the merged view, once it was replaced. - fn resync(&mut self, id: Id) { - let merged = self.views.get(&id); - for (_, heard) in self.heard.range_mut((id, N::MIN)..=(id, N::MAX)) { - heard.resync(merged); - } - } - - /// Count a forget against each shown claim of `id` overlapping none of `kept`. - fn forgot_outside(&mut self, id: Id, kept: &BTreeSet) { - self.reader.forget(id, |claim| within_any(claim, kept)); - } - - /// The view this node shows and forwards for `id`, having selected `route`: what - /// it knows certified, and each claim a certifier carries uncertified where no - /// floor it knows covers the claim. - fn adopted(&self, id: Id, route: Option>) -> Option>> { - if let Some(origin) = self.origins.get(&id) { - return Some(Cow::Borrowed(&origin.view)); - } - if !self.live(id, route) { - return None; - } - let merged = self.views.get(&id); - let mut loose = self.loose(id, route, merged).peekable(); - if loose.peek().is_none() { - return merged.map(Cow::Borrowed); - } - let mut view = merged.cloned().unwrap_or_default(); - for (claim, item, _) in loose { - view.items.entry(claim.clone()).or_insert_with(|| item.clone()); - } - Some(Cow::Owned(view)) - } - - /// The uncertified claims of `id` its certifiers carry on streams answering the - /// current interest where `merged` certifies nothing, the selected next hop's - /// first, and whether the carrier is held. - fn loose<'a>( - &'a self, - id: Id, - route: Option>, - merged: Option<&'a View>, - ) -> impl Iterator + 'a { - let mut certifiers = self.certifiers(id); - if let Some(next) = route.and_then(|route| route.next) { - certifiers.sort_by_key(|&n| n != next); - } - let routes = self.rx.get(&id); - certifiers.into_iter().flat_map(move |next| { - let held = routes.is_some_and(|routes| routes[&next].held); - self.heard - .get(&(id, next)) - .filter(|heard| heard.interest == self.interest) - .into_iter() - .flat_map(|heard| heard.view().uncertified()) - .filter(move |(claim, _)| merged.is_none_or(|merged| merged.as_of(claim).is_none())) - .map(move |(claim, item)| (claim, item, held)) - }) - } - - /// Whether this node sends `id` to `neighbor` at all: its own identities to - /// everyone, learned ones only to a proxied neighbor. - fn serves(&self, neighbor: &Peer, id: Id) -> bool { - neighbor.link.proxy || self.origins.contains_key(&id) - } - - /// One identity's entries in a map keyed identity first. - fn of(map: &BTreeMap<(Id, N), V>, id: Id) -> impl Iterator { - map.range((id, N::MIN)..=(id, N::MAX)) - } - - /// Every identity this node keeps any state for. - fn ids(&self) -> BTreeSet { - let mut ids: BTreeSet = self.rx.keys().copied().collect(); - ids.extend(self.origins.keys()); - ids.extend(self.fd.keys()); - ids.extend(self.views.keys()); - ids.extend(self.pending.keys()); - ids.extend(self.sent.keys().map(|(id, _)| *id)); - ids.extend(self.unacked.keys().map(|(id, _)| *id)); - ids.extend(self.heard.keys().map(|(id, _)| *id)); - ids.extend(self.reader.shown().keys().map(|(id, _)| *id)); - ids - } - - /// Recompute every identity: for a change that touches them all, such as a - /// new interest. - fn update_all(&mut self) { - let ids = self.ids(); - self.update(ids); - } - - /// Mark `claims` of `id` changed. - fn mark(&mut self, id: Id, claims: impl IntoIterator) { - let dirty = self.dirty.entry(id).or_default(); - if !dirty.all { - dirty.claims.extend(claims); - } - } - - /// Mark every claim of `id` changed. - fn mark_all(&mut self, id: Id) { - *self.dirty.entry(id).or_default() = Dirty { - all: true, - claims: BTreeSet::new(), - }; - } - - /// Recompute each of `ids`: its selection, the feasibility distance for what - /// this node advertises, what each neighbor is sent, its seqno request, and - /// what the reader sees. - fn update(&mut self, ids: impl IntoIterator) { - self.recompute(ids, true); - } - - /// [`Self::update`], letting go of what no neighbor certifies only when `prune`. A - /// neighbor's change of hops or RTT alone moves no certifier, so it lets go of - /// nothing: only a message that could have cost a region its certifier does. - fn recompute(&mut self, ids: impl IntoIterator, prune: bool) { - let peers: Vec = self - .neighbors - .iter() - .filter(|(_, peer)| peer.watch.is_some()) - .map(|(n, _)| *n) - .collect(); - let ids: BTreeSet = ids.into_iter().collect(); - #[cfg(test)] - { - self.touched += ids.len(); - } - for id in ids { - if prune { - self.prune(id); - } - self.record(id, &peers); - let ctx = self.ctx(id, self.selected(id)); - let mut dirty = self.dirty.remove(&id).unwrap_or_default(); - dirty.all |= self.ctxs.get(&id) != Some(&ctx); - self.advertise(id, &peers, &ctx, &dirty); - // Drop a feasibility distance once nothing here routes or advertises - // the identity and every neighbor echoed its ROUTE_END: no node then - // holds an advertisement from this one, so no older one can return as - // a loop. - if !self.rx.contains_key(&id) - && !self.origins.contains_key(&id) - && Self::of(&self.sent, id).next().is_none() - && Self::of(&self.unacked, id).next().is_none() - { - self.fd.remove(&id); - self.advertised.remove(&id); - } - self.requests(id); - self.show(id, &ctx, &dirty); - // Nothing to present or show is left to compare against. - let empty = !ctx.own && !ctx.live && !self.views.contains_key(&id); - match empty { - true => self.ctxs.remove(&id), - false => self.ctxs.insert(id, ctx), - }; - } - } - - /// The context `id`'s claims are presented and shown in, having selected `route`. - fn ctx(&self, id: Id, route: Option>) -> Ctx { - let mut certifiers = self.certifiers(id); - if let Some(next) = route.and_then(|route| route.next) { - certifiers.sort_by_key(|&n| n != next); - } - let routes = self.rx.get(&id); - let certifiers = (certifiers.into_iter()) - .map(|next| { - let held = routes.is_some_and(|routes| routes[&next].held); - let answers = (self.heard.get(&(id, next))).is_some_and(|heard| heard.interest == self.interest); - (next, held, answers) - }) - .collect(); - Ctx { - own: self.origins.contains_key(&id), - live: self.live(id, route), - usable: route.is_some(), - certifiers, - } - } - - /// The floors of the view this node shows and forwards for `id`. - fn adopted_floors(&self, id: Id) -> Option<&BTreeMap> { - self.view(id).map(|view| &view.floors) - } - - /// The claim at `claim` in the view this node shows and forwards for `id`, as - /// [`Self::adopted`] has it, without building the view. - fn adopted_at<'a>(&'a self, id: Id, ctx: &Ctx, claim: &Path) -> Option<&'a A> { - if let Some(origin) = self.origins.get(&id) { - return origin.view.items.get(claim); - } - if !ctx.live { - return None; - } - let merged = self.views.get(&id); - if let Some(item) = merged.and_then(|merged| merged.items.get(claim)) { - return Some(item); - } - self.loose_at(id, ctx, merged, claim).map(|(item, _)| item) - } - - /// The uncertified claim at `claim` its certifiers carry where `merged` certifies - /// nothing, as [`Self::loose`] finds it first, and whether a carrier is not held. - fn loose_at<'a>(&'a self, id: Id, ctx: &Ctx, merged: Option<&View>, claim: &Path) -> Option<(&'a A, bool)> { - if merged.is_some_and(|merged| merged.as_of(claim).is_some()) { - return None; - } - let mut found: Option<(&A, bool)> = None; - for &(next, held, answers) in &ctx.certifiers { - if !answers { - continue; - } - let view = self.heard[&(id, next)].view(); - if let Some(item) = view.items.get(claim) - && view.as_of(claim).is_none() - { - let found = found.get_or_insert((item, false)); - found.1 |= !held; - } - } - found - } - - /// Whether `id` meets the interest `watch`: a stream answering it is presented a - /// claim of the view this node forwards, inside the interest or covering it. - fn meets(&self, id: Id, ctx: &Ctx, watch: &Path) -> bool { - let Some(floors) = self.adopted_floors(id).or(ctx.live.then_some(&EMPTY_FLOORS)) else { - return false; - }; - if covering(floors, watch, |claim| self.adopted_at(id, ctx, claim)).is_some() { - return true; - } - if let Some(origin) = self.origins.get(&id) { - return below(&origin.view.items, watch).next().is_some(); - } - if !ctx.live { - return false; - } - let merged = self.views.get(&id); - if merged.is_some_and(|merged| below(&merged.items, watch).next().is_some()) { - return true; - } - ctx.certifiers - .iter() - .filter(|&&(_, _, answers)| answers) - .any(|&(next, _, _)| { - let view = self.heard[&(id, next)].view(); - // A floor over the whole interest certifies everything below it. - !view.floors.keys().any(|region| within(watch, region)) - && below(&view.items, watch).any(|(claim, _)| { - view.as_of(claim).is_none() && merged.is_none_or(|merged| merged.as_of(claim).is_none()) - }) - }) - } - - /// Record the feasibility distance for the route this node selected for `id`, if - /// it sends that route to any neighbor. - fn record(&mut self, id: Id, peers: &[N]) { - let Some(selected) = self.selected(id) else { - return; - }; - let ctx = self.ctx(id, Some(selected)); - // As `advertise` decides, including that a route ends on a stream until its end is echoed. - let sends = peers.iter().any(|to| { - let peer = &self.neighbors[to]; - self.serves(peer, id) - && !(peer.link.upstream && selected.down_only) - && (self.sent.contains_key(&(id, *to)) || !self.unacked.contains_key(&(id, *to))) - && self.meets(id, &ctx, peer.watch.as_ref().expect("a watching peer")) - }); - if !sends { - return; - } - // Babel's feasibility distance: reset by a newer seqno, lowered by a better - // metric. It tracks finite updates actually sent (RFC 8966 s3.7.3). - let fd = self.fd.entry(id).or_insert((selected.seqno, selected.metric)); - if selected.seqno > fd.0 || (selected.seqno == fd.0 && selected.metric < fd.1) { - *fd = (selected.seqno, selected.metric); - } - } - - /// Queue any change to what each watching neighbor is sent for `id`, in `ctx`, - /// with `dirty` claims changed since the last time. - /// - /// A route goes to a stream only while it meets the stream's interest: it starts - /// before its first claim there, and once the last one goes, the batch whose floor - /// certifies that withdrawal precedes ROUTE_END. Each stream is sent only what - /// changed among the dirty claims, unless any could have. - fn advertise(&mut self, id: Id, peers: &[N], ctx: &Ctx, dirty: &Dirty) { - let selected = self.selected(id); - // The selected route's hops, then this node: its own route starts with it. - let through = selected.and_then(|route| route.next); - let hops: Arc<[Id]> = (through.iter()) - .flat_map(|next| self.rx[&id][next].advert.hops.iter().copied()) - .chain([self.hop]) - .collect(); - let floors = self.adopted_floors(id).cloned().unwrap_or_default(); - // Built only for a stream sent a whole snapshot, once per interest. - let mut adopted: Option> = None; - let mut presented: Vec<(Path, View)> = Vec::new(); - let mut plans = Vec::new(); - - for &to in peers { - let peer = &self.neighbors[&to]; - let serves = self.serves(peer, id); - let started = self.sent.contains_key(&(id, to)); - if !serves && !started { - continue; - } - let reach = match selected { - Some(route) if serves && !(peer.link.upstream && route.down_only) => Some(Reach::Finite(Advert { - seqno: route.seqno, - metric: route.metric, - down_only: route.down_only, - rtt: self.route_rtt(id, &route), - hops: hops.clone(), - })), - None if serves && ctx.live && started => Some(Reach::Held), - _ => None, - }; - // A route starts again only once the neighbor echoed its last end there, so - // one that withholds echoes holds at most one end per identity. - let reach = reach.filter(|_| started || !self.unacked.contains_key(&(id, to))); - let watch = peer.watch.as_ref().expect("a watching peer"); - let sent = self.sent_views.get(&(id, to)); - let (changes, meets) = match (&reach, sent) { - (None, _) => (Vec::new(), false), - // A stream holding nothing that the route still does not meet. - (Some(_), None) if !self.meets(id, ctx, watch) => (Vec::new(), false), - (Some(_), Some(sent)) if !dirty.all => { - let mut held = sent.items.len(); - let mut changes = Vec::new(); - let paths = (dirty.claims.iter()).filter(|claim| within(claim, watch) && *claim != watch); - for claim in paths { - let now = self.adopted_at(id, ctx, claim); - changes.push((claim, now)); - } - changes.push((watch, covering(&floors, watch, |claim| self.adopted_at(id, ctx, claim)))); - changes.retain(|(claim, now)| sent.items.get(*claim) != *now); - for (claim, now) in &changes { - match (sent.items.contains_key(*claim), now.is_some()) { - (false, true) => held += 1, - (true, false) => held -= 1, - _ => {} - } - } - // Starts and changes, then ends, as a whole snapshot's diff orders them. - changes.sort_by_key(|(_, now)| now.is_none()); - let changes = (changes.into_iter()) - .map(|(claim, now)| (claim.clone(), now.cloned())) - .collect(); - (changes, held > 0) - } - (Some(_), sent) => { - let index = match presented.iter().position(|(interest, _)| interest == watch) { - Some(index) => index, - None => { - let adopted = adopted.get_or_insert_with(|| { - self.adopted(id, selected).map(Cow::into_owned).unwrap_or_default() - }); - presented.push((watch.clone(), adopted.present(watch))); - presented.len() - 1 - } - }; - let view = &presented[index].1; - let changes = view.diff(sent.unwrap_or(&View::default())); - (changes, !view.items.is_empty()) - } - }; - let floors = narrow(&floors, watch); - #[cfg(test)] - if reach.is_some() { - let adopted = self.adopted(id, selected).map(Cow::into_owned).unwrap_or_default(); - let mut check = sent.cloned().unwrap_or_default(); - check.apply(changes.clone(), floors.clone()); - assert_eq!(check, adopted.present(watch), "incremental presentation to {to:?}"); - } - plans.push((to, started, reach, meets, changes, floors, watch.clone())); - } - - for (to, started, reach, meets, changes, floors, watch) in plans { - let key = (id, to); - let reaching = reach.is_some(); - let send = reach.filter(|_| meets); - let sending = send.is_some(); - if let Some(send) = send - && self.sent.get(&key) != Some(&send) - { - if id & QUALIFIED != 0 - && let Reach::Finite(advert) = &send - { - self.advertised.insert(id, advert.hops.clone()); - } - self.sent.insert(key, send.clone()); - self.out.push(Out::Route { - to, - id, - reach: Some(send), - }); - } - // A batch goes while the route does, the last one just before its ROUTE_END. - if sending || reaching && started && self.sent_views.contains_key(&key) { - let sent = self.sent_views.entry(key).or_default(); - let floor = sent.floor(&watch); - sent.apply(changes.clone(), floors); - let now = sent.floor(&watch); - if !changes.is_empty() || now != floor { - self.out.push(Out::Batch { - to, - id, - changes, - floor: now, - }); - } - } - // Announcements ride on the route and end with it. - if !sending { - if started { - *self.unacked.entry(key).or_default() += 1; - self.sent.remove(&key); - self.out.push(Out::Route { to, id, reach: None }); - } - self.sent_views.remove(&key); - } - } - } - - /// Start, resend, or clear this node's seqno request for `id`. - fn requests(&mut self, id: Id) { - if self.origins.contains_key(&id) { - self.pending.remove(&id); - return; - } - let route = self.selected(id); - let starved = route.is_none() && self.infeasible(id).next().is_some(); - // Nothing follows the last seqno: its origin rotates to a new identity instead, so - // there is nothing to ask for. - if starved && let Some(want) = self.fd[&id].0.checked_add(1) { - let pending = self.pending.entry(id).or_insert(Pending { - seqno: want, - from: BTreeSet::new(), - asked: None, - }); - if pending.seqno < want { - pending.seqno = want; - pending.asked = None; - } - } - let Some(pending) = self.pending.get(&id) else { - return; - }; - let answered = route.is_some_and(|route| route.seqno >= pending.seqno); - if answered || (pending.from.is_empty() && !starved) { - self.pending.remove(&id); - return; - } - // A request follows the next hop even back through a requester: next hops - // form no loop, one pending request per hop stops a bounce, and no - // requester ever cancels, so excluding one can strand a request while every - // route still runs through it. - let Some(target) = self.target(id) else { - self.pending.remove(&id); - return; - }; - let pending = self.pending.get_mut(&id).expect("pending"); - if pending.asked != Some(target) { - pending.asked = Some(target); - let seqno = pending.seqno; - let named = self.name(target, id); - self.out.push(Out::Request { - to: target, - id: named, - seqno, - }); - } - } - - /// Assert an incremental reader view, `now` over the dirty claims, matches a full recompute. - #[cfg(test)] - fn check_shown(&self, id: Id, ctx: &Ctx, dirty: &Dirty, now: &BTreeMap>>) { - if dirty.all { - return; - } - let route = self.selected(id); - let mut all: BTreeSet = self.reader.claims(id).cloned().collect(); - if let Some(view) = self.adopted(id, route) { - all.extend(view.items.keys().cloned()); - } - self.reader.check(id, now, self.shown_now(id, ctx, all.iter())); - } - - /// Recompute what this node's reader sees of `id` in `ctx` and log each change: - /// only the `dirty` claims, unless any could have changed. - fn show(&mut self, id: Id, ctx: &Ctx, dirty: &Dirty) { - let claims: BTreeSet = match dirty.all { - true => { - let mut claims: BTreeSet = self.reader.claims(id).cloned().collect(); - if let Some(view) = self.adopted(id, self.selected(id)) { - claims.extend(view.items.keys().cloned()); - } - claims - } - false => dirty.claims.clone(), - }; - let now = self.shown_now(id, ctx, claims.iter()); - #[cfg(test)] - self.check_shown(id, ctx, dirty, &now); - - // A claim no routed neighbor's region covers was let go, not ended, and so - // was one the view never certified: only its stream left. - let regions = self.certified(id, true); - let views = &self.views; - let uncertified = |claim: &Path| { - regions.as_ref().is_some_and(|regions| !within_any(claim, regions)) - || !ctx.own && views.get(&id).is_none_or(|view| view.as_of(claim).is_none()) - }; - self.reader.apply( - id, - |claim| within(claim, &self.watch), - now, - |claim| ctx.live && !uncertified(claim), - ); - } - - /// How the reader now sees each of `claims` of `id`, in `ctx`. - fn shown_now<'a>( - &self, - id: Id, - ctx: &Ctx, - claims: impl Iterator, - ) -> BTreeMap>> { - let merged = match ctx.own { - true => None, - false => self.views.get(&id), - }; - // A certified claim is available only through a certifier that is not held: - // another scope's usable route cannot serve it. - let serving = match ctx.usable && !ctx.own { - true => self.certified(id, false), - false => None, - }; - let current = self.current.get(&id); - let at = |claim: &Path| { - if !within(claim, &self.watch) { - return None; - } - let item = self.adopted_at(id, ctx, claim)?; - // An uncertified claim is available through a carrier that is not held. - let loose = match !ctx.own && ctx.live { - true => self.loose_at(id, ctx, merged, claim).map(|(_, unheld)| unheld), - false => None, - }; - // A claim a stream answering the current interest carries is current. - let certified = - loose.is_some() || current.is_none_or(|regions| regions.iter().any(|region| within(claim, region))); - if !certified { - return None; - } - let available = ctx.usable - && match loose { - Some(unheld) => unheld || serving.is_none(), - None => serving.as_ref().is_none_or(|regions| within_any(claim, regions)), - }; - Some(Shown { - item: item.clone(), - available, - presented: merged.is_some_and(|merged| merged.floors.contains_key(claim)), - }) - }; - claims.map(|claim| (claim.clone(), at(claim))).collect() - } -} diff --git a/rs/moq-net/src/route/node.rs b/rs/moq-net/src/route/node.rs index 77648e49721..ba8153ba30e 100644 --- a/rs/moq-net/src/route/node.rs +++ b/rs/moq-net/src/route/node.rs @@ -1,37 +1,42 @@ //! One origin node's router, shared by every lite-07 announce stream on its origin. //! -//! Each announce stream is its own neighbor key, one per direction: a stream this -//! node asked for (it receives routes and claims, and sends ROUTE_REQUEST) and a -//! stream it answers (it sends routes and claims, and receives ROUTE_REQUEST and -//! the ROUTE_END echo). Keying by stream lets a replacement stream open before -//! the old one closes, so no route drops in between. +//! Each announce stream is its own neighbour key on the [`Router`], one per direction: a +//! stream this node asked for receives routes and claims, and a stream it answers sends +//! them. Every stream of one session shares the session's [`Peer`] tag: a hold on what a +//! stream this node answers was sent waits on the routes the peer sends back on the +//! session's newest stream this node asked for. A stream that ends while its session +//! stays up retires quietly, as ROUTE_ENDs for its routes would; the streams that end +//! with their session report its hop dead. //! //! The node does no I/O. A stream's session feeds it what the peer sent and takes -//! [`Output`]s from its queue: wire messages to send, and the shown claims it -//! serves into the origin as the next hop of a feasible route. A claim is served -//! through every stream with a usable feasible route to its identity, each at its -//! own metric, so the origin ranks them and fails over at once when one goes; any -//! feasible route forwards without a loop. A stream that stops carrying one lets go -//! only once another serves it, so the origin always holds a route for it. +//! [`Output`]s from its queue: wire messages to send, and the shown claims it serves into +//! the origin. A claim is served through each live route covering it, the narrowest one +//! marked primary: a reader here follows that one, and a request forwarded from a peer +//! follows the route it names ([`Node::forward`]). A stream that stops carrying one lets +//! go only once another serves it, so the origin always holds a route for it. //! -//! A claim of another identity reaches the other streams only once a stream serving -//! it confirmed that instance, so a peer never names an instance this node's origin -//! cannot route yet. Until then its batches hold the claim back and their floors -//! certify nothing. +//! A claim of another identity reaches the other streams only once a stream serving it +//! confirmed that instance, so a peer never names an instance this node's origin cannot +//! route yet. Until then its batches hold the claim back and their floors certify +//! nothing. use std::{ collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque}, sync::{Arc, Mutex, MutexGuard}, task::{Poll, ready}, + time::Duration, }; +use kio::clock::{Deadlines, Instant}; + use crate::{Instance, Path}; use rand::RngExt; use super::{ - Error, GENERATED, Id, Link, Metric, Out, QUALIFIED, Reach, Rev, Router, Seqno, - view::{above, ancestors, below}, + Error, GENERATED, Id, Link, MAX_HOPS, Metric, QUALIFIED, Rev, + path::{Dead, Doubt, Forward, Hop, Offer, Reach, RouteId, Router}, + view::{ancestors, below}, }; /// One announce stream in one direction. @@ -59,51 +64,86 @@ pub(crate) struct Claim { pub chain: crate::Hops, } -/// A shown claim as the origin serves it through the selected route. +/// A shown claim as the origin serves it through one live route. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) struct Served { pub claim: Claim, - /// The selected route's metric: a path through the claim costs the Announce - /// Cost plus this. + /// The route's metric: a path through the claim costs the Announce Cost plus this. pub metric: Metric, - /// The round trip to the route's origin node, informational. - pub rtt: Option, + /// The round trip to the route's publisher, informational. + pub rtt: Option, /// Whether a usable route serves it; a held one leaves it shown but unavailable. pub available: bool, + /// A node on the route's path sent GOAWAY: it ranks below every route avoiding one. + pub draining: bool, + /// The lowest priority of a link on the route's path, the one it arrives on + /// included: a claim ranks by it after drain, as a route does. + pub priority: i8, + /// A route at the narrowest prefix with a live route covering the claim: one a + /// reader here follows, and a request naming no route. + pub primary: bool, + /// The route this node selected for its prefix; another is an alternate, which a + /// reader fails over to at once when the selected one goes. + pub selected: bool, + /// The origin nodes the route's path crosses: a request from one of them never + /// follows it. + pub path: Arc<[u64]>, } -/// What a stream's session acts on, in order. One identity's route changes and -/// batches coalesce while the session leaves them unread. +/// What a stream's session acts on, in order. One route's changes and batches coalesce +/// while the session leaves them unread, but never across a DEAD, which applies to the +/// routes the peer holds when it reads it. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) enum Output { - /// Send ROUTE_START, ROUTE_UPDATE, ROUTE_HOLD, or for `None` ROUTE_END. - Route { id: Id, reach: Option }, + /// Send ROUTE_START or ROUTE_UPDATE, ROUTE_HOLD (`Held`), or for `None` ROUTE_END, + /// for the route to `id` covering `prefix`. + Route { id: Id, prefix: Path, reach: Option }, /// Send these claim changes for the route, closed by ANNOUNCE_FLOOR. Batch { id: Id, + prefix: Path, changes: Vec<(Path, Option)>, floor: Option, }, - /// Echo the peer's ROUTE_END for the identity it named `id`. - Echo { id: Id }, - /// Serve `claim` of `id` into the origin through this stream, or update how it - /// is served. A request forwarded through it names `named`, the identity as - /// the peer advertised it. Confirm with [`Stream::serving`]. + /// Send this DEAD. + Dead(Dead), + /// Serve `claim` of `id` into the origin through this stream, along its route covering + /// `prefix`, or update how it is served. A request forwarded through it names `named`, + /// the identity as the peer advertised it. Confirm with [`Stream::serving`]. Serve { id: Id, named: Id, claim: Path, + prefix: Path, served: Served, }, - /// Stop serving `claim` of `id` through this stream. `ended` when the claim's origin - /// withdrew it, as far as this node knows; otherwise the node let go of it here. - Unserve { id: Id, claim: Path, ended: bool }, + /// Stop serving `claim` of `id` along the route covering `prefix` through this stream. + /// `ended` when the claim's origin withdrew it, as far as this node knows; otherwise + /// the node let go of it here. + Unserve { + id: Id, + claim: Path, + prefix: Path, + ended: bool, + }, +} + +/// The session a stream belongs to. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct Peer { + /// Shared by every stream of one session. + pub tag: u64, + /// How the peer names itself in a path: the Node ID its SETUP declared, or 0 when it + /// declared none, in which case the router takes it from the first route it sends. + pub node: Id, } /// The node every lite-07 session of one origin shares. #[derive(Clone)] pub(crate) struct Node { state: Arc>, + /// When the router's next timer fires, for the task that fires it. + timer: kio::Shared>, } impl std::fmt::Debug for Node { @@ -115,20 +155,17 @@ impl std::fmt::Debug for Node { /// Each stream's outputs, by its key. type Queues = HashMap>; -/// What one stream has yet to read. Route changes and claim batches coalesce per -/// identity, and what the reader was never sent leaves nothing when it ends: a route -/// started and ended unread is acked to the router at once, as the reader would. -/// Serving changes coalesce per claim. A stream whose reader stalls holds one entry -/// per route it holds or will start and per claim it serves, however much the node -/// changes meanwhile. +/// What one stream has yet to read. Route changes and claim batches coalesce per route +/// between DEADs, and a route started and ended unread leaves nothing. Serving changes +/// coalesce per claim and route. A stream whose reader stalls holds one entry per route +/// and claim per DEAD queued meanwhile, and its session closes it once it stalls too long. #[derive(Default)] struct Outbox { - /// Identities with route changes or claims pending, by when they first queued. - order: BTreeMap, - next: u64, - pending: HashMap, - /// Outputs to read in order: the other kinds, and a pending entry once taken. + /// Route changes pending, the DEAD closing each segment read after its routes. + segments: VecDeque, + /// Outputs to read in order: a pending entry once taken, or a DEAD. ready: VecDeque, + next: u64, /// Each claim's last Serve or Unserve, read after the rest, in queue order. A job /// queues its Serve before the Unserve or restart of the claim it continues, and the /// session needs the job's route in place before the claim's goes, so a reader @@ -136,34 +173,151 @@ struct Outbox { /// same instance and epoch keeps its place, while an Unserve, a restart, or a Serve /// after an Unserve goes last. serves: BTreeMap, - /// Where each claim's last change sits in `serves`. - serving: HashMap<(Id, Path), u64>, - /// Each route the reader holds, and the claims it was sent under it. - delivered: HashMap>, + /// Where each claim's last change on each route sits in `serves`. + serving: HashMap<(Id, Path, Path), u64>, + /// The routes the reader holds once it reads everything queued, each with its path, + /// so a DEAD ending the rest of a hop's routes ends them here too. + known: HashMap>, + /// The claims the reader holds under each route once it reads everything queued, so + /// a withdrawal of a claim it was never sent cancels the start instead of queueing: + /// what a stream keeps for its claims stays within the live ones. + claims: HashMap>, /// Claim starts and restarts held back until a stream serving them confirms their - /// instance, by identity, with the identity's last floor, which certifies nothing - /// while any is held. Bounded like `pending`: one per claim. The node's `waiting` - /// indexes them. - withheld: HashMap)>, - /// ROUTE_REQUESTs to send, by the identity as the peer advertised it. A higher - /// seqno subsumes a lower one, so a writer that stalls holds one per identity. - requests: BTreeMap, + /// instance, by route, with the route's last floor, which certifies nothing while any + /// is held. Bounded like the segments: one per claim. The node's `waiting` indexes + /// them. + withheld: HashMap, + /// What the segments a DEAD closed hold unread: each DEAD, its moved routes, and the + /// changes queued ahead of it for routes the reader still holds once everything + /// queued is read. A route that ends is read once and gone, so from then on its + /// changes count against nothing, whichever DEAD sealed them. See [`Self::overrun`]. + sealed: usize, + /// Each route's entries sealed while it counted in `sealed`, in the order they are read. + sealing: HashMap, } -/// One identity's route changes and claim batches since the reader last took it. +/// The floor a route's last batch closed with while claims of it were held back, which +/// their release closes with, and when that batch first queued: only a later batch +/// replaces it, whether sealed by a DEAD or read. +#[derive(Clone, Copy, Default)] +struct HeldFloor { + rev: Option, + seq: u64, +} + +impl HeldFloor { + fn set(&mut self, rev: Option, seq: u64) { + if seq >= self.seq { + *self = Self { rev, seq }; + } + } +} + +/// One route's entries that count in [`Outbox::sealed`]: those sealed, those read, and +/// how many of the first sealed no longer count, since the route ended after. +#[derive(Default)] +struct Sealing { + sealed: u64, + read: u64, + void: u64, +} + +/// Route changes queued before the next DEAD. +#[derive(Default)] +struct Segment { + /// Routes with changes pending, by when they first queued. + order: BTreeMap, + pending: HashMap, + /// The DEAD read once the routes are. + dead: Option, +} + +/// One route's changes and claim batches since the segment began. #[derive(Default)] struct Pending { - /// A ROUTE_END goes first, ending the route the reader holds and its claims. + /// The reader held the route when the segment began. + was: bool, + /// The route the reader held ended: a ROUTE_END goes first, after `closing`. ended: bool, - /// The last batch for the route the reader holds, read just before its ROUTE_END: - /// its floor certifies the withdrawals that ended the route. + /// The last batch for the route the reader held, read just before its ROUTE_END: its + /// floor certifies the withdrawals that ended the route. closing: Option, - /// The last finite reach since, which starts the route if the reader holds none. - finite: Option, + /// The last offer since, which starts the route if the reader holds none. + finite: Option, /// The route was held after that. held: bool, /// The batches since, merged: each claim's last change, and the last floor. batch: Option, + /// The claims `batch` starts that the reader did not hold when the entry began. + fresh: BTreeSet, + /// A DEAD sealed the entry while the reader still held its route: it counts in + /// [`Outbox::sealing`] until read. + sealed: bool, + /// A DEAD sealed the entry while claims of its route were held back, which go after + /// the DEAD once confirmed: its floor certifies nothing, since it cannot cover them, + /// and closes their release instead. + uncertified: bool, +} + +impl Pending { + fn is_empty(&self) -> bool { + !self.ended && self.finite.is_none() && !self.held && self.batch.is_none() + } + + /// Whether the reader held a claim at `path` when the entry began, given what it + /// holds once everything queued is read: a change queued here that did not start the + /// claim changed or withdrew one it held. + fn had(&self, holds: &BTreeSet, path: &Path) -> bool { + let queued = self.batch.as_ref().is_some_and(|(merged, _)| merged.contains_key(path)); + !self.fresh.contains(path) && (queued || holds.contains(path)) + } + + /// Queue `claim` at `path`, given the claims the reader holds once everything queued + /// is read, which it updates: a withdrawal of a claim the reader was not sent when the + /// entry began cancels its start, or is dropped. + fn change(&mut self, holds: &mut BTreeSet, path: Path, claim: Option) { + let had = self.had(holds, &path); + let (merged, _) = self.batch.get_or_insert_default(); + match claim { + Some(claim) => { + if !had { + self.fresh.insert(path.clone()); + } + holds.insert(path.clone()); + merged.insert(path, Some(claim)); + } + None => { + holds.remove(&path); + self.fresh.remove(&path); + match had { + true => merged.insert(path, None), + false => merged.remove(&path), + }; + } + } + } + + /// Drop what is queued at `path`, which the reader keeps as it held it when the entry + /// began. + fn unqueue(&mut self, holds: &mut BTreeSet, path: &Path) { + let had = self.had(holds, path); + if let Some((merged, _)) = &mut self.batch { + merged.remove(path); + } + self.fresh.remove(path); + match had { + true => holds.insert(path.clone()), + false => holds.remove(path), + }; + } +} + +/// How many routes a DEAD moves. +fn moved(dead: &Dead) -> usize { + match dead { + Dead::Closed { moved, .. } => moved.len(), + Dead::Draining { .. } => 0, + } } /// Merged batches: each claim's last change, and the last floor. @@ -175,237 +329,352 @@ type Changes = BTreeMap>; /// Claims held back, by path. type Held = BTreeMap; -/// The streams holding back a claim of each identity, by the path each holds it at: -/// one entry per held claim. A change to a home is looked up as a range of paths. -type Waiting = HashMap>>; +/// The streams holding back a claim of each identity, by the path each holds it at, +/// with the prefix of the route each holds it under: one entry per held claim. A change +/// to a home is looked up as a range of paths. +type Waiting = HashMap>>; impl Outbox { fn is_empty(&self) -> bool { - self.ready.is_empty() && self.order.is_empty() && self.serves.is_empty() + self.ready.is_empty() && self.segments.is_empty() && self.serves.is_empty() } - /// The pending entry for `id`, queued behind the others when new. - fn entry(&mut self, id: Id) -> &mut Pending { - let (order, next) = (&mut self.order, &mut self.next); - let (_, entry) = self.pending.entry(id).or_insert_with(|| { + /// The open segment, a new one after a DEAD. + fn segment(&mut self) -> &mut Segment { + if self.segments.back().is_none_or(|segment| segment.dead.is_some()) { + self.segments.push_back(Segment::default()); + } + self.segments.back_mut().expect("just pushed") + } + + /// The pending entry for `key` in the open segment, queued behind the others when new. + fn entry(&mut self, key: &RouteId) -> &mut Pending { + let was = self.known.contains_key(key); + let next = &mut self.next; + let segment = { + if self.segments.back().is_none_or(|segment| segment.dead.is_some()) { + self.segments.push_back(Segment::default()); + } + self.segments.back_mut().expect("just pushed") + }; + let Segment { order, pending, .. } = segment; + let (_, entry) = pending.entry(key.clone()).or_insert_with(|| { let seq = *next; *next += 1; - order.insert(seq, id); - (seq, Pending::default()) + order.insert(seq, key.clone()); + ( + seq, + Pending { + was, + ..Pending::default() + }, + ) }); entry } - /// Queue `output`. - fn push(&mut self, output: Output) { - match output { - Output::Route { reach: None, .. } => unreachable!("a route end is queued with Outbox::end"), - Output::Route { - id, - reach: Some(Reach::Held), - } => { - self.entry(id).held = true; - } - Output::Route { - id, - reach: Some(finite), - } => { - let entry = self.entry(id); - entry.finite = Some(finite); - entry.held = false; - } - Output::Batch { .. } => unreachable!("a batch is queued with Outbox::batch"), - // Each serving change replaces the last: the session acts on where the claim - // ends up, in place when it keeps its instance. - Output::Serve { id, ref claim, .. } | Output::Unserve { id, ref claim, .. } => { - let key = (id, claim.clone()); - if let Some(&seq) = self.serving.get(&key) - && let Output::Serve { served, .. } = &output - && let Some(Output::Serve { served: queued, .. }) = self.serves.get(&seq) - && queued.claim.instance == served.claim.instance - && queued.claim.epoch == served.claim.epoch - { - self.serves.insert(seq, output); - return; - } - let seq = self.next; - self.next += 1; - if let Some(old) = self.serving.insert(key, seq) { - self.serves.remove(&old); - } - self.serves.insert(seq, output); - } - Output::Echo { .. } => { - self.ready.push_back(output); - } + /// Drop the open segment's entry for `key` once it holds nothing. + fn prune(&mut self, key: &RouteId) { + let Some(segment) = self.segments.back_mut() else { + return; + }; + if segment.pending.get(key).is_some_and(|(_, pending)| pending.is_empty()) + && let Some((seq, _)) = segment.pending.remove(key) + { + segment.order.remove(&seq); } } - /// Queue the end of `id`'s route, for the stream `key`. Returns `id` when the reader - /// will never read it, since it never read the route it ends, for the caller to ack - /// in its place. - fn end(&mut self, key: Key, id: Id, waiting: &mut Waiting) -> Option { - // The peer drops the route's claims, so nothing held back is owed. - let held = self.withheld.remove(&id); - for path in held.iter().flat_map(|(held, _)| held.keys()) { - unwait(waiting, id, path, key); + /// Queue a serving change: each replaces the last for its claim and route, the session + /// acting on where the claim ends up, in place when it keeps its instance. + fn serve(&mut self, output: Output) { + let (Output::Serve { id, claim, prefix, .. } | Output::Unserve { id, claim, prefix, .. }) = &output else { + unreachable!("only serving changes are queued here"); + }; + let key = (*id, claim.clone(), prefix.clone()); + if let Some(&seq) = self.serving.get(&key) + && let Output::Serve { served, .. } = &output + && let Some(Output::Serve { served: queued, .. }) = self.serves.get(&seq) + && queued.claim.instance == served.claim.instance + && queued.claim.epoch == served.claim.epoch + { + self.serves.insert(seq, output); + return; } - let holds = self.delivered.contains_key(&id); - let entry = self.entry(id); - // The reader holds the route once it reads what is pending: an end of that keeps - // any end before, an end of a start it never read is acked. - let ended = entry.ended; - let acked = !holds || ended; - let closing = match ended { - true => entry.closing.take(), - // Its floor certifies nothing if a claim was held back. - false => { - (entry.batch.take().filter(|_| holds)).map(|(merged, floor)| (merged, floor.filter(|_| held.is_none()))) + let seq = self.next; + self.next += 1; + if let Some(old) = self.serving.insert(key, seq) { + self.serves.remove(&old); + } + self.serves.insert(seq, output); + } + + /// Queue a route change for the stream `stream`. + fn route(&mut self, stream: Key, key: RouteId, reach: Option, waiting: &mut Waiting) { + match reach { + Some(Reach::Finite(offer)) => { + let path = offer.path.clone(); + let entry = self.entry(&key); + entry.finite = Some(offer); + entry.held = false; + self.known.insert(key, path); + } + Some(Reach::Held) => { + self.entry(&key).held = true; + } + None => { + self.unwithhold(stream, &key, waiting); + // Whether the reader holds the route is read before it is forgotten: an end + // queued alone, with no batch ahead of it, opens the entry. + let entry = self.entry(&key); + match entry.was && !entry.ended { + // The reader holds the route: it ends, after the floor of its last batch. + true => { + entry.ended = true; + entry.closing = entry.batch.take(); + entry.finite = None; + entry.held = false; + } + // A start the reader never read: it reads nothing of it. + false => { + entry.finite = None; + entry.held = false; + entry.batch = None; + } + } + // Its claims end with it. + entry.fresh.clear(); + self.known.remove(&key); + self.unseal(&key); + self.claims.remove(&key); + self.prune(&key); } - }; - *entry = Pending { - ended: holds, - closing, - ..Pending::default() - }; - if !holds && let Some((seq, _)) = self.pending.remove(&id) { - self.order.remove(&seq); } - acked.then_some(id) } - /// Queue a batch of `id`'s claim changes, closed by `floor`, for the stream `key`, - /// holding back each start or restart `awaits` a confirmation. + /// Queue a batch of the route's claim changes, closed by `floor`, for the stream + /// `stream`, holding back each start or restart `awaits` a confirmation. fn batch( &mut self, - key: Key, - id: Id, + stream: Key, + key: RouteId, (changes, floor): (Vec<(Path, Option)>, Option), awaits: impl Fn(&Path, &Claim) -> Option, waiting: &mut Waiting, ) { - let ended = self.entry(id).ended; - let Self { - pending, - delivered, - withheld, - .. - } = self; - let (_, entry) = pending.get_mut(&id).expect("just queued"); - let sent = delivered.get(&id); - let (merged, last) = entry.batch.get_or_insert_default(); + let mut withheld = self.withheld.remove(&key); + let mut holds = self.claims.remove(&key).unwrap_or_default(); + let entry = self.entry(&key); for (path, claim) in changes { // A newer change replaces one held back. - if let Some((held, _)) = withheld.get_mut(&id) + if let Some((held, _)) = &mut withheld && held.remove(&path).is_some() { - unwait(waiting, id, &path, key); + unwait(waiting, key.0, &path, stream, &key.1); } if let Some(item) = &claim && awaits(&path, item) == Some(true) { // The reader keeps what it was sent until then. - merged.remove(&path); + entry.unqueue(&mut holds, &path); waiting - .entry(id) + .entry(key.0) .or_default() .entry(path.clone()) .or_default() - .insert(key); - withheld.entry(id).or_default().0.insert(path, item.clone()); + .insert((stream, key.1.clone())); + withheld.get_or_insert_default().0.insert(path, item.clone()); continue; } - // The reader holds nothing here once the entry reaches it. - let unsent = ended || sent.is_none_or(|sent| !sent.contains(&path)); - match claim { - None if unsent => merged.remove(&path), - claim => merged.insert(path, claim), - }; + entry.change(&mut holds, path, claim); } - *last = floor; - if withheld.get(&id).is_some_and(|(held, _)| held.is_empty()) { - withheld.remove(&id); + entry.batch.get_or_insert_default().1 = floor; + if !holds.is_empty() { + self.claims.insert(key.clone(), holds); + } + if let Some(withheld) = withheld.filter(|(held, _)| !held.is_empty()) { + self.withheld.insert(key, withheld); } } - /// Queue the claim of `id` held back at `path`, now confirmed. - fn release(&mut self, id: Id, path: &Path) { - let Some((held, floor)) = self.withheld.get_mut(&id) else { + /// Queue the claim of the route held back at `path`, now confirmed. + fn release(&mut self, key: &RouteId, path: &Path) { + let Some((held, floor)) = self.withheld.get_mut(key) else { return; }; let Some(item) = held.remove(path) else { return; }; - let floor = *floor; + let floor = floor.rev; if held.is_empty() { - self.withheld.remove(&id); + self.withheld.remove(key); } + let mut holds = self.claims.remove(key).unwrap_or_default(); + let entry = self.entry(key); // A batch still queued closes with a newer floor. - let (merged, _) = self.entry(id).batch.get_or_insert_with(|| (Changes::new(), floor)); - merged.insert(path.clone(), Some(item)); + entry.batch.get_or_insert_with(|| (Changes::new(), floor)); + entry.change(&mut holds, path.clone(), Some(item)); + self.claims.insert(key.clone(), holds); + } + + /// The peer drops the route's claims, so nothing held back is owed. + fn unwithhold(&mut self, stream: Key, key: &RouteId, waiting: &mut Waiting) { + if let Some((held, _)) = self.withheld.remove(key) { + for path in held.keys() { + unwait(waiting, key.0, path, stream, &key.1); + } + } + } + + /// Queue a DEAD for the stream `stream`, read once every route change queued before + /// it is. The reader's routes through its hop move as it says, and with `gone` the + /// rest end with it. + fn dead(&mut self, stream: Key, dead: Dead, waiting: &mut Waiting) { + if let Dead::Closed { hop, moved, gone, .. } = &dead { + let mut kept = HashSet::new(); + for moved in moved { + let key = (moved.id, moved.prefix.clone()); + if let Some(path) = self.known.get_mut(&key) { + *path = moved.offer.path.clone(); + } + kept.insert(key); + } + if *gone { + let ended: Vec = (self.known.iter()) + .filter(|(key, path)| !kept.contains(*key) && path.contains(hop)) + .map(|(key, _)| key.clone()) + .collect(); + for key in ended { + self.known.remove(&key); + self.unseal(&key); + self.claims.remove(&key); + self.unwithhold(stream, &key, waiting); + } + } + } + self.segment(); + let segment = self.segments.back_mut().expect("just opened"); + let mut sealed = 1 + moved(&dead); + for (key, (seq, pending)) in &mut segment.pending { + pending.sealed = self.known.contains_key(key); + if pending.sealed { + self.sealing.entry(key.clone()).or_default().sealed += 1; + sealed += 1; + } + // Claims held back go after the DEAD, so the floor waits for them: the release + // closes them with it. + if let Some((_, last)) = self.withheld.get_mut(key) { + pending.uncertified = true; + if let Some((_, floor)) = &pending.batch { + last.set(*floor, *seq); + } + } + } + segment.dead = Some(dead); + self.sealed += sealed; + } + + /// The reader stops holding the route `key` once everything queued is read, so what a + /// DEAD sealed for it no longer counts in [`Self::sealed`]. + fn unseal(&mut self, key: &RouteId) { + if let Some(sealing) = self.sealing.get_mut(key) { + self.sealed -= (sealing.sealed - sealing.read.max(sealing.void)) as usize; + sealing.void = sealing.sealed; + } + } + + /// Whether the reader fell behind repeated failures of the routes it holds. Each + /// crosses at most [`MAX_HOPS`] hops, and each failed hop costs it at most a DEAD, a + /// moved entry, and a change queued after, so a reader holding more than that unread + /// for every route it holds is slower than the routes fail: a peer that reads a + /// trickle while its routes churn would otherwise hold a DEAD per failure forever. + fn overrun(&self) -> bool { + self.sealed > 3 * MAX_HOPS * (self.known.len() + 1) } fn pop(&mut self) -> Option { - if self.ready.is_empty() { - match self.order.pop_first() { - Some((_, id)) => { - let (_, pending) = self.pending.remove(&id).expect("every queued identity is pending"); - self.take(id, pending); + while self.ready.is_empty() { + let Some(segment) = self.segments.front_mut() else { + let (_, output) = self.serves.pop_first()?; + if let Output::Serve { id, claim, prefix, .. } | Output::Unserve { id, claim, prefix, .. } = &output { + self.serving.remove(&(*id, claim.clone(), prefix.clone())); + } + return Some(output); + }; + match segment.order.pop_first() { + Some((_, key)) => { + let (seq, pending) = segment.pending.remove(&key).expect("every queued route is pending"); + if pending.sealed { + let sealing = self.sealing.get_mut(&key).expect("a sealed entry is counted"); + sealing.read += 1; + if sealing.read > sealing.void { + self.sealed -= 1; + } + if sealing.read == sealing.sealed { + self.sealing.remove(&key); + } + } + self.take(key, seq, pending); } None => { - let (_, output) = self.serves.pop_first()?; - if let Output::Serve { id, claim, .. } | Output::Unserve { id, claim, .. } = &output { - self.serving.remove(&(*id, claim.clone())); + let segment = self.segments.pop_front().expect("matched above"); + if let Some(dead) = segment.dead { + self.sealed -= 1 + moved(&dead); + self.ready.push_back(Output::Dead(dead)); } - return Some(output); } } } self.ready.pop_front() } - /// Queue one identity's pending entry to read, as the router would have sent it. - fn take(&mut self, id: Id, pending: Pending) { + /// Queue one route's pending entry to read, as the router would have sent it. + fn take(&mut self, (id, prefix): RouteId, seq: u64, pending: Pending) { if pending.ended { - if let Some(closing) = pending.closing { + if let Some((changes, floor)) = pending.closing { self.ready.push_back(Output::Batch { id, - changes: ordered(closing.0), - floor: closing.1, + prefix: prefix.clone(), + changes: ordered(changes), + floor, }); } - self.delivered.remove(&id); - self.ready.push_back(Output::Route { id, reach: None }); + self.ready.push_back(Output::Route { + id, + prefix: prefix.clone(), + reach: None, + }); } - if let Some(reach) = pending.finite { - self.delivered.entry(id).or_default(); - self.ready.push_back(Output::Route { id, reach: Some(reach) }); + if let Some(offer) = pending.finite { + self.ready.push_back(Output::Route { + id, + prefix: prefix.clone(), + reach: Some(Reach::Finite(offer)), + }); } if pending.held { self.ready.push_back(Output::Route { id, + prefix: prefix.clone(), reach: Some(Reach::Held), }); } if let Some((merged, floor)) = pending.batch { // What is held back leaves the floor certifying nothing until it goes. - let floor = match self.withheld.get_mut(&id) { + let floor = match self.withheld.get_mut(&(id, prefix.clone())) { Some((_, last)) => { - *last = floor; + last.set(floor, seq); None } + None if pending.uncertified => None, None => floor, }; - let changes = ordered(merged); - if let Some(sent) = self.delivered.get_mut(&id) { - for (path, claim) in &changes { - match claim { - Some(_) => sent.insert(path.clone()), - None => sent.remove(path), - }; - } - } - self.ready.push_back(Output::Batch { id, changes, floor }); + self.ready.push_back(Output::Batch { + id, + prefix, + changes: ordered(merged), + floor, + }); } } } @@ -418,20 +687,21 @@ fn awaits(homes: &Homes, id: Id, path: &Path, instance: &Instance, epoch: u64) - let same = |claim: &Claim| claim.instance == *instance && claim.epoch == epoch; let home = ancestors(path).rev().find_map(|shown| { let home = homes.get(&(id, shown))?; - home.keys.values().any(|slot| same(&slot.served.claim)).then_some(home) + home.slots.values().any(|slot| same(&slot.served.claim)).then_some(home) })?; let confirmed = - (home.keys.values()).any(|slot| (slot.confirmed.as_ref()).is_some_and(|(i, e)| i == instance && *e == epoch)); + (home.slots.values()).any(|slot| (slot.confirmed.as_ref()).is_some_and(|(i, e)| i == instance && *e == epoch)); Some(!confirmed) } -/// The stream `key` no longer holds back a claim of `id` at `path`. -fn unwait(waiting: &mut Waiting, id: Id, path: &Path, key: Key) { +/// The stream `stream` no longer holds back a claim of `id` at `path` under the route +/// covering `prefix`. +fn unwait(waiting: &mut Waiting, id: Id, path: &Path, stream: Key, prefix: &Path) { let Some(paths) = waiting.get_mut(&id) else { return; }; if let Some(keys) = paths.get_mut(path) { - keys.remove(&key); + keys.remove(&(stream, prefix.clone())); if keys.is_empty() { paths.remove(path); } @@ -453,21 +723,25 @@ fn recheck(homes: &Homes, queues: &Queues, waiting: &mut Waiting, id: Id, change for claim in changed { let at = paths.get_key_value(claim).into_iter(); for (path, keys) in at.chain(below(paths, claim)) { - held.extend(keys.iter().map(|key| (path.clone(), *key))); + held.extend( + keys.iter() + .map(|(stream, prefix)| (path.clone(), *stream, prefix.clone())), + ); } } - for (path, key) in held { - let Some(queue) = queues.get(&key) else { + for (path, stream, prefix) in held { + let Some(queue) = queues.get(&stream) else { continue; }; let mut outbox = queue.lock(); - let confirmed = (outbox.withheld.get(&id)) + let key = (id, prefix); + let confirmed = (outbox.withheld.get(&key)) .and_then(|(held, _)| held.get(&path)) .and_then(|item| awaits(homes, id, &path, &item.instance, item.epoch)) == Some(false); if confirmed { - outbox.release(id, &path); - unwait(waiting, id, &path, key); + outbox.release(&key, &path); + unwait(waiting, id, &path, stream, &key.1); } } } @@ -483,6 +757,7 @@ fn ordered(changes: Changes) -> Vec<(Path, Option)> { struct Locked<'a> { // Dropped first, being declared first, so the held wakes go out unlocked. state: MutexGuard<'a, State>, + timer: &'a kio::Shared>, _hold: kio::Hold, } @@ -500,46 +775,65 @@ impl std::ops::DerefMut for Locked<'_> { } } +impl Drop for Locked<'_> { + fn drop(&mut self) { + // Whatever changed, the timer task waits for the router's next deadline. + let deadline = self.state.router.deadline(); + if *self.timer.read() != deadline { + *self.timer.lock() = deadline; + } + } +} + struct State { router: Router, + /// The origin's clock, which dates every input the router times. + clock: Deadlines, next_key: Key, queues: Queues, + /// Each stream's session tag. + streams: HashMap, + /// Each session's streams. + sessions: HashMap, /// Each shown claim of another identity, and the streams serving it. homes: Homes, /// What the streams hold back, by the confirmation it waits on. waiting: Waiting, - /// The usable feasible routes each identity's claims were last served through, - /// with the name each stream's peer gives it: while they stay the same, only a - /// claim whose shown state changed is served differently. + /// The live routes each identity's claims were last served through: while they stay + /// the same, only a claim whose shown state changed is served differently. routes: HashMap, - /// The streams whose close changes a path to this node, which then advances its - /// seqno so a node remembering a shorter path accepts the longer one without a - /// ROUTE_REQUEST: one whose peer forwarded it a route, or, on a node that does - /// not forward, one this node's route goes out on. A new stream lengthens no path. - links: HashSet, - /// Answer streams of a node that does not forward, which join `links` once this - /// node's route goes out on them. - origins: HashSet, +} + +/// One session's streams on the node. +#[derive(Default)] +struct Session { + /// The newest stream this node asked the peer for, whose routes answer the holds on + /// what the streams answering the peer were sent. + receive: Option, + /// The streams answering the peer. + answers: BTreeSet, + /// How many streams it has open. + open: usize, /// How many [`Closing`] guards are held: while any is, a dropped stream joins - /// `closed` instead of disconnecting on its own. + /// `closed` instead of retiring on its own. closing: usize, - /// The streams dropped while a [`Closing`] guard was held, disconnected together - /// once the last is released. + /// The streams dropped while a guard was held, disconnected together once the last is + /// released: the session closed. closed: Vec, } /// Each shown claim of another identity, and the streams serving it. type Homes = BTreeMap<(Id, Path), Home>; -/// An identity's usable feasible routes: each stream, its metric and RTT, and the -/// peer's name. -type Routes = Vec<(Key, Metric, Option, Id)>; +/// An identity's live routes, by prefix: the stream each goes through, its offer, and +/// the identity as that stream's peer names it. +type Routes = Vec<(Path, Key, Offer, Id)>; -/// Where one shown claim is served: each stream, what it was last sent, and -/// whether it confirmed serving it. +/// Where one shown claim is served: each stream and route prefix, what it was last +/// sent, and whether it confirmed serving it. #[derive(Default)] struct Home { - keys: BTreeMap, + slots: BTreeMap<(Key, Path), Slot>, } struct Slot { @@ -548,24 +842,29 @@ struct Slot { named: Id, /// The instance and epoch the stream last confirmed serving. confirmed: Option<(Instance, u64)>, - /// No longer a route to the claim's identity: let go once a wanted stream confirms. + /// No longer a live route covering the claim: let go once a wanted slot confirms. leaving: bool, } impl Home { - /// Unserve every leaving stream once a wanted one serves the claim. + /// Unserve every leaving slot once a wanted one serves the claim. fn release(&mut self, queues: &Queues, id: Id, claim: &Path) { - if !self.keys.values().any(|slot| !slot.leaving && slot.confirmed.is_some()) { + if !self + .slots + .values() + .any(|slot| !slot.leaving && slot.confirmed.is_some()) + { return; } - self.keys.retain(|key, slot| { + self.slots.retain(|(stream, prefix), slot| { if slot.leaving { - push( + serve( queues, - *key, + *stream, Output::Unserve { id, claim: claim.clone(), + prefix: prefix.clone(), ended: false, }, ); @@ -576,44 +875,47 @@ impl Home { } impl Node { - /// A node declaring a fresh identity, which also names it in the hops of the routes - /// it forwards, `seed` keying its rotation. - pub fn new(id: Id, seed: u64) -> Self { + /// A node declaring a fresh identity, which also names it in the paths of the routes + /// it forwards, `seed` keying its rotation, dating what it hears by `clock`. + pub fn new(id: Id, seed: u64, clock: Deadlines) -> Self { debug_assert!( id != 0 && id & QUALIFIED == 0, "a declared identity is non-zero and unqualified" ); - let mut router = Router::new(id, seed); - router.originate(id, 0); + let mut router = Router::new(id, seed, Doubt::default()); + router.clock(clock.now()); + router.originate(id); Self { state: Arc::new(Mutex::new(State { router, + clock, next_key: 1, queues: HashMap::new(), + streams: HashMap::new(), + sessions: HashMap::new(), homes: BTreeMap::new(), waiting: HashMap::new(), routes: HashMap::new(), - links: HashSet::new(), - origins: HashSet::new(), - closing: 0, - closed: Vec::new(), })), + timer: kio::Shared::new(None), } } /// A node declaring the origin's hop id, so the origin has one identity on every /// wire. A declared identity is never qualified, so an id carrying that marker /// loses it, or is replaced by a random one if nothing is left. - pub fn origin(hop: std::num::NonZeroU64) -> Self { + pub fn origin(hop: std::num::NonZeroU64, clock: Deadlines) -> Self { let mut rng = kio::clock::rng(); let id = declared_for(hop).unwrap_or_else(|| rng.random_range(1..=GENERATED)); - Self::new(id, rng.random()) + Self::new(id, rng.random(), clock) } fn lock(&self) -> Locked<'_> { + let state = self.state.lock().unwrap_or_else(|poison| poison.into_inner()); Locked { _hold: kio::hold(), - state: self.state.lock().unwrap_or_else(|poison| poison.into_inner()), + timer: &self.timer, + state, } } @@ -625,12 +927,18 @@ impl Node { .expect("a node always declares an identity") } + /// How this node names itself in the paths of the routes it forwards. + pub fn hop(&self) -> Id { + self.lock().router.hop() + } + /// This node's own claim at `path` changed: `None` withdraws it. pub fn own(&self, path: Path, claim: Option) { let mut state = self.lock(); let changed = match claim { Some(claim) => { - let view = state.router.declared().and_then(|id| state.router.view(id)); + let declared = state.router.declared(); + let view = declared.and_then(|id| state.router.view(id, &Path::empty())); let current = view.and_then(|view| view.items.get(&path)); let changed = current != Some(&claim); changed && state.router.announce(path, claim).is_ok() @@ -642,60 +950,95 @@ impl Node { } } - /// Register a stream this node asked the peer for under `prefix`. - pub fn receive(&self, link: Link, prefix: Path) -> Result { - self.register(link, false, |router, key| router.asks(key, prefix)) - } - - /// Register a stream answering the peer's request for `prefix`. - pub fn answer(&self, link: Link, prefix: Path) -> Result { - // A node that forwards nothing knows no better than that every stream carrying - // its route is a path to it; one whose prefix matches no claim carries none yet. - let origin = !link.proxy; - self.register(link, origin, |router, key| router.peer_interest(key, prefix)) + /// Register a stream this node asked `peer` for, which becomes the session's stream + /// whose routes answer the holds on what its other streams were sent. + pub fn receive(&self, peer: Peer, link: Link) -> Result { + let mut state = self.lock(); + let key = state.register(peer, link)?; + let session = state.sessions.get_mut(&peer.tag).expect("registered"); + session.receive = Some(key); + let answers: Vec = session.answers.iter().copied().collect(); + for answer in answers { + state.router.pair(answer, key); + } + state.flush(); + Ok(self.stream(&state, key)) } - fn register( - &self, - link: Link, - origin: bool, - open: impl FnOnce(&mut Router, Key) -> Result<(), Error>, - ) -> Result { + /// Register a stream answering `peer`'s request for `prefix`. + pub fn answer(&self, peer: Peer, link: Link, prefix: Path) -> Result { let mut state = self.lock(); - let key = state.next_key; - state.next_key += 1; - state.router.connect(key, link)?; - let queue = kio::Shared::default(); - state.queues.insert(key, queue.clone()); - if origin { - state.origins.insert(key); + let key = state.register(peer, link)?; + let session = state.sessions.get_mut(&peer.tag).expect("registered"); + session.answers.insert(key); + if let Some(receive) = session.receive { + state.router.pair(key, receive); } - if let Err(err) = open(&mut state.router, key) { - state.router.disconnect([key]); - state.queues.remove(&key); - state.origins.remove(&key); - state.links.remove(&key); + let edit = super::path::Edit { + remove: Vec::new(), + add: vec![prefix], + }; + if let Err(err) = state.router.edit(key, edit) { + state.unregister(key); + state.router.retire(key); return Err(err); } state.flush(); - Ok(Stream { + Ok(self.stream(&state, key)) + } + + fn stream(&self, state: &State, key: Key) -> Stream { + Stream { node: self.clone(), key, - queue, - }) + queue: state.queues[&key].clone(), + clock: None, + } } - /// Hold back the disconnect of every stream dropped until the guard is released, - /// then disconnect them together: a session closing many streams at once costs the - /// route layer one recompute rather than one per stream. - pub fn closing(&self) -> Closing { - self.lock().closing += 1; - Closing { node: self.clone() } + /// Hold back the disconnect of every stream of the session `tag` dropped until the + /// guard is released, then disconnect them together, reporting the session's hops + /// dead: the session closed. A stream dropped outside one retires quietly. + pub fn closing(&self, tag: u64) -> Closing { + self.lock().sessions.entry(tag).or_default().closing += 1; + Closing { + node: self.clone(), + tag, + } } - #[cfg(test)] - pub(crate) fn retained(&self) -> super::Retained { - self.lock().router.retained() + /// Where a request for `claim` of `id` that the session `tag`'s peer sent goes, naming + /// the route this node offered it covering `prefix`: here, along this node's route + /// covering a prefix, or `None` when that route is gone or would lead back to the peer. + pub fn forward(&self, tag: u64, id: Id, claim: &Path, prefix: &Path) -> Option { + let state = self.lock(); + let session = state.sessions.get(&tag)?; + // The stream that was sent the route, else any answering the peer. + let from = (session.answers.iter()) + .find(|key| state.router.sent(**key, id, prefix)) + .or_else(|| session.answers.first()) + .copied()?; + match state.router.forward(id, claim, Some((from, prefix.clone())))? { + Forward::Here => Some(Via::Here), + Forward::Next { prefix, .. } => Some(Via::Route { prefix }), + } + } + + /// Fire the router's timers due by the origin's clock. + pub fn timeout(&self) { + let mut state = self.lock(); + let now = state.clock.now(); + state.router.timeout(now); + state.flush(); + } + + /// When the router's next timer fires, waking `waiter` once that changes from `armed`. + pub fn poll_deadline(&self, armed: Option, waiter: &kio::Waiter) -> Poll> { + let deadline = ready!(self.timer.poll(waiter, |at| match **at != armed { + true => Poll::Ready(()), + false => Poll::Pending, + })); + Poll::Ready(*deadline) } #[cfg(test)] @@ -706,11 +1049,25 @@ impl Node { } } -/// One announce stream's key on the node. Dropping it disconnects the stream. +/// Where a request a peer sent goes; see [`Node::forward`]. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) enum Via { + /// This node publishes it. + Here, + /// Along this node's route covering `prefix`. + Route { prefix: Path }, +} + +/// One announce stream's key on the node. Dropping it retires the stream, or disconnects +/// it with its session; see [`Node::closing`]. pub(crate) struct Stream { node: Node, key: Key, queue: kio::Shared, + /// The clock of the session the stream belongs to. The origin's clock moves only as + /// its driver polls, so an idle origin reads an old instant: what a session hands the + /// node is dated by the later of the two, a closed session's doubt included. + clock: Option, } impl std::fmt::Debug for Stream { @@ -726,8 +1083,23 @@ impl Stream { true => Poll::Pending, false => Poll::Ready(()), }; - let mut outbox = ready!(self.queue.poll(waiter, ready)); - Poll::Ready(outbox.pop().expect("an outbox holding something pops it")) + loop { + // What was queued may have come to nothing, a route started and ended unread, + // which leaves the outbox empty to wait on. + if let Some(output) = ready!(self.queue.poll(waiter, ready)).pop() { + return Poll::Ready(output); + } + } + } + + /// Ready once the reader fell behind repeated failures of its routes + /// (`Outbox::overrun`): the session cuts it off, as it does a reader that stops. + pub fn poll_overrun(&self, waiter: &kio::Waiter) -> Poll<()> { + let overrun = |outbox: &kio::Ref<'_, Outbox>| match outbox.overrun() { + true => Poll::Ready(()), + false => Poll::Pending, + }; + self.queue.poll(waiter, overrun).map(|_| ()) } /// The next output already queued, without waiting. @@ -738,81 +1110,69 @@ impl Stream { self.queue.lock().pop() } - /// The ROUTE_REQUESTs queued since the last call, each identity's highest seqno. - /// Take them only once the writer has room, so they keep coalescing meanwhile. - pub fn poll_requests(&self, waiter: &kio::Waiter) -> Poll> { - let ready = |outbox: &kio::Ref<'_, Outbox>| match outbox.requests.is_empty() { - true => Poll::Pending, - false => Poll::Ready(()), - }; - let mut outbox = ready!(self.queue.poll(waiter, ready)); - Poll::Ready(std::mem::take(&mut outbox.requests)) + /// Date what the stream hands the node by its session's clock too. + pub fn with_clock(mut self, clock: Deadlines) -> Self { + self.clock = Some(clock); + self + } + + /// The later of the origin's clock and the session's. + fn now(&self, state: &State) -> Instant { + let origin = state.clock.now(); + self.clock.as_ref().map_or(origin, |clock| origin.max(clock.now())) } - fn with(&self, f: impl FnOnce(&mut Router, Key) -> R) -> R { + fn with(&self, f: impl FnOnce(&mut Router, Key, Instant) -> R) -> R { let mut state = self.node.lock(); - let out = f(&mut state.router, self.key); + let now = self.now(&state); + state.router.clock(now); + let out = f(&mut state.router, self.key, now); state.flush(); out } - /// ROUTE_START, ROUTE_UPDATE, ROUTE_HOLD (`Held`), or ROUTE_END (`None`) for - /// the identity the peer names `named`. - pub fn route(&self, named: Id, reach: Option) { - let mut state = self.node.lock(); - // Only the peer's own route has metric 0: any other means the peer forwards. - if matches!(&reach, Some(Reach::Finite(advert)) if advert.metric > 0) { - state.links.insert(self.key); - } - state.router.route(self.key, named, reach); - state.flush(); + /// ROUTE_START or ROUTE_UPDATE, ROUTE_HOLD (`Held`), or ROUTE_END (`None`) for the + /// route the peer names `named` covering `prefix`, as the peer names it. + pub fn route(&self, named: Id, prefix: Path, reach: Option) { + self.with(|router, key, _| router.route(key, named, prefix, reach)); } - /// `next`, of the same session, takes this stream's place: it inherits whether - /// this one was a route link, so closing this one changes no path. - pub fn replaced_by(&self, next: &Stream) { - let mut state = self.node.lock(); - if state.links.remove(&self.key) { - state.links.insert(next.key); - } - if state.origins.remove(&self.key) { - state.origins.insert(next.key); - } + /// A batch of claim changes for the route the peer names `named` covering `prefix`, + /// closed by its floor. + pub fn batch(&self, named: Id, prefix: Path, changes: Vec<(Path, Option)>, floor: Option) { + self.with(|router, key, _| router.batch(key, named, prefix, changes, floor)); } - /// A batch of claim changes for the route the peer names `named`, closed by - /// its floor. - pub fn batch(&self, named: Id, changes: Vec<(Path, Option)>, floor: Option) { - self.with(|router, key| { - let interest = router.interest(); - router.batch(key, named, changes, floor, interest) - }); + /// How the peer names itself in a path, or 0 while the node has not learned it. + pub fn peer(&self) -> Id { + self.node.lock().router.hop_of(self.key) } - /// The peer sent ROUTE_REQUEST for `id`, as this node advertised it. - pub fn request(&self, id: Id, seqno: Seqno) { - self.with(|router, key| router.request(key, id, seqno)); + /// The peer sent DEAD. + pub fn dead(&self, dead: Dead) { + self.with(|router, key, now| router.dead(key, dead, now)); } - /// The peer echoed this node's ROUTE_END for `id`. - pub fn echo(&self, id: Id) { - self.with(|router, key| router.echo(key, id)); + /// Whether the node applies a DEAD from the peer naming `hop` and `far`, as + /// [`Router::credits`] says; the node drops any other. + pub fn credits(&self, hop: Hop, far: Id) -> bool { + self.with(|router, key, _| router.credits(key, hop, far)) } /// The peer sent GOAWAY: its routes rank below every other. pub fn goaway(&self) { - self.with(|router, key| router.goaway(key)); + self.with(|router, key, now| router.goaway(key, now)); } /// The link to the peer has `rtt`, or `None` unmeasured. - pub fn rtt(&self, rtt: Option) { - self.with(|router, key| router.rtt(key, rtt)); + pub fn rtt(&self, rtt: Option) { + self.with(|router, key, _| router.rtt(key, rtt)); } - /// This stream's session now serves `claim` of `id` at `served`'s instance, so the - /// other streams are sent that instance, and a stream that no longer carries a route - /// to it lets go. - pub fn serving(&self, id: Id, claim: &Path, served: &Claim) { + /// This stream's session now serves `claim` of `id` along its route covering `prefix` + /// at `served`'s instance, so the other streams are sent that instance, and a slot + /// that no longer carries a live route to it lets go. + pub fn serving(&self, id: Id, claim: &Path, prefix: &Path, served: &Claim) { let mut state = self.node.lock(); let State { homes, queues, waiting, .. @@ -820,14 +1180,14 @@ impl Stream { let Some(home) = homes.get_mut(&(id, claim.clone())) else { return; }; - let Some(slot) = home.keys.get_mut(&self.key) else { + let Some(slot) = home.slots.get_mut(&(self.key, prefix.clone())) else { return; }; let confirmed = (served.instance.clone(), served.epoch); if slot.confirmed.as_ref() == Some(&confirmed) { return; } - slot.confirmed = Some(confirmed.clone()); + slot.confirmed = Some(confirmed); home.release(queues, id, claim); recheck(homes, queues, waiting, id, std::slice::from_ref(claim)); } @@ -836,114 +1196,167 @@ impl Stream { impl Drop for Stream { fn drop(&mut self) { let mut state = self.node.lock(); - match state.closing { - 0 => state.close(vec![self.key]), - _ => state.closed.push(self.key), + let Some(&tag) = state.streams.get(&self.key) else { + return; + }; + let now = self.now(&state); + state.router.clock(now); + let session = state.sessions.get_mut(&tag).expect("a registered stream's session"); + match session.closing { + 0 => state.close(vec![self.key], false), + _ => session.closed.push(self.key), } } } -/// Holds back the disconnect of each stream dropped meanwhile, then disconnects them -/// together on drop. See [`Node::closing`]. +/// Holds back the disconnect of each stream of one session dropped meanwhile, then +/// disconnects them together on drop. See [`Node::closing`]. pub(crate) struct Closing { node: Node, + tag: u64, } impl Drop for Closing { fn drop(&mut self) { let mut state = self.node.lock(); - state.closing -= 1; - if state.closing == 0 && !state.closed.is_empty() { - let closed = std::mem::take(&mut state.closed); - state.close(closed); + let Some(session) = state.sessions.get_mut(&self.tag) else { + return; + }; + session.closing -= 1; + if session.closing > 0 { + return; + } + let closed = std::mem::take(&mut session.closed); + if !closed.is_empty() { + state.close(closed, true); } + state.forget(self.tag); } } -/// Queue `output` for the stream `key`. -fn push(queues: &Queues, key: Key, output: Output) { +/// Queue a serving change for the stream `key`. +fn serve(queues: &Queues, key: Key, output: Output) { if let Some(queue) = queues.get(&key) { - queue.lock().push(output); + queue.lock().serve(output); } } impl State { - /// Route the router's messages and shown claims to the streams. Claims are served - /// before the batches carrying them go, so a batch knows what is not served yet. - /// Disconnect the dropped streams `keys`: their sessions are gone, and what they - /// served with them. - fn close(&mut self, keys: Vec) { + /// Connect a stream of `peer`'s session. + fn register(&mut self, peer: Peer, link: Link) -> Result { + let key = self.next_key; + self.next_key += 1; + let now = self.clock.now(); + self.router.clock(now); + let link = Link { + session: peer.tag, + ..link + }; + self.router.connect(key, peer.node, link)?; + // Lite-07 asks for one prefix per stream, on opening it: what the router asks the + // peer for takes effect at once, and no later edit comes. + self.router.edited(key); + self.queues.insert(key, kio::Shared::default()); + self.streams.insert(key, peer.tag); + self.sessions.entry(peer.tag).or_default().open += 1; + Ok(key) + } + + /// Forget the stream `key` and its place in its session; the router keeps it until + /// the caller retires or disconnects it. + fn unregister(&mut self, key: Key) { + if let Some(queue) = self.queues.remove(&key) { + for ((id, prefix), (held, _)) in &queue.read().withheld { + for path in held.keys() { + unwait(&mut self.waiting, *id, path, key, prefix); + } + } + } + let Some(tag) = self.streams.remove(&key) else { + return; + }; + let session = self.sessions.get_mut(&tag).expect("a registered stream's session"); + session.open -= 1; + session.answers.remove(&key); + if session.receive == Some(key) { + session.receive = None; + } + self.forget(tag); + } + + /// Drop the session `tag` once it has no stream and no guard. + fn forget(&mut self, tag: u64) { + if self + .sessions + .get(&tag) + .is_some_and(|session| session.open == 0 && session.closing == 0) + { + self.sessions.remove(&tag); + } + } + + /// The dropped streams `keys` ended: with their session when `closed`, which reports + /// their hops dead, else quietly. What they served goes with them. + fn close(&mut self, keys: Vec, closed: bool) { for &key in &keys { - if let Some(queue) = self.queues.remove(&key) { - for (id, (held, _)) in &queue.read().withheld { - for path in held.keys() { - unwait(&mut self.waiting, *id, path, key); - } + self.unregister(key); + } + match closed { + true => { + // The streams dropping dated the router by their sessions' clocks. + let now = self.clock.now().max(self.router.now()); + self.router.disconnect(keys.iter().copied(), now); + } + false => { + for &key in &keys { + self.router.retire(key); } } } - self.router.disconnect(keys.iter().copied()); let keys: HashSet = keys.into_iter().collect(); for home in self.homes.values_mut() { - home.keys.retain(|key, _| !keys.contains(key)); - } - self.origins.retain(|key| !keys.contains(key)); - let linked = self.links.len(); - self.links.retain(|key| !keys.contains(key)); - if self.links.len() != linked { - let _ = self.router.bump(); + home.slots.retain(|(key, _), _| !keys.contains(key)); } self.flush(); } + /// Route the router's messages and shown claims to the streams. Claims are served + /// before the batches carrying them go, so a batch knows what is not served yet. fn flush(&mut self) { - loop { - let outs = self.router.drain(); - self.serve(); - let mut acked = Vec::new(); - for out in outs { - let (key, output) = match out { - Out::Route { to, id, reach: None } => { - if let Some(queue) = self.queues.get(&to) - && let Some(id) = queue.lock().end(to, id, &mut self.waiting) - { - acked.push((to, id)); - } - continue; + let outs = self.router.drain(); + self.serve(); + for out in outs { + match out { + super::path::Out::Route { to, id, prefix, reach } => { + if let Some(queue) = self.queues.get(&to) { + queue.lock().route(to, (id, prefix), reach, &mut self.waiting); } - Out::Route { to, id, reach } => { - if Some(id) == self.router.declared() && self.origins.contains(&to) { - self.links.insert(to); - } - (to, Output::Route { id, reach }) - } - Out::Batch { to, id, changes, floor } => { - if let Some(queue) = self.queues.get(&to) { - let homes = &self.homes; - let awaits = - |path: &Path, item: &Claim| awaits(homes, id, path, &item.instance, item.epoch); - queue.lock().batch(to, id, (changes, floor), awaits, &mut self.waiting); - } - continue; + } + super::path::Out::Batch { + to, + id, + prefix, + changes, + floor, + } => { + if let Some(queue) = self.queues.get(&to) { + let homes = &self.homes; + let awaits = |path: &Path, item: &Claim| awaits(homes, id, path, &item.instance, item.epoch); + queue + .lock() + .batch(to, (id, prefix), (changes, floor), awaits, &mut self.waiting); } - Out::Request { to, id, seqno } => { - if let Some(queue) = self.queues.get(&to) { - let mut outbox = queue.lock(); - let want = outbox.requests.entry(id).or_insert(seqno); - *want = (*want).max(seqno); - } - continue; + } + super::path::Out::Dead { to, dead } => { + if let Some(queue) = self.queues.get(&to) { + queue.lock().dead(to, dead, &mut self.waiting); } - Out::Echo { to, id } => (to, Output::Echo { id }), - }; - push(&self.queues, key, output); - } - if acked.is_empty() { - break; - } - // A ROUTE_END coalesced away before its reader read the route: no peer holds it. - for (key, id) in acked { - self.router.echo(key, id); + } + // The stream answering the peer sends its own once its initial set is out. + super::path::Out::Synced { .. } => {} + // Each stream's request is its only edit, sent as the session opens it, and + // takes effect as the stream starts: lite-07 has no marker to send. + super::path::Out::Ask { .. } | super::path::Out::Edited { .. } => {} } } } @@ -965,13 +1378,17 @@ impl State { } } - /// Serve each shown claim of `id` through every stream with a usable feasible - /// route to it that can carry the claim, keeping it where it is, unavailable, while - /// there is none. Only `claims`, whose shown state changed, are served anew, unless - /// the routes changed too. + /// Serve each shown claim of `id` through each live route covering it, keeping it + /// where it is, unavailable, while there is none. Only `claims`, whose shown state + /// changed, are served anew, unless the routes changed too. fn home(&mut self, id: Id, claims: BTreeSet, ended: &BTreeMap<(Id, Path), bool>) { - let routes: Routes = (self.router.routes(id).into_iter()) - .map(|(key, metric, rtt)| (key, metric, rtt, self.router.name(key, id))) + let routes: Routes = (self.router.routes(id)) + .flat_map(|(prefix, _)| { + let usable = self.router.usable(&(id, prefix.clone())); + usable + .into_iter() + .map(|(next, offer)| (prefix.clone(), next, offer, self.router.name(next, id))) + }) .collect(); let last = self.routes.get(&id).map_or(&[][..], Vec::as_slice); let claims = match last == routes.as_slice() { @@ -992,48 +1409,56 @@ impl State { let changed: Vec = claims.iter().cloned().collect(); let (shown, gone): (Vec, Vec) = (claims.into_iter()).partition(|claim| self.router.shown().contains_key(&(id, claim.clone()))); - // Each route by the prefix its stream asked for, so a claim looks up the routes - // that may carry it per segment rather than checking every route. - let mut asked: BTreeMap> = BTreeMap::new(); - for (index, (key, ..)) in routes.iter().enumerate() { - asked.entry(self.router.asked(*key)).or_default().push(index); + // Each prefix's usable routes, the selected one first. + let mut by_prefix: BTreeMap<&Path, Vec<&(Path, Key, Offer, Id)>> = BTreeMap::new(); + for route in &routes { + by_prefix.entry(&route.0).or_default().push(route); } // Most specific first: a job continuing its claim's instance is served before // the claim moves on, so a reader resolved through the claim finds the job's route. for claim in shown.into_iter().rev() { let shown = &self.router.shown()[&(id, claim.clone())]; let (item, available) = (shown.item.clone(), shown.available); - let mut indices: Vec = above(&asked, &claim) - .flat_map(|(_, indices)| indices) - .copied() - .collect(); - indices.sort_unstable(); - let carrying: Vec<(Key, Metric, Option, Id)> = (indices.into_iter()) - .map(|index| routes[index]) - .filter(|(key, ..)| self.router.carries(*key, &claim)) - .collect(); + // The usable routes covering the claim, the narrowest prefix's last. + let covering: Vec<&Vec<&(Path, Key, Offer, Id)>> = + ancestors(&claim).filter_map(|prefix| by_prefix.get(&prefix)).collect(); let home = self.homes.entry((id, claim.clone())).or_default(); - // With no route left, the router no longer knows the name a stream's peer - // gave the identity, so it keeps the one it was served under. - let wanted: Vec<(Key, Metric, Option, bool, Id)> = match carrying.is_empty() { - true => (home.keys.iter()) - .map(|(key, slot)| (*key, slot.served.metric, slot.served.rtt, false, slot.named)) + // With no live route left, the router no longer knows the name a stream's peer + // gave the identity, so each slot keeps the one it was served under. + let wanted: Vec<((Key, Path), Served, Id)> = match covering.last() { + None => (home.slots.iter()) + .map(|(at, slot)| { + let served = Served { + claim: item.clone(), + available: false, + ..slot.served.clone() + }; + (at.clone(), served, slot.named) + }) .collect(), - false => (carrying.into_iter()) - .map(|(key, metric, rtt, named)| (key, metric, rtt, available, named)) + Some(narrowest) => (covering.iter()) + .flat_map(|routes| routes.iter().enumerate()) + .map(|(rank, (prefix, next, offer, named))| { + let served = Served { + claim: item.clone(), + metric: offer.metric, + rtt: offer.rtt, + available, + draining: offer.draining, + priority: offer.priority, + primary: prefix == &narrowest[0].0, + selected: rank == 0, + path: offer.path.iter().map(|hop| hop.node).collect(), + }; + ((*next, prefix.clone()), served, *named) + }) .collect(), }; - for slot in home.keys.values_mut() { + for slot in home.slots.values_mut() { slot.leaving = true; } - for (key, metric, rtt, available, named) in wanted { - let served = Served { - claim: item.clone(), - metric, - rtt, - available, - }; - match home.keys.get_mut(&key) { + for (at, served, named) in wanted { + match home.slots.get_mut(&at) { Some(slot) if slot.served == served && slot.named == named => { slot.leaving = false; continue; @@ -1050,16 +1475,18 @@ impl State { confirmed: None, leaving: false, }; - home.keys.insert(key, slot); + home.slots.insert(at.clone(), slot); } } - let serve = Output::Serve { + let (stream, prefix) = at; + let output = Output::Serve { id, named, claim: claim.clone(), + prefix, served, }; - push(&self.queues, key, serve); + serve(&self.queues, stream, output); } home.release(&self.queues, id, &claim); } @@ -1068,22 +1495,18 @@ impl State { continue; }; let ended = ended.get(&(id, claim.clone())).copied().unwrap_or(false); - for key in home.keys.into_keys() { - push( - &self.queues, - key, - Output::Unserve { - id, - claim: claim.clone(), - ended, - }, - ); + for (stream, prefix) in home.slots.into_keys() { + let output = Output::Unserve { + id, + claim: claim.clone(), + prefix, + ended, + }; + serve(&self.queues, stream, output); } } // A claim held back may now wait on another home, or on none. recheck(&self.homes, &self.queues, &mut self.waiting, id, &changed); - #[cfg(test)] - self.check(id, &routes); // Nothing left to recover: let the identity go. let shows = (self.router.shown().range((id, Path::empty())..).next()).is_some_and(|((shown, _), _)| *shown == id); @@ -1092,50 +1515,6 @@ impl State { self.router.collect(id); } } - - /// Every claim of `id` is served as serving it anew would: only shown claims have - /// a home, and each is served through every route carrying it, or kept unavailable. - #[cfg(test)] - fn check(&self, id: Id, routes: &Routes) { - let homes = (self.homes.range((id, Path::empty())..)).take_while(|((home, _), _)| *home == id); - for ((_, claim), _) in homes { - assert!( - self.router.shown().contains_key(&(id, claim.clone())), - "{claim:?} has a home but is not shown" - ); - } - let shown = (self.router.shown().range((id, Path::empty())..)).take_while(|((shown, _), _)| *shown == id); - for ((_, claim), shown) in shown { - let home = self.homes.get(&(id, claim.clone())); - let slots = home.into_iter().flat_map(|home| home.keys.iter()); - let carrying: Vec<_> = (routes.iter()) - .filter(|(key, ..)| self.router.carries(*key, claim)) - .collect(); - if carrying.is_empty() { - for (_, slot) in slots { - assert_eq!(slot.served.claim, shown.item, "{claim:?} kept as shown"); - assert!(!slot.served.available, "{claim:?} kept unavailable"); - } - continue; - } - let slots: BTreeMap = slots.filter(|(_, slot)| !slot.leaving).map(|(k, s)| (*k, s)).collect(); - let want: BTreeMap = (carrying.iter()) - .map(|&&(key, metric, rtt, named)| { - let served = Served { - claim: shown.item.clone(), - metric, - rtt, - available: shown.available, - }; - (key, (served, named)) - }) - .collect(); - let got: BTreeMap = (slots.iter()) - .map(|(key, slot)| (*key, (slot.served.clone(), slot.named))) - .collect(); - assert_eq!(got, want, "{claim:?} served through every route carrying it"); - } - } } /// The identity a node created for the origin hop `hop` declares, or `None` when nothing @@ -1166,883 +1545,4 @@ pub(crate) fn hop(id: Id) -> crate::Hop { } #[cfg(test)] -mod tests { - use super::*; - use crate::route::Advert; - - fn p(path: &'static str) -> Path { - Path::from_static(path) - } - - fn claim(instance: &Instance, cost: u64) -> Claim { - Claim { - instance: instance.clone(), - epoch: 0, - cost, - upstream: false, - learned: false, - chain: crate::Hops::new(), - } - } - - fn outputs(stream: &Stream) -> Vec { - std::iter::from_fn(|| stream.try_output()).collect() - } - - /// Deliver everything `from` would send on the wire to the stream receiving it, - /// returning what else it queued. - fn pump(from: &Stream, to: &Stream) -> Vec { - let mut rest = Vec::new(); - for output in outputs(from) { - match output { - Output::Route { id, reach } => to.route(id, reach), - Output::Batch { id, changes, floor } => to.batch(id, changes, floor), - other => rest.push(other), - } - } - rest - } - - /// The serve commands among a stream's outputs. - fn served(outputs: Vec) -> Vec { - (outputs.into_iter()) - .filter(|output| matches!(output, Output::Serve { .. } | Output::Unserve { .. })) - .collect() - } - - #[test] - fn a_claim_is_served_through_the_stream_its_route_arrived_on() { - let instance = Instance::random(); - let (origin, relay) = (Node::new(1, 1), Node::new(2, 2)); - let answer = origin.answer(Link::default(), Path::empty()).unwrap(); - let receive = relay.receive(Link::default(), Path::empty()).unwrap(); - origin.own(p("live/x"), Some(claim(&instance, 5))); - pump(&answer, &receive); - assert_eq!( - served(outputs(&receive)), - [Output::Serve { - id: 1, - named: 1, - claim: p("live/x"), - served: Served { - claim: claim(&instance, 5), - metric: 1, - rtt: None, - available: true, - }, - }] - ); - - origin.own(p("live/x"), None); - pump(&answer, &receive); - assert_eq!( - served(outputs(&receive)), - [Output::Unserve { - id: 1, - claim: p("live/x"), - ended: true - }] - ); - } - - #[test] - fn a_claim_is_served_through_every_feasible_route_at_its_own_metric() { - let instance = Instance::random(); - let origin = Node::new(1, 1); - let relay = Node::new(2, 2); - let far = relay - .receive( - Link { - cost: 3, - ..Link::default() - }, - Path::empty(), - ) - .unwrap(); - let far_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - let near = relay.receive(Link::default(), Path::empty()).unwrap(); - let near_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - origin.own(p("live/x"), Some(claim(&instance, 0))); - pump(&far_answer, &far); - pump(&near_answer, &near); - let metric = |outputs: Vec| match &served(outputs)[..] { - [.., Output::Serve { served, .. }] => Some(served.metric), - _ => None, - }; - assert_eq!(metric(outputs(&far)), Some(3)); - assert_eq!(metric(outputs(&near)), Some(1)); - far.serving(1, &p("live/x"), &claim(&instance, 0)); - - // A stream whose route ends lets go only once the other confirms it serves the claim. - far.route(1, None); - assert!(served(outputs(&far)).is_empty(), "nothing else serves it yet"); - near.serving(1, &p("live/x"), &claim(&instance, 0)); - assert_eq!( - served(outputs(&far)), - [Output::Unserve { - id: 1, - claim: p("live/x"), - ended: false - }] - ); - assert!(served(outputs(&near)).is_empty(), "the remaining route serves on"); - } - - /// Two streams reach one origin node for different prefixes: each claim is served - /// only through the streams that asked for it, however cheap the other. - #[test] - fn a_claim_is_served_only_through_streams_that_carry_it() { - let instance = Instance::random(); - let origin = Node::new(1, 1); - let relay = Node::new(2, 2); - let a = relay.receive(Link::default(), p("live/a")).unwrap(); - let a_answer = origin.answer(Link::default(), p("live/a")).unwrap(); - let costly = Link { - cost: 3, - ..Link::default() - }; - let b = relay.receive(costly, p("live/b")).unwrap(); - let b_answer = origin.answer(Link::default(), p("live/b")).unwrap(); - origin.own(p("live/a/x"), Some(claim(&instance, 0))); - origin.own(p("live/b/y"), Some(claim(&instance, 0))); - pump(&a_answer, &a); - pump(&b_answer, &b); - let claims = |outputs: Vec| -> Vec { - (served(outputs).into_iter()) - .filter_map(|output| match output { - Output::Serve { claim, .. } => Some(claim), - _ => None, - }) - .collect() - }; - assert_eq!(claims(outputs(&a)), [p("live/a/x")]); - assert_eq!(claims(outputs(&b)), [p("live/b/y")]); - } - - /// A stalled stream reads a job's Serve before the Unserve of the claim it - /// continues, even once the job changed again meanwhile, so a reader resolved - /// through the claim finds the job's route before the claim's goes. - #[test] - fn a_stalled_stream_reads_a_job_before_its_claim_goes() { - let instance = Instance::random(); - let origin = Node::new(1, 1); - let relay = Node::new(2, 2); - let fast = relay.receive(Link::default(), Path::empty()).unwrap(); - let stalled = relay.receive(Link::default(), Path::empty()).unwrap(); - let fast_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - let stalled_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - origin.own(p("live"), Some(claim(&instance, 0))); - pump(&fast_answer, &fast); - pump(&stalled_answer, &stalled); - outputs(&fast); - outputs(&stalled); - - origin.own(p("live/x"), Some(claim(&instance, 0))); - origin.own(p("live"), None); - pump(&fast_answer, &fast); - origin.own(p("live/x"), Some(claim(&instance, 5))); - pump(&fast_answer, &fast); - let read: Vec<(Path, Option)> = (served(outputs(&stalled)).into_iter()) - .map(|output| match output { - Output::Serve { claim, served, .. } => (claim, Some(served.claim.cost)), - Output::Unserve { claim, .. } => (claim, None), - other => panic!("not a serving change: {other:?}"), - }) - .collect(); - assert_eq!(read, [(p("live/x"), Some(5)), (p("live"), None)]); - } - - /// A job and its claim's restart arriving together are read job first, even past a - /// change to the claim still queued, so a reader resolved through the claim's old - /// instance finds the job's route before the claim's moves on. - #[test] - fn a_job_is_read_before_its_claims_restart() { - let (old, new) = (Instance::random(), Instance::random()); - let origin = Node::new(1, 1); - let relay = Node::new(2, 2); - let fast = relay.receive(Link::default(), Path::empty()).unwrap(); - let stalled = relay.receive(Link::default(), Path::empty()).unwrap(); - let fast_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - let stalled_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - origin.own(p("live"), Some(claim(&old, 0))); - pump(&fast_answer, &fast); - pump(&stalled_answer, &stalled); - outputs(&fast); - outputs(&stalled); - - // A change to the claim the stalled stream leaves queued. - origin.own(p("live"), Some(claim(&old, 1))); - pump(&fast_answer, &fast); - pump(&stalled_answer, &stalled); - outputs(&fast); - - origin.own(p("live/x"), Some(claim(&old, 0))); - origin.own(p("live"), Some(claim(&new, 0))); - pump(&fast_answer, &fast); - pump(&stalled_answer, &stalled); - for stream in [&fast, &stalled] { - let read: Vec<(Path, Instance)> = (served(outputs(stream)).into_iter()) - .map(|output| match output { - Output::Serve { claim, served, .. } => (claim, served.claim.instance), - other => panic!("not a serve: {other:?}"), - }) - .collect(); - assert_eq!(read, [(p("live/x"), old.clone()), (p("live"), new.clone())]); - } - } - - /// A claim of a scoped peer left with no usable route stays where it is served, - /// under the name that peer gave its identity, so the requests it serves meanwhile - /// still name it as the peer does. - #[test] - fn a_claim_without_a_route_keeps_the_peers_name() { - let instance = Instance::random(); - let origin = Node::new(1, 1); - let relay = Node::new(2, 2); - let scoped = || Link { - scope: Some(super::super::Scope { - regions: vec![Path::empty()], - qualify: super::super::qualifier([7; 32]), - }), - ..Link::default() - }; - let first = relay.receive(scoped(), Path::empty()).unwrap(); - let second = relay.receive(scoped(), Path::empty()).unwrap(); - let first_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - let second_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - origin.own(p("live/x"), Some(claim(&instance, 0))); - pump(&first_answer, &first); - pump(&second_answer, &second); - second.route(1, Some(Reach::Held)); - let names = |outputs: Vec| -> Vec<(Id, bool)> { - (served(outputs).into_iter()) - .filter_map(|output| match output { - Output::Serve { named, served, .. } => Some((named, served.available)), - _ => None, - }) - .collect() - }; - assert_eq!(names(outputs(&first)), [(1, true)]); - - // The second still holds a route it may recover through. - first.route(1, None); - assert_eq!(names(outputs(&first)), [(1, false)]); - } - - /// A stream whose reader stalls holds one serving change per claim, however often - /// the claim changes through another stream meanwhile. - #[test] - fn a_stalled_stream_holds_one_serve_per_claim() { - let instance = Instance::random(); - let origin = Node::new(1, 1); - let relay = Node::new(2, 2); - let fast = relay.receive(Link::default(), Path::empty()).unwrap(); - let stalled = relay.receive(Link::default(), Path::empty()).unwrap(); - let fast_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - let stalled_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - origin.own(p("live/x"), Some(claim(&instance, 0))); - pump(&fast_answer, &fast); - pump(&stalled_answer, &stalled); - for cost in 1..1000 { - origin.own(p("live/x"), Some(claim(&instance, cost))); - pump(&fast_answer, &fast); - outputs(&fast); - } - pump(&stalled_answer, &stalled); - let read = served(outputs(&stalled)); - assert_eq!(read.len(), 1, "{read:?}"); - assert!( - matches!(&read[0], Output::Serve { served, .. } if served.claim.cost == 999), - "{read:?}" - ); - } - - /// A reader that stops reading holds one entry for the route, however often its - /// claims change: the last change to each, and none for a claim it was never sent. - #[test] - fn a_stalled_reader_holds_the_net_change() { - let instance = Instance::random(); - let origin = Node::new(1, 1); - let answer = origin.answer(Link::default(), Path::empty()).unwrap(); - outputs(&answer); - for cost in 0..1000 { - origin.own(p("live/x"), Some(claim(&instance, cost))); - origin.own(p("live/churn"), Some(claim(&instance, cost))); - origin.own(p("live/churn"), None); - } - let read = outputs(&answer); - let batch = |read: &[Output]| match read { - [.., Output::Batch { changes, .. }] => changes.clone(), - _ => panic!("no batch: {read:?}"), - }; - assert!(read.len() <= 2, "{read:?}"); - assert_eq!(batch(&read), [(p("live/x"), Some(claim(&instance, 999)))]); - - // Once sent, a claim's end is kept, behind what started. - for cost in 0..1000 { - origin.own(p("live/y"), Some(claim(&instance, cost))); - } - origin.own(p("live/x"), None); - let read = outputs(&answer); - assert_eq!(read.len(), 1, "{read:?}"); - assert_eq!( - batch(&read), - [(p("live/y"), Some(claim(&instance, 999))), (p("live/x"), None)] - ); - } - - /// Routes that start and end while the reader stalls leave nothing for it, and the - /// router is acked at once as the reader would: churning fresh identities holds - /// nothing. A route the reader holds ends once, however often it restarted unread. - #[test] - fn routes_ended_unread_leave_nothing() { - let instance = Instance::random(); - let relay = Node::new(2, 2); - let edge = relay - .answer( - Link { - proxy: true, - ..Link::default() - }, - Path::empty(), - ) - .unwrap(); - outputs(&edge); - let advert = |seqno| { - Some(Reach::Finite(Advert { - seqno, - metric: 0, - down_only: false, - rtt: None, - hops: [].into(), - })) - }; - // An identity that stays, after the churned ones in the node's order. - let standing = relay.receive(Link::default(), Path::empty()).unwrap(); - standing.route(200, advert(0)); - standing.batch(200, vec![(p("live/s"), Some(claim(&instance, 0)))], Some(1)); - standing.serving(200, &p("live/s"), &claim(&instance, 0)); - outputs(&edge); - let kept = relay.retained(); - for origin in 10..110 { - let receive = relay.receive(Link::default(), Path::empty()).unwrap(); - receive.route(origin, advert(0)); - receive.batch(origin, vec![(p("live/x"), Some(claim(&instance, 0)))], Some(1)); - drop(receive); - } - let of = |read: Vec, of: fn(Id) -> bool| -> Vec { - (read.into_iter()) - .filter(|output| matches!(output, Output::Route { id, .. } | Output::Batch { id, .. } if of(*id))) - .collect() - }; - let read = of(outputs(&edge), |id| (10..110).contains(&id)); - assert_eq!(read, [], "the churned routes"); - assert_eq!(relay.retained(), kept, "nothing of the churned identities is kept"); - - // The reader holds a route, which ends, restarts, and ends again unread. - let receive = relay.receive(Link::default(), Path::empty()).unwrap(); - receive.route(7, advert(0)); - receive.batch(7, vec![(p("live/x"), Some(claim(&instance, 0)))], Some(1)); - assert_eq!(of(outputs(&edge), |id| id == 7).len(), 2, "the route and its batch"); - receive.route(7, None); - receive.route(7, advert(1)); - receive.batch(7, vec![(p("live/x"), Some(claim(&instance, 0)))], Some(2)); - receive.route(7, None); - let read = of(outputs(&edge), |id| id == 7); - assert_eq!(read, [Output::Route { id: 7, reach: None }]); - assert_eq!( - relay.retained().echoes, - 1, - "only the end the reader reads awaits its echo" - ); - edge.echo(7); - assert_eq!(relay.retained().echoes, 0); - } - - /// A route goes on a stream only while it has a claim the stream's interest - /// meets. Once the last one is withdrawn, the reader reads the floor certifying - /// the withdrawal before the route's end, even when both queued unread. - #[test] - fn a_route_ends_after_the_floor_certifying_its_last_claim() { - let instance = Instance::random(); - let origin = Node::new(1, 1); - let edge = origin.answer(Link::default(), Path::empty()).unwrap(); - let other = origin.answer(Link::default(), p("other")).unwrap(); - assert_eq!(outputs(&edge), [], "no claim, no route"); - - origin.own(p("live/x"), Some(claim(&instance, 0))); - assert_eq!(outputs(&edge).len(), 2, "the route, then its claim"); - assert_eq!(outputs(&other), [], "the interest meets no claim"); - - origin.own(p("live/x"), None); - assert_eq!( - outputs(&edge), - [ - Output::Batch { - id: 1, - changes: vec![(p("live/x"), None)], - floor: Some(2), - }, - Output::Route { id: 1, reach: None }, - ] - ); - } - - /// A peer that withholds ROUTE_END echoes holds one end per identity however - /// often its only claim flaps: the route starts again only once the end is echoed. - #[test] - fn a_route_restarts_only_once_its_end_is_echoed() { - let instance = Instance::random(); - let origin = Node::new(1, 1); - let edge = origin.answer(Link::default(), Path::empty()).unwrap(); - origin.own(p("live/x"), Some(claim(&instance, 0))); - outputs(&edge); - // The peer reads everything, and echoes nothing. - let mut ends = 0; - for _ in 0..100 { - origin.own(p("live/x"), None); - origin.own(p("live/x"), Some(claim(&instance, 0))); - ends += (outputs(&edge).into_iter()) - .filter(|output| matches!(output, Output::Route { reach: None, .. })) - .count(); - } - assert_eq!(ends, 1); - assert_eq!(origin.retained().echoes, 1); - - edge.echo(1); - let read = outputs(&edge); - assert!(matches!(read[0], Output::Route { id: 1, reach: Some(_) }), "{read:?}"); - assert_eq!(read.len(), 2, "the route, then its claim"); - } - - /// A relay sends another stream a claim only once a stream serving it confirmed - /// its instance, so a request naming that instance always finds a route: first when - /// the claim starts, again when it restarts at a new instance, and also where a - /// narrower interest is shown the claim covering it. - #[test] - fn a_claim_goes_on_only_once_served() { - let (old, new) = (Instance::random(), Instance::random()); - let origin = Node::new(1, 1); - let relay = Node::new(2, 2); - let receive = relay.receive(Link::default(), Path::empty()).unwrap(); - let answer = origin.answer(Link::default(), Path::empty()).unwrap(); - let edge = relay.answer(Link::default(), Path::empty()).unwrap(); - let narrow = relay.answer(Link::default(), p("live/x")).unwrap(); - let both = || [instances(&edge), instances(&narrow)]; - - origin.own(p("live"), Some(claim(&old, 0))); - pump(&answer, &receive); - assert_eq!(both(), [[(vec![], None)], [(vec![], None)]], "held until served"); - receive.serving(1, &p("live"), &claim(&old, 0)); - assert_eq!( - both(), - [ - [(vec![(p("live"), Some(old.clone()))], Some(1))], - [(vec![(p("live/x"), Some(old.clone()))], Some(1))] - ] - ); - - origin.own(p("live"), Some(claim(&new, 0))); - pump(&answer, &receive); - assert_eq!( - both(), - [[(vec![], None)], [(vec![], None)]], - "the old instance stays until the new one is served, the floor certifying nothing" - ); - receive.serving(1, &p("live"), &claim(&old, 0)); - assert_eq!(both(), [[], []], "a stale confirmation releases nothing"); - receive.serving(1, &p("live"), &claim(&new, 0)); - assert_eq!( - both(), - [ - [(vec![(p("live"), Some(new.clone()))], Some(2))], - [(vec![(p("live/x"), Some(new.clone()))], Some(2))] - ] - ); - - // A route that ends with a claim held back certifies nothing on its way out. - origin.own(p("live"), Some(claim(&old, 1))); - pump(&answer, &receive); - receive.route(1, None); - assert!(matches!( - &outputs(&edge)[..], - [Output::Batch { floor: None, .. }, Output::Route { reach: None, .. }] - )); - } - - /// A claim presented at a narrower interest waits on the shown claim covering it, - /// whichever that is by the time it is confirmed: a confirmed one that covers it - /// again once the claim between goes, and one that moves to a narrower path before - /// it is confirmed. - #[test] - fn a_held_claim_follows_the_claim_covering_it() { - let (first, second) = (Instance::random(), Instance::random()); - let origin = Node::new(1, 1); - let relay = Node::new(2, 2); - let receive = relay.receive(Link::default(), Path::empty()).unwrap(); - let answer = origin.answer(Link::default(), Path::empty()).unwrap(); - origin.own(p("live"), Some(claim(&first, 0))); - pump(&answer, &receive); - receive.serving(1, &p("live"), &claim(&first, 0)); - origin.own(p("live/x"), Some(claim(&first, 0))); - pump(&answer, &receive); - let narrow = relay.answer(Link::default(), p("live/x/y")).unwrap(); - assert_eq!(instances(&narrow), [(vec![], None)], "held on the unconfirmed live/x"); - origin.own(p("live/x"), None); - pump(&answer, &receive); - assert_eq!( - instances(&narrow), - [(vec![(p("live/x/y"), Some(first.clone()))], Some(3))], - "covered by the confirmed live again" - ); - - origin.own(p("live"), Some(claim(&second, 0))); - pump(&answer, &receive); - assert_eq!(instances(&narrow), [(vec![], None)]); - origin.own(p("live/x"), Some(claim(&second, 0))); - origin.own(p("live"), None); - pump(&answer, &receive); - assert_eq!( - instances(&narrow), - [(vec![], None)], - "now held on the unconfirmed live/x" - ); - receive.serving(1, &p("live/x"), &claim(&second, 0)); - assert_eq!( - instances(&narrow), - [(vec![(p("live/x/y"), Some(second.clone()))], Some(6))] - ); - } - - /// A batch's changes, each claim by its instance, and its floor. - type Instances = (Vec<(Path, Option)>, Option); - - /// The batches for identity 1 a stream would send. - fn instances(stream: &Stream) -> Vec { - (outputs(stream).into_iter()) - .filter_map(|output| match output { - Output::Batch { id: 1, changes, floor } => { - let changes = (changes.into_iter()) - .map(|(path, claim)| (path, claim.map(|claim| claim.instance))) - .collect(); - Some((changes, floor)) - } - _ => None, - }) - .collect() - } - - /// The seqnos of the finite ROUTE messages for `id` a stream would send. - fn routes(stream: &Stream, id: Id) -> Vec { - (outputs(stream).into_iter()) - .filter_map(|output| match output { - Output::Route { id: to, reach } if to == id => match reach { - Some(Reach::Finite(advert)) => Some(advert.seqno), - reach => panic!("unexpected reach {reach:?}"), - }, - _ => None, - }) - .collect() - } - - fn finite(seqno: Seqno, metric: Metric) -> Option { - Some(Reach::Finite(Advert { - seqno, - metric, - down_only: false, - rtt: None, - hops: [].into(), - })) - } - - /// A route that already crossed as many nodes as one may is not forwarded, since this - /// node's hop would take it past what any peer can decode, so a session never fails to - /// encode one. One hop fewer forwards at exactly the limit. - #[test] - fn a_route_at_the_hop_limit_is_not_forwarded() { - use crate::route::MAX_HOPS; - for (hops, forwarded) in [(MAX_HOPS - 1, Some(MAX_HOPS)), (MAX_HOPS, None)] { - let relay = Node::new(2, 2); - let to_b = relay.answer(Link::default(), Path::empty()).unwrap(); - let from_a = relay.receive(Link::default(), Path::empty()).unwrap(); - from_a.route( - 5, - Some(Reach::Finite(Advert { - seqno: 1, - metric: 1, - down_only: false, - rtt: None, - hops: (100..100 + hops as u64).collect(), - })), - ); - // A route goes on the wire only with a claim it shows, once that is served. - let live = claim(&Instance::random(), 0); - from_a.batch(5, vec![(p("live"), Some(live.clone()))], Some(0)); - from_a.serving(5, &p("live"), &live); - let sent: Vec<_> = (outputs(&to_b).into_iter()) - .filter_map(|output| match output { - Output::Route { - id: 5, - reach: Some(Reach::Finite(advert)), - } => Some(advert), - _ => None, - }) - .collect(); - assert_eq!( - sent.iter().map(|advert| advert.hops.len()).max(), - forwarded, - "{hops} hops" - ); - for advert in sent { - let start = crate::lite::AnnounceBroadcast::RouteStart { node: 5, advert }; - crate::coding::Encode::encode_bytes(&start, crate::lite::Version::Lite07).unwrap(); - } - } - } - - /// A relay's own seqno advances only when a session over which another node - /// learned a path to it closes: a client that forwards nothing costs no ROUTE. - #[test] - fn a_relay_bumps_only_when_a_route_link_closes() { - let relay = Node::new(2, 2); - relay.own(p("live/r"), Some(claim(&Instance::random(), 0))); - let forwarding = Link::default(); - // Two peer relays, each forwarding a route of a third node. - let to_a = relay.answer(forwarding.clone(), Path::empty()).unwrap(); - let from_a = relay.receive(Link::default(), Path::empty()).unwrap(); - let to_b = relay.answer(forwarding.clone(), Path::empty()).unwrap(); - let from_b = relay.receive(Link::default(), Path::empty()).unwrap(); - from_a.route(5, finite(0, 1)); - from_b.route(6, finite(0, 1)); - assert_eq!(routes(&to_b, 2), [0]); - outputs(&to_a); - - // A client sending only its own route connects and leaves. - let to_client = relay.answer(forwarding, Path::empty()).unwrap(); - let from_client = relay.receive(Link::default(), Path::empty()).unwrap(); - from_client.route(9, finite(0, 0)); - drop((to_client, from_client)); - assert!(routes(&to_a, 2).is_empty()); - assert!(routes(&to_b, 2).is_empty()); - - // A permit change refiling a peer's stream is no link change either. - // The refile replays no forwarded route, yet the session stays a route link. - let refiled = relay.receive(Link::default(), Path::empty()).unwrap(); - from_a.replaced_by(&refiled); - drop(from_a); - assert!(routes(&to_b, 2).is_empty()); - - // Losing a forwarding peer does change a path. - drop((to_a, refiled)); - assert_eq!(routes(&to_b, 2), [1]); - drop(from_b); - } - - /// A node that forwards nothing advances its seqno whenever a session carrying - /// its route closes, so moving to a farther relay needs no ROUTE_REQUEST. - #[test] - fn a_publisher_bumps_when_any_session_closes() { - let publisher = Node::new(9, 9); - publisher.own(p("live/p"), Some(claim(&Instance::random(), 0))); - let origin = Link { - proxy: false, - ..Link::default() - }; - let first = publisher.answer(origin.clone(), Path::empty()).unwrap(); - let second = publisher.answer(origin, Path::empty()).unwrap(); - assert_eq!(routes(&second, 9), [0]); - drop(first); - assert_eq!(routes(&second, 9), [1]); - - // A session whose prefix matches no claim carries no route, so its close moves no path. - let unmatched = publisher - .answer( - Link { - proxy: false, - ..Link::default() - }, - p("other"), - ) - .unwrap(); - assert!(outputs(&unmatched).is_empty()); - // One whose prefix a claim meets only later, after another session closed. - let later = publisher - .answer( - Link { - proxy: false, - ..Link::default() - }, - p("later"), - ) - .unwrap(); - drop(unmatched); - assert!(routes(&second, 9).is_empty()); - - // It carries the route from then on, so its close does move a path. - publisher.own(p("later/p"), Some(claim(&Instance::random(), 0))); - assert_eq!(routes(&later, 9), [1]); - drop(later); - assert_eq!(routes(&second, 9), [2]); - } - - #[test] - fn a_route_lost_with_its_session_leaves_the_others_serving() { - let instance = Instance::random(); - let origin = Node::new(1, 1); - let relay = Node::new(2, 2); - let first = relay.receive(Link::default(), Path::empty()).unwrap(); - let first_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - let second = relay.receive(Link::default(), Path::empty()).unwrap(); - let second_answer = origin.answer(Link::default(), Path::empty()).unwrap(); - origin.own(p("live/x"), Some(claim(&instance, 0))); - pump(&first_answer, &first); - pump(&second_answer, &second); - outputs(&first); - assert!(matches!(served(outputs(&second))[..], [.., Output::Serve { .. }])); - second.serving(1, &p("live/x"), &claim(&instance, 0)); - - drop(first); - assert!(served(outputs(&second)).is_empty(), "already served where it still is"); - assert!(relay.shown().contains_key(&(1, p("live/x")))); - } - - #[test] - fn a_held_route_keeps_serving_where_it_was_but_unavailable() { - let instance = Instance::random(); - let relay = Node::new(2, 2); - let receive = relay.receive(Link::default(), Path::empty()).unwrap(); - let advert = Advert { - seqno: 0, - metric: 0, - down_only: false, - rtt: None, - hops: [].into(), - }; - receive.route(1, Some(Reach::Finite(advert))); - receive.batch(1, vec![(p("live/x"), Some(claim(&instance, 0)))], Some(1)); - assert!(matches!(served(outputs(&receive))[..], [Output::Serve { .. }])); - receive.serving(1, &p("live/x"), &claim(&instance, 0)); - - receive.route(1, Some(Reach::Held)); - assert!(matches!( - &served(outputs(&receive))[..], - [Output::Serve { served, .. }] if !served.available - )); - receive.route(1, None); - assert!(matches!(served(outputs(&receive))[..], [Output::Unserve { .. }])); - } - - /// What a stream holds back leaves nothing waiting once it goes, whether withdrawn, - /// confirmed, or closed with the stream, so nothing builds up as claims churn. - #[test] - fn a_held_claim_leaves_nothing_waiting_once_it_goes() { - let instance = Instance::random(); - let origin = Node::new(1, 1); - let relay = Node::new(2, 2); - let receive = relay.receive(Link::default(), Path::empty()).unwrap(); - let answer = origin.answer(Link::default(), Path::empty()).unwrap(); - let waiting = || -> usize { - (relay.lock().waiting.values()) - .flat_map(BTreeMap::values) - .map(BTreeSet::len) - .sum() - }; - - let edge = relay.answer(Link::default(), Path::empty()).unwrap(); - origin.own(p("live"), Some(claim(&instance, 0))); - pump(&answer, &receive); - assert_eq!(waiting(), 1); - receive.serving(1, &p("live"), &claim(&instance, 0)); - assert_eq!(waiting(), 0, "confirmed"); - - // The route stays up through the confirmed claim while the one below churns. - origin.own(p("live/a"), Some(claim(&instance, 0))); - pump(&answer, &receive); - assert_eq!(waiting(), 1); - origin.own(p("live/a"), None); - pump(&answer, &receive); - assert_eq!(waiting(), 0, "withdrawn"); - - origin.own(p("live/a"), Some(claim(&instance, 0))); - pump(&answer, &receive); - assert_eq!(waiting(), 1); - drop(edge); - assert_eq!(waiting(), 0, "closed"); - } - - /// A peer's GOAWAY moves the route this node forwards onto another stream at once. - #[test] - fn a_goaway_moves_the_forwarded_route_to_another_stream() { - let instance = Instance::random(); - let relay = Node::new(2, 2); - let edge = relay.answer(Link::default(), Path::empty()).unwrap(); - let first = relay.receive(Link::default(), Path::empty()).unwrap(); - let second = relay.receive(Link::default(), Path::empty()).unwrap(); - for (stream, hop) in [(&first, 10), (&second, 20)] { - let advert = Advert { - seqno: 0, - metric: 1, - down_only: false, - rtt: None, - hops: [hop].into(), - }; - stream.route(5, Some(Reach::Finite(advert))); - stream.batch(5, vec![(p("live/x"), Some(claim(&instance, 0)))], Some(1)); - } - let hops = |stream: &Stream| -> Vec> { - (outputs(stream).into_iter()) - .filter_map(|output| match output { - Output::Route { - id: 5, - reach: Some(Reach::Finite(advert)), - } => Some(advert.hops.to_vec()), - _ => None, - }) - .collect() - }; - assert_eq!(hops(&edge), [vec![10, 2]]); - - first.goaway(); - assert_eq!(hops(&edge), [vec![20, 2]]); - } - - /// Streams dropped while a session's streams close disconnect together, once the - /// last guard held over them goes. - #[test] - fn dropped_streams_disconnect_once_the_last_closing_guard_goes() { - let instance = Instance::random(); - let relay = Node::new(2, 2); - let receive = relay.receive(Link::default(), Path::empty()).unwrap(); - receive.route(5, finite(0, 0)); - receive.batch(5, vec![(p("live/x"), Some(claim(&instance, 0)))], Some(1)); - let (outer, inner) = (relay.closing(), relay.closing()); - drop(receive); - drop(inner); - assert!(relay.shown().contains_key(&(5, p("live/x"))), "another guard is held"); - drop(outer); - assert!(relay.shown().is_empty()); - } - - /// The origin declares its hop id, without the qualified marker, or a random - /// identity when nothing is left of it. - #[test] - fn the_origin_declares_its_hop_id_unqualified() { - let hop = |id| std::num::NonZeroU64::new(id).unwrap(); - assert_eq!(Node::origin(hop(5)).declared(), 5); - assert_eq!(Node::origin(hop(QUALIFIED | 5)).declared(), 5); - let drawn = Node::origin(hop(QUALIFIED)).declared(); - assert!((1..=GENERATED).contains(&drawn), "{drawn}"); - } - - /// A declared identity travels as itself on a lite-06 wire, and each qualified key - /// as its own hop id within the bits a generated one keeps, so a relay recognizes - /// only its own routes coming back. - #[test] - fn an_identity_travels_as_its_own_hop() { - assert_eq!(hop(5).id(), 5); - assert_eq!(hop(GENERATED + 1).id(), GENERATED + 1); - let hops: BTreeSet = (0..256).map(|key| hop(QUALIFIED | key).id()).collect(); - assert_eq!(hops.len(), 256, "one hop per key"); - assert!(hops.iter().all(|id| (1..=GENERATED).contains(id)), "{hops:?}"); - } -} +mod tests; diff --git a/rs/moq-net/src/route/node/tests.rs b/rs/moq-net/src/route/node/tests.rs new file mode 100644 index 00000000000..ea98a73b29f --- /dev/null +++ b/rs/moq-net/src/route/node/tests.rs @@ -0,0 +1,808 @@ +use super::*; + +fn p(path: &'static str) -> Path { + Path::from_static(path) +} + +fn claim(instance: &Instance, cost: u64) -> Claim { + Claim { + instance: instance.clone(), + epoch: 0, + cost, + upstream: false, + learned: false, + chain: crate::Hops::new(), + } +} + +fn clock() -> Deadlines { + Deadlines::new(Instant::ZERO) +} + +fn peer(tag: u64, node: Id) -> Peer { + Peer { tag, node } +} + +fn outputs(stream: &Stream) -> Vec { + std::iter::from_fn(|| stream.try_output()).collect() +} + +/// Deliver everything `from` would send on the wire to the stream receiving it, +/// returning what else it queued. +fn pump(from: &Stream, to: &Stream) -> Vec { + let mut rest = Vec::new(); + for output in outputs(from) { + match output { + Output::Route { id, prefix, reach } => to.route(id, prefix, reach), + Output::Batch { + id, + prefix, + changes, + floor, + } => to.batch(id, prefix, changes, floor), + Output::Dead(dead) => to.dead(dead), + other => rest.push(other), + } + } + rest +} + +/// The serve commands among a stream's outputs. +fn served(outputs: Vec) -> Vec { + (outputs.into_iter()) + .filter(|output| matches!(output, Output::Serve { .. } | Output::Unserve { .. })) + .collect() +} + +/// The wire messages among a stream's outputs. +fn wire(outputs: Vec) -> Vec { + (outputs.into_iter()) + .filter(|output| !matches!(output, Output::Serve { .. } | Output::Unserve { .. })) + .collect() +} + +/// The claim as a relay (node 2) serves it through the origin's (node 1) route. +fn served_at(claim: Claim, metric: Metric, primary: bool) -> Served { + Served { + claim, + metric, + rtt: None, + available: true, + draining: false, + priority: 0, + primary, + selected: true, + path: std::sync::Arc::new([1, 2]), + } +} + +#[test] +fn a_claim_is_served_through_the_stream_its_route_arrived_on() { + let instance = Instance::random(); + let (origin, relay) = (Node::new(1, 1, clock()), Node::new(2, 2, clock())); + let answer = origin.answer(peer(1, 2), Link::default(), Path::empty()).unwrap(); + let receive = relay.receive(peer(1, 1), Link::default()).unwrap(); + origin.own(p("live/x"), Some(claim(&instance, 5))); + pump(&answer, &receive); + assert_eq!( + served(outputs(&receive)), + [Output::Serve { + id: 1, + named: 1, + claim: p("live/x"), + prefix: Path::empty(), + served: served_at(claim(&instance, 5), 1, true), + }] + ); + + origin.own(p("live/x"), None); + pump(&answer, &receive); + assert_eq!( + served(outputs(&receive)), + [Output::Unserve { + id: 1, + claim: p("live/x"), + prefix: Path::empty(), + ended: true + }] + ); +} + +/// A claim covered by routes at two prefixes is served along both, and a reader follows +/// the narrower one. +#[test] +fn the_narrowest_live_route_is_primary() { + let instance = Instance::random(); + let (origin, relay) = (Node::new(1, 1, clock()), Node::new(2, 2, clock())); + let wide = origin.answer(peer(1, 2), Link::default(), Path::empty()).unwrap(); + let narrow = origin.answer(peer(2, 2), Link::default(), p("live")).unwrap(); + let via_wide = relay.receive(peer(1, 1), Link::default()).unwrap(); + let via_narrow = relay.receive(peer(2, 1), Link::default()).unwrap(); + origin.own(p("live/x"), Some(claim(&instance, 0))); + pump(&wide, &via_wide); + pump(&narrow, &via_narrow); + let serve = |prefix: &'static str, primary| Output::Serve { + id: 1, + named: 1, + claim: p("live/x"), + prefix: p(prefix), + served: served_at(claim(&instance, 0), 1, primary), + }; + assert_eq!(served(outputs(&via_wide)), [serve("", false)]); + assert_eq!(served(outputs(&via_narrow)), [serve("live", true)]); + + assert_eq!( + relay.forward(3, 1, &p("live/x"), &Path::empty()), + None, + "no stream answers that session" + ); + assert_eq!(origin.forward(1, 1, &p("live/x"), &Path::empty()), Some(Via::Here)); +} + +/// A request a peer sends follows the route this node offered it under the prefix it names. +#[test] +fn a_forwarded_request_follows_the_route_it_names() { + let instance = Instance::random(); + let (origin, relay) = (Node::new(1, 1, clock()), Node::new(2, 2, clock())); + let answer = origin.answer(peer(1, 2), Link::default(), Path::empty()).unwrap(); + let receive = relay.receive(peer(1, 1), Link::default()).unwrap(); + let downstream = relay.answer(peer(5, 3), Link::default(), Path::empty()).unwrap(); + origin.own(p("live/x"), Some(claim(&instance, 0))); + pump(&answer, &receive); + outputs(&receive) + .into_iter() + .filter_map(|output| match output { + Output::Serve { + id, + claim, + prefix, + served, + .. + } => Some((id, claim, prefix, served)), + _ => None, + }) + .for_each(|(id, claim, prefix, served)| receive.serving(id, &claim, &prefix, &served.claim)); + assert!(!wire(outputs(&downstream)).is_empty(), "the relay offers its route"); + assert_eq!( + relay.forward(5, 1, &p("live/x"), &Path::empty()), + Some(Via::Route { prefix: Path::empty() }) + ); +} + +/// A stream that ends while its session stays up ends its routes quietly; one that ends +/// with its session reports the hop dead to every neighbour sent a route through it. +#[test] +fn only_a_closed_session_reports_its_hop_dead() { + for closed in [false, true] { + let instance = Instance::random(); + let (origin, relay, leaf) = ( + Node::new(1, 1, clock()), + Node::new(2, 2, clock()), + Node::new(3, 3, clock()), + ); + let answer = origin.answer(peer(1, 2), Link::default(), Path::empty()).unwrap(); + let receive = relay.receive(peer(1, 1), Link::default()).unwrap(); + let down = relay.answer(peer(2, 3), Link::default(), Path::empty()).unwrap(); + let up = leaf.receive(peer(1, 2), Link::default()).unwrap(); + origin.own(p("live/x"), Some(claim(&instance, 0))); + pump(&answer, &receive); + for output in served(outputs(&receive)) { + if let Output::Serve { + id, + claim, + prefix, + served, + .. + } = output + { + receive.serving(id, &claim, &prefix, &served.claim); + } + } + pump(&down, &up); + outputs(&up); + + let guard = closed.then(|| relay.closing(1)); + drop(receive); + drop(guard); + let sent = wire(outputs(&down)); + let dead = sent + .iter() + .any(|output| matches!(output, Output::Dead(Dead::Closed { .. }))); + assert_eq!(dead, closed, "{sent:?}"); + assert_eq!( + relay.lock().router.deadline().is_some(), + closed, + "only a closed session casts doubt" + ); + } +} + +/// A route started and ended while the reader stalled leaves it nothing to read. +#[test] +fn a_route_ended_unread_leaves_nothing() { + let instance = Instance::random(); + let origin = Node::new(1, 1, clock()); + let answer = origin.answer(peer(1, 2), Link::default(), Path::empty()).unwrap(); + outputs(&answer); + origin.own(p("live/x"), Some(claim(&instance, 0))); + origin.own(p("live/x"), None); + // The origin's own route stays for the prefix; its claims net out. + let read = outputs(&answer); + let starts = (read.iter()) + .filter( + |output| matches!(output, Output::Batch { changes, .. } if changes.iter().any(|(_, claim)| claim.is_some())), + ) + .count(); + assert_eq!(starts, 0, "{read:?}"); +} + +/// However many changes queue while the reader stalls, it reads the net change. +#[test] +fn a_stalled_reader_holds_the_net_change() { + let origin = Node::new(1, 1, clock()); + let answer = origin.answer(peer(1, 2), Link::default(), Path::empty()).unwrap(); + outputs(&answer); + let instance = Instance::random(); + for cost in 0..1000 { + origin.own(p("live/x"), Some(claim(&instance, cost))); + } + let read = outputs(&answer); + let batches: Vec<_> = (read.iter()) + .filter_map(|output| match output { + Output::Batch { changes, .. } => Some(changes.clone()), + _ => None, + }) + .collect(); + assert_eq!(batches, [vec![(p("live/x"), Some(claim(&instance, 999)))]]); +} + +/// A DEAD applies to the routes the peer holds when it reads it, so no route change +/// queued after it is read before it. +#[test] +fn nothing_moves_across_a_dead() { + let offer = |metric| Offer { + path: [super::super::path::Hop { node: 1, session: 0 }].into(), + draining: false, + priority: 0, + metric, + down_only: false, + rtt: None, + }; + let mut outbox = Outbox::default(); + let mut waiting = Waiting::default(); + let key = (1, Path::empty()); + outbox.route(7, key.clone(), Some(Reach::Finite(offer(1))), &mut waiting); + outbox.dead(7, Dead::Draining { node: 1 }, &mut waiting); + outbox.route(7, key.clone(), Some(Reach::Finite(offer(2))), &mut waiting); + let read: Vec = std::iter::from_fn(|| outbox.pop()).collect(); + let route = |metric| Output::Route { + id: 1, + prefix: Path::empty(), + reach: Some(Reach::Finite(offer(metric))), + }; + assert_eq!(read, [route(1), Output::Dead(Dead::Draining { node: 1 }), route(2)]); +} + +/// A reader that falls behind repeated failures of the routes it holds overruns its +/// stream, while one behind a failure of every hop a path can cross does not, and +/// reading what was queued clears it. A route a DEAD ends counts no longer, so neither +/// does a publisher that came and went. +#[test] +fn a_reader_behind_repeated_failures_overruns() { + use super::super::path::Hop; + let offer = |session| Offer { + path: [Hop { node: 1, session: 0 }, Hop { node: 2, session }].into(), + draining: false, + priority: 0, + metric: 1, + down_only: false, + rtt: None, + }; + let dead = |session, gone| Dead::Closed { + hop: Hop { node: 2, session }, + far: 3, + moved: Vec::new(), + gone, + }; + + // One route whose hop keeps failing, the route moving on after each. + let mut outbox = Outbox::default(); + let mut waiting = Waiting::default(); + let key = (1, Path::empty()); + outbox.route(7, key.clone(), Some(Reach::Finite(offer(0))), &mut waiting); + let mut failures = 0; + while !outbox.overrun() { + failures += 1; + outbox.dead(7, dead(failures, false), &mut waiting); + outbox.route(7, key.clone(), Some(Reach::Finite(offer(failures))), &mut waiting); + } + assert!(failures > MAX_HOPS as u64, "overran {failures} failures behind"); + assert_eq!(std::iter::from_fn(|| outbox.pop()).count() as u64, 2 * failures + 1); + assert!(!outbox.overrun()); + assert_eq!(outbox.sealed, 0, "everything read"); + + // Publishers that each come and go with their session, the DEAD ending each route. + let mut outbox = Outbox::default(); + let mut publishers = 0; + while !outbox.overrun() { + publishers += 1; + assert!(publishers < 1000, "a reader holding no route never overran"); + let key = (publishers, Path::empty()); + outbox.route(7, key, Some(Reach::Finite(offer(publishers))), &mut waiting); + outbox.dead(7, dead(publishers, true), &mut waiting); + } + assert!(outbox.known.is_empty(), "every route ended with its DEAD"); + + // Many routes through one hop queued unread, which one DEAD ends together: a + // single failure, however many routes it ends, never overruns. + let mut outbox = Outbox::default(); + let routes = 4 * MAX_HOPS as u64; + for id in 1..=routes { + outbox.route(7, (id, Path::empty()), Some(Reach::Finite(offer(1))), &mut waiting); + } + outbox.dead(7, dead(1, true), &mut waiting); + assert!(!outbox.overrun(), "{} sealed", outbox.sealed); + let read = std::iter::from_fn(|| outbox.pop()).count() as u64; + assert_eq!(read, routes + 1, "each route's start, then the DEAD that ends it"); + assert_eq!(outbox.sealed, 0, "everything read"); + + // The same routes sealed by an earlier DEAD, which leaves them up, then ended by a + // later one: what the first sealed stops counting once they end. + let mut outbox = Outbox::default(); + for id in 1..=routes { + outbox.route(7, (id, Path::empty()), Some(Reach::Finite(offer(1))), &mut waiting); + } + outbox.dead(7, Dead::Draining { node: 9 }, &mut waiting); + outbox.dead(7, dead(1, true), &mut waiting); + assert!(!outbox.overrun(), "{} sealed", outbox.sealed); + let read = std::iter::from_fn(|| outbox.pop()).count() as u64; + assert_eq!(read, routes + 2, "each route's start, then both DEADs"); + assert_eq!(outbox.sealed, 0, "everything read"); + assert!(outbox.sealing.is_empty()); +} + +/// A claim started and withdrawn before the reader read it leaves nothing queued, so a +/// route whose claims churn at fresh paths keeps only its live ones however slowly the +/// peer reads. A withdrawal of a claim the reader holds still goes, and so does one of a +/// claim sent ahead of a DEAD the reader has yet to read. +#[test] +fn a_claim_started_and_withdrawn_unread_leaves_nothing() { + let offer = Offer { + path: [super::super::path::Hop { node: 1, session: 0 }].into(), + draining: false, + priority: 0, + metric: 1, + down_only: false, + rtt: None, + }; + let instance = Instance::random(); + let mut outbox = Outbox::default(); + let mut waiting = Waiting::default(); + let key = (1, Path::empty()); + let mut batch = |outbox: &mut Outbox, path: Path, claim: Option, floor| { + let changes = (vec![(path, claim)], Some(floor)); + outbox.batch(7, key.clone(), changes, |_, _| None, &mut waiting); + }; + let read = |changes: Vec<(Path, Option)>, floor| Output::Batch { + id: 1, + prefix: Path::empty(), + changes, + floor: Some(floor), + }; + outbox.route(7, key.clone(), Some(Reach::Finite(offer)), &mut Waiting::default()); + batch(&mut outbox, p("live/held"), Some(claim(&instance, 0)), 1); + assert_eq!(std::iter::from_fn(|| outbox.pop()).count(), 2); + + for n in 0..100 { + let path = Path::new(format!("live/{n}").as_str()).unwrap(); + batch(&mut outbox, path.clone(), Some(claim(&instance, 0)), 2 + 2 * n); + batch(&mut outbox, path, None, 3 + 2 * n); + } + batch(&mut outbox, p("live/held"), None, 500); + let sent: Vec = std::iter::from_fn(|| outbox.pop()).collect(); + assert_eq!(sent, [read(vec![(p("live/held"), None)], 500)]); + assert!(outbox.claims.is_empty(), "the reader holds no claim"); + + batch(&mut outbox, p("live/a"), Some(claim(&instance, 0)), 501); + outbox.dead(7, Dead::Draining { node: 1 }, &mut Waiting::default()); + batch(&mut outbox, p("live/a"), None, 502); + let sent: Vec = std::iter::from_fn(|| outbox.pop()).collect(); + assert_eq!( + sent, + [ + read(vec![(p("live/a"), Some(claim(&instance, 0)))], 501), + Output::Dead(Dead::Draining { node: 1 }), + read(vec![(p("live/a"), None)], 502), + ] + ); +} + +/// A route the reader read ends even when its ROUTE_END is queued alone, with no batch +/// withdrawing its claims ahead of it, as when a hold ends. +#[test] +fn a_read_route_ends_alone() { + let offer = Offer { + path: [super::super::path::Hop { node: 1, session: 0 }].into(), + draining: false, + priority: 0, + metric: 1, + down_only: false, + rtt: None, + }; + let mut outbox = Outbox::default(); + let mut waiting = Waiting::default(); + let key = (1, Path::empty()); + outbox.route(7, key.clone(), Some(Reach::Finite(offer)), &mut waiting); + outbox.dead(7, Dead::Draining { node: 1 }, &mut waiting); + assert_eq!(std::iter::from_fn(|| outbox.pop()).count(), 2); + outbox.route(7, key, None, &mut waiting); + let read: Vec = std::iter::from_fn(|| outbox.pop()).collect(); + assert_eq!( + read, + [Output::Route { + id: 1, + prefix: Path::empty(), + reach: None + }] + ); +} + +/// A relay sends a claim on only once a stream serving it confirmed its instance, so a +/// peer never names an instance this relay's origin cannot route yet. +#[test] +fn a_claim_goes_on_only_once_served() { + let instance = Instance::random(); + let (origin, relay) = (Node::new(1, 1, clock()), Node::new(2, 2, clock())); + let answer = origin.answer(peer(1, 2), Link::default(), Path::empty()).unwrap(); + let receive = relay.receive(peer(1, 1), Link::default()).unwrap(); + let down = relay.answer(peer(2, 3), Link::default(), Path::empty()).unwrap(); + outputs(&down); + origin.own(p("live/x"), Some(claim(&instance, 0))); + pump(&answer, &receive); + let claims = |read: Vec| -> Vec { + (read.into_iter()) + .filter_map(|output| match output { + Output::Batch { changes, .. } => Some(changes), + _ => None, + }) + .flatten() + .filter_map(|(path, claim)| claim.map(|_| path)) + .collect() + }; + assert!(claims(outputs(&down)).is_empty(), "held back until served"); + for output in served(outputs(&receive)) { + if let Output::Serve { + id, + claim, + prefix, + served, + .. + } = output + { + receive.serving(id, &claim, &prefix, &served.claim); + } + } + assert_eq!(claims(outputs(&down)), [p("live/x")]); +} + +/// The router's next timer reaches the task that fires it, and firing it lifts the doubt. +#[test] +fn the_doubt_timer_fires_on_the_origins_clock() { + let instance = Instance::random(); + let clock = clock(); + let (origin, relay) = ( + Node::new(1, 1, Deadlines::new(Instant::ZERO)), + Node::new(2, 2, clock.clone()), + ); + let answer = origin.answer(peer(1, 2), Link::default(), Path::empty()).unwrap(); + let receive = relay.receive(peer(1, 1), Link::default()).unwrap(); + origin.own(p("live/x"), Some(claim(&instance, 0))); + pump(&answer, &receive); + let guard = relay.closing(1); + drop(receive); + drop(guard); + let waiter = kio::Waiter::noop(); + let Poll::Ready(Some(at)) = relay.poll_deadline(None, &waiter) else { + panic!("a doubt is timed"); + }; + assert_eq!(at, Instant::ZERO.saturating_add(Doubt::DEFAULT.timer)); + clock.advance(at); + relay.timeout(); + assert_eq!(relay.poll_deadline(Some(at), &waiter), Poll::Ready(None)); +} + +/// A route the peer could not extend is not forwarded: with this node's hop it leaves +/// no room for the peer's, which would refuse it, and one already at the limit would +/// take it past what any peer can decode, so a session never fails to encode one. One +/// hop fewer forwards, a hop short of the limit. +#[test] +fn a_route_at_the_hop_limit_is_not_forwarded() { + use crate::route::MAX_HOPS; + for (hops, forwarded) in [ + (MAX_HOPS - 2, Some(MAX_HOPS - 1)), + (MAX_HOPS - 1, None), + (MAX_HOPS, None), + ] { + let relay = Node::new(2, 2, clock()); + let from_a = relay.receive(peer(1, 100 + hops as Id - 1), Link::default()).unwrap(); + let to_b = relay.answer(peer(2, 3), Link::default(), Path::empty()).unwrap(); + let offer = Offer { + path: (0..hops as Id) + .map(|i| super::super::path::Hop { + node: 100 + i, + session: 1, + }) + .collect(), + draining: false, + priority: i8::MAX, + metric: 1, + down_only: false, + rtt: None, + }; + from_a.route(100, Path::empty(), Some(Reach::Finite(offer))); + // A route goes on the wire only with a claim it shows, once that is served. + let live = claim(&Instance::random(), 0); + from_a.batch(100, Path::empty(), vec![(p("live"), Some(live.clone()))], Some(0)); + from_a.serving(100, &p("live"), &Path::empty(), &live); + let sent: Vec = (outputs(&to_b).into_iter()) + .filter_map(|output| match output { + Output::Route { + reach: Some(Reach::Finite(offer)), + .. + } => Some(offer), + _ => None, + }) + .collect(); + assert_eq!( + sent.iter().map(|offer| offer.path.len()).max(), + forwarded, + "{hops} hops" + ); + for offer in sent { + let start = crate::lite::AnnounceBroadcast::RouteStart { + node: 100, + suffix: Path::empty(), + offer, + }; + crate::coding::Encode::encode_bytes(&start, crate::lite::Version::Lite07).unwrap(); + } + } +} + +/// Serve every claim a stream's outputs ask it to, as its session would. +fn serve_all(stream: &Stream) { + for output in served(outputs(stream)) { + if let Output::Serve { + id, + claim, + prefix, + served, + .. + } = output + { + stream.serving(id, &claim, &prefix, &served.claim); + } + } +} + +/// What a stream holds back leaves nothing waiting once it goes, whether withdrawn, +/// confirmed, or closed with the stream, so nothing builds up as claims churn. +#[test] +fn a_held_claim_leaves_nothing_waiting_once_it_goes() { + let instance = Instance::random(); + let (origin, relay) = (Node::new(1, 1, clock()), Node::new(2, 2, clock())); + let answer = origin.answer(peer(1, 2), Link::default(), Path::empty()).unwrap(); + let receive = relay.receive(peer(1, 1), Link::default()).unwrap(); + let waiting = || -> usize { + (relay.lock().waiting.values()) + .flat_map(BTreeMap::values) + .map(BTreeSet::len) + .sum() + }; + + let edge = relay.answer(peer(2, 3), Link::default(), Path::empty()).unwrap(); + origin.own(p("live"), Some(claim(&instance, 0))); + pump(&answer, &receive); + assert_eq!(waiting(), 1); + serve_all(&receive); + assert_eq!(waiting(), 0, "confirmed"); + + // The route stays up through the confirmed claim while the one below churns. + origin.own(p("live/a"), Some(claim(&instance, 0))); + pump(&answer, &receive); + assert_eq!(waiting(), 1); + origin.own(p("live/a"), None); + pump(&answer, &receive); + assert_eq!(waiting(), 0, "withdrawn"); + + origin.own(p("live/a"), Some(claim(&instance, 0))); + pump(&answer, &receive); + assert_eq!(waiting(), 1); + drop(edge); + assert_eq!(waiting(), 0, "closed"); +} + +/// A peer's GOAWAY moves the route this node forwards onto another stream at once. +#[test] +fn a_goaway_moves_the_forwarded_route_to_another_stream() { + let instance = Instance::random(); + let relay = Node::new(2, 2, clock()); + let edge = relay.answer(peer(3, 3), Link::default(), Path::empty()).unwrap(); + let first = relay.receive(peer(1, 10), Link::default()).unwrap(); + let second = relay.receive(peer(2, 20), Link::default()).unwrap(); + for (stream, node, metric) in [(&first, 10, 1), (&second, 20, 2)] { + let offer = Offer { + path: [Hop { node: 5, session: 0 }, Hop { node, session: 1 }].into(), + draining: false, + priority: i8::MAX, + metric, + down_only: false, + rtt: None, + }; + let live = claim(&instance, 0); + stream.route(5, Path::empty(), Some(Reach::Finite(offer))); + stream.batch(5, Path::empty(), vec![(p("live/x"), Some(live.clone()))], Some(1)); + stream.serving(5, &p("live/x"), &Path::empty(), &live); + } + let paths = |stream: &Stream| -> Vec> { + (outputs(stream).into_iter()) + .filter_map(|output| match output { + Output::Route { + id: 5, + reach: Some(Reach::Finite(offer)), + .. + } => Some(offer.path.iter().map(|hop| hop.node).collect()), + _ => None, + }) + .collect() + }; + assert_eq!(paths(&edge), [vec![5, 10, 2]]); + + first.goaway(); + assert_eq!(paths(&edge), [vec![5, 20, 2]]); +} + +/// Streams dropped while a session's streams close disconnect together, once the +/// last guard held over them goes. +#[test] +fn dropped_streams_disconnect_once_the_last_closing_guard_goes() { + let instance = Instance::random(); + let relay = Node::new(2, 2, clock()); + let receive = relay.receive(peer(1, 1), Link::default()).unwrap(); + let offer = Offer { + path: [Hop { node: 1, session: 0 }].into(), + draining: false, + priority: i8::MAX, + metric: 0, + down_only: false, + rtt: None, + }; + receive.route(1, Path::empty(), Some(Reach::Finite(offer))); + receive.batch( + 1, + Path::empty(), + vec![(p("live/x"), Some(claim(&instance, 0)))], + Some(1), + ); + let (outer, inner) = (relay.closing(1), relay.closing(1)); + drop(receive); + drop(inner); + assert!(relay.shown().contains_key(&(1, p("live/x"))), "another guard is held"); + drop(outer); + assert!(relay.shown().is_empty()); +} + +/// The origin declares its hop id, without the qualified marker, or a random +/// identity when nothing is left of it. +#[test] +fn the_origin_declares_its_hop_id_unqualified() { + let id = |id| std::num::NonZeroU64::new(id).unwrap(); + assert_eq!(Node::origin(id(5), clock()).declared(), 5); + assert_eq!(Node::origin(id(QUALIFIED | 5), clock()).declared(), 5); + let drawn = Node::origin(id(QUALIFIED), clock()).declared(); + assert!((1..=GENERATED).contains(&drawn), "{drawn}"); +} + +/// A declared identity travels as itself on a lite-06 wire, and each qualified key +/// as its own hop id within the bits a generated one keeps, so a relay recognizes +/// only its own routes coming back. +#[test] +fn an_identity_travels_as_its_own_hop() { + assert_eq!(hop(5).id(), 5); + assert_eq!(hop(GENERATED + 1).id(), GENERATED + 1); + let hops: BTreeSet = (0..256).map(|key| hop(QUALIFIED | key).id()).collect(); + assert_eq!(hops.len(), 256, "one hop per key"); + assert!(hops.iter().all(|id| (1..=GENERATED).contains(id)), "{hops:?}"); +} + +/// A claim held back for confirmation when a DEAD seals its batch goes after the DEAD +/// once confirmed, so the sealed batch's floor certifies nothing, since it cannot cover +/// a claim read only later, and the release closes with that floor instead. +#[test] +fn a_floor_sealed_while_a_claim_is_held_back_certifies_nothing() { + let offer = Offer { + path: [super::super::path::Hop { node: 1, session: 0 }].into(), + draining: false, + priority: 0, + metric: 1, + down_only: false, + rtt: None, + }; + let instance = Instance::random(); + let mut outbox = Outbox::default(); + let mut waiting = Waiting::default(); + let key = (1, Path::empty()); + outbox.route(7, key.clone(), Some(Reach::Finite(offer)), &mut waiting); + let changes = vec![ + (p("live/x"), Some(claim(&instance, 0))), + (p("live/y"), Some(claim(&instance, 0))), + ]; + let awaits = |path: &Path, _: &Claim| Some(*path == p("live/y")); + outbox.batch(7, key.clone(), (changes, Some(5)), awaits, &mut waiting); + outbox.dead(7, Dead::Draining { node: 1 }, &mut waiting); + outbox.release(&key, &p("live/y")); + + let batches: Vec<(Vec, Option)> = (std::iter::from_fn(|| outbox.pop())) + .filter_map(|output| match output { + Output::Batch { changes, floor, .. } => Some((changes.into_iter().map(|(path, _)| path).collect(), floor)), + _ => None, + }) + .collect(); + assert_eq!(batches.len(), 2, "{batches:?}"); + assert_eq!( + batches[0], + (vec![p("live/x")], None), + "sealed before live/y was confirmed" + ); + assert_eq!( + batches[1], + (vec![p("live/y")], Some(5)), + "the release closes with the sealed floor" + ); +} + +/// The floor a held-back claim's release closes with is the newest batch's, whatever +/// order the reader reads the older batches in: reading a batch sealed before the one +/// that held the claim back does not take the release's floor back. +#[test] +fn a_release_closes_with_the_newest_floor_after_older_batches_are_read() { + let offer = Offer { + path: [super::super::path::Hop { node: 1, session: 0 }].into(), + draining: false, + priority: 0, + metric: 1, + down_only: false, + rtt: None, + }; + let instance = Instance::random(); + let mut outbox = Outbox::default(); + let mut waiting = Waiting::default(); + let key = (1, Path::empty()); + let awaits = |path: &Path, _: &Claim| Some(*path == p("live/y")); + outbox.route(7, key.clone(), Some(Reach::Finite(offer)), &mut waiting); + let x = vec![(p("live/x"), Some(claim(&instance, 0)))]; + outbox.batch(7, key.clone(), (x, Some(5)), awaits, &mut waiting); + outbox.dead(7, Dead::Draining { node: 1 }, &mut waiting); + let y = vec![(p("live/y"), Some(claim(&instance, 0)))]; + outbox.batch(7, key.clone(), (y, Some(6)), awaits, &mut waiting); + outbox.dead(7, Dead::Draining { node: 2 }, &mut waiting); + + // The reader reads up to the first DEAD, and live/y is confirmed only then. + let mut read = Vec::new(); + while let Some(output) = outbox.pop() { + let first = matches!(output, Output::Dead(Dead::Draining { node: 1 })); + read.push(output); + if first { + break; + } + } + outbox.release(&key, &p("live/y")); + read.extend(std::iter::from_fn(|| outbox.pop())); + let last = (read.into_iter()).rev().find_map(|output| match output { + Output::Batch { changes, floor, .. } => { + Some((changes.into_iter().map(|(path, _)| path).collect::>(), floor)) + } + _ => None, + }); + assert_eq!(last, Some((vec![p("live/y")], Some(6)))); +} diff --git a/rs/moq-net/src/route/path.rs b/rs/moq-net/src/route/path.rs index 3cf5da46656..2707df16d36 100644 --- a/rs/moq-net/src/route/path.rs +++ b/rs/moq-net/src/route/path.rs @@ -41,11 +41,11 @@ //! claims, and a wider route carries its claims and floor to each added prefix //! inside it. So a swap re-sends nothing both sets cover. //! -//! `rs/moq-net/verify/route/RouteHandoff.tla` model-checks the rule. Nothing drives -//! this router on the wire yet: sessions run [`super::Router`] until the switch -//! (`quest/m2/cluster-routes/switch.md`). - -#![cfg_attr(not(test), allow(dead_code, reason = "only the fixtures drive it until the switch"))] +//! `rs/moq-net/verify/route/RouteHandoff.tla` model-checks the rule. Every lite-07 +//! session drives it through [`super::node::Node`]. Lite-07 opens one announce +//! stream per prefix, so the node files each stream under a key of its own whose +//! one edit is the prefix it opened with, in effect from the start; edits on one +//! stream wait for the route stream (`quest/m2/cluster-routes/route-stream.md`). use std::{ cmp::Reverse, @@ -94,7 +94,7 @@ pub(crate) struct Offer { impl Offer { /// A received path must leave room for this node's hop. - fn extendable(&self) -> bool { + pub(crate) fn extendable(&self) -> bool { !self.path.is_empty() && self.path.len() < MAX_HOPS } @@ -226,6 +226,12 @@ impl Default for Doubt { } } +/// The longest a node holds a route with no live path: a neighbour that has not answered +/// by then counts as answering with nothing. A hold protects presence, not loop freedom, +/// so ending it early is safe, and a neighbour that never answers holds nothing for good +/// (epic Plan item 30). +pub(crate) const HOLD: Duration = Duration::from_secs(10); + /// Where a request for a claim goes from a node. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) enum Forward { @@ -252,6 +258,8 @@ struct Heard { state: State, /// Its claims as presented at its prefix, with one floor or none yet. view: View, + /// Whether its sender closed a batch of claims yet, so `view` is what it carries. + batched: bool, } /// A dead hop keeps its own deadline; another failure of the far node cannot extend it. @@ -422,7 +430,7 @@ impl Route { } #[derive(Clone, Debug)] -struct Session { +struct Session { /// How this node names the session in the hops of the routes it learns there. number: u64, /// How the neighbour names itself in a path. @@ -435,11 +443,14 @@ struct Session { asked: BTreeMap, /// The set each edit sent and not yet marked leaves, oldest first. edits: VecDeque>, + /// The key whose heard routes answer what this one was sent: another key of the same + /// session when the session answers each of its peer's streams under a key of its own. + listen: Option, /// The routes it offered, so closing it costs what it held. heard: BTreeSet, } -impl Session { +impl Session { /// The prefixes this node asks the neighbour for once every edit in flight lands. fn requested(&self) -> BTreeSet { match self.edits.back() { @@ -538,7 +549,7 @@ pub(crate) struct Router { hop: Id, doubt: Doubt, opened: u64, - sessions: BTreeMap, + sessions: BTreeMap>, /// The sessions currently answering an interest. answering: BTreeMap>, origins: BTreeMap>, @@ -566,6 +577,10 @@ pub(crate) struct Router { doubted: BTreeMap, /// Nodes that sent GOAWAY, kept while a heard route names one or it is in doubt. draining: BTreeSet, + /// When each held route lets go whatever is left unanswered. + held_until: BTreeMap, + /// The latest time any input carried. + now: Instant, /// This node's interest: the prefixes its reader sees, none inside another. watch: BTreeSet, reader: Reader, @@ -594,7 +609,7 @@ fn names(path: &[Hop], node: Id) -> bool { } /// Whether the node that offered `path` learned it from `node`. -fn child(path: &[Hop], node: Id) -> bool { +pub(crate) fn child(path: &[Hop], node: Id) -> bool { path.len() >= 2 && path[path.len() - 2].node == node } @@ -608,6 +623,7 @@ pub(super) fn meet(a: &Path, b: &Path) -> Option { } /// Whether a prefix of `set` lies inside another. +#[cfg(test)] fn nested(set: &BTreeSet) -> bool { set.iter() .any(|prefix| ancestors(prefix).any(|above| above != *prefix && set.contains(&above))) @@ -621,6 +637,9 @@ trait Carried: Clone { fn held(&self) -> bool; /// Its claims, unless it was ended. fn view(&mut self) -> Option<&mut View>; + /// It moved to a wider key with no floor: what its sender carries there is + /// unknown until the sender closes a batch at that key. + fn widened(&mut self) {} } impl Carried for Heard { @@ -635,6 +654,10 @@ impl Carried for Heard { fn view(&mut self) -> Option<&mut View> { Some(&mut self.view) } + + fn widened(&mut self) { + self.batched = false; + } } impl Carried for Sent { @@ -725,6 +748,7 @@ fn rekey>( }; view.items.remove(prefix); view.floors.clear(); + route.widened(); arriving.entry((*id, wider)).or_default().push((key.clone(), route)); continue; } @@ -809,6 +833,8 @@ impl Router { dead: BTreeMap::new(), doubted: BTreeMap::new(), draining: BTreeSet::new(), + held_until: BTreeMap::new(), + now: Instant::ZERO, watch: BTreeSet::from([Path::empty()]), reader: Reader::default(), shown_claims: BTreeMap::new(), @@ -837,9 +863,20 @@ impl Router { self.reader.changed() } + /// The time is `now`: a hold started on the next input lets go at its deadline from here. + pub fn clock(&mut self, now: Instant) { + self.now = self.now.max(now); + } + + /// The latest instant this router was handed. + pub fn now(&self) -> Instant { + self.now + } + // --- Sessions --- - /// A session to `neighbor`, which names itself `hop` in a path, opened. + /// A session to `neighbor`, which names itself `hop` in a path, opened. A `hop` of 0 + /// is learned from the last hop of the first route the neighbour sends. pub fn connect(&mut self, neighbor: N, hop: Id, link: Link) -> Result<(), Error> { if link.cost == 0 { return Err(Error::ZeroCost); @@ -853,6 +890,7 @@ impl Router { draining: false, asked: BTreeMap::new(), edits: VecDeque::new(), + listen: None, heard: BTreeSet::new(), }; self.sessions.insert(neighbor, session); @@ -861,9 +899,72 @@ impl Router { Ok(()) } + /// `answer` answers one of the streams the session `session` opened: the routes the + /// peer sends back on `session` are what answers a hold on what `answer` was sent. + /// A session whose peer opens several streams answers each under a key of its own. + pub fn pair(&mut self, answer: N, session: N) { + if let Some(entry) = self.sessions.get_mut(&answer) { + entry.listen = (answer != session).then_some(session); + } + } + + /// How `neighbor` names itself in a path: as it declared, else as the stream its + /// session's routes arrive on learned, or 0 while this node has learned neither. + pub fn hop_of(&self, neighbor: N) -> Id { + match self.sessions.get(&neighbor).map_or(0, |session| session.hop) { + 0 => self + .sessions + .get(&self.listen(neighbor)) + .map_or(0, |session| session.hop), + hop => hop, + } + } + + /// The key whose heard routes answer what `neighbor` was sent. + fn listen(&self, neighbor: N) -> N { + self.sessions + .get(&neighbor) + .and_then(|session| session.listen) + .unwrap_or(neighbor) + } + + /// Drop `neighbor`, a stream that ended while its session stays up: what it heard + /// goes as ROUTE_ENDs would take it, and what it was sent as a cancelled interest. + /// No hop is reported dead and no node doubted, since no session closed. + pub fn retire(&mut self, neighbor: N) { + let rx = self.listen(neighbor); + let Some(session) = self.sessions.remove(&neighbor) else { + return; + }; + let mut ids = BTreeSet::new(); + for route in &session.heard { + ids.insert(route.0); + self.forget_heard(route, neighbor); + self.filed.remove(&(route.0, neighbor)); + self.unwait(neighbor, route); + } + if let Some(answer) = self.answering.remove(&neighbor) { + for (key, sent) in answer.tx { + ids.insert(key.0); + if let Some(offer) = sent.offer() { + self.unindex_sent(neighbor, &key, &offer.path); + } + if self.child_pending(key.0, rx, &key.1) { + self.waiting + .entry((key.0, sent.src.clone())) + .or_default() + .insert((rx, key.1.clone())); + } + self.unreflect(neighbor, &key, &sent.src); + } + } + self.update(ids, Vec::new(), None); + } + /// The sessions to `neighbors` closed at `now`: everything learned or sent on them /// goes, and each names its hop to the others. pub fn disconnect(&mut self, neighbors: impl IntoIterator, now: Instant) { + self.clock(now); let mut ids = BTreeSet::new(); let mut hops = Vec::new(); for neighbor in neighbors { @@ -889,7 +990,12 @@ impl Router { session: session.number, }; ids.extend(self.fail(hop, session.hop, None, Vec::new(), false, now)); - hops.push(Report::Closed(hop)); + // A scoped peer names itself as it likes, so its close neither doubts that node + // nor names it to the neighbours: the routes it sent end one by one instead. + if session.link.scope.is_none() { + ids.extend(self.suspect(session.hop, now)); + hops.push(Report::Closed(hop)); + } } self.update(ids, hops, None); } @@ -925,13 +1031,14 @@ impl Router { } self.unreflect(neighbor, key, &sent.src); } - let rekeyed = rekey(&old, &new, tx.clone(), self.sessions[&neighbor].hop); + let rekeyed = rekey(&old, &new, tx.clone(), self.hop_of(neighbor)); // A child that learned a route this stream no longer carries under its key // may still owe an answer to the route it reflected. + let rx = self.listen(neighbor); for key in rekeyed.gone { - if self.child_pending(key.0, neighbor, &key.1) { + if self.child_pending(key.0, rx, &key.1) { let src = (key.0, tx[&key].src.clone()); - self.waiting.entry(src).or_default().insert((neighbor, key.1)); + self.waiting.entry(src).or_default().insert((rx, key.1)); } } let tx = rekeyed.routes; @@ -1002,12 +1109,20 @@ impl Router { /// once only onto a path that avoids it, and tells the neighbours it sent a route /// through it. pub fn goaway(&mut self, neighbor: N, now: Instant) { + self.clock(now); let Some(session) = self.sessions.get_mut(&neighbor) else { return; }; session.draining = true; let node = session.hop; + let scoped = session.link.scope.is_some(); let mut ids = self.through(neighbor); + // A scoped peer's GOAWAY drains only the routes it sent: the node it names itself + // could be any, and routes through that node outside its scope are not its to rank. + if scoped { + self.update(ids, Vec::new(), None); + return; + } ids.extend(self.drains(node, now)); self.update(ids, vec![Report::Draining(node)], None); } @@ -1036,6 +1151,7 @@ impl Router { /// it asks each neighbour for to match. What this node knows outside them is /// forgotten. A route a neighbour sends stays, but is believed for a region only /// once every edit in flight that could drop it has taken effect. + #[cfg(test)] pub fn watch(&mut self, prefixes: BTreeSet) -> Result<(), Error> { if nested(&prefixes) { return Err(Error::Nested); @@ -1068,6 +1184,7 @@ impl Router { /// Whether `neighbor` sent the initial set of `prefix`, which this node asks it /// for, or `None` when it does not ask for it. + #[cfg(test)] pub fn is_synced(&self, neighbor: N, prefix: &Path) -> Option { self.sessions.get(&neighbor)?.asked.get(prefix).copied() } @@ -1103,12 +1220,18 @@ impl Router { self.scopes.entry(id).or_default().insert(Path::empty()); } + /// How this node names itself in a path. + pub fn hop(&self) -> Id { + self.hop + } + /// The identity this node declares. pub fn declared(&self) -> Option { self.origins.keys().next().copied() } /// The declared identity's claim revision. + #[cfg(test)] pub fn revision(&self) -> Option { let origin = self.origins.values().next()?; origin.view.floors.get(&Path::empty()).copied() @@ -1178,24 +1301,32 @@ impl Router { // --- Received messages --- /// A route `from` sent for `named` under `prefix`: ROUTE_START or ROUTE_UPDATE, - /// ROUTE_HOLD, or ROUTE_END for `None`. One outside what this node asked `from` - /// for, as of the last marker, is refused. + /// ROUTE_HOLD, or ROUTE_END for `None`, filed as [`Self::admit`] says. pub fn route(&mut self, from: N, named: Id, prefix: Path, reach: Option) { - let Some(session) = self.sessions.get(&from) else { + let Some((id, prefixes)) = self.admit(from, named, &prefix) else { return; }; - if !ancestors(&prefix).any(|above| session.asked.contains_key(&above)) { - return; + for prefix in prefixes { + self.route_at(from, (named, id), prefix, reach.clone()); + } + } + + /// [`Router::route`] for the route filed as `id` at `prefix`. + fn route_at(&mut self, from: N, (named, id): (Id, Id), prefix: Path, reach: Option) { + if let Some(Reach::Finite(offer)) = &reach + && let Some(last) = offer.path.last() + { + let session = self.sessions.get_mut(&from).expect("checked above"); + if session.hop == 0 { + session.hop = last.node; + } } - let Some(id) = self.admit(from, named, &prefix) else { - return; - }; let key = (id, prefix); let entry = self.rx.get(&key).and_then(|routes| routes.get(&from)); match reach { // A path a node could not extend is refused, as if ended. Some(Reach::Finite(offer)) if offer.extendable() => { - let view = entry.map(|entry| entry.view.clone()).unwrap_or_default(); + let (view, batched) = entry.map_or_else(Default::default, |entry| (entry.view.clone(), entry.batched)); if entry.is_some() { self.forget_heard(&key, from); } @@ -1203,7 +1334,13 @@ impl Router { true => State::Dead, false => State::Live, }; - self.hear(key.clone(), from, Heard { offer, state, view }); + let heard = Heard { + offer, + state, + view, + batched, + }; + self.hear(key.clone(), from, heard); if id != named { self.filed.insert((id, from), named); } @@ -1228,9 +1365,26 @@ impl Router { /// A batch of claim changes for the route `from` sent for `named` under `prefix`, /// closed by its floor. pub fn batch(&mut self, from: N, named: Id, prefix: Path, changes: Vec<(Path, Option)>, floor: Option) { - let Some(id) = self.admit(from, named, &prefix) else { + let Some((id, prefixes)) = self.admit(from, named, &prefix) else { return; }; + if let [only] = prefixes.as_slice() + && *only == prefix + { + return self.batch_at(from, id, prefix, changes, floor); + } + // Narrowed to regions: each sees only the changes within it. + for prefix in prefixes { + let changes = (changes.iter()) + .filter(|(claim, _)| within(claim, &prefix)) + .cloned() + .collect(); + self.batch_at(from, id, prefix, changes, floor); + } + } + + /// [`Router::batch`] for the route filed as `id` at `prefix`. + fn batch_at(&mut self, from: N, id: Id, prefix: Path, changes: Vec<(Path, Option)>, floor: Option) { let key = (id, prefix.clone()); let Some(heard) = self.rx.get_mut(&key).and_then(|routes| routes.get_mut(&from)) else { return; @@ -1239,6 +1393,7 @@ impl Router { let claims: Vec = changes.iter().map(|(claim, _)| claim.clone()).collect(); let floors = floor.map(|rev| BTreeMap::from([(prefix, rev)])).unwrap_or_default(); heard.view.apply(changes, floors); + heard.batched = true; let delta = Delta { route: key.clone(), from, @@ -1248,11 +1403,32 @@ impl Router { self.run([id].into(), Vec::new(), Some(delta), BTreeMap::new(), Some(key)); } + /// Whether this node applies a DEAD from `from` naming `hop` and `far`: a peer only + /// reports hops it advertised to us, so one of the routes heard from it must cross + /// `hop` right after `far`. Binding the far node to that path keeps unrelated DEADs + /// from allocating hop or doubt state. Any other DEAD is dropped whole. + pub fn credits(&self, from: N, hop: Hop, far: Id) -> bool { + let credited = self.rx_hops.get(&hop).and_then(|peers| peers.get(&from)); + credited.is_some_and(|keys| { + keys.iter().any(|key| { + self.rx[key][&from] + .offer + .path + .windows(2) + .any(|pair| pair[1] == hop && pair[0].node == far) + }) + }) + } + /// A [`Dead`] from `from`, at `now`. pub fn dead(&mut self, from: N, dead: Dead, now: Instant) { + self.clock(now); if !self.sessions.contains_key(&from) { return; } + // A scoped peer's news covers only the routes it sent: it could otherwise name a hop + // it only read in a path and drop healthy routes outside its scope (epic Plan item 30). + let scoped = self.sessions[&from].link.scope.is_some(); let (hop, far, moved, gone) = match dead { Dead::Closed { hop, far, moved, gone } => (hop, far, moved, gone), Dead::Draining { node } => { @@ -1261,6 +1437,22 @@ impl Router { if node == self.hop || !credited { return; } + if scoped { + let keys: BTreeSet = (self.rx_hops.range(hop_range(node))) + .filter_map(|(_, peers)| peers.get(&from)) + .flatten() + .cloned() + .collect(); + for key in &keys { + let heard = self.rx.get_mut(key).and_then(|routes| routes.get_mut(&from)); + if let Some(heard) = heard { + heard.offer.draining = true; + } + } + let ids = keys.into_iter().map(|(id, _)| id).collect(); + self.update(ids, Vec::new(), None); + return; + } // A node already draining ranks and is doubted as before: the news only // reaches a neighbour not yet told. Each upstream relays it, so a repeat // must not revisit every route sent through the node. @@ -1273,27 +1465,22 @@ impl Router { return; } }; - // A peer only reports hops it previously advertised to us. Also bind the far - // node to that path: unrelated DEADs must not allocate hop or doubt state. - let credited = self.rx_hops.get(&hop).and_then(|peers| peers.get(&from)); - if !credited.is_some_and(|keys| { - keys.iter().any(|key| { - self.rx[key][&from] - .offer - .path - .windows(2) - .any(|pair| pair[1] == hop && pair[0].node == far) - }) - }) { + if !self.credits(from, hop, far) { return; } let mut filed = Vec::new(); for moved in moved { - if let Some(id) = self.admit(from, moved.id, &moved.prefix) { - filed.push(((id, moved.prefix), moved.offer)); + if let Some((id, prefixes)) = self.admit(from, moved.id, &moved.prefix) { + filed.extend(prefixes.into_iter().map(|prefix| ((id, prefix), moved.offer.clone()))); } } - let ids = self.fail(hop, far, Some(from), filed, gone, now); + if scoped { + let ids = self.fail_heard(hop, from, filed, gone); + self.update(ids, Vec::new(), None); + return; + } + let mut ids = self.fail(hop, far, Some(from), filed, gone, now); + ids.extend(self.suspect(far, now)); self.update(ids, vec![Report::Closed(hop)], None); } @@ -1333,8 +1520,8 @@ impl Router { // A route this node used or held through `from` moves with the route it heard, // so a hold or a doubt carries over rather than letting go of its claims. Of // several moving to one key, the one in use goes: only it may stay on a path - // through a doubted node. - let mut carried: BTreeMap> = BTreeMap::new(); + // through a doubted node. A hold keeps its deadline. + let mut carried: BTreeMap, RouteId)> = BTreeMap::new(); for (was, now) in rekeyed.moved { let Some(route) = self.routes.get(&was).filter(|route| route.next() == Some(from)) else { continue; @@ -1342,12 +1529,15 @@ impl Router { if self.routes.contains_key(&now) { continue; } - if carried.get(&now).is_none_or(|kept| !kept.live() && route.live()) { - carried.insert(now, route.clone()); + if carried.get(&now).is_none_or(|(kept, _)| !kept.live() && route.live()) { + carried.insert(now, (route.clone(), was)); } } - for (now, route) in carried { + for (now, (route, was)) in carried { self.scopes.entry(now.0).or_default().insert(now.1.clone()); + if let Some(&at) = self.held_until.get(&was) { + self.held_until.insert(now.clone(), at); + } self.routes.insert(now, route); } for (id, named) in filed { @@ -1360,6 +1550,7 @@ impl Router { } /// `from` sent the initial set of `prefix`, which this node asks it for. + #[cfg(test)] pub fn synced(&mut self, from: N, prefix: Path) { let synced = (self.sessions.get_mut(&from)).and_then(|session| session.asked.get_mut(&prefix)); if let Some(synced) = synced { @@ -1369,6 +1560,11 @@ impl Router { /// Fire every doubt and dead-hop timer due by `now`. pub fn timeout(&mut self, now: Instant) { + self.clock(now); + let lapsed: BTreeSet = (self.held_until.iter()) + .filter(|(_, at)| **at <= now) + .map(|((id, _), _)| *id) + .collect(); let due: Vec = self .doubted .iter() @@ -1379,10 +1575,10 @@ impl Router { .iter() .filter_map(|(hop, state)| state.deadline().filter(|at| *at <= now).map(|_| *hop)) .collect(); - if due.is_empty() && expired.is_empty() { + if due.is_empty() && expired.is_empty() && lapsed.is_empty() { return; } - let mut ids = BTreeSet::new(); + let mut ids = lapsed; for node in due { self.doubted.remove(&node); ids.extend(self.naming(node)); @@ -1400,6 +1596,7 @@ impl Router { .values() .map(|state| state.deadline()) .chain(self.dead.values().filter_map(|state| state.deadline())) + .chain(self.held_until.values().copied()) .min() } @@ -1419,6 +1616,7 @@ impl Router { // --- Queries --- /// The route this node uses for `id` under `prefix`. + #[cfg(test)] pub fn route_for(&self, id: Id, prefix: &Path) -> Option<&Route> { self.routes.get(&(id, prefix.clone())) } @@ -1521,7 +1719,7 @@ impl Router { let route = self.routes.get(&(id, prefix.clone()))?; let offer = route.offer()?; if let Some(prev) = prev - && names(&offer.path, self.sessions[&prev].hop) + && names(&offer.path, self.hop_of(prev)) { return None; } @@ -1533,6 +1731,13 @@ impl Router { }) } + /// Whether this node sent `to` a live route for `id` covering `prefix`. + pub fn sent(&self, to: N, id: Id, prefix: &Path) -> bool { + (self.answering.get(&to)) + .and_then(|answer| answer.tx.get(&(id, prefix.clone()))) + .is_some_and(|sent| sent.offer().is_some()) + } + /// The identity this node names for `id` to `next`: the one `next` sent, if this /// node qualified it. pub fn name(&self, next: N, id: Id) -> Id { @@ -1610,6 +1815,8 @@ impl Router { dead, doubted, draining, + held_until, + now: _, watch, reader, shown_claims, @@ -1626,9 +1833,10 @@ impl Router { let doubted: Vec<_> = (doubted.iter()) .map(|(node, state)| (node, matches!(state, Suspicion::Doubted(_)), left(state.deadline()))) .collect(); + let held: Vec<_> = (held_until.iter()).map(|(key, at)| (key, left(*at))).collect(); super::digest(hasher, &(hop, doubt, opened, sessions, answering, origins, rx)); super::digest(hasher, &(rx_hops, tx_hops, tx_nodes, reflects, waiting, routes, views)); - super::digest(hasher, &(scopes, filed, dead, doubted, draining, watch)); + super::digest(hasher, &(scopes, filed, dead, doubted, draining, held, watch)); super::digest(hasher, &(reader, shown_claims, rng)); } @@ -1671,21 +1879,42 @@ impl Router { // --- Internals: what is heard --- /// The key this node files `named` under when `from` sends it a route covering - /// `prefix`, or `None` when refused: a scoped neighbour's identities are all - /// qualified, one already qualified is refused, and so is a route outside its scope. - /// A node also ignores routes to its own identities. - fn admit(&self, from: N, named: Id, prefix: &Path) -> Option { + /// `prefix`, and the prefixes it files the route at, or `None` when refused: a scoped + /// neighbour's identities are all qualified, one already qualified is refused, and + /// its route counts only within its regions, so one covering a region is filed at + /// that region and one outside them all is refused. One outside what this node asked + /// `from` for, as of the last marker, is refused too. A node also ignores routes to + /// its own identities. + fn admit(&self, from: N, named: Id, prefix: &Path) -> Option<(Id, Vec)> { let session = self.sessions.get(&from)?; - let id = match &session.link.scope { + let (id, prefixes) = match &session.link.scope { Some(scope) => { - if named & QUALIFIED != 0 || !scope.regions.iter().any(|region| within(prefix, region)) { + if named & QUALIFIED != 0 { return None; } - (scope.qualify)(named) | QUALIFIED + let prefixes: Vec = match scope.regions.iter().any(|region| within(prefix, region)) { + true => vec![prefix.clone()], + false => { + let inside: BTreeSet<&Path> = + (scope.regions.iter()).filter(|region| within(region, prefix)).collect(); + // A region inside another is already covered by it. + (inside.iter()) + .filter(|region| !inside.iter().any(|above| above != *region && within(region, above))) + .map(|region| (*region).clone()) + .collect() + } + }; + if prefixes.is_empty() { + return None; + } + ((scope.qualify)(named) | QUALIFIED, prefixes) } - None => named, + None => (named, vec![prefix.clone()]), }; - (!self.origins.contains_key(&id)).then_some(id) + let prefixes: Vec = (prefixes.into_iter()) + .filter(|prefix| ancestors(prefix).any(|above| session.asked.contains_key(&above))) + .collect(); + (!prefixes.is_empty() && !self.origins.contains_key(&id)).then_some((id, prefixes)) } /// Whether `path` holds a hop this node knows is dead. @@ -1799,9 +2028,9 @@ impl Router { self.naming(node) } - /// The session `hop` names closed. `from` reported it, with what became of its own - /// routes through it, or this node saw it close. It casts doubt on `far`. Returns - /// the identities touched. + /// The session `hop` names closed, at the node `far` at its other end. `from` + /// reported it, with what became of its own routes through it, or this node saw it + /// close. Returns the identities touched. fn fail( &mut self, hop: Hop, @@ -1828,32 +2057,8 @@ impl Router { for (key, peer) in through { self.step(); ids.insert(key.0); - let reported = from == Some(peer); - if let Some(offer) = moved.get(&key).filter(|_| reported) { - if offer.extendable() { - self.repath(&key, peer, offer.clone()); - } else { - self.forget_heard(&key, peer); - } - continue; - } - if reported { - if gone { - self.forget_heard(&key, peer); - } else { - let hop = self.hop; - let heard = self - .rx - .get_mut(&key) - .and_then(|routes| routes.get_mut(&peer)) - .expect("heard"); - // A neighbour that holds under this node answers once those behind it have. - // The neighbour cannot tell whether this node qualified the key, so - // both sides decide by the path alone, as the sender's report does. - if !child(&heard.offer.path, hop) { - heard.state = State::Held; - } - } + if from == Some(peer) { + self.reported(&key, peer, moved.get(&key), gone); continue; } let heard = self @@ -1865,12 +2070,64 @@ impl Router { heard.state = State::Dead; } } - if far != self.hop { - let at = now.saturating_add(self.doubt.timer); - let at = self.doubted.get(&far).map_or(at, |state| state.deadline().max(at)); - self.doubted.insert(far, Suspicion::Doubted(at)); - // Any route through that node may have become unusable. - ids.extend(self.naming(far)); + ids + } + + /// A session to `far` closed, as of `now`: no route through it is taken until the + /// timer fires. Returns the identities whose routes rank anew. + fn suspect(&mut self, far: Id, now: Instant) -> BTreeSet { + if far == self.hop { + return BTreeSet::new(); + } + let at = now.saturating_add(self.doubt.timer); + let at = self.doubted.get(&far).map_or(at, |state| state.deadline().max(at)); + self.doubted.insert(far, Suspicion::Doubted(at)); + // Any route through that node may have become unusable. + self.naming(far) + } + + /// What became of the route `peer` sent under `key` through a hop it reported dead: + /// its new path if it moved, else ended if `gone`, else held unless this node is the + /// one it learned the route from. + fn reported(&mut self, key: &RouteId, peer: N, moved: Option<&Offer>, gone: bool) { + if let Some(offer) = moved { + match offer.extendable() { + true => self.repath(key, peer, offer.clone()), + false => self.forget_heard(key, peer), + } + return; + } + if gone { + self.forget_heard(key, peer); + return; + } + let hop = self.hop; + let heard = (self.rx.get_mut(key)) + .and_then(|routes| routes.get_mut(&peer)) + .expect("heard"); + // A neighbour that holds under this node answers once those behind it have. + // The neighbour cannot tell whether this node qualified the key, so both sides + // decide by the path alone, as the sender's report does. + if !child(&heard.offer.path, hop) { + heard.state = State::Held; + } + } + + /// A scoped peer reported `hop` dead: only the routes it sent through the hop change, + /// and nothing is remembered or doubted. Returns the identities touched. + fn fail_heard(&mut self, hop: Hop, from: N, moved: Vec<(RouteId, Offer)>, gone: bool) -> BTreeSet { + let moved: BTreeMap = moved.into_iter().collect(); + let through: Vec = (self.rx_hops.get(&hop)) + .and_then(|peers| peers.get(&from)) + .into_iter() + .flatten() + .cloned() + .collect(); + let mut ids = BTreeSet::new(); + for key in through { + self.step(); + ids.insert(key.0); + self.reported(&key, from, moved.get(&key), gone); } ids } @@ -1902,56 +2159,94 @@ impl Router { } /// Pick the best-ranked usable live route ([`Offer::rank`]), staying on the next - /// hop among equals. + /// hop among equals: on its node, through the latest of its sessions there + /// ([`Link::session`]), and on the stream it uses there. /// Doubt keeps this node from moving to a route, not on the one it uses. fn select(&self, key: &RouteId) -> Option> { let was = self.routes.get(key); let current = was.and_then(Route::next); let using = current.filter(|_| was.is_some_and(Route::live)); - type Rank = ((bool, Reverse, Metric), bool, N); + // The node the next hop names itself, once known. + let staying = (current.and_then(|next| self.sessions.get(&next))) + .map(|session| session.hop) + .filter(|hop| *hop != 0); + type Rank = ((bool, Reverse, Metric), bool, Reverse, bool, N); let mut best: Option<(Rank, Offer)> = None; for (&peer, heard) in self.rx.get(key).into_iter().flatten() { self.step(); - if heard.state != State::Live || names(&heard.offer.path, self.hop) { - continue; - } - let session = &self.sessions[&peer]; - let Some(metric) = heard.offer.metric.checked_add(session.link.cost) else { + let Some(offer) = self.usable_offer(peer, heard, using) else { continue; }; - if Some(peer) != using && self.doubts_path(&heard.offer.path) { - // It may come back once the doubt lifts. - continue; - } - let draining = session.draining || heard.offer.draining || self.drains_path(&heard.offer.path); - let priority = session.link.priority.min(heard.offer.priority); - let rank = ((draining, Reverse(priority), metric), Some(peer) != current, peer); + let session = &self.sessions[&peer]; + let stays = Some(session.hop) == staying; + // The model has one session per pair of nodes: a later one to the next hop's + // node replaces the one in use, as a GOAWAY migration leaves it. + let latest = Reverse(if stays { session.link.session } else { 0 }); + let rank = (offer.rank(), !stays, latest, Some(peer) != current, peer); if best.as_ref().is_some_and(|(best, _)| *best <= rank) { continue; } - let path: Arc<[Hop]> = (heard.offer.path.iter().copied()) - .chain([Hop { - node: self.hop, - session: session.number, - }]) - .collect(); - let offer = Offer { - path, - draining, - priority, - metric, - down_only: session.link.upstream || heard.offer.down_only, - rtt: heard.offer.rtt.zip(session.rtt).and_then(|(a, b)| a.checked_add(b)), - }; best = Some((rank, offer)); } best.map(|((.., next), offer)| Route::Live { next, offer }) } + /// The route `heard` from `peer` gives this node, extended by its hop, or `None` when + /// it is not usable: not live, naming this node, unreachable, or through a node in + /// doubt that this node is not already using it through (`using`). + fn usable_offer(&self, peer: N, heard: &Heard, using: Option) -> Option { + if heard.state != State::Live || names(&heard.offer.path, self.hop) { + return None; + } + let session = &self.sessions[&peer]; + let metric = heard.offer.metric.checked_add(session.link.cost)?; + if Some(peer) != using && self.doubts_path(&heard.offer.path) { + // It may come back once the doubt lifts. + return None; + } + let draining = session.draining || heard.offer.draining || self.drains_path(&heard.offer.path); + let path: Arc<[Hop]> = (heard.offer.path.iter().copied()) + .chain([Hop { + node: self.hop, + session: session.number, + }]) + .collect(); + Some(Offer { + path, + draining, + priority: session.link.priority.min(heard.offer.priority), + metric, + down_only: session.link.upstream || heard.offer.down_only, + rtt: heard.offer.rtt.zip(session.rtt).and_then(|(a, b)| a.checked_add(b)), + }) + } + + /// Every usable route for `key` while this node has a live one, each as this node + /// would take it, the selected one first: a reader here fails over among them at + /// once when the selected one goes. + pub fn usable(&self, key: &RouteId) -> Vec<(N, Offer)> { + let Some(Route::Live { next, offer }) = self.routes.get(key) else { + return Vec::new(); + }; + let mut out = vec![(*next, offer.clone())]; + for (&peer, heard) in self.rx.get(key).into_iter().flatten() { + if peer == *next { + continue; + } + if let Some(offer) = self.usable_offer(peer, heard, Some(*next)) { + out.push((peer, offer)); + } + } + out + } + /// With no live route, keep what an upstream or a child has yet to answer. fn retain(&self, key: &RouteId) -> Option> { - // Only a node that had the route holds it. + // Only a node that had the route holds it, and only until its deadline. let was = self.routes.get(key)?; + if self.held_until.get(key).is_some_and(|at| *at <= self.now) { + return None; + } let path = was.path().clone(); // Its next hop holds, or has yet to say what became of a dead route, on the // path this node's route was built on. A next hop that has since sent another @@ -1984,6 +2279,7 @@ impl Router { } fn child_pending(&self, id: Id, peer: N, prefix: &Path) -> bool { + let peer = self.listen(peer); self.rx .get(&(id, prefix.clone())) .and_then(|routes| routes.get(&peer)) @@ -2050,7 +2346,7 @@ impl Router { return prefix.clone(); } let covering = || ancestors(prefix).rev().map(|above| (id, above)); - let hop = self.sessions[&peer].hop; + let hop = self.hop_of(peer); let best = |usable| { covering() .filter_map(|key| { @@ -2099,6 +2395,26 @@ impl Router { .map(|((_, prefix), view)| (prefix, view)) } + /// What the live route this node selected covering `prefix` carries there, with + /// nothing certified, or `None` while it has none, does not believe it, or its sender + /// has not yet closed a batch of its claims. + fn carried(&self, id: Id, prefix: &Path) -> Option> { + let (above, next) = + ancestors(prefix) + .rev() + .find_map(|above| match self.routes.get(&(id, above.clone()))? { + Route::Live { next, .. } => Some((above, *next)), + _ => None, + })?; + let heard = self.rx.get(&(id, above))?.get(&next)?; + if !heard.batched || !self.believed(next, heard, prefix) { + return None; + } + let mut view = heard.view.present(prefix); + view.floors.clear(); + Some(view) + } + /// The narrowest view of `id` covering `prefix`, or the publisher's own. fn covering_view(&self, id: Id, prefix: &Path) -> Option<(Path, &View)> { if let Some(origin) = self.origins.get(&id) { @@ -2119,7 +2435,7 @@ impl Router { /// Bring every view of `id` up to date: each prefix within the interest that a route /// covers keeps the newest floor a believed route covering it gave, kept whole while - /// a route covers it. Returns whether the views changed shape, and marks changed + /// a route covers it, or what the selected live route carries while none certifies it. Returns whether the views changed shape, and marks changed /// claims in `dirty`. fn know(&mut self, id: Id, keys: &BTreeSet, delta: Option<&Delta>, dirty: &mut Dirty) { if self.origins.contains_key(&id) { @@ -2153,8 +2469,10 @@ impl Router { }; self.scopes.entry(id).or_default().insert(prefix.clone()); let floor = base.floors.get(prefix).copied(); - // The newest floor any believed route covering this prefix gave. + // The newest floor any believed route covering this prefix gave, and whether + // any gave one at all. let mut best: Option<(Rev, RouteId, N)> = None; + let mut floored = false; for above in ancestors(prefix) { let route = (id, above); for (&peer, heard) in self.rx.get(&route).into_iter().flatten() { @@ -2162,7 +2480,11 @@ impl Router { let Some(&rev) = heard.view.floors.get(&route.1) else { continue; }; - if !self.believed(peer, heard, prefix) || Some(rev) <= floor { + if !self.believed(peer, heard, prefix) { + continue; + } + floored = true; + if Some(rev) <= floor { continue; } if best.as_ref().is_none_or(|(best, ..)| rev > *best) { @@ -2171,7 +2493,24 @@ impl Router { } } let Some((_, route, peer)) = best else { - self.views.insert(key, base); + // While no believed route certifies the prefix, it shows what the live + // route this node selected carries there: a sender that cannot vouch for + // its whole interest, such as one refusing a claim it was sent, still + // shows its claims, uncertified. + let view = match floored { + true => None, + false => self.carried(id, prefix), + }; + let view = match view { + Some(view) => { + for (claim, _) in view.diff(&base) { + dirty.claims.insert(claim); + } + view + } + None => base, + }; + self.views.insert(key, view); continue; }; let heard = &self.rx[&route][&peer].view; @@ -2269,11 +2608,9 @@ impl Router { // A route qualified here is a different key from the one this peer // offered. Returning its own path under the qualified key would look like // a child of another route. The peer cannot use that path anyway. - if self.name(peer, id) != id - && (route.next() == Some(peer) - || route - .offer() - .is_some_and(|offer| names(&offer.path, self.sessions[&peer].hop))) + if self.name(self.listen(peer), id) != id + && (route.next() == Some(self.listen(peer)) + || route.offer().is_some_and(|offer| names(&offer.path, self.hop_of(peer)))) { return Want::End; } @@ -2281,12 +2618,12 @@ impl Router { // A route goes to the node that published it no more than to an upstream link // when down-only: neither would use it. Some(offer) if session.link.upstream && offer.down_only => Want::End, - Some(offer) if offer.path[0].node == session.hop => Want::End, + Some(offer) if offer.path[0].node == self.hop_of(peer) => Want::End, // The neighbour could not extend it, so it would refuse it. Some(offer) if !offer.extendable() => Want::End, Some(offer) => Want::Path(offer.clone(), src.clone()), // The next hop it holds under hears nothing until those behind this node answered. - None if route.next() == Some(peer) => { + None if route.next() == Some(self.listen(peer)) => { let key = (id, src.clone()); match self.pending(&key) || self.hopes(&key) { true => return Want::Keep, @@ -2411,7 +2748,7 @@ impl Router { if !sent.held() { // The neighbour it learned the route from takes silence to mean // this node holds under it, so it is told outright. - let parent = child(&sent.offer().expect("sent").path, self.sessions[&peer].hop); + let parent = child(&sent.offer().expect("sent").path, self.hop_of(peer)); holds.push(key.clone()); if parent { parents.push(key.clone()); @@ -2430,8 +2767,7 @@ impl Router { Want::Keep => { // The neighbour takes a first DEAD to mean this node holds every route it // did not learn from the neighbour, so what was sent says so too. - if first && !sent.held() && !child(&sent.offer().expect("sent").path, self.sessions[&peer].hop) - { + if first && !sent.held() && !child(&sent.offer().expect("sent").path, self.hop_of(peer)) { let sent = self .answering .get_mut(&peer) @@ -2688,13 +3024,7 @@ impl Router { let Some(sent) = self.answering[&peer].tx.get(&key) else { continue; }; - if sent.offer().is_some() - || self - .rx - .get(&key) - .and_then(|routes| routes.get(&peer)) - .is_some_and(|heard| heard.state != State::Held && self.is_child(id, peer, &heard.offer.path)) - { + if sent.offer().is_some() || self.child_pending(id, peer, &key.1) { continue; } let src = sent.src.clone(); @@ -2706,8 +3036,12 @@ impl Router { /// Send a final withdrawing batch before retiring a sent offer. fn end_sent(&mut self, peer: N, key: &RouteId, batch: Option>) { + // A floor the neighbour already holds there, with no change, tells it nothing. + let held = (self.answering[&peer].tx.get(key)) + .and_then(Sent::active) + .and_then(|sent| sent.view.floors.get(&key.1).copied()); if let Some((_, changes, floor)) = batch - && (!changes.is_empty() || floor.is_some()) + && (!changes.is_empty() || floor.is_some() && floor != held) { self.out.push(Out::Batch { to: peer, @@ -2873,6 +3207,7 @@ impl Router { route_changed |= self.routes.get(&key) != live.as_ref(); match live { Some(route) => { + self.held_until.remove(&key); self.routes.insert(key, route); } None => { @@ -2922,9 +3257,13 @@ impl Router { for key in held { match self.retain(&key) { Some(route) => { + self.held_until + .entry(key.clone()) + .or_insert(self.now.saturating_add(HOLD)); self.routes.insert(key, route); } None => { + self.held_until.remove(&key); self.routes.remove(&key); } } @@ -3186,7 +3525,9 @@ impl Router { Some((claim.clone(), at)) } (Some((above, wider)), _) => Some((above.clone(), wider)), - (None, at) => Some((claim.clone(), at.or_else(|| self.views.get(&(id, claim.clone())))?)), + (None, Some(at)) => Some((claim.clone(), at)), + // Nothing certified: the narrowest view covering the claim shows what it carries. + (None, None) => self.covering_view(id, claim), } } @@ -3244,9 +3585,15 @@ impl Router { .collect(); #[cfg(test)] self.check_shown(id, dirty, &now); - // A claim no longer shown under a route that still covers it was withdrawn. - let routes = &self.routes; - let ended = |claim: &Path| ancestors(claim).any(|above| routes.contains_key(&(id, above))); + // A claim no longer shown under a route that still covers it was withdrawn, if a + // floor certified that; one that only stopped being carried goes offline. + let (routes, views, origin) = (&self.routes, &self.views, self.origins.contains_key(&id)); + let ended = |claim: &Path| { + ancestors(claim).any(|above| routes.contains_key(&(id, above))) + && (origin + || of(views, id) + .any(|((_, prefix), view)| within(claim, prefix) && view.floors.contains_key(prefix))) + }; for (claim, shown) in &now { match shown { Some(_) => { diff --git a/rs/moq-net/src/route/tests/cache.rs b/rs/moq-net/src/route/tests/cache.rs index 53bdc015200..192b4a580ba 100644 --- a/rs/moq-net/src/route/tests/cache.rs +++ b/rs/moq-net/src/route/tests/cache.rs @@ -213,7 +213,7 @@ impl Mesh { /// before its links. pub fn origin(&mut self, node: Node, id: Id) { self.sim.leaf(node); - self.sim.originate(node, id, 0); + self.sim.originate(node, id); self.sim.announce(node, "live", 0, None); self.origins.insert(id, node); } diff --git a/rs/moq-net/src/route/tests/compare.rs b/rs/moq-net/src/route/tests/compare.rs index 400997f50a1..9e9e777147d 100644 --- a/rs/moq-net/src/route/tests/compare.rs +++ b/rs/moq-net/src/route/tests/compare.rs @@ -1,6 +1,5 @@ -//! One script on both rules: [`Router`], which keeps Babel's distance, and -//! [`path::Router`], which keeps its doubt until the timer fires. The ignored -//! benchmarks also compare the rejected early-collection policy. +//! One script on [`path::Router`], which keeps its doubt until the timer fires. The +//! ignored benchmarks also compare the rejected early-collection policy. //! //! A [`Lab`] builds the same mesh on each, applies one change at a time, and //! delivers in rounds: every message queued when a round starts crosses its link @@ -13,23 +12,19 @@ use std::collections::BTreeSet; use super::*; -/// The production router and its accepted replacement, checked in ordinary CI. -const RULES: [Rule; 2] = [Rule::Distance, Rule::PATH]; +/// The production rule, checked in ordinary CI. +const RULES: [Rule; 1] = [Rule::PATH]; /// The rules compared, in the report's order. -pub(super) const COMPARED: [Rule; 3] = [Rule::Distance, Rule::EARLY, Rule::PATH]; +pub(super) const COMPARED: [Rule; 2] = [Rule::EARLY, Rule::PATH]; -/// A queue of messages, or of echoed ROUTE_ENDs, from one node to another. -#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] -pub(super) enum Chan { - Msg(Node, Node), - Ack(Node, Node), -} +/// A queue of messages from one node to another. +pub(super) type Chan = (Node, Node); /// What a router sent and showed so far. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] pub(super) struct Sent { - /// Route messages of every kind, requests and echoes included. + /// Route messages of every kind. pub msgs: usize, /// Routes those messages name. pub items: usize, @@ -115,8 +110,6 @@ impl Net for Sim { fn close(&mut self, a: Node, b: Node) { self.disconnect(a, b); - self.closed(a); - self.closed(b); } fn mute(&mut self, node: Node, peer: Node) { @@ -126,7 +119,6 @@ impl Net for Sim { fn sever(&mut self, node: Node, peer: Node) { Sim::sever(self, node, peer); - self.closed(peer); } fn reset(&mut self, node: Node) { @@ -134,7 +126,7 @@ impl Net for Sim { } fn publish(&mut self, node: Node, id: Id) { - self.originate(node, id, 0); + self.originate(node, id); } fn claim(&mut self, node: Node, claim: &str, live: bool) { @@ -145,19 +137,14 @@ impl Net for Sim { } fn ready(&self) -> Vec<(Chan, usize)> { - let msgs = (self.busy().into_iter()).map(|(from, to)| (Chan::Msg(from, to), self.queued(from, to))); - let acks = (self.busy_acks().into_iter()).map(|(from, to)| (Chan::Ack(from, to), self.queued_acks(from, to))); - let held = |chan: &Chan| match *chan { - Chan::Msg(from, to) | Chan::Ack(from, to) => self.held.contains(&(from, to)), - }; - msgs.chain(acks).filter(|(chan, _)| !held(chan)).collect() + (self.busy().into_iter()) + .filter(|chan| !self.held.contains(chan)) + .map(|(from, to)| ((from, to), self.queued(from, to))) + .collect() } - fn step(&mut self, chan: Chan) { - match chan { - Chan::Msg(from, to) => self.deliver(from, to), - Chan::Ack(from, to) => self.deliver_ack(from, to), - }; + fn step(&mut self, (from, to): Chan) { + self.deliver(from, to); } fn tick(&mut self) -> bool { @@ -176,8 +163,8 @@ impl Net for Sim { fn sent(&self) -> Sent { let (offline, unavailable, starts) = tally(self.events.iter().map(|event| event.kind)); Sent { - msgs: self.stats.route_msgs + self.stats.requests, - items: self.stats.route_items + self.stats.requests, + msgs: self.stats.route_msgs, + items: self.stats.route_items, claims: self.stats.claim_items, offline, unavailable, @@ -186,17 +173,6 @@ impl Net for Sim { } } -impl Sim { - /// A publisher advances its seqno when a session its route went out on closes, - /// as `route::Node` has it do. A new session lengthens no path, so it does not. - /// The path rule has no seqno. - fn closed(&mut self, node: Node) { - if self.declared[node].is_some() && !self.paths() { - self.bump(node); - } - } -} - /// A mesh of relays, each publisher attached to one of them. #[derive(Clone, Debug)] pub(super) struct Mesh { @@ -858,15 +834,11 @@ pub(super) fn churn(rule: Rule, cases: std::ops::Range) -> Churn { total } -/// Under churn every rule settles and is right once it does, and the distance -/// never loops, even for a step. +/// Under churn every rule settles and is right once it does. #[test] fn every_rule_settles_after_churn() { for rule in RULES { - let cost = churn(rule, 0..60); - if rule == Rule::Distance { - assert_eq!(cost.loops, 0); - } + churn(rule, 0..60); } } @@ -913,7 +885,7 @@ fn path_changes_keep_claims_and_withdrawals_keep_routes() { for costs in &costs { assert!(costs.values().all(|cost| cost.settled && cost.loops == 0)); } - for path in &costs[1..] { + for path in &costs { for change in [Change::Leave, Change::Detour, Change::Home] { assert_eq!(path[&change].sent.claims, 0); } diff --git a/rs/moq-net/src/route/tests/handoff.rs b/rs/moq-net/src/route/tests/handoff.rs index 885a8c85b2f..e2740830f0a 100644 --- a/rs/moq-net/src/route/tests/handoff.rs +++ b/rs/moq-net/src/route/tests/handoff.rs @@ -12,7 +12,7 @@ use std::collections::BTreeSet; use super::path::meet; use super::*; -both!( +fixtures!( an_older_complete_snapshot_revives_no_known_withdrawal, trailing_announcements_report_no_false_absence, a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it, @@ -36,8 +36,8 @@ both!( stale_replay_after_collection_lasts_only_until_the_withdrawal_behind_it, holds_retire_once_nothing_can_recover, exhausted_counters_rotate_the_identity, - a_quiet_origin_is_reached_by_a_kept_request, - sweep_requests_reach_a_quiet_origin, + a_quiet_origin_is_reached_once_a_withdrawal_lands, + sweep_relays_reach_a_quiet_origin, announcement_cost_follows_sessions_and_live_claims, sweep_scoped_handoff, sweep_handoff, @@ -51,7 +51,7 @@ both!( ); /// An origin reaching a node through two relays, the first preferred and the -/// second a feasible successor. +/// second a standby. fn two_relays(rule: Rule) -> Sim { let (origin, a, b, node) = (0, 1, 2, 3); let mut sim = Sim::with(rule, 4); @@ -60,7 +60,7 @@ fn two_relays(rule: Rule) -> Sim { sim.join(origin, b, 1); sim.join(node, a, 1); sim.join(node, b, 2); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); assert!(sim.run()); sim } @@ -111,8 +111,8 @@ fn trailing_announcements_report_no_false_absence(rule: Rule) { sim.connect(node, b); assert!(sim.run()); assert!(sim.queued(b, node) >= 2, "a ROUTE then its announcements"); - // Under the path rule the stream opens with b's own ask and the marker answering node's. - for _ in 0..1 + 2 * usize::from(sim.paths()) { + // The stream opens with b's own ask and the marker answering node's. + for _ in 0..3 { assert!(sim.deliver(b, node), "only the ROUTE"); } sim.disconnect(node, a); @@ -127,12 +127,10 @@ fn trailing_announcements_report_no_false_absence(rule: Rule) { assert!(events.is_empty(), "{events:?}"); } -/// The partitioned local mesh, with a client behind the starved relay. The -/// CDN detour is infeasible until the origin advances its seqno, and through -/// that window the relay and its client keep the claim unavailable rather -/// than ending it. The relay's seqno request recovers it through the CDN. Under -/// the path rule the detour crosses the far end of the closed session, so the -/// relay holds until that doubt lifts, and needs no request. +/// The partitioned local mesh, with a client behind the starved relay. The CDN +/// detour crosses the far end of the closed session, so the relay holds until that +/// doubt lifts, and through that window the relay and its client keep the claim +/// unavailable rather than ending it. fn a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it(rule: Rule) { let (c1, c2, l1, l2, origin, client) = (0, 1, 2, 3, 4, 5); let mut sim = Sim::with(rule, 6); @@ -144,7 +142,7 @@ fn a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it(rule sim.leaf(client); sim.join(origin, l2, 1); sim.join(client, l1, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "local/x", 0, None); assert!(sim.run()); assert_eq!(sim.shows(client, 7, "local/x"), Some(true)); @@ -153,17 +151,8 @@ fn a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it(rule let before = sim.events.len(); sim.hold(l1, c1); sim.disconnect(l1, l2); - match sim.paths() { - true => { - assert!(sim.flow()); - assert!(sim.holds(l1, 7) && !sim.routed(l1, 7), "held while l2 is in doubt"); - } - false => { - assert!(sim.run()); - assert!(sim.starved(l1, 7)); - assert!(sim.grounded(client, 7)); - } - } + assert!(sim.flow()); + assert!(sim.holds(l1, 7) && !sim.routed(l1, 7), "held while l2 is in doubt"); for node in [l1, client] { assert_eq!(sim.shows(node, 7, "local/x"), Some(false), "node {node}"); } @@ -171,9 +160,6 @@ fn a_starved_relay_keeps_presence_unavailable_until_its_request_recovers_it(rule sim.release(l1, c1); assert!(sim.run()); - if !sim.paths() { - assert_eq!(sim.seqno(origin), 1, "the request reached the origin"); - } assert_eq!(sim.next_hop(l1, 7), Some(c1)); let events: Vec = sim.events[before..] .iter() @@ -194,7 +180,7 @@ fn lagging(rule: Rule, watch: &str) -> Sim { sim.join(client, r1, 1); sim.join(client, r2, 1); sim.watch(client, watch); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim } @@ -262,8 +248,8 @@ fn cancelling_interest_retires_its_region(rule: Rule) { [Kind::Start, Kind::End, Kind::Start, Kind::End] ); assert_eq!(revivals(&sim.events), 0, "a retired region, like a collection"); - // The path rule keeps a view only while a route covers it, and none does. - assert_eq!(sim.floors(client, 7), usize::from(!sim.paths())); + // A view stays only while a route covers it, and none does. + assert_eq!(sim.floors(client, 7), 0); sim.check().unwrap(); } @@ -274,7 +260,7 @@ fn narrowing_past_every_claim_sends_only_the_routes_end() { let (origin, client) = (0, 1); let mut sim = Sim::new(2); sim.join(origin, client, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/a/x", 0, None); assert!(sim.run()); assert_eq!(sim.shows(client, 7, "live/a/x"), Some(true)); @@ -359,7 +345,7 @@ fn widening_behind_a_scoped_grant_shows_the_rest_of_the_scope(rule: Rule) { }; sim.link(client, relay, SimLink::cost(1).grant(grant), SimLink::cost(1)); sim.watch(client, "live/a/x/"); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/a/x/1", 0, None); sim.announce(origin, "live/a/y", 0, None); assert!(sim.run()); @@ -370,10 +356,7 @@ fn widening_behind_a_scoped_grant_shows_the_rest_of_the_scope(rule: Rule) { sim.watch(client, ""); assert!(sim.run()); assert_eq!(sim.shows(client, id, "live/a/y"), Some(true)); - let guard = |sim: &Sim| match sim.paths() { - true => sim.kept().views, - false => sim.retained().certified, - }; + let guard = |sim: &Sim| sim.kept().views; let certified = guard(&sim); for _ in 0..8 { sim.announce(origin, "live/a/z", 0, None); @@ -433,7 +416,7 @@ fn scoped_pair(rule: Rule, scopes: [&'static str; 2]) -> Sim { }; sim.link(client, relay, SimLink::cost(1).grant(grant), SimLink::cost(1)); } - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim } @@ -480,7 +463,7 @@ fn two_scoped_uplinks(rule: Rule) -> Sim { sim.link(relay, local, SimLink::cost(1).grant(grant), SimLink::cost(1)); sim.join(relay, z, 1); } - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim } @@ -498,10 +481,6 @@ fn a_disjoint_interest_is_answered_with_nothing(rule: Rule) { assert_eq!(sim.shows(z, id, "live/"), None); assert_eq!(sim.shows(z, id, "live/a/"), Some(true)); assert_eq!(sim.shows(z, id, "live/b/"), Some(true)); - assert!( - sim.paths() || sim.babel(z).views().all(|(_, view)| view.items.is_empty()), - "nothing certified" - ); sim.watch(z, "live/c"); assert!(sim.run()); @@ -518,7 +497,7 @@ fn an_interest_matches_whole_segments(rule: Rule) { let mut sim = Sim::with(rule, 2); sim.leaf(origin); sim.join(origin, client, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/a/x", 0, None); sim.announce(origin, "live/ab", 0, None); sim.watch(client, "live/a"); @@ -562,52 +541,28 @@ fn a_lost_scope_keeps_newer_knowledge_where_still_certified(rule: Rule) { sim.check().unwrap(); } -/// When one scoped uplink closes, its relay is starved and holds, and the -/// third relay knows that scope's claims only uncertified, through the starved -/// relay. Once the other relay's newer seqno makes the starved one infeasible, -/// nothing carries the lost scope and the third lets it go, whether or not the -/// starved relay's own requests get through. Certifiers are feasible neighbors -/// only, so the two never keep the scope alive through each other. The path rule -/// sends no request, and its relay holds until the third answers. +/// When one scoped uplink closes, its relay holds, and the third relay knows that +/// scope's claims only through the holding relay. Once the third answers, nothing +/// carries the lost scope and the third lets it go: the two never keep the scope +/// alive through each other. fn a_relay_lets_go_of_a_scope_its_uplinks_lost(rule: Rule) { let (origin, local, x, z) = (0, 1, 2, 4); - let requests: &[bool] = match rule { - Rule::Distance => &[false, true], - Rule::Path(_) => &[true], - }; - for &request in requests { - let mut sim = two_scoped_uplinks(rule); - sim.announce(origin, "live/a/x", 0, None); - sim.announce(origin, "live/b/z", 0, None); - assert!(sim.run()); - let id = qualified("local", 7); - assert_eq!(sim.shows(z, id, "live/a/x"), Some(true)); - assert_eq!(sim.shows(z, id, "live/b/z"), Some(true)); + let mut sim = two_scoped_uplinks(rule); + sim.announce(origin, "live/a/x", 0, None); + sim.announce(origin, "live/b/z", 0, None); + assert!(sim.run()); + let id = qualified("local", 7); + assert_eq!(sim.shows(z, id, "live/a/x"), Some(true)); + assert_eq!(sim.shows(z, id, "live/b/z"), Some(true)); - if !request { - // Hold every request at the starved relay. - for peer in [z, local] { - sim.hold(x, peer); - } - } - sim.disconnect(x, local); - sim.unannounce(origin, "live/a/x"); - assert!(sim.run()); - if !request { - // Only the ROUTE update reaches z, not the requests. - while sim.deliver(x, z) {} - assert!(sim.run()); - } - assert_eq!(sim.shows(z, id, "live/a/x"), None, "request {request}"); - assert_eq!( - life(&sim, z, "live/a/x").last(), - Some(&Kind::Offline), - "request {request}" - ); - assert_eq!(sim.shows(z, id, "live/b/z"), Some(true), "request {request}"); - assert_eq!(false_ends(&sim.events), 0, "request {request}"); - sim.check().unwrap(); - } + sim.disconnect(x, local); + sim.unannounce(origin, "live/a/x"); + assert!(sim.run()); + assert_eq!(sim.shows(z, id, "live/a/x"), None); + assert_eq!(life(&sim, z, "live/a/x").last(), Some(&Kind::Offline)); + assert_eq!(sim.shows(z, id, "live/b/z"), Some(true)); + assert_eq!(false_ends(&sim.events), 0); + sim.check().unwrap(); } /// Every lite-07 claim covers the paths below it, so a claim above an @@ -644,7 +599,7 @@ fn widening_past_a_changed_claim_moves_it_up(rule: Rule) { sim.leaf(client); sim.join(origin, relay, 1); sim.join(client, relay, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/", 0, None); sim.watch(client, "live/b/z"); assert!(sim.run()); @@ -671,7 +626,7 @@ fn widening_while_the_relay_lost_the_origin_moves_the_claim_up(rule: Rule) { sim.leaf(client); sim.join(origin, relay, 1); sim.join(client, relay, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); assert!(sim.run()); sim.announce(origin, "live/", 2, None); sim.watch(client, "live/b"); @@ -710,7 +665,7 @@ fn a_claim_withdrawn_under_a_live_one_ends(rule: Rule) { sim.leaf(client); sim.join(origin, relay, 1); sim.join(client, relay, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/", 0, None); sim.announce(origin, "live/a/x", 0, None); assert!(sim.run()); @@ -736,8 +691,8 @@ fn wildcard_claims_end_per_origin_and_are_certified_below(rule: Rule) { sim.join(client, r1, 1); sim.join(client, r2, 1); sim.watch(client, "live/a/"); - sim.originate(o1, 10, 0); - sim.originate(o2, 11, 0); + sim.originate(o1, 10); + sim.originate(o2, 11); sim.announce(o1, "live/", 0, Some(1)); sim.announce(o2, "live/", 0, Some(1)); assert!(sim.run()); @@ -769,10 +724,8 @@ fn wildcard_claims_end_per_origin_and_are_certified_below(rule: Rule) { /// An origin process restarts with a fresh identity: its claims start a new /// revision history and the old identity's go offline once nothing can -/// recover them. Reusing the identity would restart its revision below the -/// floors other nodes hold: they keep the dead process's claims and refuse -/// the new ones. The path rule keeps no view once nothing covers it, so a -/// reused identity starts over everywhere. +/// recover them. No view stays once nothing covers it, so a reused identity +/// starts over everywhere too. fn origin_restart_takes_a_fresh_identity_for_its_claims(rule: Rule) { let (origin, a, b) = (0, 1, 2); for fresh in [false, true] { @@ -780,7 +733,7 @@ fn origin_restart_takes_a_fresh_identity_for_its_claims(rule: Rule) { sim.leaf(origin); sim.join(origin, a, 1); sim.join(a, b, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/x", 0, None); sim.unannounce(origin, "live/x"); sim.announce(origin, "live/y", 0, None); @@ -797,30 +750,21 @@ fn origin_restart_takes_a_fresh_identity_for_its_claims(rule: Rule) { assert_eq!(false_ends(&sim.events), 0); let id = if fresh { 8 } else { 7 }; - sim.originate(origin, id, 10); + sim.originate(origin, id); sim.reconnect(origin); sim.announce(origin, "live/x", 0, None); assert!(sim.run()); for node in [a, b] { - if fresh { - assert_eq!(sim.shows(node, 8, "live/x"), Some(true)); - assert_eq!(sim.shown(node).len(), 1); - } else if sim.paths() { - assert_eq!(sim.shows(node, 7, "live/x"), Some(true)); - assert_eq!(sim.shown(node).len(), 1); - } else { - assert_eq!(sim.shows(node, 7, "live/y"), Some(true), "the dead process's claim"); - assert_eq!(sim.shows(node, 7, "live/x"), None); - } + assert_eq!(sim.shows(node, id, "live/x"), Some(true)); + assert_eq!(sim.shown(node).len(), 1); } } } /// Node b learned that `live/x` ended, then lost every route and collected the /// origin. A session still delivering older messages brings the claim back, -/// but only until the withdrawal queued behind it lands. Without collection, -/// b's retained floor refuses the replay outright. The path rule forgets the view -/// with the last route, so it replays either way. +/// but only until the withdrawal queued behind it lands. The view goes with the +/// last route, so it replays whether or not b collected the origin. fn stale_replay_after_collection_lasts_only_until_the_withdrawal_behind_it(rule: Rule) { let (origin, a, b, c) = (0, 1, 2, 3); for collect in [false, true] { @@ -831,7 +775,7 @@ fn stale_replay_after_collection_lasts_only_until_the_withdrawal_behind_it(rule: sim.join(a, c, 1); sim.join(c, b, 1); sim.hold(a, b); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/x", 0, None); assert!(sim.run()); sim.unannounce(origin, "live/x"); @@ -844,29 +788,23 @@ fn stale_replay_after_collection_lasts_only_until_the_withdrawal_behind_it(rule: if collect { sim.collect(b, 7); } - // The held ROUTE, then the snapshot carrying `live/x`: under the path rule after - // a's own ask, the marker answering b's, and SYNCED, which open its stream. - for _ in 0..2 + 3 * usize::from(sim.paths()) { + // a's own ask, the marker answering b's, and SYNCED, which open its stream, then + // the held ROUTE and the snapshot carrying `live/x`. + for _ in 0..5 { assert!(sim.deliver(a, b)); } - let forgot = collect || sim.paths(); - assert_eq!(sim.shows(b, 7, "live/x").is_some(), forgot); + assert!(sim.shows(b, 7, "live/x").is_some()); assert!(sim.deliver(a, b), "the withdrawal"); assert_eq!(sim.shows(b, 7, "live/x"), None); sim.release(a, b); assert!(sim.run()); - let replayed: &[Kind] = if forgot { - &[Kind::Start, Kind::End, Kind::Start, Kind::End] - } else { - &[Kind::Start, Kind::End] - }; - assert_eq!(kinds(&at(&sim, b)), replayed); + assert_eq!(kinds(&at(&sim, b)), [Kind::Start, Kind::End, Kind::Start, Kind::End]); assert!(!sim.holds(b, 7)); } } -/// Holds follow feasible advertisements only, so they cannot prop each other -/// up. When the origin dies, every node in a ring goes offline and keeps no +/// Holds follow only the path they were built on, so they cannot prop each +/// other up. When the origin dies, every node in a ring goes offline and keeps no /// route state, with no timer, and none reports an END the origin did not send. fn holds_retire_once_nothing_can_recover(rule: Rule) { let origin = 4; @@ -876,7 +814,7 @@ fn holds_retire_once_nothing_can_recover(rule: Rule) { } sim.leaf(origin); sim.join(origin, 0, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/x", 0, None); assert!(sim.run()); @@ -885,19 +823,14 @@ fn holds_retire_once_nothing_can_recover(rule: Rule) { for relay in 0..4 { assert!(sim.shown(relay).is_empty()); assert!(!sim.holds(relay, 7), "relay {relay}"); - assert!( - sim.paths() || !sim.remembers(relay, 7), - "relay {relay} drained its distance" - ); } assert_eq!(false_ends(&sim.events), 0); assert_eq!(sim.pending(), 0); } -/// Seqnos and revisions are varints. An origin never advertises the last -/// value: a bump, a claim change, or a request that would reach it rotates the -/// identity instead, so every node lets the old one go and sees the claims -/// again under the new one. Nothing wraps. +/// Revisions are varints. An origin never advertises the last value: a claim +/// change that would reach it rotates the identity instead, so every node lets the +/// old one go and sees the claims again under the new one. Nothing wraps. fn exhausted_counters_rotate_the_identity(rule: Rule) { let (origin, a, b, c, client) = (0, 1, 2, 3, 4); let rotated = |sim: &Sim| { @@ -909,7 +842,7 @@ fn exhausted_counters_rotate_the_identity(rule: Rule) { assert!(!sim.routed(a, 7)); sim.check().unwrap(); }; - let build = |seqno| { + let build = || { let mut sim = Sim::with(rule, 5); sim.leaf(origin); sim.leaf(client); @@ -919,24 +852,13 @@ fn exhausted_counters_rotate_the_identity(rule: Rule) { sim.join(origin, c, 1); sim.join(client, a, 1); sim.disconnect(origin, c); - sim.originate(origin, 7, seqno); + sim.originate(origin, 7); sim.announce(origin, "live/x", 0, None); assert!(sim.run()); sim }; - // The path rule has no seqno. - if rule == Rule::Distance { - let mut sim = build(MAX - 2); - sim.bump(origin); - assert!(sim.run()); - assert_eq!(sim.proven(origin), 7); - sim.bump(origin); - assert!(sim.run()); - rotated(&sim); - } - - let mut sim = build(0); + let mut sim = build(); sim.age(origin, MAX - 2); sim.announce(origin, "live/y", 0, None); assert!(sim.run()); @@ -944,28 +866,13 @@ fn exhausted_counters_rotate_the_identity(rule: Rule) { sim.unannounce(origin, "live/y"); assert!(sim.run()); rotated(&sim); - if sim.paths() { - return; - } - - // The origin moves to a farther attachment, and the starved node asks for - // the last seqno. - let mut sim = build(MAX - 1); - sim.disconnect(origin, a); - sim.connect(origin, c); - assert!(sim.run()); - rotated(&sim); - assert!(sim.routed(a, sim.proven(origin))); } -/// Node b forwards a request along its route through x, whose own route just -/// failed but whose retraction to b is still in flight, so x drops it. When -/// the retraction lands, b moves to an equal-metric route through y and -/// advertises nothing new. Keeping the request at every hop that forwarded -/// it, and resending it when that hop's next hop changes, reaches the origin -/// with no timer. The path rule needs no request: n's route through b never -/// named z, the far end of the session it lost. -fn a_quiet_origin_is_reached_by_a_kept_request(rule: Rule) { +/// Node b's route through x just failed, but its retraction to b is still in +/// flight. When it lands, b moves to an equal-metric route through y and n +/// reaches the origin with no request: n's route through b never named z, the +/// far end of the session it lost. +fn a_quiet_origin_is_reached_once_a_withdrawal_lands(rule: Rule) { let (origin, w, x, y2, y, b, n, z, client) = (0, 1, 2, 3, 4, 5, 6, 7, 8); let mut sim = Sim::with(rule, 9); sim.leaf(origin); @@ -980,7 +887,7 @@ fn a_quiet_origin_is_reached_by_a_kept_request(rule: Rule) { sim.join(n, z, 1); sim.join(z, origin, 1); sim.join(n, client, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/x", 0, None); assert!(sim.run()); assert_eq!(sim.next_hop(n, 7), Some(z)); @@ -990,19 +897,17 @@ fn a_quiet_origin_is_reached_by_a_kept_request(rule: Rule) { sim.hold(x, b); sim.disconnect(w, x); assert!(sim.run()); - assert!(sim.paths() || sim.starved(n, 7)); sim.release(x, b); assert!(sim.run()); assert_eq!(sim.next_hop(b, 7), Some(y)); assert_eq!(sim.shows(client, 7, "live/x"), Some(true)); - assert!(sim.paths() || sim.seqno(origin) == 1); assert!(sim.routed(n, 7)); } /// Random meshes churn sessions and hold and deliver messages out of step, /// then release every hold. Every relay that can still reach a quiet origin -/// ends with a route to it, which only a seqno request can bring. -fn sweep_requests_reach_a_quiet_origin(rule: Rule) { +/// ends with a route to it. +fn sweep_relays_reach_a_quiet_origin(rule: Rule) { const RELAYS: usize = 5; let mut stranded = 0; for seed in 0..300u64 { @@ -1027,7 +932,7 @@ fn sweep_requests_reach_a_quiet_origin(rule: Rule) { links.push((relay, origin)); } } - sim.originate(origin, origin as u64, 0); + sim.originate(origin, origin as u64); // A route goes out only while a claim meets the interest. sim.announce(origin, "", 0, None); } @@ -1110,16 +1015,15 @@ fn announcement_cost_follows_sessions_and_live_claims(rule: Rule) { sim.leaf(client); sim.join(client, relays / 2, 1); sim.watch(client, "none/"); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); for prefix in &PREFIXES[..claims] { sim.announce(origin, prefix, 0, None); } assert!(sim.run()); - // Each relay sends to both ring neighbors, and the origin and its relay to - // each other: without split horizon the relay echoes the origin's claims. - // The path rule sends no route to its own publisher. - let sessions = 2 * relays + 2 - usize::from(sim.paths()); + // Each relay sends to both ring neighbors, and the origin to its relay: no + // route goes back to its own publisher. + let sessions = 2 * relays + 1; let retained = sim.state(); assert_eq!(retained.items, relays * claims); // The client's interest meets no claim, so it holds no route and no floor. @@ -1143,7 +1047,7 @@ fn announcement_cost_follows_sessions_and_live_claims(rule: Rule) { let after = sim.state(); assert_eq!(after.items, relays * (claims - 1)); assert_eq!(after.floors, retained.floors, "no tombstones"); - assert_eq!(after.pending, 0, "every ROUTE_END echo drained"); + assert_eq!(after.pending, 0, "every dead hop and doubt collected"); sim.check().unwrap(); // The client cycles through disjoint regions, one with a live claim. @@ -1174,7 +1078,6 @@ fn announcement_cost_follows_sessions_and_live_claims(rule: Rule) { /// the path rule the client's interest is a set, edited in place. fn sweep_scoped_handoff(rule: Rule) { const CLAIMS: [&str; 5] = ["live/a/x", "live/a/y/", "live/b/z", "live/c", "live/"]; - const WATCHES: [&[&str]; 5] = [&[""], &["live/"], &["live/a/"], &["live/b/z"], &["live/c"]]; const SETS: [&[&str]; 8] = [ &[""], &["live/"], @@ -1185,7 +1088,7 @@ fn sweep_scoped_handoff(rule: Rule) { &["live/a/x", "live/a/y", "live/c"], &["live/a/", "live/b/", "live/c"], ]; - let watches: &[&[&str]] = if rule == Rule::Distance { &WATCHES } else { &SETS }; + let watches: &[&[&str]] = &SETS; const SCOPES: [&str; 2] = ["live/a/", "live/b/"]; let (origin, r1, r2, client) = (0, 1, 2, 3); let mut rng = XorShift(0x5c0b_ed00); @@ -1205,7 +1108,7 @@ fn sweep_scoped_handoff(rule: Rule) { sim.link(client, relay, SimLink::cost(1).grant(grant), SimLink::cost(1)); } let links = [(origin, r1), (origin, r2), (r1, r2), (client, r1), (client, r2)]; - sim.originate(origin, 7, 0); + sim.originate(origin, 7); assert!(sim.run()); for _ in 0..40 { match rng.next(8) { @@ -1245,10 +1148,6 @@ fn sweep_scoped_handoff(rule: Rule) { } sim.settle(1); sim.check().unwrap_or_else(|err| panic!("scenario {scenario}: {err}")); - assert!( - sim.paths() || sim.retained().certified <= SCOPES.len() + 1, - "scenario {scenario}: the widening guard grows" - ); } for (a, b) in links { sim.release(a, b); @@ -1298,7 +1197,7 @@ const CLAIMS: [&str; 4] = ["live/", "live/a/", "live/a/x", "live/b"]; /// shows every live origin's current claims and holds nothing for a dead one. fn sweep_handoff(rule: Rule) { for scenario in 0..scenarios() { - let mut sim = churn(rule, scenario, rule != Rule::Distance); + let mut sim = churn(rule, scenario, true); let label = format!("scenario {scenario}"); assert_eq!(revivals(&sim.events), 0, "{label}"); assert_eq!(false_ends(&sim.events), 0, "{label}"); @@ -1306,18 +1205,6 @@ fn sweep_handoff(rule: Rule) { } } -/// Scenario 16108 of the distance rule's [`sweep_handoff`], past what a gate run -/// reaches, pinned so every run has it: a batch that leaves its neighbor's floors -/// alone still moves the merged view's, which changes how claims it did not name -/// are presented, so every claim is checked again. -#[test] -fn a_batch_moving_the_merged_floors_rechecks_every_claim() { - let mut sim = churn(Rule::Distance, 16108, false); - assert_eq!(revivals(&sim.events), 0); - assert_eq!(false_ends(&sim.events), 0); - settled(&mut sim, "pinned churn"); -} - /// A neighbour's unchanged claims at a newer floor must displace what another /// path taught us in between, even when that other path now comes through us. #[test] @@ -1327,7 +1214,7 @@ fn an_unchanged_snapshot_advances_the_floor_others_need() { sim.join(origin, primary, 1); sim.join(primary, other, 1); sim.join(origin, other, 3); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/a", 0, None); assert!(sim.run()); @@ -1349,30 +1236,10 @@ fn an_unchanged_snapshot_advances_the_floor_others_need() { assert_eq!(sim.reaches(other, 7, &p("live/a")), Some(origin)); } -/// Both rules run the same external script and settle to the same claims. Their -/// different message graphs interleave snapshots differently, so reader histories -/// need not match, but neither may revive a known withdrawal or report a false END. -#[test] -fn the_rules_agree_on_every_script() { - for scenario in 0..scenarios() { - let label = format!("scenario {scenario}"); - let sims = [Rule::Distance, Rule::PATH].map(|rule| { - let mut sim = churn(rule, scenario, false); - assert_eq!(revivals(&sim.events), 0, "{label}: {rule:?}"); - assert_eq!(false_ends(&sim.events), 0, "{label}: {rule:?}"); - settled(&mut sim, &label); - sim - }); - for node in 0..RELAYS + 3 { - assert_eq!(sims[0].shown(node), sims[1].shown(node), "{label}: node {node}"); - } - } -} - /// Origins sit at `RELAYS` and `RELAYS + 1`, the client at `RELAYS + 2`. The script, -/// every change made from outside, depends on `scenario` alone, so both rules run -/// the same one; which queued message arrives next is drawn apart from it. With -/// `sets`, which only the path rule takes, the client watches sets of prefixes. +/// every change made from outside, depends on `scenario` alone; which queued message +/// arrives next is drawn apart from it. With `sets`, the client watches sets of +/// prefixes. fn churn(rule: Rule, scenario: usize, sets: bool) -> Sim { const SETS: [&[&str]; 6] = [ &[""], @@ -1405,8 +1272,8 @@ fn churn(rule: Rule, scenario: usize, sets: bool) -> Sim { sim.join(a, b, 1 + rng.next(3)); } let mut ids = [100, 200]; - sim.originate(first, ids[0], 0); - sim.originate(first + 1, ids[1], 0); + sim.originate(first, ids[0]); + sim.originate(first + 1, ids[1]); assert!(sim.run()); for _ in 0..40 { @@ -1452,22 +1319,17 @@ fn churn(rule: Rule, scenario: usize, sets: bool) -> Sim { let origin = rng.next(2) as usize; sim.kill(first + origin); ids[origin] += 1; - sim.originate(first + origin, ids[origin], 0); + sim.originate(first + origin, ids[origin]); sim.reconnect(first + origin); } _ => { - // Echoed ROUTE_ENDs interleave freely with the echoing node's routes. for _ in 0..rng.next(6) { - let busy: Vec<(bool, usize, usize)> = (sim.busy().into_iter().map(|(from, to)| (false, from, to))) - .chain(sim.busy_acks().into_iter().map(|(from, to)| (true, from, to))) - .collect(); + let busy = sim.busy(); if busy.is_empty() { break; } - match busy[pick.next(busy.len() as u64) as usize] { - (false, from, to) => sim.deliver(from, to), - (true, from, to) => sim.deliver_ack(from, to), - }; + let (from, to) = busy[pick.next(busy.len() as u64) as usize]; + sim.deliver(from, to); } } } @@ -1515,11 +1377,7 @@ fn settled(sim: &mut Sim, label: &str) { shown, expected.keys().collect(), "{label}: node {node}\n{}", - if sim.paths() { - sim.path(node).dump() - } else { - String::new() - } + sim.path(node).dump() ); for ((id, claim), shown) in sim.shown(node) { assert!(shown.available, "{label}: node {node} {id} {claim}"); @@ -1531,10 +1389,6 @@ fn settled(sim: &mut Sim, label: &str) { } for id in (100..110).chain(200..210).filter(|id| !live.contains(id)) { assert!(!sim.holds(node, id), "{label}: node {node} holds dead {id}"); - assert!( - sim.paths() || !sim.remembers(node, id), - "{label}: node {node} keeps dead {id}'s distance" - ); } } assert_eq!(sim.pending(), 0, "{label}"); @@ -1555,7 +1409,7 @@ fn a_link_flap_costs_routes_not_broadcasts(rule: Rule) { sim.join(a, c, 1); sim.join(b, c, 1); sim.join(client, c, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); for claim in 0..CLAIMS { sim.announce(origin, &format!("live/{claim}"), 0, None); } @@ -1598,7 +1452,7 @@ fn a_restarted_route_sends_a_complete_snapshot(rule: Rule) { sim.leaf(origin); sim.join(origin, relay, 1); sim.join(relay, node, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/x", 0, None); // Keeps the route up once `live/x` goes. sim.announce(origin, "live/y", 0, None); @@ -1632,7 +1486,7 @@ fn a_covering_claim_moves_with_a_disjoint_interest(rule: Rule) { sim.join(origin, relay, 1); sim.join(relay, client, 1); sim.watch(client, "live/a"); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live", 0, None); assert!(sim.run()); assert_eq!(sim.shows(client, 7, "live/a"), Some(true)); @@ -1677,7 +1531,6 @@ fn a_restarted_floorless_route_drops_what_it_no_longer_carries(rule: Rule) { assert!(sim.run()); sim.unannounce(origin, "live/a/x"); assert!(sim.run()); - while sim.deliver_ack(z, x) {} sim.connect(x, local); assert!(sim.run()); sim.release(z, x); @@ -1689,9 +1542,9 @@ fn a_restarted_floorless_route_drops_what_it_no_longer_carries(rule: Rule) { } /// Relay 4 hears origin 7's claims from relay 0, which certifies them all, and from -/// relay 3, which asked only for `b`. While 4's own distance sets 0 aside, 0 moves -/// between two equal routes and says so. That change of hops moves no certifier, so -/// 4 lets go of nothing, where pruning on it lost the root claim for good. +/// relay 3, which asked only for `b`. A path names the session too, so the route +/// through 3 ends when it closes rather than changing its hops alone, and 4 lets go +/// of nothing, where pruning on a change of hops lost the root claim for good. fn a_change_of_hops_alone_lets_go_of_no_claim(rule: Rule) { let (origin, other) = (6, 7); let mut sim = Sim::with(rule, 8); @@ -1707,22 +1560,16 @@ fn a_change_of_hops_alone_lets_go_of_no_claim(rule: Rule) { sim.join(2, 3, 1); sim.join(2, 5, 1); sim.link(3, 4, SimLink::cost(1), SimLink::cost(2)); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "", 0, None); - sim.originate(other, 8, 0); + sim.originate(other, 8); sim.announce(other, "", 0, None); sim.watch(3, "b"); assert!(sim.run()); assert_eq!(sim.shows(4, 7, ""), Some(true)); - let hops = sim.stats.hop_updates; sim.disconnect(origin, 3); assert!(sim.run()); - // The path rule names the session too, so the route through 3 ends instead. - assert!( - sim.paths() || sim.stats.hop_updates > hops, - "a route changed its hops alone" - ); assert_eq!(sim.shows(4, 7, ""), Some(true)); sim.disconnect(2, 3); assert!(sim.run()); @@ -1736,7 +1583,7 @@ fn triangle(rule: Rule, ab: SimLink) -> Sim { sim.join(origin, a, 1); sim.join(origin, b, 1); sim.link(a, b, ab, ab); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "x", 0, None); assert!(sim.run()); sim @@ -1768,8 +1615,7 @@ fn mutually_preferred_links_settle(rule: Rule) { } /// A relay leaves a draining neighbour for a costlier path that does not drain, -/// and that path, not draining itself, is what it offers on. The distance rule -/// keeps the edge on the draining neighbour until a seqno refresh instead. +/// and that path, not draining itself, is what it offers on. #[test] fn goaway_moves_a_relay_to_a_path_that_does_not_drain() { let (origin, near, far, relay, edge) = (0, 1, 2, 3, 4); @@ -1780,7 +1626,7 @@ fn goaway_moves_a_relay_to_a_path_that_does_not_drain() { sim.join(relay, far, 2); sim.join(edge, relay, 1); sim.join(edge, near, 3); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "x", 0, None); assert!(sim.run()); assert_eq!(sim.toward(relay, 7), Some(Some(near))); @@ -1811,7 +1657,7 @@ fn goaway_around_a_ring(sim: &mut Sim, hub: Node, leaving: Node) { for (i, &relay) in ring.iter().enumerate() { sim.link(relay, ring[(i + 1) % 3], SimLink::cost(1), SimLink::cost(2)); } - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "x", 0, None); assert!(sim.run()); sim.goaway(leaving); diff --git a/rs/moq-net/src/route/tests/interest.rs b/rs/moq-net/src/route/tests/interest.rs index 5c8d384c53d..0899e96f0de 100644 --- a/rs/moq-net/src/route/tests/interest.rs +++ b/rs/moq-net/src/route/tests/interest.rs @@ -16,7 +16,7 @@ fn chain(watch: &[&str], per: usize) -> Sim { sim.join(origin, relay, 1); sim.join(relay, client, 1); sim.watch_all(client, watch); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); for room in ROOMS { for i in 0..per { sim.announce(origin, &format!("{room}/{i}"), 0, None); @@ -152,7 +152,7 @@ fn an_edit_crossing_a_dead_hop_and_a_hold_settles() { sim.join(r2, client, 1); sim.join(r1, r2, 1); sim.watch_all(client, &["live/a"]); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/a/x", 0, None); sim.announce(origin, "live/b/y", 0, None); assert!(sim.run()); @@ -204,7 +204,7 @@ fn narrowing_keeps_the_route_the_sender_would_reflect() { sim.join(origin, client, 1); sim.join(r1, client, 1); sim.watch_all(client, &["live/a"]); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "live/a/x", 0, None); assert!(sim.run()); assert_eq!(sim.next_hop(client, 7), Some(origin)); diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index 294ce834909..f52b824c8ea 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -1,6 +1,5 @@ -//! The routing contract's fixtures, run in a simulated network against either -//! rule: [`Router`], which sessions run today, or [`path::Router`], the rule the -//! routes epic's Plan item 27 picked. +//! The routing contract's fixtures, run in a simulated network against +//! [`path::Router`], the rule the routes epic's Plan item 27 picked. //! //! [`Sim`] wires one router per node through ordered, holdable message queues, //! so a fixture can deliver messages out of step and replay them late. Each round @@ -35,12 +34,9 @@ pub(super) type Node = usize; /// How long one round of deliveries takes on the mocked clock. pub(super) const HOP: Duration = Duration::from_millis(25); -/// Which route rule every node of a [`Sim`] runs. +/// Which route rule every node of a [`Sim`] runs: [`path::Router`], with its doubt. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(super) enum Rule { - /// [`Router`]: Babel's distance and seqno, which sessions run today. - Distance, - /// [`path::Router`], with its doubt. Path(Doubt), } @@ -55,7 +51,6 @@ impl Rule { pub fn as_str(self) -> &'static str { match self { - Self::Distance => "distance", Self::Path(Doubt { lift: Lift::Unnamed, .. }) => "path+early", @@ -64,115 +59,22 @@ impl Rule { } } -/// Run each fixture named once per rule, as a test per rule: the rule sessions run -/// today, and the one that replaces it. -macro_rules! both { +/// Run each fixture named under the path rule, as a test each. +macro_rules! fixtures { ($($name:ident),* $(,)?) => { - mod distance { - $(#[test] fn $name() { super::$name(super::Rule::Distance) })* - } mod path { $(#[test] fn $name() { super::$name(super::Rule::PATH) })* } }; } -pub(super) use both; - -/// One node's router, under either rule. -#[derive(Clone)] -enum Relay { - Distance(Router), - Path(path::Router), -} - -impl Relay { - fn new(rule: Rule, hop: Id, seed: u64) -> Self { - match rule { - Rule::Distance => Self::Distance(Router::new(hop, seed)), - Rule::Path(doubt) => Self::Path(path::Router::new(hop, seed, doubt)), - } - } - - /// The distance rule's interest generation. - fn interest(&self) -> u32 { - match self { - Self::Distance(router) => router.interest(), - Self::Path(_) => unreachable!("the path rule edits its interest in place"), - } - } +pub(super) use fixtures; - fn declared(&self) -> Option { - match self { - Self::Distance(router) => router.declared(), - Self::Path(router) => router.declared(), - } - } - - fn revision(&self) -> Option { - match self { - Self::Distance(router) => router.revision(), - Self::Path(router) => router.revision(), - } - } - - /// The declared identity's own claims. - fn own(&self, id: Id) -> Option<&View> { - match self { - Self::Distance(router) => router.view(id), - Self::Path(router) => router.view(id, &Path::empty()), - } - } +/// A node's router. +type Relay = path::Router; - fn events(&mut self) -> Vec> { - // The simulator reads full views directly. Consume change notifications - // alongside events, as a driver does, so they retain no script history. - match self { - Self::Distance(router) => { - drop(router.changed()); - router.events() - } - Self::Path(router) => { - drop(router.changed()); - router.events() - } - } - } - - fn holds(&self, id: Id) -> bool { - match self { - Self::Distance(router) => router.holds(id), - Self::Path(router) => router.holds(id), - } - } - - fn shown(&self) -> &BTreeMap<(Id, Path), Shown> { - match self { - Self::Distance(router) => router.shown(), - Self::Path(router) => router.shown(), - } - } - - fn touched(&self) -> usize { - match self { - Self::Distance(router) => router.touched(), - Self::Path(router) => router.touched(), - } - } - - fn deadline(&self) -> Option { - match self { - Self::Distance(_) => None, - Self::Path(router) => router.deadline(), - } - } - - /// Every view of another publisher's claims, by the identity it files them under. - fn views(&self) -> Vec<(Id, &View)> { - match self { - Self::Distance(router) => router.views().collect(), - Self::Path(router) => router.views().map(|((id, _), view)| (*id, view)).collect(), - } - } +fn relay(rule: Rule, hop: Id, seed: u64) -> Relay { + let Rule::Path(doubt) = rule; + path::Router::new(hop, seed, doubt) } /// A claim's attributes, plus the revision its life began at so fixtures can @@ -281,28 +183,13 @@ impl SimLink { #[derive(Clone, Debug)] enum Msg { - /// A [`Router`] route. - Route { - id: Id, - reach: Option, - }, - Batch { - id: Id, - changes: Vec<(Path, Option)>, - floor: Option, - interest: u32, - }, - Request { - id: Id, - seqno: Seqno, - }, - /// A [`path::Router`] route. + /// A route. Path { id: Id, prefix: Path, reach: Option, }, - /// A [`path::Router`] route's claims. + /// A route's claims. Claims { id: Id, prefix: Path, @@ -332,7 +219,7 @@ pub(super) struct SimEvent { pub withdrawn: bool, } -/// What either rule retains, summed over every node. +/// What the routers retain, summed over every node. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] pub(super) struct State { /// Claims in views of other publishers. @@ -343,14 +230,14 @@ pub(super) struct State { pub heard: usize, /// Claims on routes sent. pub sent: usize, - /// What waits on another node: requests and echoes, or dead hops and doubts. + /// What waits on another node: dead hops and doubts. pub pending: usize, } /// Messages sent, as protocol counts. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] pub(super) struct Stats { - /// Route messages: ROUTE, DEAD, and echoed ROUTE_ENDs. + /// Route messages: ROUTE and DEAD. pub route_msgs: usize, /// Routes those messages name: a DEAD counts itself and each route it moved. pub route_items: usize, @@ -358,10 +245,7 @@ pub(super) struct Stats { pub hop_updates: usize, pub claim_msgs: usize, pub claim_items: usize, - pub requests: usize, - /// Requests a node sent the same neighbor again for the same seqno. - pub repeated_requests: usize, - /// Messages and echoes delivered. + /// Messages delivered. pub delivered: usize, } @@ -386,15 +270,10 @@ pub(super) struct Sim { grants: BTreeSet<&'static str>, up: BTreeSet<(Node, Node)>, queues: BTreeMap<(Node, Node), VecDeque>, - /// Echoed ROUTE_ENDs: unordered against the echoing node's own routes. - acks: BTreeMap<(Node, Node), VecDeque>, held: BTreeSet<(Node, Node)>, history: BTreeMap>, - /// Every request sent, to spot a repeat. - requested: BTreeSet<(Node, Node, Id, Seqno)>, - /// The last finite route each node sent each neighbor, to spot a change of hops alone. - routed: BTreeMap<(Node, Node, Id), Advert>, - /// The same for [`path::Router`], by prefix. + /// The last finite route each node sent each neighbor, by prefix, to spot a change + /// of hops alone. offered: BTreeMap<(Node, Node, Id, Path), path::Offer>, pub stats: Stats, /// The work of routers a restart replaced. @@ -403,9 +282,9 @@ pub(super) struct Sim { } impl Sim { - /// `nodes` nodes running the rule sessions run today. + /// `nodes` nodes running the path rule. pub fn new(nodes: usize) -> Self { - Self::with(Rule::Distance, nodes) + Self::with(Rule::PATH, nodes) } /// `nodes` nodes running `rule`. @@ -413,7 +292,7 @@ impl Sim { Self { rule, nodes: (0..nodes) - .map(|node| Relay::new(rule, hop(node, 0), node as u64 + 1)) + .map(|node| relay(rule, hop(node, 0), node as u64 + 1)) .collect(), now: Instant::ZERO, leaf: vec![false; nodes], @@ -424,11 +303,8 @@ impl Sim { grants: BTreeSet::new(), up: BTreeSet::new(), queues: BTreeMap::new(), - acks: BTreeMap::new(), held: BTreeSet::new(), history: BTreeMap::new(), - requested: BTreeSet::new(), - routed: BTreeMap::new(), offered: BTreeMap::new(), stats: Stats::default(), spent: 0, @@ -436,36 +312,9 @@ impl Sim { } } - pub fn rule(&self) -> Rule { - self.rule - } - - /// Whether the nodes run [`path::Router`]. - pub fn paths(&self) -> bool { - matches!(self.rule, Rule::Path(_)) - } - - /// `node`'s [`Router`]. - pub fn babel(&self, node: Node) -> &Router { - match &self.nodes[node] { - Relay::Distance(router) => router, - Relay::Path(_) => panic!("node {node} runs the path rule"), - } - } - - fn babel_mut(&mut self, node: Node) -> &mut Router { - match &mut self.nodes[node] { - Relay::Distance(router) => router, - Relay::Path(_) => panic!("node {node} runs the path rule"), - } - } - - /// `node`'s [`path::Router`]. + /// `node`'s router. pub fn path(&self, node: Node) -> &path::Router { - match &self.nodes[node] { - Relay::Path(router) => router, - Relay::Distance(_) => panic!("node {node} runs the distance rule"), - } + &self.nodes[node] } /// A node that sends only identities it serves, like a browser. @@ -503,38 +352,23 @@ impl Sim { regions: vec![p(grant.scope)], qualify: Arc::new(move |id| qualified(grant.subject, id)), }), + // One session to a peer at a time. + ..Link::default() } } - /// Open a session: each side asks for its interest, the path rule by an edit it - /// sends itself. + /// Open a session: each side asks for its interest by an edit it sends itself. pub fn connect(&mut self, a: Node, b: Node) { assert!(!self.connected(a, b), "{a}-{b} already up"); self.up.insert(Self::pair(a, b)); for (x, y) in [(a, b), (b, a)] { let link = self.config(x, y); let named = hop(y, self.restarts[y]); - match &mut self.nodes[x] { - Relay::Distance(router) => router.connect(y, link), - Relay::Path(router) => router.connect(y, named, link), - } - .expect("a positive cost"); - } - if !self.paths() { - for (x, y) in [(a, b), (b, a)] { - self.ask(y, x); - } + self.nodes[x].connect(y, named, link).expect("a positive cost"); } self.flush(); } - /// Under the distance rule, `node` asks `peer` for announcements under its - /// interest on a new stream. - fn ask(&mut self, node: Node, peer: Node) { - let watch = self.watching(node).clone(); - self.babel_mut(peer).peer_interest(node, watch).unwrap(); - } - pub fn connected(&self, a: Node, b: Node) -> bool { self.up.contains(&Self::pair(a, b)) } @@ -544,12 +378,8 @@ impl Sim { self.up.remove(&Self::pair(a, b)); for (x, y) in [(a, b), (b, a)] { self.queues.remove(&(x, y)); - self.acks.remove(&(x, y)); self.held.remove(&(x, y)); - match &mut self.nodes[y] { - Relay::Distance(router) => router.disconnect([x]), - Relay::Path(router) => router.disconnect([x], self.now), - } + self.nodes[y].disconnect([x], self.now); } self.flush(); } @@ -561,18 +391,11 @@ impl Sim { self.up.remove(&Self::pair(node, peer)); for key in [(node, peer), (peer, node)] { self.queues.remove(&key); - self.acks.remove(&key); self.held.remove(&key); } - match &mut self.nodes[peer] { - Relay::Distance(router) => router.disconnect([node]), - Relay::Path(router) => router.disconnect([node], self.now), - } - } - match &mut self.nodes[node] { - Relay::Distance(router) => router.disconnect(peers), - Relay::Path(router) => router.disconnect(peers, self.now), + self.nodes[peer].disconnect([node], self.now); } + self.nodes[node].disconnect(peers, self.now); self.flush(); } @@ -586,17 +409,10 @@ impl Sim { self.declared[node] = None; let restarts = self.restarts[node]; let seed = fnv("restart", &[node as u64, u64::from(restarts)]); - if let Relay::Path(router) = &self.nodes[node] { - self.spent += router.work(); - } - self.nodes[node] = Relay::new(self.rule, hop(node, restarts), seed); + self.spent += self.nodes[node].work(); + self.nodes[node] = relay(self.rule, hop(node, restarts), seed); let watch = self.watch[node].clone(); - match &mut self.nodes[node] { - Relay::Distance(router) => { - router.watch(watch.into_iter().next().expect("one prefix")); - } - Relay::Path(router) => router.watch(watch).expect("no prefix inside another"), - }; + self.nodes[node].watch(watch).expect("no prefix inside another"); self.flush(); } @@ -614,38 +430,16 @@ impl Sim { (0..self.nodes.len()).filter(move |&peer| self.links[node][peer].is_some() && self.connected(node, peer)) } - /// `node` starts originating `id`, at `seqno` under the distance rule. - pub fn originate(&mut self, node: Node, id: Id, seqno: Seqno) { - match &mut self.nodes[node] { - Relay::Distance(router) => router.originate(id, seqno), - Relay::Path(router) => router.originate(id), - } + /// `node` starts originating `id`. + pub fn originate(&mut self, node: Node, id: Id) { + self.nodes[node].originate(id); self.declared[node] = Some(id); self.history.entry(id).or_default(); self.flush(); } - /// `node` advertises `id` as if serving it, claiming the root. - pub fn forge(&mut self, node: Node, id: Id, seqno: Seqno) { - let item = Item { - seed: 0, - epoch: None, - since: 0, - }; - self.babel_mut(node).forge(id, seqno, Path::empty(), item); - self.flush(); - } - - pub fn bump(&mut self, node: Node) { - self.babel_mut(node).bump().unwrap(); - self.flush(); - } - pub fn age(&mut self, node: Node, rev: Rev) { - match &mut self.nodes[node] { - Relay::Distance(router) => router.age(rev), - Relay::Path(router) => router.age(rev), - } + self.nodes[node].age(rev); self.flush(); } @@ -654,35 +448,22 @@ impl Sim { self.nodes[node].declared().expect("not an origin") } - pub fn seqno(&self, node: Node) -> Seqno { - self.babel(node).seqno().expect("not an origin") - } - pub fn announce(&mut self, node: Node, claim: &str, seed: Metric, epoch: Option) { let claim = p(claim); let router = &self.nodes[node]; let id = router.declared().expect("not an origin"); let rev = router.revision().unwrap() + 1; - let since = router - .own(id) + let since = (router.view(id, &Path::empty())) .and_then(|view| view.items.get(&claim)) .map_or(rev, |item| item.since); let item = Item { seed, epoch, since }; - match &mut self.nodes[node] { - Relay::Distance(router) => router.announce(claim.clone(), item), - Relay::Path(router) => router.announce(claim.clone(), item), - } - .unwrap(); + self.nodes[node].announce(claim.clone(), item).unwrap(); self.record(node, claim, Some(item)); } pub fn unannounce(&mut self, node: Node, claim: &str) { let claim = p(claim); - match &mut self.nodes[node] { - Relay::Distance(router) => router.unannounce(claim.clone()), - Relay::Path(router) => router.unannounce(claim.clone()), - } - .unwrap(); + self.nodes[node].unannounce(claim.clone()).unwrap(); self.record(node, claim, None); } @@ -696,29 +477,17 @@ impl Sim { self.flush(); } - /// The prefix `node` asks its neighbors for announcements under. Under the - /// distance rule each neighbor answers on a new stream with a complete snapshot; - /// under the path rule the node edits what it asks each neighbour for. + /// The prefix `node` asks its neighbors for announcements under: the node edits + /// what it asks each neighbour for. pub fn watch(&mut self, node: Node, prefix: &str) { self.watch_all(node, &[prefix]); } - /// `node`'s interest becomes `prefixes`, none inside another. The distance rule - /// takes one prefix. + /// `node`'s interest becomes `prefixes`, none inside another. pub fn watch_all(&mut self, node: Node, prefixes: &[&str]) { let set: BTreeSet = prefixes.iter().map(|prefix| p(prefix)).collect(); self.watch[node] = set.clone(); - match &mut self.nodes[node] { - Relay::Distance(router) => { - assert_eq!(set.len(), 1, "the distance rule watches one prefix"); - router.watch(set.into_iter().next().expect("one prefix")); - let peers: Vec = self.peers(node).collect(); - for peer in peers { - self.ask(node, peer); - } - } - Relay::Path(router) => router.watch(set).expect("no prefix inside another"), - } + self.nodes[node].watch(set).expect("no prefix inside another"); self.flush(); } @@ -739,40 +508,34 @@ impl Sim { let now = self.now; let peers: Vec = self.peers(node).collect(); for peer in peers { - match &mut self.nodes[peer] { - Relay::Distance(router) => router.goaway(node), - Relay::Path(router) => router.goaway(node, now), - } + self.nodes[peer].goaway(node, now); } self.flush(); } pub fn collect(&mut self, node: Node, id: Id) { - match &mut self.nodes[node] { - Relay::Distance(router) => router.collect(id), - Relay::Path(router) => router.collect(id), - } + self.nodes[node].collect(id); self.flush(); } /// `node` measures its link to `peer` at `ms`, or nothing. pub fn rtt(&mut self, node: Node, peer: Node, ms: Option) { let rtt = ms.map(Duration::from_millis); - match &mut self.nodes[node] { - Relay::Distance(router) => router.rtt(peer, rtt), - Relay::Path(router) => router.rtt(peer, rtt), - } + self.nodes[node].rtt(peer, rtt); self.flush(); } - /// What `node` last sent `peer` for `id`. - pub fn sent(&self, node: Node, peer: Node, id: Id) -> Option { - self.babel(node).sent.get(&(id, peer)).cloned() + /// The last finite route `node` sent `peer` for `id` covering `prefix`, while it + /// stands. + pub fn offer(&self, node: Node, peer: Node, id: Id, prefix: &Path) -> Option<&path::Offer> { + self.offered.get(&(node, peer, id, prefix.clone())) } - /// The streams `node` sends a route for `id` on. + /// The streams `node` sends a finite route for `id` on. pub fn sending(&self, node: Node, id: Id) -> usize { - Router::::of(&self.babel(node).sent, id).count() + (self.offered.keys()) + .filter(|(from, _, key, _)| *from == node && *key == id) + .count() } pub fn hold(&mut self, from: Node, to: Node) { @@ -789,67 +552,35 @@ impl Sim { return false; }; let now = self.now; - match (&mut self.nodes[to], msg) { - (Relay::Distance(router), Msg::Route { id, reach }) => router.route(from, id, reach), - ( - Relay::Distance(router), - Msg::Batch { - id, - changes, - floor, - interest, - }, - ) => router.batch(from, id, changes, floor, interest), - (Relay::Distance(router), Msg::Request { id, seqno }) => router.request(from, id, seqno), - (Relay::Path(router), Msg::Path { id, prefix, reach }) => router.route(from, id, prefix, reach), - ( - Relay::Path(router), - Msg::Claims { - id, - prefix, - changes, - floor, - }, - ) => router.batch(from, id, prefix, changes, floor), - (Relay::Path(router), Msg::Dead(dead)) => router.dead(from, dead, now), + let router = &mut self.nodes[to]; + router.clock(now); + match msg { + Msg::Path { id, prefix, reach } => router.route(from, id, prefix, reach), + Msg::Claims { + id, + prefix, + changes, + floor, + } => router.batch(from, id, prefix, changes, floor), + Msg::Dead(dead) => router.dead(from, dead, now), // A router sends only edits it may: a refused one would close the session. - (Relay::Path(router), Msg::Ask(edit)) => router.edit(from, edit).expect("a valid edit"), - (Relay::Path(router), Msg::Edited) => router.edited(from), - (Relay::Path(router), Msg::Synced { prefix }) => router.synced(from, prefix), - (_, msg) => panic!("node {to} cannot read {msg:?}"), + Msg::Ask(edit) => router.edit(from, edit).expect("a valid edit"), + Msg::Edited => router.edited(from), + Msg::Synced { prefix } => router.synced(from, prefix), } self.stats.delivered += 1; self.flush(); true } - /// Deliver one echoed ROUTE_END from `from` to `to`, even if held. - pub fn deliver_ack(&mut self, from: Node, to: Node) -> bool { - let Some(id) = self.acks.get_mut(&(from, to)).and_then(VecDeque::pop_front) else { - return false; - }; - self.babel_mut(to).echo(from, id); - self.stats.delivered += 1; - self.flush(); - true - } - pub fn queued(&self, from: Node, to: Node) -> usize { self.queues.get(&(from, to)).map_or(0, VecDeque::len) } - pub fn queued_acks(&self, from: Node, to: Node) -> usize { - self.acks.get(&(from, to)).map_or(0, VecDeque::len) - } - pub fn busy(&self) -> Vec<(Node, Node)> { Self::nonempty(&self.queues) } - pub fn busy_acks(&self) -> Vec<(Node, Node)> { - Self::nonempty(&self.acks) - } - fn nonempty(queues: &BTreeMap<(Node, Node), VecDeque>) -> Vec<(Node, Node)> { queues .iter() @@ -881,12 +612,7 @@ impl Sim { fn rounds(&mut self, rounds: usize, wait: bool) -> bool { for _ in 0..rounds { let ready: Vec<(Node, Node)> = self.busy().into_iter().filter(|key| !self.held.contains(key)).collect(); - let acks: Vec<(Node, Node)> = self - .busy_acks() - .into_iter() - .filter(|key| !self.held.contains(key)) - .collect(); - if ready.is_empty() && acks.is_empty() { + if ready.is_empty() { match self.deadline().filter(|_| wait) { Some(at) => { self.now = self.now.max(at); @@ -899,9 +625,6 @@ impl Sim { for (from, to) in ready { self.deliver(from, to); } - for (from, to) in acks { - self.deliver_ack(from, to); - } self.now = self.now.saturating_add(HOP); self.tick(); } @@ -914,7 +637,7 @@ impl Sim { /// The next doubt timer on any node. pub fn deadline(&self) -> Option { - self.nodes.iter().filter_map(Relay::deadline).min() + self.nodes.iter().filter_map(|router| router.deadline()).min() } /// Let `elapsed` pass, firing each doubt timer due by then. @@ -929,19 +652,15 @@ impl Sim { let at = self.nodes[node].deadline().expect("a timer"); self.now = self.now.max(at); let now = self.now; - if let Relay::Path(router) = &mut self.nodes[node] { - router.timeout(now); - } + self.nodes[node].timeout(now); self.flush(); } /// Fire every doubt timer due by now. fn tick(&mut self) { let now = self.now; - for relay in &mut self.nodes { - if let Relay::Path(router) = relay - && router.deadline().is_some_and(|at| at <= now) - { + for router in &mut self.nodes { + if router.deadline().is_some_and(|at| at <= now) { router.timeout(now); } } @@ -951,83 +670,13 @@ impl Sim { /// Move every router's output onto the queues and log its reader events. fn flush(&mut self) { for node in 0..self.nodes.len() { - if self.paths() { - self.flush_paths(node); - } else { - self.flush_routes(node); - } + self.flush_paths(node); self.flush_events(node); } } - fn flush_routes(&mut self, node: Node) { - { - let outs = match &mut self.nodes[node] { - Relay::Distance(router) => router.drain(), - Relay::Path(_) => unreachable!(), - }; - for out in outs { - let (to, msg) = match out { - Out::Route { to, id, reach } => { - // A route names every node it crossed once, the sender last. - if let Some(Reach::Finite(advert)) = &reach { - let me = hop(node, self.restarts[node]); - assert_eq!(advert.hops.last(), Some(&me), "node {node}'s hops for {id}"); - let named: BTreeSet = advert.hops.iter().copied().collect(); - assert_eq!(named.len(), advert.hops.len(), "node {node} sent {id} round a loop"); - let was = self.routed.insert((node, to, id), advert.clone()); - let same = |was: Advert| { - Advert { - hops: advert.hops.clone(), - ..was - } == *advert - }; - self.stats.hop_updates += usize::from(was.is_some_and(same)); - } else { - self.routed.remove(&(node, to, id)); - } - self.stats.route_msgs += 1; - self.stats.route_items += 1; - (to, Msg::Route { id, reach }) - } - Out::Batch { to, id, changes, floor } => { - self.stats.claim_msgs += 1; - self.stats.claim_items += changes.len(); - let interest = self.nodes[to].interest(); - ( - to, - Msg::Batch { - id, - changes, - floor, - interest, - }, - ) - } - Out::Request { to, id, seqno } => { - self.stats.requests += 1; - if !self.requested.insert((node, to, id, seqno)) { - self.stats.repeated_requests += 1; - } - (to, Msg::Request { id, seqno }) - } - Out::Echo { to, id } => { - self.stats.route_msgs += 1; - self.stats.route_items += 1; - self.acks.entry((node, to)).or_default().push_back(id); - continue; - } - }; - self.queues.entry((node, to)).or_default().push_back(msg); - } - } - } - fn flush_paths(&mut self, node: Node) { - let outs = match &mut self.nodes[node] { - Relay::Path(router) => router.drain(), - Relay::Distance(_) => unreachable!(), - }; + let outs = self.nodes[node].drain(); for out in outs { let (to, msg) = match out { path::Out::Route { to, id, prefix, reach } => { @@ -1100,8 +749,7 @@ impl Sim { let id = declared.expect("rotated"); let router = &self.nodes[node]; let rev = router.revision().unwrap(); - let changes = router - .own(id) + let changes = (router.view(id, &Path::empty())) .unwrap() .items .iter() @@ -1114,6 +762,9 @@ impl Sim { self.history.insert(id, changes); self.declared[node] = declared; } + // The simulator reads full views directly. Consume change notifications + // alongside events, as a driver does, so they retain no script history. + drop(self.nodes[node].changed()); for event in self.nodes[node].events() { let origin = self.source(event.id).unwrap_or(event.id); let since = event.item.since; @@ -1133,42 +784,27 @@ impl Sim { } } - /// Routes, prefixes, and streams visited across every node, under the path rule. + /// Routes, prefixes, and streams visited across every node. pub fn work(&self) -> usize { self.spent + (0..self.nodes.len()).map(|node| self.path(node).work()).sum::() } /// Identities recomputed across every node. pub fn touched(&self) -> usize { - self.nodes.iter().map(Relay::touched).sum() - } - - pub fn route(&self, node: Node, id: Id) -> Option> { - self.babel(node).selected(id) + self.nodes.iter().map(|router| router.touched()).sum() } - pub fn export_route(&self, node: Node, id: Id) -> Option> { - self.babel(node).export(id) - } - - /// The neighbour `node` routes `id` through: under the path rule, the widest - /// live route's. + /// The neighbour `node` routes `id` through: the widest live route's. pub fn next_hop(&self, node: Node, id: Id) -> Option { - match &self.nodes[node] { - Relay::Distance(router) => router.selected(id)?.next, - Relay::Path(router) => (router.routes(id).find(|(_, route)| route.offer().is_some()))?.1.next(), - } + (self.nodes[node].routes(id).find(|(_, route)| route.offer().is_some()))? + .1 + .next() } /// The neighbour `node` routes `id` through, `Some(None)` at its publisher, or - /// `None` with no route: under the path rule, no live one. + /// `None` with no live route. pub fn toward(&self, node: Node, id: Id) -> Option> { - match &self.nodes[node] { - Relay::Distance(router) => router.selected(id).map(|route| route.next), - Relay::Path(router) => { - (router.routes(id).find(|(_, route)| route.offer().is_some())).map(|(_, route)| route.next()) - } - } + (self.nodes[node].routes(id).find(|(_, route)| route.offer().is_some())).map(|(_, route)| route.next()) } /// `peer` notices its session to `node` closed, which `node` has yet to. @@ -1176,22 +812,15 @@ impl Sim { self.up.remove(&Self::pair(node, peer)); for key in [(node, peer), (peer, node)] { self.queues.remove(&key); - self.acks.remove(&key); self.held.remove(&key); } - match &mut self.nodes[peer] { - Relay::Distance(router) => router.disconnect([node]), - Relay::Path(router) => router.disconnect([node], self.now), - } + self.nodes[peer].disconnect([node], self.now); self.flush(); } - /// Whether `node` has a route for `id`: under the path rule, a live one. + /// Whether `node` has a live route for `id`. pub fn routed(&self, node: Node, id: Id) -> bool { - match &self.nodes[node] { - Relay::Distance(router) => router.selected(id).is_some(), - Relay::Path(router) => router.routes(id).any(|(_, route)| route.offer().is_some()), - } + self.nodes[node].routes(id).any(|(_, route)| route.offer().is_some()) } /// The node a request from `node`'s reader for `claim` of `id` ends at, following @@ -1210,36 +839,13 @@ impl Sim { panic!("a request for {id} {claim:?} from node {node} loops"); } - pub fn starved(&self, node: Node, id: Id) -> bool { - self.babel(node).starved(id) - } - - pub fn grounded(&self, node: Node, id: Id) -> bool { - self.babel(node).grounded(id) - } - pub fn holds(&self, node: Node, id: Id) -> bool { self.nodes[node].holds(id) } - pub fn remembers(&self, node: Node, id: Id) -> bool { - self.babel(node).remembers(id) - } - - /// The floors `node` keeps for `id`: one per view under the path rule. + /// The floors `node` keeps for `id`: one per view. pub fn floors(&self, node: Node, id: Id) -> usize { - match &self.nodes[node] { - Relay::Distance(router) => router.floors(id), - Relay::Path(router) => router.views().filter(|((key, _), _)| *key == id).count(), - } - } - - pub fn identities(&self, node: Node) -> usize { - self.babel(node).identities() - } - - pub fn aliases(&self, node: Node) -> usize { - self.babel(node).aliases() + self.nodes[node].views().filter(|((key, _), _)| *key == id).count() } pub fn shown(&self, node: Node) -> &BTreeMap<(Id, Path), Shown> { @@ -1251,34 +857,19 @@ impl Sim { self.shown(node).get(&(id, claim)).map(|shown| shown.available) } - /// What either rule retains, summed over every node, in the terms they share. + /// What the routers retain, summed over every node. pub fn state(&self) -> State { - match self.rule { - Rule::Distance => { - let r = self.retained(); - State { - items: r.items, - floors: r.floors, - heard: r.heard, - sent: r.sent, - pending: r.echoes + r.pending, - } - } - Rule::Path(_) => { - let r = self.kept(); - State { - items: r.items, - floors: r.views, - heard: r.heard, - sent: r.sent, - pending: r.dead + r.doubt, - } - } + let r = self.kept(); + State { + items: r.items, + floors: r.views, + heard: r.heard, + sent: r.sent, + pending: r.dead + r.doubt, } } - /// What waits on another node: requests and echoes under the distance rule, dead - /// hops and doubts under the path rule. + /// What waits on another node: dead hops and doubts. pub fn pending(&self) -> usize { self.state().pending } @@ -1300,23 +891,6 @@ impl Sim { out } - /// Retained state summed over every node. - pub fn retained(&self) -> Retained { - let mut out = Retained::default(); - for node in 0..self.nodes.len() { - let r = self.babel(node).retained(); - out.items += r.items; - out.floors += r.floors; - out.heard += r.heard; - out.sent += r.sent; - out.echoes += r.echoes; - out.pending += r.pending; - out.distances += r.distances; - out.certified += r.certified; - } - out - } - /// The origin's state of `claim` as of revision `rev`. pub fn truth(&self, id: Id, claim: &str, rev: Rev) -> Option { let claim = p(claim); @@ -1373,7 +947,8 @@ impl Sim { pub fn check(&self) -> Result, String> { let mut checked = BTreeSet::new(); for (node, router) in self.nodes.iter().enumerate() { - for (key, view) in router.views() { + for ((key, _), view) in router.views() { + let key = *key; let Some(id) = self.source(key) else { continue; }; @@ -1487,36 +1062,23 @@ pub(super) fn false_ends(events: &[SimEvent]) -> usize { .count() } -/// Every node's next-hop chain toward `id` ends without revisiting a node. Under the -/// path rule a chain may come round only through a stale route, one whose next hop -/// has moved since offering it: the model's `LoopsAreStale`. +/// Every node's next-hop chain toward `id` ends without revisiting a node. A chain may +/// come round only through a stale route, one whose next hop has moved since offering +/// it: the model's `LoopsAreStale`. pub(super) fn assert_loop_free(sim: &Sim, nodes: usize, id: Id) { - if sim.paths() { - let claims: BTreeSet = (0..nodes) - .flat_map(|node| { - sim.path(node).routes(id).map(|(prefix, _)| prefix.clone()).chain( - sim.path(node) - .shown() - .keys() - .filter_map(move |(key, claim)| (*key == id).then_some(claim.clone())), - ) - }) - .collect(); - for node in 0..nodes { - for claim in &claims { - assert_scoped_chain(sim, node, id, claim); - } - } - return; - } + let claims: BTreeSet = (0..nodes) + .flat_map(|node| { + sim.path(node).routes(id).map(|(prefix, _)| prefix.clone()).chain( + sim.path(node) + .shown() + .keys() + .filter_map(move |(key, claim)| (*key == id).then_some(claim.clone())), + ) + }) + .collect(); for node in 0..nodes { - let mut chain = vec![node]; - while let Some(next) = sim.next_hop(*chain.last().expect("a node"), id) { - if let Some(start) = chain.iter().position(|&seen| seen == next) { - let cycle = &chain[start..]; - panic!("{id} loops from node {node} round {cycle:?}"); - } - chain.push(next); + for claim in &claims { + assert_scoped_chain(sim, node, id, claim); } } } diff --git a/rs/moq-net/src/route/tests/residual.rs b/rs/moq-net/src/route/tests/residual.rs index a47cf104add..5d9125cee6c 100644 --- a/rs/moq-net/src/route/tests/residual.rs +++ b/rs/moq-net/src/route/tests/residual.rs @@ -1,14 +1,12 @@ //! What the reviews of the hop list found on meshes where neighbours know different -//! parts of a publisher's claims, run on both rules. +//! parts of a publisher's claims. //! -//! The first three were bugs of [`Router`], which the route per scope fixes: each -//! asserts the fix under the path rule and still pins the bug under the distance -//! rule, which sessions run until the switch (`quest/m2/cluster-routes/switch.md`) -//! deletes it. The others are rules either must keep. +//! The first three were bugs of Babel's distance rule, which the route per scope +//! fixes: each asserts the fix. The others are rules the path rule must keep. use super::*; -both!( +fixtures!( a_seqno_advance_storms_between_two_grants, a_relay_behind_a_scoped_neighbour_is_blind, a_relay_forgets_a_claim_a_fresh_relay_would_show, @@ -20,12 +18,9 @@ both!( /// The cost of `node`'s route for `id` under `prefix`. fn metric(sim: &Sim, node: Node, id: Id, prefix: &str) -> Option { - match sim.rule() { - Rule::Distance => sim.route(node, id).map(|route| route.metric), - Rule::Path(_) => (sim.path(node).route_for(id, &p(prefix))) - .and_then(|route| route.offer()) - .map(|offer| offer.metric), - } + (sim.path(node).route_for(id, &p(prefix))) + .and_then(|route| route.offer()) + .map(|offer| offer.metric) } /// One advance of the origin's seqno and this mesh never goes quiet again under @@ -57,18 +52,11 @@ fn a_seqno_advance_storms_between_two_grants(rule: Rule) { sim.link(5, 6, SimLink::cost(1).grant(ga), SimLink::cost(1)); sim.link(5, 7, SimLink::cost(1).grant(gb), SimLink::cost(1)); sim.link(6, 7, SimLink::cost(3), SimLink::cost(1)); - sim.originate(origin, 8, 0); + sim.originate(origin, 8); sim.announce(origin, "", 0, None); sim.watch(4, "a"); sim.watch(7, "b"); assert!(sim.settle(3000)); - if rule == Rule::Distance { - assert_eq!(sim.seqno(origin), 0, "it converges with no request"); - sim.bump(origin); - assert!(!sim.settle(500), "the storm is fixed: drop the residual"); - assert!(sim.seqno(origin) > 20, "each lap asks the origin for another seqno"); - return; - } let shown = |sim: &Sim| [4, 7].map(|node| sim.shown(node).clone()); let before = shown(&sim); sim.disconnect(origin, 4); @@ -91,20 +79,14 @@ fn a_relay_behind_a_scoped_neighbour_is_blind(rule: Rule) { sim.join(far, wide, 1); sim.join(wide, relay, 1); sim.join(relay, tail, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "a/x", 0, None); sim.announce(origin, "b/x", 0, None); assert!(sim.run()); assert_eq!(sim.shows(wide, 7, "b/x"), Some(true)); for node in [relay, tail] { assert_eq!(sim.shows(node, 7, "a/x"), Some(true), "node {node}"); - let blind = rule == Rule::Distance; - assert_eq!(sim.shows(node, 7, "b/x"), (!blind).then_some(true), "node {node}"); - } - if rule == Rule::Distance { - let routes: Vec = sim.babel(relay).routes(7).into_iter().map(|(next, ..)| next).collect(); - assert_eq!(routes, [narrow], "the wide neighbour is no route either"); - return; + assert_eq!(sim.shows(node, 7, "b/x"), Some(true), "node {node}"); } assert_eq!(sim.reaches(relay, 7, &p("b/x")), Some(origin)); let first = sim.path(relay).forward(7, &p("b/x"), None); @@ -142,9 +124,9 @@ fn forgetful(rule: Rule, cut: bool, narrow: bool) -> Sim { sim.join(2, 3, 1); sim.join(2, 5, 1); sim.link(3, 4, SimLink::cost(1), SimLink::cost(2)); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "", 0, None); - sim.originate(other, 8, 0); + sim.originate(other, 8); sim.announce(other, "", 0, None); assert!(sim.run()); sim @@ -169,8 +151,7 @@ fn a_relay_forgets_a_claim_a_fresh_relay_would_show(rule: Rule) { assert_eq!(metric(sim, 0, 7, ""), Some(4)); } assert_eq!(fresh.shows(0, 7, ""), Some(true)); - let forgot = rule == Rule::Distance; - assert_eq!(sim.shows(0, 7, ""), (!forgot).then_some(true)); + assert_eq!(sim.shows(0, 7, ""), Some(true)); } /// A neighbour that stops certifying and returns brings back what it certified. @@ -201,26 +182,17 @@ fn a_seqno_advance_leaves_a_relays_claims_alone(rule: Rule) { sim.join(origin, narrow, 1); sim.join(wide, relay, 1); sim.join(narrow, relay, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "", 0, None); assert!(sim.run()); - let claim = match rule { - Rule::Distance => "a", - Rule::Path(_) => "", - }; - assert_eq!(sim.shows(relay, 7, claim), Some(true)); + assert_eq!(sim.shows(relay, 7, ""), Some(true)); let before = sim.events.len(); - match rule { - Rule::Distance => sim.bump(origin), - Rule::Path(_) => { - sim.disconnect(origin, narrow); - sim.connect(origin, narrow); - } - } + sim.disconnect(origin, narrow); + sim.connect(origin, narrow); assert!(sim.settle(500)); assert!(sim.events[before..].iter().all(|event| event.node != relay)); - assert_eq!(sim.shows(relay, 7, claim), Some(true)); + assert_eq!(sim.shows(relay, 7, ""), Some(true)); assert_eq!(sim.shown(relay).len(), 1); } @@ -238,10 +210,9 @@ fn two_narrowed_interests_settle(rule: Rule) { sim.join(relay, b, 1); sim.join(b, tail, 1); sim.link(tail, a, SimLink::cost(4), SimLink::cost(1)); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "", 0, None); assert!(sim.settle(3000), "never settles"); - assert!(sim.paths() || sim.seqno(origin) <= 1); } /// Two relays behind a relay whose only way to the origin asked for `a`. One of @@ -258,7 +229,7 @@ fn two_relays_behind_a_scoped_relay_let_a_dead_claim_go(rule: Rule) { sim.join(relay, right, 1); sim.join(left, right, 1); sim.join(origin, right, 5); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "a/x", 0, None); sim.announce(origin, "b/x", 0, None); assert!(sim.run()); diff --git a/rs/moq-net/src/route/tests/routing.rs b/rs/moq-net/src/route/tests/routing.rs index b1ab2f68e84..a2f3b1ac7fa 100644 --- a/rs/moq-net/src/route/tests/routing.rs +++ b/rs/moq-net/src/route/tests/routing.rs @@ -1,5 +1,5 @@ -//! Route selection: identity, feasibility, seqno requests, GOAWAY, the -//! down-only bit, and qualified identities. +//! Route selection: GOAWAY, the down-only bit, qualified identities, the hop limit, +//! and how a view merges what each neighbour certifies. use super::*; @@ -10,386 +10,67 @@ const LOCAL: Grant = Grant { /// Originate `id` at `node` with a root claim, which meets every interest: a /// route goes to a stream only while a claim meets the stream's interest. -fn originate(sim: &mut Sim, node: Node, id: Id, seqno: Seqno) { - sim.originate(node, id, seqno); +fn originate(sim: &mut Sim, node: Node, id: Id) { + sim.originate(node, id); sim.announce(node, "", 0, None); } fn next(sim: &Sim, node: Node, id: Id) -> Option { - sim.route(node, id).and_then(|route| route.next) + sim.next_hop(node, id) } -/// A restart that reuses its routing identity restarts its seqno below the -/// feasibility distance every other node still holds, so its routes are -/// refused until a seqno request lifts it past the old process's seqno. A -/// fresh identity has no history to lose to. -#[test] -fn restart_reusing_identity_starves_until_requested() { - let (origin, a, b) = (0, 1, 2); - let mut sim = Sim::new(3); - sim.join(origin, a, 1); - sim.join(a, b, 1); - originate(&mut sim, origin, 7, 0); - for _ in 0..3 { - sim.bump(origin); - } - assert!(sim.run()); - assert!(sim.route(b, 7).is_some()); - - sim.kill(origin); - originate(&mut sim, origin, 7, 0); - sim.reconnect(origin); - sim.hold(a, origin); - assert!(sim.run()); - assert!(sim.starved(a, 7), "seqno 0 is older than the 3 a remembers"); - assert!(sim.route(b, 7).is_none()); - sim.release(a, origin); - assert!(sim.run()); - assert_eq!(sim.seqno(origin), 4, "the request names the seqno after a's distance"); - assert!(sim.route(b, 7).is_some()); - - sim.kill(origin); - originate(&mut sim, origin, 8, 0); - sim.reconnect(origin); - let requests = sim.stats.requests; - assert!(sim.run()); - assert!(sim.route(b, 8).is_some()); - assert_eq!(sim.stats.requests, requests, "a fresh identity needs no request"); -} - -/// The same process moving to a farther attachment keeps its identity and -/// seqno, so the node next to its old attachment holds a feasibility distance -/// shorter than any route it now has and starves until its seqno request -/// reaches the origin. -#[test] -fn reconnect_without_restart_needs_a_seqno_advance() { - let (origin, a, b, c) = (0, 1, 2, 3); - let mut sim = Sim::new(4); - sim.join(a, b, 1); - sim.join(b, c, 1); - sim.join(origin, a, 1); - sim.join(origin, c, 1); - sim.disconnect(origin, c); - originate(&mut sim, origin, 7, 0); - assert!(sim.run()); - - sim.hold(a, b); - sim.disconnect(origin, a); - sim.connect(origin, c); - assert!(sim.run()); - assert!(sim.starved(a, 7), "a advertised one hop; every new route is longer"); - - sim.release(a, b); - assert!(sim.run()); - assert_eq!(sim.seqno(origin), 1); - assert_eq!(next(&sim, a, 7), Some(b)); -} - -/// A dead incarnation's route can return after a node collects it, but only -/// from a message already in flight on a live session, and that session then -/// delivers the retraction behind it. -#[test] -fn collected_incarnation_returns_only_while_its_retraction_is_in_flight() { - let (origin, a, b, c) = (0, 1, 2, 3); - let mut sim = Sim::new(4); - sim.join(origin, a, 1); - sim.join(a, b, 1); - sim.join(a, c, 1); - sim.join(c, b, 1); - // Everything a sends b waits: first the route, then its retraction. - sim.hold(a, b); - originate(&mut sim, origin, 7, 0); - assert!(sim.run()); - assert_eq!(next(&sim, b, 7), Some(c)); - - sim.kill(origin); - assert!(sim.run()); - assert!(sim.route(b, 7).is_none()); - - sim.collect(b, 7); - assert!(sim.deliver(a, b), "the held update"); - assert!(sim.route(b, 7).is_some(), "the dead incarnation is back at b"); - assert!(sim.run()); - assert!(sim.route(a, 7).is_none()); - assert_loop_free(&sim, 4, 7); - - sim.release(a, b); - assert!(sim.run()); - for node in 0..4 { - assert!(sim.route(node, 7).is_none(), "node {node} routes to a dead incarnation"); - } -} - -/// No route entry is not proof that no advertisement derived from this node's -/// own is still in flight. b keeps its feasibility distance until every -/// neighbor it advertised to echoes the ROUTE_END back, so its own echo from c -/// is refused rather than looping. A leaf never advertises the identity back, -/// so only its explicit echo ends the drain. -#[test] -fn collection_keeps_the_feasibility_distance_against_a_queued_echo() { - let (origin, b, c, leaf) = (0, 1, 2, 3); - let mut sim = Sim::new(4); - sim.leaf(leaf); - sim.join(origin, b, 1); - sim.join(b, c, 1); - sim.join(b, leaf, 1); - sim.hold(c, b); - originate(&mut sim, origin, 7, 0); - assert!(sim.run()); - assert_eq!(next(&sim, c, 7), Some(b)); - assert!(sim.queued(c, b) > 0, "c's echo waits"); - - sim.hold(b, c); - sim.hold(leaf, b); - sim.disconnect(origin, b); - assert!(!sim.holds(b, 7)); - sim.collect(b, 7); - assert!(sim.remembers(b, 7), "draining: the ROUTE_ENDs are not echoed yet"); - assert!(sim.deliver(c, b), "the echo of b's own advertisement"); - assert!(sim.route(b, 7).is_none(), "b refuses its own echo"); - assert!(sim.starved(b, 7)); - - sim.release(b, c); - sim.release(c, b); - assert!(sim.run()); - assert!(sim.queued_acks(leaf, b) > 0, "the leaf's echo waits"); - assert!(sim.remembers(b, 7), "the leaf may still hold b's advertisement"); - sim.release(leaf, b); - assert!(sim.run()); - for node in [b, c, leaf] { - assert!( - sim.route(node, 7).is_none(), - "node {node} routes to an unreachable origin" - ); - assert!(!sim.holds(node, 7), "node {node}"); - assert!(!sim.remembers(node, 7), "node {node} keeps a drained distance"); - } +/// The cost of `node`'s root route for `id`. +fn metric(sim: &Sim, node: Node, id: Id) -> Option { + let route = sim.path(node).route_for(id, &Path::empty())?; + Some(route.offer()?.metric) } -/// A restarted relay loses every feasibility distance, yet stays loop-free: -/// its sessions close with the process, and each neighbor drops what it -/// learned on them. A stale echo still reaches it through a third node's -/// session that outlived it, and is safe, because every node on that echo's -/// path kept its own distance. -#[test] -fn restart_drops_the_feasibility_distance_with_every_session() { - let (origin, b, c, d) = (0, 1, 2, 3); - let mut sim = Sim::new(4); - sim.join(origin, b, 1); - sim.join(b, c, 1); - sim.join(c, d, 1); - sim.join(d, b, 10); - originate(&mut sim, origin, 7, 0); - assert!(sim.run()); - assert_eq!(next(&sim, d, 7), Some(c)); - - // The update c sends d after losing b waits on a session that outlives b. - sim.hold(c, d); - sim.kill(b); - assert!(sim.run()); - assert_loop_free(&sim, 4, 7); - assert!(sim.route(c, 7).is_none()); - assert_eq!(next(&sim, d, 7), Some(c), "stale"); - - sim.connect(b, d); - assert!(sim.run()); - assert_eq!( - next(&sim, b, 7), - Some(d), - "the old advertisement reaches the new process" - ); - assert_loop_free(&sim, 4, 7); - sim.connect(b, c); - assert!(sim.run()); - assert!(sim.route(c, 7).is_none(), "the distance at c refuses the longer copy"); - assert_loop_free(&sim, 4, 7); - - sim.release(c, d); - assert!(sim.run()); - assert_loop_free(&sim, 4, 7); - sim.connect(origin, b); - assert!(sim.run()); - assert_loop_free(&sim, 4, 7); - for node in [b, c, d] { - assert!(sim.route(node, 7).is_some(), "node {node} recovers"); - } +/// A route along `hops`, the publisher first and the sender last, costing `metric`. +fn offer(hops: impl IntoIterator, metric: Metric) -> Option { + let path = (hops.into_iter().enumerate()) + .map(|(session, node)| path::Hop { + node, + session: session as u64, + }) + .collect(); + Some(path::Reach::Finite(path::Offer { + path, + draining: false, + priority: 0, + metric, + down_only: false, + rtt: None, + })) } -/// Random meshes churn links, hold and deliver messages out of step, and -/// collect route-less identities. Each node drops a feasibility distance only -/// once every neighbor echoed its ROUTE_END, so no selection ever loops. -#[test] -fn sweep_forgetting_distances() { - const RELAYS: usize = 5; - let mut rng = XorShift(0x5eed_f0d5); - let mut collected = 0; - for _ in 0..200 { - let nodes = RELAYS + 2; - let mut sim = Sim::new(nodes); - let mut links = Vec::new(); - for a in 0..RELAYS { - for b in a + 1..RELAYS { - if b == a + 1 || rng.chance(40) { - sim.join(a, b, 1 + rng.next(3)); - links.push((a, b)); - } - } - } - let ids = [RELAYS as u64, RELAYS as u64 + 1]; - for origin in RELAYS..nodes { - let relay = rng.next(RELAYS as u64) as usize; - sim.join(origin, relay, 1); - links.push((relay, origin)); - originate(&mut sim, origin, origin as u64, 0); - } - assert!(sim.run()); - - for _ in 0..60 { - match rng.next(8) { - 0 => { - let (a, b) = links[rng.next(links.len() as u64) as usize]; - if sim.connected(a, b) { - sim.disconnect(a, b); - } else { - sim.connect(a, b); - } - } - 1 | 2 => { - let (a, b) = links[rng.next(links.len() as u64) as usize]; - let (from, to) = if rng.chance(50) { (a, b) } else { (b, a) }; - if rng.chance(60) { - sim.hold(from, to); - } else { - sim.release(from, to); - } - } - 3 => { - let busy = sim.busy(); - if !busy.is_empty() { - let (from, to) = busy[rng.next(busy.len() as u64) as usize]; - sim.deliver(from, to); - } - } - 4 => sim.bump(RELAYS + rng.next(2) as usize), - _ => { - let node = rng.next(RELAYS as u64) as usize; - let id = ids[rng.next(2) as usize]; - if !sim.holds(node, id) { - sim.collect(node, id); - collected += 1; - } - } - } - sim.settle(1); - for &id in &ids { - assert_loop_free(&sim, nodes, id); - } - } - } - assert!(collected > 0); +fn shows(router: &path::Router, id: Id, claim: &str) -> Option { + router.shown().get(&(id, p(claim))).map(|shown| shown.available) } -/// Babel's feasibility needs every link to increase the metric, so a -/// zero-cost link is refused when it is configured. +/// A selection rule that never converges on a zero-cost link refuses one when it is +/// configured. #[test] fn zero_cost_links_are_refused() { - let mut router: Router = Router::new(99, 1); + let mut router: path::Router = path::Router::new(99, 1, Doubt::default()); let link = Link { cost: 0, ..Link::default() }; - assert_eq!(router.connect(1, link), Err(Error::ZeroCost)); + assert_eq!(router.connect(1, 101, link), Err(Error::ZeroCost)); } -/// A total reaching the infinite metric is unreachable, never saturated, and -/// not waiting on a seqno. +/// A total reaching the infinite metric is unreachable, never saturated. #[test] fn overflow_is_unreachable() { let (origin, a, b) = (0, 1, 2); let mut sim = Sim::new(3); sim.join(origin, a, MAX - 1); sim.join(a, b, 5); - originate(&mut sim, origin, 7, 0); + originate(&mut sim, origin, 7); assert!(sim.run()); - assert_eq!(sim.route(a, 7).map(|route| route.metric), Some(MAX - 1)); - assert!(sim.route(b, 7).is_none()); - assert!(!sim.starved(b, 7), "unreachable, not waiting on a seqno"); -} - -/// Redundant cores are feasible successors: losing one fails over at once, -/// with no seqno request. -#[test] -fn redundant_cores_fail_over_without_a_seqno_request() { - let (ingest, core_a, core_b, edge) = (0, 1, 2, 3); - let mut sim = Sim::new(4); - sim.join(ingest, core_a, 1); - sim.join(ingest, core_b, 1); - sim.join(core_a, core_b, 1); - sim.link(edge, core_a, SimLink::cost(1).upstream(), SimLink::cost(1)); - sim.link(edge, core_b, SimLink::cost(1).upstream(), SimLink::cost(1)); - originate(&mut sim, ingest, 7, 0); - assert!(sim.run()); - let primary = next(&sim, edge, 7).unwrap(); - - let requests = sim.stats.requests; - sim.disconnect(edge, primary); - assert!( - sim.route(edge, 7).is_some(), - "the other core advertises no more than the lost one" - ); - assert!(sim.run()); - assert_eq!(sim.stats.requests, requests); -} - -/// A publisher reached directly over P2P and through the CDN. A browser -/// advertises nothing it learned, so losing the P2P session falls back at -/// once. A routing node advertised the one-hop route, so the CDN detour is -/// infeasible until its seqno request reaches the publisher. -#[test] -fn p2p_fallback_is_immediate_for_leaves_and_needs_a_seqno_for_transit() { - for transit in [false, true] { - let (publisher, edge, subscriber) = (0, 1, 2); - let mut sim = Sim::new(3); - sim.leaf(publisher); - if !transit { - sim.leaf(subscriber); - } - sim.join(publisher, edge, 1); - sim.link(subscriber, edge, SimLink::cost(1).upstream(), SimLink::cost(1)); - sim.link(subscriber, publisher, SimLink::cost(1).priority(1), SimLink::cost(1)); - originate(&mut sim, publisher, 9, 0); - assert!(sim.run()); - assert_eq!(next(&sim, subscriber, 9), Some(publisher)); - - sim.disconnect(subscriber, publisher); - assert_eq!(sim.starved(subscriber, 9), transit); - assert!(sim.run()); - assert_eq!(next(&sim, subscriber, 9), Some(edge)); - assert_eq!(sim.seqno(publisher), u64::from(transit)); - } -} - -/// A relay whose only peers are upstream sends no update for its down-only -/// route, so it holds no feasibility distance and takes a longer route at the -/// same seqno once its best uplink goes. -#[test] -fn feasibility_tracks_only_updates_actually_sent() { - let (relay, c1, c2, origin) = (0, 1, 2, 3); - let mut sim = Sim::new(4); - sim.link(relay, c1, SimLink::cost(1).upstream(), SimLink::cost(1)); - sim.link(relay, c2, SimLink::cost(1).upstream(), SimLink::cost(1)); - sim.leaf(origin); - sim.join(origin, c1, 1); - sim.join(origin, c2, 5); - originate(&mut sim, origin, 7, 0); - assert!(sim.run()); - assert_eq!(next(&sim, relay, 7), Some(c1)); - assert!(!sim.remembers(relay, 7)); - - sim.disconnect(relay, c1); - assert_eq!(next(&sim, relay, 7), Some(c2)); - assert!(sim.run()); - assert_eq!(sim.seqno(origin), 0); + assert_eq!(metric(&sim, a, 7), Some(MAX - 1)); + assert!(!sim.routed(b, 7)); + assert_eq!(sim.shows(b, 7, ""), None, "unreachable, not shown"); } /// A GOAWAY ranks the peer below every eligible peer, priority included, and @@ -403,7 +84,7 @@ fn goaway_fails_over_and_keeps_the_draining_peer_last() { sim.join(ingest, core_b, 1); sim.link(edge, core_a, SimLink::cost(1).upstream().priority(1), SimLink::cost(1)); sim.link(edge, core_b, SimLink::cost(1).upstream(), SimLink::cost(1)); - originate(&mut sim, ingest, 7, 0); + originate(&mut sim, ingest, 7); assert!(sim.run()); assert_eq!(next(&sim, edge, 7), Some(core_a)); @@ -420,61 +101,42 @@ fn goaway_fails_over_and_keeps_the_draining_peer_last() { ); } -/// An edge serving a client through two uplinks, where the second uplink's -/// route is longer than the one the edge advertised. A GOAWAY keeps the -/// draining uplink serving while the alternative is infeasible. The refresh -/// reaches the edge through both uplinks: if the alternative's arrives first -/// the edge moves; if the draining uplink's does, the edge keeps it until its -/// session closes and then waits for one more refresh, which its request asks -/// for. +/// Among equal routes a node stays on its next hop, and moves to a later session to +/// that node: the peer a GOAWAY's sender asked to leave reconnects, and the sender +/// moves off the old session, which ranks no lower there, so the old one can close. +/// Another stream of the session in use, an earlier session, or another node's equal +/// route moves nothing. #[test] -fn goaway_keeps_serving_until_a_refresh_makes_the_alternative_feasible() { - let (origin, a, b, edge, client) = (0, 1, 2, 3, 4); - let build = || { - let mut sim = Sim::new(5); - sim.leaf(origin); - sim.leaf(client); - sim.join(origin, a, 1); - sim.join(origin, b, 5); - sim.link(edge, a, SimLink::cost(1).upstream(), SimLink::cost(1)); - sim.link(edge, b, SimLink::cost(1).upstream(), SimLink::cost(1)); - sim.join(edge, client, 1); - originate(&mut sim, origin, 7, 0); - assert!(sim.run()); - sim +fn a_later_session_to_the_next_hop_takes_over_among_equals() { + let session = |session| Link { + session, + ..Link::default() }; + let next = |router: &path::Router| router.route_for(7, &Path::empty()).and_then(path::Route::next); + let mut router: path::Router = path::Router::new(99, 1, Doubt::default()); + router.connect(1, 101, session(5)).unwrap(); + router.edited(1); + router.route(1, 7, Path::empty(), offer([100, 101], 1)); + assert_eq!(next(&router), Some(1)); - for draining_first in [false, true] { - let mut sim = build(); - sim.goaway(a); - assert!(sim.run()); - assert_eq!( - next(&sim, edge, 7), - Some(a), - "b advertised 5, above the edge's distance of 2" - ); - assert!(sim.route(client, 7).is_some()); + // Another stream of the same session, and another node's equal route. + router.connect(2, 101, session(5)).unwrap(); + router.edited(2); + router.route(2, 7, Path::empty(), offer([100, 101], 1)); + router.connect(3, 102, session(9)).unwrap(); + router.edited(3); + router.route(3, 7, Path::empty(), offer([100, 102], 1)); + assert_eq!(next(&router), Some(1), "only a later session to the next hop moves"); - let late = if draining_first { b } else { a }; - sim.hold(late, edge); - sim.bump(origin); - assert!(sim.run()); - sim.release(late, edge); - assert!(sim.run()); - assert!(sim.route(client, 7).is_some()); - if draining_first { - assert_eq!( - next(&sim, edge, 7), - Some(a), - "the refresh through a lowered the distance first" - ); - sim.disconnect(edge, a); - assert!(sim.starved(edge, 7), "only another refresh moves the edge to b"); - assert!(sim.run()); - assert_eq!(sim.seqno(origin), 2, "the edge's request asked for it"); - } - assert_eq!(next(&sim, edge, 7), Some(b)); - } + router.connect(4, 101, session(6)).unwrap(); + router.edited(4); + router.route(4, 7, Path::empty(), offer([100, 101], 1)); + assert_eq!(next(&router), Some(4), "the later session takes over"); + + router.connect(5, 101, session(2)).unwrap(); + router.edited(5); + router.route(5, 7, Path::empty(), offer([100, 101], 1)); + assert_eq!(next(&router), Some(4), "an earlier session does not"); } /// Two CDN relays, each the uplink of one relay in a local mesh, reaching a @@ -497,7 +159,7 @@ fn two_uplinks(l2_marks_upstream: bool, grant: Option) -> Sim { sim.link(l2, c2, up, admit); sim.leaf(origin); sim.join(origin, c1, 1); - originate(&mut sim, origin, 7, 0); + originate(&mut sim, origin, 7); sim } @@ -509,150 +171,11 @@ fn two_uplinks_never_carry_cdn_transit() { let mut sim = two_uplinks(true, None); assert!(sim.run()); assert_eq!(next(&sim, c2, 7), Some(c1)); - assert!(sim.route(l2, 7).is_some()); - assert!(sim.route(l2, 7).unwrap().down_only); -} - -/// A local relay whose preferred route is down-only exports nothing, so the -/// CDN hauls the origin across its expensive internal link while a LAN path -/// sits unadvertised. Ranking local peers above a backup uplink avoids it. -/// Babel selects only feasible routes, so a transit relay's priority for a -/// longer route holds only if that route arrived before the shorter one was -/// advertised. -#[test] -fn preferred_down_only_route_hides_the_exportable_alternative() { - let (c1, c2, l1, l2, origin) = (0, 1, 2, 3, 4); - let build = |lan_first: bool, prefer_uplink: bool| { - let mut sim = Sim::new(5); - sim.join(c1, c2, 10); - sim.leaf(origin); - sim.join(origin, l1, 1); - originate(&mut sim, origin, 7, 0); - let (uplink, lan) = if prefer_uplink { (1, 0) } else { (0, 1) }; - let join_lan = |sim: &mut Sim| sim.link(l2, l1, SimLink::cost(1).priority(lan), SimLink::cost(1)); - if lan_first { - join_lan(&mut sim); - } - sim.link(l1, c1, SimLink::cost(1).upstream(), SimLink::cost(1)); - sim.link(l2, c2, SimLink::cost(1).upstream().priority(uplink), SimLink::cost(1)); - assert!(sim.run()); - if !lan_first { - join_lan(&mut sim); - } - assert!(sim.run()); - sim - }; - - let sim = build(false, true); - assert_eq!(next(&sim, l2, 7), Some(c2)); - assert!(sim.export_route(l2, 7).is_none()); - assert_eq!(next(&sim, c2, 7), Some(c1)); - assert_eq!(sim.route(c2, 7).map(|route| route.metric), Some(12)); - - let sim = build(false, false); - assert_eq!(next(&sim, l2, 7), Some(l1)); - assert_eq!(next(&sim, c2, 7), Some(l2)); - - // Arrival order decides the transit priority's outcome. - assert_eq!(next(&build(true, true), l2, 7), Some(l1)); -} - -/// A scoped peer forwards a victim's identity with a newer seqno. Unscoped, -/// the forgery captures the victim's route. Filed under a qualified key, it is -/// a separate identity, and the victim's route is untouched. A scoped peer's -/// own identity is qualified too, since nothing proves it. -#[test] -fn forged_identity_is_filed_apart_under_a_scoped_grant() { - for scoped in [false, true] { - let (victim, core, edge, forger) = (0, 1, 2, 3); - let mut sim = Sim::new(4); - sim.leaf(victim); - sim.leaf(forger); - sim.join(victim, core, 5); - sim.join(core, edge, 1); - let grant = Grant { - subject: "forger", - scope: "forger/", - }; - let admission = if scoped { - SimLink::cost(1).grant(grant) - } else { - SimLink::cost(1) - }; - sim.link(edge, forger, admission, SimLink::cost(1)); - originate(&mut sim, victim, 7, 0); - originate(&mut sim, forger, 3, 0); - assert!(sim.run()); - - sim.forge(forger, 7, 1000); - assert!(sim.run()); - let own = if scoped { qualified("forger", 3) } else { 3 }; - assert!(sim.route(core, own).is_some()); - assert_eq!( - sim.route(core, 3).is_some(), - !scoped, - "an unproven own identity is qualified" - ); - if scoped { - assert_eq!(next(&sim, core, 7), Some(victim)); - assert_eq!(next(&sim, core, qualified("forger", 7)), Some(edge)); - } else { - assert_eq!( - next(&sim, core, 7), - Some(edge), - "the forger captures the victim's route" - ); - } - - sim.kill(forger); - assert!(sim.run()); - assert_eq!(next(&sim, core, 7), Some(victim)); - } -} - -/// A local mesh publishing through two uplinks, each admitted under the same -/// subject. Both CDN relays file the local origin under one key, so it -/// survives losing an uplink, and so does the local relay's own identity; a -/// retracted identity leaves no alias behind. -#[test] -fn scoped_delivery_through_two_uplinks_reaches_one_key() { - let (c1, c2, l1, l2, origin) = (0, 1, 2, 3, 4); - let mut sim = Sim::new(5); - sim.join(c1, c2, 10); - sim.join(l1, l2, 1); - let admit = SimLink::cost(1).grant(LOCAL); - sim.link(l1, c1, SimLink::cost(1).upstream(), admit); - sim.link(l2, c2, SimLink::cost(1).upstream(), admit); - sim.leaf(origin); - sim.join(origin, l1, 1); - originate(&mut sim, origin, 9, 0); - originate(&mut sim, l1, 20, 0); - assert!(sim.run()); - - let key = qualified(LOCAL.subject, 9); - assert!(sim.route(c1, 9).is_none()); - assert_eq!(next(&sim, c1, key), Some(l1)); - assert_eq!(next(&sim, c2, key), Some(l2)); - assert_eq!(next(&sim, c1, qualified(LOCAL.subject, 20)), Some(l1)); - - sim.disconnect(l1, c1); - assert!(sim.run()); - assert_eq!( - next(&sim, c1, qualified(LOCAL.subject, 20)), - Some(c2), - "the local relay's own fails over" - ); - sim.connect(l1, c1); - assert!(sim.run()); - - sim.disconnect(l2, c2); - assert!(sim.run()); - assert_eq!(next(&sim, c2, key), Some(c1)); - - assert_eq!(sim.aliases(c1), 2, "the origin's and the local relay's own"); - sim.kill(origin); - assert!(sim.run()); - assert_eq!(sim.aliases(c1), 1); + let route = sim + .path(l2) + .route_for(7, &Path::empty()) + .and_then(|route| route.offer()); + assert!(route.expect("a route").down_only); } /// A local relay missing its `upstream` mark leaks CDN routes back to the @@ -675,23 +198,8 @@ fn misconfigured_uplink_is_contained_by_qualified_identities() { ); } -/// A qualified key the CDN sent down comes back up through the misconfigured -/// uplink. A scoped connection forwarding a qualified key has that route -/// refused, so the leak is filed once and never again. -#[test] -fn scoped_connections_cannot_requalify_a_qualified_identity() { - let c2 = 1; - let mut sim = two_uplinks(true, Some(LOCAL)); - assert!(sim.run()); - let marked = sim.identities(c2); - - let mut sim = two_uplinks(false, Some(LOCAL)); - assert!(sim.run()); - assert_eq!(sim.identities(c2), marked + 1, "the leak is filed once, never again"); -} - -/// A partitioned local mesh reaches its other half through the CDN once a -/// seqno request reaches the origin, and heals back onto the LAN. +/// A partitioned local mesh reaches its other half through the CDN once the doubt +/// on the far end of the closed session lifts, and heals back onto the LAN. #[test] fn partitioned_local_mesh_reaches_itself_through_the_cdn() { let (c1, c2, l1, l2, origin) = (0, 1, 2, 3, 4); @@ -702,14 +210,12 @@ fn partitioned_local_mesh_reaches_itself_through_the_cdn() { sim.link(l2, c2, SimLink::cost(1).upstream(), SimLink::cost(1)); sim.leaf(origin); sim.join(origin, l2, 1); - originate(&mut sim, origin, 7, 0); + originate(&mut sim, origin, 7); assert!(sim.run()); assert_eq!(next(&sim, l1, 7), Some(l2)); sim.disconnect(l1, l2); - assert!(sim.starved(l1, 7), "the CDN detour is longer than the LAN route"); assert!(sim.run()); - assert_eq!(sim.seqno(origin), 1); assert_eq!(next(&sim, l1, 7), Some(c1)); sim.connect(l1, l2); @@ -717,137 +223,20 @@ fn partitioned_local_mesh_reaches_itself_through_the_cdn() { assert_eq!(next(&sim, l1, 7), Some(l2), "heals"); } -/// A five-node ring loses its preferred origin link while a longer backup -/// remains. The backup's route is infeasible, so the starved node's request, -/// with duplicates suppressed at every hop, reaches the origin while the -/// backup never loses its session. -#[test] -fn a_ring_recovers_its_longer_backup_through_one_request() { - let origin = 5; - let mut sim = Sim::new(6); - for relay in 0..5 { - sim.join(relay, (relay + 1) % 5, 1); - } - sim.leaf(origin); - sim.join(origin, 0, 1); - sim.join(origin, 2, 10); - originate(&mut sim, origin, 7, 0); - assert!(sim.run()); - assert_eq!(next(&sim, 2, 7), Some(1)); - - let requests = sim.stats.requests; - sim.disconnect(origin, 0); - assert!(sim.run()); - for relay in 0..5 { - assert!(sim.route(relay, 7).is_some(), "relay {relay}"); - } - assert_eq!(sim.seqno(origin), 1); - assert!(sim.stats.requests > requests); - assert_eq!(sim.stats.repeated_requests, 0, "duplicates suppressed"); - assert_eq!(sim.retained().pending, 0); -} - -/// A request for an origin no usable route reaches stops once no usable -/// route is left, leaving nothing pending. -#[test] -fn an_unanswerable_request_stops() { - let origin = 4; - let mut sim = Sim::new(5); - for relay in 0..4 { - sim.join(relay, (relay + 1) % 4, 1); - } - sim.leaf(origin); - sim.join(origin, 0, 1); - originate(&mut sim, origin, 7, 0); - assert!(sim.run()); - sim.disconnect(origin, 0); - assert!(sim.run()); - for relay in 0..4 { - assert!(sim.route(relay, 7).is_none()); - assert!(!sim.holds(relay, 7)); - } - assert_eq!(sim.retained().pending, 0); -} - -/// A request newer than the one pending raises it and is sent on again, so the -/// newer requester is not left waiting on an answer that only meets the older -/// one; a repeat of the pending Seqno, or an older one, is a duplicate. -#[test] -fn a_newer_request_raises_the_pending_one() { - let (up, a, b) = (0, 1, 2); - let mut relay = Router::::new(99, 9); - for neighbor in [up, a, b] { - relay.connect(neighbor, Link::default()).unwrap(); - } - let advert = Advert { - seqno: 0, - metric: 1, - down_only: false, - rtt: None, - hops: [].into(), - }; - relay.route(up, 7, Some(Reach::Finite(advert))); - relay.drain(); - - let requests = |relay: &mut Router| -> Vec<(Node, Seqno)> { - relay - .drain() - .into_iter() - .filter_map(|out| match out { - Out::Request { to, id: 7, seqno } => Some((to, seqno)), - _ => None, - }) - .collect() - }; - - relay.request(a, 7, 1); - assert_eq!(requests(&mut relay), [(up, 1)]); - relay.request(b, 7, 1); - assert_eq!(requests(&mut relay), [], "a duplicate"); - relay.request(b, 7, 2); - assert_eq!(requests(&mut relay), [(up, 2)], "raised and sent again"); - relay.request(a, 7, 1); - assert_eq!(requests(&mut relay), [], "older than the pending one"); -} - -/// A route starved at the last seqno asks for nothing newer, since its origin rotates -/// to a new identity rather than advance past it. -#[test] -fn a_route_starved_at_the_last_seqno_requests_nothing() { - let (origin, a, b) = (0, 1, 2); - let mut sim = Sim::new(3); - sim.join(origin, a, 1); - sim.join(a, b, 1); - originate(&mut sim, origin, 7, MAX); - assert!(sim.run()); - assert!(sim.route(b, 7).is_some()); - - sim.kill(origin); - originate(&mut sim, origin, 7, 0); - sim.reconnect(origin); - let requests = sim.stats.requests; - assert!(sim.run()); - assert!(sim.starved(a, 7), "seqno 0 is older than the last one a remembers"); - assert_eq!(sim.stats.requests, requests, "nothing follows the last seqno"); -} - -/// Each message recomputes only the identity it names, so the work a message -/// costs does not grow with how many other origins a node routes to, on either rule. +/// A message to one identity recomputes that identity alone, however many others +/// the node routes. #[test] fn a_message_recomputes_only_its_identity() { - for (rule, origins) in [Rule::Distance, Rule::PATH] - .into_iter() - .flat_map(|rule| [(rule, 1), (rule, 50)]) - { + for origins in [1, 50] { let (a, b, client) = (0, 1, 2); - let mut sim = Sim::with(rule, 3 + origins); + let mut sim = Sim::new(3 + origins); sim.join(a, b, 1); sim.leaf(client); sim.join(client, b, 1); for origin in 3..3 + origins { sim.leaf(origin); sim.join(origin, a, 1); - sim.originate(origin, origin as Id, 0); + sim.originate(origin, origin as Id); sim.announce(origin, &format!("live/{origin}"), 0, None); } assert!(sim.run()); @@ -859,26 +248,16 @@ fn a_message_recomputes_only_its_identity() { assert_eq!( sim.touched() - touched, sim.stats.delivered - delivered + 1, - "{rule:?}: one identity per message, plus the origin's own change" + "one identity per message, plus the origin's own change" ); } } /// A route whose metric plus the link cost reaches the infinite metric is -/// unreachable, so it neither certifies regions nor keeps an identity held: -/// only the reachable neighbor's region shows, and a held overflowing route -/// is let go. +/// unreachable, so it shows nothing: only the reachable neighbour's region shows, +/// and a held overflowing route keeps nothing. #[test] fn an_overflowing_route_neither_certifies_nor_holds() { - let finite = |metric| { - Some(Reach::Finite(Advert { - seqno: 0, - metric, - down_only: false, - rtt: None, - hops: [].into(), - })) - }; let item = Item { seed: 0, epoch: None, @@ -890,428 +269,142 @@ fn an_overflowing_route_neither_certifies_nor_holds() { ..Link::default() }; - let mut router: Router = Router::new(99, 1); - router.connect(near, Link::default()).unwrap(); - router.connect(far, far_link.clone()).unwrap(); - router.asks(near, p("live/a")).unwrap(); - router.route(near, 7, finite(1)); - router.batch(near, 7, vec![(p("live/a/x"), Some(item))], Some(1), 0); - router.route(far, 7, finite(1)); - router.batch(far, 7, vec![(p("live/b/y"), Some(item))], Some(1), 0); - assert_eq!(router.selected(7).and_then(|route| route.next), Some(near)); - assert_eq!(router.shows(7, &p("live/a/x")), Some(true)); + let mut router: path::Router = path::Router::new(99, 1, Doubt::default()); + router.connect(near, 101, Link::default()).unwrap(); + router.edited(near); + router.connect(far, 102, far_link.clone()).unwrap(); + router.edited(far); + router.route(near, 7, p("live/a"), offer([100, 101], 1)); + router.batch(near, 7, p("live/a"), vec![(p("live/a/x"), Some(item))], Some(1)); + router.route(far, 7, Path::empty(), offer([100, 102], 1)); + router.batch(far, 7, Path::empty(), vec![(p("live/b/y"), Some(item))], Some(1)); + assert_eq!(shows(&router, 7, "live/a/x"), Some(true)); assert_eq!( - router.shows(7, &p("live/b/y")), + shows(&router, 7, "live/b/y"), None, - "only the unreachable route certifies it" + "only the unreachable route carries it" ); - let mut router: Router = Router::new(99, 1); - router.connect(far, far_link).unwrap(); - router.route(far, 7, finite(1)); - router.batch(far, 7, vec![(p("live/x"), Some(item))], Some(1), 0); - router.route(far, 7, Some(Reach::Held)); - assert!(router.selected(7).is_none()); - assert!(!router.grounded(7), "an unreachable held route recovers nothing"); - assert_eq!(router.shows(7, &p("live/x")), None); + let mut router: path::Router = path::Router::new(99, 1, Doubt::default()); + router.connect(far, 102, far_link).unwrap(); + router.edited(far); + router.route(far, 7, Path::empty(), offer([100, 102], 1)); + router.batch(far, 7, Path::empty(), vec![(p("live/x"), Some(item))], Some(1)); + router.route(far, 7, Path::empty(), Some(path::Reach::Held)); + assert_eq!(shows(&router, 7, "live/x"), None); } -/// A route grows to the hop limit with nothing else about it changed, so its sender -/// stops certifying. What only that neighbor certified is let go at once, though a -/// change of hops alone otherwise lets go of nothing. +/// A path that crossed as many nodes as one may is refused like an ended route: what +/// only that neighbour carried goes at once, while one hop fewer moves nothing. #[test] fn a_route_grown_to_the_hop_limit_certifies_nothing() { - let finite = |hops: usize| { - Some(Reach::Finite(Advert { - seqno: 0, - metric: 1, - down_only: false, - rtt: None, - hops: (1..=hops as Id).collect(), - })) - }; let item = Item { seed: 0, epoch: None, since: 1, }; let (narrow, wide) = (1, 2); - let mut router: Router = Router::new(99, 1); - router.connect(narrow, Link::default()).unwrap(); - router.connect(wide, Link::default()).unwrap(); - router.asks(narrow, p("live/a")).unwrap(); - router.route(narrow, 7, finite(2)); - router.batch(narrow, 7, vec![(p("live/a/x"), Some(item))], Some(1), 0); - router.route(wide, 7, finite(MAX_HOPS - 2)); - router.batch(wide, 7, vec![(p("live/b/y"), Some(item))], Some(1), 0); - assert_eq!(router.shows(7, &p("live/b/y")), Some(true)); - - router.route(wide, 7, finite(MAX_HOPS - 1)); + let hops = |count: usize| (0..count as Id).map(|hop| 200 + hop).chain([102]); + let mut router: path::Router = path::Router::new(99, 1, Doubt::default()); + router.connect(narrow, 101, Link::default()).unwrap(); + router.edited(narrow); + router.connect(wide, 102, Link::default()).unwrap(); + router.edited(wide); + router.route(narrow, 7, p("live/a"), offer([100, 101], 1)); + router.batch(narrow, 7, p("live/a"), vec![(p("live/a/x"), Some(item))], Some(1)); + router.route(wide, 7, Path::empty(), offer(hops(MAX_HOPS - 3), 1)); + router.batch(wide, 7, Path::empty(), vec![(p("live/b/y"), Some(item))], Some(1)); + assert_eq!(shows(&router, 7, "live/b/y"), Some(true)); + + router.route(wide, 7, Path::empty(), offer(hops(MAX_HOPS - 2), 1)); + assert_eq!(shows(&router, 7, "live/b/y"), Some(true), "one hop more moves nothing"); + router.route(wide, 7, Path::empty(), offer(hops(MAX_HOPS - 1), 1)); assert_eq!( - router.shows(7, &p("live/b/y")), - Some(true), - "one hop more moves nothing" - ); - router.route(wide, 7, finite(MAX_HOPS)); - assert_eq!( - router.shows(7, &p("live/b/y")), + shows(&router, 7, "live/b/y"), None, "the route crossed as many nodes as one may" ); - assert_eq!(router.shows(7, &p("live/a/x")), Some(true)); -} - -/// A fresh claim arrives only on a stream that certifies nothing, while an -/// older floor on another stream proves it absent. The merge rule trusts the -/// floor, so the claim stays hidden until the certifying neighbor refreshes -/// past it or lets go: that is the bound on how late an uncertified claim -/// shows. -#[test] -fn a_fresh_uncertified_claim_waits_on_an_older_floor() { - let finite = |metric| { - Some(Reach::Finite(Advert { - seqno: 0, - metric, - down_only: false, - rtt: None, - hops: [].into(), - })) - }; - let item = Item { - seed: 0, - epoch: None, - since: 2, - }; - let (certifier, carrier) = (1, 2); - for refresh in [true, false] { - let mut router: Router = Router::new(99, 1); - router.connect(certifier, Link::default()).unwrap(); - router.connect(carrier, Link::default()).unwrap(); - router.route(certifier, 7, finite(1)); - router.batch(certifier, 7, Vec::new(), Some(1), 0); - router.route(carrier, 7, finite(1)); - router.batch(carrier, 7, vec![(p("live/x"), Some(item))], None, 0); - assert_eq!(router.shows(7, &p("live/x")), None, "the older floor hides it"); - - match refresh { - true => router.batch(certifier, 7, vec![(p("live/x"), Some(item))], Some(2), 0), - false => router.disconnect([certifier]), - } - assert_eq!(router.shows(7, &p("live/x")), Some(true), "refresh {refresh}"); - } + assert_eq!(shows(&router, 7, "live/a/x"), Some(true)); } /// Closing a session's streams together leaves the node as closing them one at a -/// time does: each stream asks its own prefix, and some share one, so a region one -/// closes stays certified by another while the rest let go. Once none is left, the -/// reader sees the same; the view stays, as a node with no route keeps it until a -/// route returns or it collects the identity. +/// time does: each stream carries its own prefix, and some share one, so a region +/// one closes stays carried by another while the rest let go. #[test] fn closing_streams_together_matches_closing_them_in_turn() { - let finite = Some(Reach::Finite(Advert { - seqno: 0, - metric: 1, - down_only: false, - rtt: None, - hops: [].into(), - })); let item = |seed| Item { seed, epoch: None, since: 1, }; + let now = Instant::ZERO; let opened = || { - let mut router: Router = Router::new(99, 1); + let mut router: path::Router = path::Router::new(99, 1, Doubt::default()); for stream in 0..6 { - let prefix = format!("live/{}", stream % 4); - router.connect(stream, Link::default()).unwrap(); - router.asks(stream, Path::new(&prefix).unwrap()).unwrap(); - router.route(stream, 7, finite.clone()); + let prefix = p(&format!("live/{}", stream % 4)); + router.connect(stream, 101, Link::default()).unwrap(); + router.edited(stream); + router.route(stream, 7, prefix.clone(), offer([100, 101], 1)); let claims = (0..3) - .map(|j| (Path::new(&format!("{prefix}/{j}")).unwrap(), Some(item(j)))) + .map(|j| (prefix.join(&p(&j.to_string())).unwrap(), Some(item(j)))) .collect(); - router.batch(stream, 7, claims, Some(1), 0); + router.batch(stream, 7, prefix, claims, Some(1)); } router }; for closed in [vec![0, 1, 4], vec![0, 1, 2, 3, 4, 5]] { let mut together = opened(); - together.disconnect(closed.iter().copied()); + together.disconnect(closed.iter().copied(), now); let mut in_turn = opened(); for &stream in &closed { - in_turn.disconnect([stream]); + in_turn.disconnect([stream], now); } assert_eq!(together.shown(), in_turn.shown(), "closed {closed:?}"); - if closed.len() < 6 { - assert_eq!(together.view(7), in_turn.view(7), "closed {closed:?}"); - assert_eq!(together.retained(), in_turn.retained(), "closed {closed:?}"); - } else { + assert_eq!(together.retained(), in_turn.retained(), "closed {closed:?}"); + if closed.len() == 6 { assert!(together.shown().is_empty()); } } } -/// An echo travels apart from the echoing node's routes, so it can overtake a -/// route that node derived from this one's. b learned its route from a and sent it -/// back, then ended it on a's ROUTE_END. Its echo arrives first, so a drops its -/// feasibility distance, and b's old route arrives looking like a fresh, longer one -/// to an origin neither reaches. Its hops name a, so a refuses it, where two relays -/// once counted up without end. -#[test] -fn a_route_that_crossed_a_node_is_refused_once_its_distance_is_gone() { - let (origin, a, b) = (0, 1, 2); - let mut sim = Sim::new(3); - sim.join(origin, a, 1); - sim.join(origin, b, 1); - sim.join(a, b, 1); - // b hears the origin only through a, and b's route to a waits. - sim.hold(origin, b); - sim.hold(b, a); - originate(&mut sim, origin, 7, 0); - assert!(sim.run()); - assert_eq!(next(&sim, b, 7), Some(a)); - - sim.disconnect(origin, a); - assert!(sim.run()); - assert!(!sim.holds(b, 7), "b let go on a's ROUTE_END"); - assert!(sim.deliver_ack(b, a), "b's echo overtakes its own route"); - assert!(!sim.remembers(a, 7), "a dropped its distance"); - assert!(sim.deliver(b, a)); - assert!(sim.holds(a, 7), "b's route stays open until b ends it"); - assert!(sim.route(a, 7).is_none(), "a refuses its own route, come back"); - assert!(!sim.starved(a, 7), "no seqno would help"); - assert_eq!(sim.shows(a, 7, ""), None); - - // b still reaches the origin, and a takes the route b learns there. - sim.release(b, a); - sim.release(origin, b); - assert!(sim.run()); - assert_eq!( - sim.route(a, 7).map(|route| (route.next, route.metric)), - Some((Some(b), 2)) - ); -} - -/// The origin sits behind relay a, in a triangle with b and c. An echo proves only -/// that nothing derived is in flight on its own link: here c's copy of a's route -/// waits on its way to b while both echo a's ROUTE_END, so a drops its distance, and -/// then b takes c's copy and offers it to a. It crossed a, so a refuses it and the -/// copies die out, where the three relays once handed it round at a growing metric. -#[test] -fn a_route_handed_round_a_triangle_is_refused_where_it_began() { - let (origin, a, b, c) = (0, 1, 2, 3); - let mut sim = Sim::new(4); - sim.join(origin, a, 1); - sim.join(a, b, 2); - sim.join(a, c, 1); - sim.join(b, c, 2); - for (from, to) in [(c, b), (b, c), (b, a), (c, a)] { - sim.hold(from, to); - } - originate(&mut sim, origin, 7, 0); - assert!(sim.run()); - assert_eq!(sim.route(b, 7).map(|route| route.metric), Some(3)); - - sim.disconnect(origin, a); - assert!(sim.run()); - assert!(!sim.holds(b, 7) && !sim.holds(c, 7), "both let go on a's ROUTE_END"); - assert!(sim.deliver_ack(b, a) && sim.deliver_ack(c, a)); - assert!(!sim.remembers(a, 7), "every neighbor echoed, so a dropped its distance"); - - // c's copy, sent before the end, is shorter than what b advertised: feasible there. - assert!(sim.deliver(c, b)); - assert_eq!( - sim.route(b, 7).map(|route| (route.next, route.metric)), - Some((Some(c), 4)) - ); - - // b may offer it to a once a echoes the ROUTE_END b sent it. - let mut peak = 0; - for _ in 0..100 { - for (from, to) in [(b, a), (c, a), (a, b), (a, c)] { - sim.deliver(from, to); - sim.deliver_ack(from, to); - peak = peak.max(sim.route(a, 7).map_or(0, |route| route.metric)); - } - } - assert_eq!(peak, 0, "a never takes back a route that crossed it"); - - for (from, to) in [(c, b), (b, c), (b, a), (c, a)] { - sim.release(from, to); - } - assert!(sim.run()); - for node in [a, b, c] { - assert!(!sim.holds(node, 7), "node {node}"); - assert!(!sim.remembers(node, 7), "node {node}"); - } -} - -/// An origin names itself in its route's hops like any relay, so once it rotated -/// to a fresh identity and forgot the old one, a copy of the old route is still -/// refused where it began. -#[test] -fn a_retired_identitys_route_is_refused_by_the_node_that_originated_it() { - let (origin, b) = (0, 1); - let mut sim = Sim::new(2); - sim.join(origin, b, 1); - sim.hold(b, origin); - originate(&mut sim, origin, 7, MAX - 1); - assert!(sim.run()); - assert_eq!(next(&sim, b, 7), Some(origin)); - - sim.bump(origin); - assert_ne!(sim.proven(origin), 7, "the last seqno rotates the identity"); - assert!(sim.run()); - assert!(sim.deliver_ack(b, origin), "b's echo overtakes its own route"); - assert!(!sim.remembers(origin, 7)); - assert!(sim.deliver(b, origin)); - assert!(sim.holds(origin, 7), "b's copy of the old route arrived"); - assert!(sim.route(origin, 7).is_none(), "it crossed the node that began it"); - - sim.release(b, origin); - assert!(sim.run()); - assert!(!sim.holds(origin, 7) && !sim.remembers(origin, 7)); -} - -/// A scoped peer hands a relay's own route back under the key the relay files that -/// peer's identities by. The hops are never qualified, so the relay still finds -/// itself in them and refuses the copy. +/// A relay hands a peer under a scoped grant the origin's route, which the peer +/// files under a qualified key. The copy it would hand back names the relay, so the +/// relay never takes it for a route of its own. #[test] fn a_route_that_returns_through_a_scoped_peer_is_refused() { let (origin, relay, peer) = (0, 1, 2); let mut sim = Sim::new(3); sim.join(origin, relay, 1); sim.link(relay, peer, SimLink::cost(1).grant(LOCAL), SimLink::cost(1)); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "local/x", 0, None); assert!(sim.run()); let key = qualified(LOCAL.subject, 7); assert_eq!(next(&sim, peer, 7), Some(relay)); - assert!(sim.holds(relay, key), "the peer sent the route back"); - assert!(sim.route(relay, key).is_none(), "it crossed the relay"); + assert!(!sim.routed(relay, key), "it crossed the relay"); assert_eq!(sim.shows(relay, key, "local/x"), None); sim.disconnect(origin, relay); assert!(sim.run()); assert!(!sim.holds(relay, key) && !sim.holds(peer, 7)); -} - -/// A route that already crossed as many nodes as one may goes no further: the next -/// relay cannot add itself, so it is unreachable there, not waiting on a seqno. -#[test] -fn a_route_stops_at_the_hop_limit() { - let last = MAX_HOPS + 1; - let mut sim = Sim::new(last + 1); - for node in 0..last { - sim.join(node, node + 1, 1); - } - originate(&mut sim, 0, 7, 0); - assert!(sim.run()); - let full = last - 2; - assert_eq!(sim.route(full, 7).map(|route| route.metric), Some(full as Metric)); - assert!( - sim.route(full + 1, 7).is_none(), - "the route crossed as many nodes as one may" - ); - assert!(sim.holds(full + 1, 7) && !sim.starved(full + 1, 7)); - assert!(!sim.holds(last, 7), "nothing is sent on"); -} - -/// A relay sends the origin's route to a peer that files it under a qualified key -/// and hands it back. While the peer got there by another relay the copy is a route -/// like any other; once the peer takes this relay's own, the copy has crossed it. No -/// seqno changes that, so the relay lets the key go without asking for one. -#[test] -fn a_scoped_copy_that_crossed_a_relay_asks_for_no_seqno() { - let (origin, near, far, cdn) = (0, 1, 2, 3); - let mut sim = Sim::new(4); - sim.join(origin, near, 1); - sim.join(origin, far, 1); - sim.link(near, cdn, SimLink::cost(1), SimLink::cost(1).grant(LOCAL)); - sim.link(far, cdn, SimLink::cost(1), SimLink::cost(2).grant(LOCAL)); - // The peer hears the far relay first. - sim.hold(near, cdn); - sim.originate(origin, 7, 0); - sim.announce(origin, "local/x", 0, None); - assert!(sim.run()); - let key = qualified(LOCAL.subject, 7); - assert_eq!(next(&sim, cdn, key), Some(far)); - assert_eq!(next(&sim, near, key), Some(cdn)); - assert_eq!( - sim.shows(near, key, "local/x"), - Some(true), - "the far relay's copy is a route" - ); - - sim.release(near, cdn); - assert!(sim.run()); - assert_eq!(next(&sim, cdn, key), Some(near), "the peer took the shorter route"); - assert!(sim.route(near, key).is_none(), "which crossed the near relay"); - assert!(!sim.starved(near, key)); - assert_eq!(sim.seqno(origin), 0, "nothing failed, so nothing was asked"); - assert_eq!(sim.retained().pending, 0); - assert_eq!(sim.shows(near, 7, "local/x"), Some(true)); -} - -/// The same copy, arriving at a metric the relay's distance refuses: the peer had -/// the origin through the far relay, and falls back on the near one's route when -/// that goes. The copy is infeasible, but it extends nothing the near relay sent -/// under the key, so the peer will never move off it: the relay does not wait. -#[test] -fn a_scoped_copy_that_crossed_a_relay_is_not_waited_on() { - let (origin, near, far, cdn) = (0, 1, 2, 3); - let mut sim = Sim::new(4); - sim.join(origin, near, 5); - sim.join(origin, far, 1); - // The near relay is nearer the origin than the peer, so its route is feasible there. - sim.link(near, cdn, SimLink::cost(1), SimLink::cost(3).grant(LOCAL)); - sim.link(far, cdn, SimLink::cost(1), SimLink::cost(5).grant(LOCAL)); - sim.originate(origin, 7, 0); - sim.announce(origin, "local/x", 0, None); - assert!(sim.run()); - let key = qualified(LOCAL.subject, 7); - assert_eq!(next(&sim, cdn, key), Some(far)); - assert_eq!(sim.route(near, key).map(|route| route.metric), Some(7)); - assert_eq!(sim.shows(near, key, "local/x"), Some(true)); - - sim.disconnect(far, cdn); - assert!(sim.run()); - assert_eq!( - sim.route(cdn, key).map(|route| (route.next, route.metric)), - Some((Some(near), 8)) - ); - assert!(sim.holds(near, key) && sim.route(near, key).is_none()); - assert!( - !sim.starved(near, key), - "the peer's route stands whatever the relay does" - ); - assert_eq!(sim.seqno(origin), 0); - assert_eq!(sim.retained().pending, 0); - assert_eq!(sim.shows(near, 7, "local/x"), Some(true)); -} - -/// A relay files a scoped origin under a qualified key and sends that key on to a -/// peer, which routes it through the relay and sends it back. The copy extends what -/// the relay sent under the key, so once the relay's own route goes, it waits for -/// the peer to hear and move off, as for any identity. -#[test] -fn a_scoped_key_sent_back_through_its_relay_is_waited_on() { - let (origin, relay, peer) = (0, 1, 2); - let mut sim = Sim::new(3); - sim.link(origin, relay, SimLink::cost(1), SimLink::cost(1).grant(LOCAL)); - sim.join(relay, peer, 1); - sim.originate(origin, 7, 0); - sim.announce(origin, "local/x", 0, None); - assert!(sim.run()); - let key = qualified(LOCAL.subject, 7); - assert_eq!(next(&sim, relay, key), Some(origin)); - assert_eq!(next(&sim, peer, key), Some(relay)); - assert!(sim.holds(relay, key) && sim.holds(peer, key)); +} - sim.disconnect(origin, relay); - assert!(sim.starved(relay, key), "the peer still routes through the relay"); +/// A route that already crossed as many nodes as one may, its origin included, goes +/// no further: the next relay cannot add itself, so it refuses the path, and nothing +/// is sent on. +#[test] +fn a_route_stops_at_the_hop_limit() { + let last = MAX_HOPS + 1; + let mut sim = Sim::new(last + 1); + for node in 0..last { + sim.join(node, node + 1, 1); + } + originate(&mut sim, 0, 7); assert!(sim.run()); - assert!(!sim.holds(relay, key) && !sim.holds(peer, key)); + let full = MAX_HOPS - 1; + assert_eq!(metric(&sim, full, 7), Some(full as Metric)); + assert!(!sim.routed(full + 1, 7), "the route crossed as many nodes as one may"); + assert!(!sim.holds(last, 7), "nothing is sent on"); } /// The cost of the longest loop-free path from `from` through nodes not yet `seen`, @@ -1330,11 +423,10 @@ fn longest(sim: &Sim, from: Node, seen: &mut Vec) -> Metric { best } -/// Random meshes with the origin behind a relay deliver every message and echo in a -/// random order while links flap and relays restart and collect. A route names the -/// relays it crossed, so no relay ever selects one longer than a loop-free path: a -/// node that forgot the origin cannot take back a copy of its own old route. Without -/// the hops this finds a count to infinity within a dozen cases. Every fourth mesh +/// Random meshes with the origin behind a relay deliver every message in a random +/// order while links flap and relays restart and collect. A route names the relays +/// it crossed, so no relay ever selects one longer than a loop-free path: a node that +/// forgot the origin cannot take back a copy of its own old route. Every fourth mesh /// scopes some relays under a grant, which hands a relay its own route back under a /// qualified key: nothing may wait on that copy once the mesh is quiet. #[test] @@ -1373,11 +465,11 @@ fn sweep_no_route_outgrows_a_loop_free_path() { } // A restart is a new relay as far as hops go, which a route may cross once more. let mut bound = longest(&sim, origin, &mut Vec::new()); - originate(&mut sim, origin, 7, 0); + originate(&mut sim, origin, 7); let check = |sim: &Sim, step: usize, bound: Metric| { for node in 0..relays { - let metric = sim.route(node, 7).map_or(0, |route| route.metric); + let metric = metric(sim, node, 7).unwrap_or(0); assert!( metric <= bound, "case {case} step {step}: node {node} selected a route of {metric}, past {bound}" @@ -1406,8 +498,7 @@ fn sweep_no_route_outgrows_a_loop_free_path() { collected += 1; } } - 3 => sim.bump(origin), - 4 if case % 4 == 0 && rng.chance(25) => { + 3 if case % 4 == 0 && rng.chance(25) => { let node = rng.next(relays as u64) as usize; sim.kill(node); sim.reconnect(node); @@ -1415,12 +506,10 @@ fn sweep_no_route_outgrows_a_loop_free_path() { restarts += 1; } _ => { - let (busy, acks) = (sim.busy(), sim.busy_acks()); - let pick = rng.next((busy.len() + acks.len()).max(1) as u64) as usize; - if let Some(&(from, to)) = busy.get(pick) { + let busy = sim.busy(); + if !busy.is_empty() { + let (from, to) = busy[rng.next(busy.len() as u64) as usize]; sim.deliver(from, to); - } else if let Some(&(from, to)) = acks.get(pick - busy.len()) { - sim.deliver_ack(from, to); } } } @@ -1430,7 +519,7 @@ fn sweep_no_route_outgrows_a_loop_free_path() { check(&sim, usize::MAX, bound); // Quiet: a relay the origin still reaches routes to it and shows its claim, and - // every other one holds nothing, its distance included. No request is left. + // every other one holds nothing. Nothing is left in doubt. let mut reached = vec![origin]; let mut next = 0; while let Some(&at) = reached.get(next) { @@ -1443,29 +532,22 @@ fn sweep_no_route_outgrows_a_loop_free_path() { } cut += usize::from(reached.len() <= relays); for node in 0..relays { - for &id in sim.babel(node).rx.keys() { - assert!(!sim.starved(node, id), "case {case}: node {node} waits on {id}"); - } // A scoped link files the origin under another key past it. if scoped { continue; } match reached.contains(&node) { true => { - assert!(sim.route(node, 7).is_some(), "case {case}: node {node} lost its route"); + assert!(sim.routed(node, 7), "case {case}: node {node} lost its route"); assert_eq!(sim.shows(node, 7, ""), Some(true), "case {case}: node {node}"); } false => { assert!(!sim.holds(node, 7), "case {case}: node {node} holds a dead route"); - assert!( - !sim.remembers(node, 7), - "case {case}: node {node} keeps a dead distance" - ); assert_eq!(sim.shows(node, 7, ""), None, "case {case}: node {node}"); } } } - assert_eq!(sim.retained().pending, 0, "case {case}"); + assert_eq!(sim.pending(), 0, "case {case}"); } assert!( flaps > 0 && collected > 0 && cut > 0 && restarts > 0, @@ -1473,9 +555,10 @@ fn sweep_no_route_outgrows_a_loop_free_path() { ); } -/// A relay restarts with no memory and hands node 4 a reflection of the route 4 sent -/// before its last one. It crossed 4 and is infeasible there, so 4 waits on it like -/// any infeasible route, and its readers see the claim unavailable, never gone. +/// A relay restarts with no memory. Its neighbours hold the claim, so their readers +/// see it unavailable first, never gone at once. A hold ends once the neighbours that +/// learned the route answered, which can come before the relay's new route does; that +/// route is taken once the doubt on the relay lifts. #[test] fn a_relay_restart_keeps_the_claim_unavailable() { let origin = 6; @@ -1486,7 +569,7 @@ fn a_relay_restart_keeps_the_claim_unavailable() { sim.link(1, 5, SimLink::cost(1).priority(1), SimLink::cost(3).priority(-1)); sim.join(3, 4, 1); sim.join(4, 5, 1); - originate(&mut sim, origin, 7, 0); + originate(&mut sim, origin, 7); assert!(sim.run()); let before = sim.events.len(); @@ -1498,424 +581,9 @@ fn a_relay_restart_keeps_the_claim_unavailable() { .filter(|event| event.node == node) .map(|event| event.kind) .collect(); - assert_eq!(seen, [Kind::Unavailable, Kind::Available], "node {node}"); - } -} - -/// Every identity a rotation draws is fresh: never zero, never one still -/// declared or just retired, and within the 53 bits a lite-06 hop carries as -/// itself. The seed makes the first draw zero, and the high bits of every draw -/// set, which a small seed never reaches. -#[test] -fn rotated_identities_are_fresh_and_in_range() { - // One xorshift64 draw, and its inverse to pick the seed before a chosen draw. - fn draw(mut x: u64) -> u64 { - x ^= x << 13; - x ^= x >> 7; - x ^ x << 17 - } - fn undraw(y: u64) -> u64 { - let unshift = |y: u64, shift: &dyn Fn(u64) -> u64| (0..64).fold(y, |x, _| y ^ shift(x)); - let x = unshift(y, &|x| x << 17); - let x = unshift(x, &|x| x >> 7); - unshift(x, &|x| x << 13) - } - // The router forces its seed odd, so take the first zero draw an odd seed reaches. - let seed = (1..) - .map(|high: u64| undraw(high << 53)) - .find(|seed| seed & 1 == 1) - .expect("an odd seed"); - assert_eq!(draw(seed) & GENERATED, 0); - - let item = Item { - seed: 0, - epoch: None, - since: 0, - }; - // Eight rotations, each from aging the declared identity to its last revision. - fn check(rule: &str, mut rotate: impl FnMut() -> Id) { - let mut seen = BTreeSet::from([7]); - for _ in 0..8 { - let fresh = rotate(); - assert_ne!(fresh, 0, "{rule}"); - assert!(fresh <= GENERATED, "{rule}: {fresh:#x} is beyond a generated hop id"); - assert!(seen.insert(fresh), "{rule}: {fresh:#x} was drawn before"); - } - } - - let mut router: Router = Router::new(hop(0, 0), seed); - router.originate(7, 0); - router.announce(p("live/x"), item).unwrap(); - check("distance", || { - router.age(MAX - 1); - router.announce(p("live/x"), item).unwrap(); - router.declared().expect("declared") - }); - let mut router = path::Router::::new(hop(0, 0), seed, Doubt::DEFAULT); - router.originate(7); - router.announce(p("live/x"), item).unwrap(); - check("path", || { - router.age(MAX - 1); - router.announce(p("live/x"), item).unwrap(); - router.declared().expect("declared") - }); -} - -/// One batch may withdraw a stream's only claim and start another: the stream -/// moves to the new claim, and the route it rides on stays up throughout. -#[test] -fn a_batch_replacing_a_streams_only_claim_keeps_its_route() { - let (up, down) = (0, 1); - let item = |seed| Item { - seed, - epoch: None, - since: 1, - }; - let mut router: Router = Router::new(hop(2, 0), 1); - router.connect(up, Link::default()).unwrap(); - router.connect(down, Link::default()).unwrap(); - router.peer_interest(down, Path::empty()).unwrap(); - let interest = router.interest(); - let advert = Advert { - seqno: 0, - metric: 1, - down_only: false, - rtt: None, - hops: Arc::from([hop(0, 0)]), - }; - router.route(up, 7, Some(Reach::Finite(advert))); - router.batch(up, 7, vec![(p("live/a"), Some(item(1)))], Some(1), interest); - assert!( - router - .drain() - .iter() - .any(|out| matches!(out, Out::Route { to, reach: Some(_), .. } if *to == down)) - ); - - router.batch( - up, - 7, - vec![(p("live/a"), None), (p("live/b"), Some(item(2)))], - Some(2), - interest, - ); - let out = router.drain(); - assert!( - !out.iter() - .any(|out| matches!(out, Out::Route { to, reach: None, .. } if *to == down)), - "{out:?}" - ); - assert!(out.contains(&Out::Batch { - to: down, - id: 7, - changes: vec![(p("live/b"), Some(item(2))), (p("live/a"), None)], - floor: Some(2), - })); -} - -/// A relay sends a route only to a stream whose interest one of its claims meets, -/// and records a feasibility distance only for a route it sends. Here no stream's -/// interest meets the claim, so the relay commits to nothing it never advertised. -#[test] -fn a_route_no_interest_meets_records_no_distance() { - let (origin, a, b) = (0, 1, 2); - let mut sim = Sim::new(3); - sim.leaf(origin); - sim.watch(origin, "other/"); - sim.watch(b, "other/"); - // The relay's own interest certifies nothing over theirs. - sim.watch(a, "live/"); - sim.join(origin, a, 1); - sim.join(a, b, 1); - sim.originate(origin, 7, 0); - sim.announce(origin, "live/x", 0, None); - assert!(sim.run()); - assert_eq!(next(&sim, a, 7), Some(origin)); - assert_eq!(sim.sent(a, b, 7), None); - assert!(!sim.remembers(a, 7)); -} - -/// Two neighbors with feasible routes carry different uncertified attributes for one -/// claim: the relay shows and forwards its selected next hop's, whichever neighbor -/// sorts first. -#[test] -fn an_uncertified_claim_comes_from_the_selected_next_hop() { - let (near, far, down) = (5, 0, 1); - let item = |seed| Item { - seed, - epoch: None, - since: 1, - }; - let mut router: Router = Router::new(hop(2, 0), 1); - // Both routes stay feasible: each sender's metric is below any this relay advertises. - router - .connect( - far, - Link { - cost: 5, - ..Link::default() - }, - ) - .unwrap(); - router.connect(near, Link::default()).unwrap(); - router.connect(down, Link::default()).unwrap(); - router.peer_interest(down, Path::empty()).unwrap(); - let interest = router.interest(); - for (from, seed) in [(far, 1), (near, 2)] { - let advert = Advert { - seqno: 0, - metric: 1, - down_only: false, - rtt: None, - hops: Arc::from([hop(9, 0), hop(from, 0)]), - }; - router.route(from, 7, Some(Reach::Finite(advert))); - // No floor: the claim lives only while the stream carries it. - router.batch(from, 7, vec![(p("live/x"), Some(item(seed)))], None, interest); - } - assert_eq!(router.selected(7).and_then(|route| route.next), Some(near)); - assert_eq!(router.shown()[&(7, p("live/x"))].item, item(2)); - // Echo any end the relay sent downstream meanwhile, so it starts the route again. - let mut out = router.drain(); - router.echo(down, 7); - out.extend(router.drain()); - let last = (out.into_iter()) - .filter_map(|out| match out { - Out::Batch { to, changes, .. } if to == down => Some(changes), - _ => None, - }) - .flatten() - .rfind(|(claim, _)| *claim == p("live/x")); - assert_eq!(last, Some((p("live/x"), Some(item(2))))); -} - -/// A relay forwarding a neighbor's seqno request along its route. `close` names the -/// session that then closes, and whatever it was, nothing is left to ask for: the -/// relay keeps no request, though nothing else it knows names the identity. -fn forwarded_request_closes(close: Node) -> usize { - let (up, requester) = (0, 1); - let mut router: Router = Router::new(hop(2, 0), 1); - router.connect(up, Link::default()).unwrap(); - router.connect(requester, Link::default()).unwrap(); - let advert = Advert { - seqno: 0, - metric: 1, - down_only: false, - rtt: None, - hops: Arc::from([hop(9, 0), hop(up, 0)]), - }; - router.route(up, 7, Some(Reach::Finite(advert))); - router.request(requester, 7, 1); - assert!(router.drain().contains(&Out::Request { - to: up, - id: 7, - seqno: 1 - })); - assert_eq!(router.retained().pending, 1); - router.disconnect([close]); - router.retained().pending -} - -#[test] -fn a_forwarded_request_goes_with_its_requesters_session() { - assert_eq!(forwarded_request_closes(1), 0); -} - -#[test] -fn a_forwarded_request_goes_with_the_route_it_follows() { - assert_eq!(forwarded_request_closes(0), 0); -} - -/// A relay keeps nothing for an identity once the only session that routed it -/// closes: no state outlives what a peer told it. -#[test] -fn a_relay_keeps_nothing_once_its_only_route_closes() { - let up = 0; - let mut router: Router = Router::new(hop(2, 0), 1); - router.connect(up, Link::default()).unwrap(); - let advert = Advert { - seqno: 0, - metric: 1, - down_only: false, - rtt: None, - hops: Arc::from([hop(9, 0), hop(up, 0)]), - }; - router.route(up, 7, Some(Reach::Finite(advert))); - assert!(!router.keeps_nothing(7)); - router.disconnect([up]); - assert!(router.keeps_nothing(7)); -} - -/// A claim only ever carried uncertified, as a stream's own, goes offline when the -/// stream drops it: that proves only the stream left it, not that the origin -/// withdrew it. That holds while another certifier has yet to answer, so which -/// regions are certified is unknown. -#[test] -fn an_uncertified_claim_dropped_goes_offline_while_a_certifier_is_unanswered() { - let (answered, silent) = (0, 1); - let item = Item { - seed: 0, - epoch: None, - since: 1, - }; - let mut router: Router = Router::new(hop(2, 0), 1); - router.connect(answered, Link::default()).unwrap(); - router.connect(silent, Link::default()).unwrap(); - let interest = router.interest(); - for from in [answered, silent] { - let advert = Advert { - seqno: 0, - metric: 1, - down_only: false, - rtt: None, - hops: Arc::from([hop(9, 0), hop(from, 0)]), - }; - router.route(from, 7, Some(Reach::Finite(advert))); - } - router.batch(answered, 7, vec![(p("live/x"), Some(item))], None, interest); - assert_eq!(router.shows(7, &p("live/x")), Some(true)); - router.events(); - - router.batch(answered, 7, vec![(p("live/x"), None)], None, interest); - let kinds: Vec = router.events().into_iter().map(|event| event.kind).collect(); - assert_eq!(kinds, [Kind::Offline]); -} - -/// A finite route `from` sends, as its next hop toward an origin further on. -fn relayed(from: Node, seqno: Seqno, metric: Metric) -> Option { - Some(Reach::Finite(Advert { - seqno, - metric, - down_only: false, - rtt: None, - hops: Arc::from([hop(9, 0), hop(from, 0)]), - })) -} - -/// A claim only a held neighbor certifies is shown unavailable: a held route cannot -/// serve it. Once the selected neighbor's floor covers the claim, it is available, -/// though that batch names no claim at all. -#[test] -fn a_floor_naming_no_claim_makes_a_held_neighbors_claim_available() { - let (selected, held) = (0, 1); - let item = Item { - seed: 0, - epoch: None, - since: 1, - }; - let mut router: Router = Router::new(hop(2, 0), 1); - router.connect(selected, Link::default()).unwrap(); - router.connect(held, Link::default()).unwrap(); - let interest = router.interest(); - for from in [selected, held] { - router.route(from, 7, relayed(from, 0, 1)); - } - router.batch(held, 7, vec![(p("live/x"), Some(item))], Some(5), interest); - // The selected neighbor carries the claim, but certifies nothing yet. - router.batch(selected, 7, vec![(p("live/x"), Some(item))], None, interest); - router.route(held, 7, Some(Reach::Held)); - assert_eq!(router.selected(7).and_then(|route| route.next), Some(selected)); - assert_eq!(router.shows(7, &p("live/x")), Some(false)); - router.events(); - - router.batch(selected, 7, Vec::new(), Some(5), interest); - assert_eq!(router.shows(7, &p("live/x")), Some(true)); - let kinds: Vec = router.events().into_iter().map(|event| event.kind).collect(); - assert_eq!(kinds, [Kind::Available]); -} - -/// A relay that narrows its interest keeps a floor outside it while that floor is -/// newer than every floor inside, then lets it go, with its claims, once a floor -/// inside catches up. A stream watching wider than the relay hears those claims -/// end. The relay asked one neighbor for a sibling region, and that neighbor has -/// yet to answer the narrowed interest, so no pruning lets the region go first. -#[test] -fn a_relay_lets_go_outside_its_interest_once_a_floor_inside_catches_up() { - let (live, other, down) = (0, 1, 3); - let item = |seed| Item { - seed, - epoch: None, - since: 1, - }; - let mut router: Router = Router::new(hop(2, 0), 1); - for neighbor in [live, other, down] { - router.connect(neighbor, Link::default()).unwrap(); - } - router.asks(live, p("live")).unwrap(); - router.asks(other, p("other")).unwrap(); - router.peer_interest(down, Path::empty()).unwrap(); - let interest = router.interest(); - for from in [live, other] { - router.route(from, 7, relayed(from, 0, 1)); + assert_eq!(seen.first(), Some(&Kind::Unavailable), "node {node}: {seen:?}"); + assert_eq!(sim.shows(node, 7, ""), Some(true), "node {node}: {seen:?}"); } - router.batch(other, 7, vec![(p("other/x"), Some(item(1)))], Some(10), interest); - router.batch(live, 7, vec![(p("live/x"), Some(item(2)))], Some(3), interest); - - let interest = router.watch(p("live")); - router.batch(live, 7, vec![(p("live/x"), Some(item(2)))], Some(3), interest); - let view = router.view(7).expect("a view"); - assert_eq!(view.floors, BTreeMap::from([(p("live"), 3), (p("other"), 10)])); - assert!(view.items.contains_key(&p("other/x"))); - router.drain(); - - router.batch(live, 7, vec![(p("live/y"), Some(item(3)))], Some(11), interest); - let view = router.view(7).expect("a view"); - assert_eq!(view.floors, BTreeMap::from([(p("live"), 11)])); - assert_eq!( - view.items.keys().cloned().collect::>(), - [p("live/x"), p("live/y")] - ); - let changes: Vec<(Path, Option)> = (router.drain().into_iter()) - .filter_map(|out| match out { - Out::Batch { to, changes, .. } if to == down => Some(changes), - _ => None, - }) - .flatten() - .collect(); - assert_eq!(changes, [(p("live/y"), Some(item(3))), (p("other/x"), None)]); -} - -/// A relay forwarding a request for a newer seqno than its own starvation needs -/// keeps asking for the newer one: a route at the seqno it needs ends its -/// starvation but answers nothing its requester asked for. -#[test] -fn a_starving_relay_keeps_forwarding_the_newer_seqno_its_requester_asked_for() { - let (up, requester) = (0, 1); - let item = Item { - seed: 0, - epoch: None, - since: 1, - }; - let mut router: Router = Router::new(hop(2, 0), 1); - router.connect(up, Link::default()).unwrap(); - router.connect(requester, Link::default()).unwrap(); - router.peer_interest(requester, Path::empty()).unwrap(); - let interest = router.interest(); - router.route(up, 7, relayed(up, 0, 1)); - // A claim meets the requester's interest, so the relay sends it the route and - // records a feasibility distance of seqno 0. - router.batch(up, 7, vec![(p("live/x"), Some(item))], Some(1), interest); - assert!(router.remembers(7)); - router.request(requester, 7, 5); - assert!(router.drain().contains(&Out::Request { - to: up, - id: 7, - seqno: 5 - })); - - // A longer route at the same seqno is infeasible: the relay starves, needing only seqno 1. - router.route(up, 7, relayed(up, 0, 5)); - assert!(router.starved(7)); - let requests: Vec<_> = (router.drain().into_iter()) - .filter(|out| matches!(out, Out::Request { .. })) - .collect(); - assert_eq!(requests, [], "the request for seqno 5 already covers it"); - - router.route(up, 7, relayed(up, 1, 5)); - assert_eq!(router.selected(7).map(|route| route.seqno), Some(1)); - assert_eq!(router.retained().pending, 1, "seqno 5 is still owed to the requester"); - router.route(up, 7, relayed(up, 5, 5)); - assert_eq!(router.retained().pending, 0); } /// A path route the next hop offers again at a cost that overflows with the link's @@ -2030,29 +698,114 @@ fn a_dead_alternative_does_not_keep_a_lost_route_held() { assert!(router.shown().is_empty()); } +/// Every identity a rotation draws is fresh: never zero, never one still +/// declared or just retired, and within the 53 bits a lite-06 hop carries as +/// itself. The seed makes the first draw zero, and the high bits of every draw +/// set, which a small seed never reaches. +#[test] +fn rotated_identities_are_fresh_and_in_range() { + // One xorshift64 draw, and its inverse to pick the seed before a chosen draw. + fn draw(mut x: u64) -> u64 { + x ^= x << 13; + x ^= x >> 7; + x ^ x << 17 + } + fn undraw(y: u64) -> u64 { + let unshift = |y: u64, shift: &dyn Fn(u64) -> u64| (0..64).fold(y, |x, _| y ^ shift(x)); + let x = unshift(y, &|x| x << 17); + let x = unshift(x, &|x| x >> 7); + unshift(x, &|x| x << 13) + } + // The router forces its seed odd, so take the first zero draw an odd seed reaches. + let seed = (1..) + .map(|high: u64| undraw(high << 53)) + .find(|seed| seed & 1 == 1) + .expect("an odd seed"); + assert_eq!(draw(seed) & GENERATED, 0); + + let item = Item { + seed: 0, + epoch: None, + since: 0, + }; + let mut router = path::Router::::new(99, seed, Doubt::DEFAULT); + router.originate(7); + router.announce(p("live/x"), item).unwrap(); + // Eight rotations, each from aging the declared identity to its last revision. + let mut seen = BTreeSet::from([7]); + for _ in 0..8 { + router.age(MAX - 1); + router.announce(p("live/x"), item).unwrap(); + let fresh = router.declared().expect("declared"); + assert_ne!(fresh, 0); + assert!(fresh <= GENERATED, "{fresh:#x} is beyond a generated hop id"); + assert!(seen.insert(fresh), "{fresh:#x} was drawn before"); + } +} + +/// One batch may withdraw a stream's only claim and start another: the stream +/// moves to the new claim, and the route it rides on stays up throughout. +#[test] +fn a_batch_replacing_a_streams_only_claim_keeps_its_route() { + let (up, down) = (0, 1); + let item = |seed| Item { + seed, + epoch: None, + since: 1, + }; + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + router.connect(up, 9, Link::default()).unwrap(); + router.edited(up); + router.connect(down, 8, Link::default()).unwrap(); + reask(&mut router, down, &[""]).unwrap(); + let hops = [(7, 0), (9, 1)]; + router.route(up, 7, p(""), Some(path::Reach::Finite(along(&hops, 1)))); + router.batch(up, 7, p(""), vec![(p("live/a"), Some(item(1)))], Some(1)); + assert!( + (router.drain().iter()).any(|out| matches!(out, path::Out::Route { to, reach: Some(_), .. } if *to == down)) + ); + + router.batch( + up, + 7, + p(""), + vec![(p("live/a"), None), (p("live/b"), Some(item(2)))], + Some(2), + ); + let out = router.drain(); + assert!( + !out.iter() + .any(|out| matches!(out, path::Out::Route { to, reach: None, .. } if *to == down)), + "{out:?}" + ); + let changes: Vec<_> = (out.into_iter()) + .filter_map(|out| match out { + path::Out::Batch { to, changes, .. } if to == down => Some(changes), + _ => None, + }) + .flatten() + .collect(); + assert!(changes.contains(&(p("live/b"), Some(item(2)))), "{changes:?}"); + assert!(changes.contains(&(p("live/a"), None)), "{changes:?}"); +} + /// A driver hears of every identity whose routes were recomputed, even when no shown /// claim changed, so it can serve those claims through the routes it has now. #[test] fn a_route_change_alone_names_its_identity() { let up = 0; - let mut router: Router = Router::new(hop(2, 0), 1); - router.connect(up, Link::default()).unwrap(); - let interest = router.interest(); - let advert = Advert { - seqno: 0, - metric: 1, - down_only: false, - rtt: None, - hops: Arc::from([hop(0, 0)]), - }; - router.route(up, 7, Some(Reach::Finite(advert.clone()))); + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + router.connect(up, 9, Link::default()).unwrap(); + router.edited(up); + let hops = [(7, 0), (9, 1)]; + router.route(up, 7, p(""), Some(path::Reach::Finite(along(&hops, 1)))); let item = Item { seed: 1, epoch: None, since: 1, }; - router.batch(up, 7, vec![(p("live/a"), Some(item))], Some(1), interest); + router.batch(up, 7, p(""), vec![(p("live/a"), Some(item))], Some(1)); assert!(router.changed()[&7].contains(&p("live/a"))); - router.route(up, 7, Some(Reach::Finite(Advert { metric: 2, ..advert }))); + router.route(up, 7, p(""), Some(path::Reach::Finite(along(&hops, 2)))); assert_eq!(router.changed(), BTreeMap::from([(7, BTreeSet::new())])); } diff --git a/rs/moq-net/src/route/tests/rtt.rs b/rs/moq-net/src/route/tests/rtt.rs index c5deb080de5..903316971cb 100644 --- a/rs/moq-net/src/route/tests/rtt.rs +++ b/rs/moq-net/src/route/tests/rtt.rs @@ -6,7 +6,7 @@ use std::time::Duration; use super::*; fn originate(sim: &mut Sim, node: Node, id: Id) { - sim.originate(node, id, 0); + sim.originate(node, id); sim.announce(node, "", 0, None); } @@ -16,12 +16,19 @@ fn ms(ms: u64) -> Option { /// The RTT `node` advertises to `peer` for `id`. fn advertised(sim: &Sim, node: Node, peer: Node, id: Id) -> Option { - match sim.sent(node, peer, id) { - Some(Reach::Finite(advert)) => advert.rtt, - other => panic!("{node} sends {peer} {other:?}"), + match sim.offer(node, peer, id, &Path::empty()) { + Some(offer) => offer.rtt, + None => panic!("{node} sends {peer} no route for {id}"), } } +/// The next hop, metric, and RTT of `node`'s route for `id`. +fn route(sim: &Sim, node: Node, id: Id) -> Option<(Option, Metric, Option)> { + let route = sim.path(node).route_for(id, &Path::empty())?; + let offer = route.offer()?; + Some((route.next(), offer.metric, offer.rtt)) +} + /// A chain origin, a, b, c: each relay measures its link toward the origin. fn chain() -> Sim { let (origin, a, b, c) = (0, 1, 2, 3); @@ -44,8 +51,7 @@ fn each_relay_adds_its_link_to_the_sum() { assert_eq!(advertised(&sim, origin, a, 7), ms(0), "the origin's own route"); assert_eq!(advertised(&sim, a, b, 7), ms(10)); assert_eq!(advertised(&sim, b, c, 7), ms(30)); - let routes = sim.babel(c).routes(7); - assert_eq!(routes, [(b, 3, ms(60))]); + assert_eq!(route(&sim, c, 7), Some((Some(b), 3, ms(60)))); } /// A shift on one link sends one ROUTE_UPDATE on each stream carrying the route from @@ -77,7 +83,7 @@ fn an_unknown_link_ends_the_sum() { assert!(sim.run()); assert_eq!(advertised(&sim, a, b, 7), ms(10)); assert_eq!(advertised(&sim, b, c, 7), None); - assert_eq!(sim.babel(c).routes(7)[0].2, None); + assert_eq!(route(&sim, c, 7), Some((Some(b), 3, None))); } /// The metric selects; a far lower RTT through a costlier link changes nothing. @@ -93,7 +99,7 @@ fn rtt_never_selects_a_route() { sim.rtt(fast, origin, Some(1)); originate(&mut sim, origin, 7); assert!(sim.run()); - assert_eq!(sim.route(edge, 7).and_then(|route| route.next), Some(cheap)); + assert_eq!(sim.next_hop(edge, 7), Some(cheap)); assert_eq!(advertised(&sim, cheap, edge, 7), ms(200)); } diff --git a/rs/moq-net/src/route/tests/scope.rs b/rs/moq-net/src/route/tests/scope.rs index aaeca087d71..58bf92001b2 100644 --- a/rs/moq-net/src/route/tests/scope.rs +++ b/rs/moq-net/src/route/tests/scope.rs @@ -253,7 +253,7 @@ fn follower(prefixes: usize) -> (Sim, Node) { sim.link(2 + spoke, hub, SimLink::cost(1).grant(grant), SimLink::cost(1)); sim.join(2 + spoke, follower, 1); } - sim.originate(origin, 7, 0); + sim.originate(origin, 7); for spoke in 0..prefixes { sim.announce(origin, &format!("live/{spoke}/x"), 0, None); } @@ -481,7 +481,7 @@ fn another_grant_does_not_turn_a_raw_route_into_a_qualified_child() { } sim.join(origin, relay, 2); sim.join(origin, second, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "b/x", 0, None); assert!(sim.run()); let key = qualified("ga", 7); @@ -503,7 +503,7 @@ fn widening_past_a_childs_narrower_interest_releases_the_old_scope() { sim.join(relay, child, 1); sim.watch(relay, "a"); sim.watch(child, "a/x"); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "a/x", 0, None); sim.announce(origin, "a/y", 0, None); assert!(sim.run()); @@ -535,7 +535,7 @@ fn overlapping_scopes_do_not_oscillate_between_covering_paths() { for node in [2, 3, 4] { sim.watch(node, "b"); } - sim.originate(5, 7, 0); + sim.originate(5, 7); sim.announce(5, "b/x", 0, None); assert!(sim.run(), "{}", mesh(&sim)); for node in 0..5 { @@ -558,7 +558,7 @@ fn widening_a_relay_does_not_oscillate_its_former_prefix() { sim.join(4, 1, 1); sim.watch(0, "a"); sim.watch(2, "a"); - sim.originate(4, 7, 0); + sim.originate(4, 7); sim.announce(4, "a/x", 0, None); assert!(sim.run()); @@ -587,7 +587,7 @@ fn a_repathed_scoped_route_keeps_its_original_key() { }), SimLink::cost(1), ); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "a/x", 0, None); assert!(sim.run()); let key = qualified("ga", 7); @@ -613,7 +613,7 @@ fn qualification_does_not_turn_another_key_into_a_child() { sim.link(relay, raw, SimLink::cost(4).grant(grant), SimLink::cost(1)); sim.link(alternate, origin, SimLink::cost(1).grant(grant), SimLink::cost(1)); sim.join(alternate, relay, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "a/x", 0, None); assert!(sim.run()); let key = qualified("ga", 7); @@ -639,7 +639,7 @@ fn an_unwatched_grant_retires_its_previous_stream() { }), SimLink::cost(1), ); - sim.originate(0, 7, 0); + sim.originate(0, 7); sim.announce(0, "b/x", 0, None); assert!(sim.run()); let key = qualified("ga", 7); @@ -666,7 +666,7 @@ fn a_cancelled_stream_cannot_restore_a_route() { }), SimLink::cost(1), ); - sim.originate(0, 7, 0); + sim.originate(0, 7); sim.announce(0, "b/x", 0, None); // The original snapshot is still in flight when the grant is cancelled. sim.watch(1, "a"); @@ -875,7 +875,7 @@ fn sweep_scoped_meshes_show_what_the_rule_says() { } } } - sim.originate(origin, 7, 0); + sim.originate(origin, 7); for claim in CLAIMS { if rng.chance(40) { sim.announce(origin, claim, rng.next(3), None); @@ -964,7 +964,7 @@ fn holds_never_close_a_cycle() { sim.join(origin, hub, 1); sim.watch(0, "a/"); sim.watch(1, "a/"); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "a/x", 0, None); assert!(sim.run()); quiet(&sim, origin, "connected"); @@ -980,6 +980,35 @@ fn holds_never_close_a_cycle() { } } +/// A peer under a scoped grant names itself as it likes. Declaring a core relay's Node +/// ID, its GOAWAY ranks only the routes it sent as draining, and its close doubts no +/// node, so the relay's own routes carry on as they were. +#[test] +fn a_scoped_peers_goaway_and_close_touch_only_its_routes() { + let mut router = path::Router::::new(99, 1, Doubt::DEFAULT); + router.connect(1, 1, Link::default()).unwrap(); + router.edited(1); + let scoped = Link { + scope: Some(Scope { + regions: vec![p("a")], + qualify: super::super::qualifier([1; 32]), + }), + ..Link::default() + }; + router.connect(2, 1, scoped).unwrap(); + router.edited(2); + router.route(1, 7, p(""), Some(offered(1, 0))); + router.batch(1, 7, p(""), vec![(p("a"), Some(1))], Some(1)); + let draining = |router: &path::Router| router.routes(7).next().unwrap().1.offer().unwrap().draining; + assert!(!draining(&router)); + + router.goaway(2, Instant::ZERO); + assert!(!draining(&router), "the relay's route does not drain"); + router.disconnect([2], Instant::ZERO); + assert_eq!(router.retained().doubt, 0, "no node is doubted"); + assert!(router.shown()[&(7, p("a"))].available); +} + /// An offer through a closed session is dead until its sender says what became of /// it, even once the doubt lifts, and a stale repeat of it stays dead: the hop is /// remembered while any heard route names it. @@ -1374,7 +1403,7 @@ fn a_qualified_reflection_stays_live_through_a_first_dead() { sim.join(other, scoped, 5); sim.join(origin, spare, 5); sim.join(spare, relay, 5); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "a/x", 0, None); assert!(sim.run()); let key = qualified("ga", 7); diff --git a/rs/moq-net/src/route/tests/search.rs b/rs/moq-net/src/route/tests/search.rs index 6168fe57533..8d677eeab90 100644 --- a/rs/moq-net/src/route/tests/search.rs +++ b/rs/moq-net/src/route/tests/search.rs @@ -16,6 +16,7 @@ use std::{ panic::{AssertUnwindSafe, catch_unwind}, }; +use super::super::view::within_any; use super::*; /// How far the search may go, as `RouteHandoff.cfg` bounds the model. @@ -35,8 +36,6 @@ pub(super) struct Bounds { #[derive(Clone, Debug, PartialEq, Eq)] pub(super) enum Step { Deliver(Node, Node), - /// An echoed ROUTE_END, under the distance rule. - Ack(Node, Node), Announce { origin: Node, claim: &'static str, @@ -56,7 +55,6 @@ impl fmt::Display for Step { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match self { Self::Deliver(from, to) => write!(f, "sim.deliver({from}, {to});"), - Self::Ack(from, to) => write!(f, "sim.deliver_ack({from}, {to});"), Self::Announce { origin, claim, seed } => write!(f, "sim.announce({origin}, {claim:?}, {seed}, None);"), Self::Unannounce { origin, claim } => write!(f, "sim.unannounce({origin}, {claim:?});"), Self::Connect(a, b) => write!(f, "sim.connect({a}, {b});"), @@ -124,17 +122,14 @@ impl World { fn fingerprint(&self) -> u64 { let mut hasher = DefaultHasher::new(); let sim = &self.sim; - for relay in &sim.nodes { - match relay { - Relay::Distance(router) => router.fingerprint(&mut hasher), - Relay::Path(router) => router.fingerprint(sim.now, &mut hasher), - } + for router in &sim.nodes { + router.fingerprint(sim.now, &mut hasher); } digest( &mut hasher, &(&sim.watch, &sim.restarts, &sim.declared, &sim.up, &sim.held), ); - digest(&mut hasher, &(&sim.queues, &sim.acks, &sim.history)); + digest(&mut hasher, &(&sim.queues, &sim.history)); digest( &mut hasher, &(self.changes, self.flaps, &self.ended, self.burst, self.churn), @@ -143,18 +138,15 @@ impl World { } fn quiet(&self) -> bool { - self.sim.busy().is_empty() && self.sim.busy_acks().is_empty() + self.sim.busy().is_empty() } - /// Messages and echoes queued, by direction. + /// Messages queued, by direction. fn queued(&self) -> BTreeMap<(Node, Node), usize> { let mut out = BTreeMap::new(); for (key, queue) in &self.sim.queues { *out.entry(*key).or_default() += queue.len(); } - for (key, queue) in &self.sim.acks { - *out.entry(*key).or_default() += queue.len(); - } out } } @@ -252,7 +244,6 @@ impl Search<'_> { .into_iter() .map(|(from, to)| Step::Deliver(from, to)) .collect(); - out.extend(sim.busy_acks().into_iter().map(|(from, to)| Step::Ack(from, to))); for node in 0..sim.nodes.len() { if sim.nodes[node].deadline().is_some() && self.patient(sim, node) { out.push(Step::Fire(node)); @@ -301,9 +292,7 @@ impl Search<'_> { if !self.bounds.patient { return true; } - let Relay::Path(router) = &sim.nodes[node] else { - return true; - }; + let router = &sim.nodes[node]; let at = sim.now.max(router.deadline().expect("a timer")); let due = router.due(at); !sim.queues.values().flatten().any(|msg| match msg { @@ -331,7 +320,7 @@ impl Search<'_> { for (key, now) in world.queued() { let was = queued.get(&key).copied().unwrap_or(0); let consumed = match step { - Step::Deliver(from, to) | Step::Ack(from, to) if (*from, *to) == key => 1, + Step::Deliver(from, to) if (*from, *to) == key => 1, _ => 0, }; sent += now - (was - consumed); @@ -370,7 +359,6 @@ impl Search<'_> { let sim = &mut world.sim; match *step { Step::Deliver(from, to) => assert!(sim.deliver(from, to)), - Step::Ack(from, to) => assert!(sim.deliver_ack(from, to)), Step::Announce { origin, claim, seed } => { sim.announce(origin, claim, seed, None); world.changes += 1; @@ -399,7 +387,7 @@ impl Search<'_> { Step::Fire(_) => true, Step::Deliver(from, to) => match (&sim.nodes[to], sim.queues[&(from, to)].front()) { ( - Relay::Path(router), + router, Some(Msg::Path { id, prefix, @@ -413,13 +401,6 @@ impl Search<'_> { router.hears(key, prefix, from) } (_, Some(Msg::Dead(path::Dead::Draining { .. }))) => true, - ( - Relay::Distance(router), - Some(Msg::Route { - id, - reach: Some(Reach::Finite(_)), - }), - ) => router.rx.get(id).is_some_and(|routes| routes.contains_key(&from)), _ => false, }, _ => false, @@ -486,7 +467,7 @@ impl Search<'_> { /// at that revision as a stream answering the route's prefix presents them. fn offered(&self, sim: &Sim) -> Result<(), String> { for (node, relay) in sim.nodes.iter().enumerate() { - let Relay::Path(router) = relay else { continue }; + let router = relay; for ((key, prefix), view) in router.heard() { let Some(id) = sim.source(*key) else { continue }; for (region, &rev) in &view.floors { @@ -521,38 +502,19 @@ impl Search<'_> { /// Where a request for `claim` of `key` from `node`'s reader goes. fn ask(&self, sim: &Sim, node: Node, key: Id, claim: &Path) -> Ask { let mut chain = vec![node]; - match &sim.nodes[node] { - Relay::Distance(_) => { - let mut at = node; - while let Some(next) = sim.next_hop(at, key) { - if chain.contains(&next) { - chain.push(next); - return Ask::Loop(chain); - } + let (mut at, mut key, mut from) = (node, key, None); + loop { + match sim.path(at).forward(key, claim, from.take()) { + None => return Ask::Stuck, + Some(path::Forward::Here) => return Ask::Origin, + Some(path::Forward::Next { next, named, prefix }) => { + let looped = chain.contains(&next); chain.push(next); - at = next; - } - match at == self.origin { - true => Ask::Origin, - false => Ask::Stuck, - } - } - Relay::Path(_) => { - let (mut at, mut key, mut from) = (node, key, None); - loop { - match sim.path(at).forward(key, claim, from.take()) { - None => return Ask::Stuck, - Some(path::Forward::Here) => return Ask::Origin, - Some(path::Forward::Next { next, named, prefix }) => { - let looped = chain.contains(&next); - chain.push(next); - if looped { - return Ask::Loop(chain); - } - from = Some((at, prefix)); - (at, key) = (next, named); - } + if looped { + return Ask::Loop(chain); } + from = Some((at, prefix)); + (at, key) = (next, named); } } } @@ -597,15 +559,12 @@ impl Search<'_> { )); } for (key, _) in shown { - if sim.paths() && self.ask(sim, node, key, &claim) != Ask::Origin { + if self.ask(sim, node, key, &claim) != Ask::Origin { return Err(format!("a request for {claim} from node {node} stops short")); } } } } - if !sim.paths() { - return Ok(()); - } for node in (0..sim.nodes.len()).filter(|&node| node != self.origin) { let router = sim.path(node); for key in self.keys(sim) { @@ -645,8 +604,9 @@ impl Sim { /// The newest floor `node`'s kept views of the publisher hold over a region /// that covers the claim or lies within it. fn floor(&self, node: Node, (id, claim): &(Id, Path)) -> Option { - (self.nodes[node].views().into_iter()) - .filter(|(view_id, _)| view_id == id) + self.nodes[node] + .views() + .filter(|((view_id, _), _)| view_id == id) .flat_map(|(_, view)| &view.floors) .filter(|(region, _)| within(claim, region) || within(region, claim)) .map(|(_, &rev)| rev) @@ -662,7 +622,7 @@ fn triangle(dear: Metric, watch: &str, grant: Option) -> Mesh { // The model starts with every interest applied, so the sessions' asks are // answered before there is anything to route. assert!(mesh.sim.run()); - mesh.sim.originate(mesh.origin, mesh.id, 0); + mesh.sim.originate(mesh.origin, mesh.id); mesh } @@ -714,7 +674,7 @@ const TWO: Bounds = Bounds { #[ignore = "Nightly: about ten seconds in release"] fn every_order_from_the_first_asks_keeps_the_rules() { let mut mesh = unanswered(1, "", None); - mesh.sim.originate(mesh.origin, mesh.id, 0); + mesh.sim.originate(mesh.origin, mesh.id); run(mesh, &ONE); } @@ -730,7 +690,7 @@ fn every_order_behind_a_relay_keeps_the_rules() { // The model starts with every interest applied, so the sessions' asks are // answered before there is anything to route. assert!(sim.run()); - sim.originate(0, 7, 0); + sim.originate(0, 7); run(Mesh { sim, origin: 0, id: 7 }, &ONE); } @@ -779,22 +739,22 @@ fn every_order_under_a_scoped_grant_keeps_the_rules() { #[test] #[ignore = "Nightly"] fn every_order_past_a_narrowed_relay_shows_what_a_path_carries() { - run(blind(Rule::PATH), &BLIND); + run(blind(), &BLIND); } /// `a_relay_behind_a_scoped_neighbour_is_blind`'s mesh settled with every relay /// asking for everything, and then the narrow relay asking for `a` alone, with /// nothing delivered since. -fn blind(rule: Rule) -> Mesh { +fn blind() -> Mesh { let (origin, narrow, wide, far, relay, tail) = (0, 1, 2, 3, 4, 5); - let mut sim = Sim::with(rule, 6); + let mut sim = Sim::with(Rule::PATH, 6); sim.join(origin, narrow, 1); sim.join(narrow, relay, 1); sim.join(origin, far, 1); sim.join(far, wide, 1); sim.join(wide, relay, 1); sim.join(relay, tail, 1); - sim.originate(origin, 7, 0); + sim.originate(origin, 7); sim.announce(origin, "a/x", 0, None); sim.announce(origin, "b/x", 0, None); assert!(sim.run()); @@ -809,12 +769,3 @@ const BLIND: Bounds = Bounds { flaps: 0, patient: true, }; - -/// The search's self-check: on the distance rule, which the path rule replaces, -/// it finds the blind relay `residual.rs` pins. -#[test] -#[cfg_attr(miri, ignore = "searches thousands of states")] -fn the_search_finds_the_blind_relay() { - let failure = search(blind(Rule::Distance), &BLIND).expect_err("the distance rule is blind"); - assert!(failure.rule.contains("node 4 shows b/x as []"), "{failure}"); -} diff --git a/rs/moq-net/src/route/view.rs b/rs/moq-net/src/route/view.rs index 0e6ed5c099c..d41ad7e3dfd 100644 --- a/rs/moq-net/src/route/view.rs +++ b/rs/moq-net/src/route/view.rs @@ -57,14 +57,6 @@ impl View { } } - /// The view presented to the union of `regions`. - #[cfg(test)] - pub fn present_all(&self, regions: &[Path]) -> Self { - let mut out = Self::default(); - out.represent(self, regions, self.items.keys()); - out - } - /// Bring this view of `raw`, presented to the union of `regions`, up to date once /// `claims` of `raw` changed and its floors were replaced. /// @@ -99,126 +91,12 @@ impl View { } } - /// What the view certifies within `prefix`: the view as presented there, less - /// the claims no kept floor certifies. - pub fn restrict(&self, prefix: &Path) -> Self { - let mut out = self.present(prefix); - let floors = &out.floors; - out.items.retain(|claim, _| above(floors, claim).next().is_some()); - out - } - - /// The view restricted to the union of `regions`, as merging each region's - /// restriction would leave it, in one pass. - /// - /// A region within another adds nothing the wider one's restriction lacks: the - /// floors over it narrow to the wider region, where they certify the claim at the - /// region itself at its own path. So only the widest regions count, and those are - /// disjoint, so their restrictions merge as a union. A floor a wider one at least - /// as new covers then goes, as [`Self::merge`] drops it. - pub fn restrict_all(&self, regions: &BTreeSet) -> Self { - let mut out = Self::default(); - for region in regions { - let wider = heads(region).filter(|&head| head != region.as_str()); - if wider.into_iter().any(|head| regions.contains(head)) { - continue; - } - let Self { floors, items } = self.restrict(region); - out.floors.extend(floors); - out.items.extend(items); - } - settle(&mut out.floors); - out - } - /// The newest revision this view knows `claim` at, if a region certifies it. + #[cfg(test)] pub fn as_of(&self, claim: &Path) -> Option { above(&self.floors, claim).map(|(_, &floor)| floor).max() } - /// The claims no floor certifies. - pub fn uncertified(&self) -> impl Iterator { - self.items.iter().filter(|(claim, _)| self.as_of(claim).is_none()) - } - - /// The single revision the view certifies all of `prefix` at, if one floor - /// covers it and nothing within it is known at another revision. - pub fn floor(&self, prefix: &Path) -> Option { - match self.floors.iter().next() { - Some((region, &rev)) if self.floors.len() == 1 && region == prefix => Some(rev), - _ => None, - } - } - - /// Take `other`'s state for every claim it knows at a newer revision, then - /// raise the floors and drop any region a wider one at least as new covers. - /// An uncertified claim of `other` is known at no revision, so it is not taken. - /// Returns the claims whose state changed. - pub fn merge(&mut self, other: &Self) -> Vec { - let claims: BTreeSet = self.items.keys().chain(other.items.keys()).cloned().collect(); - self.merge_claims(other, &claims) - } - - /// Every floor of `other` lies within a newer one here, so `other` knows no claim - /// at a newer revision and merging it changes nothing. - fn outdates(&self, other: &Self) -> bool { - (other.floors.iter()).all(|(region, &floor)| above(&self.floors, region).any(|(_, &newer)| newer > floor)) - } - - /// [`Self::merge`], considering only `claims`: the same once they include every - /// claim whose state differs between the two views, since a claim held alike is - /// never taken. - pub fn merge_claims<'a>(&mut self, other: &Self, claims: impl IntoIterator) -> Vec { - // A lagging view costs nothing to merge, however far it lags. - if self.outdates(other) { - return Vec::new(); - } - let mut changed = Vec::new(); - for claim in claims { - let (theirs, ours) = (other.as_of(claim), self.as_of(claim)); - // At one revision, a claim exactly on a wider region outranks the claim - // above it that a floor on this path presented here. - let exact = theirs == ours && self.floors.contains_key(claim) && !other.floors.contains_key(claim); - if (theirs > ours || exact) && self.items.get(claim) != other.items.get(claim) { - changed.push(claim.clone()); - } - } - for claim in &changed { - match other.items.get(claim) { - Some(item) => self.items.insert(claim.clone(), item.clone()), - None => self.items.remove(claim), - }; - } - for (region, &floor) in &other.floors { - let entry = self.floors.entry(region.clone()).or_insert(floor); - *entry = (*entry).max(floor); - } - settle(&mut self.floors); - changed - } - - /// What a node keeps of its view once its interest is `watch`. - /// - /// A floor outside the interest may certify a claim above both at a newer - /// revision than any floor inside, so it stays, with what it certifies, - /// until a floor inside catches up; dropping it would leave that claim - /// known only at an older revision than the item it holds. Once no - /// neighbor certifies it, pruning lets it go. - pub fn retire(&self, watch: &Path) -> Self { - let mut kept = self.restrict(watch); - let newest = kept.floors.values().max().copied(); - for (region, &floor) in &self.floors { - if Some(floor) > newest && !kept.floors.contains_key(region) { - kept.floors.insert(region.clone(), floor); - let exact = self.items.get_key_value(region); - for (claim, item) in exact.into_iter().chain(below(&self.items, region)) { - kept.items.insert(claim.clone(), item.clone()); - } - } - } - kept - } - /// The changes that move a receiver from `old` to `self`: every changed or /// added claim with its attributes, and every removed claim with `None`. pub fn diff(&self, old: &Self) -> Vec<(Path, Option)> { @@ -364,18 +242,6 @@ fn heads(path: &Path) -> impl Iterator { (std::iter::once(0).chain(ends).chain((!s.is_empty()).then_some(s.len()))).map(move |end| &s[..end]) } -/// Drop each region a wider one at least as new covers: what it certifies, the -/// wider one already does. -fn settle(floors: &mut BTreeMap) { - let covered: Vec = (floors.iter()) - .filter(|&(region, &floor)| above(floors, region).any(|(wider, &newer)| wider != region && newer >= floor)) - .map(|(region, _)| region.clone()) - .collect(); - for region in covered { - floors.remove(®ion); - } -} - #[cfg(test)] mod tests { use super::*; @@ -396,7 +262,6 @@ mod tests { let v = view(&[("", 5)], &[("live/a/x", 1), ("live", 2), ("other/x", 3)]); let r = v.present(&p("live/a")); assert_eq!(r, view(&[("live/a", 5)], &[("live/a/x", 1), ("live/a", 2)])); - assert_eq!(r.floor(&p("live/a")), Some(5)); } #[test] @@ -408,70 +273,12 @@ mod tests { assert_eq!(v.present(&p("live/a")), view(&[("live/a", 5)], &[("live/a", 1)])); } - #[test] - fn restrict_moves_a_certified_covering_claim_and_drops_an_uncertified_one() { - let v = view(&[("", 5)], &[("live/a/x", 1), ("live", 2)]); - assert_eq!( - v.restrict(&p("live/a")), - view(&[("live/a", 5)], &[("live/a", 2), ("live/a/x", 1)]) - ); - // The floor proves no claim covers its region, so the claim above is stale. - let v = view(&[("live/a", 5)], &[("live/a/x", 1), ("live", 2)]); - assert_eq!(v.restrict(&p("live/a/y")), view(&[("live/a/y", 5)], &[])); - } - - #[test] - fn a_wider_floor_at_one_revision_replaces_a_claim_moved_onto_a_narrower_one() { - let mut a = view(&[("live/b/z", 5)], &[("live/b/z", 1)]); - a.merge(&view(&[("live/b", 5)], &[("live/b", 1)])); - assert_eq!(a, view(&[("live/b", 5)], &[("live/b", 1)])); - } - - #[test] - fn retire_keeps_an_outside_floor_only_while_it_is_newer_than_every_floor_inside() { - let v = view( - &[("live", 3), ("other/a", 5), ("other/b", 3), ("other/c", 2)], - &[("live/x", 1), ("other/a/x", 2), ("other/b/x", 3), ("other/c/x", 4)], - ); - assert_eq!( - v.retire(&p("live")), - view(&[("live", 3), ("other/a", 5)], &[("live/x", 1), ("other/a/x", 2)]) - ); - } - #[test] fn a_floor_certifies_only_what_lies_within_it() { let v = view(&[("live/a", 5)], &[("live/a", 1), ("live/a/x", 2), ("live", 3)]); assert_eq!(v.as_of(&p("live/a")), Some(5)); assert_eq!(v.as_of(&p("live/a/x")), Some(5)); assert_eq!(v.as_of(&p("live")), None); - assert_eq!( - v.uncertified().map(|(claim, _)| claim.as_str()).collect::>(), - ["live"] - ); - // Two floors certify no single revision. - let v = view(&[("live", 5), ("live/a", 6)], &[]); - assert_eq!(v.floor(&p("live")), None); - } - - #[test] - fn merge_keeps_the_newer_knowledge_per_claim() { - let mut a = view(&[("live", 3)], &[("live/x", 1)]); - let b = view(&[("live", 4)], &[]); - a.merge(&b); - assert_eq!(a, view(&[("live", 4)], &[])); - - // An older view revives nothing. - let c = view(&[("live", 2)], &[("live/x", 1)]); - a.merge(&c); - assert_eq!(a, view(&[("live", 4)], &[])); - } - - #[test] - fn merge_drops_a_region_a_wider_newer_floor_covers() { - let mut a = view(&[("live/a", 3)], &[]); - a.merge(&view(&[("", 4)], &[])); - assert_eq!(a.floors, view(&[("", 4)], &[]).floors); } #[test] @@ -487,16 +294,6 @@ mod tests { assert_eq!(got, gone); } - #[test] - fn restrict_all_ignores_a_region_within_another() { - // A newer floor over `live` but not `live/a`, and an older one exactly there: - // restricting to `live/a` alone presents `live`'s claim at `live/a`, which the - // wider region knows is no claim of its own. - let v = view(&[("live", 5), ("live/a", 3)], &[("live", 1)]); - let wide = view(&[("live", 5)], &[("live", 1)]); - assert_eq!(v.restrict_all(&BTreeSet::from([p("live/a"), p("live")])), wide); - } - /// Random views over paths that share segments and string prefixes (`a` and `ab`). #[test] fn lookups_match_a_scan_of_every_floor() { @@ -533,15 +330,6 @@ mod tests { } assert_eq!(narrow(&v.floors, &path), narrowed); } - let regions: BTreeSet = PATHS.iter().filter(|_| next(3) == 0).map(|path| p(path)).collect(); - let widest: Vec<&Path> = (regions.iter()) - .filter(|®ion| !regions.iter().any(|wider| wider != region && within(region, wider))) - .collect(); - let mut folded = View::default(); - for region in widest.iter().rev() { - folded.merge(&v.restrict(region)); - } - assert_eq!(v.restrict_all(®ions), folded); } } } diff --git a/rs/moq-net/src/test_interop.rs b/rs/moq-net/src/test_interop.rs index d1eac3fbb34..6886ac8b308 100644 --- a/rs/moq-net/src/test_interop.rs +++ b/rs/moq-net/src/test_interop.rs @@ -570,6 +570,7 @@ fn lite_tracks_interop() { broadcast: crate::Path::from_static("room"), instance: None, node: 0, + depth: 0, } .encode_bytes(version) .unwrap(); diff --git a/rs/moq-net/tests/integration/byte_holes.rs b/rs/moq-net/tests/integration/byte_holes.rs index 413aa9767bc..6bb37a83088 100644 --- a/rs/moq-net/tests/integration/byte_holes.rs +++ b/rs/moq-net/tests/integration/byte_holes.rs @@ -112,6 +112,8 @@ async fn a_half_read_frame_continues_from_its_byte() { } assert!(got == payload, "the frame continues at its byte"); + // A publisher bills what the transport acknowledged, which trails what `E` read. + settle().await; let rest = (SIZE - HALF) as u64; let sent = sent(®istry); assert!(sent >= rest, "{sent} bytes cannot carry the {rest} left"); diff --git a/rs/moq-net/tests/integration/catalog_handover.rs b/rs/moq-net/tests/integration/catalog_handover.rs index 2147dfd0166..2587a5ac480 100644 --- a/rs/moq-net/tests/integration/catalog_handover.rs +++ b/rs/moq-net/tests/integration/catalog_handover.rs @@ -25,6 +25,9 @@ const TEST_TIMEOUT: Duration = Duration::from_secs(300); /// How long any one read may take, in virtual time. const READ_TIMEOUT: Duration = Duration::from_secs(10); +/// How long a lite-07 relay doubts the far end of a closed session. +const DOUBT: Duration = Duration::from_secs(1); + /// Short readers in each burst, re-reading the catalog and leaving. const BURST: usize = 4; @@ -151,7 +154,12 @@ impl Topology { async fn trigger(&mut self, trigger: Trigger) { match trigger { Trigger::Disconnect => abort(self.a_to_r.take().unwrap()), - Trigger::Unannounce => abort(self.p_to_a.take().unwrap()), + Trigger::Unannounce => { + abort(self.p_to_a.take().unwrap()); + // `P` is the far end of `A`'s closed session, and `B`'s route crosses it too: + // on lite-07 `R` holds the claim until the doubt lifts, then moves to `B`. + kio::clock::sleep(DOUBT).await; + } Trigger::BetterRoute => { let direct = link(self.version, &self.publisher, &self.subscriber, None).await; self.links.push(direct); @@ -161,6 +169,9 @@ impl Topology { settle().await; let a_to_r = link(self.version, &self.relay_a, &self.subscriber, None).await; self.a_to_r = Some(a_to_r); + // `R` doubts `A` for a second after its session closed, so the routes the + // new session carries count only once that lifts. + kio::clock::sleep(DOUBT).await; } } settle().await; diff --git a/rs/moq-net/tests/integration/instance.rs b/rs/moq-net/tests/integration/instance.rs index f0f98a1d325..4bdcfb3f313 100644 --- a/rs/moq-net/tests/integration/instance.rs +++ b/rs/moq-net/tests/integration/instance.rs @@ -140,14 +140,14 @@ async fn next_hop_lost(version: &str) { let _to_preferred = link(version, &publisher, &preferred).await; let _to_standby = link(version, &publisher, &standby).await; let mut announced = relay.consume().routes(); - let first_hop = link( + let first_hop = link(version, &preferred, &relay).await; + let first = started(&mut announced).await; + let _second_hop = link( version, - &preferred, - &relay.clone().with_preference(origin::Preference::new(1)), + &standby, + &relay.clone().with_preference(origin::Preference::new(-1)), ) .await; - let first = started(&mut announced).await; - let _second_hop = link(version, &standby, &relay).await; if let Ok(event) = kio::clock::timeout(Duration::from_secs(1), announced.next()).await { panic!("{version}: the standby route changed the preferred one: {event:?}"); } diff --git a/rs/moq-net/tests/integration/main.rs b/rs/moq-net/tests/integration/main.rs index d676ca85dc3..6acf754bf36 100644 --- a/rs/moq-net/tests/integration/main.rs +++ b/rs/moq-net/tests/integration/main.rs @@ -54,6 +54,7 @@ mod model; mod mount; mod multi_cdn; mod partial_group; +mod path_routes; mod quiet_catalog; mod range_anchors; mod range_fill; diff --git a/rs/moq-net/tests/integration/multi_cdn.rs b/rs/moq-net/tests/integration/multi_cdn.rs index 5f9face586e..ff1eac34806 100644 --- a/rs/moq-net/tests/integration/multi_cdn.rs +++ b/rs/moq-net/tests/integration/multi_cdn.rs @@ -3,9 +3,10 @@ //! An endpoint linked to two CDNs uses its preferred one and fails over to the next. //! //! A publisher `P` pushes one broadcast, under one instance, to CDN relays `A` and `B`. A -//! subscriber `S` links to both, ranking `A` first with +//! subscriber `S` links to both, ranking `A` first by demoting `B` with //! [`origin::Producer::with_preference`] while pricing `B` cheaper, so only the -//! preference puts `A` ahead. Losing `A` moves `S` to `B`, and `A` returning moves it +//! preference puts `A` ahead. A lite-07 route ranks by the lowest preference on its path, +//! and `P`'s links to both CDNs are neutral, so promoting `A` instead would gain nothing. Losing `A` moves `S` to `B`, and `A` returning moves it //! back. Between lite-07 routes, which carry the instance, each move resumes the //! subscription. A moq-transport `B` carries none, so `S` mints one for it, and the move //! onto it is a restart that ends the subscription rather than stitching onto it. @@ -179,8 +180,8 @@ impl Topology { push(lite, &publisher, &cdn_a).await, push(secondary, &publisher, &cdn_b).await, ]; - let s_to_a = dial(lite, &cdn_a, &subscriber, 1, A_COST).await; - let s_to_b = dial(secondary, &cdn_b, &subscriber, 0, 1).await; + let s_to_a = dial(lite, &cdn_a, &subscriber, 0, A_COST).await; + let s_to_b = dial(secondary, &cdn_b, &subscriber, -1, 1).await; settle().await; Self { @@ -250,7 +251,7 @@ impl Topology { async fn restore_a(&mut self) { let lite: Version = "moq-lite-07-wip".parse().unwrap(); - self.s_to_a = Some(dial(lite, &self.cdn_a, &self.subscriber, 1, A_COST).await); + self.s_to_a = Some(dial(lite, &self.cdn_a, &self.subscriber, 0, A_COST).await); settle().await; } } diff --git a/rs/moq-net/tests/integration/path_routes.rs b/rs/moq-net/tests/integration/path_routes.rs new file mode 100644 index 00000000000..a5db1c75806 --- /dev/null +++ b/rs/moq-net/tests/integration/path_routes.rs @@ -0,0 +1,401 @@ +#![cfg(feature = "async")] + +//! Lite-07 routes by their path, on real sessions with time paused. +//! +//! A route names every relay it crossed and the session each learned it on. A relay that +//! loses a session names that hop in one DEAD, every relay drops the routes through it, +//! and the node at the closed session's far end is doubted for a second, so nobody moves +//! onto a stale route through it. A relay left without a route holds its claims, +//! unavailable, until the neighbours that learned the route from it answer, or for +//! [`HOLD`] at most. + +use crate::route_change::{link, produce_origin}; +use crate::support; + +use std::time::Duration; + +use futures::channel::mpsc; +use moq_net::{Error, Pattern, Patterns, Version, auth::Grant, origin}; +use support::harness::{MockConnectOptions, MockPair, connect_mock, peer, peer_with_latency}; + +/// How long a relay holds a route no neighbour answers for, and lets an announce +/// stream stall before closing its session. +const HOLD: Duration = Duration::from_secs(10); + +/// How long a closed session's far end is doubted. +const DOUBT: Duration = Duration::from_secs(1); + +fn lite07() -> Version { + "moq-lite-07-wip".parse().unwrap() +} + +/// One change a node showed at a path: how long after the cut, its kind, and whether a +/// usable route reached the publisher. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +struct Seen { + at: Duration, + kind: support::Kind, + available: bool, +} + +/// The changes a node shows at a path. +struct Watch { + rx: mpsc::UnboundedReceiver<(kio::clock::Instant, support::Kind, bool)>, +} + +impl Watch { + /// What the node showed since the last call, timed from `start`. + fn seen(&mut self, start: kio::clock::Instant) -> Vec { + std::iter::from_fn(|| self.rx.try_recv().ok()) + .map(|(at, kind, available)| Seen { + at: at.saturating_duration_since(start), + kind, + available, + }) + .collect() + } +} + +/// Watch every change `node` shows at `path`, past what it shows now. +async fn record(node: &origin::Producer, path: &'static str) -> Watch { + let mut routes = node.consume().routes(); + let (tx, rx) = mpsc::unbounded(); + kio::clock::sim::spawn(async move { + while let Some(event) = routes.next().await { + if event.announce.prefix.as_str() != path { + continue; + } + let kind = match event.kind { + origin::RouteKind::Start => support::Kind::Start, + origin::RouteKind::Update => support::Kind::Update, + origin::RouteKind::Restart => support::Kind::Restart, + origin::RouteKind::End => support::Kind::End, + origin::RouteKind::Offline => support::Kind::Offline, + }; + let seen = (kio::clock::now(), kind, event.announce.route.available()); + if tx.unbounded_send(seen).is_err() { + break; + } + } + }); + let mut watch = Watch { rx }; + land().await; + watch.seen(kio::clock::now()); + watch +} + +fn withdrawn(seen: &Seen) -> bool { + matches!(seen.kind, support::Kind::End | support::Kind::Offline) +} + +/// Close a session at both ends, as a dropped connection does. +fn cut(pair: &MockPair) { + pair.server.abort(Error::Cancel); + pair.client.abort(Error::Cancel); +} + +/// Publish `path` on `node`. +fn publish(node: &origin::Producer, path: &str) -> origin::Route { + let route = node.route(path, origin::RouteOptions::hidden()).unwrap(); + route.announce().unwrap(); + route +} + +/// Wait long enough for every message in flight to land, without reaching the doubt. +async fn land() { + kio::clock::sleep(Duration::from_millis(100)).await; +} + +/// Let the network settle, doubts and holds included. +async fn quiesce() { + kio::clock::sleep(HOLD + DOUBT).await; +} + +/// `W` reaches `P` through `A`, and through `C` and `D`, which never cross `A`. The session +/// from `A` to `W` closes: `W` doubts `A`, the far end, and the detour avoids it, so `W` +/// moves onto it at once and the claim never turns unavailable. +#[kio::test] +async fn a_detour_avoiding_the_far_end_is_taken_at_once() { + let version = lite07(); + let [p, a, c, d, w] = [1, 2, 3, 4, 5].map(produce_origin); + let _live = publish(&p, "live"); + let _p_a = peer(version, &p, &a).await; + let a_w = peer(version, &a, &w).await; + let _p_c = peer(version, &p, &c).await; + let _c_d = peer(version, &c, &d).await; + let _d_w = peer(version, &d, &w).await; + quiesce().await; + + let mut watch = record(&w, "live").await; + let start = kio::clock::now(); + cut(&a_w); + land().await; + let seen = watch.seen(start); + assert!( + seen.iter().all(|seen| seen.available && !withdrawn(seen)), + "the detour serves at once: {seen:?}" + ); +} + +/// `W` reaches `P` through `A`, and through `B`, whose route crosses `A` too. The session +/// from `A` to `W` closes: `W` cannot tell `A` from a dead relay whose news is still on its +/// way, so it moves onto `B`'s route only once the doubt lifts, holding the claim meanwhile. +#[kio::test] +async fn a_detour_through_the_far_end_waits_out_the_doubt() { + let version = lite07(); + let [p, a, b, w] = [1, 2, 3, 4].map(produce_origin); + let _live = publish(&p, "live"); + let _p_a = peer(version, &p, &a).await; + let a_w = peer(version, &a, &w).await; + let _a_b = peer(version, &a, &b).await; + let _b_w = peer(version, &b, &w).await; + quiesce().await; + + let mut watch = record(&w, "live").await; + let start = kio::clock::now(); + cut(&a_w); + quiesce().await; + let seen = watch.seen(start); + assert!( + !seen.iter().any(withdrawn), + "the claim is held, not withdrawn: {seen:?}" + ); + let held = seen.iter().find(|seen| !seen.available).expect("held meanwhile"); + let usable = seen + .iter() + .find(|seen| seen.available) + .expect("B's route serves in the end"); + assert!(usable.at >= held.at + DOUBT, "not before the doubt lifts: {seen:?}"); +} + +/// `X` is `P`'s only way out and neighbours `R1`, `R2` and `R3`, which mesh with each other. +/// `X` dies: each relay hears its own session close and its neighbours' DEADs, and none +/// moves onto another's route through `X` on the way. Each withdraws the claim once. +#[kio::test] +async fn a_dead_relay_reaches_every_neighbour_as_withdrawals() { + let version = lite07(); + let [p, x, r1, r2, r3] = [1, 2, 3, 4, 5].map(produce_origin); + let _live = publish(&p, "live"); + let mut dying = vec![peer(version, &p, &x).await]; + for r in [&r1, &r2, &r3] { + dying.push(peer(version, &x, r).await); + } + let _mesh = [ + peer(version, &r1, &r2).await, + peer(version, &r2, &r3).await, + peer(version, &r1, &r3).await, + ]; + quiesce().await; + + let mut watched = Vec::new(); + for r in [&r1, &r2, &r3] { + watched.push(record(r, "live").await); + } + let start = kio::clock::now(); + dying.iter().for_each(cut); + quiesce().await; + for (i, watch) in watched.iter_mut().enumerate() { + let seen = watch.seen(start); + assert!( + !seen.iter().any(|seen| seen.available && !withdrawn(seen)), + "R{}: no move onto a route through X: {seen:?}", + i + 1 + ); + assert_eq!( + seen.iter().filter(|seen| withdrawn(seen)).count(), + 1, + "R{}: {seen:?}", + i + 1 + ); + assert!(withdrawn(seen.last().unwrap()), "R{}: {seen:?}", i + 1); + } +} + +/// What `B` does while `A` holds the route for it. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Child { + /// Answers, over a link with a one-way [`LAG`]. + Answers, + /// Never answers, and its session closes [`LAG`] after the cut. + Closes, + /// Never answers. + Silent, +} + +/// How long `B` takes in [`hold_behind`]. +const LAG: Duration = Duration::from_secs(2); + +/// `P` feeds `A`, which feeds `C`, and peers with `B`, which learned the route from `A` and +/// offers it back. `P`'s session to `A` closes: `A` holds the route for `B`, so `C` sees the +/// claim unavailable, then withdrawn once the hold ends. Returns how long after the cut that +/// was. +async fn hold_behind(child: Child) -> Duration { + let version = lite07(); + let [p, a, b, c] = [1, 2, 3, 4].map(produce_origin); + let _live = publish(&p, "live"); + let p_a = link(version, &p, &a).await; + let lag = if child == Child::Answers { LAG } else { Duration::ZERO }; + let a_b = peer_with_latency(version, &a, &b, lag).await; + let _a_c = link(version, &a, &c).await; + // A reader at `B` asks `A` for its routes. + let _reader = record(&b, "live").await; + quiesce().await; + + let mut watch = record(&c, "live").await; + if child != Child::Answers { + a_b.server_transport.hold_writes(); + } + let start = kio::clock::now(); + cut(&p_a); + if child == Child::Closes { + kio::clock::sleep(LAG).await; + cut(&a_b); + } + quiesce().await; + let seen = watch.seen(start); + assert!( + matches!( + seen.first(), + Some(Seen { + kind: support::Kind::Update, + available: false, + .. + }) + ), + "held first: {seen:?}" + ); + assert_eq!(seen.iter().filter(|seen| withdrawn(seen)).count(), 1, "{seen:?}"); + let ended = seen.last().unwrap(); + assert!(withdrawn(ended), "{seen:?}"); + ended.at +} + +/// `A` loses its only route and holds the claim for `B`, which learned it from `A`. `B` +/// answers once the DEAD reaches it, a round trip later, and the hold ends then. +#[kio::test] +async fn a_hold_ends_behind_an_answering_neighbour() { + let ended = hold_behind(Child::Answers).await; + assert!(ended >= 2 * LAG && ended < 2 * LAG + DOUBT, "ended at {ended:?}"); +} + +/// [`a_hold_ends_behind_an_answering_neighbour`], with `B` silent until its session closes: +/// nothing is left to wait for. +#[kio::test] +async fn a_hold_ends_behind_a_closed_session() { + let ended = hold_behind(Child::Closes).await; + assert!(ended >= LAG && ended < LAG + DOUBT, "ended at {ended:?}"); +} + +/// [`a_hold_ends_behind_an_answering_neighbour`], with `B` silent: the hold ends at its +/// deadline. +#[kio::test] +async fn a_hold_ends_at_its_deadline_behind_a_silent_neighbour() { + let ended = hold_behind(Child::Silent).await; + assert!(ended >= HOLD && ended < HOLD + DOUBT, "ended at {ended:?}"); +} + +/// `W` reaches `P` through `A`. `Y` and `Z` peer with `W` and with each other, and carry +/// nothing of `P`'s: their session closing names a hop no route of `W`'s crosses, which +/// changes nothing there. +#[kio::test] +async fn an_unknown_hop_in_a_dead_changes_nothing() { + let version = lite07(); + let [p, a, w, y, z] = [1, 2, 3, 4, 5].map(produce_origin); + let _live = publish(&p, "live"); + let _p_a = peer(version, &p, &a).await; + let _a_w = peer(version, &a, &w).await; + let _w_y = peer(version, &w, &y).await; + let _w_z = peer(version, &w, &z).await; + let y_z = peer(version, &y, &z).await; + quiesce().await; + + let mut watch = record(&w, "live").await; + let start = kio::clock::now(); + cut(&y_z); + quiesce().await; + assert_eq!(watch.seen(start), [], "W keeps its route as it was"); +} + +/// A chain one relay longer than a route's hops allow: the last relay refuses the route +/// whose path is full, and every relay before it uses its own. +#[kio::test] +async fn a_path_at_the_hop_limit_is_refused() { + let version = lite07(); + let nodes: Vec<_> = (1..=33).map(produce_origin).collect(); + let mut watched = Vec::new(); + for node in &nodes { + watched.push(record(node, "live").await); + } + let start = kio::clock::now(); + let _live = publish(&nodes[0], "live"); + let mut links = Vec::new(); + for pair in nodes.windows(2) { + links.push(link(version, &pair[0], &pair[1]).await); + } + quiesce().await; + + let reached: Vec = (watched.iter_mut()) + .map(|watch| watch.seen(start).iter().any(|seen| seen.available)) + .collect(); + // The publisher names itself in one hop, and each relay adds its own, so the 32nd + // relay past it hears a path of 32 hops. + assert!(reached[..32].iter().all(|reached| *reached), "{reached:?}"); + assert!(!reached[32], "the full path is refused"); +} + +/// A peer whose session is limited to `a` reports its own session closing. That DEAD +/// covers only the routes that peer sent: `W` keeps the route through `B`, which crosses +/// the same hop, until `B` says so too, and moves onto it meanwhile. +#[kio::test] +async fn a_scoped_peers_dead_leaves_other_sessions_routes_alone() { + let version = lite07(); + let [p, s, b, w] = [1, 2, 3, 4].map(produce_origin); + let _live = publish(&p, "a/live"); + let p_s = link(version, &p, &s).await; + let _s_b = link(version, &s, &b).await; + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(b.consume()); + options.client_subscribe = Some(w.clone()); + options.latency = Duration::from_millis(300); + let _b_w = connect_mock(options).await; + let mut options = MockConnectOptions::new(version); + options.client_publish = Some(s.consume()); + options.server_subscribe = Some(w.clone()); + let s_w = connect_mock(options).await; + let mut grant = Grant::default(); + grant.publish = [Pattern::subtree("a").unwrap()].into_iter().collect::(); + s_w.server.auth().authorize(&grant); + quiesce().await; + + let mut watch = record(&w, "a/live").await; + let start = kio::clock::now(); + cut(&p_s); + kio::clock::sleep(Duration::from_millis(200)).await; + let early = watch.seen(start); + assert!( + early.iter().all(|seen| seen.available && !withdrawn(seen)), + "B's route serves until B reports the hop: {early:?}" + ); + quiesce().await; + let late = watch.seen(start); + assert!(late.iter().any(withdrawn), "B's report withdraws it: {late:?}"); +} + +/// A peer that stops reading its announce stream stalls every route its relay sends it: +/// the relay closes the session once nothing moved for [`HOLD`]. +#[kio::test] +async fn a_peer_that_stops_reading_is_closed() { + let version = lite07(); + let [p, w] = [1, 2].map(produce_origin); + let _first = publish(&p, "first"); + let p_w = link(version, &p, &w).await; + quiesce().await; + + p_w.server_transport.hold_writes(); + let start = kio::clock::now(); + let _second = publish(&p, "second"); + let closed = kio::clock::timeout(HOLD * 2, p_w.server.closed()).await; + let after = kio::clock::now().saturating_duration_since(start); + assert!(closed.is_ok(), "the session closes"); + assert!(after >= HOLD, "not before the hold: {after:?}"); +} diff --git a/rs/moq-net/tests/integration/route_change.rs b/rs/moq-net/tests/integration/route_change.rs index 23df5a1a333..f2ee1a5fb73 100644 --- a/rs/moq-net/tests/integration/route_change.rs +++ b/rs/moq-net/tests/integration/route_change.rs @@ -563,6 +563,41 @@ async fn better_route_restarts(version: &str) { ); } +/// A route through the far end of a closed link waits out the doubt. `A` loses its +/// session to `P`, the far end, and `B`'s route names `P` too: `R` cannot tell `P` from a +/// dead relay whose news is still on its way, so it holds the claim and moves to `B` only +/// once the doubt lifts. The subscription through `A` ends meanwhile; a new one reads on +/// through `B`. +async fn waits_out_the_doubt(version: &str) { + let version: Version = version.parse().unwrap(); + let (mut topology, sub) = Topology::new(version).await; + let mut rx = read(sub); + topology.standby().await; + + let mut group = topology.track.append_next().unwrap(); + group.write_frame(group.at_start(), payload(0, 0)).unwrap(); + group.close().unwrap(); + assert_eq!(next(&mut rx).await, (0, payload(0, 0))); + + topology.trigger(Trigger::Unannounce).await; + kio::clock::sleep(Duration::from_secs(1)).await; + let mut group = topology.track.append_next().unwrap(); + group.write_frame(group.at_start(), payload(1, 0)).unwrap(); + group.close().unwrap(); + let remote = topology.subscriber.consume().request("live", None).await.unwrap(); + let mut fresh = read(remote.track("video").unwrap().subscribe(None).await.unwrap()); + assert_eq!(next(&mut fresh).await, (1, payload(1, 0)), "{version}: through B"); + match kio::clock::timeout(Duration::from_secs(10), rx.next()) + .await + .expect("the subscription through A hung") + { + Some((_, Err(_))) => {} + None => panic!("{version}: the subscription through A ended cleanly"), + Some((group, Ok(_))) => panic!("{version}: the subscription through A read group {group}"), + } + assert!(rx.try_recv().is_err(), "{version}: a delivery after the error"); +} + macro_rules! route_change_tests { ($($name:ident: $version:literal,)*) => { $( @@ -600,13 +635,10 @@ macro_rules! route_change_tests { } #[kio::test] - async fn unannounce_between_groups() { - run(Trigger::Unannounce, Position::BetweenGroups).await; - } - - #[kio::test] - async fn unannounce_mid_group() { - run(Trigger::Unannounce, Position::MidGroup).await; + async fn unannounce_waits_out_the_doubt() { + kio::clock::timeout(TEST_TIMEOUT, waits_out_the_doubt($version)) + .await + .expect("timed out"); } #[kio::test] diff --git a/rs/moq-net/verify/route/Blind.cfg b/rs/moq-net/verify/route/Blind.cfg index cb12ae926fa..864c24c5ece 100644 --- a/rs/moq-net/verify/route/Blind.cfg +++ b/rs/moq-net/verify/route/Blind.cfg @@ -11,6 +11,7 @@ CONSTANTS Dear = {} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {1} Editors = {} diff --git a/rs/moq-net/verify/route/Dead.cfg b/rs/moq-net/verify/route/Dead.cfg index 6cbc4b87fce..2d950cd66cb 100644 --- a/rs/moq-net/verify/route/Dead.cfg +++ b/rs/moq-net/verify/route/Dead.cfg @@ -12,6 +12,7 @@ CONSTANTS Dear = {{0, 4}} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {1} Editors = {} diff --git a/rs/moq-net/verify/route/Drain.cfg b/rs/moq-net/verify/route/Drain.cfg index 8f74661dd0e..1bed20da7f0 100644 --- a/rs/moq-net/verify/route/Drain.cfg +++ b/rs/moq-net/verify/route/Drain.cfg @@ -11,6 +11,7 @@ CONSTANTS Dear = {} Ring = {} Preferred = {} + Upstream = {} Leaving = {0} Scoped = {} Editors = {} diff --git a/rs/moq-net/verify/route/DrainRing.cfg b/rs/moq-net/verify/route/DrainRing.cfg index 99f4beb7d95..3f1af4ca367 100644 --- a/rs/moq-net/verify/route/DrainRing.cfg +++ b/rs/moq-net/verify/route/DrainRing.cfg @@ -14,6 +14,7 @@ CONSTANTS Dear = {} Ring = {1, 2, 3} Preferred = {} + Upstream = {} Leaving = {0} Scoped = {} Editors = {} diff --git a/rs/moq-net/verify/route/Early.cfg b/rs/moq-net/verify/route/Early.cfg index 4defd365574..1e328a4022e 100644 --- a/rs/moq-net/verify/route/Early.cfg +++ b/rs/moq-net/verify/route/Early.cfg @@ -8,6 +8,7 @@ CONSTANTS Dear = {} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {} Editors = {} diff --git a/rs/moq-net/verify/route/Edit.cfg b/rs/moq-net/verify/route/Edit.cfg index b0bcf3684ba..940aec4574c 100644 --- a/rs/moq-net/verify/route/Edit.cfg +++ b/rs/moq-net/verify/route/Edit.cfg @@ -15,6 +15,7 @@ CONSTANTS Dear = {{0, 2}} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {2} Editors = {2} diff --git a/rs/moq-net/verify/route/EditNarrow.cfg b/rs/moq-net/verify/route/EditNarrow.cfg index a5af453f665..875c3fd2967 100644 --- a/rs/moq-net/verify/route/EditNarrow.cfg +++ b/rs/moq-net/verify/route/EditNarrow.cfg @@ -10,6 +10,7 @@ CONSTANTS Dear = {{0, 2}} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {} Editors = {2} diff --git a/rs/moq-net/verify/route/EditWiden.cfg b/rs/moq-net/verify/route/EditWiden.cfg index 82366c4ec2b..0d2c4b8c437 100644 --- a/rs/moq-net/verify/route/EditWiden.cfg +++ b/rs/moq-net/verify/route/EditWiden.cfg @@ -10,6 +10,7 @@ CONSTANTS Dear = {{0, 2}} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {2} Editors = {2} diff --git a/rs/moq-net/verify/route/HoldCycle.cfg b/rs/moq-net/verify/route/HoldCycle.cfg index b3af2339860..b00b16c71fe 100644 --- a/rs/moq-net/verify/route/HoldCycle.cfg +++ b/rs/moq-net/verify/route/HoldCycle.cfg @@ -13,6 +13,7 @@ CONSTANTS Dear = {{2, 4}, {3, 5}} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {4, 5} Editors = {} diff --git a/rs/moq-net/verify/route/Lag.cfg b/rs/moq-net/verify/route/Lag.cfg index 18279bc1f34..a7cc3d4deb1 100644 --- a/rs/moq-net/verify/route/Lag.cfg +++ b/rs/moq-net/verify/route/Lag.cfg @@ -9,6 +9,7 @@ CONSTANTS Dear = {} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {} Editors = {} diff --git a/rs/moq-net/verify/route/LagWide.cfg b/rs/moq-net/verify/route/LagWide.cfg index 029c045f88c..3e1983c1be8 100644 --- a/rs/moq-net/verify/route/LagWide.cfg +++ b/rs/moq-net/verify/route/LagWide.cfg @@ -7,6 +7,7 @@ CONSTANTS Dear = {} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {1} Editors = {} diff --git a/rs/moq-net/verify/route/Loop.cfg b/rs/moq-net/verify/route/Loop.cfg index 62123c8c897..ed0a1ad1a93 100644 --- a/rs/moq-net/verify/route/Loop.cfg +++ b/rs/moq-net/verify/route/Loop.cfg @@ -9,6 +9,7 @@ CONSTANTS Dear = {{1, 2}} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {} Editors = {} diff --git a/rs/moq-net/verify/route/Overlap.cfg b/rs/moq-net/verify/route/Overlap.cfg index fe11d1bf7ab..34fdaa5937d 100644 --- a/rs/moq-net/verify/route/Overlap.cfg +++ b/rs/moq-net/verify/route/Overlap.cfg @@ -10,6 +10,7 @@ CONSTANTS Dear = {{0, 1}, {0, 2}, {1, 3}, {2, 3}, {2, 4}, {3, 4}} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {2, 3, 4} Editors = {} diff --git a/rs/moq-net/verify/route/Prefer.cfg b/rs/moq-net/verify/route/Prefer.cfg index 39091a62f1a..5dd008cf6a3 100644 --- a/rs/moq-net/verify/route/Prefer.cfg +++ b/rs/moq-net/verify/route/Prefer.cfg @@ -10,6 +10,7 @@ CONSTANTS Dear = {} Ring = {} Preferred = {{1, 2}} + Upstream = {} Leaving = {} Scoped = {} Editors = {} diff --git a/rs/moq-net/verify/route/Ring.cfg b/rs/moq-net/verify/route/Ring.cfg index 4b1eedcf83e..682095cfed3 100644 --- a/rs/moq-net/verify/route/Ring.cfg +++ b/rs/moq-net/verify/route/Ring.cfg @@ -9,6 +9,7 @@ CONSTANTS Dear = {} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {} Editors = {} diff --git a/rs/moq-net/verify/route/RouteHandoff.cfg b/rs/moq-net/verify/route/RouteHandoff.cfg index 8a53686af0a..9b96f8f9d4b 100644 --- a/rs/moq-net/verify/route/RouteHandoff.cfg +++ b/rs/moq-net/verify/route/RouteHandoff.cfg @@ -9,6 +9,7 @@ CONSTANTS Dear = {} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {} Editors = {} diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index dc787779b22..c5f41ddebb2 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -17,7 +17,10 @@ (* best live route whose path does not name it: one crossing no draining *) (* session, then the highest lowest link priority, then the cheapest. An *) (* offer carries its drain; its priority and cost follow from its path. *) -(* There is no distance, seqno, ROUTE_REQUEST, or echo. Instead: *) +(* A route learned on a link its learner marks upstream, or from a *) +(* down-only route, is down-only, and is never sent over a link its sender *) +(* marks upstream. There is no distance, seqno, ROUTE_REQUEST, or echo. *) +(* Instead: *) (* *) (* - A relay that loses a session names that hop in one DEAD message per *) (* neighbour, listing which of its routes through it moved to another *) @@ -44,6 +47,7 @@ CONSTANTS Ring, \* relays round a ring in number order: each one's link to the \* next costs 1 there and 2 at the next, overriding `Dear` Preferred, \* the links both ends give priority 1; the rest have 0 + Upstream, \* links whose higher-numbered end marks them upstream Leaving, \* nodes that may send GOAWAY, once on each session Scoped, \* relays that start out watching prefix "a" alone Editors, \* relays that may change what they watch @@ -61,6 +65,8 @@ Cost == [l \in Links |-> IF l \in Dear THEN 2 ELSE 1] Succ(n) == LET after == {m \in Ring : m > n} from == IF after # {} THEN after ELSE Ring IN CHOOSE m \in from : \A k \in from : m <= k +\* Whether n marks its link to m upstream. +Up(n, m) == {n, m} \in Upstream /\ n > m \* What node n's link to m costs n: links round the ring cost by direction. CostAt(n, m) == IF n \in Ring /\ m \in Ring /\ m = Succ(n) THEN 1 ELSE IF n \in Ring /\ m \in Ring /\ n = Succ(m) THEN 2 @@ -101,19 +107,19 @@ OHop == <> \* A route a neighbour offered (`Heard`). `state` is "live", "dead" (its path \* holds a hop this node knows is dead, and the neighbour has not said what \* became of it), or "held". -NoHeard == [on |-> FALSE, path |-> <<>>, drain |-> FALSE, state |-> "live", rev |-> None, items |-> NoItems] +NoHeard == [on |-> FALSE, path |-> <<>>, drain |-> FALSE, down |-> FALSE, state |-> "live", rev |-> None, items |-> NoItems] \* The route a node uses for a key (`Route`). Held: `path` is the last live \* route's, and `next` is the neighbour it holds under, or None on its own account. -NoRoute == [on |-> FALSE, held |-> FALSE, next |-> None, path |-> <<>>, drain |-> FALSE] +NoRoute == [on |-> FALSE, held |-> FALSE, next |-> None, path |-> <<>>, drain |-> FALSE, down |-> FALSE] \* What a node last sent a neighbour for a key (`Sent`). `src` is the key of \* the node's own route the offer reflects, kept after an end. -NoSent(k) == [on |-> FALSE, held |-> FALSE, path |-> <<>>, drain |-> FALSE, rev |-> None, items |-> NoItems, src |-> k] +NoSent(k) == [on |-> FALSE, held |-> FALSE, path |-> <<>>, drain |-> FALSE, down |-> FALSE, rev |-> None, items |-> NoItems, src |-> k] NotShown == [item |-> None, avail |-> FALSE] \* Messages: one record shape so TLC can compare them. -Blank == [t |-> "", k |-> "root", path |-> <<>>, drain |-> FALSE, rev |-> None, items |-> NoItems, +Blank == [t |-> "", k |-> "root", path |-> <<>>, drain |-> FALSE, down |-> FALSE, rev |-> None, items |-> NoItems, hop |-> OHop, moved |-> {}, gone |-> FALSE, node |-> None, set |-> {}] -OfferMsg(k, path, drain) == [Blank EXCEPT !.t = "offer", !.k = k, !.path = path, !.drain = drain] +OfferMsg(k, path, drain, down) == [Blank EXCEPT !.t = "offer", !.k = k, !.path = path, !.drain = drain, !.down = down] HoldMsg(k) == [Blank EXCEPT !.t = "hold", !.k = k] EndMsg(k) == [Blank EXCEPT !.t = "end", !.k = k] ClaimsMsg(k, v) == [Blank EXCEPT !.t = "claims", !.k = k, !.rev = v.floor, !.items = v.items] @@ -198,18 +204,23 @@ Streams(s, p) == {k \in Keys : Within(k, s.ans[p])} \* has a live one not naming the receiver, else its root route, which covers "a". \* A narrower path cannot outrank a better covering path: that preference can \* make neighbours oscillate between scopes. Equal ranks prefer the narrow key. +\* A down-only route is never a source toward an upstream link; with no other +\* live one, a held route is, so a child can answer its loss. Cheapest(s, ks) == LET R(k) == Rank(s.rt[k].path, s.rt[k].drain) IN CHOOSE k \in ks : \A j \in ks : \/ Below(R(k), R(j)) \/ R(k) = R(j) /\ (k = "a" \/ j = "root") -SrcKey(s, p, k) == +SrcKey(n, s, p, k) == IF k # "a" \/ "a" \notin s.ans[p] THEN k - ELSE LET live == {j \in Keys : Live(s.rt[j])} + ELSE LET live == {j \in Keys : Live(s.rt[j]) /\ ~(Up(n, p) /\ s.rt[j].down)} usable == {j \in live : ~Names(s.rt[j].path, p)} + held(j) == s.rt[j].on /\ s.rt[j].held IN IF usable # {} THEN Cheapest(s, usable) ELSE IF live # {} THEN Cheapest(s, live) + ELSE IF held("a") THEN "a" + ELSE IF held("root") THEN "root" ELSE IF s.rt["a"].on THEN "a" ELSE "root" \* A neighbour that learned this node's route for k has yet to answer its loss. @@ -233,6 +244,8 @@ SelectKey(n, s, k, gens) == C == {m \in Nodes : Usable(n, s, m, k) /\ (m = using \/ ~Doubted(s, s.rx[m][k].path))} Path(m) == Append(s.rx[m][k].path, <>) Drain(m) == s.rx[m][k].drain \/ m \in s.drain \/ Drains(s, s.rx[m][k].path) + \* Sticky: learned on an upstream link, or from a down-only route. + Down(m) == s.rx[m][k].down \/ Up(n, m) R(m) == Rank(Path(m), Drain(m)) best == CHOOSE m \in C : \A x \in C : \/ Below(R(m), R(x)) @@ -246,14 +259,15 @@ SelectKey(n, s, k, gens) == /\ Len(was.path) > 0 /\ s.rx[current][k].path = SubSeq(was.path, 1, Len(was.path) - 1) IN IF C # {} - THEN [on |-> TRUE, held |-> FALSE, next |-> best, path |-> Path(best), drain |-> Drain(best)] + THEN [on |-> TRUE, held |-> FALSE, next |-> best, path |-> Path(best), drain |-> Drain(best), down |-> Down(best)] ELSE IF ~was.on THEN NoRoute ELSE IF ~under /\ ~Hopes(n, s, k) /\ ~Pending(n, s, k) THEN NoRoute - ELSE [on |-> TRUE, held |-> TRUE, next |-> IF under THEN current ELSE None, path |-> was.path, drain |-> FALSE] + ELSE [on |-> TRUE, held |-> TRUE, next |-> IF under THEN current ELSE None, path |-> was.path, + drain |-> FALSE, down |-> FALSE] \* The publisher's own route covers everything. OwnRoutes == [k \in Keys |-> IF k = "root" - THEN [on |-> TRUE, held |-> FALSE, next |-> None, path |-> <>, drain |-> FALSE] + THEN [on |-> TRUE, held |-> FALSE, next |-> None, path |-> <>, drain |-> FALSE, down |-> FALSE] ELSE NoRoute] \* Plan every live route before assigning children to the routes they reflect. @@ -261,7 +275,7 @@ OwnRoutes == [k \in Keys |-> IF k = "root" PlanKey(n, s, k, gens) == LET live == SelectKey(n, s, k, gens) IN IF Live(live) THEN live - ELSE IF s.rt[k].on THEN [s.rt[k] EXCEPT !.held = TRUE, !.drain = FALSE] + ELSE IF s.rt[k].on THEN [s.rt[k] EXCEPT !.held = TRUE, !.drain = FALSE, !.down = FALSE] ELSE NoRoute Select(n, s, gens) == @@ -269,7 +283,7 @@ Select(n, s, gens) == ELSE LET planned == [s EXCEPT !.rt = [k \in Keys |-> PlanKey(n, s, k, gens)]] sourced == [planned EXCEPT !.tx = [p \in Nodes |-> [k \in Keys |-> IF planned.tx[p][k].on - THEN [planned.tx[p][k] EXCEPT !.src = SrcKey(planned, p, k)] + THEN [planned.tx[p][k] EXCEPT !.src = SrcKey(n, planned, p, k)] ELSE planned.tx[p][k]]]] IN [sourced EXCEPT !.rt = [k \in Keys |-> SelectKey(n, sourced, k, gens)]] @@ -323,25 +337,27 @@ ShownNow(s) == \* What this node wants a neighbour p to hold for key k: a path, a hold, an \* end, or to keep what it was sent while the next hop this node holds under -\* waits for those behind it. A path or a hold goes only where this node keeps -\* a view: one it does not watch ends (path.rs: `Want::Empty` with no covering view). +\* waits for those behind it. A down-only route goes to no upstream link. A +\* path or a hold goes only where this node keeps a view: one it does not watch +\* ends (path.rs: `Want::Empty` with no covering view). Want0(n, s, k, p) == - LET src == SrcKey(s, p, k) + LET src == SrcKey(n, s, p, k) r == s.rt[src] - IN IF ~r.on THEN [kind |-> "end", path |-> <<>>, drain |-> FALSE, src |-> src] - ELSE IF ~r.held THEN [kind |-> "path", path |-> r.path, drain |-> r.drain, src |-> src] + Kind(kind) == [kind |-> kind, path |-> <<>>, drain |-> FALSE, down |-> FALSE, src |-> src] + IN IF ~r.on THEN Kind("end") + ELSE IF ~r.held + THEN IF Up(n, p) /\ r.down THEN Kind("end") + ELSE [kind |-> "path", path |-> r.path, drain |-> r.drain, down |-> r.down, src |-> src] ELSE IF r.next = p - THEN [kind |-> IF Pending(n, s, src) \/ Hopes(n, s, src) THEN "keep" ELSE "end", - path |-> <<>>, drain |-> FALSE, src |-> src] + THEN Kind(IF Pending(n, s, src) \/ Hopes(n, s, src) THEN "keep" ELSE "end") \* A hold names no path, so a stream holds only the path it last sent. - ELSE IF s.tx[p][k].on /\ s.tx[p][k].path # r.path - THEN [kind |-> "end", path |-> <<>>, drain |-> FALSE, src |-> src] - ELSE [kind |-> "hold", path |-> <<>>, drain |-> FALSE, src |-> src] + ELSE IF s.tx[p][k].on /\ s.tx[p][k].path # r.path THEN Kind("end") + ELSE Kind("hold") Want(n, s, k, p) == LET w == Want0(n, s, k, p) IN IF w.kind \in {"path", "hold"} /\ s.view[k].floor = Absent - THEN [w EXCEPT !.kind = "end", !.path = <<>>, !.drain = FALSE] + THEN [w EXCEPT !.kind = "end", !.path = <<>>, !.drain = FALSE, !.down = FALSE] ELSE w \* `advertise` for one key on the stream to p. @@ -357,11 +373,11 @@ Advertise1(n, s, p, k) == IF t.on THEN [tx |-> NoSent(t.src), out |-> <>] ELSE [tx |-> t, out |-> <<>>] [] OTHER -> - LET offer == ~t.on \/ t.held \/ t.path # w.path \/ t.drain # w.drain + LET offer == ~t.on \/ t.held \/ t.path # w.path \/ t.drain # w.drain \/ t.down # w.down claims == t.rev # v.floor \/ t.items # v.items - IN [tx |-> [on |-> TRUE, held |-> FALSE, path |-> w.path, drain |-> w.drain, + IN [tx |-> [on |-> TRUE, held |-> FALSE, path |-> w.path, drain |-> w.drain, down |-> w.down, rev |-> v.floor, items |-> v.items, src |-> w.src], - out |-> (IF offer THEN <> ELSE <<>>) + out |-> (IF offer THEN <> ELSE <<>>) \o (IF claims THEN <> ELSE <<>>)] Advertise(n, s, ups) == @@ -384,7 +400,7 @@ Report1(n, s, p, hop) == LET t == s.tx[p] through == {k \in Keys : t[k].on /\ HasHop(t[k].path, hop)} w == [k \in Keys |-> Want(n, s, k, p)] - moved == {<> : k \in {j \in through : w[j].kind = "path"}} + moved == {<> : k \in {j \in through : w[j].kind = "path"}} holds == {k \in through : w[k].kind = "hold" /\ ~t[k].held} \* The neighbour this node learned the route from takes silence to mean \* it holds under it, so it is told outright. @@ -395,7 +411,7 @@ Report1(n, s, p, hop) == tx1 == [k \in Keys |-> IF k \notin through THEN t[k] ELSE CASE w[k].kind = "path" -> [t[k] EXCEPT !.path = w[k].path, !.drain = w[k].drain, - !.held = FALSE, !.src = w[k].src] + !.down = w[k].down, !.held = FALSE, !.src = w[k].src] [] w[k].kind = "hold" -> [t[k] EXCEPT !.held = TRUE] [] w[k].kind = "end" -> NoSent(t[k].src) \* p takes a first DEAD to mean every route it did not @@ -406,7 +422,7 @@ Report1(n, s, p, hop) == THEN <> \o (IF gone THEN <<>> ELSE Each(parents, LAMBDA k : <>) \o Each(ends, LAMBDA k : <>)) - ELSE Each({k \in through : w[k].kind = "path"}, LAMBDA k : <>) + ELSE Each({k \in through : w[k].kind = "path"}, LAMBDA k : <>) \o Each(holds, LAMBDA k : <>) \o Each(ends, LAMBDA k : <>) IN IF through = {} THEN [tx |-> t, told |-> s.told[p], out |-> <<>>] @@ -613,7 +629,7 @@ Fail(n, s, hop, from, moved, gone) == ELSE IF p = from /\ Moved(k) # {} THEN LET mv == CHOOSE mv \in Moved(k) : TRUE path == mv[2] - IN [h EXCEPT !.path = path, !.drain = mv[3], + IN [h EXCEPT !.path = path, !.drain = mv[3], !.down = mv[4], !.state = IF DeadIn(path, s.dead) THEN "dead" ELSE "live"] ELSE IF p = from THEN IF gone THEN NoHeard @@ -639,7 +655,7 @@ Recv(n, s, from, msg) == IN CASE msg.t = "offer" /\ ~Within(k, s.asked[from]) -> s [] msg.t = "offer" -> [s EXCEPT !.rx[from][k] = - [on |-> TRUE, path |-> msg.path, drain |-> msg.drain, + [on |-> TRUE, path |-> msg.path, drain |-> msg.drain, down |-> msg.down, state |-> IF DeadIn(msg.path, s.dead) THEN "dead" ELSE "live", rev |-> IF h.on THEN h.rev ELSE None, items |-> IF h.on THEN h.items ELSE NoItems]] @@ -945,6 +961,34 @@ SimplePaths == /\ q[d][i].t = "offer" => IsPath(d[1], q[d][i].path) /\ q[d][i].t = "dead" => \A mv \in q[d][i].moved : IsPath(d[1], mv[2]) +\* The path crosses a link its learner marks upstream. +DownPath(path) == \E i \in DOMAIN path : i > 1 /\ Up(path[i][1], path[i][2]) +\* No hop is learned on a link its sender marks upstream after one learned on a +\* link its learner marks upstream: the route never climbs back up. +ValleyFree(path) == + ~\E i, j \in DOMAIN path : + /\ 1 < i /\ i < j + /\ Up(path[i][1], path[i][2]) + /\ Up(path[j][2], path[j][1]) + +\* A route is down-only exactly when its path came down an upstream link, and +\* no down-only route is sent, in flight, or heard over a link its sender marks +\* upstream: no relay carries traffic between two upstreams. +DownOnly == + /\ \A n \in Relays, k \in Keys : Live(st[n].rt[k]) => (st[n].rt[k].down <=> DownPath(st[n].rt[k].path)) + /\ \A n, p \in Nodes, k \in Keys : st[n].tx[p][k].on => + /\ st[n].tx[p][k].down <=> DownPath(st[n].tx[p][k].path) + /\ Up(n, p) => ~st[n].tx[p][k].down + /\ \A n, m \in Nodes, k \in Keys : st[n].rx[m][k].on => + /\ st[n].rx[m][k].down <=> DownPath(st[n].rx[m][k].path) + /\ ValleyFree(st[n].rx[m][k].path) + /\ Up(m, n) => ~st[n].rx[m][k].down + /\ \A d \in Dirs : \A i \in DOMAIN q[d] : + /\ q[d][i].t = "offer" => + (q[d][i].down <=> DownPath(q[d][i].path)) /\ ~(Up(d[1], d[2]) /\ q[d][i].down) + /\ q[d][i].t = "dead" => \A mv \in q[d][i].moved : + (mv[4] <=> DownPath(mv[2])) /\ ~(Up(d[1], d[2]) /\ mv[4]) + \* A hop whose session has closed: the link is down or opened again since. Gone(h) == h # OHop /\ ({h[1], h[2]} \notin up \/ gen[{h[1], h[2]}] # h[3]) Stale(path) == \E i \in DOMAIN path : Gone(path[i]) @@ -957,7 +1001,7 @@ RECURSIVE Fwd(_, _, _, _) Fwd(cur, k, prev, i) == IF cur = Origin THEN "origin" ELSE IF i = 0 THEN "loop" - ELSE LET src == IF prev = None THEN k ELSE SrcKey(st[cur], prev, k) + ELSE LET src == IF prev = None THEN k ELSE SrcKey(cur, st[cur], prev, k) r == st[cur].rt[src] IN IF ~Live(r) THEN "stuck" ELSE IF prev # None /\ Names(r.path, prev) THEN "refused" @@ -971,7 +1015,7 @@ Request(n, c) == RECURSIVE Walk(_, _, _, _) Walk(cur, k, prev, i) == IF cur = Origin \/ i = 0 THEN <<>> - ELSE LET src == IF prev = None THEN k ELSE SrcKey(st[cur], prev, k) + ELSE LET src == IF prev = None THEN k ELSE SrcKey(cur, st[cur], prev, k) r == st[cur].rt[src] IN IF ~Live(r) \/ (prev # None /\ Names(r.path, prev)) THEN <<>> ELSE <<[node |-> cur, path |-> r.path]>> \o Walk(r.next, src, cur, i - 1) @@ -1002,7 +1046,7 @@ Chased(n, m, k, i) == \/ q[<>] # <<>> \/ /\ i > 0 /\ m # Origin - /\ LET src == SrcKey(st[m], n, k) + /\ LET src == SrcKey(m, st[m], n, k) r == st[m].rt[src] IN /\ Live(r) /\ Stale(r.path) @@ -1015,11 +1059,18 @@ StaleIsChased == Usable(n, st[n], m, k) /\ Stale(st[n].rx[m][k].path) => Chased(n, m, k, Cardinality(Nodes)) \* The nodes a path over open sessions from the publisher reaches, through -\* nodes that watch claim c. +\* nodes that watch claim c, and never over a link its sender marks upstream +\* after one its learner does. Each is <>. RECURSIVE Grow(_, _, _) -Grow(R, c, i) == IF i = 0 THEN R - ELSE Grow(R \cup {n \in Relays : CoversSet(st[n].watch, c) /\ \E m \in R : {m, n} \in up}, c, i - 1) -Carried(c) == Grow({Origin}, c, Cardinality(Nodes)) +Grow(R, c, i) == + IF i = 0 THEN R + ELSE Grow(R \cup {<> : + x \in {y \in R \X Relays : + /\ CoversSet(st[y[2]].watch, c) + /\ {y[1][1], y[2]} \in up + /\ ~(y[1][2] /\ Up(y[1][1], y[2]))}}, + c, i - 1) +Carried(c) == {x[1] : x \in Grow({<>}, c, Cardinality(Nodes))} \* Once nothing is in flight and no doubt is left: each relay shows exactly the \* publisher's claims that a path carries to it, available; a request for each diff --git a/rs/moq-net/verify/route/Twice.cfg b/rs/moq-net/verify/route/Twice.cfg index b1191c9713d..fef277fe656 100644 --- a/rs/moq-net/verify/route/Twice.cfg +++ b/rs/moq-net/verify/route/Twice.cfg @@ -10,6 +10,7 @@ CONSTANTS Dear = {} Ring = {} Preferred = {} + Upstream = {} Leaving = {} Scoped = {} Editors = {} diff --git a/rs/moq-net/verify/route/Upstream.cfg b/rs/moq-net/verify/route/Upstream.cfg new file mode 100644 index 00000000000..69b742d49b3 --- /dev/null +++ b/rs/moq-net/verify/route/Upstream.cfg @@ -0,0 +1,43 @@ +\* Two uplinks: the publisher (0) sits on a CDN relay joined by a dear link to +\* another (1). Relays 2 and 3 form a local mesh, each with an upstream link to +\* one CDN relay. Every route 2 and 3 hold came down an upstream link, so it is +\* down-only even one LAN hop on, and 3 never sends it up to 1: with the CDN +\* link down, 1 has no route rather than one through the local mesh, while 2 +\* still reaches the publisher through 3 and 1 when its own uplink closes. The +\* Rust fixture is two_uplinks_never_carry_cdn_transit. +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {2, 3}, {1, 3}} + Dear = {{0, 1}} + Ring = {} + Preferred = {} + Upstream = {{0, 2}, {1, 3}} + Leaving = {} + Scoped = {} + Editors = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 1 + MaxEdits = 0 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + Mirrored + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + DownOnly + LoopsAreStale + StaleIsChased + RequestsLoopFree + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/UpstreamScoped.cfg b/rs/moq-net/verify/route/UpstreamScoped.cfg new file mode 100644 index 00000000000..08c73ae5bdc --- /dev/null +++ b/rs/moq-net/verify/route/UpstreamScoped.cfg @@ -0,0 +1,39 @@ +\* Upstream.cfg where the second CDN relay (1) asks for "a" alone, so relay 3's +\* stream to it, over a link 3 marks upstream, picks its source among covering +\* routes that are all down-only and ends rather than send one up. +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Links = {{0, 1}, {0, 2}, {2, 3}, {1, 3}} + Dear = {{0, 1}} + Ring = {} + Preferred = {} + Upstream = {{0, 2}, {1, 3}} + Leaving = {} + Scoped = {1} + Editors = {} + Claims = {"x"} + Narrow = {"x"} + MaxRev = 1 + MaxFlaps = 1 + MaxEdits = 0 + Patient = TRUE + +SPECIFICATION Spec + +INVARIANTS + Truthful + Mirrored + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + DownOnly + LoopsAreStale + StaleIsChased + RequestsLoopFree + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-relay/src/cluster.rs b/rs/moq-relay/src/cluster.rs index 823e48420e0..a04c3bb85ed 100644 --- a/rs/moq-relay/src/cluster.rs +++ b/rs/moq-relay/src/cluster.rs @@ -2471,9 +2471,9 @@ async fn lan_discovery( /// The param is dial-side configuration, not something the peer reads off the /// URL (it rides SETUP instead), so it is stripped before connecting. An /// unparseable value is an error rather than a silent default: a mispriced link -/// skews routing for every broadcast crossing it. So is 0: route selection needs -/// every link to cost something, or a route could match its own feasibility -/// distance. +/// skews routing for every broadcast crossing it. So is 0: route selection +/// converges only because every link costs something, so extending a route never +/// leaves it as cheap as the one it came from. fn take_cost(url: &mut Url) -> anyhow::Result> { let Some(value) = url .query_pairs() diff --git a/test/drill/mutations/relay-withdraws-lost-publisher.patch b/test/drill/mutations/relay-withdraws-lost-publisher.patch index 4d6afd9ef7d..a4c3e2329e0 100644 --- a/test/drill/mutations/relay-withdraws-lost-publisher.patch +++ b/test/drill/mutations/relay-withdraws-lost-publisher.patch @@ -7,10 +7,9 @@ # name stays announced with nothing behind it. Explicit withdrawals still consume # the handle, so only the implicit retraction on session loss is removed. diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs -index 085c27fd18..c976671e9d 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs -@@ -9891,7 +9891,7 @@ impl Announced { +@@ -11417,7 +11417,7 @@ }; self.held.pop(); if let Held::Attached(entry) = held { @@ -19,7 +18,7 @@ index 085c27fd18..c976671e9d 100644 } } -@@ -9973,7 +9973,7 @@ fn mint_sources(subscriber: &Subscriber, entry +@@ -11503,7 +11503,7 @@ let validity = if subscriber.version.has_instance() { Default::default() } else { @@ -28,7 +27,7 @@ index 085c27fd18..c976671e9d 100644 }; let source = subscriber.origin.create_source(path.clone(), validity); // The handler exists before the requester sees the source, so a track it -@@ -10001,7 +10001,8 @@ struct AnnouncedRoute { +@@ -11532,7 +11532,8 @@ /// without recomputing the chain. route: crate::origin::RouteInfo, /// Dropping it retracts the route and rejects its queued requests. @@ -38,7 +37,7 @@ index 085c27fd18..c976671e9d 100644 /// Closes as the route goes, which closes every source it minted. live: kio::Producer<()>, /// Queues this route on its prefix's ready set when its request queue wakes. -@@ -10019,7 +10020,7 @@ impl AnnouncedRoute { +@@ -11550,7 +11551,7 @@ ) -> Self { Self { route, @@ -48,19 +47,18 @@ index 085c27fd18..c976671e9d 100644 waker: std::task::Waker::from(wake.clone()), wake, diff --git a/rs/moq-net/src/lite/subscriber/routed.rs b/rs/moq-net/src/lite/subscriber/routed.rs -index ee42ed8829..1cad960cf8 100644 --- a/rs/moq-net/src/lite/subscriber/routed.rs +++ b/rs/moq-net/src/lite/subscriber/routed.rs -@@ -423,7 +423,7 @@ impl Routed { - if let Some(served) = self.served.remove(&(id, claim)) +@@ -706,7 +706,7 @@ + if let Some(served) = self.served.remove(&(id, claim, prefix)) && ended { - served.route.dynamic.withdrawn(); + std::mem::ManuallyDrop::into_inner(served.route.dynamic).withdrawn(); } } - // The ROUTE_END echo goes out as the peer's ROUTE_END is read, and a stream -@@ -457,7 +457,7 @@ impl Routed { + // A stream this node asked for is never sent routes. +@@ -751,7 +751,7 @@ { tracing::debug!(route = %subscriber.log_path(&path), node = id, "dropping looped claim"); for entry in self.previous.remove(&key).into_iter().chain(self.served.remove(&key)) { diff --git a/test/drill/mutations/subscriber-leaks-broadcasts.patch b/test/drill/mutations/subscriber-leaks-broadcasts.patch index 8738695e886..aa27c38ba3c 100644 --- a/test/drill/mutations/subscriber-leaks-broadcasts.patch +++ b/test/drill/mutations/subscriber-leaks-broadcasts.patch @@ -11,7 +11,7 @@ diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs -@@ -2042,7 +2042,8 @@ +@@ -2651,7 +2651,8 @@ /// it comes from the connect config or the peer's SETUP, neither of which /// changes for the life of the session. link_cost: u64, @@ -21,7 +21,7 @@ diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber. // Lite06+: announce ids. Each received `active` implicitly assigns the next // per-stream ordinal; `ended`/`restart` reference it instead of repeating the // path, and lite-07 bases name it too. Tracked even for announces we drop -@@ -2145,7 +2146,10 @@ +@@ -2814,7 +2815,10 @@ let mut run: PrefixRun = PrefixRun { responder_origin, link_cost, @@ -33,26 +33,26 @@ diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber. decoder: lite::AnnounceDecoder::default(), auth_epoch: 0, initial, -@@ -2301,7 +2305,8 @@ - path: Path, - instance: Option, +@@ -2983,7 +2987,8 @@ node: u64, + /// How many leading segments of the path make up the route's prefix (lite-07). + depth: u64, - source: crate::model::broadcast::SourceGuard, + // MUTATION: never dropped, so a session that ends keeps every source it minted alive. + source: std::mem::ManuallyDrop, dynamic: crate::broadcast::Requests, route: kio::Consumer<()>, // A dedicated close-watch handle, since each pending operation needs its own. -@@ -2451,7 +2456,7 @@ - path: minted.path, +@@ -3129,7 +3134,7 @@ instance: minted.instance, node: minted.node, + depth: minted.depth, - source: minted.source, + source: std::mem::ManuallyDrop::new(minted.source), datagram_requests: minted.dynamic.datagrams(), dynamic: minted.dynamic, route: minted.route, -@@ -2560,7 +2565,7 @@ +@@ -3237,7 +3242,7 @@ let serve = TrackServe { validity: self.dynamic.validity().clone(), route: Some(self.route.clone()),