diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index b833d5d91cf..31acd93f803 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -419,7 +419,7 @@ The publisher replies with a single ANNOUNCE_OK message followed by the matching - ANNOUNCE_RESTART: a claim now serves another Instance, or a higher Epoch of the same one. - ANNOUNCE_FLOOR: a batch of a route's claims is complete. - WARM_START: the publisher keeps a copy of a broadcast whether or not the subscriber pulls through it (see [Warm Routes](#warm-routes)). -- WARM_UPDATE: a warm route's Serves or Warm Cost changed. +- WARM_UPDATE: a warm route's Warm Cost, carrier, or Hops changed. - WARM_END: the publisher withdrew a warm route. - SYNCED: the initial set is complete. @@ -581,38 +581,33 @@ An advertiser signals capacity through its claim alone, withdrawing or re-pricin #### Warm Routes {#warm-routes} A relay MAY pull each broadcast through a neighbor of its choosing rather than along its selected route to the origin node, so a broadcast crosses fewer relays and neighboring relays share one copy. -A warm route tells a neighbor that the sender carries a broadcast and keeps it whether or not that neighbor pulls through it. -An endpoint that does not choose per broadcast ignores every WARM_START, WARM_UPDATE, and WARM_END, and sends a `Serves` of Unknown unless it serves only its own readers. - -Every SUBSCRIBE names whom it serves in `Serves`: - -- Alone: demand of the subscriber's own, either a local reader or pulls that trace back to two or more relays. -- Relay: demand that traces back to one relay alone, its root, named by its Node ID. A downstream pull that serves Alone traces back to the relay that sent it, and one that serves Relay to its root. -- Unknown: the subscriber does not say, or some of its demand came from a pull that serves Unknown and the rest traces back to fewer than two relays. - -A copy kept after its demand is gone serves whom it last served. - -A relay that pulls a broadcast for demand MAY send a warm route for it, under its route to the broadcast's origin node, on an Announce Stream it answers whose request covers the path. -It sends one only while its `Serves` is not Unknown and its selected route to the origin node is usable, and only to a neighbor that has no usable route to that origin node or whose last route to it has a Metric at least the relay's own. -A warm route travels one hop: a receiver never forwards it. -It carries the sender's `Serves` and its `Warm Cost`, the price it pays for its own pull, as defined below. -The sender sends a WARM_UPDATE when either changes, and a WARM_END once it no longer pulls the broadcast. -A change that narrows whom the copy serves, from Alone to a Relay, from one Relay to another, or to no warm route at all, waits until the sender has kept the copy without the old demand for as long as it keeps any copy whose demand is gone, so a neighbor that just joined it does not move straight back; until then the warm route stands as last sent, `Warm Cost` included. - -A receiver counts a neighbor warm for a broadcast when the neighbor's warm route serves Alone, or serves a Relay that is neither the receiver nor the root of the receiver's own pull while that pull does not serve Unknown. -Demand a receiver causes, through any number of relays, then never warms a neighbor toward it. -The receiver prices a pull through each neighbor as: - -- the link cost alone, through a neighbor it counts warm, whose copy is paid for already; -- the link cost plus the neighbor's `Warm Cost`, through a neighbor with a warm route it does not count warm, whose copy depends on the receiver's demand; -- otherwise its metric through that neighbor. - -A relay MUST pull a broadcast only through a neighbor eligible for it: one whose last route to the request's origin node is feasible (see [Routing](#routing)), or was advertised with exactly the Seqno and Metric of the relay's feasibility distance while the neighbor's Rendezvous Hash for the broadcast's path is lower than the relay's own. -The Rendezvous Hash is as defined in [Resolution](#resolution), with the neighbor's or the relay's own Node ID (see [Node ID Parameter](#node-id-parameter)). -A feasibility distance only falls within a Seqno, so the Seqno rises, or the Metric and then the hash fall, along every pull, and the pulls of a broadcast never form a loop, even over stale routes. - -Among the eligible neighbors, a relay SHOULD prefer a session that has not received a GOAWAY, then the lowest price, then a neighbor it counts warm, then the lowest Rendezvous Hash. -It SHOULD move an existing pull only to a neighbor strictly better by GOAWAY and then price, so two relays pricing each other's copies settle instead of swapping. +A warm route offers a neighbor a copy of one broadcast: the Hops it crossed, its `Warm Cost`, and its carrier. +An endpoint that does not choose per broadcast ignores every WARM_START, WARM_UPDATE, and WARM_END, sends none, and pulls each broadcast along its selected route. + +A relay's **source** for a broadcast is where it pulls it from: its next hop on its selected route to the origin node, or a neighbor whose warm route it joins. +Pulling through a neighbor gets whatever that neighbor pulls, so pulling through the next hop joins its copy when it offers one. +A copy's **carrier** is the relay with a local reader where the copy left the origin node's route: a relay with a reader whose next hop offers it no copy. +A copy's Hops are those of the route as far as its carrier, then the Hop ID of each relay that joined it, the sender last, as for a route's Hops (see [Routing](#routing)). +Its `Warm Cost` is 0 at its carrier, and what joining it cost every relay after: the `Warm Cost` it joined plus the cost of the link it joined over, so a copy's cost rises with every link from where it left the route. + +A relay with a source offers its copy, if it has one, on every Announce Stream it answers whose request covers the broadcast's path, under its route to the origin node: the copy it joined, or, as a carrier, the one it pulls. +It offers the copy even to a neighbor its Hops name, which tells that neighbor that pulling through the relay leads back to it. +A relay with no demand left offers the copy it would join, so a neighbor can price it before it pulls, and keeps its pull only for as long as it keeps any copy whose demand is gone. +It sends a WARM_UPDATE when the copy it offers changes, and a WARM_END once it offers none, so a broadcast no relay carries costs no message of its own. + +A relay MUST NOT join a copy whose Hops name it or a neighbor whose subscription to the broadcast it serves, or whose Hops it would refuse for a route (see [Routing](#routing)), and MUST NOT join a copy of another Instance. +Among its next hop and the copies it may join, it SHOULD pull through the cheapest: a copy costs its `Warm Cost` plus the link's cost, and the next hop without a copy the relay's metric to the origin node. +A tie stays on the current source, then prefers a copy. +It SHOULD NOT pull through a copy offered on a session that received a GOAWAY, unless it is its next hop's, and moves a pull already on one. + +Two relays that start carrying in the same instant would each join the other's copy for good, so copies are ranked by the Rendezvous Hash of the broadcast's path and their carrier's Hop ID, as defined in [Resolution](#resolution). +A relay on a copy, as its carrier or by joining one still offered, MUST NOT move to another copy, its next hop's included, unless that copy's carrier is the same or ranks lower. +A relay that pulls along its route with no reader of its own hands whatever it joins to the neighbors that pull through it, or would on their selected routes, so it is bound by their carriers instead: each such neighbor offers back its copy with Hops that begin with the relay's own route's. +It MUST NOT join a copy other than its next hop's unless its carrier ranks at or below those of all of these offers, nor while a neighbor whose subscription it serves has offered none. +An offer whose Hops do not begin with the relay's route was sent before the relay took that route, and counts as none: it bounds no carrier, and from a neighbor whose subscription the relay serves it forbids the join. + +The Hops keep the pulls of a broadcast free of loops once the copies offered are delivered: no relay joins a copy that passes through itself or a neighbor it serves. +Two relays acting on each other's stale offers can still join each other until the WARM_END or WARM_UPDATE crossing their link arrives, and the pull then moves on. ### Subscribe A subscriber opens Subscribe Streams to request a Track. @@ -1514,7 +1509,7 @@ The origin's Revision at which the floor certifies the stream's claims for the r ## WARM_START {#warm-start} -A publisher sends a WARM_START message to tell the subscriber that it carries a broadcast and keeps it whether or not the subscriber pulls through it (see [Warm Routes](#warm-routes)). +A publisher sends a WARM_START message to offer the subscriber its copy of a broadcast (see [Warm Routes](#warm-routes)). Each WARM_START implicitly assigns the next Warm ID on the stream, later referenced by WARM_UPDATE and WARM_END (see [Announce](#announce)). ~~~ @@ -1525,8 +1520,10 @@ WARM_START Message { Broadcast Path Suffix (s), Instance (b), Minted (8), - Serves (..), Warm Cost (i), + Carrier Index (i), + Hop Count (i), + Hop ID (i) ..., } ~~~ @@ -1543,19 +1540,26 @@ Combined with the requested prefix to form the broadcast's path. **Instance** and **Minted**: The Instance of the copy the publisher carries, as defined for [ANNOUNCE_START](#announce-start). -**Serves**: -Whom the publisher's own pull of the broadcast serves, as defined for [SUBSCRIBE](#subscribe). -Unknown is a PROTOCOL_VIOLATION here. - **Warm Cost**: -The publisher's price for its own pull of the broadcast (see [Warm Routes](#warm-routes)), in the units of the route's Metric. +What pulling the copy cost the publisher, 0 when it is the copy's carrier (see [Warm Routes](#warm-routes)), in the units of the route's Metric. -A stream has at most one live warm route per route and path. -A subscriber that uses warm routes MUST close the session with a PROTOCOL_VIOLATION on a WARM_START naming a Route ID that was never assigned or already retired, or a path its route already has a live warm route for. +**Carrier Index**: +The position of the copy's carrier among the Hop IDs that follow, 0 for the first. +A value not below `Hop Count` is a PROTOCOL_VIOLATION. + +**Hop Count**: +The number of Hop IDs that follow. +More than 32 is a PROTOCOL_VIOLATION. + +**Hop ID**: +The copy's Hops (see [Warm Routes](#warm-routes)), in order, the publisher's own last. + +A stream has at most one live warm route per route and Broadcast: a relay still pulling an Instance for readers pinned to it offers that copy beside the one of the Instance that replaced it. +A subscriber that uses warm routes MUST close the session with a PROTOCOL_VIOLATION on a WARM_START naming a Route ID that was never assigned or already retired, or a path and Instance its route already has a live warm route for. ## WARM_UPDATE {#warm-update} -A publisher sends a WARM_UPDATE message when a warm route's `Serves` or `Warm Cost` changes, referencing its Warm ID. +A publisher sends a WARM_UPDATE message when the copy a warm route offers changes, referencing its Warm ID. The id stays live. ~~~ @@ -1563,8 +1567,10 @@ WARM_UPDATE Message { Type (i) = 0xD Message Length (i) Warm ID (i), - Serves (..), Warm Cost (i), + Carrier Index (i), + Hop Count (i), + Hop ID (i) ..., } ~~~ @@ -1574,7 +1580,7 @@ Set to 0xD to indicate a WARM_UPDATE message. **Warm ID**: The ordinal implicitly assigned by a prior WARM_START on this stream. -**Serves** and **Warm Cost**: +**Warm Cost**, **Carrier Index**, **Hop Count**, and **Hop ID**: As defined for [WARM_START](#warm-start), replacing the warm route's. @@ -1618,7 +1624,6 @@ SUBSCRIBE Message { Range Count (i) Range (..) ... Update (i) - Serves (..) } Range { @@ -1634,10 +1639,6 @@ Anchor { [Timestamp (i)] } -Serves { - Serves Type (i) - [Node ID (i)] -} ~~~ **Subscribe ID**: @@ -1696,15 +1697,6 @@ A Range resolves once: an update that leaves it live does not resolve it again. The number of the subscription's terms this message sets: 0 on SUBSCRIBE, and on each SUBSCRIBE_UPDATE a larger value than the one before it. A [SUBSCRIBE_DROP](#subscribe-drop) echoes the number in effect when its groups were given up, so the subscriber can tell a stale verdict on its current terms from one an update has since replaced. -**Serves Type**: -Whom the subscription serves (see [Warm Routes](#warm-routes)), and which field follows: - -- Unknown (0x0): no fields. -- Alone (0x1): no fields. -- Relay (0x2): the `Node ID` of the one relay its demand traces back to. 0 is a PROTOCOL_VIOLATION. - -Any other value is a PROTOCOL_VIOLATION. - A `Start` is a floor, not a request; `Subscriber Max Age` is the only thing that asks for data. The publisher SHOULD start at the oldest group at or above the floor that [Expiration](#expiration) has not expired, and MUST NOT deliver an older one. A `Subscriber Max Age` of 0 therefore starts at the latest group, since every older group is already stale. @@ -1740,7 +1732,6 @@ SUBSCRIBE_UPDATE Message { Range (..) ... Update (i) Wanted Group (i) - Serves (..) } ~~~ @@ -2443,7 +2434,7 @@ An application-specific payload. - The Cost parameter must be at least 1. - Replaced the shortest-path, Hop ID, Instance Hash, and Chain Hash tie-breaks with the local priority of the route's session, the Announce Cost plus the route's metric, and a Rendezvous Hash of the requested path and the origin's Node ID. - Removed the standby-ordering deployment requirements on Route Cost seeds and bounds. -- Added warm routes: WARM_START (0xC), WARM_UPDATE (0xD), and WARM_END (0xE) on the Announce Stream, and `Serves` on SUBSCRIBE and SUBSCRIBE_UPDATE. A relay MAY pull each broadcast through a neighbor that already carries it, priced by whom that neighbor's copy serves, and only through a neighbor whose route is feasible or tied at its feasibility distance with a lower Rendezvous Hash, so the pulls never loop. +- Added warm routes: WARM_START (0xC), WARM_UPDATE (0xD), and WARM_END (0xE) on the Announce Stream. A relay MAY pull each broadcast through a neighbor that offers a copy of it, carrying the copy's Hops, its Warm Cost, and its carrier. A relay never joins a copy whose Hops name it or a neighbor it serves, and moves between copies only toward a carrier with a lower Rendezvous Hash, so the pulls of a broadcast are free of loops once the copies offered are delivered. - Added the temporary Lite-06 Chain Setup Parameter (0x9) and a trailing Lite-06 Chain on ANNOUNCE_START, ANNOUNCE_UPDATE, and ANNOUNCE_RESTART: the moq-lite-06 Hop ID list of a claim learned over moq-lite-06, which a node drops when it names that node. ### Tracks {#changelog-07-tracks} diff --git a/js/net/src/lite/subscribe.test.ts b/js/net/src/lite/subscribe.test.ts index 8cf45ace599..57cb6e1a841 100644 --- a/js/net/src/lite/subscribe.test.ts +++ b/js/net/src/lite/subscribe.test.ts @@ -200,11 +200,10 @@ test("lite-07 carries the span as its only range, exclusive at the end", async ( test("a lite-07 list decodes as the span covering it", async () => { // Priority, order, max delay, and an unstated min delay as SubscribeUpdate writes them, - // then no Range ID to retire, the ranges it adds, the update number, an absent wanted group, and - // a pull that serves Alone. + // then no Range ID to retire, the ranges it adds, the update number, and an absent wanted group. const head = (await encodeMessage(Version.DRAFT_07, new SubscribeUpdate({ priority: 0, ranges: [] }))).slice(1, 5); const update = async (ranges: number[]) => { - const body = [...head, 0, ...ranges, 0, 0, 1]; + const body = [...head, 0, ...ranges, 0, 0]; const got = await SubscribeUpdate.decode( new Reader(undefined, new Uint8Array([body.length, ...body]), Version.DRAFT_07), Version.DRAFT_07, diff --git a/js/net/src/lite/subscribe.ts b/js/net/src/lite/subscribe.ts index f5334b72f60..0cc7bcb918e 100644 --- a/js/net/src/lite/subscribe.ts +++ b/js/net/src/lite/subscribe.ts @@ -28,7 +28,6 @@ import { hasRanges, hasRefresh, hasSkip, - hasWarm, resolvesStart, Version, } from "./version.ts"; @@ -43,27 +42,6 @@ async function encodeUpdate(w: Writer, version: Version, update: bigint) { if (hasSkip(version)) await w.u62(update); } -// `Serves Type`: whom a pull serves. A relay names the one relay its demand traces back to. -const SERVES_UNKNOWN = 0n; -const SERVES_ALONE = 1n; -const SERVES_RELAY = 2n; - -/** Encode whom a pull serves: Alone, since this endpoint forwards nothing and every pull is its own. */ -async function encodeServes(w: Writer, version: Version) { - if (hasWarm(version)) await w.u62(SERVES_ALONE); -} - -/** Decode whom a pull serves, and ignore it: only a relay choosing a parent per broadcast prices by it. */ -async function decodeServes(r: Reader, version: Version) { - if (!hasWarm(version)) return; - const serves = await r.u62(); - if (serves === SERVES_RELAY) { - if ((await r.u62()) === 0n) throw new ProtocolViolation("serves names relay 0"); - } else if (serves !== SERVES_UNKNOWN && serves !== SERVES_ALONE) { - throw new ProtocolViolation(`unknown serves type ${serves}`); - } -} - /** * Encode the `Group Start` field shared by SUBSCRIBE and SUBSCRIBE_UPDATE. * @@ -753,7 +731,6 @@ export class SubscribeUpdate { } await encodeUpdate(w, version, this.update); if (hasRefresh(version)) await w.u62(this.wantedGroup === undefined ? 0n : this.wantedGroup + 1n); - await encodeServes(w, version); break; } } @@ -780,7 +757,6 @@ export class SubscribeUpdate { } const update = await decodeUpdate(r, version); const wanted = hasRefresh(version) ? await r.u62() : 0n; - await decodeServes(r, version); return new SubscribeUpdate({ wantedGroup: wanted === 0n ? undefined : wanted - 1n, priority, @@ -918,7 +894,6 @@ export class Subscribe { break; } await encodeUpdate(w, version, this.update); - await encodeServes(w, version); } static async #decode(r: Reader, version: Version): Promise { @@ -939,7 +914,6 @@ export class Subscribe { const minDelay = await decodeMinDelay(r, version); const span = await decodeSpan(r, version); const update = await decodeUpdate(r, version); - await decodeServes(r, version); return new Subscribe({ id, broadcast, diff --git a/js/net/src/lite/version.ts b/js/net/src/lite/version.ts index 5a15033b1ea..56dc6455bdc 100644 --- a/js/net/src/lite/version.ts +++ b/js/net/src/lite/version.ts @@ -488,23 +488,6 @@ export function hasAnnounceCompression(version: Version): boolean { } } -/** Whether the announce stream carries warm routes (WARM_START, WARM_UPDATE, and WARM_END), and - * SUBSCRIBE and SUBSCRIBE_UPDATE say whom each pull serves. Added in lite-07. */ -export function hasWarm(version: Version): boolean { - // Explicitly list older versions so future versions keep the lite-07+ behavior. - switch (version) { - case Version.DRAFT_01: - case Version.DRAFT_02: - case Version.DRAFT_03: - case Version.DRAFT_04: - case Version.DRAFT_05: - case Version.DRAFT_06: - return false; - default: - return true; - } -} - /** Whether TRACK_INFO carries the refresh ratio and SUBSCRIBE_UPDATE a wanted group. Added in lite-07. * Older peers see a track that never refreshes. */ export function hasRefresh(version: Version): boolean { diff --git a/quest/m2/cache-routing-proof.md b/quest/m2/cache-routing-proof.md index 43d2526c853..25bf364906d 100644 --- a/quest/m2/cache-routing-proof.md +++ b/quest/m2/cache-routing-proof.md @@ -3,8 +3,8 @@ ## Goal Sweeps over the route layer show that the [epic](/quest/m3/cache-routing/README.md)'s -warm routes, which propagate per broadcast, stay loop-free at every step, -under stale, reordered, and in-flight messages. Once demand and topology stop +warm routes, which propagate per broadcast, stay loop-free once the offers in +flight are delivered, under stale, reordered, and in-flight messages. Once demand and topology stop changing, every demanded broadcast is delivered, parent choices stop changing, and a relay carries a broadcast only when it has a local reader, feeds two or more downstream sessions, or is its consumers' cheapest route @@ -30,41 +30,102 @@ picked 2026-11-11 on removes the seqno, ROUTE_REQUEST, and the ROUTE_END echo. [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) proved the -one-hop warm route first decided (epic items 8 to 12): -`rs/moq-net/src/route/tests/cache.rs` models demand, parents, warm routes per -neighbor, linger, drain, and legacy peers over the real `Router`, through the -crate-private `Router::eligible`, and lite-07 carries its wire (WARM_START, -WARM_UPDATE, WARM_END, and `Serves` on SUBSCRIBE), which Rust reads and -ignores. Rewrite the model, its fixtures, and that wire for the pick. Start -from `rs/moq-net/src/route/tests/copies.rs`, which runs a relay joining a -copy at a neighbour farther from the publisher than itself, two relays that -start carrying together, and a copy that goes, on the prototype router. It -ranks carriers by a hash of the broadcast and the carrier; the proof settles -whether that rank is the rule. +one-hop warm route first decided (epic items 8 to 12). -The fixtures in `cache.rs` keep their meshes: +Found 2026-11-12 ([moq-dev/moq#8615](https://github.com/moq-dev/moq/pull/8615)), +rewriting `rs/moq-net/src/route/tests/cache.rs` on the path prototype in +`paths.rs` (Rules item 27 picked: causes, holds, a 4-hop doubt timer), with +demand, subscriptions on their own streams, lingers, GOAWAY, legacy peers, +and instances. A path alone does not settle copies that propagate; each rule +below was forced by a sweep scenario that otherwise never settles or loops +for good: -- A chain of three relays plus a skip link, priced 3 and 5: cold skip, warm - join, and the return to the skip when the skipped relay's reader leaves. -- A relay whose only consumer is N withdraws its warm route toward N. -- Two relays at equal cost, both warm: the rank picks one carrier, and they - never adopt each other. -- A ring of relays each holding a stale report of a neighbor's cost: no loop - and no broadcast left without a source. -- A warm withdrawal in flight while N's request is on its way. -- A route loss, and a relay that drains (GOAWAY makes a neighbor least - preferred) while broadcasts are being pulled. -- A warm route for another instance of the same path is never joined. -- k LAN relays with their own uplinks: one uplink copy per broadcast in a - full mesh; record the copies a sparse ring keeps. +- **Monotone copy costs.** Only a relay with a reader whose pull follows the + publisher's route offers its copy at no cost; a relay that joined a copy + offers it at what its own pull costs. Pricing every reader's copy at 0 + made a dispute wheel round a ring of five, two consistent states that a + lockstep schedule flips between forever (scenario 77). Joining a copy + still costs only the links from where it left the publisher's route. +- **Carrier rank.** Every copy names its carrier, the relay with a reader + where it left the publisher's route. A relay on a copy, its own included, + moves onto another, its next hop's included, only if that copy's carrier + ranks below by the Rendezvous Hash. A rank per pair of neighbours, or only + between readers, left longer crossing cycles trading places (scenarios 139 + and 72). +- **A relay off a copy is bound by the copies it feeds.** A relay pulling + along its route with no reader of its own, or with no demand at all, hands + whatever it joins to every neighbour that pulls through it or whose route's + next hop it is, since a next hop's copy is no choice. It joins a copy other + than its next hop's only if that copy's carrier ranks at or below the + carrier each such neighbour offers back along this relay's route, and none + while a neighbour that pulls through it has offered none. Free, a relay + with no reader joined a higher-ranked copy and relabelled the carrier + behind it, and a copy circulated round a ring of five (scenario 246 of the + long sweep); an offer back along another path is an echo of the relay's + own old copy, which kept two relays crossing every four rounds. +- **Truthful offers.** A relay offers every neighbour that reads copies its + current copy, even one whose path names that neighbour (poisoned reverse, + not split horizon), and pulling through a neighbour means its copy when it + offers one, its route only when it does not. With split horizon a request + that follows a route reaches a relay that forwards it into a copy leading + back, a cycle that outlives quiescence (scenario 1563 of the long sweep). + A copy whose path names a neighbour the relay serves is unusable. A relay + serving a legacy neighbour may still join a copy: the path check keeps out + one through that neighbour, and refusing every copy while its subscription + came and went kept a ring flipping. +- **GOAWAY** moves every pull off a draining neighbour's copy other than + the next hop's, rather than ranking it last ahead of cost, which let two + relays each prefer the other's propagation of the draining carrier's copy + (scenario 149 without holds). +- **A pull on another relay's behalf offers nothing new**, so `Serves` and + its roots are not needed: a relay with no reader offers only a copy it + joined, at its cost. -Add a relay that joins a copy at a neighbour farther from the publisher than -itself. Make every sweep scenario settle: `UNSETTLED` in `cache.rs` names -scenario 2251, which never does under the one-hop rule, and the sweep's long -run (`sweep_cache_routing_long`) leaves the merge gate's ignore list for -nightly once it is empty. Report warm messages per demand change and confirm -an idle broadcast sends none. Update the draft's Warm Routes section and the -Rust and `js/net` codecs in the same PR. +Decided 2026-11-13 (maintainer, /quest-run): ✅ accept a cycle that lasts +until the crossing withdrawal is delivered, as routes item 27 accepted for +selected routes, and word the Goal "loop-free once the offers in flight are +delivered" (rejected: a per-broadcast distance, the carrier's rank then the +cost, with a reset by echo or timer, which brings back the seqno and echo +item 27 removed). Two relays acting on each other's stale offers both join +in the same instant, which only such a distance prevents. + +Measured (`sweep_cache_routing_long`, 3000 scenarios each with and without +held links, the gate's seed): every scenario settles on the steady state, an +idle broadcast sends no warm message, and about 3.2 warm messages go out per +reader change. A broadcast's pulls form a cycle after 0.39% of churn steps with +held links and 0.50% without, each ending when the withdrawal crossing its +link is delivered, at most 53 deliveries later in the sweep's random order. +Each subscription is a stream of its own, so a replacement pull can overtake +the cancel of the stream it replaces, and the steady state holds one stream +per neighbour fed. The +LAN fixture keeps one uplink copy per broadcast in a full mesh of 6, and 15 +copies for 8 broadcasts in a ring of 6. `settle` lets a hop of delay pass every +64 rounds of deliveries, so a doubt lifts during a long exchange as it would. + +Lite-07 carries the rule: WARM_START and WARM_UPDATE carry the copy's Warm +Cost, its carrier's position in its Hops, and the Hops, and `Serves` left +SUBSCRIBE and SUBSCRIBE_UPDATE +([moq-dev/moq#8615](https://github.com/moq-dev/moq/pull/8615)). + +Remaining: + +- Under other seeds the long sweep still finds scenarios that never settle, + 2 in 48,000 across 8 seeds (the rules first proven here failed every seed + within 1,200 scenarios): `MOQ_ROUTE_SEED=305419896` scenarios 196 and + 1391, both without held links. Each rule above closed one such class and + exposed a rarer one, and binding a + relay by the copy a neighbour routed through it sits on now, wherever it + came from, made a third seed fail, so the next step is an argument that + the rules converge, such as a TLA+ model of the copy choice alongside + `verify-route`'s, rather than another rule per scenario. Replay one with + `MOQ_ROUTE_SEED= cargo test -p moq-net --release --lib + route::tests::cache::sweep_cache_routing_long -- --ignored`. +- Once fresh seeds settle, run the long sweep nightly from + `test/route-sweep.sh` with a fresh `MOQ_ROUTE_SEED`, and drop its line from + `sh/gh/test-inventory.allow`. +- The [epic](/quest/m3/cache-routing/README.md)'s items 2, 3, 9, and 10 + describe the one-hop warm route these rules replace; its item 14 records + the replacement. ## Related diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index cf67f9e5d64..0c244a6744d 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -30,7 +30,7 @@ Decided 2026-11-08 (maintainer, /quest-plan): cache-dependent route prices. (rejected: per-broadcast forwarding with a cache-only subscribe and no advertisement; keeping routing cache-agnostic) 2. ✅ **A warm route is per neighbor and travels one hop.** Reopened by - item 13. It goes only to + item 13; replaced by item 14. It goes only to direct neighbors and is never forwarded. B is warm toward N only while B has demand besides N's (a local reader or another downstream session), and withdraws once that demand has been gone for the copy's existing @@ -40,7 +40,7 @@ Decided 2026-11-08 (maintainer, /quest-plan): warm because it still feeds N, and N never skips it again. (rejected: broadcast-wide warmth, as the pre-#3225 design had; forwarding warmth k hops, which floods every warm transition) -3. ✅ **One cost per link, priced marginally.** Pulling through a warm +3. ✅ **One cost per link, priced marginally.** Replaced by item 14. Pulling through a warm neighbor costs the link alone, since that neighbor's cold metric is already paid; a cold neighbor costs the link plus its cold metric. A tie goes to the warm route, then to item 4's hash. A topology that wants both @@ -94,7 +94,7 @@ Decided 2026-11-10 (maintainer, /quest-iterate on [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)), from the proof's proposals: -9. ✅ **Each pull names whom it serves**, amending item 2: a trailing +9. ✅ **Each pull names whom it serves**, amending item 2; replaced by item 14: a trailing `Serves` field on SUBSCRIBE and SUBSCRIBE_UPDATE. (rejected: no field) "Demand besides N's" can't be told from one hop: if B's only demand is a relay pulling on N's behalf, B counts warm toward N, N joins B, B's demand @@ -105,7 +105,7 @@ proof's proposals: narrowed warm route stands until the copy's linger runs out, as the withdrawal already does. 10. ✅ **Price a dependent copy at its cost, and move only on strict - improvement**, amending item 3; the Goal's "cheapest cold route" became + improvement**, amending item 3; replaced by item 14; the Goal's "cheapest cold route" became "cheapest route given the copies the others hold". (rejected: dropping mutual warmth so only a relay with a local reader advertises, which converges trivially but loses the LAN sharing that feeding two sessions @@ -121,7 +121,7 @@ proof's proposals: narrowed route standing while two relays swap between them. The sweep names them in `UNSETTLED` and checks each still never settles, and the [proof's rewrite](/quest/m2/cache-routing-proof.md) settles them first. -11. ✅ **The wire is in moq-lite-07 itself**, overturning the proof's choice +11. ✅ **The wire is in moq-lite-07 itself** (its warm fields replaced by item 14), overturning the proof's choice of a SETUP extension. (rejected: an additive Warm Routes SETUP extension) Lite-07 is unpublished, so its shape can still change until the cut, and a later change goes to the next version. The draft's @@ -180,6 +180,20 @@ Measured by the proof: [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457) proves items 2 and 4 as first decided. +14. **The proven rule** ([proof](/quest/m2/cache-routing-proof.md), + [moq-dev/moq#8615](https://github.com/moq-dev/moq/pull/8615)), on the + path prototype with item 27's routes. A warm route offers a copy's Hops, + its Warm Cost (0 at its carrier, the relay with a reader where it left the + route; else the cost of the sender's own pull), and its carrier. A relay + pulls through the cheapest of its next hop and the copies whose Hops name + neither it nor a neighbor it serves, offers its copy to every neighbor + even one the Hops name, moves between copies only toward a carrier with a + lower Rendezvous Hash, and, pulling along its route, is bound by the + carriers of the neighbors behind it. `Serves` is gone. The pulls are free + of loops once the copies offered are delivered, which the maintainer + accepted on 2026-11-13. The proof's remaining work is a few scenarios + under fresh seeds that never settle. + Prior art: warm/cold route prices ([moq-dev/moq#2925](https://github.com/moq-dev/moq/pull/2925)) were deleted by prefix routes ([moq-dev/moq#3225](https://github.com/moq-dev/moq/pull/3225)), diff --git a/rs/moq-net/src/lite/announce.rs b/rs/moq-net/src/lite/announce.rs index 1efe5edbb8d..0405a720205 100644 --- a/rs/moq-net/src/lite/announce.rs +++ b/rs/moq-net/src/lite/announce.rs @@ -1,3 +1,5 @@ +use std::sync::Arc; + use num_enum::{IntoPrimitive, TryFromPrimitive}; use crate::{ @@ -8,7 +10,7 @@ use crate::{ }; use super::{ - Message, Serves, Version, + Message, Version, message::{MAX_MESSAGE_SIZE, decode_size}, }; @@ -118,33 +120,59 @@ pub enum AnnounceBroadcast { Floor { id: u64, revision: Option }, /// WARM_START (lite-07+): under the route with Route ID `route`, the sender keeps its /// copy of the broadcast at `suffix` whether or not the receiver pulls through it, and - /// assigns it the next Warm ID. `serves` is never [`Serves::Unknown`]. + /// assigns it the next Warm ID. See [`Warm`] for what the copy carries. WarmStart { route: u64, suffix: Path, instance: Instance, - serves: Serves, - cost: u64, + warm: Warm, }, - /// WARM_UPDATE (lite-07+): the warm route with this id serves `serves` at `cost` now. - WarmUpdate { id: u64, serves: Serves, cost: u64 }, + /// WARM_UPDATE (lite-07+): the warm route with this id carries `warm` now. + WarmUpdate { id: u64, warm: Warm }, /// WARM_END (lite-07+): the sender withdrew the warm route with this id. WarmEnd { id: u64 }, } -/// Encode whom a warm route's copy serves, which it always names. -fn encode_warm_serves(w: &mut Encoder<'_>, serves: Serves) -> Result<(), EncodeError> { - match serves { - Serves::Unknown => Err(EncodeError::InvalidState), - serves => serves.encode(w), +/// A copy of a broadcast a warm route offers. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct Warm { + /// What pulling the copy through the sender costs it: 0 at the copy's carrier, else + /// its own pull's cost, in the units of the route's metric. + pub cost: u64, + /// The position in `hops` of the carrier, the relay with a reader where the copy left + /// its origin's route. + pub carrier: u64, + /// Every node the copy crossed, the origin node first and the sender last. + pub hops: Arc<[route::Id]>, +} + +/// Encode a warm route's cost, carrier, and hops. No relay offers a copy that crossed too +/// many hops, or whose carrier is not among them. +fn encode_warm(w: &mut Encoder<'_>, warm: &Warm) -> Result<(), EncodeError> { + if warm.hops.len() > route::MAX_HOPS || warm.carrier >= warm.hops.len() as u64 { + return Err(EncodeError::InvalidState); + } + w.varint(warm.cost)?; + w.varint(warm.carrier)?; + w.varint(warm.hops.len() as u64)?; + for hop in warm.hops.iter() { + w.varint(*hop)?; } + Ok(()) } -fn decode_warm_serves(r: &mut Decoder<'_>) -> Result { - match Serves::decode(r)? { - Serves::Unknown => Err(DecodeError::InvalidValue), - serves => Ok(serves), +fn decode_warm(r: &mut Decoder<'_>) -> Result { + let cost = r.varint()?; + let carrier = r.varint()?; + let count = r.varint()?; + if count > route::MAX_HOPS as u64 { + return Err(DecodeError::TooMany); } + if carrier >= count { + return Err(DecodeError::InvalidValue); + } + let hops = (0..count).map(|_| r.varint()).collect::>()?; + Ok(Warm { cost, carrier, hops }) } /// ROUTE_START and ROUTE_UPDATE flag: the route is down-only. @@ -450,19 +478,16 @@ impl Encode for AnnounceBroadcast { route, suffix, instance, - serves, - cost, + warm, } => { w.varint(*route)?; suffix.encode(w, version)?; super::instance::encode_copy(w, instance)?; - encode_warm_serves(w, *serves)?; - w.varint(*cost)?; + encode_warm(w, warm)?; } - Self::WarmUpdate { id, serves, cost } => { + Self::WarmUpdate { id, warm } => { w.varint(*id)?; - encode_warm_serves(w, *serves)?; - w.varint(*cost)?; + encode_warm(w, warm)?; } Self::WarmEnd { id } => w.varint(*id)?, Self::Ended { .. } | Self::Skipped => unreachable!("refused above"), @@ -596,13 +621,11 @@ impl<'a> Decode<'a, Version> for AnnounceBroadcast { route: body.varint()?, suffix: Path::decode(&mut body, version)?, instance: super::instance::decode_copy(&mut body)?, - serves: decode_warm_serves(&mut body)?, - cost: body.varint()?, + warm: decode_warm(&mut body)?, }, WARM_UPDATE if version.has_warm() => Self::WarmUpdate { id: body.varint()?, - serves: decode_warm_serves(&mut body)?, - cost: body.varint()?, + warm: decode_warm(&mut body)?, }, WARM_END if version.has_warm() => Self::WarmEnd { id: body.varint()? }, // Unknown types are skipped by length so an earlier Lite06 build @@ -1256,9 +1279,9 @@ mod tests { } /// The draft's warm route layouts, byte for byte: WARM_START is the Route ID, the path - /// suffix, the instance and its minted flag, whom the copy serves, and the cost; - /// WARM_UPDATE the Warm ID, serves, and cost; WARM_END the Warm ID. A warm route always - /// names whom it serves, and lite-06 skips the types as unknown. + /// suffix, the instance and its minted flag, then the copy's cost, its carrier's + /// position in its hops, and the hops; WARM_UPDATE the Warm ID and the same three; + /// WARM_END the Warm ID. Lite-06 skips the types as unknown. #[test] fn lite07_warm_route_layout() { let encode = |msg: &AnnounceBroadcast, version: Version| { @@ -1266,22 +1289,22 @@ mod tests { msg.encode(&mut Encoder::new(&mut buf, version.into()), version) .map(|()| buf) }; + let warm = Warm { + cost: 4, + carrier: 1, + hops: Arc::from([5, 9, 6]), + }; let start = AnnounceBroadcast::WarmStart { route: 3, suffix: Path::from_static("c"), instance: "a".parse().unwrap(), - serves: Serves::Relay(9), - cost: 4, - }; - let update = AnnounceBroadcast::WarmUpdate { - id: 7, - serves: Serves::Alone, - cost: 4, + warm: warm.clone(), }; + let update = AnnounceBroadcast::WarmUpdate { id: 7, warm }; let end = AnnounceBroadcast::WarmEnd { id: 7 }; let layouts: [(&AnnounceBroadcast, &[u8]); 3] = [ - (&start, &[12, 9, 3, 1, b'c', 1, b'a', 0, 2, 9, 4]), - (&update, &[13, 3, 7, 1, 4]), + (&start, &[12, 12, 3, 1, b'c', 1, b'a', 0, 4, 1, 3, 5, 9, 6]), + (&update, &[13, 7, 7, 4, 1, 3, 5, 9, 6]), (&end, &[14, 1, 7]), ]; for (msg, wire) in layouts { @@ -1292,20 +1315,39 @@ mod tests { crate::coding::decode_buf(&mut &wire[..], Version::Lite06, |r, v| AnnounceBroadcast::decode(r, v)); assert_eq!(got.unwrap(), AnnounceBroadcast::Skipped); } + } - let unknown = AnnounceBroadcast::WarmUpdate { + /// A warm route's carrier is one of its hops, of which there are at most 32. + #[test] + fn lite07_warm_route_names_its_carrier_among_its_hops() { + let outside = AnnounceBroadcast::WarmUpdate { id: 7, - serves: Serves::Unknown, - cost: 4, + warm: Warm { + cost: 4, + carrier: 3, + hops: Arc::from([5, 9, 6]), + }, }; + let mut buf = Vec::new(); assert!(matches!( - encode(&unknown, Version::Lite07), + outside.encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07), Err(EncodeError::InvalidState) )); let decode = |wire: &[u8]| { crate::coding::decode_buf(&mut &wire[..], Version::Lite07, |r, v| AnnounceBroadcast::decode(r, v)) }; - assert!(decode(&[13, 3, 7, 0, 4]).is_err()); + assert!(decode(&[13, 4, 7, 4, 0, 0]).is_err(), "no hops, so no carrier"); + assert!( + decode(&[13, 7, 7, 4, 3, 3, 5, 9, 6]).is_err(), + "a carrier past the hops" + ); + let mut long = vec![13, 37, 7, 4, 0, 33]; + long.extend([1; 33]); + assert!(decode(&long).is_err(), "33 hops"); + long[1] = 36; + long[5] = 32; + long.pop(); + decode(&long).unwrap(); } /// Lite-07 carries a lite-06 chain after the cost, only when it is not empty, so an diff --git a/rs/moq-net/src/lite/subscribe.rs b/rs/moq-net/src/lite/subscribe.rs index 3b1d83307ff..4ba90ab9ef6 100644 --- a/rs/moq-net/src/lite/subscribe.rs +++ b/rs/moq-net/src/lite/subscribe.rs @@ -120,62 +120,6 @@ impl Range { } } -/// Whom a pull serves, which a relay choosing a parent per broadcast prices warm routes by. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum Serves { - /// The subscriber does not say. - Unknown, - /// Demand of the subscriber's own: a local reader, or pulls tracing back to two or more - /// relays. - Alone, - /// Demand tracing back to the relay with this Node ID alone. - Relay(u64), -} - -impl Serves { - pub(super) fn encode(self, w: &mut Encoder<'_>) -> Result<(), EncodeError> { - match self { - Self::Unknown => w.varint(0), - Self::Alone => w.varint(1), - Self::Relay(0) => Err(EncodeError::InvalidState), - Self::Relay(node) => { - w.varint(2)?; - w.varint(node) - } - } - } - - pub(super) fn decode(r: &mut Decoder<'_>) -> Result { - Ok(match r.varint()? { - 0 => Self::Unknown, - 1 => Self::Alone, - 2 => match r.varint()? { - 0 => return Err(DecodeError::InvalidValue), - node => Self::Relay(node), - }, - _ => return Err(DecodeError::InvalidValue), - }) - } -} - -/// Encode whom a pull serves. This endpoint chooses no parent per broadcast, so it does not -/// say. -fn encode_serves(w: &mut Encoder<'_>, version: Version) -> Result<(), EncodeError> { - match version.has_warm() { - true => Serves::Unknown.encode(w), - false => Ok(()), - } -} - -/// Decode whom a pull serves, and ignore it: only a relay choosing a parent per broadcast -/// prices by it, which this endpoint does not do. -fn decode_serves(r: &mut Decoder<'_>, version: Version) -> Result<(), DecodeError> { - if version.has_warm() { - Serves::decode(r)?; - } - Ok(()) -} - impl Version { /// Whether this version's SUBSCRIBE carries the subscriber's max delay preference. /// @@ -209,7 +153,6 @@ impl<'a> Message<'a> for Subscribe<'a> { } }; let update = decode_update(r, version)?; - decode_serves(r, version)?; Ok(Self { id, @@ -245,7 +188,6 @@ impl<'a> Message<'a> for Subscribe<'a> { } } encode_update(w, version, self.update)?; - encode_serves(w, version)?; Ok(()) } @@ -820,7 +762,6 @@ impl<'a> Message<'a> for SubscribeUpdate { true => r.varint()?.checked_sub(1), false => None, }; - decode_serves(r, version)?; Ok(Self { wanted_group, @@ -861,7 +802,7 @@ impl<'a> Message<'a> for SubscribeUpdate { if version.has_refresh() { w.varint_opt(self.wanted_group)?; } - encode_serves(w, version) + Ok(()) } } @@ -1945,10 +1886,10 @@ mod test { msg.ranges = vec![Range::OLDEST]; let mut buf = encode(&msg, version).unwrap(); - // The list is last but for the update number and whom the pull serves: a count of one, - // then two anchors of type 0. - let at = buf.len() - 5; - assert_eq!(&buf[at..], [1, 0, 0, 0, 0]); + // The list is last but for the update number: a count of one, then two anchors of + // type 0. + let at = buf.len() - 4; + assert_eq!(&buf[at..], [1, 0, 0, 0]); buf[at] = 0; buf.truncate(at + 1); assert!(matches!( @@ -1957,7 +1898,7 @@ mod test { )); let mut buf = encode(&msg, version).unwrap(); - let last = buf.len() - 3; + let last = buf.len() - 2; buf[last] = 4; assert!(matches!( crate::coding::decode_from(&mut buf.as_slice(), version, Subscribe::decode_msg), @@ -2011,27 +1952,6 @@ mod test { ); } - /// Lite-07 ends SUBSCRIBE with whom the pull serves: this endpoint says Unknown, and reads - /// and ignores any well-formed value. An unknown type or a relay named 0 is malformed, and - /// older versions carry nothing. - #[test] - fn serves_trails_subscribe_only_on_lite07() { - let msg = subscribe_sample(); - let lite07 = encode(&msg, Version::Lite07).unwrap(); - assert_eq!(lite07.last(), Some(&0)); - let head = &lite07[..lite07.len() - 1]; - let decode = |serves: &[u8]| { - let buf = [head, serves].concat(); - crate::coding::decode_from(&mut buf.as_slice(), Version::Lite07, Subscribe::decode_msg).map(|_| ()) - }; - for serves in [&[0][..], &[1], &[2, 9]] { - decode(serves).unwrap(); - } - for serves in [&[3][..], &[2, 0], &[]] { - assert!(decode(serves).is_err(), "{serves:?}"); - } - } - /// Lite-07 carries `Wanted Group` on SUBSCRIBE_UPDATE; older versions drop it. #[test] fn wanted_group_roundtrips_only_on_lite07() { diff --git a/rs/moq-net/src/lite/version.rs b/rs/moq-net/src/lite/version.rs index d5887ed8e72..03e35385345 100644 --- a/rs/moq-net/src/lite/version.rs +++ b/rs/moq-net/src/lite/version.rs @@ -398,8 +398,7 @@ impl Version { } /// Whether the announce stream carries warm routes (WARM_START, WARM_UPDATE, and - /// WARM_END), and SUBSCRIBE and SUBSCRIBE_UPDATE say whom each pull serves. Added in - /// lite-07. + /// WARM_END). Added in lite-07. pub(crate) fn has_warm(self) -> bool { !matches!( self, diff --git a/rs/moq-net/src/route/mod.rs b/rs/moq-net/src/route/mod.rs index c905dc0bf29..c1b36c31417 100644 --- a/rs/moq-net/src/route/mod.rs +++ b/rs/moq-net/src/route/mod.rs @@ -221,20 +221,6 @@ pub(crate) struct Selected { pub down_only: bool, } -/// A usable route a request for one of an identity's broadcasts may go through. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub(crate) struct Eligible { - pub next: N, - /// This node's metric through it. - pub metric: Metric, - /// This node's RTT through it, when known. - pub rtt: Option, - /// Advertised at exactly this node's feasibility distance: a request goes through - /// it only when the neighbor's rendezvous hash for the broadcast is below this - /// node's, so the hash orders the tie and no two tied nodes pull through each other. - pub tied: bool, -} - /// How a node's reader sees one announced claim. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) struct Shown { @@ -1104,7 +1090,7 @@ impl Router { type Rank = (bool, Reverse, Metric, N); let mut best: Option<(Rank, Selected)> = None; for (&next, entry) in self.rx.get(&id)? { - if entry.held || !self.selectable(id, entry) { + if entry.held || !self.eligible(id, entry) { continue; } let peer = &self.neighbors[&next]; @@ -1126,29 +1112,18 @@ impl Router { best.map(|(_, selected)| selected) } - /// Every usable route to `id` a request may go through without a loop, if it - /// [carries](Self::carries) the claim: each feasible one, and each tied with this - /// node's feasibility distance. - pub fn eligible(&self, id: Id) -> Vec> { + /// Every usable feasible route to `id`, with this node's metric and RTT through + /// each. Any of them forwards without a loop, if it [carries](Self::carries) the + /// claim. + pub fn routes(&self, id: Id) -> Vec<(N, Metric, Option)> { if self.origins.contains_key(&id) { return Vec::new(); } - let fd = self.fd.get(&id); (self.rx.get(&id).into_iter().flatten()) - .filter(|(_, entry)| !entry.held && !entry.crossed) + .filter(|(_, entry)| !entry.held && self.eligible(id, entry)) .filter_map(|(&next, entry)| { - let tied = fd.is_some_and(|&fd| fd == (entry.advert.seqno, entry.advert.metric)); - if !tied && !self.feasible(id, &entry.advert) { - return None; - } - let metric = self.metric(next, entry)?; let rtt = through_rtt(&entry.advert, &self.neighbors[&next]); - Some(Eligible { - next, - metric, - rtt, - tied, - }) + Some((next, self.metric(next, entry)?, rtt)) }) .collect() } @@ -1311,7 +1286,7 @@ impl Router { /// Whether this node may select or certify through `entry`: a feasible route that /// did not cross it. One that crossed it and is feasible all the same is no route /// at all, since no seqno would change where it has been. - fn selectable(&self, id: Id, entry: &Entry) -> bool { + fn eligible(&self, id: Id, entry: &Entry) -> bool { !entry.crossed && self.feasible(id, &entry.advert) } @@ -1387,7 +1362,7 @@ impl Router { /// Whether `entry`'s last finite advertisement from `next` is feasible here /// and reachable across the link, as selection requires of a usable route. fn reachable(&self, id: Id, next: N, entry: &Entry) -> bool { - self.selectable(id, entry) && self.metric(next, entry).is_some() + self.eligible(id, entry) && self.metric(next, entry).is_some() } /// The regions of `id` that this node's certifiers certify, those whose diff --git a/rs/moq-net/src/route/node.rs b/rs/moq-net/src/route/node.rs index 366d873e697..8b202c32a18 100644 --- a/rs/moq-net/src/route/node.rs +++ b/rs/moq-net/src/route/node.rs @@ -970,10 +970,8 @@ impl State { /// there is none. Only `claims`, whose shown state changed, are served anew, unless /// the routes changed too. fn home(&mut self, id: Id, claims: BTreeSet, ended: &BTreeMap<(Id, Path), bool>) { - // A tied route needs a broadcast's hash, which the claim ranking here lacks. - let routes: Routes = (self.router.eligible(id).into_iter()) - .filter(|route| !route.tied) - .map(|route| (route.next, route.metric, route.rtt, self.router.name(route.next, id))) + let routes: Routes = (self.router.routes(id).into_iter()) + .map(|(key, metric, rtt)| (key, metric, rtt, self.router.name(key, id))) .collect(); let last = self.routes.get(&id).map_or(&[][..], Vec::as_slice); let claims = match last == routes.as_slice() { diff --git a/rs/moq-net/src/route/tests/cache.rs b/rs/moq-net/src/route/tests/cache.rs index 216c0b9106f..fb667831552 100644 --- a/rs/moq-net/src/route/tests/cache.rs +++ b/rs/moq-net/src/route/tests/cache.rs @@ -1,176 +1,97 @@ -//! Cache-aware routing over the route layer, as `quest/m3/cache-routing/README.md` -//! decides it. +//! Cache-aware routing on the path prototype in `paths.rs`, as +//! `quest/m3/cache-routing/README.md` decides it: a warm route is per broadcast, +//! propagates, and is kept free of loops by its path, with what +//! `quest/m2/cache-routing-proof.md` found a path needs besides. //! -//! [`Mesh`] adds per-broadcast demand and forwarding to [`Sim`]: readers join and -//! leave, and each relay pulls a broadcast it has demand for through one neighbor, -//! its parent. A relay picks the cheapest neighbor its eligibility rule allows -//! ([`Router::eligible`], a tied route only toward the lower rendezvous hash), where -//! a warm neighbor costs the link alone and a cold one the link plus its cold -//! metric; ties go to warm, then to the lower hash. A copy with no demand left -//! lingers. +//! A relay with a reader whose pull follows the publisher's route offers its copy +//! at no cost, as its carrier, with the path the copy took. Each relay pulls a +//! broadcast it has demand for (a reader, or a neighbour's subscription) through +//! the cheapest of its next hop toward the publisher and each copy offered, and +//! offers a copy it joined on at what its pull costs, so a copy's cost rises with +//! every link from where it left the publisher's route. Pulling through a +//! neighbour goes wherever that neighbour pulls, so each relay offers every +//! neighbour its copy, even one whose path names that neighbour, which can then +//! use neither. A relay on one copy moves onto another only if that copy's carrier +//! ranks below by the Rendezvous Hash, so copies crossing on their way never trade +//! places; a relay pulling along its route is held to the carriers of the +//! neighbours behind it, which those neighbours' offers back name. A copy with no +//! demand left lingers. //! -//! Every pull names whom it serves ([`Serves`]): a pull whose whole demand traces -//! back to one relay names that relay, its root. A carrier tells each neighbor whose -//! cold metric is at least its own that it is warm, naming whom its copy serves, and -//! the neighbor counts it warm unless that is the neighbor's own root or itself: -//! demand a neighbor causes, through any number of relays, never warms a carrier -//! toward it. A carrier narrows what it tells only once the linger runs out. -//! -//! Each relay recomputes its choices in the same step as its router changes, as the -//! router's own node does, and every step checks that each broadcast's parents form -//! no cycle. Warm routes ride the stream carrying the sender's routes, so a -//! ROUTE_END retires them; subscriptions ride streams of their own, unordered -//! against both. Lingers run out only when a fixture or sweep says time passed. +//! [`Mesh`] adds what a sweep checks to [`PathSim`]: after every step, whether a +//! broadcast's pulls form a cycle, which the path allows only while a withdrawal +//! crosses a link; once quiet, every reader is fed and nothing would change. +//! Subscriptions ride streams of their own, unordered against the routes and +//! copies. +use super::paths::{Cast, PathSim, Rules, rendezvous}; use super::*; -/// A path its origin serves under one instance: two broadcasts sharing a path -/// and origin are two instances of it. -#[derive(Clone, Copy, Debug)] -struct Broadcast { - path: &'static str, - origin: Node, -} - -/// Whom a pull or a copy serves. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] -pub(super) enum Serves { - /// Demand of its own: a reader, or pulls tracing back to two or more relays. - #[default] - Alone, - /// Demand that traces back to this one relay alone. - Relay(Node), - /// Demand some of which came from a relay that names no one. - Unknown, -} +/// The rule item 27 of the routes epic picked: holds, causes, and a doubt that lifts +/// after 4 hops of delay. +const PICK: Rules = Rules::waits(4); -/// A warm route: whom the sender's copy serves, and what keeping it costs a -/// receiver it serves alone. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub(super) struct Warm { - pub serves: Serves, - /// The sender's price for its own pull, the cost of the copy to the demand it serves. - pub cost: Metric, -} - -/// One node's state for one broadcast. -#[derive(Clone, Debug, Default)] -struct Carry { - reader: bool, - /// Neighbors whose subscription arrived and was not cancelled, and whom each serves. - downstream: BTreeMap, - /// The neighbor this node pulls the broadcast through. - parent: Option, - /// Whom this node's pull serves, as it last told its parent. - serves: Serves, - /// Demand is gone and the copy lingers. - lingering: bool, - /// What this node advertises warm, or `None` when cold. - advert: Option, - /// The advert is narrower than it was, and the old one stands until the linger - /// runs out. - fading: bool, - /// What each neighbor was last told. - told: BTreeMap, - /// Each neighbor's warm route that arrived. - warm: BTreeMap, -} - -/// What the cache layer sent and moved. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] -pub(super) struct CacheStats { - /// Warm route starts, updates, and ends. - pub warm: usize, - /// Subscriptions, updates, and cancels. - pub subscribes: usize, - /// A relay moved a copy from one parent to another. - pub moves: usize, +/// A queue the mesh delivers from. +#[derive(Clone, Debug)] +enum Queue { + Route(Node, Node), + /// One subscription stream: from, to, broadcast, stream. + Sub(Node, Node, Cast, u64), } pub(super) struct Mesh { - pub sim: Sim, - broadcasts: Vec, - carry: Vec>, - /// Subscriptions, each broadcast's on a stream of its own. - subs: BTreeMap<(Node, Node, usize), VecDeque>, - /// `(node, neighbor)`: the neighbor sent `node` GOAWAY. - draining: BTreeSet<(Node, Node)>, - /// Relays that never negotiated warm routes: each pulls through its selected - /// route, sends no warm route, and its pulls name no one. - legacy: BTreeSet, - pub stats: CacheStats, - /// Warm messages per broadcast. - warm: Vec, -} - -/// The draft's Rendezvous Hash that orders a tie, by node index standing in for the Node ID. -fn hash(path: &str, node: Node) -> u64 { - let mut hash: u64 = 0x420C_0DEC_B00B; - for byte in path.bytes().chain((node as u64).to_le_bytes()) { - hash = (hash ^ u64::from(byte)).wrapping_mul(0x0100_0000_01b3); - } - hash -} - -/// A change to one subscription. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -enum Sub { - /// Pull, or update whom the pull serves. - Pull(Serves), - Cancel, -} - -/// A neighbor a node may pull a broadcast through. -#[derive(Clone, Copy, Debug)] -struct Candidate { - next: Node, - /// The node's metric through it. - cold: Metric, - /// The link to it alone, what pulling through it costs when it is warm. - link: Metric, - draining: bool, + pub sim: PathSim, + /// Each publisher's node. + origins: BTreeMap, + casts: Vec, + /// Steps after which some broadcast's pulls formed a cycle. + pub loops: usize, + pub steps: usize, + /// Of those, steps after which a reader's pulls led into the cycle. + pub starved: usize, + /// The most deliveries in a row after which a cycle remained, and the current run. + pub longest: usize, + run: usize, + delivered: bool, } impl Mesh { - pub fn new(nodes: usize) -> Self { + pub fn new(nodes: usize, rules: Rules) -> Self { Self { - sim: Sim::new(nodes), - broadcasts: Vec::new(), - carry: vec![Vec::new(); nodes], - subs: BTreeMap::new(), - draining: BTreeSet::new(), - legacy: BTreeSet::new(), - stats: CacheStats::default(), - warm: Vec::new(), + sim: PathSim::new(nodes, rules), + origins: BTreeMap::new(), + casts: Vec::new(), + loops: 0, + steps: 0, + starved: 0, + longest: 0, + run: 0, + delivered: false, } } fn nodes(&self) -> usize { - self.carry.len() + self.sim.nodes.len() } - /// A publisher at `node`, a leaf that originates `id` with a claim covering - /// every broadcast path. Called before its links, which a leaf configures. + /// A publisher at `node`, a leaf that originates `id` and claims `live`. Called + /// before its links. pub fn origin(&mut self, node: Node, id: Id) { self.sim.leaf(node); - self.sim.originate(node, id, 0); - self.sim.announce(node, "live/", 0, None); - self.step(); + self.sim.originate(node, id); + self.sim.announce(node, "live"); + self.origins.insert(id, node); } - /// `node` never negotiates warm routes. + /// `node` never sends or reads a copy. pub fn legacy(&mut self, node: Node) { - self.legacy.insert(node); + self.sim.nodes[node].legacy = true; } - /// A broadcast at `path` served by the origin at `origin`. - pub fn broadcast(&mut self, path: &'static str, origin: Node) -> usize { - self.broadcasts.push(Broadcast { path, origin }); - for carry in &mut self.carry { - carry.push(Carry::default()); - } - self.warm.push(0); - self.broadcasts.len() - 1 + /// A broadcast at `path` of the publisher `id`, under `instance`. + pub fn broadcast(&mut self, id: Id, path: &str, instance: u64) -> Cast { + let cast = (id, p(path), instance); + self.casts.push(cast.clone()); + cast } pub fn join(&mut self, a: Node, b: Node, cost: Metric) { @@ -178,52 +99,34 @@ impl Mesh { self.step(); } - /// A reader at `node` starts or stops reading `broadcast`. - pub fn read(&mut self, node: Node, broadcast: usize, reading: bool) { - self.carry[node][broadcast].reader = reading; - self.step(); - } - - pub fn reading(&self, node: Node, broadcast: usize) -> bool { - self.carry[node][broadcast].reader - } - pub fn connect(&mut self, a: Node, b: Node) { self.sim.connect(a, b); self.step(); } - /// The session closes: everything on it goes, subscriptions and warm routes too. pub fn disconnect(&mut self, a: Node, b: Node) { self.sim.disconnect(a, b); - for (x, y) in [(a, b), (b, a)] { - self.subs.retain(|&(from, to, _), _| (from, to) != (x, y)); - self.draining.remove(&(x, y)); - for carry in &mut self.carry[x] { - carry.downstream.remove(&y); - carry.told.remove(&y); - carry.warm.remove(&y); - } - } self.step(); } - /// `node` sends GOAWAY on every session: its neighbors rank it last. pub fn goaway(&mut self, node: Node) { - let peers: Vec = self.sim.peers(node).collect(); - for peer in peers { - self.draining.insert((peer, node)); - } self.sim.goaway(node); self.step(); } - /// The origin at `node` advances its seqno. - pub fn bump(&mut self, node: Node) { - self.sim.bump(node); + /// A reader at `node` starts or stops reading `cast`. + pub fn read(&mut self, node: Node, cast: &Cast, reading: bool) { + match reading { + true => self.sim.carry(node, cast), + false => self.sim.uncarry(node, cast), + } self.step(); } + pub fn reading(&self, node: Node, cast: &Cast) -> bool { + self.sim.nodes[node].reading(cast) + } + pub fn hold(&mut self, from: Node, to: Node) { self.sim.hold(from, to); } @@ -232,574 +135,333 @@ impl Mesh { self.sim.release(from, to); } - /// The linger at `node` for `broadcast` runs out: a copy with no demand goes, and - /// a narrowed advert takes effect. - pub fn tick(&mut self, node: Node, broadcast: usize) { - let carry = &mut self.carry[node][broadcast]; - let parent = match carry.lingering { - true => { - carry.lingering = false; - carry.parent.take() - } - false => None, - }; - if carry.fading { - carry.fading = false; - self.carry[node][broadcast].advert = self.desired(node, broadcast); - } - if let Some(parent) = parent { - self.subscribe(node, parent, broadcast, Sub::Cancel); - } + /// The linger for `cast` at `node` runs out. + pub fn expire(&mut self, node: Node, cast: &Cast) { + self.sim.expire(node, cast); self.step(); } - /// Deliver the next message on `from`'s route stream to `to`. - pub fn deliver(&mut self, from: Node, to: Node) -> bool { - let delivered = self.sim.deliver(from, to); + /// One hop of delay passes everywhere. Whether a doubt still waits to lift. + pub fn tick(&mut self) -> bool { + let waiting = self.sim.tick(); self.step(); - delivered + waiting } - pub fn deliver_ack(&mut self, from: Node, to: Node) -> bool { - let delivered = self.sim.deliver_ack(from, to); + pub fn deliver_sub(&mut self, from: Node, to: Node, cast: &Cast) -> bool { + let delivered = self.sim.deliver_sub(from, to, cast); self.step(); delivered } - /// Deliver the next subscription change for `broadcast` from `from` to `to`. - pub fn deliver_sub(&mut self, from: Node, to: Node, broadcast: usize) -> bool { - let Some(sub) = self.subs.get_mut(&(from, to, broadcast)).and_then(VecDeque::pop_front) else { - return false; - }; - self.subs.retain(|_, queue| !queue.is_empty()); - let downstream = &mut self.carry[to][broadcast].downstream; - match sub { - Sub::Pull(serves) => downstream.insert(from, serves), - Sub::Cancel => downstream.remove(&from), - }; + pub fn deliver_stream(&mut self, from: Node, to: Node, cast: &Cast, stream: u64) -> bool { + let delivered = self.sim.deliver_stream(from, to, cast, stream); self.step(); - true + delivered + } + + /// The subscription streams from `from` to `to` with a change to deliver, oldest first. + pub fn streams(&self, from: Node, to: Node) -> Vec { + (self.sim.pending_subs().into_iter()) + .filter(|(f, t, ..)| (*f, *t) == (from, to)) + .map(|(.., stream)| stream) + .collect() } - /// Every queue with something to deliver: routes, echoes, then subscriptions. + /// Every unheld queue with something to deliver. fn busy(&self) -> Vec { (self.sim.busy().into_iter().map(|(from, to)| Queue::Route(from, to))) - .chain(self.sim.busy_acks().into_iter().map(|(from, to)| Queue::Ack(from, to))) - .chain(self.subs.keys().map(|&(from, to, b)| Queue::Sub(from, to, b))) + .chain( + (self.sim.busy_subs().into_iter()).map(|(from, to, cast, stream)| Queue::Sub(from, to, cast, stream)), + ) .collect() } - fn deliver_one(&mut self, queue: Queue) { + fn deliver_one(&mut self, queue: &Queue) { + self.delivered = true; match queue { - Queue::Route(from, to) => self.deliver(from, to), - Queue::Ack(from, to) => self.deliver_ack(from, to), - Queue::Sub(from, to, b) => self.deliver_sub(from, to, b), + Queue::Route(from, to) => self.sim.deliver(*from, *to), + Queue::Sub(from, to, cast, stream) => self.sim.deliver_stream(*from, *to, cast, *stream), }; + self.step(); } /// Deliver round-robin until every unheld queue is empty, letting no linger run - /// out. False if it never settles. + /// out and no time pass. False if it never empties. pub fn run(&mut self) -> bool { - for _ in 0..10_000 { - let ready: Vec = (self.busy().into_iter()) - .filter(|queue| !self.sim.held.contains(&queue.pair())) - .collect(); + self.rounds(10_000) + } + + /// Deliver up to `rounds` rounds, one message from each unheld queue per round. + /// Whether every unheld queue emptied. + fn rounds(&mut self, rounds: usize) -> bool { + for _ in 0..rounds { + let ready = self.busy(); if ready.is_empty() { return true; } - for queue in ready { + for queue in &ready { self.deliver_one(queue); } } false } - /// Run, then let every linger run out, until nothing changes. + /// Run, let time pass until every doubt lifts, and let every linger run out, + /// until nothing changes. A round of deliveries takes a hop of delay, so a long + /// exchange lets time pass too, if more slowly. False if it never settles. pub fn settle(&mut self) -> bool { - for _ in 0..100 { - if !self.run() { - return false; + for _ in 0..1000 { + if !self.rounds(64) { + self.tick(); + continue; } - let lingering: Vec<(Node, usize)> = (0..self.nodes()) - .flat_map(|node| (0..self.broadcasts.len()).map(move |b| (node, b))) - .filter(|&(node, b)| { - let carry = &self.carry[node][b]; - carry.lingering || carry.fading - }) + if self.tick() { + continue; + } + let lingering: Vec<(Node, Cast)> = (0..self.nodes()) + .flat_map(|node| self.sim.nodes[node].lingers().into_iter().map(move |cast| (node, cast))) .collect(); - if lingering.is_empty() { + if lingering.is_empty() && self.busy().is_empty() { return true; } - for (node, b) in lingering { - self.tick(node, b); + for (node, cast) in lingering { + self.expire(node, &cast); } } false } - pub fn parent(&self, node: Node, broadcast: usize) -> Option { - self.carry[node][broadcast].parent - } - - /// Whether `node` pulls `broadcast`. - pub fn carries(&self, node: Node, broadcast: usize) -> bool { - self.parent(node, broadcast).is_some() + pub fn parent(&self, node: Node, cast: &Cast) -> Option { + self.sim.nodes[node].parent(cast) } - /// Whether `node` counts `from` warm for `broadcast`: its copy stays whether or not - /// `node` pulls through it. - pub fn warm_at(&self, node: Node, from: Node, broadcast: usize) -> bool { - let carry = &self.carry[node][broadcast]; - carry.warm.get(&from).is_some_and(|warm| match warm.serves { - Serves::Alone => true, - Serves::Relay(root) => match carry.serves { - Serves::Unknown => false, - Serves::Alone => root != node, - Serves::Relay(own) => root != node && root != own, - }, - Serves::Unknown => false, - }) + /// Whether `node` pulls `cast`. + pub fn carries(&self, node: Node, cast: &Cast) -> bool { + self.parent(node, cast).is_some() } - /// What pulling `broadcast` through `c` costs `node`: the link alone through a warm - /// neighbor, the link plus the copy's cost through one whose copy serves only - /// `node`'s demand, and the cold metric through any other. - fn price(&self, node: Node, broadcast: usize, c: &Candidate) -> Metric { - match self.carry[node][broadcast].warm.get(&c.next) { - _ if self.warm_at(node, c.next, broadcast) => c.link, - Some(warm) if warm.serves != Serves::Unknown => c.link + warm.cost, - _ => c.cold, - } - } - - /// The relays pulling `broadcast` straight from its origin. - pub fn copies(&self, broadcast: usize) -> usize { - let origin = self.broadcasts[broadcast].origin; + /// The relays pulling `cast` straight from its publisher. + pub fn copies(&self, cast: &Cast) -> usize { + let origin = self.origins[&cast.0]; (0..self.nodes()) - .filter(|&node| self.parent(node, broadcast) == Some(origin)) + .filter(|&node| self.parent(node, cast) == Some(origin)) .count() } - /// Warm messages sent for `broadcast`. - pub fn warm_sent(&self, broadcast: usize) -> usize { - self.warm[broadcast] + /// Copy offers and their ends sent for `cast`. + pub fn warm_sent(&self, cast: &Cast) -> usize { + self.sim.warm.get(cast).copied().unwrap_or(0) } - fn id(&self, broadcast: usize) -> Id { - self.sim.proven(self.broadcasts[broadcast].origin) + /// Whether `node` is offered a copy of `cast` by `from`. + pub fn offered(&self, node: Node, from: Node, cast: &Cast) -> bool { + self.sim.nodes[node].offers(from, cast).1.is_some() } - fn subscribe(&mut self, from: Node, to: Node, broadcast: usize, sub: Sub) { - self.subs.entry((from, to, broadcast)).or_default().push_back(sub); - self.stats.subscribes += 1; + /// Count a step after which some broadcast's pulls form a cycle. + fn step(&mut self) { + self.steps += 1; + let cycle = self.cycle().is_some(); + self.loops += usize::from(cycle); + self.run = match cycle { + true => self.run + usize::from(std::mem::take(&mut self.delivered)), + false => 0, + }; + self.delivered = false; + self.longest = self.longest.max(self.run); + self.starved += usize::from(cycle && self.starving()); } - /// Take the deliveries, recompute every node's choices, and check for a loop. - fn step(&mut self) { - for note in std::mem::take(&mut self.sim.notes) { - match note { - Note::Warm { - from, - to, - broadcast, - warm, - } => { - let map = &mut self.carry[to][broadcast].warm; - match warm { - Some(warm) => map.insert(from, warm), - None => map.remove(&from), - }; - } - Note::Ended { from, to, id } => { - for b in 0..self.broadcasts.len() { - if self.id(b) == id { - self.carry[to][b].warm.remove(&from); - } + /// Whether some reader's pulls lead into a cycle. + fn starving(&self) -> bool { + self.casts.iter().any(|cast| { + (0..self.nodes()).filter(|&node| self.reading(node, cast)).any(|node| { + let mut at = node; + for _ in 0..=self.nodes() { + match self.parent(at, cast) { + Some(next) => at = next, + None => return false, } } - } - } - for node in 0..self.nodes() { - for b in 0..self.broadcasts.len() { - self.choose(node, b); - self.tell(node, b); - } - } - self.check_loops(); - } - - /// The neighbors `node` may pull `broadcast` through. - fn candidates(&self, node: Node, broadcast: usize) -> Vec { - let path = self.broadcasts[broadcast].path; - (self.sim.nodes[node].eligible(self.id(broadcast)).into_iter()) - .filter(|route| !route.tied || hash(path, route.next) < hash(path, node)) - .map(|route| Candidate { - next: route.next, - cold: route.metric, - link: self.sim.links[node][route.next].expect("a configured link").cost, - draining: self.draining.contains(&(node, route.next)), - }) - .collect() - } - - /// The neighbor `node` pulls `broadcast` through: its current parent unless another - /// candidate is strictly cheaper, otherwise the cheapest, ties going to warm and - /// then to the lower hash. - fn best(&self, node: Node, broadcast: usize) -> Option { - if self.legacy.contains(&node) { - return self.sim.nodes[node] - .selected(self.id(broadcast)) - .and_then(|route| route.next); - } - let path = self.broadcasts[broadcast].path; - let ranked: Vec<_> = (self.candidates(node, broadcast).into_iter()) - .map(|c| { - let warm = self.warm_at(node, c.next, broadcast); - ( - (c.draining, self.price(node, broadcast, &c)), - !warm, - hash(path, c.next), - c.next, - ) + true }) - .collect(); - let best = ranked.iter().min()?; - let parent = self.carry[node][broadcast].parent; - match ranked.iter().find(|rank| Some(rank.3) == parent) { - Some(current) if current.0 <= best.0 => parent, - _ => Some(best.3), - } - } - - /// What `node` pays for its own pull of `broadcast`. - fn cost(&self, node: Node, broadcast: usize) -> Option { - let parent = self.carry[node][broadcast].parent?; - let c = (self.candidates(node, broadcast).into_iter()).find(|c| c.next == parent)?; - Some(self.price(node, broadcast, &c)) - } - - /// Whom `node`'s demand for `broadcast` serves: itself for a reader, the one root - /// its pulls trace back to, or no one in particular across two or more roots. - fn served(&self, node: Node, broadcast: usize) -> Serves { - let carry = &self.carry[node][broadcast]; - if self.legacy.contains(&node) { - return Serves::Unknown; - } - if carry.reader { - return Serves::Alone; - } - let mut roots = BTreeSet::new(); - let mut unknown = false; - for (&from, &serves) in &carry.downstream { - match serves { - Serves::Alone => roots.insert(from), - Serves::Relay(root) => roots.insert(root), - Serves::Unknown => { - unknown = true; - false - } - }; - } - match (roots.len(), unknown) { - (2.., _) => Serves::Alone, - (1, false) => Serves::Relay(*roots.first().expect("one root")), - _ => Serves::Unknown, - } - } - - /// What `node` would advertise for `broadcast` now: warm while it pulls for demand - /// it can name. - fn desired(&self, node: Node, broadcast: usize) -> Option { - let carry = &self.carry[node][broadcast]; - let demand = carry.reader || !carry.downstream.is_empty(); - if node == self.broadcasts[broadcast].origin || self.legacy.contains(&node) || !demand { - return None; - } - let serves = self.served(node, broadcast); - let cost = self.cost(node, broadcast)?; - (serves != Serves::Unknown).then_some(Warm { serves, cost }) - } - - /// Recompute `node`'s parent for `broadcast`, and move or update the subscription. - fn choose(&mut self, node: Node, broadcast: usize) { - if node == self.broadcasts[broadcast].origin { - return; - } - let carry = &mut self.carry[node][broadcast]; - let demand = carry.reader || !carry.downstream.is_empty(); - if demand { - carry.lingering = false; - } else if carry.parent.is_some() { - carry.lingering = true; - } - let want = demand || carry.lingering; - // A lingering copy serves whom it last served. - let serves = match demand { - true => self.served(node, broadcast), - false => self.carry[node][broadcast].serves, - }; - let parent = match want { - true => self.best(node, broadcast), - false => None, - }; - let carry = &mut self.carry[node][broadcast]; - let (old, told) = (carry.parent, carry.serves); - carry.parent = parent; - carry.serves = serves; - if parent != old - && let Some(old) = old - && self.sim.connected(node, old) - { - self.subscribe(node, old, broadcast, Sub::Cancel); - } - if let Some(parent) = parent - && (Some(parent) != old || serves != told) - { - self.subscribe(node, parent, broadcast, Sub::Pull(serves)); - } - if old.is_some() && parent.is_some() && parent != old { - self.stats.moves += 1; - } - } - - /// Update `node`'s advert for `broadcast`, widening at once and narrowing once the - /// linger runs out, and tell each neighbor that may use it. - fn tell(&mut self, node: Node, broadcast: usize) { - let desired = self.desired(node, broadcast); - let carry = &mut self.carry[node][broadcast]; - // Narrower: some neighbor that counted the copy warm would no longer. - let narrower = match (carry.advert, desired) { - (Some(old), Some(new)) => old.serves != new.serves && new.serves != Serves::Alone, - (Some(_), None) => true, - (None, _) => false, - }; - carry.fading = narrower; - if !narrower { - carry.advert = desired; - } - let advert = carry.advert; - let id = self.id(broadcast); - let peers: Vec = (self.sim.peers(node)) - .filter(|&peer| !self.sim.leaf[peer] && !self.legacy.contains(&peer)) - .collect(); - for to in peers { - let router = &self.sim.nodes[node]; - // A warm route names the route it travels under, which a ROUTE_END retires. - let routed = router.sent.contains_key(&(id, to)); - let own = router.selected(id).map(|route| route.metric); - // A neighbor with no usable route to the origin has an infinite cold metric. - let behind = (router.rx.get(&id).and_then(|routes| routes.get(&to))) - .is_none_or(|entry| entry.held || own.is_some_and(|own| entry.advert.metric >= own)); - let carry = &mut self.carry[node][broadcast]; - if !routed { - carry.told.remove(&to); - } - let want = advert.filter(|_| routed && own.is_some() && behind); - if want == carry.told.get(&to).copied() { - continue; - } - match want { - Some(warm) => carry.told.insert(to, warm), - None => carry.told.remove(&to), - }; - let msg = Msg::Warm { broadcast, warm: want }; - self.sim.queues.entry((node, to)).or_default().push_back(msg); - self.stats.warm += 1; - self.warm[broadcast] += 1; - } + }) } - /// Each broadcast's parents form no cycle. - fn check_loops(&self) { - for b in 0..self.broadcasts.len() { + /// A broadcast whose pulls form a cycle, and each relay's parent for it. + fn cycle(&self) -> Option<(Cast, Vec>)> { + for cast in &self.casts { for node in 0..self.nodes() { let mut at = node; for hop in 0.. { - let Some(next) = self.parent(at, b) else { + let Some(next) = self.parent(at, cast) else { break; }; - assert!( - hop <= self.nodes(), - "broadcast {b} loops from node {node}: {:?}", - (0..self.nodes()).map(|n| self.parent(n, b)).collect::>() - ); + if hop > self.nodes() { + let parents = (0..self.nodes()).map(|n| self.parent(n, cast)).collect(); + return Some((cast.clone(), parents)); + } at = next; } } } + None } - /// The steady state once demand and topology stop changing: every reader is - /// delivered, subscriptions and warm routes match what their senders hold, nothing - /// would change, every carrier has demand, and no relay could pull more cheaply - /// given the copies the others hold. + /// The steady state once demand and topology stop changing: every reader is fed + /// from its publisher, subscriptions and copies offered match what their senders + /// hold, no copy lingers, and each relay pulls exactly what it has demand for, + /// from its cheapest source. So a relay carries a broadcast only for a reader, or + /// for neighbours whose cheapest source it is. pub fn steady(&mut self, label: &str) { - for (b, broadcast) in self.broadcasts.iter().enumerate() { + assert!(self.busy().is_empty(), "{label}: still busy"); + assert_eq!(self.cycle(), None, "{label}"); + for cast in &self.casts { + let origin = self.origins[&cast.0]; for node in 0..self.nodes() { - let carry = &self.carry[node][b]; - if carry.reader { + let relay = &self.sim.nodes[node]; + if relay.reading(cast) { let mut at = node; - while let Some(next) = self.parent(at, b) { + while let Some(next) = self.parent(at, cast) { at = next; } - assert_eq!(at, broadcast.origin, "{label}: broadcast {b} reaches node {node}"); + assert_eq!(at, origin, "{label}: {cast:?} reaches node {node}"); } - let feeds: BTreeMap = (0..self.nodes()) - .filter(|&n| self.parent(n, b) == Some(node)) - .map(|n| (n, self.carry[n][b].serves)) + let feeds: BTreeSet = (0..self.nodes()) + .filter(|&n| self.parent(n, cast) == Some(node)) .collect(); - assert_eq!(carry.downstream, feeds, "{label}: broadcast {b} at node {node}"); - assert!( - !carry.fading && !carry.lingering, - "{label}: broadcast {b} at node {node}" + assert_eq!(relay.downstream(cast), feeds, "{label}: {cast:?} at node {node}"); + assert_eq!( + relay.streams(cast), + feeds.len(), + "{label}: {cast:?} at node {node}: one stream each" ); - for to in 0..self.nodes() { + assert!(!relay.lingering(cast), "{label}: {cast:?} at node {node}"); + if node != origin { + let demand = relay.reading(cast) || !feeds.is_empty(); + let source = relay.source(cast).map(|source| source.next); assert_eq!( - carry.told.get(&to), - self.carry[to][b].warm.get(&node), - "{label}: broadcast {b} warm from {node} at {to}" + relay.parent(cast), + source.filter(|_| demand), + "{label}: {cast:?} at node {node}" ); } - if node == broadcast.origin || carry.parent.is_none() { - continue; - } - assert!( - carry.reader || !feeds.is_empty(), - "{label}: node {node} carries broadcast {b} for no one" - ); - } - for node in 0..self.nodes() { - if node == broadcast.origin || self.legacy.contains(&node) { - continue; - } - let Some(cost) = self.cost(node, b) else { - continue; - }; - let parent = self.parent(node, b); - let draining = parent.is_some_and(|parent| self.draining.contains(&(node, parent))); - for c in self.candidates(node, b) { - assert!( - (c.draining, self.price(node, b, &c)) >= (draining, cost), - "{label}: node {node} would move broadcast {b} from {parent:?} to {}", - c.next - ); + for peer in 0..self.nodes() { + if !self.sim.connected(node, peer) || self.sim.nodes[peer].legacy { + continue; + } + let sent = relay.offers(peer, cast).0; + let heard = self.sim.nodes[peer].offers(node, cast).1; + assert_eq!(sent, heard, "{label}: {cast:?} offered from {node} to {peer}"); } } } - let stats = self.stats; - let queued: usize = self.sim.queues.values().map(VecDeque::len).sum(); - self.step(); - assert_eq!(self.stats, stats, "{label}: a choice changed at quiescence"); - assert_eq!(queued, self.sim.queues.values().map(VecDeque::len).sum::()); - } -} - -/// A queue the cache layer delivers from. -#[derive(Clone, Copy, Debug)] -enum Queue { - Route(Node, Node), - Ack(Node, Node), - Sub(Node, Node, usize), -} - -impl Queue { - fn pair(self) -> (Node, Node) { - match self { - Self::Route(from, to) | Self::Ack(from, to) | Self::Sub(from, to, _) => (from, to), - } + let stats = self.sim.stats; + self.tick(); + assert_eq!(self.sim.stats, stats, "{label}: something changed at quiescence"); } } -/// An origin behind relay `a`, then relays `a`, `b`, `c` in a chain priced 3, with -/// a skip from `a` to `c` priced `skip`, and one broadcast. -fn chain(skip: Metric) -> (Mesh, usize) { +/// An origin behind relay `a`, then relays `a`, `b`, `c` in a chain priced 3, with a +/// skip from `a` to `c` priced `skip`, and one broadcast. +fn chain(skip: Metric) -> (Mesh, Cast) { let (o, a, b, c) = (0, 1, 2, 3); - let mut mesh = Mesh::new(4); + let mut mesh = Mesh::new(4, PICK); mesh.origin(o, 7); mesh.join(o, a, 1); mesh.join(a, b, 3); mesh.join(b, c, 3); mesh.join(a, c, skip); - let x = mesh.broadcast("live/x", o); + let x = mesh.broadcast(7, "live/x", 0); assert!(mesh.settle()); (mesh, x) } -/// A cold broadcast takes the skip. A reader on the skipped relay makes it pull -/// through `a` and warm toward `c`, which joins it. When that reader leaves, its -/// warm route goes after the linger and `c` returns to the skip. +/// A cold broadcast takes the skip. A reader on the skipped relay `b` makes `c` join +/// its copy, and when that reader leaves, `c` returns to the skip. Readers arriving +/// the other way round join the other way: `b` joins `c`'s copy, which is farther +/// from the publisher than `b` is. #[test] -fn a_skipped_relay_draws_its_neighbor_while_it_has_a_reader() { +fn a_skipped_relay_draws_its_neighbour_while_it_has_a_reader() { let (a, b, c) = (1, 2, 3); let (mut mesh, x) = chain(5); - mesh.read(c, x, true); + mesh.read(c, &x, true); assert!(mesh.settle()); - assert_eq!(mesh.parent(c, x), Some(a), "a cold broadcast takes the skip"); - assert!(!mesh.carries(b, x)); + assert_eq!(mesh.parent(c, &x), Some(a), "a cold broadcast takes the skip"); + assert!(!mesh.carries(b, &x)); mesh.steady("cold"); - mesh.read(b, x, true); + mesh.read(b, &x, true); assert!(mesh.settle()); - assert_eq!(mesh.parent(b, x), Some(a)); - assert_eq!(mesh.parent(c, x), Some(b), "c joins b's copy"); - mesh.steady("warm"); + assert_eq!(mesh.parent(b, &x), Some(c), "b joins c's copy, though c is farther"); + assert_eq!(mesh.parent(c, &x), Some(a)); + mesh.steady("b joins c"); - mesh.read(b, x, false); + let (mut mesh, x) = chain(5); + mesh.read(b, &x, true); + assert!(mesh.settle()); + assert_eq!(mesh.parent(b, &x), Some(a)); + mesh.read(c, &x, true); assert!(mesh.settle()); - assert_eq!(mesh.parent(c, x), Some(a), "back to the skip"); - assert!(!mesh.carries(b, x)); + assert_eq!(mesh.parent(c, &x), Some(b), "c joins b's copy"); + mesh.steady("c joins b"); + + mesh.read(b, &x, false); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, &x), Some(a), "back to the skip"); + assert!(!mesh.carries(b, &x)); mesh.steady("cold again"); + assert_eq!(mesh.loops, 0); } -/// A relay whose only consumer is `c` withdraws its warm route toward `c`. With -/// a cheaper skip, `c` bypasses it; when the relay is `c`'s cheapest cold route, -/// `c` stays and the relay keeps carrying for it alone. +/// A relay whose only consumer is `c` offers `c` nothing once its own reader goes. +/// With a cheaper skip, `c` bypasses it; when the relay is `c`'s cheapest route to +/// the publisher, `c` stays and the relay keeps pulling for it alone. #[test] -fn a_relay_feeding_one_consumer_stays_only_on_its_cheapest_cold_route() { +fn a_relay_feeding_one_consumer_stays_only_on_its_cheapest_route() { let (a, b, c) = (1, 2, 3); for (skip, stays) in [(5, false), (9, true)] { let (mut mesh, x) = chain(skip); - mesh.read(b, x, true); - mesh.read(c, x, true); + mesh.read(b, &x, true); + mesh.read(c, &x, true); assert!(mesh.settle()); - assert_eq!(mesh.parent(c, x), Some(b), "skip {skip}"); - mesh.read(b, x, false); + assert_eq!(mesh.parent(c, &x), Some(b), "skip {skip}"); + mesh.read(b, &x, false); assert!(mesh.settle()); - assert!(!mesh.warm_at(c, b, x), "skip {skip}: b withdrew"); - assert_eq!(mesh.parent(c, x), Some(if stays { b } else { a }), "skip {skip}"); - assert_eq!(mesh.carries(b, x), stays, "skip {skip}"); + assert_eq!(mesh.parent(c, &x), Some(if stays { b } else { a }), "skip {skip}"); + assert_eq!(mesh.carries(b, &x), stays, "skip {skip}"); mesh.steady(&format!("skip {skip}")); } } -/// `r2`'s cheapest cold route runs through `r1` to `r3`, so `r3`'s only demand is -/// `r1`'s pull on `r2`'s behalf. Counted as demand besides `r2`'s, it would warm `r3` -/// toward `r2`: `r2` would join `r3` over its direct link, `r1`'s pull and with it -/// `r3`'s warmth would go, and `r2` would return to `r1`, once per linger forever. -/// A pull names the relay it serves, so `r3` stays cold toward `r2`. +/// `r2`'s cheapest route runs through `r1` to `r3`, so `r3` pulls only on `r2`'s +/// behalf, and offers `r2` nothing: were it to, `r2` would join it over the direct +/// link, `r1`'s pull would go, and with it the reason for `r3`'s copy. A reader of +/// `r1`'s own makes it offer a copy, which `r2` already pulls. #[test] -fn a_pull_on_a_neighbors_behalf_never_warms_its_parent_toward_it() { +fn a_pull_on_a_neighbours_behalf_offers_nothing() { let (o, r1, r2, r3) = (0, 1, 2, 3); - let mut mesh = Mesh::new(4); + let mut mesh = Mesh::new(4, PICK); mesh.origin(o, 7); mesh.join(o, r3, 1); mesh.join(r3, r1, 1); mesh.join(r1, r2, 1); mesh.join(r2, r3, 3); - let x = mesh.broadcast("live/x", o); - mesh.read(r2, x, true); + let x = mesh.broadcast(7, "live/x", 0); + assert!(mesh.settle()); + mesh.read(r2, &x, true); assert!(mesh.settle()); - assert_eq!(mesh.parent(r2, x), Some(r1)); - assert_eq!(mesh.parent(r1, x), Some(r3)); - assert!(!mesh.warm_at(r2, r3, x)); + assert_eq!(mesh.parent(r2, &x), Some(r1)); + assert_eq!(mesh.parent(r1, &x), Some(r3)); + assert!(!mesh.offered(r2, r3, &x) && !mesh.offered(r2, r1, &x)); + assert_eq!(mesh.warm_sent(&x), 2, "r2's own copy, offered to r1 and r3"); mesh.steady("behalf"); - // A reader of its own makes r1's pull stand alone and r3 warm toward r2, but r1, - // warm too, is the cheaper join. - mesh.read(r1, x, true); + mesh.read(r1, &x, true); assert!(mesh.settle()); - assert!(mesh.warm_at(r2, r3, x)); - assert_eq!(mesh.parent(r2, x), Some(r1)); + assert!(mesh.offered(r2, r1, &x)); + assert_eq!(mesh.parent(r2, &x), Some(r1)); mesh.steady("own reader"); } @@ -807,58 +469,69 @@ const PATHS: [&str; 8] = [ "live/0", "live/1", "live/2", "live/3", "live/4", "live/5", "live/6", "live/7", ]; -/// Two relays at equal cold metric, both warm toward each other: per broadcast, -/// only the lower hash keeps its uplink and the other joins it, so the pair pulls -/// one copy, and the owner differs across broadcasts. +/// Two relays at equal cost, each with a reader of every broadcast, whose readers +/// arrive in the same instant: per broadcast only the lower hash keeps its uplink and +/// the other joins it, so the pair pulls one copy, and the owner differs across +/// broadcasts. Unranked, the two trade copies for good. #[test] fn equal_relays_pull_one_copy_owned_by_the_lower_hash() { let (o, x, y) = (0, 1, 2); - let mut mesh = Mesh::new(3); - mesh.origin(o, 7); - mesh.join(o, x, 3); - mesh.join(o, y, 3); - mesh.join(x, y, 1); - let broadcasts: Vec = PATHS.iter().map(|path| mesh.broadcast(path, o)).collect(); - for &b in &broadcasts { - mesh.read(x, b, true); - mesh.read(y, b, true); - } + let build = |rules: Rules| { + let mut mesh = Mesh::new(3, rules); + mesh.origin(o, 7); + mesh.join(o, x, 3); + mesh.join(o, y, 3); + mesh.join(x, y, 1); + let casts: Vec = PATHS.iter().map(|path| mesh.broadcast(7, path, 0)).collect(); + for cast in &casts { + mesh.sim.carry(x, cast); + mesh.sim.carry(y, cast); + } + (mesh, casts) + }; + let (mut mesh, casts) = build(PICK); assert!(mesh.settle()); let mut owners = BTreeSet::new(); - for (&b, path) in broadcasts.iter().zip(PATHS) { - let (low, high) = match hash(path, x) < hash(path, y) { + for cast in &casts { + let rank = |node: Node| rendezvous(&cast.1, hop(node, 0)); + let (low, high) = match rank(x) < rank(y) { true => (x, y), false => (y, x), }; - assert_eq!(mesh.parent(low, b), Some(o), "{path}"); - assert_eq!(mesh.parent(high, b), Some(low), "{path}"); - assert_eq!(mesh.copies(b), 1, "{path}"); + assert_eq!(mesh.parent(low, cast), Some(o), "{cast:?}"); + assert_eq!(mesh.parent(high, cast), Some(low), "{cast:?}"); + assert_eq!(mesh.copies(cast), 1, "{cast:?}"); owners.insert(low); } assert_eq!(owners.len(), 2, "the hash spreads the owner"); mesh.steady("pair"); + assert_eq!(mesh.loops, 0); + + let (mut mesh, _) = build(Rules { ranked: false, ..PICK }); + assert!(!mesh.run(), "unranked, the two trade copies for good"); } -/// A ring of relays, each holding its neighbors' reports from while every relay -/// had its own link to the origin, loses all but one of those links. Each report -/// is now lower than the truth, the hazard that let a whole ring let go at once -/// under the old rank. Delivered in any order, no broadcast loops, and once quiet -/// every reader is fed through the one link left. +/// A ring of relays, each with its own link to the publisher and a reader, loses all +/// but one of those links while every link of the ring is held, so each relay acts on +/// what it last heard. Delivered in any order, once quiet every reader is fed through +/// the one link left, and the pulls formed a cycle only while a withdrawal was still +/// crossing a link. #[test] -fn a_ring_of_stale_lower_reports_forms_no_loop() { +fn a_ring_of_stale_offers_settles_on_the_link_left() { const RING: usize = 5; let o = 0; let relays: Vec = (1..=RING).collect(); + let mut loops = 0; for seed in 0..20u64 { - let mut mesh = Mesh::new(RING + 1); + let mut mesh = Mesh::new(RING + 1, PICK); mesh.origin(o, 7); for (i, &relay) in relays.iter().enumerate() { mesh.join(o, relay, 1); mesh.join(relay, relays[(i + 1) % RING], 1); } - let x = mesh.broadcast("live/x", o); + let x = mesh.broadcast(7, "live/x", 0); for &relay in &relays { - mesh.read(relay, x, true); + mesh.read(relay, &x, true); } assert!(mesh.settle()); for (i, &relay) in relays.iter().enumerate() { @@ -869,80 +542,122 @@ fn a_ring_of_stale_lower_reports_forms_no_loop() { for &relay in &relays[1..] { mesh.disconnect(o, relay); } + for (i, &relay) in relays.iter().enumerate() { + let next = relays[(i + 1) % RING]; + mesh.release(relay, next); + mesh.release(next, relay); + } let mut rng = XorShift(0x5717_e000 + seed); loop { let busy = mesh.busy(); if busy.is_empty() { break; } - mesh.deliver_one(busy[rng.next(busy.len() as u64) as usize]); + mesh.deliver_one(&busy[rng.next(busy.len() as u64) as usize]); } - assert!(mesh.settle()); + assert!(mesh.settle(), "seed {seed}"); mesh.steady(&format!("seed {seed}")); - assert_eq!(mesh.copies(x), 1, "seed {seed}"); - assert_eq!(mesh.parent(relays[0], x), Some(o), "seed {seed}"); + assert_eq!(mesh.copies(&x), 1, "seed {seed}"); + assert_eq!(mesh.parent(relays[0], &x), Some(o), "seed {seed}"); + loops += mesh.loops; } + assert!(loops <= 20, "{loops} steps with a cycle across 20 seeds"); } -/// `b` withdraws its warm route toward `c` while `c`'s subscription to it is on -/// its way. The subscription reaches `b` with no copy left, and `b` pulls it -/// along its own route, as for any request; `c` then moves back to the skip. +/// `b` withdraws its copy while `c`'s subscription to it is on its way. The +/// subscription reaches `b` with no copy left, and `b` pulls along its own route, as +/// for any request; `c` then moves back to the skip. #[test] -fn a_request_crossing_a_warm_withdrawal_is_forwarded_without_a_loop() { +fn a_request_crossing_a_withdrawal_is_forwarded_without_a_loop() { let (a, b, c) = (1, 2, 3); let (mut mesh, x) = chain(5); - mesh.read(b, x, true); + mesh.read(b, &x, true); assert!(mesh.settle()); - assert!(mesh.warm_at(c, b, x)); + assert!(mesh.offered(c, b, &x)); mesh.hold(c, b); mesh.hold(b, c); - mesh.read(c, x, true); + mesh.read(c, &x, true); assert!(mesh.run()); - assert_eq!(mesh.parent(c, x), Some(b), "c's subscription waits"); + assert_eq!(mesh.parent(c, &x), Some(b), "c's subscription waits"); - mesh.read(b, x, false); - mesh.tick(b, x); + mesh.read(b, &x, false); + mesh.expire(b, &x); assert!(mesh.run()); - assert!(!mesh.carries(b, x), "b let its copy go"); - assert!(mesh.warm_at(c, b, x), "its withdrawal waits"); - - assert!(mesh.deliver_sub(c, b, x)); - assert_eq!(mesh.parent(b, x), Some(a), "b forwards the request along its own route"); + assert!(!mesh.carries(b, &x), "b let its copy go"); + assert!(mesh.offered(c, b, &x), "its withdrawal waits"); + + assert!(mesh.deliver_sub(c, b, &x)); + assert_eq!( + mesh.parent(b, &x), + Some(a), + "b forwards the request along its own route" + ); mesh.release(c, b); mesh.release(b, c); assert!(mesh.settle()); - assert_eq!(mesh.parent(c, x), Some(a)); - assert!(!mesh.carries(b, x)); + assert_eq!(mesh.parent(c, &x), Some(a)); + assert!(!mesh.carries(b, &x)); mesh.steady("crossed"); + assert_eq!(mesh.loops, 0); } -/// A warm route for another instance of the same path is never joined: `c`'s -/// reader of the new instance takes the skip although `b` is warm for the old one. +/// `c` moves off `b` and back while its subscriptions to `b` wait. The new stream's +/// pull reaches `b` before the old stream's cancel, and `b` keeps feeding `c`. #[test] -fn a_warm_route_for_another_instance_is_never_joined() { - let (o, a, b, c) = (0, 1, 2, 3); +fn a_replacement_pull_overtaking_the_old_cancel_keeps_its_feed() { + let (a, b, c) = (1, 2, 3); + let (mut mesh, x) = chain(5); + mesh.read(b, &x, true); + mesh.read(c, &x, true); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, &x), Some(b)); + mesh.hold(c, b); + + mesh.read(b, &x, false); + assert!(mesh.run()); + assert_eq!(mesh.parent(c, &x), Some(a), "c leaves for the skip"); + mesh.read(b, &x, true); + assert!(mesh.run()); + assert_eq!(mesh.parent(c, &x), Some(b), "c comes back"); + + let streams = mesh.streams(c, b); + assert_eq!(streams.len(), 2, "the old cancel and the new pull wait"); + assert!(mesh.deliver_stream(c, b, &x, streams[1])); + assert!(mesh.deliver_stream(c, b, &x, streams[0])); + assert_eq!(mesh.sim.nodes[b].downstream(&x), BTreeSet::from([c])); + mesh.release(c, b); + assert!(mesh.settle()); + assert_eq!(mesh.parent(c, &x), Some(b)); + mesh.steady("overtaken"); + assert_eq!(mesh.loops, 0); +} + +/// A copy of another instance of the same path is never joined: `c`'s reader of the +/// new instance takes the skip although `b` offers the old one. +#[test] +fn a_copy_of_another_instance_is_never_joined() { + let (a, b, c) = (1, 2, 3); let (mut mesh, old) = chain(5); - let new = mesh.broadcast("live/x", o); - mesh.read(b, old, true); - mesh.read(c, new, true); + let new = mesh.broadcast(7, "live/x", 1); + mesh.read(b, &old, true); + mesh.read(c, &new, true); assert!(mesh.settle()); - assert!(mesh.warm_at(c, b, old)); - assert_eq!(mesh.parent(c, new), Some(a)); - assert!(!mesh.carries(b, new)); + assert!(mesh.offered(c, b, &old)); + assert_eq!(mesh.parent(c, &new), Some(a)); + assert!(!mesh.carries(b, &new)); mesh.steady("instances"); } -/// `k` relays on a LAN, each with its own uplink and a reader of every broadcast. -/// In a full mesh each broadcast crosses one uplink. In a ring a relay joins only a -/// neighbor with a lower hash, so each relay whose neighbors both hash higher keeps -/// an uplink copy, which the epic accepts. +/// `k` relays on a LAN, each with its own uplink and a reader of every broadcast. In +/// a full mesh each broadcast crosses one uplink. In a ring some relays keep an +/// uplink copy of their own, which the epic accepts. #[test] fn lan_relays_share_uplink_copies() { const K: usize = 6; let o = 0; let relays: Vec = (1..=K).collect(); for full in [true, false] { - let mut mesh = Mesh::new(K + 1); + let mut mesh = Mesh::new(K + 1, PICK); mesh.origin(o, 7); for &relay in &relays { mesh.join(o, relay, 3); @@ -954,94 +669,240 @@ fn lan_relays_share_uplink_copies() { } } } - let broadcasts: Vec = PATHS.iter().map(|path| mesh.broadcast(path, o)).collect(); - for &b in &broadcasts { + let casts: Vec = PATHS.iter().map(|path| mesh.broadcast(7, path, 0)).collect(); + for cast in &casts { for &relay in &relays { - mesh.read(relay, b, true); + mesh.read(relay, cast, true); } } assert!(mesh.settle()); - mesh.steady(if full { "full mesh" } else { "ring" }); - let mut copies = 0; - for (&b, path) in broadcasts.iter().zip(PATHS) { - let minima = (0..K) - .filter(|&i| { - let h = hash(path, relays[i]); - h < hash(path, relays[(i + 1) % K]) && h < hash(path, relays[(i + K - 1) % K]) - }) - .count(); - assert_eq!(mesh.copies(b), if full { 1 } else { minima }, "{path}"); - copies += mesh.copies(b); - } - // Recorded: one copy per broadcast in the full mesh, 12 for 8 in the ring. - assert_eq!(copies, if full { 8 } else { 12 }); + let label = if full { "full mesh" } else { "ring" }; + mesh.steady(label); + let copies: Vec = casts.iter().map(|cast| mesh.copies(cast)).collect(); + // Recorded: one uplink copy per broadcast in the full mesh, and a few in the + // ring, where a relay joins only a copy whose carrier ranks below its own. + // Readers arriving everywhere at once cross their offers on the way, and the + // pulls form a cycle until those cross back. + let total: usize = copies.iter().sum(); + assert_eq!( + (total, mesh.loops), + if full { (8, 5) } else { (15, 0) }, + "{label}: {copies:?}" + ); } } -/// While a broadcast is pulled: its origin advances its seqno, a relay loses its -/// route, and a relay drains. The seqno reaches `c` through the skip before `b` -/// passes it on, and `b`'s older route is then ineligible, so `c` moves off `b` -/// and back: what staying loop-free across seqnos costs a joined copy. +/// While a broadcast is pulled, a relay loses its link toward the publisher, and +/// later a relay drains (GOAWAY moves pulls off its copy). #[test] -fn a_seqno_advance_a_lost_route_and_a_drain_keep_pulls_loop_free() { - let (o, a, b, c) = (0, 1, 2, 3); +fn a_lost_route_and_a_drain_move_pulls_without_a_loop() { + let (a, b, c) = (1, 2, 3); let (mut mesh, x) = chain(5); - mesh.read(b, x, true); - mesh.read(c, x, true); + mesh.read(b, &x, true); + mesh.read(c, &x, true); assert!(mesh.settle()); - assert_eq!(mesh.parent(c, x), Some(b)); - - let moves = mesh.stats.moves; - mesh.bump(o); - assert!(mesh.settle()); - assert_eq!(mesh.parent(c, x), Some(b)); - assert_eq!(mesh.stats.moves - moves, 2, "c left b while b's route was older"); - mesh.steady("seqno"); + assert_eq!(mesh.parent(c, &x), Some(b)); mesh.disconnect(a, b); assert!(mesh.settle()); - assert_eq!(mesh.parent(c, x), Some(a)); - assert_eq!(mesh.parent(b, x), Some(c), "b reaches the origin only through c"); + assert_eq!(mesh.parent(c, &x), Some(a)); + assert_eq!(mesh.parent(b, &x), Some(c), "b reaches the publisher only through c"); mesh.steady("route loss"); mesh.connect(a, b); assert!(mesh.settle()); - assert_eq!(mesh.parent(c, x), Some(b)); + mesh.steady("reconnected"); mesh.goaway(b); assert!(mesh.settle()); - assert_eq!(mesh.parent(c, x), Some(a), "c leaves the draining relay"); + assert_eq!(mesh.parent(c, &x), Some(a), "c leaves the draining relay"); mesh.steady("drain"); + assert_eq!(mesh.loops, 0); +} + +const HOME: Node = 0; +const NEAR: Node = 1; +const FAR: Node = 2; +const TAIL: Node = 3; +const ORIGIN: Node = 4; +const SIDE: Node = 5; + +/// Two relays with an uplink each to the publisher's relay, `NEAR`'s the cheaper, and +/// a cheap link between them. `TAIL` hangs off `FAR`, and `SIDE` off `NEAR`. +fn uplinks(rules: Rules) -> (Mesh, Cast) { + let mut mesh = Mesh::new(6, rules); + mesh.origin(ORIGIN, 7); + mesh.join(HOME, NEAR, 10); + mesh.join(HOME, FAR, 12); + mesh.join(NEAR, FAR, 3); + mesh.join(FAR, TAIL, 1); + mesh.join(NEAR, SIDE, 1); + mesh.join(ORIGIN, HOME, 1); + let x = mesh.broadcast(7, "live", 0); + assert!(mesh.settle()); + (mesh, x) } -/// Random relay meshes with two origins under churn: readers come and go, +/// `FAR` carries the broadcast and is farther from the publisher than `NEAR`. A reader +/// at `SIDE` pulls through `NEAR`, which joins `FAR`'s copy over the cheap link rather +/// than pull its own over its uplink. Babel's distance cannot allow this: `FAR`'s cost +/// to the publisher is not below `NEAR`'s. +#[test] +fn a_relay_joins_a_copy_farther_from_the_publisher() { + for rules in [PICK, Rules::EPOCHS] { + let (mut mesh, x) = uplinks(rules); + mesh.read(FAR, &x, true); + assert!(mesh.settle()); + let before = mesh.sim.stats; + mesh.read(SIDE, &x, true); + assert!(mesh.settle()); + assert_eq!(mesh.parent(SIDE, &x), Some(NEAR), "{rules:?}"); + assert_eq!( + mesh.parent(NEAR, &x), + Some(FAR), + "{rules:?}: NEAR joins the farther copy" + ); + assert_eq!(mesh.parent(FAR, &x), Some(HOME), "{rules:?}"); + assert_eq!(mesh.sim.nodes[NEAR].cost(7), Some(11)); + assert_eq!(mesh.sim.nodes[FAR].cost(7), Some(13)); + mesh.steady(&format!("{rules:?}")); + // SIDE's copy came through NEAR, its only neighbour, so it offers it to no one. + assert_eq!(mesh.sim.stats.warm_msgs - before.warm_msgs, 0, "{rules:?}"); + + mesh.read(FAR, &x, false); + assert!(mesh.settle()); + assert_eq!(mesh.parent(NEAR, &x), Some(HOME), "{rules:?}"); + assert!(!mesh.carries(FAR, &x), "{rules:?}"); + mesh.steady(&format!("{rules:?}: FAR's reader left")); + assert_eq!(mesh.loops, 0, "{rules:?}"); + } +} + +/// Both relays carry a copy, `NEAR`'s pulled from `FAR`'s whenever `FAR` ranks below +/// it. When `FAR`'s uplink closes the two swap if they must, each still fed from the +/// publisher, and the pulls loop only while the news crosses their link. +#[test] +fn two_copies_never_feed_each_other_once_quiet() { + for rules in [PICK, Rules::EPOCHS] { + let (mut mesh, x) = uplinks(rules); + mesh.read(FAR, &x, true); + assert!(mesh.settle()); + mesh.read(NEAR, &x, true); + mesh.read(TAIL, &x, true); + assert!(mesh.settle()); + mesh.steady(&format!("{rules:?}")); + assert_eq!(mesh.loops, 0, "{rules:?}"); + + mesh.disconnect(HOME, FAR); + assert!(mesh.settle()); + assert_eq!(mesh.parent(NEAR, &x), Some(HOME), "{rules:?}"); + assert_eq!(mesh.parent(FAR, &x), Some(NEAR), "{rules:?}"); + assert_eq!(mesh.parent(TAIL, &x), Some(FAR), "{rules:?}"); + mesh.steady(&format!("{rules:?}: FAR's uplink closed")); + assert!(mesh.loops <= 3, "{rules:?}: {} steps with a cycle", mesh.loops); + } +} + +/// `NEAR` pulls `FAR`'s copy, whose path crosses `HOME`, and then loses its own +/// session to `HOME`, which casts doubt on `HOME`. Doubt keeps a reader from moving +/// to a copy, not on the one it pulls, so `NEAR` stays on `FAR`'s throughout. +#[test] +fn a_doubt_leaves_the_copy_in_use_alone() { + let (mut mesh, x) = uplinks(PICK); + mesh.read(FAR, &x, true); + mesh.read(SIDE, &x, true); + assert!(mesh.settle()); + assert_eq!(mesh.parent(NEAR, &x), Some(FAR)); + + mesh.disconnect(HOME, NEAR); + assert_eq!(mesh.parent(NEAR, &x), Some(FAR)); + for _ in 0..64 { + match mesh.busy().first().cloned() { + Some(queue) => mesh.deliver_one(&queue), + None => { + mesh.tick(); + } + }; + assert_eq!(mesh.parent(NEAR, &x), Some(FAR)); + } + assert!(mesh.settle()); + mesh.steady("doubt"); + assert_eq!(mesh.loops, 0); +} + +/// The publisher leaves while two relays carry a copy. Every copy goes with its +/// route, nothing is left pulling, and nothing remembers a dead hop. +#[test] +fn a_copy_that_goes_takes_its_offers_along() { + let (mut mesh, x) = uplinks(PICK); + mesh.read(FAR, &x, true); + mesh.read(NEAR, &x, true); + mesh.read(TAIL, &x, true); + assert!(mesh.settle()); + + mesh.disconnect(ORIGIN, HOME); + assert!(mesh.settle()); + for relay in [HOME, NEAR, FAR, TAIL, SIDE] { + assert!(!mesh.carries(relay, &x), "relay {relay}"); + assert_eq!(mesh.sim.nodes[relay].retained(), (0, 0), "relay {relay}"); + } +} + +/// Random relay meshes with two publishers under churn: readers come and go, /// sessions drop and return, messages are held and delivered out of step, lingers -/// run out, origins advance their seqno, and relays drain. No broadcast's parents -/// ever form a cycle. Once churn stops the mesh settles on the steady state, and a -/// broadcast nobody reads has sent no warm route. +/// run out, time passes, and relays drain, with links held or not. Once churn stops +/// every mesh settles on the steady state, and a broadcast nobody reads has sent no +/// copy offer. The pulls form a cycle only while a withdrawal crosses a link, which +/// the sweep counts. #[test] fn sweep_cache_routing() { - sweep(100); + for holds in [true, false] { + sweep(SEED, 100, holds); + } } -/// The sweep at length, too slow for the merge gate. +/// The gate's seed, and the long sweep's unless `MOQ_ROUTE_SEED` names another. +const SEED: u64 = 0xcac4_e5ee; + +/// The sweep at length from `MOQ_ROUTE_SEED` over `MOQ_ROUTE_CASES` scenarios each way, +/// which prints what it measured. #[test] -#[ignore = "3000 scenarios take minutes"] +#[ignore = "slow: nightly"] fn sweep_cache_routing_long() { - sweep(3000); + let setting = |name: &str, fallback: u64| std::env::var(name).map_or(fallback, |value| value.parse().expect(name)); + let seed = setting("MOQ_ROUTE_SEED", SEED); + let cases = setting("MOQ_ROUTE_CASES", 3000) as usize; + println!("MOQ_ROUTE_SEED={seed} MOQ_ROUTE_CASES={cases}"); + for holds in [true, false] { + let out = sweep(seed, cases, holds); + println!("holds {holds}: {out:?}"); + } } -/// The scenarios that stay loop-free but never settle: two carriers each keep a -/// narrowed advert standing while two relays swap between them. The sweep checks -/// that each still never settles, so the list shrinks as convergence is fixed. -const UNSETTLED: [usize; 1] = [2251]; +/// What a sweep measured. +#[derive(Debug, Default)] +struct Sweep { + /// Steps after which some broadcast's pulls formed a cycle, of all steps. + loops: usize, + steps: usize, + /// Scenarios with any such step. + looped: usize, + /// Steps after which a reader's pulls led into a cycle. + starved: usize, + /// The most deliveries in a row after which a cycle remained. + longest: usize, + /// Copy offers and their ends sent, and reader changes, during churn. + warm: usize, + reads: usize, +} -fn sweep(scenarios: usize) { +fn sweep(seed: u64, scenarios: usize, holds: bool) -> Sweep { const RELAYS: usize = 6; - let mut rng = XorShift(0xcac4_e5ee); + let mut rng = XorShift(seed); + let mut out = Sweep::default(); for scenario in 0..scenarios { - let label = format!("scenario {scenario}"); + let label = format!("seed {seed} holds {holds} scenario {scenario}"); let origins = [RELAYS, RELAYS + 1]; - let mut mesh = Mesh::new(RELAYS + 2); + let mut mesh = Mesh::new(RELAYS + 2, PICK); mesh.origin(origins[0], 100); mesh.origin(origins[1], 200); if rng.chance(30) { @@ -1063,21 +924,23 @@ fn sweep(scenarios: usize) { mesh.join(a, b, 1 + rng.next(5)); } let read = [ - mesh.broadcast("live/a", origins[0]), - mesh.broadcast("live/b", origins[0]), - mesh.broadcast("live/a", origins[1]), + mesh.broadcast(100, "live/a", 0), + mesh.broadcast(100, "live/b", 0), + mesh.broadcast(200, "live/a", 0), // Another instance of the first. - mesh.broadcast("live/a", origins[0]), + mesh.broadcast(100, "live/a", 1), ]; - let idle = mesh.broadcast("live/idle", origins[0]); + let idle = mesh.broadcast(100, "live/idle", 0); assert!(mesh.settle(), "{label}"); + let warm = mesh.sim.stats.warm_msgs; for _ in 0..60 { match rng.next(12) { 0..=2 => { - let (relay, b) = (rng.next(RELAYS as u64) as usize, read[rng.next(4) as usize]); - let reading = mesh.reading(relay, b); - mesh.read(relay, b, !reading); + let (relay, cast) = (rng.next(RELAYS as u64) as usize, &read[rng.next(4) as usize]); + let reading = mesh.reading(relay, cast); + mesh.read(relay, cast, !reading); + out.reads += 1; } 3 => { let (a, b) = links[rng.next(links.len() as u64) as usize]; @@ -1090,12 +953,15 @@ fn sweep(scenarios: usize) { let (a, b) = links[rng.next(links.len() as u64) as usize]; let (from, to) = if rng.chance(50) { (a, b) } else { (b, a) }; match rng.chance(70) { - true => mesh.hold(from, to), + true if holds => mesh.hold(from, to), + true => {} false => mesh.release(from, to), } } - 5 => mesh.tick(rng.next(RELAYS as u64) as usize, read[rng.next(4) as usize]), - 6 if rng.chance(50) => mesh.bump(origins[rng.next(2) as usize]), + 5 => mesh.expire(rng.next(RELAYS as u64) as usize, &read[rng.next(4) as usize]), + 6 if rng.chance(50) => { + mesh.tick(); + } 6 if rng.chance(20) => mesh.goaway(rng.next(RELAYS as u64) as usize), _ => { for _ in 0..rng.next(8) { @@ -1103,11 +969,12 @@ fn sweep(scenarios: usize) { if busy.is_empty() { break; } - mesh.deliver_one(busy[rng.next(busy.len() as u64) as usize]); + mesh.deliver_one(&busy[rng.next(busy.len() as u64) as usize]); } } } } + out.warm += mesh.sim.stats.warm_msgs - warm; for &(a, b) in &links { mesh.release(a, b); @@ -1116,12 +983,14 @@ fn sweep(scenarios: usize) { mesh.connect(a, b); } } - if UNSETTLED.contains(&scenario) { - assert!(!mesh.settle(), "{label} settles: drop it from UNSETTLED"); - continue; - } assert!(mesh.settle(), "{label}"); mesh.steady(&label); - assert_eq!(mesh.warm_sent(idle), 0, "{label}"); - } + assert_eq!(mesh.warm_sent(&idle), 0, "{label}"); + out.loops += mesh.loops; + out.steps += mesh.steps; + out.looped += usize::from(mesh.loops > 0); + out.starved += mesh.starved; + out.longest = out.longest.max(mesh.longest); + } + out } diff --git a/rs/moq-net/src/route/tests/copies.rs b/rs/moq-net/src/route/tests/copies.rs deleted file mode 100644 index a7652355b0d..00000000000 --- a/rs/moq-net/src/route/tests/copies.rs +++ /dev/null @@ -1,240 +0,0 @@ -//! The cached-copy case of cache-aware routing, on the prototype in `paths.rs`. -//! -//! A relay with a reader for a broadcast carries a copy, and offers its neighbours -//! a route to that copy alone: the path the copy took from the publisher, and no -//! cost of its own. A relay pulls the broadcast from whichever is cheaper, the -//! publisher's route or a copy whose path does not name it, and passes a copy's -//! route on for as long as it beats the publisher's. - -use super::paths::{PathSim, Rules, Source}; -use super::*; - -const HOME: Node = 0; -const NEAR: Node = 1; -const FAR: Node = 2; -const TAIL: Node = 3; -const ORIGIN: Node = 4; - -/// Two relays with an uplink each to the publisher's relay, `NEAR`'s the cheaper, -/// and a cheap link between them. `TAIL` hangs off `FAR`. -fn uplinks(rules: Rules) -> PathSim { - let mut sim = PathSim::new(5, rules); - sim.leaf(ORIGIN); - sim.join(HOME, NEAR, 10); - sim.join(HOME, FAR, 12); - sim.join(NEAR, FAR, 3); - sim.join(FAR, TAIL, 1); - sim.join(ORIGIN, HOME, 1); - sim.originate(ORIGIN, 7); - sim.announce(ORIGIN, "live"); - assert!(sim.run()); - sim -} - -fn pulls(sim: &PathSim, node: Node) -> Option<(Node, Metric, bool)> { - let source: Source = sim.source(node, 7, "live")?; - Some((source.next, source.cost, source.warm)) -} - -/// How following each relay's source for the broadcast fares: relays whose sources -/// lead round a loop, and relays whose sources lead nowhere. -fn astray(sim: &PathSim) -> (usize, usize) { - let (mut loops, mut lost) = (0, 0); - for relay in [HOME, NEAR, FAR, TAIL] { - let mut at = relay; - let mut hops = 0; - loop { - match sim.source(at, 7, "live") { - None if at == relay => break, - None => { - lost += 1; - break; - } - Some(source) if source.next == ORIGIN => break, - Some(source) if !sim.connected(at, source.next) => { - lost += 1; - break; - } - Some(source) => at = source.next, - } - hops += 1; - if hops > 5 { - loops += 1; - break; - } - } - } - (loops, lost) -} - -/// Deliver one message at a time, and sum what was astray after each. -fn settle(sim: &mut PathSim) -> (usize, usize) { - let (mut loops, mut lost) = astray(sim); - for _ in 0..10_000 { - let Some(&(from, to)) = sim.busy().first() else { - return (loops, lost); - }; - sim.deliver(from, to); - let (a, b) = astray(sim); - loops += a; - lost += b; - } - panic!("never settles"); -} - -/// `FAR` carries the broadcast and is farther from the publisher than `NEAR`. -/// `NEAR` joins that copy over the cheap link rather than pull its own over its -/// uplink. Babel's distance cannot allow this: `FAR`'s cost to the publisher is -/// not below what `NEAR` advertised, which is the check cache-aware routing's -/// eligibility rule makes (moq-dev/moq#8457). -#[test] -fn a_relay_joins_a_copy_no_closer_to_the_publisher() { - for rules in [Rules::BARE, Rules::CAUSES, Rules::EPOCHS] { - let mut sim = uplinks(rules); - assert_eq!(pulls(&sim, NEAR), Some((HOME, 11, false))); - assert_eq!(pulls(&sim, FAR), Some((HOME, 13, false))); - - let before = sim.stats; - sim.carry(FAR, 7, "live"); - assert!(sim.run()); - assert_eq!(pulls(&sim, FAR), Some((HOME, 13, false)), "{rules:?}"); - assert_eq!(pulls(&sim, NEAR), Some((FAR, 3, true)), "{rules:?}"); - assert_eq!(pulls(&sim, TAIL), Some((FAR, 1, true)), "{rules:?}"); - // One message to each neighbour the copy's path does not name, and none onward: - // every further neighbour is on that path. - assert_eq!(sim.stats.warm_msgs - before.warm_msgs, 2, "{rules:?}"); - assert_eq!(sim.stats.route_msgs - before.route_msgs, 2, "{rules:?}"); - - sim.uncarry(FAR, 7, "live"); - assert!(sim.run()); - assert_eq!(pulls(&sim, NEAR), Some((HOME, 11, false)), "{rules:?}"); - assert_eq!(pulls(&sim, TAIL), Some((FAR, 14, false)), "{rules:?}"); - assert_eq!(sim.stats.warm_msgs - before.warm_msgs, 4, "{rules:?}"); - } - - // The same mesh on today's router: the neighbour with the copy is the farther one. - let mut sim = Sim::new(5); - sim.join(HOME, NEAR, 10); - sim.join(HOME, FAR, 12); - sim.join(NEAR, FAR, 3); - sim.join(FAR, TAIL, 1); - sim.join(ORIGIN, HOME, 1); - sim.originate(ORIGIN, 7, 0); - sim.announce(ORIGIN, "live", 0, None); - assert!(sim.run()); - let (near, far) = (sim.route(NEAR, 7).unwrap(), sim.route(FAR, 7).unwrap()); - assert_eq!((near.metric, far.metric), (11, 13)); - assert!(far.metric >= near.metric, "so the distance refuses the join"); -} - -/// Both relays carry a copy, `NEAR`'s pulled from `FAR`'s. `FAR` never takes -/// `NEAR`'s in return, since that copy came through `FAR`. When `FAR`'s uplink -/// closes the two swap, each still fed from the publisher. The swap loops for as -/// long as the news takes to cross their link, which a cause shortens. -#[test] -fn two_copies_never_feed_each_other() { - for rules in [Rules::BARE, Rules::CAUSES, Rules::EPOCHS] { - let mut sim = uplinks(rules); - sim.carry(FAR, 7, "live"); - assert_eq!(settle(&mut sim), (0, 0), "{rules:?}"); - sim.carry(NEAR, 7, "live"); - assert_eq!(settle(&mut sim), (0, 0), "{rules:?}"); - assert_eq!(pulls(&sim, FAR), Some((HOME, 13, false)), "{rules:?}"); - assert_eq!(pulls(&sim, NEAR), Some((FAR, 3, true)), "{rules:?}"); - - // Until `NEAR` hears, it still pulls from `FAR`, which now pulls from it. - sim.disconnect(HOME, FAR); - let (loops, _) = settle(&mut sim); - let limit = match rules.causes { - true => 3, - false => 6, - }; - assert!( - loops <= limit, - "{rules:?}: {loops} relays astray, summed over deliveries" - ); - assert_eq!(pulls(&sim, NEAR), Some((HOME, 11, false)), "{rules:?}"); - assert_eq!(pulls(&sim, FAR), Some((NEAR, 3, true)), "{rules:?}"); - assert_eq!(pulls(&sim, TAIL), Some((FAR, 1, true)), "{rules:?}"); - assert_eq!(astray(&sim), (0, 0), "{rules:?}"); - } -} - -/// `NEAR` pulls `FAR`'s copy, whose path crosses `HOME`, and then loses its own -/// session to `HOME`, which casts doubt on `HOME`. Doubt keeps a reader from moving -/// to a copy, not on the one it pulls, so `NEAR` stays on `FAR`'s throughout. -#[test] -fn a_doubt_leaves_the_copy_in_use_alone() { - for rules in [Rules::waits(4), Rules::EPOCHS] { - let mut sim = uplinks(rules); - sim.carry(FAR, 7, "live"); - assert_eq!(settle(&mut sim), (0, 0), "{rules:?}"); - sim.carry(NEAR, 7, "live"); - assert_eq!(settle(&mut sim), (0, 0), "{rules:?}"); - assert_eq!(pulls(&sim, NEAR), Some((FAR, 3, true)), "{rules:?}"); - - sim.disconnect(HOME, NEAR); - assert_eq!(pulls(&sim, NEAR), Some((FAR, 3, true)), "{rules:?}"); - for _ in 0..64 { - match sim.busy().first() { - Some(&(from, to)) => sim.deliver(from, to), - None => sim.tick(), - }; - assert_eq!(pulls(&sim, NEAR), Some((FAR, 3, true)), "{rules:?}"); - } - assert!(sim.busy().is_empty() && !sim.tick(), "{rules:?}"); - assert_eq!(astray(&sim), (0, 0), "{rules:?}"); - } -} - -/// Readers arrive at both relays in the same instant, so each pulls over its own -/// uplink and then hears of the other's copy. By their paths alone each would move -/// onto the other's, see its own copy's path come back, move off again, and so on -/// for good. Ranking the two lets exactly one move. -#[test] -fn two_copies_started_together_settle_on_one_uplink() { - for rules in [Rules::BARE, Rules::CAUSES, Rules::EPOCHS] { - let mut sim = uplinks(rules); - sim.carry(FAR, 7, "live"); - sim.carry(NEAR, 7, "live"); - assert_eq!(settle(&mut sim).0, 0, "{rules:?}"); - let (near, far) = (pulls(&sim, NEAR).unwrap(), pulls(&sim, FAR).unwrap()); - assert!( - [ - ((HOME, 11, false), (NEAR, 3, true)), - ((FAR, 3, true), (HOME, 13, false)) - ] - .contains(&(near, far)), - "{rules:?}: {near:?} {far:?}" - ); - assert_eq!(astray(&sim), (0, 0), "{rules:?}"); - - let unranked = Rules { ranked: false, ..rules }; - let mut sim = uplinks(unranked); - sim.carry(FAR, 7, "live"); - sim.carry(NEAR, 7, "live"); - assert!(!sim.run(), "{rules:?}: the two trade copies for good"); - } -} - -/// The publisher leaves while both relays carry a copy. Every route to either copy -/// goes with it and neither relay adopts the other's. A cause ends them without a -/// loop; a bare path leaves each relay to try the other's copy first. -#[test] -fn a_copy_that_goes_takes_its_routes_along() { - for rules in [Rules::BARE, Rules::CAUSES, Rules::EPOCHS] { - let mut sim = uplinks(rules); - sim.carry(FAR, 7, "live"); - assert_eq!(settle(&mut sim), (0, 0), "{rules:?}"); - sim.carry(NEAR, 7, "live"); - assert_eq!(settle(&mut sim), (0, 0), "{rules:?}"); - - sim.disconnect(ORIGIN, HOME); - let (loops, _) = settle(&mut sim); - for relay in [HOME, NEAR, FAR, TAIL] { - assert_eq!(pulls(&sim, relay), None, "{rules:?}: relay {relay}"); - assert_eq!(sim.nodes[relay].retained(), (0, 0), "{rules:?}: relay {relay}"); - } - assert_eq!(loops > 0, !rules.causes, "{rules:?}: {loops}"); - } -} diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index 900fad32d45..6c47635252b 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -7,7 +7,6 @@ mod cache; mod compare; -mod copies; mod handoff; mod paths; mod residual; @@ -115,25 +114,6 @@ enum Msg { id: Id, seqno: Seqno, }, - /// A warm route for one of [`cache`]'s broadcasts, on the stream carrying the - /// sender's routes. - Warm { - broadcast: usize, - warm: Option, - }, -} - -/// A delivery the cache layer reacts to. -#[derive(Clone, Copy, Debug)] -enum Note { - Warm { - from: Node, - to: Node, - broadcast: usize, - warm: Option, - }, - /// A ROUTE_END, which retires the warm routes under it. - Ended { from: Node, to: Node, id: Id }, } #[derive(Clone, Debug, PartialEq, Eq)] @@ -191,8 +171,6 @@ pub(super) struct Sim { routed: BTreeMap<(Node, Node, Id), Advert>, pub stats: Stats, pub events: Vec, - /// Deliveries the cache layer has yet to take. - notes: Vec, } impl Sim { @@ -216,7 +194,6 @@ impl Sim { routed: BTreeMap::new(), stats: Stats::default(), events: Vec::new(), - notes: Vec::new(), } } @@ -447,13 +424,7 @@ impl Sim { }; let router = &mut self.nodes[to]; match msg { - Msg::Route { id, reach } => { - let ended = reach.is_none(); - router.route(from, id, reach); - if ended { - self.notes.push(Note::Ended { from, to, id }); - } - } + Msg::Route { id, reach } => router.route(from, id, reach), Msg::Batch { id, changes, @@ -461,12 +432,6 @@ impl Sim { interest, } => router.batch(from, id, changes, floor, interest), Msg::Request { id, seqno } => router.request(from, id, seqno), - Msg::Warm { broadcast, warm } => self.notes.push(Note::Warm { - from, - to, - broadcast, - warm, - }), } self.stats.delivered += 1; self.flush(); diff --git a/rs/moq-net/src/route/tests/paths.rs b/rs/moq-net/src/route/tests/paths.rs index d550234db6d..4a13ac303b9 100644 --- a/rs/moq-net/src/route/tests/paths.rs +++ b/rs/moq-net/src/route/tests/paths.rs @@ -19,10 +19,12 @@ //! dead relay is reported one neighbour at a time. No relay moves to a route //! through a node in doubt until that node advances its epoch. //! - **Copies.** A relay with a reader for a broadcast offers a route to its copy, -//! which `copies.rs` exercises. +//! and pulls each broadcast it has demand for from the cheapest of its publisher's +//! route and the copies offered, which `cache.rs` proves. //! //! Left out, since the comparison does not need them: scoped grants, qualified -//! keys, interests, GOAWAY, priorities, upstream marks, and RTT. +//! keys, interests, priorities, upstream marks, and RTT. GOAWAY only moves +//! pulls off a neighbour's copy. use std::{ collections::{BTreeMap, BTreeSet, VecDeque}, @@ -61,8 +63,8 @@ pub(super) struct Rules { /// The same doubt on a timer: it lifts by itself after this many hops of delay, /// with no epoch and so no message. 0 is off. pub wait: u64, - /// A relay already pulling a broadcast moves onto another relay's copy only if - /// that relay ranks below it, so two relays never trade copies. + /// A relay on a copy moves onto another only if that copy's carrier ranks below + /// the one it is on, so relays never trade copies. pub ranked: bool, } @@ -128,17 +130,16 @@ pub(super) enum Msg { /// before that. Epoch { node: Id, epoch: u64 }, /// A copy of one broadcast is `cost` away through the sender, having come from - /// its publisher along `path`: 0 where the sender carries it. - /// `carrier` is the relay that carries it. + /// its publisher along `path`: 0 at its carrier, the relay with a reader where the + /// copy left the publisher's route. Warm { - id: Id, - claim: Path, + cast: Cast, path: Arc<[Hop]>, cost: Metric, carrier: Id, }, /// The sender offers no copy of that broadcast any more. - Cold { id: Id, claim: Path }, + Cold { cast: Cast }, /// Changes to a route's claims, complete as of the publisher's revision `rev`. Claims { id: Id, @@ -188,6 +189,8 @@ struct Session { hop: Id, /// Whether the neighbour passes routes on, as a relay does and a client does not. forwards: bool, + /// Whether the neighbour reads copies: a relay that chooses per broadcast. + reads: bool, cost: Metric, rx: BTreeMap, tx: BTreeMap, @@ -198,7 +201,7 @@ struct Session { /// The copies this neighbour offers. copies: BTreeMap, /// The copies this neighbour was offered. - casts: BTreeMap, Metric, Id)>, + casts: BTreeMap, } /// The route a relay uses for a publisher, and the claims it shows. @@ -214,8 +217,41 @@ struct Route { rev: Rev, } -/// One broadcast: its publisher and its claim. -pub(super) type Cast = (Id, Path); +/// One broadcast: its publisher, its path, and its instance. Two instances of one +/// path are two broadcasts, and a copy of one never serves the other. +pub(super) type Cast = (Id, Path, u64); + +/// A relay's demand for one broadcast, and where it pulls it from. +#[derive(Clone, Debug, Default)] +struct Carry { + /// A reader of its own, such as a session that forwards nothing: the relay + /// carries a copy, and offers it. + reader: bool, + /// Each neighbour's subscription streams that arrived and were not cancelled. A + /// neighbour that moves back here opens a new stream, which can arrive before the + /// old one's cancel. + downstream: BTreeMap>, + /// The neighbour it pulls through, and whether that joins a copy. + parent: Option<(Node, bool)>, + /// The stream of the pull through `parent`. + stream: u64, + /// Demand is gone, and the copy stays until the linger runs out. + lingering: bool, +} + +/// The draft's Rendezvous Hash of a broadcast's path and a relay, which ranks two +/// neighbours offering each other a copy. +pub(super) fn rendezvous(path: &Path, node: Id) -> u64 { + let mut hash: u64 = 0x420C_0DEC_B00B; + for byte in path.as_str().bytes().chain(node.to_le_bytes()) { + hash = (hash ^ u64::from(byte)).wrapping_mul(0x0100_0000_01b3); + } + hash +} + +/// A copy of a broadcast offered to a neighbour: the path it took from its +/// publisher, its cost from its carrier, and its carrier. +pub(super) type Offer = (Arc<[Hop]>, Metric, Id); /// A copy of a broadcast a neighbour offers. #[derive(Clone, Debug)] @@ -237,7 +273,7 @@ pub(super) struct Source { pub path: Arc<[Hop]>, /// Whether it joins a copy, rather than follow the publisher's route. pub warm: bool, - /// The relay carrying that copy. + /// That copy's carrier. pub carrier: Id, } @@ -269,9 +305,23 @@ pub(super) struct Relay { /// With a timer in place of epochs: when the doubt on each node lifts. until: BTreeMap, now: u64, - /// The broadcasts this relay has a reader for, so it carries a copy, each with - /// the neighbour it pulls from and whether that is a copy. - readers: BTreeMap>, + /// Each broadcast this relay has demand for, or still pulls while its linger runs. + carry: BTreeMap, + /// Never sends or reads a copy: it pulls along its publisher's route alone, as + /// an endpoint that does not choose per broadcast. + pub legacy: bool, + /// Neighbours that sent GOAWAY: no pull joins, or stays on, their copies. + draining: BTreeSet, + /// The source last chosen for each broadcast, pulled or only offered on: its + /// neighbour, whether it is a copy, and that copy's carrier. + via: BTreeMap, + /// Subscription changes for the request streams: a pull, or a cancel, each on + /// its stream. + subs: Vec<(Node, Cast, u64, bool)>, + /// The last subscription stream opened. + streams: u64, + /// A pull moved from one neighbour to another. + pub moves: usize, shown: BTreeMap<(Id, Path), (Rev, bool)>, out: Vec<(Node, Msg)>, events: Vec<(Id, Path, Kind)>, @@ -308,7 +358,13 @@ impl Relay { doubt: BTreeMap::new(), until: BTreeMap::new(), now: 0, - readers: BTreeMap::new(), + carry: BTreeMap::new(), + legacy: false, + draining: BTreeSet::new(), + via: BTreeMap::new(), + subs: Vec::new(), + streams: 0, + moves: 0, shown: BTreeMap::new(), out: Vec::new(), events: Vec::new(), @@ -323,12 +379,13 @@ impl Relay { std::mem::take(&mut self.events) } - pub fn connect(&mut self, peer: Node, hop: Id, forwards: bool, cost: Metric) { + pub fn connect(&mut self, peer: Node, hop: Id, forwards: bool, reads: bool, cost: Metric) { self.opened += 1; let session = Session { id: self.opened, hop, forwards, + reads, cost, rx: BTreeMap::new(), tx: BTreeMap::new(), @@ -356,6 +413,10 @@ impl Relay { }; let lost = session.rx.keys().copied().collect(); let far = (session.hop, self.epochs.get(&session.hop).copied().unwrap_or(0)); + self.draining.remove(&peer); + for carry in self.carry.values_mut() { + carry.downstream.remove(&peer); + } if self.rules.epochs && session.forwards { // This relay is still here, whatever its neighbour tells the others. A // neighbour that passes nothing on tells nobody. @@ -436,6 +497,8 @@ impl Relay { } Msg::End { id } => { session.rx.remove(&id); + // A ROUTE_END retires the copies offered under the route. + session.copies.retain(|cast, _| cast.0 != id); self.update(id); } Msg::Claims { id, changes, rev } => { @@ -464,12 +527,14 @@ impl Relay { self.warm(None); } Msg::Warm { - id, - claim, + cast, path, cost, carrier, } => { + if self.legacy { + return; + } let dead = state(&self.dead, &path) == State::Dead; let copy = Copy { path, @@ -477,12 +542,14 @@ impl Relay { carrier, dead, }; - session.copies.insert((id, claim), copy); + let id = cast.0; + session.copies.insert(cast, copy); self.warm(Some(id)); self.collect(); } - Msg::Cold { id, claim } => { - session.copies.remove(&(id, claim)); + Msg::Cold { cast } => { + let id = cast.0; + session.copies.remove(&cast); self.warm(Some(id)); self.collect(); } @@ -519,6 +586,7 @@ impl Relay { } else if reported && rules.batch { if gone || !rules.holds { session.rx.remove(&id); + session.copies.retain(|cast, _| cast.0 != id); } else if !child(&heard.path, me) { heard.state = State::Held; } @@ -612,6 +680,7 @@ impl Relay { Want::Keep => rest += 1, Want::End => { session.tx.remove(&id); + session.casts.retain(|cast, _| cast.0 != id); ended.push(id); } } @@ -654,40 +723,156 @@ impl Relay { /// A reader for `cast` arrives at this relay, which carries a copy from now on. pub fn carry(&mut self, cast: Cast) { let id = cast.0; - self.readers.insert(cast, None); + self.carry.entry(cast).or_default().reader = true; self.warm(Some(id)); } /// The reader for `cast` leaves. pub fn uncarry(&mut self, cast: &Cast) { - self.readers.remove(cast); + if let Some(carry) = self.carry.get_mut(cast) { + carry.reader = false; + } self.warm(Some(cast.0)); } - /// Where this relay pulls `cast` from: the cheapest of its publisher's route and - /// any copy whose path does not name this relay. A tie goes to the copy. A relay - /// that already pulls it stays put unless the cheaper way is the publisher's - /// route, the same neighbour, or a copy whose carrier ranks below this relay. + /// A subscription for `cast` from `from` arrives on `stream`, or its cancel. + pub fn subscribed(&mut self, from: Node, cast: Cast, stream: u64, pull: bool) { + if !self.sessions.contains_key(&from) { + return; + } + let id = cast.0; + let carry = self.carry.entry(cast).or_default(); + let streams = carry.downstream.entry(from).or_default(); + match pull { + true => streams.insert(stream), + false => streams.remove(&stream), + }; + if streams.is_empty() { + carry.downstream.remove(&from); + } + self.warm(Some(id)); + } + + /// `peer` sent GOAWAY: every pull on its copy moves off. + pub fn goaway(&mut self, peer: Node) { + if self.sessions.contains_key(&peer) && self.draining.insert(peer) { + self.warm(None); + } + } + + /// The linger for `cast` runs out: a copy with no demand left goes. + pub fn expire(&mut self, cast: &Cast) { + if let Some(carry) = self.carry.get_mut(cast) + && carry.lingering + { + carry.lingering = false; + if let Some((parent, _)) = carry.parent.take() + && self.sessions.contains_key(&parent) + { + self.subs.push((parent, cast.clone(), carry.stream, false)); + } + } + self.warm(Some(cast.0)); + } + + /// Where this relay pulls `cast` from: the cheapest of its next hop toward the + /// publisher and each copy offered. Pulling through the next hop goes wherever + /// that neighbour pulls: its copy when it offers one, else its own route. A copy + /// whose path names this relay, or a neighbour it serves, would loop. + /// + /// A relay already on a copy, its own included, joins another, its next hop's + /// included, only if that copy's carrier ranks below, so copies crossing on their + /// way never trade places round any cycle. Off a copy, it is bound the same way by + /// the carriers of the neighbours that pull, or would pull, along its route, since + /// what it joins they take with it. Left with no source and no such neighbour, it + /// chooses afresh. It neither joins nor stays on a copy of a neighbour that sent + /// GOAWAY, unless that is its next hop's. A tie stays on the current source, then + /// goes to the copy. pub fn source(&self, cast: &Cast) -> Option { + let carry = self.carry.get(cast); + let reader = carry.is_some_and(|carry| carry.reader); + let downstream = carry.map(|carry| &carry.downstream); + let serves: BTreeSet = (downstream.into_iter().flat_map(BTreeMap::keys)) + .filter_map(|peer| self.sessions.get(peer)) + .map(|session| session.hop) + .collect(); + let via = self.via.get(cast).copied(); + let pull = via.map(|(next, warm, _)| (next, warm)); + let rank = |node: Id| rendezvous(&cast.1, node); + // The carrier of the copy this relay is on: the one it joins while that is still + // offered, or its own, pulled along the publisher's route for a reader. + let own = match via { + Some((next, true, carrier)) => (self.sessions.get(&next)) + .and_then(|session| session.copies.get(cast)) + .filter(|copy| copy.carrier == carrier) + .map(|_| carrier), + Some((_, false, _)) if reader => Some(self.hop), + _ => None, + }; + let next = self + .routes + .get(&cast.0) + .and_then(|route| route.next.zip(route.path.clone())); + // Off a copy, the copies it would hand on are those of each neighbour that pulls + // through it, or would on its route, since that neighbour's next hop's copy is no + // choice: each offers its copy back along this relay's route. Joining another + // would hand each of them another carrier, so it joins only one that ranks below + // them all, and none until each that pulls through it has offered one: an offer + // along any other path is from before it took the route, and a neighbour that + // offers none may feed copies further on. `None` joins nothing it chooses. + let chosen: Option> = match own { + Some(own) => Some(Some(own)), + None => (self.sessions.iter()) + .filter_map(|(peer, session)| { + let pulls = downstream.is_some_and(|downstream| downstream.contains_key(peer)); + let behind = (session.rx.get(&cast.0)).is_some_and(|heard| child(&heard.path, self.hop)); + let fresh = (session.copies.get(cast)) + .filter(|copy| next.as_ref().is_some_and(|(_, path)| copy.path.starts_with(path))) + .map(|copy| copy.carrier); + match fresh { + Some(carrier) if pulls || behind => Some(Some(carrier)), + None if pulls => Some(None), + _ => None, + } + }) + .try_fold(None, |low: Option, carrier| { + let carrier = carrier?; + Some(Some(low.filter(|&low| rank(low) < rank(carrier)).unwrap_or(carrier))) + }), + }; let mut options = Vec::new(); - options.extend(self.routes.get(&cast.0).and_then(|route| { - Some(Source { - next: route.next?, - cost: route.cost, - path: route.path.clone()?, - warm: false, - carrier: 0, - }) - })); - let pull = self.readers.get(cast).copied().flatten(); for (&peer, session) in &self.sessions { + let hop = next.as_ref().filter(|(next, _)| *next == peer); let Some(copy) = session.copies.get(cast) else { + options.extend(hop.map(|(next, path)| Source { + next: *next, + cost: self.routes[&cast.0].cost, + path: path.clone(), + warm: false, + carrier: 0, + })); continue; }; // As in `select`, doubt keeps a reader from moving to a copy, not on the one // it pulls: that copy's own neighbour says when it goes. let doubted = pull != Some((peer, true)) && self.doubted(©.path); - if copy.dead || names(©.path, self.hop) || doubted { + let loops = copy + .path + .iter() + .any(|hop| hop.node == self.hop || serves.contains(&hop.node)); + if copy.dead || loops || doubted { + continue; + } + let ranked = |on: Id| copy.carrier == on || rank(copy.carrier) < rank(on); + let on = match hop { + Some(_) => Some(own), + None => chosen, + }; + if self.rules.ranked && !on.is_some_and(|on| on.is_none_or(ranked)) { + continue; + } + // The next hop's copy is where its route leads; any other is a choice. + if hop.is_none() && self.draining.contains(&peer) { continue; } let Some(cost) = copy.cost.checked_add(session.cost) else { @@ -706,37 +891,82 @@ impl Relay { carrier: copy.carrier, }); } - let pull = pull.filter(|pull| options.iter().any(|source| (source.next, source.warm) == *pull)); - let rank = |node: Id| fnv(cast.1.as_str(), &[cast.0, node]); - let free = |source: &Source| match pull { - Some((next, _)) if self.rules.ranked => { - !source.warm || source.next == next || rank(source.carrier) < rank(self.hop) - } - _ => true, - }; - (options.into_iter()) - .filter(free) - .min_by_key(|source| (source.cost, !source.warm, source.next)) + (options.into_iter()).min_by_key(|source| { + let stay = pull == Some((source.next, source.warm)); + (source.cost, !stay, !source.warm, source.next) + }) } - /// What this relay offers its neighbours for `cast`: its own copy at no cost, or - /// a copy it would join, for as long as that is cheaper than the publisher's route. - fn copy(&self, cast: &Cast) -> Option<(Arc<[Hop]>, Metric, Id)> { - if self.own == Some(cast.0) || self.leaf { + /// What this relay offers its neighbours for `cast`, under its route to the + /// publisher: a copy it would join, at what joining it costs, for as long as that + /// is cheaper than the publisher's route, or else, with a reader of its own, the + /// copy it pulls along the publisher's route at no cost. A copy's cost so rises + /// with every link from where it left the publisher's route. + fn copy(&self, cast: &Cast) -> Option { + if self.own == Some(cast.0) || self.leaf || self.legacy { return None; } + self.routes.get(&cast.0)?.path.as_ref()?; let source = self.source(cast)?; - match self.readers.contains_key(cast) { + match self.carry.get(cast).is_some_and(|carry| carry.reader) { + _ if source.warm => Some((source.path, source.cost, source.carrier)), true => Some((source.path, 0, self.hop)), - false if source.warm => Some((source.path, source.cost, source.carrier)), false => None, } } - /// Bring what each neighbour was offered up to date, for the copies of `id` or - /// for every copy. A neighbour is offered no copy whose path names it. + /// Follow the demand for `cast`: pull it from its source, move with the source, + /// and keep the pull while the linger runs once the demand is gone. + fn pull(&mut self, cast: &Cast) { + let source = self.source(cast); + match &source { + Some(source) => self + .via + .insert(cast.clone(), (source.next, source.warm, source.carrier)), + None => self.via.remove(cast), + }; + let source = source.map(|source| (source.next, source.warm)); + let Some(carry) = self.carry.get_mut(cast) else { + return; + }; + let demand = carry.reader || !carry.downstream.is_empty(); + if demand { + carry.lingering = false; + } else if carry.parent.is_some() { + carry.lingering = true; + } + let parent = source.filter(|_| demand || carry.lingering); + let old = std::mem::replace(&mut carry.parent, parent); + let (old, new) = (old.map(|(next, _)| next), parent.map(|(next, _)| next)); + let stream = carry.stream; + if new.is_some() && old != new { + self.streams += 1; + carry.stream = self.streams; + } + if !demand && parent.is_none() { + self.carry.remove(cast); + } + if old == new { + return; + } + if let Some(old) = old + && self.sessions.contains_key(&old) + { + self.subs.push((old, cast.clone(), stream, false)); + } + if let Some(new) = new { + self.subs.push((new, cast.clone(), self.streams, true)); + } + if old.is_some() && new.is_some() { + self.moves += 1; + } + } + + /// Bring each pull, and what each neighbour was offered, up to date, for the + /// copies of `id` or for every copy. A neighbour that reads copies is offered this + /// relay's, even one whose path names it, under the route it was sent. fn warm(&mut self, id: Option) { - let mut casts: BTreeSet = self.readers.keys().cloned().collect(); + let mut casts: BTreeSet = self.carry.keys().cloned().collect(); for session in self.sessions.values() { casts.extend(session.copies.keys().cloned()); casts.extend(session.casts.keys().cloned()); @@ -745,26 +975,40 @@ impl Relay { if id.is_some_and(|id| id != cast.0) { continue; } - let pull = self.source(&cast).map(|source| (source.next, source.warm)); - if let Some(reader) = self.readers.get_mut(&cast) { - *reader = pull; + // What this relay offers decides which offers it may join, so the two settle + // together: joining a neighbour's copy only ends what it offers that neighbour. + for _ in 0..4 { + self.pull(&cast); + if !self.offer(&cast) { + break; + } } + } + } + + /// Bring what each neighbour was offered for `cast` up to date. Whether anything changed. + fn offer(&mut self, cast: &Cast) -> bool { + let mut changed = false; + { + let cast = cast.clone(); let copy = self.copy(&cast); let peers: Vec = self.sessions.keys().copied().collect(); for peer in peers { let session = &self.sessions[&peer]; - let want = copy.clone().filter(|(path, ..)| !names(path, session.hop)); + // Even a copy whose path names the neighbour: it tells the neighbour that + // pulling through this relay leads back to it. + let routed = session.tx.contains_key(&cast.0) && session.reads; + let want = copy.clone().filter(|_| routed); if session.casts.get(&cast) == want.as_ref() { continue; } - let (id, claim) = cast.clone(); + changed = true; match want { Some((path, cost, carrier)) => { let session = self.sessions.get_mut(&peer).unwrap(); session.casts.insert(cast.clone(), (path.clone(), cost, carrier)); let msg = Msg::Warm { - id, - claim, + cast: cast.clone(), path: path.clone(), cost, carrier, @@ -774,11 +1018,60 @@ impl Relay { } None => { self.sessions.get_mut(&peer).unwrap().casts.remove(&cast); - self.out.push((peer, Msg::Cold { id, claim })); + self.out.push((peer, Msg::Cold { cast: cast.clone() })); } } } } + changed + } + + pub fn drain_subs(&mut self) -> Vec<(Node, Cast, u64, bool)> { + std::mem::take(&mut self.subs) + } + + /// The neighbour this relay pulls `cast` through, if it pulls it. + pub fn parent(&self, cast: &Cast) -> Option { + self.carry.get(cast)?.parent.map(|(next, _)| next) + } + + pub fn reading(&self, cast: &Cast) -> bool { + self.carry.get(cast).is_some_and(|carry| carry.reader) + } + + pub fn downstream(&self, cast: &Cast) -> BTreeSet { + self.carry + .get(cast) + .map(|carry| carry.downstream.keys().copied().collect()) + .unwrap_or_default() + } + + /// The subscription streams for `cast` that arrived and were not cancelled. + pub fn streams(&self, cast: &Cast) -> usize { + (self.carry.get(cast)).map_or(0, |carry| carry.downstream.values().map(BTreeSet::len).sum()) + } + + /// Whether the copy of `cast` stays only while its linger runs. + pub fn lingering(&self, cast: &Cast) -> bool { + self.carry.get(cast).is_some_and(|carry| carry.lingering) + } + + /// The broadcasts this relay pulls while its linger runs. + pub fn lingers(&self) -> Vec { + (self.carry.iter()) + .filter(|(_, carry)| carry.lingering) + .map(|(cast, _)| cast.clone()) + .collect() + } + + /// The copy of `cast` this relay offers `peer`, and the one `peer` offers it. + pub fn offers(&self, peer: Node, cast: &Cast) -> (Option, Option) { + let Some(session) = self.sessions.get(&peer) else { + return (None, None); + }; + let sent = session.casts.get(cast).cloned(); + let heard = (session.copies.get(cast)).map(|copy| (copy.path.clone(), copy.cost, copy.carrier)); + (sent, heard) } /// Forget each dead hop no neighbour's route names any more. @@ -1013,6 +1306,7 @@ impl Relay { } Want::End => { if session.tx.remove(&id).is_some() { + session.casts.retain(|cast, _| cast.0 != id); self.out.push((peer, Msg::End { id })); } continue; @@ -1127,6 +1421,8 @@ pub(super) struct PathStats { pub claim_items: usize, /// Copy offers and their ends, also counted as route messages. pub warm_msgs: usize, + /// Subscriptions and their cancels. + pub subscribes: usize, pub delivered: usize, } @@ -1138,10 +1434,15 @@ pub(super) struct PathSim { links: Vec>>, up: BTreeSet<(Node, Node)>, queues: BTreeMap<(Node, Node), VecDeque>, + /// Subscriptions, each on a stream of its own, unordered against the routes and + /// each other: true pulls, false cancels. + subs: BTreeMap<(Node, Node, Cast, u64), VecDeque>, held: BTreeSet<(Node, Node)>, /// Nodes whose process is gone and has yet to restart. stopped: BTreeSet, pub stats: PathStats, + /// Copy offers and their ends sent per broadcast. + pub warm: BTreeMap, pub events: Vec<(Node, Id, String, Kind)>, } @@ -1154,9 +1455,11 @@ impl PathSim { links: vec![vec![None; nodes]; nodes], up: BTreeSet::new(), queues: BTreeMap::new(), + subs: BTreeMap::new(), held: BTreeSet::new(), stopped: BTreeSet::new(), stats: PathStats::default(), + warm: BTreeMap::new(), events: Vec::new(), } } @@ -1187,7 +1490,8 @@ impl PathSim { let cost = self.links[x][y].expect("a configured link"); let peer = hop(y, self.restarts[y]); let forwards = !self.nodes[y].leaf; - self.nodes[x].connect(y, peer, forwards, cost); + let reads = forwards && !self.nodes[y].legacy; + self.nodes[x].connect(y, peer, forwards, reads, cost); } self.flush(); } @@ -1200,6 +1504,7 @@ impl PathSim { self.up.remove(&Self::pair(a, b)); for (x, y) in [(a, b), (b, a)] { self.queues.remove(&(x, y)); + self.subs.retain(|(from, to, ..), _| (*from, *to) != (x, y)); self.held.remove(&(x, y)); self.nodes[y].disconnect(x); } @@ -1212,6 +1517,7 @@ impl PathSim { self.up.remove(&Self::pair(node, peer)); for key in [(node, peer), (peer, node)] { self.queues.remove(&key); + self.subs.retain(|(from, to, ..), _| (*from, *to) != key); self.held.remove(&key); } self.nodes[peer].disconnect(node); @@ -1260,20 +1566,75 @@ impl PathSim { self.flush(); } - /// A reader for `claim` of `id` arrives at `node`. - pub fn carry(&mut self, node: Node, id: Id, claim: &str) { - self.nodes[node].carry((id, p(claim))); + /// A reader for `cast` arrives at `node`. + pub fn carry(&mut self, node: Node, cast: &Cast) { + self.nodes[node].carry(cast.clone()); self.flush(); } - pub fn uncarry(&mut self, node: Node, id: Id, claim: &str) { - self.nodes[node].uncarry(&(id, p(claim))); + pub fn uncarry(&mut self, node: Node, cast: &Cast) { + self.nodes[node].uncarry(cast); self.flush(); } - /// Where `node` pulls `claim` of `id` from. - pub fn source(&self, node: Node, id: Id, claim: &str) -> Option { - self.nodes[node].source(&(id, p(claim))) + /// `node` sends GOAWAY on every session. + pub fn goaway(&mut self, node: Node) { + for peer in self.peers(node) { + self.nodes[peer].goaway(node); + } + self.flush(); + } + + /// The linger for `cast` runs out at `node`. + pub fn expire(&mut self, node: Node, cast: &Cast) { + self.nodes[node].expire(cast); + self.flush(); + } + + pub fn hold(&mut self, from: Node, to: Node) { + self.held.insert((from, to)); + } + + pub fn release(&mut self, from: Node, to: Node) { + self.held.remove(&(from, to)); + } + + /// The subscription streams with a change to deliver, held or not. + pub fn pending_subs(&self) -> Vec<(Node, Node, Cast, u64)> { + self.subs.keys().cloned().collect() + } + + /// The subscription streams with a change to deliver that are not held. + pub fn busy_subs(&self) -> Vec<(Node, Node, Cast, u64)> { + (self.subs.keys()) + .filter(|(from, to, ..)| !self.held.contains(&(*from, *to))) + .cloned() + .collect() + } + + /// Deliver the next change on the oldest stream of `cast`'s subscriptions from + /// `from` to `to`. + pub fn deliver_sub(&mut self, from: Node, to: Node, cast: &Cast) -> bool { + let stream = (self.subs.keys()).find(|key| (key.0, key.1, &key.2) == (from, to, cast)); + match stream.map(|key| key.3) { + Some(stream) => self.deliver_stream(from, to, cast, stream), + None => false, + } + } + + /// Deliver the next change on one subscription stream from `from` to `to`. + pub fn deliver_stream(&mut self, from: Node, to: Node, cast: &Cast, stream: u64) -> bool { + let key = (from, to, cast.clone(), stream); + let Some(pull) = self.subs.get_mut(&key).and_then(VecDeque::pop_front) else { + return false; + }; + if self.subs[&key].is_empty() { + self.subs.remove(&key); + } + self.nodes[to].subscribed(from, cast.clone(), stream, pull); + self.stats.delivered += 1; + self.flush(); + true } /// One hop of delay passes at every running node. Whether any still waits on a timer. @@ -1310,16 +1671,20 @@ impl PathSim { true } - /// Deliver round-robin until every unheld queue is empty, or give up. + /// Deliver round-robin until every unheld queue, subscriptions included, is + /// empty, or give up. pub fn run(&mut self) -> bool { for _ in 0..10_000 { - let ready = self.busy(); - if ready.is_empty() { + let (ready, subs) = (self.busy(), self.busy_subs()); + if ready.is_empty() && subs.is_empty() { return true; } for (from, to) in ready { self.deliver(from, to); } + for (from, to, cast, stream) in subs { + self.deliver_stream(from, to, &cast, stream); + } } false } @@ -1332,10 +1697,11 @@ impl PathSim { self.stats.route_msgs += 1; self.stats.route_items += 1; } - Msg::Warm { .. } | Msg::Cold { .. } => { + Msg::Warm { cast, .. } | Msg::Cold { cast } => { self.stats.route_msgs += 1; self.stats.route_items += 1; self.stats.warm_msgs += 1; + *self.warm.entry(cast.clone()).or_default() += 1; } Msg::Dead { moved, .. } => { self.stats.route_msgs += 1; @@ -1348,6 +1714,10 @@ impl PathSim { } self.queues.entry((node, to)).or_default().push_back(msg); } + for (to, cast, stream, pull) in self.nodes[node].drain_subs() { + self.stats.subscribes += 1; + self.subs.entry((node, to, cast, stream)).or_default().push_back(pull); + } for (id, claim, kind) in self.nodes[node].events() { self.events.push((node, id, claim.as_str().to_string(), kind)); } diff --git a/rs/moq-net/src/route/tests/residual.rs b/rs/moq-net/src/route/tests/residual.rs index 34b107603f0..d6ea9c57841 100644 --- a/rs/moq-net/src/route/tests/residual.rs +++ b/rs/moq-net/src/route/tests/residual.rs @@ -75,10 +75,7 @@ fn residual_a_relay_behind_a_scoped_neighbour_is_blind() { "node {node} sees it: drop the residual" ); } - let routes: Vec = (sim.nodes[relay].eligible(7).into_iter()) - .filter(|route| !route.tied) - .map(|route| route.next) - .collect(); + let routes: Vec = sim.nodes[relay].routes(7).into_iter().map(|(next, ..)| next).collect(); assert_eq!(routes, [narrow], "the wide neighbour is no route either"); } diff --git a/rs/moq-net/src/route/tests/rtt.rs b/rs/moq-net/src/route/tests/rtt.rs index 5c64d861b1d..0abdd554600 100644 --- a/rs/moq-net/src/route/tests/rtt.rs +++ b/rs/moq-net/src/route/tests/rtt.rs @@ -44,16 +44,8 @@ fn each_relay_adds_its_link_to_the_sum() { assert_eq!(advertised(&sim, origin, a, 7), ms(0), "the origin's own route"); assert_eq!(advertised(&sim, a, b, 7), ms(10)); assert_eq!(advertised(&sim, b, c, 7), ms(30)); - let routes = sim.nodes[c].eligible(7); - assert_eq!( - routes, - [Eligible { - next: b, - metric: 3, - rtt: ms(60), - tied: false - }] - ); + let routes = sim.nodes[c].routes(7); + assert_eq!(routes, [(b, 3, ms(60))]); } /// A shift on one link sends one ROUTE_UPDATE on each stream carrying the route from @@ -85,7 +77,7 @@ fn an_unknown_link_ends_the_sum() { assert!(sim.run()); assert_eq!(advertised(&sim, a, b, 7), ms(10)); assert_eq!(advertised(&sim, b, c, 7), None); - assert_eq!(sim.nodes[c].eligible(7)[0].rtt, None); + assert_eq!(sim.nodes[c].routes(7)[0].2, None); } /// The metric selects; a far lower RTT through a costlier link changes nothing.