diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index 635f7bbfcb8..3e1c158e955 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -90,10 +90,14 @@ for paths the grant covers. Routing prefers the longest covering prefix, then a fully identified hop list over one that holds a 0 (an anonymous hop) at any depth, then the lowest cost, -then the shortest hop list, breaking any remaining tie toward the newest -announcement so a reconnecting publisher isn't outranked by the session it -replaced. An assigned identity for an anonymous peer is local selection state -and is never written into the hop list. +then the shortest hop list, then a hash of the requested path and the hop list, +breaking any remaining tie toward the newest announcement so a reconnecting +publisher isn't outranked by the session it replaced. Hashing the requested +path spreads equal-cost advertisers of one prefix, such as a transcode pool, +across its paths instead of sending every path to one of them, and every relay +that holds the same routes picks the same one for a given path. An assigned +identity for an anonymous peer is local selection state and is never written +into the hop list. ```toml [cluster] diff --git a/drafts/draft-lcurley-moq-cluster.md b/drafts/draft-lcurley-moq-cluster.md index 24ea874a0f5..abf04226e88 100644 --- a/drafts/draft-lcurley-moq-cluster.md +++ b/drafts/draft-lcurley-moq-cluster.md @@ -242,15 +242,13 @@ The expected update is a ROUTE_COST change, which is how a relay signals that it # Path Selection {#selection} A receiver resolving a SUBSCRIBE, FETCH, or track-status request consults only the most specific advertisements covering it: the longest prefix. A refusal never falls through to a less specific tier. -Nor is it retried at another advertisement of the same prefix, except for the one re-resolution NO_CAPACITY permits below. +Nor is it retried at another advertisement of the same prefix. Within that tier, a receiver SHOULD prefer a HOP_PATH that contains no 0 entry over one that does, then the lowest ROUTE_COST, breaking ties toward the shorter HOP_PATH and then toward the most recently received. This is advisory: a receiver MAY apply local policy, such as measured RTT, instead. -NO_CAPACITY ({{iana}}) refuses a request the publisher could serve but has no capacity for now. -It permits ONE re-resolution within the same tier, excluding the refusing advertiser: every route with its non-zero first Hop ID, or its session when that ID is 0. -A receiver that has spent its retry, or has no other candidate, MUST refuse downstream with a code other than NO_CAPACITY, so retries cannot compound hop by hop. -Every other refusal, including an unrecognized code, is terminal. +Every refusal, including an unrecognized code, is terminal. +A publisher signals capacity through its advertisement alone: it withdraws or re-prices it before it runs out, leaving headroom for requests already in flight, since a withdrawal and a request for the slot it gave away can cross. A receiver SHOULD NOT cache refusals. A relay MUST NOT advertise a namespace merely because it resolved it: the covering advertisement stays the only one until the publisher advertises the concrete namespace, which it SHOULD do once producing, so a later request finds the running content by its exact namespace instead of resolving a second producer. @@ -272,7 +270,7 @@ Under this extension an advertisement is a path, so a session advertises a names A namespace names one broadcast whichever publisher serves it, so several publishers of it are redundant copies: their groups are one sequence, and a subscription moves between them without a gap. A publisher MUST NOT reuse a namespace for different content, and publishes a new instance under a new namespace instead. -A refusal moves to another publisher only as {{selection}} allows: once, and only for NO_CAPACITY. +A refusal never moves to another publisher ({{selection}}). A relay that moves to another publisher MUST update its advertisement to the new path ({{updating}}). @@ -285,7 +283,7 @@ Because a relay only appends to HOP_PATH, it cannot make a competing path look s ROUTE_COST has no such protection: it is a single value the sender chooses, so a relay can advertise 0 for content it is not carrying and attract subscriptions it then has to fetch. Both cost only a suboptimal path choice, and the latter is self-limiting, since the traffic won this way must then be served. -Implementations SHOULD bound the work started by requests beneath a broad advertisement, using NO_CAPACITY when capacity is exhausted. +Implementations SHOULD bound the work started by requests beneath a broad advertisement, withdrawing it before capacity is exhausted. A receiver MUST NOT make security decisions based on Hop IDs, and a deployment spanning a trust boundary SHOULD treat a peer's ROUTE_COST as a hint to clamp or ignore rather than an accounting figure. @@ -316,14 +314,6 @@ Both are carried in PUBLISH_NAMESPACE, in REQUEST_UPDATE of a PUBLISH_NAMESPACE The Key-Value-Pair parity is load-bearing: HOP_PATH is odd, so its value is a length-prefixed byte string, while HOP_ID, RELAY_COST, and ROUTE_COST are even, so their values are bare varints. -## MOQT Error Codes - -This document requests one registration in the "REQUEST_ERROR Codes" registry. - -| Value | Name | Reference | -|:--------|:------------|:--------------| -| 0x40B5A | NO_CAPACITY | This Document | - --- back @@ -331,7 +321,7 @@ This document requests one registration in the "REQUEST_ERROR Codes" registry. ## moq-cluster-02 - A namespace names one broadcast whichever publisher serves it, and a publisher MUST NOT reuse one for different content. A receiver MAY fail a subscription over between any of its advertisements, continuing from the first object the subscriber lacks instead of at a group boundary. Replaces the first-Hop-ID publisher identity. -- Defined request resolution against the longest covering prefix and the NO_CAPACITY refusal with its single re-resolution; any other refusal is terminal, including between several publishers of one namespace. +- Defined request resolution against the longest covering prefix; every refusal is terminal, including between several publishers of one namespace, and capacity is signaled only by withdrawing or re-pricing the advertisement. - A relay does not advertise a namespace because it resolved it; the publisher advertises the concrete namespace once producing. - A change of original publisher is an ordinary update instead of a withdrawal and a new advertisement. - A relay puts a random Hop ID, picked per session, in front of an advertisement whose first HOP_PATH entry is 0, and writes that stamp followed by 0 for one with no HOP_PATH. diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index d88487b028a..08eab96eca2 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -307,8 +307,6 @@ Sent when resetting a stream (RESET_STREAM), or when refusing to receive one (ST | ------- | ------------- | ----------- | | 0x12 | MALFORMED_TRACK | The track's content could not be parsed. | | ------- | ------------- | ----------- | -| 0x30 | NO_CAPACITY | The publisher could serve this request but has no capacity for it now. Permits one re-resolution (see [Resolution](#resolution)); elsewhere it is terminal like any refusal. Bridges to NO_CAPACITY in {{I-D.lcurley-moq-cluster}}. | -| ------- | ------------- | ----------- | | 0x31 | CONTROL_TIMEOUT | The peer took too long to answer a control request. Distinct from DELIVERY_TIMEOUT, which is content that missed its deadline; it has no moq-transport value and bridges to INTERNAL_ERROR. | | ------- | ------------- | ----------- | | 0x32 | GROUP_TOO_LARGE | The group grew past the publisher's cache budget and was aborted. | @@ -423,7 +421,11 @@ The per-subscriber winner changing travels as an ANNOUNCE_UPDATE; the last quali When serving a subscription, a publisher MUST select the source by that same exclusion; if only excluded sources remain, the subscription is unroutable. Applying one rule to both advertisement and dispatch keeps advertised paths truthful, which is what prevents subscription cycles of any length. -When resolving a path covered by several routes (across any number of streams), the subscriber SHOULD prefer the most specific covering route (see [Resolution](#resolution)), then a path that contains no 0 Hop ID over one that does, then the lowest Warm Route Cost after adding each arriving link's cost (see [Cost Parameter](#cost-parameter)), breaking ties toward the lowest Cold Route Cost, then toward the shortest path, and then toward the most recently received, so a reconnecting publisher is not outranked by the stale session it replaced. +When resolving a path covered by several routes (across any number of streams), the subscriber SHOULD prefer the most specific covering route (see [Resolution](#resolution)), then a path that contains no 0 Hop ID over one that does, then the lowest Warm Route Cost after adding each arriving link's cost (see [Cost Parameter](#cost-parameter)), breaking ties toward the lowest Cold Route Cost, then toward the shortest path, then toward the lowest Spread Hash, and then toward the most recently received, so a reconnecting publisher is not outranked by the stale session it replaced. + +The Spread Hash is the 64-bit FNV-1a hash, with offset basis `0x420C0DECB00B` and the standard FNV-64 prime, of the requested path's UTF-8 bytes followed by each Hop ID of the route's path, oldest first, as 8 little-endian bytes. +It is keyed on the requested path rather than the route's prefix, so equal-cost advertisers of one prefix share its paths instead of the first one taking them all, while one path resolves to the same advertiser on every relay that holds the same routes. +When choosing which route to advertise for a prefix, the requested path is the prefix itself. Every route covering a path serves the same Broadcast, so a relay MAY move a live subscription between them, continuing from the first frame the subscriber lacks, and a route change (a reconnect, a cheaper path, a draining session) is invisible to the subscriber. @@ -433,7 +435,7 @@ A SUBSCRIBE, FETCH, or TRACK request names a path, and the receiver resolves it Only the most specific covering routes are consulted: those with the longest prefix. A route strictly inside another's paths always ranks above it, and a broadcast's exact path is the most specific route there is, so a concrete announcement shadows every broader route covering it at any cost; routes over the same prefix form one tier. The winning tier is the whole answer: a refusal from it never falls through to a less specific route, so a service refusing a path does not leak the request to a catch-all, and one unserved path costs one round trip rather than a walk down the candidates. -Nor is a refusal retried at another route of the same prefix, except for the one re-resolution NO_CAPACITY permits below. +Nor is a refusal retried at another route of the same prefix. Within the tier, cost and the tie-breaks of [Routing](#routing) order the routes. @@ -447,11 +449,8 @@ A seed of `2^32` is RECOMMENDED only when `H * C < 2^32`; otherwise the deployme Unknown, out-of-budget, and saturated routes are outside this guarantee; a receiver MUST NOT infer that they outrank standby capacity merely because they might already carry content. An advertiser that will not serve a resolved request resets the request stream with a typed code (see [Error Codes](#error-codes)). -A NO_CAPACITY reset permits the receiver ONE re-resolution, within the same tier and excluding every route whose advertiser is the refusing one. -The exclusion is what makes the retry safe, not the retraction arriving first: an advertiser's capacity and a receiver's view of it are at least half a round trip apart, so a retraction and a request for the slot it gave away necessarily cross. -Re-resolution may find no other route, and the request is then unroutable; that is a correct outcome, not a fallback list. -A receiver that has spent its re-resolution, or has nothing to spend it on, MUST reset the downstream request with a code other than NO_CAPACITY, so the single retry cannot compound hop by hop. -Every other code, and any unrecognized one, is terminal and propagates without re-resolution, so probing unserved paths costs one round trip per path. +Every refusal is terminal and propagates without re-resolution, so probing unserved paths costs one round trip per path. +An advertiser signals capacity through its route alone: it withdraws or re-prices the route before it runs out, leaving headroom for requests already in flight, since a withdrawal and a request for the slot it gave away can cross. A receiver SHOULD NOT cache refusals; rate limiting is the advertiser's concern. ### Subscribe @@ -1360,7 +1359,7 @@ The `Message Length` describes the payload size on the wire. ## moq-lite-07 - The subscriber FINs its Subscribe Stream after settling its tail; graceful session close waits for that FIN or reset. -- A refusal is not retried at another route of the same prefix either, apart from NO_CAPACITY's one re-resolution. +- A refusal is not retried at another route of the same prefix either. - Made TRACK_INFO Publisher Max Age optional, encoded as milliseconds plus one with zero meaning no limit. - Added `Largest Group` and `Largest Frame` to SUBSCRIBE_OK: the publisher's largest position when it answers, which a subscriber takes as where the live feed is. A publisher MUST answer at once when the requested start is past it. Earlier versions carry no such position, so a subscriber takes the first frame instead. - A path names one Broadcast whichever publisher serves it, and a publisher MUST NOT reuse a path for different content. A relay MAY move a subscription between any routes covering the path, continuing from the first frame the subscriber lacks instead of at a group boundary. Replaces the first-hop identity. @@ -1371,6 +1370,7 @@ The `Message Length` describes the payload size on the wire. - Added `Stream Count` to SUBSCRIBE_END: the number of Group Streams opened for the subscription. SUBSCRIBE_END is now sent once every counted Group Stream has opened, rather than as soon as the final group is known. - Removed SUBSCRIBE_DROP and its type 0x2; a group without a Group Stream is not counted. - The Subscribe Stream FIN now follows once every counted Group Stream has finished or been reset. +- Added the Spread Hash tie-break after the shortest path: a hash of the requested path and the route's Hop IDs, so equal-cost advertisers of one prefix share its paths. - Added announce compression: ANNOUNCE_START gains `Path Base` and `Path Keep` to copy the head of a live advertisement's suffix, and ANNOUNCE_START and ANNOUNCE_UPDATE gain `Hop Base` and `Hop Keep` to copy the tail of a live advertisement's Hop ID list. - Capped the SETUP Message Length at 65,536 bytes. - A relay puts a random Hop ID, picked per session, in front of an announcement whose reconstructed path starts with 0, and writes that stamp followed by 0 for an empty path. @@ -1386,7 +1386,7 @@ The `Message Length` describes the payload size on the wire. - Corrected SUBSCRIBE_END `Group` to an exclusive bound: the first sequence that will never be delivered, with 0 meaning no groups were produced. It was previously specified as the inclusive last group, which could not distinguish an empty track from one whose only group was 0. - Split ANNOUNCE_BROADCAST into three typed messages: ANNOUNCE_START (0x0), ANNOUNCE_END (0x1), and ANNOUNCE_UPDATE (0x2), each prefixed with a Type discriminator like the subscribe stream's responses. - Specified resolution of a request against the routes covering its path: only the most specific tier, the longest prefix, is consulted, so a concrete path shadows every broader route; a refusal never falls through, and cost and the existing tie-breaks order the tier. A route is always a prefix, and no message narrows one: an advertiser that serves only some of the paths beneath its prefix refuses the rest. A relay never announces a path because it resolved it; the advertiser announces the concrete path once producing. Standby ordering requires enforced deployment bounds on charged link count and cost; 2^32 is a recommended seed only when it exceeds their product. -- Split the reserved stream error range: 32 through 47 stays reserved, and 48 through 63 is moq-lite's own, assigned by the tables and mapped rather than forwarded across a bridge. Assigned 0x30 NO_CAPACITY there: it permits one re-resolution within the tier excluding the refusing advertiser, and a receiver that has spent or lacks that retry resets downstream with another code. Assigned 0x32 GROUP_TOO_LARGE: a group that grew past the publisher's cache budget is aborted. Every other code is terminal. +- Split the reserved stream error range: 32 through 47 stays reserved, and 48 through 63 is moq-lite's own, assigned by the tables and mapped rather than forwarded across a bridge. Assigned 0x32 GROUP_TOO_LARGE: a group that grew past the publisher's cache budget is aborted. Every code is terminal. - Assigned 0x33 NOT_FOUND, 0x34 OLD, and 0x35 EVICTED in the stream error table: a group the publisher cannot serve because it was never here, has been superseded, or was dropped under memory pressure. - Assigned 0x36 UNROUTABLE, 0x37 WRONG_SIZE, 0x38 FRAME_TOO_LARGE, and 0x39 TIMESTAMP_MISMATCH in the stream error table, moving them out of the reserved 32 through 47 range, which no longer carries provisional placeholders. - Assigned 0x31 CONTROL_TIMEOUT in the stream error table: a request stream torn down because the peer never answered, which DELIVERY_TIMEOUT described as late content. It has no moq-transport value and bridges to INTERNAL_ERROR. @@ -1548,7 +1548,7 @@ GOAWAY carries an optional New Session URI that asks the peer to reconnect elsew Hop IDs (see [ANNOUNCE_OK](#announce-ok) and [ANNOUNCE_START](#announce-start)) expose the relay path of a broadcast, which may reveal internal topology. A relay that does not wish to disclose its position MAY use the reserved value 0 ("unknown") instead of a stable identifier, at the cost of losing loop detection through itself (see [Routing](#routing)). The Hop ID announcement filter (see [Hop Parameter](#hop-parameter)) exists for loop avoidance, not access control: a subscriber cannot verify that a publisher honored it, so it MUST NOT be relied upon to hide a broadcast from a peer that declared its Hop ID. ## Resource Exhaustion -A peer can open many streams (subscriptions, announcements, fetches), request large announce prefixes, or advertise broad routes. Implementations SHOULD bound the number of concurrent subscriptions, announce matches, and cached groups, and SHOULD rely on QUIC flow control and stream limits to backpressure a misbehaving peer (see [ANNOUNCE_REQUEST](#announce-request)). Expiration (see [Expiration](#expiration)) bounds how long stale groups consume memory and flow control. A broad route invites a request for any covered path, each of which may start work: an advertiser SHOULD bound the work it starts and refuse beyond that with NO_CAPACITY, and a receiver re-resolves at most once per request, so a flood of requests costs the mesh one round trip each rather than a search (see [Resolution](#resolution)). +A peer can open many streams (subscriptions, announcements, fetches), request large announce prefixes, or advertise broad routes. Implementations SHOULD bound the number of concurrent subscriptions, announce matches, and cached groups, and SHOULD rely on QUIC flow control and stream limits to backpressure a misbehaving peer (see [ANNOUNCE_REQUEST](#announce-request)). Expiration (see [Expiration](#expiration)) bounds how long stale groups consume memory and flow control. A broad route invites a request for any covered path, each of which may start work: an advertiser SHOULD bound the work it starts, withdrawing its route before it runs out, and every refusal is terminal, so a flood of requests costs the mesh one round trip each rather than a search (see [Resolution](#resolution)). ## Datagram Injection Datagrams are routed to a subscription solely by Subscribe ID and carry no per-group authentication beyond that of the QUIC connection. On an unmodified QUIC/WebTransport connection this is sufficient, since datagrams are protected by the transport. A subscriber MUST silently drop any datagram with an unknown Subscribe ID and MUST deduplicate against groups received on streams (see [Datagrams](#datagrams)). diff --git a/js/net/src/error.test.ts b/js/net/src/error.test.ts index 61bb89b0564..a12f12144d8 100644 --- a/js/net/src/error.test.ts +++ b/js/net/src/error.test.ts @@ -222,11 +222,11 @@ test("the code tables match the spec", () => { // carries nothing, so no code sits there. const assignedLite: StreamCode[] = [ StreamCode.ControlTimeout, - StreamCode.NoCapacity, StreamCode.GroupTooLarge, StreamCode.NotFound, StreamCode.Old, StreamCode.Evicted, + StreamCode.Unroutable, StreamCode.FrameTooLarge, ]; for (const code of Object.values(StreamCode)) { @@ -239,11 +239,11 @@ test("the code tables match the spec", () => { } // The values the Rust `StreamError` sends for the same conditions. expect(Number(StreamCode.ControlTimeout)).toBe(0x31); - expect(Number(StreamCode.NoCapacity)).toBe(0x30); expect(Number(StreamCode.GroupTooLarge)).toBe(0x32); expect(Number(StreamCode.NotFound)).toBe(0x33); expect(Number(StreamCode.Old)).toBe(0x34); expect(Number(StreamCode.Evicted)).toBe(0x35); + expect(Number(StreamCode.Unroutable)).toBe(0x36); expect(Number(StreamCode.FrameTooLarge)).toBe(0x38); // The spaces are disjoint: 0 ends a session cleanly but fails a stream. @@ -361,11 +361,11 @@ test("toStreamCode and fromTransport agree on what a code means", () => { StreamCode.TooFarBehind, StreamCode.MalformedTrack, StreamCode.ControlTimeout, - StreamCode.NoCapacity, StreamCode.GroupTooLarge, StreamCode.NotFound, StreamCode.Old, StreamCode.Evicted, + StreamCode.Unroutable, StreamCode.FrameTooLarge, StreamCode(70), ]) { @@ -397,6 +397,7 @@ test("toStreamCode: lite-only codes do not reach an IETF peer", () => { StreamCode.Old, StreamCode.Evicted, StreamCode.GroupTooLarge, + StreamCode.Unroutable, StreamCode.FrameTooLarge, ]) { expect(toStreamCode(new StreamError(code), { version: Version.DRAFT_20 })).toBe(StreamCode.Internal); diff --git a/js/net/src/error.ts b/js/net/src/error.ts index 9912a457ef1..91873b21394 100644 --- a/js/net/src/error.ts +++ b/js/net/src/error.ts @@ -92,8 +92,8 @@ export const StreamCode = Object.freeze( Evicted: 0x35 as StreamCode, /** A frame declared a payload larger than the receiver accepts. */ FrameTooLarge: 0x38 as StreamCode, - /** The publisher could serve this request but has no capacity for it now. */ - NoCapacity: 0x30 as StreamCode, + /** The broadcast is neither announced nor served, so there is no route to it. */ + Unroutable: 0x36 as StreamCode, /** A group grew past its cache budget and was aborted. */ GroupTooLarge: 0x32 as StreamCode, } as const), diff --git a/js/net/src/ietf/publisher.ts b/js/net/src/ietf/publisher.ts index 54a7c62378a..b73ff09b303 100644 --- a/js/net/src/ietf/publisher.ts +++ b/js/net/src/ietf/publisher.ts @@ -319,7 +319,9 @@ export class Publisher { } catch (err: unknown) { const e = error(err); const condition = - e instanceof StreamError && e.code === StreamCode.NotFound ? "does_not_exist" : "internal"; + e instanceof StreamError && (e.code === StreamCode.NotFound || e.code === StreamCode.Unroutable) + ? "does_not_exist" + : "internal"; refusal = { errorCode: toRequestCode(condition, "subscribe", version), reasonPhrase: reason(e) }; } } diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index db88b3b90db..e809fe0e98c 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -225,3 +225,23 @@ export const hooks: { throw new Error("broadcast.ts not loaded"); }, }; + +/** + * Spreads equal routes across paths: FNV-1a 64 of `path` then each hop, oldest first, as 8 + * little-endian bytes. Keyed on the requested path so an equal-cost pool advertising one + * prefix shares its paths, and every node holding the same routes picks the same member. + * Mirrors `fnv_key` in `rs/moq-net`; the seed is the draft's Spread Hash offset basis. + */ +export function spreadHash(path: string, hops: readonly bigint[]): bigint { + const prime = 0x100000001b3n; + let hash = 0x420c0decb00bn; + for (const byte of new TextEncoder().encode(path)) { + hash = BigInt.asUintN(64, (hash ^ BigInt(byte)) * prime); + } + for (const hop of hops) { + for (let shift = 0n; shift < 64n; shift += 8n) { + hash = BigInt.asUintN(64, (hash ^ ((hop >> shift) & 0xffn)) * prime); + } + } + return hash; +} diff --git a/js/net/src/origin.test.ts b/js/net/src/origin.test.ts index 1220225b693..f42839d6c29 100644 --- a/js/net/src/origin.test.ts +++ b/js/net/src/origin.test.ts @@ -3,6 +3,7 @@ import { getter, Once, race, Signal } from "@moq/signals"; import { type Consumer as BroadcastConsumer, Producer as BroadcastProducer } from "./broadcast.ts"; import { StreamCode, StreamError } from "./error.ts"; import { HopSchema, Route, stampHops } from "./hop.ts"; +import { spreadHash } from "./internal.ts"; import type { Consumer, Table } from "./origin.ts"; import { Producer } from "./origin.ts"; import * as Path from "./path.ts"; @@ -1331,7 +1332,7 @@ test("reject surfaces the error from demand", async () => { const it = handle.requested(); const pending = wireOf(consumer).demand(Path.from("live/cam")); const { value: req } = await it.next(); - const err = new StreamError(StreamCode.NoCapacity, { message: "full" }); + const err = new StreamError(StreamCode.NotFound, { message: "not here" }); req?.reject(err); await expect(pending).rejects.toBe(err); @@ -1427,7 +1428,7 @@ test("advancing requested without settling rejects the previous request", async const second = await it.next(); expect(second.value?.path).toBe(Path.from("live/other")); - await expect(firstDemand).rejects.toMatchObject({ code: StreamCode.NoCapacity }); + await expect(firstDemand).rejects.toMatchObject({ code: StreamCode.Unroutable }); const produced = new BroadcastProducer(); second.value?.accept(produced); await expect(secondDemand).resolves.toBeDefined(); @@ -1631,7 +1632,7 @@ test("a refusal from a route a local broadcast superseded does not end the reque origin.close(); }); -test("close rejects queued requests with NoCapacity", async () => { +test("close rejects queued requests as unroutable", async () => { const origin = new Producer(); const handle = origin.dynamic(Path.from("live")); const waiting = handle.requested().next(); @@ -1647,7 +1648,6 @@ test("close rejects queued requests with NoCapacity", async () => { req?.accept(new BroadcastProducer()); await settle(); expect(request.active.peek()).toBeUndefined(); - expect(Number(new StreamError(StreamCode.NoCapacity).code)).toBe(0x30); request.close(); origin.close(); @@ -1668,6 +1668,62 @@ test("announced filters by arbitrary patterns and reports captures", async () => origin.close(); }); +test("the spread hash matches rs/moq-net byte for byte", () => { + expect(spreadHash("pool/job-0", [10n])).toBe(0xefb5e20a66101c32n); + expect(spreadHash("pool/job-0", [11n])).toBe(0x0eb0a91370ff6653n); +}); + +test("an equal-cost pool spreads its paths the same way on every node", async () => { + const workers = [10n, 11n, 12n, 13n].map((id) => HopSchema.parse(id)); + const paths = Array.from({ length: 64 }, (_, i) => Path.from(`pool/job-${i}`)); + + // The worker each path resolves to on an origin whose pool arrived in `order`. + async function winners(order: typeof workers): Promise { + const origin = new Producer(); + const served = new Map(); + const handles = order.map((hop) => { + const handle = wireOf(origin).receive(Path.from("pool"), { hops: [hop], cost: 3n }); + void (async () => { + for await (const request of handle.requested()) { + served.set(request.path, hop); + request.accept(new BroadcastProducer()); + } + })(); + return handle; + }); + const requests = paths.map((path) => origin.request(path)); + await settle(); + for (const request of requests) request.close(); + for (const handle of handles) handle.close(); + origin.close(); + return paths.map((path) => served.get(path) ?? -1n); + } + + const forward = await winners(workers); + const reverse = await winners([...workers].reverse()); + expect(reverse).toEqual(forward); + + // Not an assertion about any two paths, which a correct hash may put on one worker: + // only that the set does not pile onto a few. + for (const worker of workers) { + expect(forward.filter((hop) => hop === worker).length).toBeGreaterThanOrEqual(paths.length / 16); + } +}); + +test("an equal-cost pool advertises the same member whatever order it arrived in", () => { + const prefix = Path.from("pool"); + const workers = [10n, 11n].map((id) => HopSchema.parse(id)); + const expected = spreadHash(prefix, [workers[0]]) < spreadHash(prefix, [workers[1]]) ? workers[0] : workers[1]; + + for (const order of [workers, [...workers].reverse()]) { + const origin = new Producer(); + const handles = order.map((hop) => wireOf(origin).receive(prefix, { hops: [hop], cost: 3n })); + expect(origin.consume().broadcasts().peek().get(prefix)?.hops).toEqual([expected]); + for (const handle of handles) handle.close(); + origin.close(); + } +}); + test("a serving session closing releases quiet origin change listeners", async () => { const origin = new Producer(); const changed = Signal.prototype.changed; diff --git a/js/net/src/origin.ts b/js/net/src/origin.ts index 380a7afd97d..c23847301a1 100644 --- a/js/net/src/origin.ts +++ b/js/net/src/origin.ts @@ -15,7 +15,7 @@ import * as announce from "./announced.ts"; import * as broadcast from "./broadcast.ts"; import { StreamCode, StreamError } from "./error.ts"; import { isAnonymous, Route, routesEqual } from "./hop.ts"; -import { hiddenBelow, hooks, scopeCaptures, scopeHead, scopeOverlaps } from "./internal.ts"; +import { hiddenBelow, hooks, scopeCaptures, scopeHead, scopeOverlaps, spreadHash } from "./internal.ts"; import * as Path from "./path.ts"; import { type Advertised, type Advertisements, registerWire, wireOf } from "./wire.ts"; @@ -206,13 +206,19 @@ interface Candidate extends Advertised { readonly exact: boolean; } -/** Orders advertisements at one prefix: the better route, then a local broadcast on a tie, then fewer hops. */ -function compareCandidates(a: Candidate, b: Candidate): number { - return ( +/** + * Orders advertisements at `prefix`: the better route, then a local broadcast on a tie, then + * fewer hops, then the lower {@link spreadHash} of the prefix, matching `route_order` in rs/moq-net. + */ +function compareCandidates(prefix: Path.Valid, a: Candidate, b: Candidate): number { + const order = compareRoutes(a.route, b.route) || Number(b.exact) - Number(a.exact) || - a.route.hops.length - b.route.hops.length - ); + a.route.hops.length - b.route.hops.length; + if (order !== 0) return order; + const ha = spreadHash(prefix, a.route.hops); + const hb = spreadHash(prefix, b.route.hops); + return ha < hb ? -1 : ha > hb ? 1 : 0; } /** Orders two routes by preference: identified before anonymous, then lower warm cost, then lower cold cost. */ @@ -224,8 +230,16 @@ function compareRoutes(a: Route, b: Route): number { return 0; } -/** The preferred of `entries` (newest first) not skipped: the best route, then fewest hops, then newest. */ -function preferredEntry(entries: readonly RouteEntry[], skip?: (entry: RouteEntry) => boolean): RouteEntry | undefined { +/** + * The preferred of `entries` (newest first) for resolving `path`, not skipped: the best route, + * then fewest hops, then the lowest {@link spreadHash}, then newest. `path` is the requested + * path for a request, or the prefix itself for an advertisement. + */ +function preferredEntry( + path: Path.Valid, + entries: readonly RouteEntry[], + skip?: (entry: RouteEntry) => boolean, +): RouteEntry | undefined { let best: RouteEntry | undefined; for (const entry of entries) { if (skip?.(entry)) continue; @@ -235,7 +249,13 @@ function preferredEntry(entries: readonly RouteEntry[], skip?: (entry: RouteEntr } const a = entry.route.peek(); const b = best.route.peek(); - const order = compareRoutes(a, b) || a.hops.length - b.hops.length; + let order = compareRoutes(a, b) || a.hops.length - b.hops.length; + // Hashed only on a tie, so the common single-route prefix never pays for it. + if (order === 0) { + const ha = spreadHash(path, a.hops); + const hb = spreadHash(path, b.hops); + order = ha < hb ? -1 : ha > hb ? 1 : 0; + } if (order < 0) best = entry; } return best; @@ -246,8 +266,8 @@ function received(entry: RouteEntry): boolean { return !entry.originated; } -function noCapacity(): StreamError { - return new StreamError(StreamCode.NoCapacity, { message: "no capacity" }); +function unroutable(): StreamError { + return new StreamError(StreamCode.Unroutable, { message: "unroutable" }); } /** A served route from {@link Producer.dynamic}: the queue a handler drains. */ @@ -319,7 +339,7 @@ class ServeState { close(abort?: Error): void { if (this.closed.peek() !== undefined) return; - const err = abort ?? noCapacity(); + const err = abort ?? unroutable(); this.closed.set(err); const queued = [...this.pending.values()]; this.pending.clear(); @@ -425,9 +445,9 @@ class OriginState { if (list) list.push(local); else out.set(path, [local]); } - for (const list of out.values()) { + for (const [prefix, list] of out) { // Stable, so equal routes keep the table's newest-first order. - if (list.length > 1) list.sort(compareCandidates); + if (list.length > 1) list.sort((a, b) => compareCandidates(prefix, a, b)); } return out; } @@ -553,6 +573,7 @@ class OriginState { for (const [prefix, entries] of this.routes.peek() ?? []) { if (!Path.hasPrefix(prefix, path)) continue; const entry = preferredEntry( + path, entries, (candidate) => !candidate.scope.matches(path) || (skip?.(candidate) ?? false), ); @@ -1577,7 +1598,7 @@ export class Consumer { * requests beneath it. * * Drop it (or {@link close}) to retract the route and reject anything still waiting - * with {@link StreamCode.NoCapacity}. {@link update} re-prices it in place. + * with {@link StreamCode.Unroutable}. {@link update} re-prices it in place. * * @public */ @@ -1623,7 +1644,7 @@ export class Dynamic { if (!server) return; let current: Request | undefined; const drop = () => { - current?.reject(noCapacity()); + current?.reject(unroutable()); current = undefined; }; try { diff --git a/js/watch/src/broadcast.test.ts b/js/watch/src/broadcast.test.ts index 65bd62031b7..6762fc8c28f 100644 --- a/js/watch/src/broadcast.test.ts +++ b/js/watch/src/broadcast.test.ts @@ -374,6 +374,109 @@ describe("cross-broadcast renditions", () => { }); }); +// A derived rendition produced only on demand by a service that claims a covering prefix (a +// wildcard) instead of announcing each path. Nothing announces the rendition until something +// subscribes to it, so the player must list it from the claim alone. The rendition is a sibling: +// one beneath its source is already covered by the source's own announcement. The service prefix +// is hidden, as a deployment keeps it out of listings, so the claim is only seen by opting in. +describe("wildcard renditions", () => { + const name = Path.from("live/foo.hang"); + const derived = Path.from(".pro/transcode/foo.hang"); + const rel = Path.normalizeRelative("../.pro/transcode/foo.hang"); + + const watch = (owner: Origin.Producer) => + new Broadcast({ + origin: owner, + name, + enabled: true, + catalogFormat: "manual", + catalog: { + video: { + renditions: { + source: video("avc1.64001e"), + transcode: video("avc1.640028", rel), + }, + }, + } as Catalog.Root, + }); + + it("lists and demands a rendition only a wildcard covers", async () => { + const owner = new Origin.Producer(); + const main = publish(owner, name); + const source = watch(owner); + const effect = new Effect(); + + try { + await settle(); + expect(videoRenditions(source)).toEqual(["source"]); + + const worker = owner.dynamic(Path.from(".pro/transcode")); + const requests = worker.requested(); + await settle(); + expect(videoRenditions(source)).toEqual(["source", "transcode"]); + + // Selecting the rendition is what asks the worker to start, before anything announces it. + let active: Moq.Broadcast.Consumer | undefined; + effect.run((nested) => { + active = source.relativeBroadcast(nested, rel); + }); + const { value: request } = await requests.next(); + expect(request?.path).toBe(derived); + expect(active).toBeUndefined(); + + const produced = new Moq.Broadcast.Producer(); + produced.createTrack("video").writeString("frame"); + request?.accept(produced); + await settle(); + expect(active).toBeDefined(); + const track = active?.track("video").subscribe().ordered(); + expect(await track?.readString()).toBe("frame"); + track?.close(); + + // The worker announcing the path it now serves leaves the rendition where it was. + const concrete = owner.createBroadcast(derived); + concrete.announce(); + await settle(); + expect(videoRenditions(source)).toEqual(["source", "transcode"]); + expect(active).toBeDefined(); + + concrete.close(); + worker.close(); + } finally { + effect.close(); + source.close(); + main.close(); + owner.close(); + } + }); + + it("hides the rendition once the last covering wildcard is withdrawn", async () => { + const owner = new Origin.Producer(); + const main = publish(owner, name); + const source = watch(owner); + + try { + // A catch-all (an archive) and a narrower pool both cover the derived path. + const archive = owner.dynamic(Path.empty()); + const worker = owner.dynamic(Path.from(".pro/transcode")); + await settle(); + expect(videoRenditions(source)).toEqual(["source", "transcode"]); + + worker.close(); + await settle(); + expect(videoRenditions(source)).toEqual(["source", "transcode"]); + + archive.close(); + await settle(); + expect(videoRenditions(source)).toEqual(["source"]); + } finally { + source.close(); + main.close(); + owner.close(); + } + }); +}); + describe("manual catalog", () => { it("republishes a manual catalog mutated in place", async () => { const owner = new Origin.Producer(); diff --git a/js/watch/src/broadcast.ts b/js/watch/src/broadcast.ts index 15418a974b8..ba23a767cf0 100644 --- a/js/watch/src/broadcast.ts +++ b/js/watch/src/broadcast.ts @@ -158,7 +158,9 @@ export class Broadcast { const origin = effect.get(this.in.origin); if (!origin) return; - const announced = origin.announced(); + // Hidden routes count: a service claim under a `.`-named prefix is kept out of listings, but + // it still covers the renditions it would produce, and this set is never shown to anyone. + const announced = origin.announced(Path.Pattern.all(), { hidden: true }); effect.cleanup(() => announced.close()); this.#announced.set(new Set()); @@ -188,7 +190,9 @@ export class Broadcast { // Whether `path` is covered by an announced route, for `relativeBroadcast`'s // cross-broadcast refs. Announcements are prefix routes, so a route at "room/" covers - // "room/alice/cam.hang" without naming it. Opens the announcement stream on first use. + // "room/alice/cam.hang" without naming it. That is how a rendition produced only on demand + // gets selected: its service claims a covering prefix, and nothing announces the exact path + // until this subscribes. Opens the announcement stream on first use. // The blind cases (announcement gate off, no discovery) never reach here; see `#relativeTarget`. #isPathAnnounced(effect: Effect, path: Moq.Path.Valid): boolean { this.#wantAnnounced.set(true); diff --git a/quest/m0/wildcard/README.md b/quest/m0/wildcard/README.md index 43289c97e47..58ad4e8c99e 100644 --- a/quest/m0/wildcard/README.md +++ b/quest/m0/wildcard/README.md @@ -1,4 +1,4 @@ -# [L] Wildcard advertisements +# [S] Wildcard advertisements ## Goal @@ -39,10 +39,6 @@ prefix claims resolved against pattern interest. The three workloads above still hold: the transcoder claims its service prefix ([Where derived output lives](#where-derived-output-lives)), and the archive claims the root and refuses what it does not have. -Resolve and Demand are additive and land on main. Both are done on the line -branch (9d059b1b9 lists and demands covered renditions in the browser -player), so main no longer lists them; the line branch moves to -`quest/m0/wildcard/README` to match this path. ### What already exists, and what does not @@ -61,17 +57,15 @@ this, and #3770 settled the wire: an announcement carries a path prefix on every protocol, and a consumer filters announced paths against its pattern interest locally. -The routing table exists too. `Consumer::request_broadcast` resolves a local -broadcast first, then `best_server`: the longest covering prefix, filtered by -the requester's excluded hop, ordered by `route_order` -(`rs/moq-net/src/model/origin.rs:633`), served on demand by the session that -announced it and cached per prefix in `ServeState.served` (`:764`). That is the split-horizon-safe -lookup the old `origin::Dynamic` could not provide, and it is what -resolve extends rather than replaces. - -Request resolution is prefix-only (`best_server` in -`rs/moq-net/src/model/origin.rs`) and stays that way. The pattern matcher -itself exists: `moq_net::{Pattern, Patterns, Segment}` and `Path.Pattern` / +Request resolution exists too. `Consumer::request_broadcast` mints a front per +path (`rs/moq-net/src/model/front.rs`) that selects through `best_route`: a +local broadcast first, then the longest covering prefix, filtered by the +requester's excluded hop and ordered by `route_order`, whose hash is keyed on +the requested path so one prefix's pool shares its paths. A front follows the +best route and resumes through any covering one, and FETCH resolves the same +way. A standing refusal ends the front (#4875), so it never reaches a sibling +advertiser or a shorter prefix. The pattern matcher itself exists: +`moq_net::{Pattern, Patterns, Segment}` and `Path.Pattern` / `Path.Patterns` in `js/net/src/path.ts` own the shared matching, containment, specificity, and rebasing tokens and filters reuse. @@ -88,16 +82,15 @@ path, not with a route identity or a generation field. [Auth](/quest/m1/auth/README.md) line is what tokens and the consume-side filter use, matched by the shared matcher, so nothing resembles a second grammar and nothing on the wire spells a wildcard. -- **Longest prefix wins, and its refusal is final.** This is the rule - routing already follows: `best_server` filters to the longest covering prefix - before it compares cost, and the lite draft says the same, matching - longest-prefix-match wherever it appears. Advertisers of one prefix form one - pool that cost and the request hash order. A terminal refusal from the - winning tier IS the answer and never falls through to a shorter prefix, so a - transcoder refusing a path does not leak the request to the archive's - catch-all, and one unserved path still costs one round trip. The accepted - consequence: an offline derivative is not reachable through the catch-all - while a longer prefix covers it. +- **Longest prefix wins, and its refusal is final.** `best_route` filters to + the longest covering prefix before it compares cost, and the lite and + cluster drafts say the same. Advertisers of one prefix form one pool that + cost and the request hash order. A refusal from the winning tier IS the + answer and never falls through to a shorter prefix or to another + advertiser, so a transcoder refusing a path does not leak the request to + the archive's catch-all, and one unserved path costs one round trip. The + accepted consequence: an offline derivative is not reachable through the + catch-all while a longer prefix covers it. - **A claim is a POOL, not a competitor.** Several advertisers of one prefix is the normal state, not a hazard: every transcode worker claims the same prefix and takes a share. What distributes them is a @@ -119,17 +112,17 @@ path, not with a route identity or a generation field. The seed still has a floor, because standby and running claims of the same prefix do meet: a standby concrete claim (`with_cost(1000)` is the existing per-broadcast convention) shares a tier with a running publisher's - concrete announcement. The + concrete announcement and with warm-advertise's exact-path warm routes. The floor MUST exceed the deployment's enforced maximum charged-link count times its enforced maximum link cost (32 links at cost at most 5 gives a bound of 160, with producing origins seeded at 0), or a nearby standby outranks a distant running copy and the mesh starts a second encode of a stream it is already serving. That floor replaces the ad-hoc standby bias the moq.pro (downstream) transcode worker carries today. -- **A wildcard is a capability, not an inventory.** It advertises what the +- **A claim is a capability, not an inventory.** It advertises what the sender could serve, never that a given path exists. Refusal is how a specific path is denied. This is why an over-claiming advertisement is not a defect: - the catch-all `**` is legal, and answering "not that one" is the mechanism. + claiming the root is legal, and answering "not that one" is the mechanism. - **Overlap with the publish scope is what authorization checks.** An advertised prefix MUST overlap the sender's granted patterns or it is refused. A prefix wider than the grant is accepted, but it only routes @@ -138,16 +131,12 @@ path, not with a route identity or a generation field. Until [Advertise-only authorization](/quest/m3/processor/advertise-auth.md) lands, the publish scope stands in for advertising; a credential with its own advertise scope is checked against that instead. -- **Claims are visible to subscribers.** A subscriber sees every advertised - prefix under its scope, filtered locally like any other announcement. That - is the point: it tells a client it may subscribe to - matching paths, and its withdrawal tells the client the capability is gone. - This is what makes a lazily-produced rendition discoverable without the - composer waiting for an announcement that only demand would produce. The - browser player currently enforces the opposite (`js/watch`'s - `#isPathAnnounced` hides a catalog rendition with no exact-path - announcement); demand makes a covering wildcard count as - availability there. +- **Claims are visible to subscribers.** A claim is an ordinary prefix + announcement, so it tells a client it may subscribe beneath it, and its + withdrawal tells the client the capability is gone. The browser player's + gate (`js/watch`'s `#isPathAnnounced`) lists a catalog rendition under any + covering prefix, so a lazily-produced rendition is discoverable without an + announcement that only demand would produce. - **Every refusal is a terminal typed stream reset, with no negative cache.** An advertiser resets a subscribe it will not serve, and the reset carries which KIND of refusal it is (`Error::to_code` already puts a typed code on @@ -179,9 +168,7 @@ path, not with a route identity or a generation field. and any covering route resumes it (#4741). - **No reply Origin.** The lite-07 `Origin` field in SUBSCRIBE_OK and FETCH_OK, and the rule that a relay MUST NOT splice across differing Origins, are - dropped (decided 2026-10-03: nothing needs them for correctness). The line - branch carries code for Origin, `Identity`, and `Pin`, and #4050's - NO_CAPACITY re-resolution; remove it when the branch next merges main. + dropped (decided 2026-10-03: nothing needs them for correctness). - **Patterns are independent of clustering.** The `moq-pattern` crate owns the matching semantics tokens and filters share, with no draft of its own; no announce message carries a pattern on either protocol (AUTH grants on @@ -201,8 +188,8 @@ The leading `.` is deliberate. Existing customers on moq-lite-06 or older must never see `.pro/` broadcasts, which could confuse their business logic. Those versions cannot opt into hidden routes, so the relay never announces them there. Hidden routes are a moq-lite-07 feature, so the player's covering check -(Demand, done on the line branch) opts into them and sees a claim only when -lite-07 is negotiated. Finalizing lite-07 is a rollout condition, not a +opts into them and sees a claim only when lite-07 is negotiated; the token's +scope must still reach them. Finalizing lite-07 is a rollout condition, not a blocker for this line (decided in the 2026-09-30 audit), since the check works whenever lite-07 is negotiated. A customer who wants transcodes upgrades, or subscribes to the explicit `.pro//...` path, which works on any diff --git a/rs/moq-net/benches/origin.rs b/rs/moq-net/benches/origin.rs index a6cceec63b6..3572263806e 100644 --- a/rs/moq-net/benches/origin.rs +++ b/rs/moq-net/benches/origin.rs @@ -9,8 +9,12 @@ //! down to its prefix and beneath it. Competing routes at that prefix are //! ranked once per change, then each cursor selects its first visible entry. //! +//! An equal-cost pool is swept the same way, over its members and the paths it +//! already serves. +//! //! Run with `cargo bench -p moq-net --bench origin`. +use std::task::Poll; use std::time::Duration; use criterion::{BenchmarkId, Criterion, Throughput, criterion_group, criterion_main}; @@ -421,6 +425,96 @@ fn bench_request(c: &mut Criterion) { group.finish(); } +/// `(members, paths)` shapes for an equal-cost pool: how many advertisers claim +/// one prefix, against how many paths beneath it are already being served. +const POOL: [usize; 3] = [4, 32, 256]; +const POOL_PATHS: [usize; 3] = [100, 1_000, 10_000]; + +/// Hop ids for pool members, clear of the origin's own. +const POOL_HOP: u64 = 1_000; + +/// An origin where `members` equal-cost advertisers claim `pool`, already +/// serving `paths` requested paths beneath it, spread across the pool by the +/// hash on each path. Every handle is held so the fronts stay up. +struct Pool { + producer: origin::Producer, + driver: origin::Driver, + _members: Vec, + _producers: Vec, + _consumers: Vec, +} + +fn pool(members: usize, paths: usize) -> Pool { + let (producer, mut driver) = origin::Producer::new(origin::Config::new(Hop::new(1).unwrap())); + let consumer = producer.consume(); + let members: Vec<_> = (0..members) + .map(|i| producer.dynamic("pool", pool_route(POOL_HOP + i as u64)).unwrap()) + .collect(); + let waiter = kio::Waiter::noop(); + + let requests: Vec<_> = (0..paths) + .map(|i| consumer.request_broadcast(format!("pool/job-{i}"))) + .collect(); + driver.poll(moq_net::time::Instant::now(), &waiter).unwrap(); + let mut producers = Vec::with_capacity(paths); + for member in &members { + while let Poll::Ready(request) = member.poll_requested_broadcast(&waiter) { + let broadcast = broadcast::Info::new().produce(); + request.unwrap().accept(&broadcast); + producers.push(broadcast); + } + } + assert_eq!(producers.len(), paths, "every path reached a member"); + driver.poll(moq_net::time::Instant::now(), &waiter).unwrap(); + let consumers = requests + .into_iter() + .map(|request| request.now_or_never().expect("resolved once driven").expect("served")) + .collect(); + + Pool { + producer, + driver, + _members: members, + _producers: producers, + _consumers: consumers, + } +} + +/// A pool member's claim: one hop, at the same cost as every other member. +fn pool_route(hop: u64) -> origin::Route { + peer_route(hop, 3) +} + +/// A member joining and leaving a pool that already serves `paths` paths. +/// +/// Every front under the prefix watches the routes covering it, so each one +/// re-selects on both changes, and each selection scans the pool: the cost is +/// paths times members by design, which this sweep makes visible. The fronts the +/// newcomer outranks request through it, and fall back when it leaves before +/// answering. +fn bench_pool_churn(c: &mut Criterion) { + let mut group = c.benchmark_group("origin/pool_churn"); + // The widest shapes take a noticeable fraction of a second per iteration. + group.sample_size(10); + for members in POOL { + for paths in POOL_PATHS { + let id = BenchmarkId::from_parameter(format!("{members}m_{paths}p")); + group.bench_function(id, |b| { + let mut pool = pool(members, paths); + let waiter = kio::Waiter::noop(); + let hop = POOL_HOP + members as u64; + b.iter(|| { + let joined = pool.producer.dynamic("pool", pool_route(hop)).unwrap(); + pool.driver.poll(moq_net::time::Instant::now(), &waiter).unwrap(); + drop(joined); + pool.driver.poll(moq_net::time::Instant::now(), &waiter).unwrap(); + }); + }); + } + } + group.finish(); +} + /// Publisher handoff at one path: a subscriber is reading from one local /// source when a second announces at the same path and takes over (newest /// wins). Measured from the standby's attach to the subscriber receiving its @@ -601,6 +695,7 @@ criterion_group!( bench_serve_idle, bench_subscribe, bench_request, + bench_pool_churn, bench_handoff, bench_relay, bench_parked diff --git a/rs/moq-net/src/error.rs b/rs/moq-net/src/error.rs index eae8cb053b8..f569438999f 100644 --- a/rs/moq-net/src/error.rs +++ b/rs/moq-net/src/error.rs @@ -197,7 +197,6 @@ impl StreamError { Self::GoingAway => 0x4, Self::TooFarBehind => 0x5, Self::MalformedTrack => 0x12, - // 0x30 NO_CAPACITY is assigned by other work in this range. Do not reuse it. Self::ControlTimeout => 0x31, Self::GroupTooLarge => 0x32, Self::NotFound => 0x33, @@ -682,8 +681,7 @@ mod tests { assert_eq!(StreamError::from_code(err.to_code()), err, "{err:?} did not round trip"); } - // moq-lite's own 48-63 range, pinned to the draft's table. They stay off 0x30 - // (NO_CAPACITY), which other work assigns. + // moq-lite's own 48-63 range, pinned to the draft's table. for (err, code) in [ (StreamError::ControlTimeout, 0x31), (StreamError::GroupTooLarge, 0x32), diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index b8f3c848c33..e8029f0c351 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -617,20 +617,25 @@ fn fnv_key(name: &str, origins: impl IntoIterator) -> u64 { hash } -/// Ordering key for a route entry covering one prefix. Lower wins: an identified +/// Ordering key for a route entry resolving `path`. Lower wins: an identified /// chain (no 0) outranks an anonymous one regardless of cost, then the cheapest /// cost, then a broadcast published on this origin (it serves what is here, not a /// claim that has to ask), then the shortest hop chain, then a deterministic hash -/// of the prefix and chain so every node converges on the same winner, and finally +/// of `path` and the chain so every node converges on the same winner, and finally /// the newest announcement, so a reconnect under an otherwise identical route wins /// the moment it lands instead of after the transport retires the old session. -fn route_order(prefix: &Path, entry: &RouteEntry) -> (bool, Cost, bool, usize, u64, Reverse) { +/// +/// `path` is what is being resolved: the requested path for a request, the prefix +/// itself for an advertisement. Keying the hash on the requested path is what +/// spreads equal-cost advertisers of one prefix: each path picks its own winner +/// from the pool, rather than every path under the prefix hashing alike. +fn route_order(path: &Path, entry: &RouteEntry) -> (bool, Cost, bool, usize, u64, Reverse) { ( entry.is_anonymous(), entry.cost, !entry.local, entry.hops.len(), - fnv_key(prefix.as_str(), entry.hops.iter().copied()), + fnv_key(path.as_str(), entry.hops.iter().copied()), Reverse(entry.id), ) } @@ -3410,8 +3415,10 @@ impl OriginState { /// /// Only announced routes are candidates: an unannounced broadcast serves /// nobody, and does not shadow anything either. Routes `excluded` for the - /// front's path are skipped. A broadcast published on this origin competes - /// on cost like any other route and wins a tie. + /// front's path are skipped. The hash tie-break is keyed on `path`, so + /// equal-cost advertisers of one prefix share its paths. A broadcast + /// published on this origin competes on cost like any other route and wins a + /// tie. fn best_route(&self, path: &Path, horizon: Horizon, excluded: &HashSet) -> Option<&RouteEntry> { // Covering prefixes of one path form a chain, so the deepest node with a // candidate holds the unique longest prefix; walking down, the last such @@ -3430,7 +3437,7 @@ impl OriginState { if candidates.peek().is_some() { best = candidates .filter(|entry| entry.serves(path)) - .min_by_key(|entry| route_order(&entry.prefix, entry)); + .min_by_key(|entry| route_order(path, entry)); } } best @@ -5413,6 +5420,56 @@ mod tests { announced.assert_next_ended("room"); } + /// Pinned so `spreadHash` in `js/net` picks the same pool member for a path. + #[test] + fn spread_hash_matches_js() { + assert_eq!(fnv_key("pool/job-0", [origin(10)]), 0xefb5e20a66101c32); + assert_eq!(fnv_key("pool/job-0", [origin(11)]), 0x0eb0a91370ff6653); + } + + /// Equal-cost advertisers of one prefix share its paths: a set of requested + /// paths spreads across the pool, and one path always resolves to the same + /// advertiser, whatever order the routes arrived in. + #[tokio::test] + async fn equal_cost_pool_spreads_paths() { + const WORKERS: [u64; 4] = [10, 11, 12, 13]; + const PATHS: usize = 64; + + // The first hop of the route each path resolves to on a node whose pool + // arrived in `order`. + fn winners(order: impl Iterator) -> Vec { + let producer = origin(1).produce(); + let _pool: Vec = order + .map(|id| { + producer + .dynamic("pool", Route::default().with_hops(hops(&[id])).with_cost(3)) + .unwrap() + }) + .collect(); + let table = producer.shared.read(); + (0..PATHS) + .map(|i| { + let path = Path::new(&format!("pool/job-{i}")).to_owned(); + let entry = table + .best_route(&path.as_path(), Horizon::default(), &HashSet::new()) + .expect("the pool serves every path"); + entry.hops.iter().next().copied().unwrap() + }) + .collect() + } + + let forward = winners(WORKERS.into_iter()); + let reverse = winners(WORKERS.into_iter().rev()); + assert_eq!(forward, reverse, "a path must resolve the same way on every node"); + + // Not an assertion about any two paths, which a correct hash may put on + // one worker: only that the set does not pile onto a few. + for worker in WORKERS { + let share = forward.iter().filter(|hop| **hop == origin(worker)).count(); + assert!(share >= PATHS / 16, "worker {worker} took {share} of {PATHS} paths"); + } + } + #[tokio::test] async fn identical_reannounce_is_invisible() { let producer = origin(1).produce();