diff --git a/demo/web/src/stats.ts b/demo/web/src/stats.ts index bd80adc9b9f..954e5546297 100644 --- a/demo/web/src/stats.ts +++ b/demo/web/src/stats.ts @@ -5,7 +5,8 @@ * `.stats/node/` carrying JSON tracks that snapshot current activity. We * auto-discover all of those nodes (announcements under `.stats/node`), so this * works for a single relay and for a cluster alike, then aggregate each node and - * let you drill into one. + * let you drill into one. A restarted relay announces a new epoch at the same + * path, which arrives as an end then a start, so its history starts over. * * Per-node tracks we read: * publisher.json egress (relay -> downstream viewers) @@ -146,6 +147,10 @@ discovery.run((effect) => { nodeStats.mutate((s) => { delete s[node]; }); + // A restart ends the old epoch and starts a new one at the same path, + // counting from zero, so neither series may splice across it. + history.delete(node); + clusterMembership = ""; } } }); diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index a2dda75a779..ce280561a1e 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -238,7 +238,10 @@ Each node publishes `publisher.json`, `subscriber.json`, and `sessions.json` tracks (plus compressed `.json.z` twins) of cumulative counters per broadcast and auth root, split by a **tier** label chosen by the auth server's grant or `--cluster-tier`, which is what makes billing per customer or per region -possible. [Stats](/concept/stats) describes the paths, tracks, and encodings; +possible. Each run, and each group returning after its linger, announces under +a fresh [epoch](/concept/moq-lite#publisher-epochs) on its route, so a restart +is a new broadcast at the same path. +[Stats](/concept/stats) describes the paths, tracks, and encodings; read them with the [`moq-stats`](https://docs.rs/moq-stats) crate. ## \[iroh] diff --git a/doc/concept/stats.md b/doc/concept/stats.md index 8d23a0b01e9..d37d6844b51 100644 --- a/doc/concept/stats.md +++ b/doc/concept/stats.md @@ -33,16 +33,27 @@ generate more stats. prefix, so a consumer skips any path without `node` where it expects one. A group segment literally named `node` is ambiguous; don't use one. +Each announcement carries a fresh [publisher epoch](/concept/moq-lite#publisher-epochs) +on its route, so a restarted node or a group returning from idle is a new +broadcast at the same path: it replaces the old one, and no relay serves it +groups cached under the old epoch. A reader treats each epoch as its own set of +counters. The epoch rides moq-lite 07 announcements; over older versions and +moq-transport a reader sees the same change as an end and a start at the path. +An aggregating reader pins each node's subscription to its epoch, and when one +publisher is reached through both kinds of link (with and without an epoch), it +carries only that path's outgoing counters across the change. + At depth 0 the broadcast stays announced for the producer's life. At depth 1 or more, a group's broadcast is announced while that group has entries, and for a linger (five minutes by default) after its last one leaves. A group that returns within the linger keeps its broadcast, so viewer churn doesn't unannounce and re-announce it across the mesh; while it lingers empty, its tracks hold `{}`. Once the linger elapses with the group still empty, the -broadcast is unannounced. Group numbers keep increasing across recreated -tracks and group broadcasts for the producer's life; they may have gaps. A -recreated compressed track starts a new group with a full snapshot, never a -delta whose compression state belonged to its previous writer. +broadcast is unannounced and its counters dropped; a group that returns later +announces a new epoch counted from zero. Within one epoch, group numbers keep +increasing across recreated tracks; they may have gaps. A recreated compressed +track starts a new group with a full snapshot, never a delta whose compression +state belonged to its previous writer. ## Tracks @@ -132,9 +143,12 @@ Every counter is a cumulative, monotonic unsigned integer. A rate is the difference between two frames divided by the time between them, and a live count is started minus ended. A frame never shows ended above started. -A counter going **down** means the relay restarted or the entry was dropped -and re-created. Treat it as the start of a fresh segment rather than a -negative rate. +A new epoch starts every counter from zero, so a reader summing a node over +time adds each epoch's counters rather than diffing across them. Within one +epoch, a counter going **down** means the entry was dropped and re-created. +Treat it as the start of a fresh segment rather than a negative rate. On a +route without an epoch (an older relay, or a session older than moq-lite 07), +a counter going down also means the node restarted. A reader ignores unknown fields, so a newer relay can add counters, and defaults a missing field to zero, so it can read an older relay. diff --git a/doc/lib/js/net.md b/doc/lib/js/net.md index 1a67f3ba283..7adff50256f 100644 --- a/doc/lib/js/net.md +++ b/doc/lib/js/net.md @@ -49,6 +49,7 @@ for (;;) { ``` - **Origins** hold the broadcasts, not the connection: closing a session unannounces them but leaves them created for the next one. `origin.request(path)` resolves an announced local broadcast with no round trip, so a page that watches what it publishes reads its own copy, unless a cheaper route announces the same path. A refusal from the most specific route ends the request: `request.closed` settles with the handler's error and no broader route is asked. Create, populate, then `announce()` for an exact path; use `dynamic(prefix, route)` when the set of paths is not known: an exact-path subscribe before the tracks exist is refused, and nobody, local or remote, can see or reach a broadcast until it announces. +- **Epoch pinning**: `consumer.request(path, { epoch })` pins an announced publisher instance. The request stays unresolved and reports `unroutable` when the winning route names a different or missing epoch, including while an asynchronous handler answers. Omit `epoch` for unpinned lookup. The resolved broadcast's `epoch` names the route it actually resolved through, pinned or not. - **Connections** race WebTransport against WebSocket. `new Connection({ url })` pools one connection per relay URL and reconnects with backoff, which the elements use. Supplying WebTransport/WebSocket options, discovery, delay, `goaway`, or a caller-owned origin selects a private loop; explicit `share: true` refuses those options. On a relay's GOAWAY the connection dials the replacement at once while the old session keeps serving its groups until it closes or the `goaway.handover` cap (default 10s, lowered to the relay's deadline) passes; the handle and its origin stay the same. An empty GOAWAY redials the same URL through a fresh DNS resolve. A redirect is followed onto the same host by default (`goaway.redirect: "follow"` lets the relay pick the host, `"ignore"` never moves), and moves the pooled entry to the new URL; a malformed or refused one, including a host change under a pinned certificate, ends the connection with `Error.RefusedRedirect` instead of reconnecting. `closed` settles when the handle is released (`null` on a clean close); the failure that stopped retrying the current URL is `error`, and a new URL recovers the same handle. A connection owns one send-rate sampler and one `Bandwidth.Allocator`; publishers reserve against it so their encoder targets sum to the estimate instead of each matching it. - **Bandwidth** (`Bandwidth.Allocator`) divides the connection's send-rate estimate by track priority, max-min fair within a tier. An idle track claims nothing. The receive side is untouched. - **Discovery** by any pattern scope (`origin.announced(scope)`, such as `room/*/chat`; default everything). A route event's `kind` says whether the route started, updated, or ended; its `prefix` is the covered prefix relative to the origin, and `captures` reports what the scope's wildcards matched when the prefix pins them. The consumer is an async iterable. `origin.broadcasts(scope)` is a live `Getter>` of the same covered prefixes for UIs that need the current set. A borrowed `Connection.origin` also exposes `dynamic(prefix, route)` for serving paths on demand. diff --git a/doc/lib/rs/index.md b/doc/lib/rs/index.md index eb232992793..8401bb330b1 100644 --- a/doc/lib/rs/index.md +++ b/doc/lib/rs/index.md @@ -56,7 +56,7 @@ let mut announced = consumer.announced(); while let Some(event) = announced.next().await { // Skip updates and retractions. let moq_net::announce::Event::Start(update) = event else { continue }; - let broadcast = consumer.request_broadcast(&update.prefix).await?; + let broadcast = consumer.request_broadcast(&update.prefix, update.route.epoch).await?; let catalog = broadcast .track(hang::Catalog::DEFAULT_NAME)? .subscribe(hang::Catalog::default_subscription()) diff --git a/doc/lib/rs/moq-net.md b/doc/lib/rs/moq-net.md index 722f5fd4154..30bd479c332 100644 --- a/doc/lib/rs/moq-net.md +++ b/doc/lib/rs/moq-net.md @@ -17,7 +17,8 @@ above ([hang](/lib/rs/hang)); relays and CDNs implement only this. - **Origins** scope what a session can see, and merge duplicate subscriptions so a broadcast is pulled upstream once no matter how many local readers. - **Broadcasts** are created unannounced and invisible to everyone, then announced as an exact route, or served below a prefix with `dynamic`. A consumer of the same origin sees exactly what a peer sees. Discovery accepts pattern unions; events carry the advertised prefix and captures for a complete match. -- **Epochs** identify publisher instances with a canonical UUIDv7 carried on each route (`Route::epoch`): the newest wins a path, and only routes with the same epoch resume a subscription. A resolved `broadcast::Consumer` names the epoch it came through in `info().epoch`; see [publisher epochs](/concept/moq-lite#publisher-epochs). +- **Resolution** uses `consumer.request_broadcast(path, epoch)` with an owned `Epoch` to pin a publisher instance, or `None` for unpinned lookup. A pinned request refuses a different or missing epoch at request time and asynchronous resolution. +- **Epochs** identify publisher instances with a canonical UUIDv7 carried on each route (`Route::epoch`): the newest wins a path, and only routes with the same epoch resume a subscription. A resolved `broadcast::Consumer` names the epoch it came through in `info().epoch`, pinned or not; see [publisher epochs](/concept/moq-lite#publisher-epochs). - **Patterns** (`Pattern`, `Patterns`) are re-exported from [`moq-pattern`](https://docs.rs/moq-pattern). Literal `Path` stays a coordinate. - **Tracks** carry groups with a priority, an optional publisher retention window (`Info::max_age`), and a timescale, or none for an untimed track whose frames carry no timestamp. Subscribers set their own priority and max delay and can change them live. - **Groups** are written frame by frame and delivered on independent streams. Old groups are cached for fetch-by-sequence; stale groups are skipped per the subscriber's budget. diff --git a/doc/setup/upgrade.md b/doc/setup/upgrade.md index 7900428be91..a8fa63255f3 100644 --- a/doc/setup/upgrade.md +++ b/doc/setup/upgrade.md @@ -47,6 +47,13 @@ These land with the next breaking release, not the 2026-09-23 train. `producer.demand().used` and `.unused()`, as are `Group.Producer`'s, and `Allocator.reserve` takes `producer.demand()`, replacing the `Bandwidth.Demand` interface. +- **`request_broadcast` takes an epoch.** In Rust, + `origin::Consumer::request_broadcast(path)` is + `request_broadcast(path, None)`; pass an `Epoch` instead to refuse any other + publisher instance. In TypeScript, `RequestOptions.epoch` does the same. A + dynamic route updated to another epoch now refuses the requests its handler + still holds with `Unroutable`, so a handler answering them late sees its + answer dropped. - **`moq --hop` is removed; `--epoch` replaces it.** A redundant pair shares an epoch, a UUIDv7 such as `uuidgen -7` prints, instead of a Hop ID: pass the same `--epoch` (or `MOQ_EPOCH`) to both. Unlike a rename, `--hop` is now an @@ -335,7 +342,7 @@ Other changes to a deployment: `origin::Config`. - **Announcements are prefix routes** (#3225, #3770). `announce::Update` is `{ prefix, route, kind, captures }`: skip `!update.kind.is_active()` and - resolve the broadcast with `consumer.request_broadcast(&update.prefix)`. + resolve the broadcast with `consumer.request_broadcast(&update.prefix, update.route.epoch)`. Serve a subtree on demand with `origin.dynamic(prefix, route)`. - **Tracks.** `with_latency_max` / `latency_max` is `with_max_age` / `max_age`. `write_datagram(Datagram)` is `insert_datagram(sequence, timestamp, payload)` diff --git a/js/net/bench/broadcasts.ts b/js/net/bench/broadcasts.ts index 80a37a3488d..b912e5dfe0b 100644 --- a/js/net/bench/broadcasts.ts +++ b/js/net/bench/broadcasts.ts @@ -1,6 +1,7 @@ /** Sweep observers against routes for one route update, and against tracks for one demand edge. */ import { Producer as BroadcastProducer } from "../src/broadcast.ts"; +import * as Epoch from "../src/epoch.ts"; import { Producer } from "../src/origin.ts"; import * as Path from "../src/path.ts"; @@ -85,3 +86,37 @@ for (const trackCount of [8, 32, 128]) { broadcast.close(); } } + +// Resolve one touched path while both the table and that path's readers grow. +console.log("request,routes,readers,update_and_read_us"); +const epoch = Epoch.parse("01900000-0000-7000-8000-000000000001"); +for (const pinned of [false, true]) { + for (const routeCount of routeCounts) { + for (const readerCount of observerCounts) { + const origin = new Producer(); + const reader = origin.consume(); + const options = pinned ? { epoch } : {}; + const publishers = Array.from({ length: routeCount }, (_, index) => { + const source = origin.createBroadcast(Path.from(`room/${index}`)); + source.announce({ epoch }); + return source; + }); + const background = publishers.map((_, index) => reader.request(Path.from(`room/${index}`), options)); + const requests = Array.from({ length: readerCount }, () => reader.request(Path.from("room/0"), options)); + const start = performance.now(); + for (let index = 0; index < updates; index++) { + publishers[0].announce({ epoch, cost: BigInt(index + 1) }); + await Promise.resolve(); + for (const request of requests) { + if (!request.active.peek()) throw new Error("same-epoch request lost its front"); + checksum++; + } + } + console.log( + `${pinned ? "pinned" : "unpinned"},${routeCount},${readerCount},${(((performance.now() - start) * 1000) / updates).toFixed(1)}`, + ); + for (const request of [...requests, ...background]) request.close(); + origin.close(); + } + } +} diff --git a/js/net/src/broadcast.ts b/js/net/src/broadcast.ts index 369c481b060..9946e8132ac 100644 --- a/js/net/src/broadcast.ts +++ b/js/net/src/broadcast.ts @@ -4,6 +4,7 @@ * @module */ import { type Dispose, type GetPromise, type Getter, Once, Signal } from "@moq/signals"; +import type * as Epoch from "./epoch.ts"; import { NotFound } from "./error.ts"; import type { Consumer as GroupConsumer } from "./group.ts"; import { Route } from "./hop.ts"; @@ -415,6 +416,7 @@ export class Producer { interface Shared { state: BroadcastState; path: Path.Valid; + epoch?: Epoch.Valid; } // Constructs a Consumer from within this module without exposing a public constructor @@ -432,6 +434,7 @@ let makeConsumer: (shared: Shared) => Consumer; export class Consumer { #state: BroadcastState; #path: Path.Valid; + #epoch?: Epoch.Valid; // Guards against a double close() on this handle over-decrementing the consumer count. #closed = false; @@ -439,6 +442,7 @@ export class Consumer { protected constructor(shared?: never); protected constructor(shared?: Shared) { this.#path = shared?.path ?? Path.empty(); + this.#epoch = shared?.epoch; if (shared) { this.#state = shared.state; } else { @@ -461,6 +465,9 @@ export class Consumer { if (target instanceof Consumer) target.#path = path; else stampProducer(target, path); }; + hooks.stampEpoch = (target, epoch) => { + target.#epoch = epoch; + }; } /** @@ -475,6 +482,17 @@ export class Consumer { return this.#path; } + /** + * The publisher epoch of the route an origin resolved this handle through, if it has one. + * + * An origin stamps it with each request's result, so it names the publisher + * instance actually serving this handle even when the request named none. `undefined` for a + * route without an epoch or a handle not handed out by an origin. + */ + get epoch(): Epoch.Valid | undefined { + return this.#epoch; + } + /** * Settles with `null` once the broadcast closes; a broadcast end carries no cause. * Peek it synchronously (`undefined` while open), observe it reactively, or `await` it. @@ -501,7 +519,7 @@ export class Consumer { // Hand this consumer's backing state and path to a clone. Opaque (`never`) so the state type // stays unexported; a subclass passes it straight back into its own `super(...)`. protected shareState(): never { - return { state: this.#state, path: this.#path } satisfies Shared as never; + return { state: this.#state, path: this.#path, epoch: this.#epoch } satisfies Shared as never; } /** Get a lazy handle for a track on this broadcast. Repeat subscriptions dedupe onto one upstream subscription. */ diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 2cb2319cc5e..480796fbe5f 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -7,6 +7,7 @@ */ import type { Dispose, Getter } from "@moq/signals"; import type { Consumer as BroadcastConsumer, Producer as BroadcastProducer } from "./broadcast.ts"; +import type * as Epoch from "./epoch.ts"; import type { Frame, Consumer as GroupConsumer, Producer as GroupProducer } from "./group.ts"; import type { Route } from "./hop.ts"; import * as Path from "./path.ts"; @@ -186,6 +187,8 @@ export const hooks: { ) => void; /** Name a broadcast handle by the path an origin created or resolved it at. */ stampPath: (target: BroadcastProducer | BroadcastConsumer, path: Path.Valid) => void; + /** Name the epoch of the route an origin resolved a broadcast handle through. */ + stampEpoch: (target: BroadcastConsumer, epoch: Epoch.Valid | undefined) => void; } = { makeRequest: () => { throw new Error("track.ts not loaded"); @@ -232,6 +235,9 @@ export const hooks: { stampPath: () => { throw new Error("broadcast.ts not loaded"); }, + stampEpoch: () => { + throw new Error("broadcast.ts not loaded"); + }, }; /** diff --git a/js/net/src/origin.test.ts b/js/net/src/origin.test.ts index 8ea7decbc25..4560f7f5210 100644 --- a/js/net/src/origin.test.ts +++ b/js/net/src/origin.test.ts @@ -1885,3 +1885,67 @@ test("a peer naming another epoch is refused rather than served a different inst local.close(); origin.close(); }); + +test("request pins its epoch while an unpinned sibling follows the newest publisher", () => { + const origin = new Producer(); + const path = Path.from("room"); + const first = origin.createBroadcast(path); + first.announce({ epoch: EPOCH }); + const reader = origin.consume(); + const pinned = reader.request(path, { epoch: EPOCH }); + const unpinned = reader.request(path); + expect(pinned.active.peek()?.epoch).toBe(EPOCH); + expect(unpinned.active.peek()?.epoch).toBe(EPOCH); + const newerEpoch = Epoch.parse("01900000-0000-7000-8000-000000000002"); + const second = origin.createBroadcast(path); + second.announce({ epoch: newerEpoch }); + expect(pinned.active.peek()).toBeUndefined(); + expect(pinned.unroutable.peek()).toBe(true); + // The unpinned handle names the publisher it actually resolved. + expect(unpinned.active.peek()?.epoch).toBe(newerEpoch); + const stale = reader.request(path, { epoch: EPOCH }); + expect(stale.active.peek()).toBeUndefined(); + expect(stale.unroutable.peek()).toBe(true); + stale.close(); + pinned.close(); + unpinned.close(); + origin.close(); +}); + +test("request never accepts an old handler after its epoch changes asynchronously", async () => { + const origin = new Producer(); + const prefix = Path.from("room"); + const path = Path.from("room/cam"); + const dynamic = origin.dynamic(prefix, { epoch: EPOCH }); + const pending = origin.consume().request(path, { epoch: EPOCH }); + const requests = dynamic.requested(); + const old = (await requests.next()).value; + const epoch = Epoch.parse("01900000-0000-7000-8000-000000000002"); + dynamic.update({ ...dynamic.route, epoch }); + const current = origin.consume().request(path, { epoch }); + old?.accept(new BroadcastProducer().consume()); + expect(pending.active.peek()).toBeUndefined(); + expect(pending.unroutable.peek()).toBe(true); + expect(current.active.peek()).toBeUndefined(); + const next = (await requests.next()).value; + next?.accept(new BroadcastProducer().consume()); + expect(current.active.peek()?.epoch).toBe(epoch); + expect(pending.active.peek()).toBeUndefined(); + await requests.return?.(); + pending.close(); + current.close(); + dynamic.close(); + origin.close(); +}); + +test("a route without an epoch resolves a handle without one", () => { + const origin = new Producer(); + const path = Path.from("room"); + const broadcast = origin.createBroadcast(path); + broadcast.announce(); + const request = origin.consume().request(path); + expect(request.active.peek()).toBeDefined(); + expect(request.active.peek()?.epoch).toBeUndefined(); + request.close(); + origin.close(); +}); diff --git a/js/net/src/origin.ts b/js/net/src/origin.ts index 2f63a1557c0..e6ff63e54ca 100644 --- a/js/net/src/origin.ts +++ b/js/net/src/origin.ts @@ -179,7 +179,13 @@ export interface RequestSlot { announced: number; answer?: broadcast.Consumer; readonly handles: Set>; - readonly route: Signal; + readonly route: Signal; +} + +/** One path resolution and the epoch of the route that can currently serve it. */ +interface Resolution { + readonly front?: broadcast.Consumer; + readonly epoch?: Epoch.Valid; } /** @@ -349,6 +355,11 @@ class ServeState { if (this.closed.peek() !== undefined) return; const err = abort ?? unroutable(); this.closed.set(err); + this.reset(err); + } + + /** Release answers and invalidate pending handlers when the publisher instance changes. */ + reset(err: Error = unroutable()): void { const queued = [...this.pending.values()]; this.pending.clear(); this.queue.mutate((queue) => { @@ -357,11 +368,12 @@ class ServeState { for (const request of queued) { finishRequest(request, err); } - for (const [path, front] of this.served) { + const served = [...this.served]; + this.served.clear(); + for (const [path, front] of served) { front.close(); this.onChange(path); } - this.served.clear(); this.rejected.clear(); this.demanding.clear(); this.settled.update((n) => n + 1); @@ -468,7 +480,7 @@ class OriginState { // Broadcasts materialized from a served route, keyed by exact path. Shared by every // request for the path so repeats reuse one accept; dropped (and closed) when the // providing route goes away or the last request releases it. - materialized = new Map(); + materialized = new Map(); // Paths consumers asked for without waiting for an announcement; attached sessions // answer them with blind subscriptions. Never announced: an answered request is assumed @@ -612,13 +624,13 @@ class OriginState { * before the old one breaks: the current front keeps serving until the new route * answers (then swaps) or refuses (then the request ends). A retracted route swaps at once. */ - route(path: Path.Valid, slot: Pick): broadcast.Consumer | undefined { + route(path: Path.Valid, slot: Pick): Resolution | undefined { const entry = this.bestEntry(path); const local = this.local.peek()?.get(path); if (local && this.localWins(path, entry)) { // Nothing reads a remote front the local broadcast replaced, so close its session subscription. this.releaseMaterialized(path); - return local; + return { front: local, epoch: this.advertisedLocal.peek()?.get(path)?.epoch }; } let cached = this.materialized.get(path); @@ -626,21 +638,23 @@ class OriginState { this.materialized.delete(path); cached = undefined; } - if (cached && cached.entry === entry) return cached.front; + const epoch = entry?.route.peek().epoch; + if (cached && cached.entry === entry && cached.epoch === epoch) return cached; if (!entry?.server) { this.releaseMaterialized(path); - return slot.answer; + return slot.answer ? { front: slot.answer } : undefined; } const served = entry.server.served.get(path); if (served && served.closed.peek() === undefined) { cached?.front.close(); - this.materialized.set(path, { entry, front: served }); - return served; + const resolution = { entry, front: served, epoch }; + this.materialized.set(path, resolution); + return resolution; } entry.server.enqueue(path); - return cached?.front; + return { front: cached?.epoch === epoch ? cached?.front : undefined, epoch }; } } @@ -683,6 +697,8 @@ export interface Table { export interface RequestOptions { /** Wait for a routed announcement when discovery is supported; otherwise subscribe blindly. */ announced?: boolean; + /** Refuse a different publisher instance, including an asynchronous answer or a retry. */ + epoch?: Epoch.Valid; } /** @@ -1255,6 +1271,7 @@ export class Consumer { * feeds from more than one connection. */ request(path: Path.Valid, options: RequestOptions = {}): Requesting { + const epoch = options.epoch; const relative = path; path = this.#scope.path(path); const requests = this.#state.requests.peek(); @@ -1306,15 +1323,19 @@ export class Consumer { // we already made, and only a real swap clones a new one (cloning before closing the // old, so a broadcast that both routes share never briefly loses its last handle). let released = false; - let source: broadcast.Consumer | undefined; + let source: Resolution | undefined; let handle: broadcast.Consumer | undefined; - const own = (front: broadcast.Consumer | undefined): broadcast.Consumer | undefined => { + const own = (resolution: Resolution | undefined): broadcast.Consumer | undefined => { if (released) return undefined; - if (front !== source) { + const front = epoch === undefined || resolution?.epoch === epoch ? resolution?.front : undefined; + if (front !== source?.front || resolution?.epoch !== source?.epoch) { const previous = handle; - source = front; + source = front ? { front, epoch: resolution?.epoch } : undefined; handle = front?.clone(); - if (handle) hooks.stampPath(handle, relative); + if (handle) { + hooks.stampPath(handle, relative); + hooks.stampEpoch(handle, resolution?.epoch); + } previous?.close(); } return handle; @@ -1334,7 +1355,10 @@ export class Consumer { // answer. const unroutable = new Derived( [route, this.#state.answerers, closed], - (front, answerers, ended) => ended !== undefined || (!front && answerers === 0), + (resolution, answerers, ended) => + ended !== undefined || + (epoch !== undefined && resolution?.epoch !== epoch) || + (!resolution?.front && answerers === 0), ); return makeRequesting(relative, active, unroutable, closed, () => { @@ -1607,7 +1631,10 @@ export class Dynamic { */ update(route: Route | { epoch?: Route["epoch"]; hops?: Route["hops"]; cost?: Route["cost"] | bigint }): void { if (this.#closed) throw new Error("dynamic is closed"); - this.#entry.route.set(Route.normalize(route)); + const next = Route.normalize(route); + const previous = this.#entry.route.peek().epoch; + this.#entry.route.set(next); + if (previous !== next.epoch) this.#entry.server?.reset(); this.#state.rebuildOriginated(); this.#state.refreshPrefix(Path.join(this.#entry.scope.root, this.prefix)); this.#state.routes.mutate(() => {}); diff --git a/quest/m0/broadcast-epoch/README.md b/quest/m0/broadcast-epoch/README.md index 4364ad7c2dc..98a3a4e2cb8 100644 --- a/quest/m0/broadcast-epoch/README.md +++ b/quest/m0/broadcast-epoch/README.md @@ -54,9 +54,9 @@ Decided: path cannot name one. - Every first-party publisher that can restart mints its own: the apps, moq-boy, and moq-room through [Apps](/quest/m0/broadcast-epoch/apps.md), the - ingest gateways, moqsink, and the bindings below. moq-stats mints one per group announcement - through [Stats epochs](/quest/m0/broadcast-epoch/stats-epoch.md), which - also gates the release (decided 2026-10-04). + ingest gateways, moqsink, and the bindings below. moq-stats mints one per + [group announcement](/doc/concept/stats.md#broadcasts), which also gated + the release (decided 2026-10-04). - The m1 quests gating this line moved under it in the 2026-10-05 audit, and the OBS half moved to m1 as [OBS publishes under epochs](/quest/m1/obs-epoch.md). @@ -78,6 +78,12 @@ that follow the announce `Restart` (or END then START) both reach the new epoch within one RTT-scale bound rather than the idle timeout, and killing the newest epoch falls back to a still-live older one. +When the release cut carries stats epochs, drop +[#4810](https://github.com/moq-dev/moq/pull/4810)'s wall-clock group seed and +its `doc/concept/stats.md` sentence from `release`; release-to-main +back-merges keep `main`'s `rs/moq-stats` and `doc/concept/stats.md` until +then. MoQ Pro's VOD `storage.json` moves from the same seed to its own epoch. + ## Required - [Apps](/quest/m0/broadcast-epoch/apps.md) - moq-cli, the browser publish and watch components, and demo/web restart into a new epoch and reset on the switch @@ -85,5 +91,4 @@ newest epoch falls back to a still-live older one. - [Publish catalog restart](/quest/m0/broadcast-epoch/publish-catalog-restart.md) - `@moq/publish` never reuses catalog group numbers under one name and epoch after a re-announce - [TS restart](/quest/m0/broadcast-epoch/ts-restart.md) - a signalled backward TS discontinuity finishes the broadcast and continues the same input under a fresh epoch - [Bindings](/quest/m0/broadcast-epoch/bindings.md) - moq-ffi and every wrapper expose the epoch and let a publisher announce one -- [Stats epochs](/quest/m0/broadcast-epoch/stats-epoch.md) - moq-stats publishes each group announcement under its own epoch, so neither a restarted node nor a returning idle group stalls its viewers - [Stats totals and prefix tracks](/quest/m0/broadcast-epoch/stats-split.md) - the same release retires the per-path stats maps for totals and on-demand prefix tracks (decided 2026-10-05) diff --git a/quest/m0/broadcast-epoch/stats-epoch.md b/quest/m0/broadcast-epoch/stats-epoch.md deleted file mode 100644 index 8fe0f132190..00000000000 --- a/quest/m0/broadcast-epoch/stats-epoch.md +++ /dev/null @@ -1,47 +0,0 @@ -# [S] Stats epochs - -## Goal - -Each stats group broadcast at `[/]/node/` announces under -its own epoch, minted each time the group is announced, so neither a restarted -node nor a group returning from idle reuses a broadcast identity or the group -numbers a relay cached under it. - -## Plan - -- Use the shared `moq_net::Epoch`, not a stats-local UUID (decided - 2026-10-02). -- One epoch per group announcement (decided 2026-10-05, replacing 2026-10-02's - one epoch per `Producer`). A group that goes idle unannounces after its - linger and drops its totals; when it returns it announces under a new epoch - counted from zero. Why: a producer never holds an idle group's state, so its - memory is bounded by active and lingering groups, and readers see every - reset as a new broadcast rather than detecting one. The cost: a reader that - misses every frame across the linger loses that epoch's tail; billing - under-bills by that tail, consistent with the 0-bill baseline. - At depth 0 the single broadcast never unannounces, so its epoch still lasts - the producer's life. -- [#4739](https://github.com/moq-dev/moq/pull/4739)'s original commits are - prior art for the aggregator's restarted-epoch handling and the tests; its - path syntax is superseded, since the epoch is on the route. The PR landed only the - producer-wide group allocator. -- The aggregator treats a new epoch as a new node: its counters add to the - merged total instead of regressing it, and the old epoch folds into the - retired total once `aggregate::Config::grace` elapses. Test it. -- Expose each group broadcast's epoch (a per-group accessor replacing - #4739's `Producer::epoch()`) so a consumer can tell epochs apart. - MoQ Pro's VOD `storage.json` mints its own epoch rather than sharing a - producer-wide one, which no longer exists. -- `release` carries a temporary seed instead - ([#4810](https://github.com/moq-dev/moq/pull/4810)): a producer's first group - number is wall-clock microseconds, so a restarted node numbers above its - previous run. Epochs replace it, so it never reaches `main`: release-to-main - back-merges keep `main`'s `rs/moq-stats` and `doc/concept/stats.md`. Drop the - seed and its `doc/concept/stats.md` sentence from `release` when this ships - there. MoQ Pro's VOD `storage.json` seeds the same way and moves to its - own epoch with it. -- demo/web stats keys include the epoch. Update `doc/concept/stats.md`, - `doc/bin/relay/config.md`, and the relay stats config docs. - -Public API: stats consumers read the epoch from the route. Wire: none beyond -the route epoch. diff --git a/quest/m0/broadcast-epoch/stats-split.md b/quest/m0/broadcast-epoch/stats-split.md index 222bc2c2d90..852a4977ba2 100644 --- a/quest/m0/broadcast-epoch/stats-split.md +++ b/quest/m0/broadcast-epoch/stats-split.md @@ -19,7 +19,7 @@ Decided 2026-10-05 (planned from moq-dev/moq.pro#2202): reader undercounts. The maps also grow with every live broadcast, and readers like billing want only per-project sums. - **Totals.** Per group broadcast (`[/]/node/`), - per tier and role, cumulative within the [stats epoch](/quest/m0/broadcast-epoch/stats-epoch.md). + per tier and role, cumulative within the [stats epoch](/doc/concept/stats.md#broadcasts). Every group announcement, and so every restart, starts a new epoch counted from zero; nothing is serialized to disk. `Registry`'s unpruned lifetime totals (`rs/moq-net/src/stats.rs`) are node-wide, per tier and role only, so they @@ -32,7 +32,7 @@ Decided 2026-10-05 (planned from moq-dev/moq.pro#2202): ends. Today a path that left drops out of frames while the group lingers; with totals, a group that returns within the linger continues its totals. A zero linger is valid: a returning group then always takes a new epoch. After the linger it unannounces and drops its - totals; a return announces under a new [epoch](/quest/m0/broadcast-epoch/stats-epoch.md) + totals; a return announces under a new [epoch](/doc/concept/stats.md#broadcasts) counted from zero. Totals are cumulative and a lingering group's frames all repeat its final totals, so a reader that misses every frame across the linger loses that epoch's tail; billing under-bills by that tail, @@ -106,10 +106,6 @@ Broadcasts page reads one prefix per visible row and group header, its `announce reads totals only, and its customer stats feed serves this format summed across nodes. -## Required - -- [Stats epochs](/quest/m0/broadcast-epoch/stats-epoch.md) - totals are cumulative within an epoch - ## Related - [Media stats](/quest/m1/stats/README.md) - publisher and viewer media stats diff --git a/rs/hang/examples/subscribe.rs b/rs/hang/examples/subscribe.rs index 5416f19cafb..f7d403b7e65 100644 --- a/rs/hang/examples/subscribe.rs +++ b/rs/hang/examples/subscribe.rs @@ -43,14 +43,16 @@ async fn run_session(origin: moq_net::origin::Producer) -> anyhow::Result<()> { async fn run_subscribe(consumer: moq_net::origin::Consumer) -> anyhow::Result<()> { // Wait for a route to be announced, then resolve the broadcast at its path. // The convention is that a publisher announces each broadcast's exact path. - let path = match consumer.announced().next().await.context("origin closed")? { - moq_net::announce::Event::Start(announce) => announce.prefix, + let announce = match consumer.announced().next().await.context("origin closed")? { + moq_net::announce::Event::Start(announce) => announce, // Nothing else can come before the first announcement. event => anyhow::bail!("unexpected {event:?}"), }; - tracing::info!(%path, "broadcast announced"); - let broadcast = consumer.request_broadcast(&path).await?; + tracing::info!(path = %announce.prefix, "broadcast announced"); + let broadcast = consumer + .request_broadcast(&announce.prefix, announce.route.epoch) + .await?; // Read the catalog to discover available tracks. let catalog_track = broadcast diff --git a/rs/moq-bench/src/connection.rs b/rs/moq-bench/src/connection.rs index 50e0f3b6a0f..6e40b5cf89a 100644 --- a/rs/moq-bench/src/connection.rs +++ b/rs/moq-bench/src/connection.rs @@ -296,7 +296,7 @@ async fn subscribe_named(consume: moq_net::origin::Consumer, path: String, stats .routed(path.as_str()) .await .ok_or_else(|| anyhow::anyhow!("target broadcast was never announced: {path}"))?; - let broadcast = consume.request_broadcast(path.as_str()).await?; + let broadcast = consume.request_broadcast(path.as_str(), None).await?; drain(broadcast, &stats).await } @@ -352,7 +352,7 @@ async fn subscribe( let mut selected = pool.len() as u64; for path in pool { - let Ok(broadcast) = consume.request_broadcast(path.as_str()).await else { + let Ok(broadcast) = consume.request_broadcast(path.as_str(), None).await else { continue; }; spawn_drain(&mut tasks, path, broadcast, stats.clone()); @@ -370,7 +370,7 @@ async fn subscribe( if own.contains(&path) || !seen.insert(path.clone()) { continue; } - let Ok(broadcast) = consume.request_broadcast(path.as_str()).await else { + let Ok(broadcast) = consume.request_broadcast(path.as_str(), None).await else { continue; }; selected += 1; diff --git a/rs/moq-boy/src/input.rs b/rs/moq-boy/src/input.rs index 61202b11273..82bede431ac 100644 --- a/rs/moq-boy/src/input.rs +++ b/rs/moq-boy/src/input.rs @@ -75,7 +75,7 @@ pub async fn handle_viewers( let viewer_id = update.prefix.to_string(); if active { - let Ok(broadcast) = viewer_origin.request_broadcast(&update.prefix).await else { + let Ok(broadcast) = viewer_origin.request_broadcast(&update.prefix, None).await else { continue; }; tracing::info!(%viewer_id, "viewer connected"); diff --git a/rs/moq-c/src/origin.rs b/rs/moq-c/src/origin.rs index ca3f4b3183f..e10a6f6715e 100644 --- a/rs/moq-c/src/origin.rs +++ b/rs/moq-c/src/origin.rs @@ -295,7 +295,7 @@ impl Origin { mut close: oneshot::Receiver<()>, ) -> Result<(), Error> { // Resolves to an error when no announced route can serve the path. - let pending = consumer.request_broadcast(path.as_str()); + let pending = consumer.request_broadcast(path.as_str(), None); // `biased` so a pending close always wins over a ready broadcast. let broadcast = tokio::select! { diff --git a/rs/moq-cli/tests/import.rs b/rs/moq-cli/tests/import.rs index bab5846621a..6cf401b75d9 100644 --- a/rs/moq-cli/tests/import.rs +++ b/rs/moq-cli/tests/import.rs @@ -66,7 +66,7 @@ async fn import_delivers_the_catalog_finish_at_eof() { .expect("origin closed"), moq_tokio::moq_net::announce::Event::Start(_) ) {} - let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("demo")) + let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("demo", None)) .await .expect("request timed out") .expect("announced broadcast resolves"); diff --git a/rs/moq-e2ee/tests/transport.rs b/rs/moq-e2ee/tests/transport.rs index f5a4ba4f402..28ee9850c6d 100644 --- a/rs/moq-e2ee/tests/transport.rs +++ b/rs/moq-e2ee/tests/transport.rs @@ -63,7 +63,7 @@ async fn connect_protected(version: Version, track: &str) -> Fixture { } let epoch = route.epoch.unwrap_or_else(|| generation.epoch().clone()); let generation = cred.generation(epoch); - let remote = consumer.request_broadcast(&path).await.unwrap(); + let remote = consumer.request_broadcast(&path, None).await.unwrap(); let subscriber = remote .track(name.as_str()) .unwrap() diff --git a/rs/moq-ffi/src/origin.rs b/rs/moq-ffi/src/origin.rs index 4ddbd3f2027..42ebc04b9fc 100644 --- a/rs/moq-ffi/src/origin.rs +++ b/rs/moq-ffi/src/origin.rs @@ -407,7 +407,7 @@ impl MoqOriginConsumer { /// Calling this straight after connecting therefore races the session's announcements /// and can report a live broadcast as unroutable. Await `announced_broadcast` first. pub async fn request_broadcast(&self, path: String) -> Result, MoqError> { - let broadcast = self.inner.request_broadcast(path.as_str()).await?; + let broadcast = self.inner.request_broadcast(path.as_str(), None).await?; Ok(Arc::new(MoqBroadcastConsumer::routed(broadcast, self.inner.clone()))) } } diff --git a/rs/moq-ffi/src/test.rs b/rs/moq-ffi/src/test.rs index a05d9548da8..4d68f151474 100644 --- a/rs/moq-ffi/src/test.rs +++ b/rs/moq-ffi/src/test.rs @@ -2650,7 +2650,7 @@ async fn cancel_retracts_the_route_synchronously() { let dynamic = serve(&origin, ""); let inner = origin.inner().consume(); - let queued = inner.request_broadcast("x").into_inner(); + let queued = inner.request_broadcast("x", None).into_inner(); assert!( queued.poll_ok(&kio::Waiter::noop()).is_pending(), "the route must serve while the handler lives" @@ -2658,7 +2658,7 @@ async fn cancel_retracts_the_route_synchronously() { drop(queued); dynamic.cancel(); - let verdict = inner.request_broadcast("y").into_inner(); + let verdict = inner.request_broadcast("y", None).into_inner(); match verdict.poll_ok(&kio::Waiter::noop()) { std::task::Poll::Ready(Err(moq_net::Error::Unroutable)) => {} std::task::Poll::Ready(Err(err)) => panic!("unexpected error: {err}"), diff --git a/rs/moq-hls/src/export/mod.rs b/rs/moq-hls/src/export/mod.rs index c0a07b11d07..69ac18b8acf 100644 --- a/rs/moq-hls/src/export/mod.rs +++ b/rs/moq-hls/src/export/mod.rs @@ -2449,10 +2449,13 @@ mod tests { accept_sibling(&new_server, &new_media).await; // The rendition skips timeline rows until this bind resolves. Joining the // same front waits for attach, so writes below land after that. - tokio::time::timeout(Duration::from_secs(5), origin.consume().request_broadcast("media")) - .await - .expect("the rebound sibling resolves") - .expect("the replacement is routable"); + tokio::time::timeout( + Duration::from_secs(5), + origin.consume().request_broadcast("media", None), + ) + .await + .expect("the rebound sibling resolves") + .expect("the replacement is routable"); let recorder = catalog.enroll_test("video0").unwrap(); // Model source records lost during the outage: every rendition must carry the // same new timeline sequence, including a recorder that starts after this rebind. @@ -2656,10 +2659,13 @@ mod tests { let mut new_media = moq_net::broadcast::Info::new().produce(); let _ = rendition.snapshot(); accept_sibling(&new_server, &new_media).await; - tokio::time::timeout(Duration::from_secs(5), origin.consume().request_broadcast("media")) - .await - .expect("the rebound sibling resolves") - .expect("the replacement is routable"); + tokio::time::timeout( + Duration::from_secs(5), + origin.consume().request_broadcast("media", None), + ) + .await + .expect("the rebound sibling resolves") + .expect("the replacement is routable"); let recorder = catalog.enroll_test("video0").unwrap(); let _new_track = write_routed_media(&mut new_media, NEW, recorder, 6_000_000); let deadline = tokio::time::Instant::now() + Duration::from_secs(5); diff --git a/rs/moq-hls/src/server/routes.rs b/rs/moq-hls/src/server/routes.rs index 024f13e86ca..afb99720aee 100644 --- a/rs/moq-hls/src/server/routes.rs +++ b/rs/moq-hls/src/server/routes.rs @@ -772,7 +772,7 @@ mod tests { let app = Server::new(pair.sub_origin.consume(), crate::export::Config::default()).router(); wait_listed(&app, "/live/video/video0/media.m3u8", "seg/656e3d1b.0.m4s").await; - let remote = tokio::time::timeout(TIMEOUT, pair.sub_origin.consume().request_broadcast("live")) + let remote = tokio::time::timeout(TIMEOUT, pair.sub_origin.consume().request_broadcast("live", None)) .await .expect("remote resolve timed out") .expect("remote broadcast"); @@ -817,10 +817,13 @@ mod tests { let app = Server::new(pair.sub_origin.consume(), crate::export::Config::default()).router(); wait_listed(&app, "/room/live/video/video0/media.m3u8", "seg/656e3d1b.0.m4s").await; - let remote_media = tokio::time::timeout(TIMEOUT, pair.sub_origin.consume().request_broadcast("room/source")) - .await - .expect("sibling resolve timed out") - .expect("sibling broadcast"); + let remote_media = tokio::time::timeout( + TIMEOUT, + pair.sub_origin.consume().request_broadcast("room/source", None), + ) + .await + .expect("sibling resolve timed out") + .expect("sibling broadcast"); drop(media); drop(media_broadcast); tokio::time::timeout(TIMEOUT, remote_media.closed()) diff --git a/rs/moq-mux/src/catalog/hang/consumer.rs b/rs/moq-mux/src/catalog/hang/consumer.rs index 236954fe2ba..9a8ed649fbc 100644 --- a/rs/moq-mux/src/catalog/hang/consumer.rs +++ b/rs/moq-mux/src/catalog/hang/consumer.rs @@ -204,7 +204,7 @@ mod test { let origin = crate::source::produce_origin(); let dynamic = origin.dynamic("", Default::default()).unwrap(); - let requesting = origin.consume().request_broadcast("a/pub"); + let requesting = origin.consume().request_broadcast("a/pub", None); let served = broadcast.consume(); // The handler hands its route back: dropping it would retract the route and end // the broadcast it just served. diff --git a/rs/moq-mux/src/container/consumer.rs b/rs/moq-mux/src/container/consumer.rs index a83edffb103..8626d58591c 100644 --- a/rs/moq-mux/src/container/consumer.rs +++ b/rs/moq-mux/src/container/consumer.rs @@ -1661,7 +1661,7 @@ mod tests { .with_cost(5), ) .unwrap(); - let pending = origin.consume().request_broadcast("live"); + let pending = origin.consume().request_broadcast("live", None); let first = moq_net::broadcast::Info::new().produce(); let info = hang::container::track_info(hang::catalog::PRIORITY.video); let first_track = first.create_track("video", info.clone()).unwrap(); diff --git a/rs/moq-mux/src/source.rs b/rs/moq-mux/src/source.rs index 9c884ac552d..85d45d4f4b5 100644 --- a/rs/moq-mux/src/source.rs +++ b/rs/moq-mux/src/source.rs @@ -70,7 +70,7 @@ impl Source { /// Begin resolving the catalog broadcast (the one at this source's path). pub(crate) fn request_catalog(&self) -> kio::Pending { - self.origin.request_broadcast(&self.path) + self.origin.request_broadcast(&self.path, None) } /// Resolve a rendition's optional broadcast reference to an origin path. @@ -110,7 +110,7 @@ impl Source { &self, rel: Option<&moq_net::path::Relative<'_>>, ) -> crate::Result> { - Ok(self.origin.request_broadcast(&self.target(rel)?)) + Ok(self.origin.request_broadcast(&self.target(rel)?, None)) } /// The skipping counterpart to [`Self::request`], returning `None` when `rel` walks above @@ -123,7 +123,7 @@ impl Source { &self, rel: Option<&moq_net::path::Relative<'_>>, ) -> Option> { - Some(self.origin.request_broadcast(&self.resolve_reference(rel)?)) + Some(self.origin.request_broadcast(&self.resolve_reference(rel)?, None)) } /// Remove renditions whose broadcast reference escapes above the origin root. diff --git a/rs/moq-net/benches/origin.rs b/rs/moq-net/benches/origin.rs index 252acec3397..268056233bd 100644 --- a/rs/moq-net/benches/origin.rs +++ b/rs/moq-net/benches/origin.rs @@ -311,7 +311,7 @@ fn bench_announce_fronts(c: &mut Criterion) { // Served but never answered: every request under it parks a front. let _served = producer.dynamic("room", origin::Route::default()).unwrap(); let _requests: Vec<_> = (0..fronts) - .map(|i| consumer.request_broadcast(format!("room/{i}"))) + .map(|i| consumer.request_broadcast(format!("room/{i}"), None)) .collect(); let waiter = kio::Waiter::noop(); // Run each front once so it parks on its upstream request. @@ -405,7 +405,7 @@ fn bench_request(c: &mut Criterion) { let waiter = kio::Waiter::noop(); group.bench_function(BenchmarkId::new("local", publishers), |b| { b.iter(|| { - let pending = fleet.consumer.request_broadcast("room/0"); + let pending = fleet.consumer.request_broadcast("room/0", None); // The front's driver resolves the first request; later ones join it. fleet.driver.poll(moq_net::time::Instant::now(), &waiter).unwrap(); pending @@ -418,7 +418,7 @@ fn bench_request(c: &mut Criterion) { b.iter(|| { let result = fleet .consumer - .request_broadcast("room/missing") + .request_broadcast("room/missing", None) .now_or_never() .expect("fails synchronously"); assert!(matches!(result, Err(moq_net::Error::Unroutable))); @@ -456,7 +456,7 @@ fn pool(members: usize, paths: usize) -> Pool { let waiter = kio::Waiter::noop(); let requests: Vec<_> = (0..paths) - .map(|i| consumer.request_broadcast(format!("pool/job-{i}"))) + .map(|i| consumer.request_broadcast(format!("pool/job-{i}"), None)) .collect(); driver.poll(moq_net::time::Instant::now(), &waiter).unwrap(); let mut producers = Vec::with_capacity(paths); @@ -549,7 +549,7 @@ fn bench_handoff(c: &mut Criterion) { first.write_frame(Timestamp::ZERO, b"one".as_ref()).unwrap(); first.finish().unwrap(); - let resolved = consumer.request_broadcast("room/live").await.unwrap(); + let resolved = consumer.request_broadcast("room/live", None).await.unwrap(); let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); subscription.recv_group().await.unwrap().expect("first group"); @@ -604,7 +604,7 @@ fn bench_relay(c: &mut Criterion) { .map(|i| broadcast.create_track(format!("{i}"), None).unwrap()) .collect(); let mut subscriptions = runtime.block_on(async { - let resolved = producer.consume().request_broadcast("room/live").await.unwrap(); + let resolved = producer.consume().request_broadcast("room/live", None).await.unwrap(); let mut subscriptions = Vec::new(); for i in 0..tracks { let track = resolved.track(&format!("{i}")).unwrap(); @@ -661,7 +661,7 @@ fn bench_parked(c: &mut Criterion) { .collect(); let _writers: Vec<_> = sources.iter().map(|source| source.append_group().unwrap()).collect(); let (subscriptions, mut groups) = runtime.block_on(async { - let resolved = producer.consume().request_broadcast("room/live").await.unwrap(); + let resolved = producer.consume().request_broadcast("room/live", None).await.unwrap(); let mut subscriptions = Vec::new(); let mut groups = Vec::new(); for i in 0..tracks { @@ -769,7 +769,7 @@ async fn copy_walk(tracks: usize, copies: usize) -> CopyWalk { let epoch = Epoch::mint(); let first = Generation::build(&producer, tracks); first.route(&epoch); - let resolved = tokio::time::timeout(COPY_WAIT, producer.consume().request_broadcast(COPY_PATH)) + let resolved = tokio::time::timeout(COPY_WAIT, producer.consume().request_broadcast(COPY_PATH, None)) .await .expect("front did not resolve") .expect("front refused"); diff --git a/rs/moq-net/benches/viewers.rs b/rs/moq-net/benches/viewers.rs index 831ac4b5235..0b25499707f 100644 --- a/rs/moq-net/benches/viewers.rs +++ b/rs/moq-net/benches/viewers.rs @@ -107,7 +107,7 @@ impl Room { fn join(&mut self, n: usize) -> moq_net::track::Subscriber { let session = moq_net::fuzz::excluding(self.producer.consume(), Hop::random()); let path = format!("room/{}", n % self.broadcasts); - let resolved = self.drive(session.request_broadcast(path)).unwrap(); + let resolved = self.drive(session.request_broadcast(path, None)).unwrap(); let mut subscription = self.drive(resolved.track("video").unwrap().subscribe(None)).unwrap(); self.drive(subscription.recv_group()) .unwrap() diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index 43ba5b95a49..8e7ffb19c57 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -620,7 +620,7 @@ where let broadcast = match self .serving_origin() .await - .request_broadcast(&msg.track_namespace) + .request_broadcast(&msg.track_namespace, None) .await { Ok(broadcast) => broadcast, @@ -1373,7 +1373,7 @@ where // The peer must have seen the announcement to name this namespace, so this // resolves like a SUBSCRIBE does. - let broadcast = match self.serving_origin().await.request_broadcast(&namespace).await { + let broadcast = match self.serving_origin().await.request_broadcast(&namespace, None).await { Ok(broadcast) => broadcast, Err(err) => return self.reject_fetch(stream, msg.request_id, &err, &err.to_string()).await, }; diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 37f76b9aee6..5ffc6ae23a2 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -3752,11 +3752,15 @@ mod tests { let serving = kio::wait(|waiter| publisher.shared.poll_serving_origin(waiter)).await; use futures::FutureExt; assert!( - serving.request_broadcast("echoed/x").now_or_never().unwrap().is_err(), + serving + .request_broadcast("echoed/x", None) + .now_or_never() + .unwrap() + .is_err(), "served the peer its own route" ); assert!( - serving.request_broadcast("local/x").now_or_never().is_none(), + serving.request_broadcast("local/x", None).now_or_never().is_none(), "withheld an independent route" ); } diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 8341970efa7..6a0f416cab0 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -2607,7 +2607,7 @@ mod tests { assert!(announced.ready.is_empty()); let consumer = origin.consume(); - let request = moq_net_sim::spawn(async move { consumer.request_broadcast("room/7").await }); + let request = moq_net_sim::spawn(async move { consumer.request_broadcast("room/7", None).await }); let ready = kio::wait(|waiter| announced.ready.poll_pop(waiter)).await.unwrap(); assert_eq!(ready.as_str(), "room/7"); @@ -2798,7 +2798,7 @@ mod tests { // The peer's own subscription, excluding the hop the server minted for // it, is served from the local front before anything is announced back. let peer = origin.consume().excluding(assigned); - let resolved = peer.request_broadcast("room/host").await.expect("resolves"); + let resolved = peer.request_broadcast("room/host", None).await.expect("resolves"); let mut sub = resolved .track("video") .unwrap() @@ -2843,7 +2843,7 @@ mod tests { // The local front is still the one at the path, and still serving: the // peer's next request joins it rather than minting another. let still = peer - .request_broadcast("room/host") + .request_broadcast("room/host", None) .await .expect("the local front keeps serving"); assert!( diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index 5c22d4f3172..2fa46ab68dd 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -1964,6 +1964,14 @@ impl AnnounceProducer { let Some(entry) = shared.routes.entry_mut(prefix, *id) else { return Err(Error::Closed); }; + // Another publisher instance: what its server was asked, or answered, for the + // old one is not its own. Under the table lock, so no front sees the new epoch + // while an old answer can still land. + if entry.epoch != route.epoch + && let Some(server) = &entry.server + { + server.lock().renew(); + } entry.hops = route.hops.clone(); entry.epoch = route.epoch.clone(); entry.stale = stale; @@ -2386,9 +2394,11 @@ async fn serve_front(task: FrontTask) -> Vec { let mut dynamic = broadcast.dynamic(); let mut front = Front::new(track::IDLE_LINGER); let mut sources: HashMap = HashMap::new(); + // The epoch each source's route had when it was asked: the instance its content is from. + let mut epochs: HashMap> = HashMap::new(); let mut next_source = 0u64; - // The in-flight upstream request: the route and its pending channel. - let mut upstream: Option<(u64, kio::Consumer)> = None; + // The in-flight upstream request: the route, its epoch when asked, and its pending channel. + let mut upstream: Option<(u64, Option, kio::Consumer)> = None; let mut tracks: HashMap, TrackIo> = HashMap::new(); let mut deadline = crate::time::Deadline::new(&timers); // The watch generation the last selection saw. @@ -2436,16 +2446,18 @@ async fn serve_front(task: FrontTask) -> Vec { ) }; - // Whether resolving through `route` would serve an epoch a newer one already beat. - let superseded = |front: &Front, route: u64| { + // Whether an answer `route` gave for `asked` would serve an epoch a newer one already + // beat, or one the route itself has since moved off. + let superseded = |front: &Front, route: u64, asked: &Option| { let table = shared.read(); let Some(entry) = table.routes.covering(&path.as_path()).find(|entry| entry.id == route) else { return false; }; - matches!( - pick(&table, &path.as_path(), horizon, front, Some(&entry.epoch)), - Pick::Superseded - ) + entry.epoch != *asked + || matches!( + pick(&table, &path.as_path(), horizon, front, Some(&entry.epoch)), + Pick::Superseded + ) }; events.push_back(select(&mut front, &sources, &mut seen, &resolved)); @@ -2467,9 +2479,9 @@ async fn serve_front(task: FrontTask) -> Vec { .routes .covering(&path.as_path()) .find(|entry| entry.id == route && entry.live()) - .map(|entry| (entry.source.clone(), entry.server.clone())) + .map(|entry| (entry.source.clone(), entry.server.clone(), entry.epoch.clone())) }; - let Some((source, server)) = found else { + let Some((source, server, epoch)) = found else { events.push_back(Event::Resolved { route, result: Err(Refusal { @@ -2483,6 +2495,7 @@ async fn serve_front(task: FrontTask) -> Vec { let id = next_source; next_source += 1; sources.insert(id, source); + epochs.insert(id, epoch); events.push_back(Event::Resolved { route, result: Ok(id) }); continue; } @@ -2519,6 +2532,7 @@ async fn serve_front(task: FrontTask) -> Vec { let id = next_source; next_source += 1; sources.insert(id, weak.consume()); + epochs.insert(id, epoch); events.push_back(Event::Resolved { route, result: Ok(id) }); continue; } @@ -2545,13 +2559,14 @@ async fn serve_front(task: FrontTask) -> Vec { } } }; - upstream = Some((route, pending)); + upstream = Some((route, epoch, pending)); } Action::Detach { source } => { // Readers keep reading the source's copy until a replacement is // spliced, so a route being beaten keeps serving until then; a // route that died ends its copy, which says so. sources.remove(&source); + epochs.remove(&source); for io in tracks.values_mut() { if io.query.as_ref().is_some_and(|(asked, _)| asked.source == source) { io.query = None; @@ -2562,19 +2577,12 @@ async fn serve_front(task: FrontTask) -> Vec { } } Action::Resolve => { - // The broadcast is whatever its first route serves. - let table = shared.read(); + // The broadcast is whatever its first source serves, from the instance + // its route had when asked. let epoch = front - .serving_route() - .and_then(|route| { - table - .routes - .covering(&path.as_path()) - .find(|entry| entry.id == route) - .map(|entry| entry.epoch.clone()) - }) + .serving() + .and_then(|source| epochs.get(&source).cloned()) .flatten(); - drop(table); resolved = Some(epoch.clone()); if let Ok(mut pending) = request.write() && pending.resolved.is_none() @@ -2717,7 +2725,7 @@ async fn serve_front(task: FrontTask) -> Vec { if let Poll::Ready(Ok(request)) = dynamic.poll_requested_track(waiter) { return Poll::Ready(Step::Assigned(request)); } - if let Some((route, pending)) = &upstream + if let Some((route, _, pending)) = &upstream && let Poll::Ready(result) = pending.poll(waiter, |p| match &p.resolved { Some(result) => Poll::Ready(result.clone()), None => Poll::Pending, @@ -2805,12 +2813,12 @@ async fn serve_front(task: FrontTask) -> Vec { } } Step::Resolved(route, result) => { - upstream = None; + let asked = upstream.take().and_then(|(_, epoch, _)| epoch); match result { - // A newer epoch won the path while the first request was in flight: - // the route was beaten, so ask the winner rather than resolve - // requesters onto a broadcast that is already replaced. - Ok(_) if resolved.is_none() && superseded(&front, route) => Event::Resolved { + // A newer epoch won the path, or the route moved to another instance, + // while the first request was in flight: ask the winner rather than + // resolve requesters onto a broadcast that is already replaced. + Ok(_) if resolved.is_none() && superseded(&front, route, &asked) => Event::Resolved { route, result: Err(Refusal { err: Error::Unroutable, @@ -2821,6 +2829,7 @@ async fn serve_front(task: FrontTask) -> Vec { let id = next_source; next_source += 1; sources.insert(id, source); + epochs.insert(id, asked); Event::Resolved { route, result: Ok(id) } } Err(err) => Event::Resolved { @@ -3546,8 +3555,9 @@ impl Dynamic { /// At the same [`Route::epoch`], consumers observe another active update for /// the same prefix and sessions forward it as a restart, so re-pricing never /// looks like new content. Another epoch, or none, names another publisher - /// instance: consumers see the old broadcast end and a new one start, and the - /// broadcasts served through the old one end. To re-price, start from the + /// instance: consumers see the old broadcast end and a new one start, the + /// broadcasts served through the old one end, and requests still waiting on it + /// are refused with [`Error::Unroutable`]. To re-price, start from the /// current route: `dynamic.update(dynamic.route().with_cost(cost))`. /// /// The prefix is fixed at announce time: to move a route, drop this and call @@ -3620,6 +3630,11 @@ impl ServeState { if state.closed { return; } + // Refused already (the route changed instance): that answer stands and nothing + // is cached for the next request. + if state.requests.remove_if(path, |p| p.same_channel(producer)).is_none() { + return; + } let resolved = match result { Ok(broadcast) => { // If a live broadcast was already served for this path while we were @@ -3630,13 +3645,23 @@ impl ServeState { } Err(err) => Err(err), }; - state.requests.remove_if(path, |p| p.same_channel(producer)); if let Ok(mut pending) = producer.write() { pending.resolved.get_or_insert(resolved); drop(state); } } + /// Refuse every request, queued or handed to a handler, and forget the answers served, + /// because the route now names another publisher instance. + fn renew(&mut self) { + for producer in self.requests.drain_all() { + if let Ok(mut request) = producer.write() { + request.resolved.get_or_insert(Err(Error::Unroutable)); + } + } + self.served = WeakCache::default(); + } + /// Drop the still-pending entry, if it is still ours. fn forget(shared: &kio::Shared, path: &PathOwned, producer: &kio::Producer) { shared.lock().requests.remove_if(path, |p| p.same_channel(producer)); @@ -3773,9 +3798,13 @@ impl Requesting { self } - /// Stamp a resolved broadcast with the path this cursor asked for and its egress scope. - fn hand_out(&self, broadcast: broadcast::Consumer) -> broadcast::Consumer { - broadcast.with_path(self.path.clone()).with_stats(self.stats.clone()) + /// Stamp a resolved broadcast with the path this cursor asked for, its egress scope, + /// and the epoch of the route it resolved through. + fn hand_out(&self, broadcast: broadcast::Consumer, epoch: Option) -> broadcast::Consumer { + broadcast + .with_path(self.path.clone()) + .with_stats(self.stats.clone()) + .with_epoch(epoch) } /// Poll for the requested broadcast without blocking. @@ -3789,7 +3818,7 @@ impl Requesting { })) { // Another instance than the one named: never hand it out. Ok((Ok(_), epoch)) if self.epoch.is_some() && epoch != self.epoch => Err(Error::Unroutable), - Ok((result, epoch)) => result.map(|broadcast| self.hand_out(broadcast).with_epoch(epoch)), + Ok((result, epoch)) => result.map(|broadcast| self.hand_out(broadcast, epoch)), // Every handler dropped without resolving: nobody could route it. Err(_closed) => Err(Error::Unroutable), }, @@ -4241,7 +4270,7 @@ impl Consumer { let seen = watch.seen(); (watch, seen) }; - match self.request_broadcast(&path).await { + match self.request_broadcast(&path, None).await { Ok(broadcast) => return Ok(broadcast), Err(Error::Unroutable) => { kio::wait(|waiter| watch.poll_changed(waiter, seen)).await; @@ -4271,7 +4300,7 @@ impl Consumer { }) } - /// Resolve a broadcast by exact path. + /// Resolve a broadcast by exact path, optionally pinning its publisher epoch. /// /// Returns a [`kio::Pending`] future, mirroring /// [`track::Consumer::fetch_group`](track::Consumer::fetch_group). Every @@ -4290,14 +4319,23 @@ impl Consumer { /// newer epoch winning the path ends even those with [`Error::Unroutable`], so /// readers request the new broadcast instead of stalling on the old one. /// + /// Pass an owned [`crate::Epoch`] to refuse a different publisher instance, both + /// when requesting and when an asynchronous answer resolves. `None` leaves lookup + /// unpinned. A pinned request never resumes on a route whose epoch disappeared. + /// /// The returned future fails with [`Error::Unroutable`] at once when no /// announced route covers the path, including a broadcast created on this /// origin but not announced. /// A route claims capability, not inventory: resolving a covered path /// succeeds optimistically, and a path that names nothing surfaces as /// [`Error::NotFound`] on its tracks instead. - pub fn request_broadcast(&self, path: impl AsPath) -> kio::Pending { - self.request(path.as_path(), None) + pub fn request_broadcast( + &self, + path: impl AsPath, + epoch: impl Into>, + ) -> kio::Pending { + let epoch = epoch.into(); + self.request(path.as_path(), epoch.as_ref()) } /// [`Self::request_broadcast`], refused with [`Error::Unroutable`] unless the @@ -4889,7 +4927,7 @@ mod tests { broadcast.announce(Route::default()).unwrap(); consumer.announced().assert_next_wait(); - let resolved = consumer.request_broadcast(".stats/node").await.expect("resolves"); + let resolved = consumer.request_broadcast(".stats/node", None).await.expect("resolves"); assert_eq!(resolved.info().path.as_str(), ".stats/node"); } @@ -4912,8 +4950,8 @@ mod tests { let server = producer.dynamic(".svc", Route::default()).unwrap(); let project = mounted(&producer, "p1", &["**"]); - let through = project.request_broadcast(".svc/foo"); - let direct = producer.consume().request_broadcast(".svc/p1/foo"); + let through = project.request_broadcast(".svc/foo", None); + let direct = producer.consume().request_broadcast(".svc/p1/foo", None); let request = queued(&server).await; assert_eq!(request.path().as_str(), ".svc/p1/foo"); @@ -4966,7 +5004,7 @@ mod tests { announced.assert_next_wait(); // The shadowed path never resolves: the mount answers for it. - let err = project.request_broadcast(".svc/forged").await.err().unwrap(); + let err = project.request_broadcast(".svc/forged", None).await.err().unwrap(); assert!(matches!(err, Error::NotFound | Error::Unroutable), "{err:?}"); } @@ -4979,9 +5017,9 @@ mod tests { let _bar = producer.publish(".svc/p1/bar", Route::default()).unwrap(); let granted = mounted(&producer, "p1", &["foo", ".svc/foo"]); - granted.request_broadcast(".svc/foo").await.expect("granted"); + granted.request_broadcast(".svc/foo", None).await.expect("granted"); let refused = granted - .request_broadcast(".svc/bar") + .request_broadcast(".svc/bar", None) .now_or_never() .expect("refused at once"); assert!(matches!(refused, Err(Error::Unauthorized))); @@ -4991,7 +5029,7 @@ mod tests { let narrower = mounted(&producer, "p1", &["foo"]); let refused = narrower - .request_broadcast(".svc/foo") + .request_broadcast(".svc/foo", None) .now_or_never() .expect("refused at once"); assert!(matches!(refused, Err(Error::Unauthorized))); @@ -5000,7 +5038,7 @@ mod tests { // Another project's mount reaches its own target, never this one's. let other = mounted(&producer, "p2", &["**"]); other.clone().with_hidden(true).announced().assert_next_wait(); - let err = other.request_broadcast(".svc/foo").await.err().unwrap(); + let err = other.request_broadcast(".svc/foo", None).await.err().unwrap(); assert!(matches!(err, Error::Unroutable)); } @@ -5046,7 +5084,7 @@ mod tests { announced.assert_next_wait(); // Beneath the mount the target has no room: refused, never handed to a route. - let err = project.request_broadcast(".svc/x").await.err().unwrap(); + let err = project.request_broadcast(".svc/x", None).await.err().unwrap(); assert!(matches!(err, Error::BoundsExceeded(_)), "{err:?}"); let inside = project.scope(".svc/x", &Patterns::from(Pattern::all())).unwrap(); inside.announced().assert_next_wait(); @@ -5152,7 +5190,7 @@ mod tests { let mut announced = project.announced(); announced.assert_next_active(".svc/foo"); - project.request_broadcast(".svc/foo").await.expect("resolves"); + project.request_broadcast(".svc/foo", None).await.expect("resolves"); let mut report = stats::Report::default(); registry.report(&mut report); @@ -5217,7 +5255,7 @@ mod tests { peer.assert_next_ended("room/alice"); broadcast.unannounce(); announced.assert_next_wait(); - let err = consumer.request_broadcast("room/alice").await.err().unwrap(); + let err = consumer.request_broadcast("room/alice", None).await.err().unwrap(); assert!(matches!(err, Error::Unroutable)); // Back on the air, then the end of the broadcast retracts for good. @@ -5538,7 +5576,7 @@ mod tests { .dynamic("room", Route::default().with_hops(hops(&[10])).with_cost(5)) .unwrap(); - let _pending = consumer.request_broadcast("room/alice"); + let _pending = consumer.request_broadcast("room/alice", None); let request = queued(&identified).await; assert_eq!(request.path().as_str(), "room/alice"); assert!( @@ -5701,7 +5739,7 @@ mod tests { assert_eq!(route.hops, hops(&[10])); // The request goes upstream rather than to the local broadcast. - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let request = queued(&server).await; let upstream = broadcast::Info::new().produce(); request.accept(&upstream); @@ -5720,9 +5758,9 @@ mod tests { .unwrap(); let _plain = producer.publish("room/bob", Route::default()).unwrap(); - let alice = consumer.request_broadcast("room/alice").await.unwrap(); + let alice = consumer.request_broadcast("room/alice", None).await.unwrap(); assert_eq!(alice.info().epoch, Some(epoch())); - let bob = consumer.request_broadcast("room/bob").await.unwrap(); + let bob = consumer.request_broadcast("room/bob", None).await.unwrap(); assert_eq!(bob.info().epoch, None); } @@ -5737,7 +5775,7 @@ mod tests { .publish("room/alice", Route::default().with_epoch(epoch()).with_cost(5)) .unwrap(); let first = consumer - .request_broadcast("room/alice") + .request_broadcast("room/alice", None) .await .expect("resolves locally"); @@ -5751,7 +5789,7 @@ mod tests { let upstream = broadcast::Info::new().produce(); request.accept(&upstream); - let second = consumer.request_broadcast("room/alice").await.expect("resolves"); + let second = consumer.request_broadcast("room/alice", None).await.expect("resolves"); assert!(first.is_clone(&second), "the newcomer joins the front"); } @@ -5767,7 +5805,7 @@ mod tests { let server = producer.dynamic("room/alice", Route::default()).unwrap(); // Were the claim to win, the request would park on its handler forever. - let resolved = moq_net_sim::timeout(Duration::from_secs(1), consumer.request_broadcast("room/alice")) + let resolved = moq_net_sim::timeout(Duration::from_secs(1), consumer.request_broadcast("room/alice", None)) .await .expect("the newer hopless route won the tie") .expect("resolves"); @@ -5829,7 +5867,7 @@ mod tests { .unwrap(); assert!(announced.assert_next_active("room/alice").hops.is_empty()); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); assert_eq!(resolved.info().path.as_str(), "room/alice"); for _ in 0..20 { moq_net_sim::yield_now().await; @@ -5845,17 +5883,20 @@ mod tests { let consumer = producer.consume(); let broadcast = producer.publish("room/alice", Route::default()).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); broadcast.unannounce(); - let err = consumer.request_broadcast("room/alice").await.err().unwrap(); + let err = consumer.request_broadcast("room/alice", None).await.err().unwrap(); assert!(matches!(err, Error::Unroutable), "joined a retracted front: {err}"); settle(|| resolved.is_closed()).await; assert!(!broadcast.consume().is_closed(), "the broadcast itself lives on"); // Announcing again serves a fresh front. broadcast.announce(Route::default()).unwrap(); - let again = consumer.request_broadcast("room/alice").await.expect("resolves again"); + let again = consumer + .request_broadcast("room/alice", None) + .await + .expect("resolves again"); assert!(!again.is_clone(&resolved)); } @@ -5869,7 +5910,7 @@ mod tests { let broadcast = producer.publish("room/alice", Route::default()).unwrap(); let mut dynamic = broadcast.dynamic(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let track = resolved.track("video").unwrap(); let subscribing = moq_net_sim::spawn(async move { track.subscribe(None).await }); let request = moq_net_sim::timeout(Duration::from_secs(1), dynamic.requested_track()) @@ -5900,7 +5941,7 @@ mod tests { let server = producer .dynamic("room/alice", Route::default().with_hops(hops(&[10]))) .unwrap(); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let upstream = broadcast::Info::new().produce(); let dynamic = upstream.dynamic(); queued(&server).await.accept(&upstream); @@ -5998,7 +6039,7 @@ mod tests { let mut front = None; for viewer in 0..1000 { let session = producer.consume().excluding(Hop::random()); - let pending = session.request_broadcast("room/alice"); + let pending = session.request_broadcast("room/alice", None); if viewer == 0 { queued(&server).await.accept(&upstream); } @@ -6031,17 +6072,20 @@ mod tests { let fronts = || producer.shared.lock().fronts.len(); let peer = producer.consume().excluding(origin(7)); - let _local = producer.consume().request_broadcast("room/alice"); - let _peer = peer.request_broadcast("room/alice"); + let _local = producer.consume().request_broadcast("room/alice", None); + let _peer = peer.request_broadcast("room/alice", None); assert_eq!(fronts(), 1, "a hop no chain names excludes nothing"); // The same publisher, now also reached through the peer. let _echo = producer .dynamic("room", Route::default().with_hops(hops(&[10, 7]))) .unwrap(); - let _filtered = peer.request_broadcast("room/alice"); + let _filtered = peer.request_broadcast("room/alice", None); assert_eq!(fronts(), 2, "the peer still shares the plain front"); - let _viewer = producer.consume().excluding(origin(8)).request_broadcast("room/alice"); + let _viewer = producer + .consume() + .excluding(origin(8)) + .request_broadcast("room/alice", None); assert_eq!(fronts(), 2, "a viewer left the plain front"); } @@ -6058,7 +6102,7 @@ mod tests { .unwrap(); view.assert_next_active("room"); - let pending = peer.request_broadcast("room/alice"); + let pending = peer.request_broadcast("room/alice", None); let source = broadcast::Info::new().produce(); queued(&incumbent).await.accept(&source); let resolved = pending.await.expect("resolves through the incumbent"); @@ -6097,7 +6141,7 @@ mod tests { Route::default().with_epoch(epoch()).with_hops(hops(&[10])).with_cost(5), ) .unwrap(); - let pending = producer.consume().request_broadcast("room/alice"); + let pending = producer.consume().request_broadcast("room/alice", None); let upstream = broadcast::Info::new().produce(); let mut dynamic = upstream.dynamic(); queued(&first_server).await.accept(&upstream); @@ -6256,7 +6300,7 @@ mod tests { let broadcast = producer.publish("room/alice", Route::default()).unwrap(); let mut dynamic = broadcast.dynamic(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let track = resolved.track("video").unwrap(); let subscribing = moq_net_sim::spawn(async move { track.subscribe(None).await }); let request = moq_net_sim::timeout(Duration::from_secs(1), dynamic.requested_track()) @@ -6311,7 +6355,7 @@ mod tests { let consumer = producer.consume(); let broadcast = producer.publish("room/alice", Route::default()).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); broadcast.unannounce(); broadcast.announce(Route::default()).unwrap(); @@ -6319,7 +6363,7 @@ mod tests { moq_net_sim::yield_now().await; } assert!(!resolved.is_closed(), "the front ended across a reannouncement"); - let again = consumer.request_broadcast("room/alice").await.expect("resolves"); + let again = consumer.request_broadcast("room/alice", None).await.expect("resolves"); assert!(again.is_clone(&resolved)); } @@ -6331,7 +6375,7 @@ mod tests { // Created but not announced: nobody can reach it, locally included. let broadcast = producer.create_broadcast("room/alice").unwrap(); let err = consumer - .request_broadcast("room/alice") + .request_broadcast("room/alice", None) .now_or_never() .expect("unroutable is synchronous") .err() @@ -6339,13 +6383,13 @@ mod tests { assert!(matches!(err, Error::Unroutable)); broadcast.announce(Route::default()).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); assert_eq!(resolved.info().path.as_str(), "room/alice"); drop(broadcast); // Nothing covers an unknown path and no handler exists. let err = consumer - .request_broadcast("room/bob") + .request_broadcast("room/bob", None) .now_or_never() .expect("unroutable is synchronous") .err() @@ -6430,13 +6474,13 @@ mod tests { let refused = producer .consume() - .request_broadcast("room/video") + .request_broadcast("room/video", None) .now_or_never() .expect("an out-of-scope request must be refused synchronously"); assert!(matches!(refused, Err(Error::Unroutable))); assert!(dynamic.requested_broadcast().now_or_never().is_none()); - let _pending = producer.consume().request_broadcast("room/chat"); + let _pending = producer.consume().request_broadcast("room/chat", None); let request = queued(&dynamic).await; assert_eq!(request.path().as_str(), "room/chat"); } @@ -6542,7 +6586,7 @@ mod tests { let dynamic = producer.dynamic(&path, Route::default()).expect("max depth is allowed"); announced.assert_next_active(&path); - let _pending = producer.consume().request_broadcast(&path); + let _pending = producer.consume().request_broadcast(&path, None); let request = queued(&dynamic).await; assert_eq!(request.path().as_str(), path); } @@ -6621,7 +6665,7 @@ mod tests { let server = producer.dynamic("room", Route::default()).unwrap(); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let request = queued(&server).await; assert_eq!(request.path().as_str(), "room/alice"); @@ -6633,7 +6677,7 @@ mod tests { assert_eq!(resolved.info().path.as_str(), "room/alice"); // A repeat request shares the served broadcast instead of re-asking. - let again = consumer.request_broadcast("room/alice").await.expect("resolves"); + let again = consumer.request_broadcast("room/alice", None).await.expect("resolves"); assert!(again.is_clone(&resolved)); } @@ -6643,8 +6687,8 @@ mod tests { let consumer = producer.consume(); let server = producer.dynamic("room", Route::default()).unwrap(); - let first = consumer.request_broadcast("room/alice"); - let second = consumer.request_broadcast("room/alice"); + let first = consumer.request_broadcast("room/alice", None); + let second = consumer.request_broadcast("room/alice", None); let request = queued(&server).await; // Only one request reaches the server. @@ -6664,7 +6708,7 @@ mod tests { let consumer = producer.consume(); let server = producer.dynamic("room", Route::default()).unwrap(); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); drop(server); let err = pending.await.err().unwrap(); @@ -6672,7 +6716,7 @@ mod tests { // With the route gone, later requests are unroutable immediately. let err = consumer - .request_broadcast("room/alice") + .request_broadcast("room/alice", None) .now_or_never() .expect("unroutable") .err() @@ -6722,7 +6766,7 @@ mod tests { // The requester's own bytes must not be served back to it. let excluded = producer.consume().excluding(origin(7)); let err = excluded - .request_broadcast("room/alice") + .request_broadcast("room/alice", None) .now_or_never() .expect("unroutable") .err() @@ -6731,7 +6775,7 @@ mod tests { // A clean requester resolves through the route (the request queues). let clean = producer.consume().excluding(origin(8)); - let pending = clean.request_broadcast("room/alice"); + let pending = clean.request_broadcast("room/alice", None); assert!(pending.now_or_never().is_none()); } @@ -6972,7 +7016,7 @@ mod tests { let err = producer .consume() .local() - .request_broadcast("remote/alice") + .request_broadcast("remote/alice", None) .now_or_never() .expect("unroutable") .err() @@ -6981,7 +7025,7 @@ mod tests { assert!( producer .consume() - .request_broadcast("remote/alice") + .request_broadcast("remote/alice", None) .now_or_never() .is_none() ); @@ -7036,7 +7080,7 @@ mod tests { // toward an upstream, while the full view queues it on the core's route. let err = core .consume() - .request_broadcast("core/alice") + .request_broadcast("core/alice", None) .now_or_never() .expect("unroutable") .err() @@ -7045,7 +7089,7 @@ mod tests { assert!( producer .consume() - .request_broadcast("core/alice") + .request_broadcast("core/alice", None) .now_or_never() .is_none() ); @@ -7060,7 +7104,7 @@ mod tests { let consumer = producer.consume(); let server = producer.dynamic("room", Route::default()).unwrap(); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let request = queued(&server).await; request.reject(Error::Unroutable); let err = moq_net_sim::timeout(Duration::from_secs(5), pending) @@ -7071,7 +7115,7 @@ mod tests { assert!(matches!(err, Error::Unroutable)); // The route still stands and serves the next path. - let pending = consumer.request_broadcast("room/bob"); + let pending = consumer.request_broadcast("room/bob", None); let request = queued(&server).await; assert_eq!(request.path().as_str(), "room/bob"); let served = broadcast::Info::new().produce(); @@ -7089,7 +7133,7 @@ mod tests { let winner = producer.dynamic("room", Route::default()).unwrap(); let sibling = producer.dynamic("room", Route::default().with_cost(2)).unwrap(); - let mut pending = Box::pin(consumer.request_broadcast("room/alice")); + let mut pending = Box::pin(consumer.request_broadcast("room/alice", None)); assert!((&mut pending).now_or_never().is_none()); queued(&winner).await.reject(Error::NotFound); let mut result = None; @@ -7114,7 +7158,7 @@ mod tests { let consumer = producer.consume(); let stale = producer.dynamic("room", Route::default()).unwrap(); - let mut pending = Box::pin(consumer.request_broadcast("room/alice")); + let mut pending = Box::pin(consumer.request_broadcast("room/alice", None)); assert!((&mut pending).now_or_never().is_none()); let request = queued(&stale).await; @@ -7140,7 +7184,7 @@ mod tests { let consumer = producer.consume(); let stale = producer.dynamic("room", Route::default()).unwrap(); - let mut pending = Box::pin(consumer.request_broadcast("room/alice")); + let mut pending = Box::pin(consumer.request_broadcast("room/alice", None)); assert!((&mut pending).now_or_never().is_none()); let request = queued(&stale).await; @@ -7250,7 +7294,7 @@ mod tests { std::mem::forget(track); } - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); queued(&server).await.accept(&source); let resolved = pending.await.expect("resolves"); @@ -7283,7 +7327,7 @@ mod tests { let _narrow = producer.announce(".dash", Route::default()).unwrap(); let err = consumer - .request_broadcast(".dash/pid") + .request_broadcast(".dash/pid", None) .now_or_never() .expect("unroutable") .err() @@ -7291,7 +7335,7 @@ mod tests { assert!(matches!(err, Error::Unroutable)); // Everything else still routes to the broad server. - let _pending = consumer.request_broadcast("room/alice"); + let _pending = consumer.request_broadcast("room/alice", None); let request = queued(&broad_server).await; assert_eq!(request.path().as_str(), "room/alice"); } @@ -7305,7 +7349,7 @@ mod tests { // The root claim is advertised like any other prefix. announced.assert_next_active(""); - let pending = consumer.request_broadcast("anything/at/all"); + let pending = consumer.request_broadcast("anything/at/all", None); let request = queued(&dynamic).await; assert_eq!(request.path().as_str(), "anything/at/all"); @@ -7318,7 +7362,7 @@ mod tests { drop(dynamic); announced.assert_next_ended(""); let err = consumer - .request_broadcast("something/else") + .request_broadcast("something/else", None) .now_or_never() .expect("unroutable") .err() @@ -7341,7 +7385,7 @@ mod tests { // also pins that the check is segment-aware rather than textual. for path in ["tenant-b/live", "tenant-a-other/live"] { let refused = scoped - .request_broadcast(path) + .request_broadcast(path, None) .now_or_never() .expect("an out-of-scope request must be refused synchronously, not queued"); assert!(matches!(refused, Err(Error::Unauthorized))); @@ -7416,7 +7460,7 @@ mod tests { announced.assert_next_active("room"); let _server = producer.dynamic("served", Route::default()).unwrap(); - let pending = consumer.request_broadcast("served/path"); + let pending = consumer.request_broadcast("served/path", None); drop(driver); @@ -7429,7 +7473,7 @@ mod tests { assert!(matches!(producer.announce("x", Route::default()), Err(Error::Closed))); assert!(matches!(producer.create_broadcast("x"), Err(Error::Closed))); let err = consumer - .request_broadcast("y") + .request_broadcast("y", None) .now_or_never() .expect("closed") .err() @@ -7463,7 +7507,7 @@ mod tests { .dynamic("room", Route::default().with_epoch(epoch()).with_hops(hops(first))) .unwrap(); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let request = queued(&server).await; let source = broadcast::Info::new().produce(); let track = source.create_track("video", None).unwrap(); @@ -7556,7 +7600,7 @@ mod tests { let first = producer .dynamic("room", Route::default().with_hops(hops(&[10])).with_cost(5)) .unwrap(); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let source = broadcast::Info::new().produce(); let track = source.create_track("video", None).unwrap(); queued(&first).await.accept(&source); @@ -7597,7 +7641,7 @@ mod tests { ) .unwrap(); let old_track = old.create_track("video", None).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); deliver(&old_track, &mut subscription, b"old").await; @@ -7610,7 +7654,7 @@ mod tests { let new_track = new.create_track("video", None).unwrap(); assert!(matches!(next_group(&mut subscription).await, Err(Error::Unroutable))); - let replaced = consumer.request_broadcast("room/alice").await.expect("resolves"); + let replaced = consumer.request_broadcast("room/alice", None).await.expect("resolves"); assert!(!replaced.is_clone(&resolved)); let mut subscription = replaced.track("video").unwrap().subscribe(None).await.unwrap(); deliver(&new_track, &mut subscription, b"new").await; @@ -7623,19 +7667,25 @@ mod tests { async fn a_request_racing_a_newer_epoch_skips_the_old_front() { let producer = origin(1).produce(); let consumer = producer.consume(); + let first = crate::Epoch::mint(); let old = producer - .publish("room/alice", Route::default().with_epoch(crate::Epoch::mint())) + .publish("room/alice", Route::default().with_epoch(first.clone())) .unwrap(); let _old_track = old.create_track("video", None).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); + assert_eq!( + resolved.info().epoch.as_ref(), + Some(&first), + "an unpinned request names what it resolved" + ); let epoch = crate::Epoch::mint(); let new = producer .publish("room/alice", Route::default().with_epoch(epoch.clone())) .unwrap(); let new_track = new.create_track("video", None).unwrap(); - let named = consumer.request(Path::new("room/alice"), Some(&epoch)); - let bare = consumer.request_broadcast("room/alice"); + let named = consumer.request_broadcast("room/alice", epoch.clone()); + let bare = consumer.request_broadcast("room/alice", None); let named = named.await.expect("the new epoch resolves"); let bare = bare.await.expect("the new epoch resolves"); @@ -7645,6 +7695,11 @@ mod tests { ); assert!(!bare.is_clone(&resolved), "a bare request joined the superseded front"); assert!(named.is_clone(&bare), "both share the new front"); + assert_eq!( + bare.info().epoch.as_ref(), + Some(&epoch), + "an unpinned request names the new epoch" + ); let mut subscription = named.track("video").unwrap().subscribe(None).await.unwrap(); deliver(&new_track, &mut subscription, b"new").await; } @@ -7658,7 +7713,7 @@ mod tests { let old = producer .dynamic("room", Route::default().with_epoch(crate::Epoch::mint())) .unwrap(); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let request = queued(&old).await; // The answer and the newer epoch land in the same tick. @@ -7675,6 +7730,69 @@ mod tests { deliver(&new_track, &mut subscription, b"new").await; } + /// A handler asked under one publisher instance does not answer for the next: the + /// route's epoch changing while the request waits refuses it, and its late answer + /// is neither handed out under the new epoch nor cached for the next request. + #[moq_net_sim::test] + async fn a_pending_answer_does_not_outlive_its_epoch() { + let producer = origin(1).produce(); + let consumer = producer.consume(); + let dynamic = producer + .dynamic("room", Route::default().with_epoch(crate::Epoch::mint())) + .unwrap(); + let pending = consumer.request_broadcast("room/alice", None); + let request = queued(&dynamic).await; + + let epoch = crate::Epoch::mint(); + dynamic.update(dynamic.route().with_epoch(epoch.clone())).unwrap(); + let stale = broadcast::Info::new().produce(); + let _stale_track = stale.create_track("video", None).unwrap(); + request.accept(&stale); + assert!( + matches!(pending.await, Err(Error::Unroutable)), + "an answer for the old instance resolved" + ); + + let retry = consumer.request_broadcast("room/alice", None); + let request = queued(&dynamic).await; + let fresh = broadcast::Info::new().produce(); + let fresh_track = fresh.create_track("video", None).unwrap(); + request.accept(&fresh); + let resolved = retry.await.expect("the new instance resolves"); + assert_eq!(resolved.info().epoch.as_ref(), Some(&epoch)); + let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); + deliver(&fresh_track, &mut subscription, b"new").await; + } + + /// An answer that lands just before its route changes instance, but that the front + /// has not taken yet, is not stamped with the new epoch: the front asks again. + #[moq_net_sim::test] + async fn an_answer_before_an_epoch_change_is_asked_again() { + let producer = origin(1).produce(); + let consumer = producer.consume(); + let dynamic = producer + .dynamic("room", Route::default().with_epoch(crate::Epoch::mint())) + .unwrap(); + let pending = consumer.request_broadcast("room/alice", None); + let request = queued(&dynamic).await; + + // The answer and the change land in the same tick. + let stale = broadcast::Info::new().produce(); + let _stale_track = stale.create_track("video", None).unwrap(); + request.accept(&stale); + let epoch = crate::Epoch::mint(); + dynamic.update(dynamic.route().with_epoch(epoch.clone())).unwrap(); + + let request = queued(&dynamic).await; + let fresh = broadcast::Info::new().produce(); + let fresh_track = fresh.create_track("video", None).unwrap(); + request.accept(&fresh); + let resolved = pending.await.expect("the new instance resolves"); + assert_eq!(resolved.info().epoch.as_ref(), Some(&epoch)); + let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); + deliver(&fresh_track, &mut subscription, b"new").await; + } + /// Announce cursors follow the newest epoch over a cheaper one, and deliver a /// change of epoch as an end and a fresh start: a different broadcast. #[moq_net_sim::test] @@ -7715,12 +7833,26 @@ mod tests { .unwrap(); let other: crate::Epoch = "01900000-0000-7000-8000-000000000002".parse().unwrap(); - let refused = consumer.request(Path::new("room/alice"), Some(&other)).await; + let refused = consumer.request_broadcast("room/alice", other).await; assert!(matches!(refused, Err(Error::Unroutable))); - consumer - .request(Path::new("room/alice"), Some(&epoch())) + let named = consumer + .request_broadcast("room/alice", epoch()) .await .expect("the named epoch resolves"); + assert_eq!(named.info().epoch.as_ref(), Some(&epoch())); + } + + /// A route without an epoch resolves a handle without one. + #[moq_net_sim::test] + async fn a_route_without_an_epoch_resolves_without_one() { + let producer = origin(1).produce(); + let _broadcast = producer.publish("room/alice", Route::default()).unwrap(); + let resolved = producer + .consume() + .request_broadcast("room/alice", None) + .await + .expect("resolves"); + assert_eq!(resolved.info().epoch, None); } /// The driver's completion contract: it resolves once every producer handle @@ -7818,7 +7950,12 @@ mod tests { let frame = group.read_frame().await.expect("read frame").expect("frame"); assert_eq!(&frame.payload[..], b"after", "first hop {first:?}"); - let resolved = rig.producer.consume().request_broadcast("room/alice").await.unwrap(); + let resolved = rig + .producer + .consume() + .request_broadcast("room/alice", None) + .await + .unwrap(); assert!(resolved.is_clone(&rig.resolved), "first hop {first:?} left the front"); assert!( standby.poll_requested_broadcast(&kio::Waiter::noop()).is_pending(), @@ -7846,7 +7983,7 @@ mod tests { .dynamic("room", Route::default().with_epoch(epoch()).with_hops(hops(&[10, 7]))) .unwrap(); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let request = queued(&incumbent).await; let source = broadcast::Info::new().produce(); let track = source.create_track("video", None).unwrap(); @@ -7911,7 +8048,7 @@ mod tests { // Session C: a third anonymous session, excluding the hop the server // minted for it, the same split-horizon a live session applies. let consumer = producer.consume().excluding(origin(30)); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let request = queued(&server_a).await; let source_a = broadcast::Info::new().produce(); let track_a = source_a.create_track("video", None).unwrap(); @@ -7953,7 +8090,7 @@ mod tests { settle(|| consumer.get_broadcast("room/alice").is_none()).await; settle(|| { matches!( - consumer.request_broadcast("room/alice").now_or_never(), + consumer.request_broadcast("room/alice", None).now_or_never(), Some(Err(Error::Unroutable)) ) }) @@ -7963,7 +8100,7 @@ mod tests { let server_b = producer .dynamic("room", Route::default().with_hops(hops(&[20]))) .unwrap(); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let request = queued(&server_b).await; let source_b = broadcast::Info::new().produce(); let track_b = source_b.create_track("video", None).unwrap(); @@ -8056,11 +8193,11 @@ mod tests { let consumer = producer.consume(); let first = producer.publish("room/alice", Route::default()).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); // A second source at the same path joins the same front. let second = producer.publish("room/alice", Route::default()).unwrap(); - let again = consumer.request_broadcast("room/alice").await.expect("resolves"); + let again = consumer.request_broadcast("room/alice", None).await.expect("resolves"); assert!(again.is_clone(&resolved)); // Losing one source keeps the front alive; losing both closes it. @@ -8089,7 +8226,7 @@ mod tests { let broadcast = producer.publish("room/alice", Route::default()).unwrap(); let track = broadcast.create_track("video", None).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let mut subscription = resolved .track("video") .unwrap() @@ -8131,7 +8268,7 @@ mod tests { .dynamic("room", Route::default().with_hops(hops(&[10]))) .unwrap(); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let request = queued(&server).await; let source = broadcast::Info::new().produce(); let track = source.create_track("video", None).unwrap(); @@ -8177,7 +8314,7 @@ mod tests { let server = producer .dynamic("room", Route::default().with_hops(hops(&[10]))) .unwrap(); - let pending = producer.consume().request_broadcast("room/alice"); + let pending = producer.consume().request_broadcast("room/alice", None); let source = broadcast::Info::new().produce(); queued(&server).await.accept(&source); let resolved = pending.await.unwrap(); @@ -8205,7 +8342,7 @@ mod tests { group.write_frame(crate::Timestamp::ZERO, b"cached".as_ref()).unwrap(); group.finish().unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let mut subscription = resolved .track("video") .unwrap() @@ -8262,7 +8399,7 @@ mod tests { let server = producer .dynamic("live", Route::default().with_hops(hops(&[10]))) .unwrap(); - let pending = producer.consume().request_broadcast("live"); + let pending = producer.consume().request_broadcast("live", None); let source = broadcast::Info::new().produce(); let track = source.create_track("video", None).unwrap(); queued(&server).await.accept(&source); @@ -8301,7 +8438,7 @@ mod tests { let producer = origin(1).produce(); let broadcast = producer.publish("live", Route::default()).unwrap(); let track = broadcast.create_track("video", None).unwrap(); - let resolved = producer.consume().request_broadcast("live").await.unwrap(); + let resolved = producer.consume().request_broadcast("live", None).await.unwrap(); let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); let mut aborted = track.append_group().unwrap(); @@ -8340,17 +8477,20 @@ mod tests { group.finish().unwrap(); // The leaf's front view, which the next front serves from. - let leaf_front = leaf_consumer.request_broadcast("room/alice").await.expect("resolves"); + let leaf_front = leaf_consumer + .request_broadcast("room/alice", None) + .await + .expect("resolves"); let mid = origin(2).produce(); let mid_server = mid.dynamic("room", Route::default().with_hops(hops(&[10]))).unwrap(); - let mid_pending = mid.consume().request_broadcast("room/alice"); + let mid_pending = mid.consume().request_broadcast("room/alice", None); queued(&mid_server).await.accept(&leaf_front); let mid_resolved = mid_pending.await.expect("mid resolves"); let edge = origin(3).produce(); let edge_server = edge.dynamic("room", Route::default().with_hops(hops(&[20]))).unwrap(); - let edge_pending = edge.consume().request_broadcast("room/alice"); + let edge_pending = edge.consume().request_broadcast("room/alice", None); queued(&edge_server).await.accept(&mid_resolved); let edge_resolved = edge_pending.await.expect("edge resolves"); @@ -8428,7 +8568,7 @@ mod tests { .publish("room/alice", Route::default().with_epoch(epoch())) .unwrap(); let track = first.create_track("video", None).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let mut subscription = resolved .track("video") .unwrap() @@ -8478,7 +8618,7 @@ mod tests { let server = producer .dynamic("room", Route::default().with_epoch(epoch()).with_hops(hops(&[10]))) .unwrap(); - let pending = producer.consume().request_broadcast("room/alice"); + let pending = producer.consume().request_broadcast("room/alice", None); let upstream = broadcast::Info::new().produce(); let old = upstream.create_track("video", None).unwrap(); queued(&server).await.accept(&upstream); @@ -8521,7 +8661,7 @@ mod tests { #[moq_net_sim::test] async fn a_returning_reader_is_not_stranded_on_a_dropped_copy() { let (producer, _server, upstream, old, mut subscription) = remote_front().await; - let resolved = producer.consume().request_broadcast("room/alice").await.unwrap(); + let resolved = producer.consume().request_broadcast("room/alice", None).await.unwrap(); // No handler and no track: refuses "video" with NotFound. let _second = producer @@ -8582,7 +8722,7 @@ mod tests { .publish("room/alice", Route::default().with_epoch(epoch())) .unwrap(); let track = first.create_track("video", None).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); let mut group = track.append_group().unwrap(); group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); @@ -8618,7 +8758,7 @@ mod tests { .publish("room/alice", Route::default().with_epoch(epoch())) .unwrap(); let track = first.create_track("video", None).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); let mut group = track.append_group().unwrap(); group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); @@ -8673,7 +8813,7 @@ mod tests { .publish("room/alice", Route::default().with_epoch(epoch())) .unwrap(); let track = first.create_track("video", None).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); let mut group = track.append_group().unwrap(); group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); @@ -8806,7 +8946,7 @@ mod tests { let server = producer.dynamic("room", Route::default()).unwrap(); drop(producer); - let pending = consumer.request_broadcast("room/alice"); + let pending = consumer.request_broadcast("room/alice", None); let request = queued(&server).await; let source = broadcast::Info::new().produce(); request.accept(&source); @@ -8831,7 +8971,7 @@ mod tests { let consumer = producer.consume(); let broadcast = producer.publish("room/alice", Route::default()).unwrap(); let track = broadcast.create_track("video", None).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let sub = resolved.track("video").unwrap().subscribe(None).await.unwrap(); broadcast.unannounce(); @@ -8855,7 +8995,7 @@ mod tests { let consumer = producer.consume(); let broadcast = producer.publish("room/alice", Route::default()).unwrap(); let mut track = broadcast.create_track("video", None).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let sub = resolved.track("video").unwrap().subscribe(None).await.unwrap(); let pending = resolved.track("video").unwrap().subscribe(None); @@ -8882,7 +9022,7 @@ mod tests { let consumer = producer.consume(); let broadcast = producer.publish("room/alice", Route::default()).unwrap(); let track = broadcast.create_track("video", None).unwrap(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let resolved = consumer.request_broadcast("room/alice", None).await.expect("resolves"); let sub = resolved.track("video").unwrap().subscribe(None).await.unwrap(); moq_net_sim::timeout(Duration::from_secs(1), track.demand().used()) .await @@ -8952,7 +9092,7 @@ mod tests { // dropped on the spot, taking its `Watch` with it: the request must not // still be holding the table lock the watch unregisters under. drop(producer); - let _pending = consumer.request_broadcast("room/a"); + let _pending = consumer.request_broadcast("room/a", None); } #[test] diff --git a/rs/moq-net/tests/announce_to_serve.rs b/rs/moq-net/tests/announce_to_serve.rs index 395aa5a0097..02a2ad23c42 100644 --- a/rs/moq-net/tests/announce_to_serve.rs +++ b/rs/moq-net/tests/announce_to_serve.rs @@ -53,7 +53,7 @@ fn drain(announced: &mut moq_net::announce::Consumer) -> Vec { /// How a request for the path answers. async fn request(consumer: &origin::Consumer) -> Result { - consumer.request_broadcast(PATH).await.map_err(|err| match err { + consumer.request_broadcast(PATH, None).await.map_err(|err| match err { Error::Unroutable => "unroutable".to_string(), err => err.to_string(), }) diff --git a/rs/moq-net/tests/broadcast_close.rs b/rs/moq-net/tests/broadcast_close.rs index a0440bb155a..bf0700b0701 100644 --- a/rs/moq-net/tests/broadcast_close.rs +++ b/rs/moq-net/tests/broadcast_close.rs @@ -29,7 +29,7 @@ async fn close_then_lookup(publisher: moq_net::origin::Producer, reader: moq_net .await .expect("announce timeout") .expect("routed"); - let handle = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast")) + let handle = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast", None)) .await .expect("resolve timeout") .expect("broadcast resolves"); @@ -43,7 +43,7 @@ async fn close_then_lookup(publisher: moq_net::origin::Producer, reader: moq_net assert!(matches!(handle.track("video"), Err(Error::Unroutable))); assert!(matches!(handle.track("audio"), Err(Error::Unroutable))); assert!(matches!( - consumer.request_broadcast("bcast").await, + consumer.request_broadcast("bcast", None).await, Err(Error::Unroutable) )); assert!(matches!(broadcast.announce(Default::default()), Err(Error::Closed))); diff --git a/rs/moq-net/tests/datagram.rs b/rs/moq-net/tests/datagram.rs index 4f7c3e45b4d..ffcddd02130 100644 --- a/rs/moq-net/tests/datagram.rs +++ b/rs/moq-net/tests/datagram.rs @@ -48,7 +48,7 @@ async fn connect_datagram_track() -> Fixture { let consumer = consumer_origin.consume(); consumer.routed("bench").await.unwrap(); - let remote = consumer.request_broadcast("bench").await.unwrap(); + let remote = consumer.request_broadcast("bench", None).await.unwrap(); let subscriber = remote.track("datagrams").unwrap().subscribe(None).await.unwrap(); Fixture { @@ -140,7 +140,7 @@ async fn ietf_delivers_datagrams_on(version: &str) { let consumer = consumer_origin.consume(); consumer.routed("bench").await.unwrap(); - let remote = consumer.request_broadcast("bench").await.unwrap(); + let remote = consumer.request_broadcast("bench", None).await.unwrap(); let mut subscriber = remote.track("datagrams").unwrap().subscribe(None).await.unwrap(); // A group first, so the subscription's alias is bound before any datagram lands. @@ -251,7 +251,7 @@ async fn a_fetched_datagram_is_refused_per_version() { let consumer = nodes.last().unwrap().consume(); consumer.routed("room").await.unwrap(); - let remote = consumer.request_broadcast("room").await.unwrap(); + let remote = consumer.request_broadcast("room", None).await.unwrap(); let err = match remote.track("datagrams").unwrap().fetch_group(sequence, None).await { Ok(_) => panic!("{versions:?}: a datagram was fetched"), Err(err) => err, diff --git a/rs/moq-net/tests/fetch_cancel.rs b/rs/moq-net/tests/fetch_cancel.rs index 3d137daa15b..94801e0a3da 100644 --- a/rs/moq-net/tests/fetch_cancel.rs +++ b/rs/moq-net/tests/fetch_cancel.rs @@ -47,7 +47,7 @@ async fn abandoned_fetch_reaches_the_publisher(version: &str, relays: u64, stage let consumer = nodes[relays as usize] .consume() - .request_broadcast("room") + .request_broadcast("room", None) .await .unwrap(); let track = consumer.track("video").unwrap(); @@ -171,7 +171,7 @@ async fn a_refetch_within_the_linger_stays_on_the_relay(version: &str) { broadcast.announce(Default::default()).unwrap(); moq_net_sim::sleep(Duration::from_secs(1)).await; - let consumer = relay.consume().request_broadcast("room").await.unwrap(); + let consumer = relay.consume().request_broadcast("room", None).await.unwrap(); let mut waiting = Box::pin(consumer.track("video").unwrap().fetch_group(0, None)); let request = match futures::future::select(std::pin::pin!(dynamic.requested_track()), &mut waiting).await { futures::future::Either::Left((request, _)) => request.expect("the track request reaches the publisher"), diff --git a/rs/moq-net/tests/finished_broadcast_mock.rs b/rs/moq-net/tests/finished_broadcast_mock.rs index 31b6de03271..bdf7267968b 100644 --- a/rs/moq-net/tests/finished_broadcast_mock.rs +++ b/rs/moq-net/tests/finished_broadcast_mock.rs @@ -47,7 +47,7 @@ async fn round(finish_broadcast: bool) -> (Vec>, Option) .await .expect("announce timeout") .expect("routed"); - let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast")) + let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast", None)) .await .expect("resolve timeout") .expect("broadcast resolves"); diff --git a/rs/moq-net/tests/goaway.rs b/rs/moq-net/tests/goaway.rs index 2e7ca1ff267..862a9c027b2 100644 --- a/rs/moq-net/tests/goaway.rs +++ b/rs/moq-net/tests/goaway.rs @@ -342,7 +342,7 @@ async fn goaway_gates_new_subscribes_moq_lite_04() { // Subscribe BEFORE the GOAWAY and receive a first group. let sub = sub_origin.consume(); sub.routed("test").await.expect("route announced"); - let bc = sub.request_broadcast("test").await.expect("broadcast resolves"); + let bc = sub.request_broadcast("test", None).await.expect("broadcast resolves"); let mut existing = bc.track("video").unwrap().subscribe(None).await.expect("subscribe"); let mut group = track.append_group().expect("append group"); diff --git a/rs/moq-net/tests/head_fetch.rs b/rs/moq-net/tests/head_fetch.rs index 2b43733c534..2323814fa4b 100644 --- a/rs/moq-net/tests/head_fetch.rs +++ b/rs/moq-net/tests/head_fetch.rs @@ -38,7 +38,7 @@ async fn resolve(origin: &moq_net::origin::Producer) -> moq_net::broadcast::Cons .await .expect("announce timeout") .expect("routed"); - moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast")) + moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast", None)) .await .expect("resolve timeout") .expect("broadcast resolves") diff --git a/rs/moq-net/tests/history_groups.rs b/rs/moq-net/tests/history_groups.rs index 70680e7426d..0cf508dd2c7 100644 --- a/rs/moq-net/tests/history_groups.rs +++ b/rs/moq-net/tests/history_groups.rs @@ -71,7 +71,7 @@ async fn round(version: &str, relay: bool, newest_first: bool) -> Vec<(u64, usiz .await .expect("announce timeout") .expect("routed"); - let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast")) + let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast", None)) .await .expect("resolve timeout") .expect("broadcast resolves"); @@ -217,7 +217,7 @@ async fn a_late_subscriber_receives_the_relays_cached_history() { let first_pair = connect_mock(options).await; let consumer = first.consume(); consumer.routed("bcast").await.unwrap(); - let remote = consumer.request_broadcast("bcast").await.unwrap(); + let remote = consumer.request_broadcast("bcast", None).await.unwrap(); let subscription = Subscription::default() .with_max_delay(FOREVER) .with_start(Position::group(0)); @@ -257,7 +257,7 @@ async fn a_late_subscriber_receives_the_relays_cached_history() { let late_pair = connect_mock(options).await; let consumer = late.consume(); consumer.routed("bcast").await.unwrap(); - let remote = consumer.request_broadcast("bcast").await.unwrap(); + let remote = consumer.request_broadcast("bcast", None).await.unwrap(); let mut late_sub = remote.track("history").unwrap().subscribe(subscription).await.unwrap(); moq_net_sim::timeout(TIMEOUT, read_history(&mut late_sub)) .await diff --git a/rs/moq-net/tests/quiet_catalog.rs b/rs/moq-net/tests/quiet_catalog.rs index 85b34fbc9df..17eb3d8a799 100644 --- a/rs/moq-net/tests/quiet_catalog.rs +++ b/rs/moq-net/tests/quiet_catalog.rs @@ -53,7 +53,7 @@ fn publish(origin: &origin::Producer) -> (broadcast::Producer, track::Producer) async fn request(origin: &origin::Producer) -> broadcast::Consumer { let consumer = origin.consume(); consumer.routed("live").await.unwrap(); - consumer.request_broadcast("live").await.unwrap() + consumer.request_broadcast("live", None).await.unwrap() } /// Subscribe without a floor and expect the snapshot as group 0. diff --git a/rs/moq-net/tests/rejoin.rs b/rs/moq-net/tests/rejoin.rs index 3e5cd391cd7..3bddb071427 100644 --- a/rs/moq-net/tests/rejoin.rs +++ b/rs/moq-net/tests/rejoin.rs @@ -49,7 +49,7 @@ async fn rejoin_recovers_the_group_reset_on_leave() { let consumer = relay.consume(); consumer.routed("bench").await.unwrap(); - let remote = consumer.request_broadcast("bench").await.unwrap(); + let remote = consumer.request_broadcast("bench", None).await.unwrap(); let ts = |ms| Timestamp::from_millis(ms).unwrap(); let prefs = || track::Subscription::default().with_max_delay(Duration::from_secs(10)); @@ -142,7 +142,7 @@ async fn rejoin_skips_a_cache_kept_by_another_handle() { let consumer = relay.consume(); consumer.routed("bench").await.unwrap(); - let remote = consumer.request_broadcast("bench").await.unwrap(); + let remote = consumer.request_broadcast("bench", None).await.unwrap(); let ts = |ms| Timestamp::from_millis(ms).unwrap(); // Held for fetches only: it keeps the relay's copy without subscribing. @@ -214,7 +214,7 @@ async fn rejoin_mid_group_keeps_the_head_for_later_readers() { let consumer = client.consume(); consumer.routed("bench").await.unwrap(); - let remote = consumer.request_broadcast("bench").await.unwrap(); + let remote = consumer.request_broadcast("bench", None).await.unwrap(); let mut sub = remote.track("video").unwrap().subscribe(None).await.unwrap(); let mut group = sub.recv_group().await.unwrap().unwrap(); assert_eq!(group.read_frame().await.unwrap().unwrap().payload, b"a0".as_ref()); @@ -226,7 +226,7 @@ async fn rejoin_mid_group_keeps_the_head_for_later_readers() { assert_eq!(rejoined.read_frame().await.unwrap().unwrap().payload, b"a0".as_ref()); track.demand().used().await.unwrap(); - let later = relay.consume().request_broadcast("bench").await.unwrap(); + let later = relay.consume().request_broadcast("bench", None).await.unwrap(); let mut sub = later.track("video").unwrap().subscribe(None).await.unwrap(); let mut group = sub.recv_group().await.unwrap().unwrap(); assert_eq!(group.sequence, 0, "{version}"); @@ -286,7 +286,7 @@ async fn rejoin_goes_live_without_the_join_head() { let consumer = client.consume(); consumer.routed("bench").await.unwrap(); - let remote = consumer.request_broadcast("bench").await.unwrap(); + let remote = consumer.request_broadcast("bench", None).await.unwrap(); let mut sub = remote.track("video").unwrap().subscribe(None).await.unwrap(); let mut group = sub.recv_group().await.unwrap().unwrap(); assert_eq!(group.read_frame().await.unwrap().unwrap().payload, b"a0".as_ref()); @@ -313,7 +313,7 @@ async fn rejoin_goes_live_without_the_join_head() { .unwrap(); next.finish().unwrap(); - let local = relay.consume().request_broadcast("bench").await.unwrap(); + let local = relay.consume().request_broadcast("bench", None).await.unwrap(); let mut sub = local.track("video").unwrap().subscribe(None).await.unwrap(); let mut group = sub.recv_group().await.unwrap().unwrap(); assert_eq!(group.sequence, 1, "{version}"); @@ -350,7 +350,7 @@ async fn rejoin_during_the_cancel_skips_the_cache() { let consumer = relay.consume(); consumer.routed("bench").await.unwrap(); - let remote = consumer.request_broadcast("bench").await.unwrap(); + let remote = consumer.request_broadcast("bench", None).await.unwrap(); let ts = |ms| Timestamp::from_millis(ms).unwrap(); let mut group = track.append_group().unwrap(); @@ -433,7 +433,7 @@ async fn leaving_after_the_cache_window_keeps_the_latest_group() { let pair = connect_mock(options).await; let consumer = client.consume(); consumer.routed("bench").await.unwrap(); - let remote = consumer.request_broadcast("bench").await.unwrap(); + let remote = consumer.request_broadcast("bench", None).await.unwrap(); (remote, pair) } }; diff --git a/rs/moq-net/tests/route_change.rs b/rs/moq-net/tests/route_change.rs index c78543e2db5..08b9a83eebe 100644 --- a/rs/moq-net/tests/route_change.rs +++ b/rs/moq-net/tests/route_change.rs @@ -109,7 +109,7 @@ impl Topology { let consumer = subscriber.consume(); consumer.routed("live").await.unwrap(); - let remote = consumer.request_broadcast("live").await.unwrap(); + let remote = consumer.request_broadcast("live", None).await.unwrap(); let prefs = track::Subscription::default().with_max_delay(Duration::from_secs(60)); let sub = remote.track("video").unwrap().subscribe(prefs).await.unwrap(); @@ -447,7 +447,7 @@ async fn lagging_route_dies(version: Version) -> mpsc::UnboundedReceiver track::Subscriber { let consumer = origin.consume(); consumer.routed("live").await.unwrap(); - let remote = consumer.request_broadcast("live").await.unwrap(); + let remote = consumer.request_broadcast("live", None).await.unwrap(); let preferences = track::Subscription::default().with_max_delay(Duration::from_secs(60)); remote.track("video").unwrap().subscribe(preferences).await.unwrap() } @@ -585,7 +585,7 @@ async fn redundant_pair_fails_over(version: &str, loss: Loss) { settle().await; let consumer = subscriber.consume(); - let remote = consumer.request_broadcast("live").await.unwrap(); + let remote = consumer.request_broadcast("live", None).await.unwrap(); let prefs = track::Subscription::default().with_max_delay(Duration::from_secs(60)); let mut rx = read(remote.track("video").unwrap().subscribe(prefs).await.unwrap()); diff --git a/rs/moq-net/tests/session_close.rs b/rs/moq-net/tests/session_close.rs index 5795737038e..31d70bc6a8f 100644 --- a/rs/moq-net/tests/session_close.rs +++ b/rs/moq-net/tests/session_close.rs @@ -46,7 +46,7 @@ async fn setup(version: &str) -> Setup { .await .expect("announce timeout") .expect("routed"); - let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast")) + let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast", None)) .await .expect("resolve timeout") .expect("broadcast resolves"); diff --git a/rs/moq-net/tests/subscribe_tracks.rs b/rs/moq-net/tests/subscribe_tracks.rs index 3104018edd3..af34c42403e 100644 --- a/rs/moq-net/tests/subscribe_tracks.rs +++ b/rs/moq-net/tests/subscribe_tracks.rs @@ -57,7 +57,7 @@ async fn setup(version: Version) -> Setup { .await .expect("announce timeout") .expect("routed"); - let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("room")) + let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("room", None)) .await .expect("resolve timeout") .expect("broadcast resolves"); diff --git a/rs/moq-net/tests/subscription_end_integrity.rs b/rs/moq-net/tests/subscription_end_integrity.rs index 34d4f78636b..401c75a6d9c 100644 --- a/rs/moq-net/tests/subscription_end_integrity.rs +++ b/rs/moq-net/tests/subscription_end_integrity.rs @@ -64,7 +64,7 @@ async fn round(drop_session: bool) -> (Vec>, Option) { .await .expect("announce timeout") .expect("routed"); - let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast")) + let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast", None)) .await .expect("resolve timeout") .expect("broadcast resolves"); @@ -211,7 +211,10 @@ async fn killed(version: &str, local: bool) -> (Option, Option Pair { .await .expect("announce timeout") .expect("routed"); - let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast")) + let remote = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("bcast", None)) .await .expect("resolve timeout") .expect("broadcast resolves"); diff --git a/rs/moq-net/tests/unannounce_release.rs b/rs/moq-net/tests/unannounce_release.rs index 30f786d9132..a3e0f15269b 100644 --- a/rs/moq-net/tests/unannounce_release.rs +++ b/rs/moq-net/tests/unannounce_release.rs @@ -27,7 +27,7 @@ async fn release(version: Version) { let consumer = relay.consume(); consumer.routed("a/b").await.expect("routed"); - let remote = consumer.request_broadcast("a/b").await.expect("resolves"); + let remote = consumer.request_broadcast("a/b", None).await.expect("resolves"); let sub = remote.track("t").unwrap().subscribe(None).await.expect("subscribes"); track.demand().used().await.unwrap(); diff --git a/rs/moq-net/tests/untimed.rs b/rs/moq-net/tests/untimed.rs index c70bc74ff81..926ad201105 100644 --- a/rs/moq-net/tests/untimed.rs +++ b/rs/moq-net/tests/untimed.rs @@ -80,7 +80,7 @@ impl Chain { async fn subscribe(&self, track: &str) -> track::Subscriber { let consumer = self.nodes.last().unwrap().consume(); - let broadcast = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("room")) + let broadcast = moq_net_sim::timeout(TIMEOUT, consumer.request_broadcast("room", None)) .await .expect("resolve timeout") .expect("broadcast resolves"); @@ -180,7 +180,7 @@ async fn an_explicit_start_holds_on_an_untimed_track() { } moq_net_sim::sleep(Duration::from_secs(1)).await; - let consumer = chain.nodes[1].consume().request_broadcast("room").await.unwrap(); + let consumer = chain.nodes[1].consume().request_broadcast("room", None).await.unwrap(); let subscription = track::Subscription::default() .with_start(track::Position::group(2)) .with_max_delay(Duration::from_secs(30)); @@ -238,7 +238,7 @@ async fn a_fetched_untimed_frame_arrives_untimed() { broadcast.announce(Default::default()).unwrap(); moq_net_sim::sleep(Duration::from_secs(1)).await; - let consumer = chain.nodes[1].consume().request_broadcast("room").await.unwrap(); + let consumer = chain.nodes[1].consume().request_broadcast("room", None).await.unwrap(); let mut waiting = Box::pin(consumer.track("video").unwrap().fetch_group(0, None)); let request = match futures::future::select(std::pin::pin!(dynamic.requested_track()), &mut waiting).await { futures::future::Either::Left((request, _)) => request.expect("the track request reaches the publisher"), diff --git a/rs/moq-net/tests/wide_sequence.rs b/rs/moq-net/tests/wide_sequence.rs index 6fa8898a360..536b8c39a74 100644 --- a/rs/moq-net/tests/wide_sequence.rs +++ b/rs/moq-net/tests/wide_sequence.rs @@ -43,13 +43,13 @@ async fn wide_sequence_fails_only_its_subscription() { // lite-07 carries the full 64 bits, so the relay holds the group. relay.consume().routed("bench").await.unwrap(); - let cached = relay.consume().request_broadcast("bench").await.unwrap(); + let cached = relay.consume().request_broadcast("bench", None).await.unwrap(); let mut sub = cached.track("wide").unwrap().subscribe(None).await.unwrap(); assert_eq!(sub.recv_group().await.unwrap().unwrap().sequence, WIDE); let consumer = client.consume(); consumer.routed("bench").await.unwrap(); - let remote = consumer.request_broadcast("bench").await.unwrap(); + let remote = consumer.request_broadcast("bench", None).await.unwrap(); let result = moq_net_sim::timeout(Duration::from_secs(5), async { let mut sub = remote.track("wide").unwrap().subscribe(None).await?; diff --git a/rs/moq-relay/src/cluster.rs b/rs/moq-relay/src/cluster.rs index 91b1c4e6d3c..6161b25b883 100644 --- a/rs/moq-relay/src/cluster.rs +++ b/rs/moq-relay/src/cluster.rs @@ -1980,7 +1980,7 @@ mod tests { .expect("publish at the fleet path"); let subscriber = cluster.subscriber(&token).expect("subscribe grant").consume(); let broadcast = subscriber - .request_broadcast(".svc/foo") + .request_broadcast(".svc/foo", None) .await .expect("resolves through the mount"); assert_eq!(broadcast.info().path.as_str(), ".svc/foo"); @@ -2040,12 +2040,13 @@ mod tests { let path = moq_net::Path::new(".stats").join("node").join("test"); let consumer = cluster.origin.consume(); - tokio::time::timeout(std::time::Duration::from_secs(5), consumer.routed(&path)) + let route = tokio::time::timeout(std::time::Duration::from_secs(5), consumer.routed(&path)) .await .expect("stats broadcast announced within 5s") .expect("stats broadcast present"); + assert!(route.epoch.is_some(), "stats announce under an epoch"); let broadcast = consumer - .request_broadcast(&path) + .request_broadcast(&path, None) .await .expect("stats broadcast resolves"); @@ -2607,7 +2608,7 @@ mod tests { assert!(pool.used() > 0, "writes charge the constructed cache pool"); let consumer = cluster.origin.consume(); - tokio::time::timeout(Duration::from_secs(2), consumer.request_broadcast("cam")) + tokio::time::timeout(Duration::from_secs(2), consumer.request_broadcast("cam", None)) .await .expect("broadcast resolves") .expect("broadcast present"); diff --git a/rs/moq-relay/src/internal.rs b/rs/moq-relay/src/internal.rs index da6f201d32d..a76038a2a4d 100644 --- a/rs/moq-relay/src/internal.rs +++ b/rs/moq-relay/src/internal.rs @@ -1018,7 +1018,7 @@ mod tests { let (update, active) = next_update(&mut announced).await.unwrap(); assert!(active); let bc = egress - .request_broadcast(moq_net::Path::new(update.prefix.as_str())) + .request_broadcast(moq_net::Path::new(update.prefix.as_str()), None) .await .unwrap(); let mut egress_sub = bc.track("video").unwrap().subscribe(None).await.unwrap(); diff --git a/rs/moq-relay/src/stats.rs b/rs/moq-relay/src/stats.rs index 12fa7084239..6856d46e121 100644 --- a/rs/moq-relay/src/stats.rs +++ b/rs/moq-relay/src/stats.rs @@ -15,7 +15,8 @@ use serde::{Deserialize, Serialize}; /// Set `enabled = true` to attach a [`moq_stats::Producer`] to every session /// the relay accepts (and every cluster dial). The producer publishes a single /// `/node/` broadcast (or `/node` when [`Self::node`] is -/// unset) on the cluster origin. Each broadcast carries plain `.json` tracks +/// unset) on the cluster origin, under a fresh epoch each run. Each broadcast +/// carries plain `.json` tracks /// (a JSON map of broadcast path to a cumulative counter snapshot per frame) /// plus compressed `.json.z` siblings; see `moq_stats` for the wire format and /// per-field semantics. @@ -81,8 +82,9 @@ pub struct Config { /// How long a group's stats broadcast stays announced after the group's /// last session and traffic leave, e.g. "5m" or "30s". Defaults to 5 /// minutes. A group that returns within it keeps its broadcast, so viewer - /// churn doesn't unannounce and re-announce it across the mesh. Only applies - /// at `depth` 1 or more. See [`moq_stats::produce::Config::linger`]. + /// churn doesn't unannounce and re-announce it across the mesh; one that + /// returns after it announces under a new epoch. Only applies at `depth` 1 + /// or more. See [`moq_stats::produce::Config::linger`]. #[usage(skip)] #[serde(with = "crate::duration::serde_option")] pub linger: Option, diff --git a/rs/moq-relay/tests/auth_lifetime.rs b/rs/moq-relay/tests/auth_lifetime.rs index b78bd2cd4fc..1f9817c6593 100644 --- a/rs/moq-relay/tests/auth_lifetime.rs +++ b/rs/moq-relay/tests/auth_lifetime.rs @@ -252,7 +252,7 @@ async fn connect_and_round_trip(url: &url::Url) -> (moq_tokio::Connection, moq_t assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); let bc = sub_consumer - .request_broadcast("test") + .request_broadcast("test", None) .await .expect("announced broadcast resolves"); @@ -484,7 +484,7 @@ async fn a_moved_tier_retags_the_live_session() { assert_eq!(update.prefix.as_str(), "test"); assert!(announced, "expected announce, got retraction"); let bc = sub_consumer - .request_broadcast("test") + .request_broadcast("test", None) .await .expect("announced broadcast resolves"); let mut track_sub = bc.track("video").unwrap().subscribe(None).await.expect("subscribe"); diff --git a/rs/moq-relay/tests/cluster_unknown.rs b/rs/moq-relay/tests/cluster_unknown.rs index bf57b60b630..4871610de36 100644 --- a/rs/moq-relay/tests/cluster_unknown.rs +++ b/rs/moq-relay/tests/cluster_unknown.rs @@ -130,7 +130,7 @@ async fn read_first_frame(port: u16) -> Result, String> { .map_err(|_| "routed timed out".to_string())? .ok_or_else(|| "origin closed before the broadcast was announced".to_string())?; let broadcast = consumer - .request_broadcast(PATH) + .request_broadcast(PATH, None) .await .map_err(|err| format!("broadcast unroutable: {err}"))?; diff --git a/rs/moq-relay/tests/embed.rs b/rs/moq-relay/tests/embed.rs index eb4026f3d23..e9c0cf0f210 100644 --- a/rs/moq-relay/tests/embed.rs +++ b/rs/moq-relay/tests/embed.rs @@ -283,7 +283,7 @@ async fn embed_and_stop(mut config: Config) { .expect("origin closed"); assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); - let announced = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test")) + let announced = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test", None)) .await .expect("request timeout") .expect("announced broadcast resolves"); diff --git a/rs/moq-relay/tests/goaway_cluster.rs b/rs/moq-relay/tests/goaway_cluster.rs index 9433b3de8d1..5dd263a6b7b 100644 --- a/rs/moq-relay/tests/goaway_cluster.rs +++ b/rs/moq-relay/tests/goaway_cluster.rs @@ -158,7 +158,10 @@ async fn cluster_continues_a_group_split_by_goaway_inner() { let session_a = within("A accepts", accepted_a.recv()).await.expect("A accepts"); let consumer = cluster.origin.consume(); within("routed via A", consumer.routed("cam")).await.expect("routed"); - let bc = consumer.request_broadcast("cam").await.expect("broadcast resolves"); + let bc = consumer + .request_broadcast("cam", None) + .await + .expect("broadcast resolves"); let mut sub = within("subscribe", bc.track("video").expect("track").subscribe(None)) .await .expect("subscribe"); @@ -304,7 +307,10 @@ async fn cluster_migrates_on_upstream_goaway_inner() { .routed("cam") .await .expect("broadcast announced through sibling A"); - let bc = consumer.request_broadcast("cam").await.expect("broadcast resolves"); + let bc = consumer + .request_broadcast("cam", None) + .await + .expect("broadcast resolves"); let mut sub = bc .track("video") .expect("track handle") @@ -542,7 +548,7 @@ async fn cluster_diamond_goaway_seamless_failover_inner() { let bc = within("broadcast resolves on the subscriber origin", async { let consumer = sub_origin.consume(); consumer.routed("diamond").await?; - consumer.request_broadcast("diamond").await.ok() + consumer.request_broadcast("diamond", None).await.ok() }) .await .expect("broadcast announced"); @@ -771,7 +777,7 @@ async fn cluster_reconnects_on_empty_uri_goaway_inner() { let bc = within("broadcast announced", async { let consumer = cluster.origin.consume(); consumer.routed("cam").await?; - consumer.request_broadcast("cam").await.ok() + consumer.request_broadcast("cam", None).await.ok() }) .await .expect("broadcast announced"); diff --git a/rs/moq-relay/tests/public_rules.rs b/rs/moq-relay/tests/public_rules.rs index 3287f0fca5d..e9757e8099f 100644 --- a/rs/moq-relay/tests/public_rules.rs +++ b/rs/moq-relay/tests/public_rules.rs @@ -109,7 +109,10 @@ async fn first_broadcast(url: url::Url) -> String { .expect("announcement timeout"); let name = update.prefix.to_string(); - let broadcast = consumer.request_broadcast(&name).await.expect("broadcast resolves"); + let broadcast = consumer + .request_broadcast(&name, None) + .await + .expect("broadcast resolves"); let mut track = broadcast .track("video") .unwrap() diff --git a/rs/moq-relay/tests/runtime_uring.rs b/rs/moq-relay/tests/runtime_uring.rs index 2b013656cdf..51a492afa31 100644 --- a/rs/moq-relay/tests/runtime_uring.rs +++ b/rs/moq-relay/tests/runtime_uring.rs @@ -150,7 +150,7 @@ async fn uring_workers_serve_webtransport_and_raw_quic() { .expect("origin closed"); assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); - let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test")) + let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test", None)) .await .unwrap_or_else(|_| panic!("subscriber {index} request timeout")) .expect("announced broadcast resolves"); @@ -438,7 +438,7 @@ async fn an_mtls_client_authenticates_without_a_token() { .expect("origin closed"); assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); - let announced = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test")) + let announced = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test", None)) .await .expect("request timeout") .expect("announced broadcast resolves"); diff --git a/rs/moq-relay/tests/runtime_workers.rs b/rs/moq-relay/tests/runtime_workers.rs index 5cf7b159973..80bf4b02af9 100644 --- a/rs/moq-relay/tests/runtime_workers.rs +++ b/rs/moq-relay/tests/runtime_workers.rs @@ -110,7 +110,7 @@ async fn workers_serve_quic_and_share_one_origin() { .expect("origin closed"); assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); - let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test")) + let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test", None)) .await .unwrap_or_else(|_| panic!("subscriber {index} request timeout")) .expect("announced broadcast resolves"); diff --git a/rs/moq-relay/tests/smoke.rs b/rs/moq-relay/tests/smoke.rs index fdc5b566fd0..e9e1d943c03 100644 --- a/rs/moq-relay/tests/smoke.rs +++ b/rs/moq-relay/tests/smoke.rs @@ -179,7 +179,7 @@ async fn relay_websocket_round_trip_uses_newest_version() { // Auth root for `/smoke` is "smoke"; the broadcast "test" announces underneath. assert_eq!(path.as_str(), "test"); let bc = sub_consumer - .request_broadcast(&path) + .request_broadcast(&path, None) .await .expect("announced broadcast resolves"); @@ -296,7 +296,7 @@ async fn hidden_broadcasts_need_a_lite07_opt_in() { continue; } let bc = consumer - .request_broadcast(".x/y") + .request_broadcast(".x/y", None) .await .expect("hidden broadcast resolves"); let mut sub = bc.track("video").unwrap().subscribe(None).await.expect("subscribe"); @@ -472,7 +472,7 @@ async fn relay_websocket_root_path_upgrades() { assert!(active, "expected announce, got retraction"); assert_eq!(path.as_str(), "test"); let bc = sub_consumer - .request_broadcast(&path) + .request_broadcast(&path, None) .await .expect("announced broadcast resolves"); @@ -691,7 +691,7 @@ async fn internal_tcp_round_trip() { assert!(active, "expected announce, got retraction"); assert_eq!(path.as_str(), "test"); let bc = sub_consumer - .request_broadcast(&path) + .request_broadcast(&path, None) .await .expect("announced broadcast resolves"); @@ -790,7 +790,7 @@ async fn internal_unix_round_trip() { .expect("origin closed"); assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); - let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timeout") .expect("announced broadcast resolves"); diff --git a/rs/moq-room/src/room.rs b/rs/moq-room/src/room.rs index a0986ff1d94..5b0d7915272 100644 --- a/rs/moq-room/src/room.rs +++ b/rs/moq-room/src/room.rs @@ -99,7 +99,7 @@ impl Room { })); } - let mut request = self.origin.request_broadcast(&path).into_inner(); + let mut request = self.origin.request_broadcast(&path, None).into_inner(); match kio::Task::poll(&mut request, waiter) { Poll::Ready(Ok(broadcast)) => { return Poll::Ready(Some(Event { diff --git a/rs/moq-rtc/src/server/whip.rs b/rs/moq-rtc/src/server/whip.rs index 32c2c919ee5..c5b5080d399 100644 --- a/rs/moq-rtc/src/server/whip.rs +++ b/rs/moq-rtc/src/server/whip.rs @@ -221,7 +221,7 @@ mod tests { let origin = moq_tokio::origin::spawn(); let consumer = origin.consume(); let _stale_session = accept(&server, &origin, "live/cam0", &offer()).await.unwrap(); - let stale = consumer.request_broadcast("live/cam0").await.unwrap(); + let stale = consumer.request_broadcast("live/cam0", None).await.unwrap(); let mut catalog = stale .track(hang::Catalog::DEFAULT_NAME) .unwrap() @@ -243,7 +243,7 @@ mod tests { .await .expect("the stale viewer stalled"); assert!(matches!(ended, moq_net::Error::Unroutable), "{ended:?}"); - let fresh = consumer.request_broadcast("live/cam0").await.unwrap(); + let fresh = consumer.request_broadcast("live/cam0", None).await.unwrap(); assert!(!fresh.is_clone(&stale), "viewers reach the reconnected session"); } } diff --git a/rs/moq-rtmp/src/dial.rs b/rs/moq-rtmp/src/dial.rs index c7c167a739d..7ee9758c801 100644 --- a/rs/moq-rtmp/src/dial.rs +++ b/rs/moq-rtmp/src/dial.rs @@ -604,7 +604,7 @@ mod tests { .await .expect("client republish timed out") .expect("broadcast announced in client origin"); - let broadcast = announced.request_broadcast("pulled/cam0").await.unwrap(); + let broadcast = announced.request_broadcast("pulled/cam0", None).await.unwrap(); // It should carry a hang catalog track (proof the FLV demux produced real // media on the far side): subscribe to it and read one catalog frame. diff --git a/rs/moq-rtmp/src/server.rs b/rs/moq-rtmp/src/server.rs index 21873efe8be..93912fa0275 100644 --- a/rs/moq-rtmp/src/server.rs +++ b/rs/moq-rtmp/src/server.rs @@ -1751,7 +1751,7 @@ mod tests { let consumer = origin.consume(); consumer.routed("live/cam0").await.unwrap(); - let broadcast = consumer.request_broadcast("live/cam0").await.unwrap(); + let broadcast = consumer.request_broadcast("live/cam0", None).await.unwrap(); let info = broadcast.track("0.flv-v").unwrap().query().await.unwrap(); assert_eq!(info.max_age, Some(Duration::from_secs(3))); } @@ -2156,7 +2156,7 @@ mod tests { .await .expect("stale publish timed out") .unwrap(); - let stale = consumer.request_broadcast("live/cam0").await.unwrap(); + let stale = consumer.request_broadcast("live/cam0", None).await.unwrap(); let mut catalog = stale .track(hang::Catalog::DEFAULT_NAME) .unwrap() @@ -2181,7 +2181,7 @@ mod tests { assert!(matches!(ended, moq_net::Error::Unroutable), "{ended:?}"); assert!(!stale_client.is_finished(), "the stale connection is still open"); - let fresh = consumer.request_broadcast("live/cam0").await.unwrap(); + let fresh = consumer.request_broadcast("live/cam0", None).await.unwrap(); assert!(!fresh.is_clone(&stale), "viewers reach the reconnected publish"); stale_client.abort(); diff --git a/rs/moq-srt/src/ts.rs b/rs/moq-srt/src/ts.rs index e8f879bcbe9..7f6cadbd2aa 100644 --- a/rs/moq-srt/src/ts.rs +++ b/rs/moq-srt/src/ts.rs @@ -426,7 +426,7 @@ mod tests { .await .expect("announce timed out") .expect("the broadcast is announced"); - let broadcast = consumer.request_broadcast(path).await.unwrap(); + let broadcast = consumer.request_broadcast(path, None).await.unwrap(); let mut catalog = moq_mux::catalog::Consumer::::new(&broadcast, CatalogFormat::Hang) .await .unwrap(); @@ -461,7 +461,7 @@ mod tests { let origin = produce_origin(); let consumer = origin.consume(); let _stale = Publisher::new(&origin, "ingest", Default::default(), None).unwrap(); - let stale = consumer.request_broadcast("ingest").await.unwrap(); + let stale = consumer.request_broadcast("ingest", None).await.unwrap(); let mut catalog = stale .track(hang::Catalog::DEFAULT_NAME) .unwrap() @@ -482,7 +482,7 @@ mod tests { .await .expect("the stale viewer stalled"); assert!(matches!(ended, moq_net::Error::Unroutable), "{ended:?}"); - let fresh = consumer.request_broadcast("ingest").await.unwrap(); + let fresh = consumer.request_broadcast("ingest", None).await.unwrap(); assert!(!fresh.is_clone(&stale), "viewers reach the reconnected caller"); } @@ -531,7 +531,7 @@ mod tests { let consumer = origin.consume(); consumer.routed("live/cam0").await.unwrap(); - let broadcast = consumer.request_broadcast("live/cam0").await.unwrap(); + let broadcast = consumer.request_broadcast("live/cam0", None).await.unwrap(); let info = broadcast.track("0.avc3").unwrap().query().await.unwrap(); assert_eq!(info.max_age, Some(Duration::from_secs(3))); } @@ -581,7 +581,7 @@ mod tests { .await .expect("announce timed out") .expect("the ingest broadcast is announced"); - let broadcast = consumer.request_broadcast("ingest").await.unwrap(); + let broadcast = consumer.request_broadcast("ingest", None).await.unwrap(); publisher.feed(bytes::Bytes::from_static(BBB5S)).unwrap(); diff --git a/rs/moq-stats/benches/aggregate.rs b/rs/moq-stats/benches/aggregate.rs index e6ab87594a7..0102206aa6a 100644 --- a/rs/moq-stats/benches/aggregate.rs +++ b/rs/moq-stats/benches/aggregate.rs @@ -1,5 +1,6 @@ //! Cost of one merged traffic frame from [`moq_stats::aggregate`], swept over live nodes and //! nodes that have come and gone, with and without the grace that folds departed nodes away. +//! Compare known metadata with one touched publisher losing and recovering its epoch. //! //! Each node reports the same broadcast paths, like a restarted node under a new name. A bounded //! aggregate's cost follows the live nodes; an unbounded one (`grace` never elapsing) also pays for @@ -9,7 +10,7 @@ use std::time::{Duration, Instant}; -use moq_net::origin; +use moq_net::{Epoch, origin}; use moq_stats::{Role, Tier, Traffic, TrafficFrame, aggregate, traffic_track}; /// Broadcast paths each node reports. @@ -23,7 +24,9 @@ const GRACE: Duration = Duration::from_secs(1); /// A hand-published node stats broadcast with a default-tier publisher traffic track. struct Node { - _source: moq_net::broadcast::Producer, + source: moq_net::broadcast::Producer, + epoch: Epoch, + known: bool, traffic: moq_json::snapshot::Producer, frame: TrafficFrame, } @@ -33,7 +36,10 @@ impl Node { let source = origin .create_broadcast(format!(".stats/acme/node/{name}").as_str()) .unwrap(); - source.announce(origin::Route::default()).unwrap(); + let epoch = Epoch::mint(); + source + .announce(origin::Route::default().with_epoch(epoch.clone())) + .unwrap(); let track = source .create_track(traffic_track(&Tier::default(), Role::Publisher, false), None) .unwrap(); @@ -42,7 +48,9 @@ impl Node { .map(|key| (format!("acme/room-{key:02}"), Traffic::default())) .collect(); Self { - _source: source, + source, + epoch, + known: true, traffic: moq_json::snapshot::Producer::new(track, config), frame, } @@ -68,7 +76,7 @@ async fn read_until(traffic: &mut aggregate::TrafficConsumer, want: u64) { } /// Mean wall time per merged frame with `live` nodes publishing after `departed` nodes left. -async fn measure(live: usize, departed: usize, grace: Duration) -> Duration { +async fn measure(live: usize, departed: usize, grace: Duration, mixed: bool) -> Duration { let (origin, driver) = origin::Producer::new(origin::Config::default()); let driver = tokio::spawn(moq_net::time::run(driver)); let agg = aggregate::Consumer::new( @@ -102,6 +110,14 @@ async fn measure(live: usize, departed: usize, grace: Duration) -> Duration { for frame in 0..=FRAMES { let node = &mut nodes[frame as usize % live]; let start = Instant::now(); + if mixed { + node.known = !node.known; + let mut route = origin::Route::default(); + if node.known { + route.epoch = Some(node.epoch.clone()); + } + node.source.announce(route).unwrap(); + } node.publish(1); total += 1; read_until(&mut traffic, total).await; @@ -127,13 +143,15 @@ fn main() { .build() .unwrap(); - println!("live departed grace us/frame"); + println!("live departed grace metadata us/frame"); for live in [1, 16, 128] { for departed in [0, 256, 4096] { for (label, grace) in [("1s", GRACE), ("never", Duration::MAX)] { - let elapsed = runtime.block_on(measure(live, departed, grace)); - let micros = elapsed.as_secs_f64() * 1e6; - println!("{live:>4} {departed:>8} {label:>5} {micros:>11.1}"); + for (metadata, mixed) in [("known", false), ("mixed", true)] { + let elapsed = runtime.block_on(measure(live, departed, grace, mixed)); + let micros = elapsed.as_secs_f64() * 1e6; + println!("{live:>4} {departed:>8} {label:>5} {metadata:>8} {micros:>11.1}"); + } } } } diff --git a/rs/moq-stats/src/aggregate.rs b/rs/moq-stats/src/aggregate.rs index 71c49fac559..cedf4a7da2a 100644 --- a/rs/moq-stats/src/aggregate.rs +++ b/rs/moq-stats/src/aggregate.rs @@ -14,7 +14,7 @@ use std::time::Duration; use moq_net::kio::{self, Pending, Waiter}; use moq_net::stats::{Presence, Role, Tier, Traffic}; use moq_net::track::Subscribing; -use moq_net::{PathOwned, origin}; +use moq_net::{Epoch, PathOwned, origin}; use web_async::time::Instant; use crate::{Result, SessionsFrame, TrafficFrame, parse_node_path, sessions_track, traffic_track}; @@ -42,8 +42,9 @@ pub struct Config { pub compression: bool, /// How long a departed node's traffic waits for its path to return before /// folding into the retired total (default 10 minutes). A return within it - /// resumes the node's own counters; a return after it with counters intact - /// counts its earlier traffic twice, so size this above reconnect times. + /// under the same epoch (or again without one) resumes the node's own + /// counters; a return after it with counters intact counts its earlier + /// traffic twice, so size this above reconnect times. pub grace: Duration, } @@ -102,11 +103,19 @@ impl Default for Config { /// boot-lifetime counters intact never looks like new traffic. After the grace, /// a departed node's contribution folds into one retired total and the node is /// forgotten, so memory follows live nodes and the keys ever reported, not -/// every node ever seen. A node -/// returning within the grace with a lower counter (a restarted relay) -/// regresses the merged counter, the same reset contract a single node's own -/// restart follows. Presence is not sticky: a departed node stops counting -/// sessions immediately. +/// every node ever seen. Each route epoch is its own node: a restarted +/// producer, or a group returning from idle, announces a new epoch whose +/// counters add to the old epoch's kept contribution instead of regressing it. +/// A reader stays pinned to its epoch, so a replacement never feeds it. +/// When epoch metadata disappears or returns at a path (one publisher reached +/// through mixed-version links), only the immediately outgoing contribution +/// carries across; a different known epoch keeps its own entry, even when an +/// unversioned reader resolves it first. A route without an epoch (an older +/// producer, or a session older than moq-lite 07) cannot tell a reconnect from +/// a restart, so its return within the grace with a lower counter regresses the +/// merged counter, the same reset contract a single node's own restart follows. +/// Presence is not sticky: a departed node stops counting sessions +/// immediately. pub struct Consumer { origin: origin::Consumer, config: Config, @@ -246,6 +255,10 @@ struct Node { /// The announced path, relative to the announce cursor: what a re-resolve /// after the reader ends requests again. path: PathOwned, + /// The publisher instance every initial and retry resolution must serve. + epoch: Option, + /// The last known publisher identity, retained while route metadata vanishes. + identity: Option, last: Option>, /// When the path unannounced, starting its grace; cleared by a reannounce. /// Only an unannounced node folds, so a still-announced node whose reader @@ -299,8 +312,10 @@ struct Merged { name: String, config: moq_json::snapshot::consumer::Config, /// One entry per live or recently departed node broadcast, keyed by - /// absolute announced path. - nodes: HashMap>, + /// absolute announced path and route epoch, so each epoch is its own entry. + nodes: HashMap<(PathOwned, Option), Node>, + /// The immediately outgoing identity, including while it remains in grace. + last_announced: HashMap>, /// How long a departed node waits before folding into `retired`. grace: Duration, /// The summed last frames of every node whose grace elapsed. @@ -324,6 +339,7 @@ impl Merged { json }, nodes: HashMap::new(), + last_announced: HashMap::new(), grace: config.grace, retired: BTreeMap::new(), } @@ -361,9 +377,17 @@ impl Merged { let origin = &self.origin; let grace = self.grace; let retired = &mut self.retired; - self.nodes.retain(|_, node| { + let last_announced = &mut self.last_announced; + self.nodes.retain(|(path, epoch), node| { changed |= advance(node, origin, config, name, waiter); - !node.fold_expired(retired, grace, now) + if node.fold_expired(retired, grace, now) { + if last_announced.get(path) == Some(epoch) { + last_announced.remove(path); + } + false + } else { + true + } }); // Release a churn peak's buckets: walking a map costs its capacity, so a // map that once held thousands of nodes would slow every frame. @@ -391,8 +415,34 @@ impl Merged { if parse_node_path(&self.prefix, self.depth, &absolute).is_none() { return false; } + // Publisher replacement can race reader polling on another worker, so every + // reader pins the identity it contributes to, independent of announce order. + let key = (absolute, update.route.epoch); if active { + if let Some(previous) = self.last_announced.insert(key.0.clone(), key.1.clone()) { + // An unversioned route cannot identify a restart. Carry only the + // immediately outgoing contribution across loss or recovery of metadata; + // distinct known epochs remain separate, even while several are in grace. + if previous != key.1 + && (previous.is_none() || key.1.is_none()) + && let Some(mut node) = self.nodes.remove(&(key.0.clone(), previous)) + { + node.depart(); + if key.1.is_some() && node.identity.is_some() && node.identity != key.1 { + // Recovery identifies a distinct known publisher. Keep the outgoing + // lineage in grace and re-arm the incoming identity's own entry. + node.epoch = node.identity.clone(); + self.nodes.insert((key.0.clone(), node.identity.clone()), node); + } else { + node.epoch = key.1.clone(); + if key.1.is_some() { + node.identity = key.1.clone(); + } + self.nodes.insert(key.clone(), node); + } + } + } // A route update on a node already tracked (a reprice, or a takeover // with different metadata) keeps the live reader: existing // subscriptions survive a takeover, and the reader re-resolves through @@ -400,7 +450,7 @@ impl Merged { // node re-arms here, since a fresh route is new evidence that a // re-resolve could succeed. A sticky contribution carries across the // re-arm, holding the total until a fresh frame replaces it. - match self.nodes.entry(absolute) { + match self.nodes.entry(key) { Entry::Occupied(mut entry) => { let node = entry.get_mut(); // A return after the grace starts over, even when this announce @@ -408,13 +458,16 @@ impl Merged { node.fold_expired(&mut self.retired, self.grace, Instant::now()); node.departed = None; if matches!(node.reader, Reader::Ended) { - node.reader = resolve(&self.origin, &node.path); + node.reader = resolve(&self.origin, &node.path, node.epoch.clone()); } false } Entry::Vacant(entry) => { + let epoch = entry.key().1.clone(); entry.insert(Node { - reader: resolve(&self.origin, &path), + reader: resolve(&self.origin, &path, epoch.clone()), + identity: epoch.clone(), + epoch, path, last: None, departed: None, @@ -426,7 +479,7 @@ impl Merged { // Unannounce: keep the cumulative totals, retire the live gauges, and // stop reading. The entry stays so a reannounce within the grace // re-arms it. - match self.nodes.get_mut(&absolute) { + match self.nodes.get_mut(&key) { Some(node) => { node.departed = Some(Instant::now()); node.depart() @@ -436,7 +489,8 @@ impl Merged { } else { // A gauge drops its contribution, and its entry, so a departed node // stops counting and its path is not retained. - self.nodes.remove(&absolute).is_some_and(|old| old.last.is_some()) + self.last_announced.remove(&key.0); + self.nodes.remove(&key).is_some_and(|old| old.last.is_some()) } } @@ -470,6 +524,21 @@ fn advance( loop { match &mut node.reader { Reader::Resolving { pending, queued } => match pending.poll_ok(waiter) { + // An unpinned retry on a known lineage can reach a replacement publisher + // before its announcement drains here. Its counters belong to its own entry, + // which that announcement creates, so this lineage keeps its last frame. + Poll::Ready(Ok(broadcast)) + if node.epoch.is_none() + && node.identity.is_some() + && broadcast + .info() + .epoch + .as_ref() + .is_some_and(|epoch| node.identity.as_ref() != Some(epoch)) => + { + tracing::debug!(name, "stats: node resolved another publisher"); + return changed | node.depart(); + } Poll::Ready(Ok(broadcast)) => match broadcast.track(name) { Ok(track) => node.reader = Reader::Subscribing(track.subscribe(None)), Err(err) => { @@ -484,7 +553,7 @@ fn advance( // means nothing serves the path (the retraction that empties the // table also unannounces this node). Poll::Ready(Err(moq_net::Error::Unroutable)) if *queued => { - node.reader = resolve(origin, &node.path); + node.reader = resolve(origin, &node.path, node.epoch.clone()); } Poll::Ready(Err(err)) => { tracing::debug!(?err, name, "stats: node broadcast unresolvable"); @@ -529,7 +598,7 @@ fn advance( return changed; } rearmed = true; - node.reader = resolve(origin, &node.path); + node.reader = resolve(origin, &node.path, node.epoch.clone()); } Poll::Pending => return changed, }, @@ -539,8 +608,8 @@ fn advance( } /// Start resolving `path` (relative to the announce cursor) into a broadcast. -fn resolve(origin: &origin::Consumer, path: &PathOwned) -> Reader { - let pending = origin.request_broadcast(path); +fn resolve(origin: &origin::Consumer, path: &PathOwned, epoch: Option) -> Reader { + let pending = origin.request_broadcast(path, epoch); let queued = pending.is_queued(); Reader::Resolving { pending, queued } } @@ -614,7 +683,7 @@ mod tests { let (_, active) = next_update(&mut announced).await.expect("announce"); assert!(active); - let consumer = egress.request_broadcast(path).await.expect("resolve"); + let consumer = egress.request_broadcast(path, None).await.expect("resolve"); let mut sub = consumer.track("video").unwrap().subscribe(None).await.unwrap(); let mut group = track.append_group().unwrap(); @@ -668,7 +737,8 @@ mod tests { /// A hand-published node broadcast at `.stats//node/` with a /// plain default-tier traffic track, so a test controls the exact frames and /// can fail one node's reader alone (the registry-driven producer only - /// publishes whole broadcasts). Dropping it unannounces the node. + /// publishes whole broadcasts). Its route carries no epoch, so a new one at + /// the same path is that node returning. Dropping it unannounces the node. #[allow(dead_code)] struct NodeBroadcast { source: broadcast::Producer, @@ -770,10 +840,11 @@ mod tests { } #[tokio::test(start_paused = true)] - async fn reannounce_with_higher_counter_stays_monotonic() { - // A node's stats session reconnects with its cumulative counters intact - // and still growing: the total holds through the swap, then advances, - // never dipping. + async fn restarted_producer_adds_a_new_epoch() { + // A restarted node announces a new epoch with fresh counters. The new + // epoch adds to the old one's kept contribution instead of regressing + // the total, and the old epoch folds into the retired total once the + // grace elapses. let origin = produce_origin(); let node_a = node_producer(&origin, "a"); let node_b = node_producer(&origin, "b"); @@ -782,54 +853,273 @@ mod tests { let _fb = feed(&node_b, Tier::default(), "acme", "acme/room", 40).await; drive_tick().await; - let agg = Consumer::new(origin.consume(), Config::new().with_depth(1)); + let agg = Consumer::new(origin.consume(), Config::new().with_depth(1).with_grace(GRACE)); let mut traffic = agg.traffic(&Tier::default(), Role::Publisher); read_until_bytes(&mut traffic, "acme/room", 140).await; - // Node A's broadcast goes away and comes back with a higher counter. drop(fa); drop(node_a); drive_tick().await; let node_a = node_producer(&origin, "a"); - let _fa = feed(&node_a, Tier::default(), "acme", "acme/room", 120).await; + let _fa = feed(&node_a, Tier::default(), "acme", "acme/room", 30).await; drive_tick().await; - // The kept contribution holds the total at 140 until the new frame - // replaces it, landing on 120 + 40. - let frame = read_monotonic_until(&mut traffic, "acme/room", 140, 160).await; - assert_eq!(frame.get("acme/room").expect("entry").bytes, 160); + let frame = read_monotonic_until(&mut traffic, "acme/room", 140, 170).await; + assert_eq!(frame.get("acme/room").expect("entry").bytes, 170, "100 + 30 + 40"); + assert_eq!(traffic.inner.nodes.len(), 3, "the old epoch lingers for its grace"); + + tokio::time::advance(GRACE).await; + settle(&mut traffic).await; + assert_eq!(traffic.inner.nodes.len(), 2, "the old epoch folded"); + assert_eq!(traffic.inner.retired.get("acme/room").map(|t| t.bytes), Some(100)); } #[tokio::test(start_paused = true)] - async fn restarted_node_regresses_the_total() { - // A node that returns with a fresh counter (it restarted) replaces its - // contribution wholesale: the total regresses once, the existing - // fresh-segment contract. + async fn replaced_epoch_keeps_its_own_counters() { + // A node restarts while its old instance is still announced. The newer + // epoch replaces the old one at the same path; the old entry keeps its + // last contribution and never reads the new epoch, so nothing counts + // twice while the old instance keeps publishing. let origin = produce_origin(); - let node_a = node_producer(&origin, "a"); + let old_a = node_producer(&origin, "a"); let node_b = node_producer(&origin, "b"); - let fa = feed(&node_a, Tier::default(), "acme", "acme/room", 100).await; + let _old_fa = feed(&old_a, Tier::default(), "acme", "acme/room", 100).await; let _fb = feed(&node_b, Tier::default(), "acme", "acme/room", 40).await; drive_tick().await; - let agg = Consumer::new(origin.consume(), Config::new().with_depth(1)); + let agg = Consumer::new(origin.consume(), Config::new().with_depth(1).with_grace(GRACE)); let mut traffic = agg.traffic(&Tier::default(), Role::Publisher); read_until_bytes(&mut traffic, "acme/room", 140).await; - // Node A restarts: its broadcast returns with a lower counter. - drop(fa); - drop(node_a); + let new_a = node_producer(&origin, "a"); + let _new_fa = feed(&new_a, Tier::default(), "acme", "acme/room", 30).await; drive_tick().await; - let node_a = node_producer(&origin, "a"); - let _fa = feed(&node_a, Tier::default(), "acme", "acme/room", 30).await; - drive_tick().await; + let frame = read_monotonic_until(&mut traffic, "acme/room", 140, 170).await; + assert_eq!(frame.get("acme/room").expect("entry").bytes, 170, "100 + 30 + 40"); + + for _ in 0..3 { + drive_tick().await; + if let Some(frame) = settle(&mut traffic).await { + assert_eq!(frame.get("acme/room").expect("entry").bytes, 170, "counted once"); + } + } + assert_eq!(traffic.inner.nodes.len(), 3, "one entry per epoch"); + } + + #[test] + fn outgoing_epoch_metadata_migrates_without_choosing_a_grace_entry() { + let origin = produce_origin(); + let path = PathOwned::from(".stats/acme/node/a"); + let a: Epoch = "01900000-0000-7000-8000-000000000001".parse().unwrap(); + let b: Epoch = "01900000-0000-7000-8000-000000000002".parse().unwrap(); + let c: Epoch = "01900000-0000-7000-8000-000000000003".parse().unwrap(); + let d: Epoch = "01900000-0000-7000-8000-000000000004".parse().unwrap(); + let mut merged = Merged::::new(origin.consume(), &Config::new().with_depth(1), "traffic".into()); + let update = |epoch: Option| moq_net::announce::Announce { + prefix: path.clone(), + captures: None, + route: { + let mut route = origin::Route::default(); + route.epoch = epoch; + route + }, + }; + for (epoch, bytes) in [(a.clone(), 100), (b.clone(), 30), (c.clone(), 7)] { + merged.apply_announce(update(Some(epoch.clone())), true); + merged.nodes.get_mut(&(path.clone(), Some(epoch.clone()))).unwrap().last = + Some(BTreeMap::from([("room".into(), { + let mut traffic = Traffic::default(); + traffic.bytes = bytes; + traffic + })])); + merged.apply_announce(update(Some(epoch)), false); + } + assert_eq!(merged.merged()["room"].bytes, 137); + merged.apply_announce(update(None), true); + assert_eq!( + merged.nodes.len(), + 3, + "only the immediately outgoing C becomes unversioned" + ); + assert!(merged.nodes.contains_key(&(path.clone(), Some(a.clone())))); + assert!(merged.nodes.contains_key(&(path.clone(), Some(b.clone())))); + assert!(!merged.nodes.contains_key(&(path.clone(), Some(c.clone())))); + assert_eq!( + merged.merged()["room"].bytes, + 137, + "metadata loss preserves the last snapshot" + ); + + // A fresh unversioned snapshot replaces only the outgoing contribution. + merged.nodes.get_mut(&(path.clone(), None)).unwrap().last = Some(BTreeMap::from([("room".into(), { + let mut traffic = Traffic::default(); + traffic.bytes = 8; + traffic + })])); + merged.apply_announce(update(None), false); + merged.apply_announce(update(Some(a.clone())), true); + assert_eq!( + merged.nodes.len(), + 3, + "recovering A retains both outgoing C and grace B" + ); + assert!(merged.nodes.contains_key(&(path.clone(), Some(c)))); + assert_eq!( + merged.nodes[&(path.clone(), Some(a.clone()))].last.as_ref().unwrap()["room"].bytes, + 100 + ); + assert_eq!( + merged.merged()["room"].bytes, + 138, + "retained A is not overwritten by outgoing C" + ); + + merged.apply_announce(update(Some(a.clone())), false); + merged.apply_announce(update(None), true); + merged.apply_announce(update(None), false); + merged.apply_announce(update(Some(d.clone())), true); + assert_eq!( + merged.nodes.len(), + 4, + "a distinct recovered epoch gets its own contribution" + ); + assert!(merged.nodes.contains_key(&(path.clone(), Some(a)))); + assert!(merged.nodes.contains_key(&(path.clone(), Some(d)))); + assert!(!merged.nodes.contains_key(&(path, None))); + assert_eq!( + merged.merged()["room"].bytes, + 138, + "known grace entries remain counted once each" + ); + } + + /// A known replacement must not overwrite the outgoing known epoch's counters. + #[test] + fn replacement_between_announce_drain_and_reader_retry() { + replacement_while_reader_retries(false); + } + + /// Missing metadata must not let the outgoing lineage consume a recovered known epoch. + #[test] + fn unknown_replacement_between_announce_drain_and_reader_retry() { + replacement_while_reader_retries(true); + } + + /// Arrange replacement on another worker after announcements drain and before retrying. + fn replacement_while_reader_retries(drop_epoch: bool) { + use std::sync::Barrier; + + let now = std::time::Instant::now(); + let (origin, mut driver) = origin::Producer::new(origin::Config::default()); + let epoch_a: Epoch = "01900000-0000-7000-8000-000000000001".parse().unwrap(); + let epoch_b: Epoch = "01900000-0000-7000-8000-000000000002".parse().unwrap(); + let mut old = NodeBroadcast::new(&origin, "acme", "a"); + old.source + .announce(origin::Route::default().with_epoch(epoch_a.clone())) + .unwrap(); + old.publish("acme/room", 100); + let name = traffic_track(&Tier::default(), Role::Publisher, false); + let mut merged = Merged::::new(origin.consume(), &Config::new().with_depth(1), name.clone()); + let initial = futures::executor::block_on(kio::wait(|waiter| { + driver.poll(now, waiter).unwrap(); + merged.poll_next(waiter) + })) + .unwrap() + .unwrap(); + assert_eq!(initial["acme/room"].bytes, 100); + if drop_epoch { + old.source.announce(origin::Route::default()).unwrap(); + let unversioned = futures::executor::block_on(kio::wait(|waiter| { + driver.poll(now, waiter).unwrap(); + merged.poll_next(waiter) + })) + .unwrap() + .unwrap(); + assert_eq!(unversioned["acme/room"].bytes, 100); + assert!(matches!( + merged.nodes[&(PathOwned::from(".stats/acme/node/a"), None)].reader, + Reader::Active(_) + )); + } + assert!(merged.announce.poll_next(&Waiter::noop()).is_pending()); + + let ready = Barrier::new(2); + let release = Barrier::new(2); + std::thread::scope(|scope| { + let worker = scope.spawn(|| { + let mut new = NodeBroadcast::new(&origin, "acme", "a"); + new.source + .announce(origin::Route::default().with_epoch(epoch_b.clone())) + .unwrap(); + new.publish("acme/room", 30); + let pending = origin.consume().request_broadcast(".stats/acme/node/a", None); + let front = futures::executor::block_on(kio::wait(|waiter| { + driver.poll(now, waiter).unwrap(); + pending.poll_ok(waiter) + })) + .unwrap(); + let pending = front.track(&name).unwrap().subscribe(None); + let subscriber = futures::executor::block_on(kio::wait(|waiter| { + driver.poll(now, waiter).unwrap(); + pending.poll_ok(waiter) + })) + .unwrap(); + let mut warm = moq_json::snapshot::Consumer::::new(subscriber, Default::default()); + let frame = futures::executor::block_on(kio::wait(|waiter| { + driver.poll(now, waiter).unwrap(); + warm.poll_next(waiter) + })) + .unwrap() + .unwrap(); + assert_eq!(frame["acme/room"].bytes, 30); + ready.wait(); + release.wait(); + (new, front, warm, driver) + }); + ready.wait(); + let key = (PathOwned::from(".stats/acme/node/a"), (!drop_epoch).then_some(epoch_a)); + let node = merged.nodes.get_mut(&key).unwrap(); + advance(node, &merged.origin, &merged.config, &name, &Waiter::noop()); + let old_bytes = node.last.as_ref().unwrap()["acme/room"].bytes; + // Release before asserting, so a failed regression also joins its worker. + release.wait(); + let (_new, _front, _warm, mut driver) = worker.join().unwrap(); + let combined = futures::executor::block_on(kio::wait(|waiter| { + driver.poll(now, waiter).unwrap(); + merged.poll_next(waiter) + })) + .unwrap() + .unwrap(); + assert_eq!(combined["acme/room"].bytes, 130); + assert_eq!(old_bytes, 100, "the replacement frame overwrote the old epoch"); + }); + } + + #[tokio::test(start_paused = true)] + async fn unversioned_restart_regresses_the_total() { + // A node whose route carries no epoch (an older producer, or a session + // older than moq-lite 07) returns at the same path with a fresh counter: + // it replaces its contribution wholesale, and the total regresses once, + // the fresh-segment contract. + let origin = produce_origin(); + let mut node_a = NodeBroadcast::new(&origin, "acme", "a"); + let mut node_b = NodeBroadcast::new(&origin, "acme", "b"); + node_a.publish("acme/room", 100); + node_b.publish("acme/room", 40); + + let agg = Consumer::new(origin.consume(), Config::new().with_depth(1)); + let mut traffic = agg.traffic(&Tier::default(), Role::Publisher); + read_until_bytes(&mut traffic, "acme/room", 140).await; + + drop(node_a); + settle(&mut traffic).await; + let mut node_a = NodeBroadcast::new(&origin, "acme", "a"); + node_a.publish("acme/room", 30); - // The restarted frame replaces A's contribution, so the total drops to - // 30 + 40, a genuine per-node regression downstream treats as a fresh - // segment. An earlier frame may retire A's live gauges first. + // An earlier frame may retire A's live gauges first. loop { let frame = traffic.next().await.expect("read").expect("frame"); if frame.get("acme/room").map(|t| t.bytes) == Some(70) { diff --git a/rs/moq-stats/src/consume.rs b/rs/moq-stats/src/consume.rs index baf0473b53f..3d01b86c872 100644 --- a/rs/moq-stats/src/consume.rs +++ b/rs/moq-stats/src/consume.rs @@ -177,7 +177,7 @@ mod tests { let (_, active) = next_update(&mut announced).await.expect("announce"); assert!(active); - let consumer = egress.request_broadcast(path).await.expect("resolve"); + let consumer = egress.request_broadcast(path, None).await.expect("resolve"); let sub = consumer.track("video").unwrap().subscribe(None).await.unwrap(); Feed { @@ -196,7 +196,7 @@ mod tests { assert!(active); origin .consume() - .request_broadcast(moq_net::Path::new(update.prefix.as_str())) + .request_broadcast(moq_net::Path::new(update.prefix.as_str()), None) .await .expect("resolve") } diff --git a/rs/moq-stats/src/lib.rs b/rs/moq-stats/src/lib.rs index 99b2411232e..48241ba7d45 100644 --- a/rs/moq-stats/src/lib.rs +++ b/rs/moq-stats/src/lib.rs @@ -17,7 +17,10 @@ //! A [`Producer`] publishes one broadcast per node at `/node/` //! (default prefix `.stats`), or one per group of leading broadcast-path //! segments at `//node/`; parse announce paths back with -//! [`parse_node_path`]. Each [`Tier`] carries `publisher.json`, +//! [`parse_node_path`]. Each announcement carries a fresh +//! [`Epoch`](moq_net::Epoch) on its route, so a restarted node or a group +//! returning from idle is a new broadcast at the same path; a reader tells +//! them apart by the announced route's epoch. Each [`Tier`] carries `publisher.json`, //! `subscriber.json`, and `sessions.json` tracks of cumulative [`Traffic`] and //! [`Presence`] counters, plus `.json.z` siblings encoded with //! [`moq_json::snapshot`]; compute names with [`traffic_track`] / diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index 3b5ab03db93..dab7e73d18b 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -8,7 +8,7 @@ use std::time::Duration; use std::task::Poll; use moq_net::stats::{Presence, Registry, Report, Role, Tier, Traffic}; -use moq_net::{Path, PathOwned, broadcast, kio, origin, track}; +use moq_net::{Epoch, Path, PathOwned, broadcast, kio, origin, track}; use serde::Serialize; use web_async::spawn; use web_async::time::Instant; @@ -31,8 +31,9 @@ pub struct Config { pub origin: Option, /// Top-level path stats are published under (default `.stats`). The full /// advertised path is `/node/` (or `/node` when - /// `node` is unset). Also the registry's exclude prefix, so serving a - /// stats broadcast doesn't generate more stats. + /// `node` is unset), announced under a fresh [`Epoch`] each time. Also the + /// registry's exclude prefix, so serving a stats broadcast doesn't generate + /// more stats. pub prefix: PathOwned, /// Node suffix that disambiguates broadcasts from different relays sharing a /// cluster origin. Set this on every node in multi-relay deployments. May be @@ -48,8 +49,9 @@ pub struct Config { /// path. `1` publishes one broadcast per first segment at /// `//node/`, and larger values include more leading /// segments. Group broadcasts are announced while their group has live traffic, - /// plus the [`Self::linger`]; at depth `0`, the single broadcast stays - /// announced for the producer's life. + /// plus the [`Self::linger`], and a group returning after that announces + /// under a new epoch counted from zero. At depth `0`, the single broadcast + /// and its epoch last for the producer's life. pub depth: usize, /// How long a group broadcast stays announced after its group's last entry /// leaves. Default 5 minutes. @@ -58,6 +60,10 @@ pub struct Config { /// leaving and another arriving causes no unannounce and re-announce across /// the mesh. While it lingers empty, every track reads `{}`. Zero /// unannounces on the first drain the group is empty. Unused at depth `0`. + /// + /// A return after the linger is a new epoch, which an aggregating reader + /// keeps as its own node for its grace, so a short linger on a flapping + /// group multiplies that reader's state. pub linger: Duration, } @@ -205,7 +211,6 @@ impl Producer { depth, linger, interval, - sequence: Arc::new(AtomicU64::new(0)), }; spawn(task.run(Arc::downgrade(&keepalive))); @@ -232,7 +237,6 @@ struct Task { depth: usize, linger: Duration, interval: Duration, - sequence: Arc, } impl Task { @@ -439,10 +443,11 @@ impl Serialize for Frame { } } -/// Writes encoded snapshots with group numbers from the producer-wide -/// allocator. The allocator outlives on-demand tracks and group broadcasts, so -/// a recreated track resumes past any group a subscriber has cached, without -/// retaining a tombstone per requested name. +/// Writes encoded snapshots with group numbers from its group broadcast's +/// allocator. The allocator outlives on-demand tracks, so a recreated track +/// resumes past any group a subscriber has cached, without retaining a +/// tombstone per requested name. A new group broadcast is a new epoch, so its +/// allocator starts over. struct Snapshot { track: track::Producer, encoder: moq_json::snapshot::Encoder>, @@ -868,7 +873,10 @@ impl TierNames { impl GroupPublisher { fn create(task: &Task, group: &Path) -> Option { let advertised = advertised_path(&task.prefix, group, task.node()); - let broadcast = match task.origin.publish(&advertised, origin::Route::default()) { + // A fresh epoch per announcement: a restart or a return from idle is a new + // broadcast, so no relay resumes it from groups cached under the old one. + let route = origin::Route::default().with_epoch(Epoch::mint()); + let broadcast = match task.origin.publish(&advertised, route) { Ok(broadcast) => broadcast, Err(err) => { tracing::warn!(advertised = %advertised, ?err, "stats: origin rejected stats broadcast"); @@ -877,14 +885,15 @@ impl GroupPublisher { }; tracing::debug!(advertised = %advertised, "stats: publishing broadcast"); - let mut traffic = TrackFamily::new(task.sequence.clone()); - let mut sessions = TrackFamily::new(task.sequence.clone()); + let sequence = Arc::new(AtomicU64::new(0)); + let mut traffic = TrackFamily::new(sequence.clone()); + let mut sessions = TrackFamily::new(sequence.clone()); // The default tier's tracks always exist, even while idle. let tier = Tier::default(); for role in [Role::Publisher, Role::Subscriber] { let name = traffic_track(&tier, role, false); - match TrackPair::create(&broadcast, &name, task.sequence.clone()) { + match TrackPair::create(&broadcast, &name, sequence.clone()) { Ok(pair) => { traffic.tracks.insert(name, pair); } @@ -895,7 +904,7 @@ impl GroupPublisher { } } let name = sessions_track(&tier, false); - match TrackPair::create(&broadcast, &name, task.sequence.clone()) { + match TrackPair::create(&broadcast, &name, sequence) { Ok(pair) => { sessions.tracks.insert(name, pair); } @@ -1295,7 +1304,7 @@ mod tests { let (_, active) = next_update(&mut announced).await.expect("announce"); assert!(active); - let consumer = egress.request_broadcast(path).await.expect("resolve"); + let consumer = egress.request_broadcast(path, None).await.expect("resolve"); let sub = if subscribe { let mut sub = consumer @@ -1338,7 +1347,7 @@ mod tests { assert!(active); let broadcast = origin .consume() - .request_broadcast(moq_net::Path::new(update.prefix.as_str())) + .request_broadcast(moq_net::Path::new(update.prefix.as_str()), None) .await .expect("resolve"); (update.prefix.as_str().to_string(), broadcast) @@ -1428,6 +1437,89 @@ mod tests { assert_eq!(announced(&origin).await.0, ".stats/node"); } + /// A producer on `origin` publishing as node `sjc` at depth 0. + fn node_producer(origin: &origin::Producer) -> Producer { + Producer::new( + Config::new() + .with_origin(origin.clone()) + .with_node(PathOwned::from("sjc")), + ) + } + + /// A producer restarted on the same origin ends its old epoch and announces + /// the same path under a new one. + #[tokio::test(start_paused = true)] + async fn restarted_producer_announces_a_new_epoch() { + let origin = produce_origin(); + let mut events = origin.consume().with_hidden(true).announced(); + + let first = node_producer(&origin); + drive_tick().await; + let [(path, true, before)] = &take_epochs(&mut events)[..] else { + panic!("expected one announce"); + }; + assert_eq!(path, ".stats/node/sjc"); + + drop(first); + drive_tick().await; + assert_eq!(take_epochs(&mut events), [(path.clone(), false, before.clone())]); + + let _second = node_producer(&origin); + drive_tick().await; + let [(after_path, true, after)] = &take_epochs(&mut events)[..] else { + panic!("expected one announce"); + }; + assert_eq!(after_path, path); + assert!(after > before, "a restart mints a newer epoch"); + } + + /// A restart while the old instance is still announced (a route the mesh + /// has not withdrawn yet) takes the path at once: the newer epoch replaces + /// the old one and ends its subscriptions, so a reader re-requests the new + /// broadcast instead of stalling on the old one. + #[tokio::test(start_paused = true)] + async fn restart_replaces_a_still_announced_instance() { + let origin = produce_origin(); + let mut events = origin.consume().with_hidden(true).announced(); + + let first = node_producer(&origin); + let _f1 = feed(first.registry(), Tier::default(), "foo/bar", true, 1, 100).await; + drive_tick().await; + let [(path, true, before)] = &take_epochs(&mut events)[..] else { + panic!("expected one announce"); + }; + let stats = origin + .consume() + .request_broadcast(path.as_str(), None) + .await + .expect("resolve"); + let mut old = subscribe(&stats, "publisher.json").await; + while try_next_frame(&mut old).is_some() {} + + let second = node_producer(&origin); + let _f2 = feed(second.registry(), Tier::default(), "foo/bar", true, 1, 30).await; + drive_tick().await; + let [(ended, false, ended_epoch), (started, true, after)] = &take_epochs(&mut events)[..] else { + panic!("expected the old epoch to end and the new one to start"); + }; + assert_eq!((ended, ended_epoch), (path, before)); + assert_eq!(started, path); + assert!(after > before); + + // The old instance keeps publishing, but its subscription ended. + drive_tick().await; + use futures::FutureExt; + let end = old.next_group().now_or_never().expect("the old subscription ended"); + assert!(!matches!(end, Ok(Some(_))), "still reading the old epoch"); + + let stats = origin + .consume() + .request_broadcast(path.as_str(), None) + .await + .expect("resolve"); + assert_eq!(read_last_frame(&stats, "publisher.json").await["foo/bar"].bytes, 30); + } + #[tokio::test(start_paused = true)] async fn single_broadcast_path_announced() { // No matter how many broadcasts get bumped, exactly one stats @@ -1551,7 +1643,7 @@ mod tests { .with_stats(registry.tier(Tier::default()).session("viewer")); async fn view(egress: &origin::Consumer, video: &mut track::Producer, size: usize) { - let broadcast = egress.request_broadcast("foo/bar").await.expect("resolve"); + let broadcast = egress.request_broadcast("foo/bar", None).await.expect("resolve"); let mut sub = broadcast .track("video") .expect("track") @@ -1608,10 +1700,20 @@ mod tests { /// The announce events buffered so far, as `(path, active)`. fn take_events(events: &mut announce::Consumer) -> Vec<(String, bool)> { + take_epochs(events) + .into_iter() + .map(|(path, active, _)| (path, active)) + .collect() + } + + /// The announce events buffered so far in delivery order, as + /// `(path, active, epoch)`. Every stats route carries an epoch. + fn take_epochs(events: &mut announce::Consumer) -> Vec<(String, bool, Epoch)> { use futures::FutureExt; let mut out = Vec::new(); - while let Some(Some((route, active))) = next_update(events).now_or_never() { - out.push((route.prefix.as_str().to_string(), active)); + while let Some(Some((update, active))) = next_update(events).now_or_never() { + let epoch = update.route.epoch.expect("a stats route carries an epoch"); + out.push((update.prefix.as_str().to_string(), active, epoch)); } out } @@ -1628,11 +1730,14 @@ mod tests { // session rooted at `feed` (group `feed`). let first = feed(registry, Tier::default(), "acme/live", true, 1, 100).await; drive_tick().await; - let mut started = take_events(&mut events); + let mut started = take_epochs(&mut events); started.sort(); - assert_eq!(started, [(ACME.to_string(), true), (FEED.to_string(), true)]); - let acme = origin.consume().request_broadcast(ACME).await.expect("resolve"); - let sessions = origin.consume().request_broadcast(FEED).await.expect("resolve"); + let [(acme_path, true, acme_epoch), (feed_path, true, feed_epoch)] = &started[..] else { + panic!("expected both groups to announce, got {started:?}"); + }; + assert_eq!((acme_path.as_str(), feed_path.as_str()), (ACME, FEED)); + let acme = origin.consume().request_broadcast(ACME, None).await.expect("resolve"); + let sessions = origin.consume().request_broadcast(FEED, None).await.expect("resolve"); assert_eq!(read_last_frame(&acme, "publisher.json").await["acme/live"].bytes, 100); // The viewer leaves: both groups empty but stay announced, reading zero. @@ -1657,6 +1762,8 @@ mod tests { "a return within the linger re-announces nothing" ); assert_eq!(read_last_frame(&acme, "publisher.json").await["acme/live"].bytes, 50); + let mut track = subscribe(&acme, "publisher.json").await; + let old_sequence = track.next_group().await.expect("ok").expect("group").sequence; // The return re-armed the linger: it unannounces only once it elapses // again with the group still empty. @@ -1668,18 +1775,38 @@ mod tests { for _ in 0..4 { drive_tick().await; } - let mut ended = take_events(&mut events); + let mut ended = take_epochs(&mut events); ended.sort(); - assert_eq!(ended, [(ACME.to_string(), false), (FEED.to_string(), false)]); + assert_eq!( + ended, + [ + (ACME.to_string(), false, acme_epoch.clone()), + (FEED.to_string(), false, feed_epoch.clone()) + ] + ); - // A return after the linger is a new broadcast counted from zero. + // A return after the linger is a new epoch counted from zero, with group + // numbers starting over. let _third = feed(registry, Tier::default(), "acme/live", true, 1, 25).await; drive_tick().await; - let mut restarted = take_events(&mut events); + let mut restarted = take_epochs(&mut events); restarted.sort(); - assert_eq!(restarted, [(ACME.to_string(), true), (FEED.to_string(), true)]); - let acme = origin.consume().request_broadcast(ACME).await.expect("resolve"); + let [(acme_again, true, new_acme), (feed_again, true, new_feed)] = &restarted[..] else { + panic!("expected both groups to announce again, got {restarted:?}"); + }; + assert_eq!((acme_again.as_str(), feed_again.as_str()), (ACME, FEED)); + assert!( + new_acme > acme_epoch && new_feed > feed_epoch, + "a returning group mints a new epoch" + ); + let acme = origin.consume().request_broadcast(ACME, None).await.expect("resolve"); assert_eq!(read_last_frame(&acme, "publisher.json").await["acme/live"].bytes, 25); + let mut track = subscribe(&acme, "publisher.json").await; + let group = track.next_group().await.expect("ok").expect("group"); + assert!( + group.sequence < old_sequence, + "a new epoch numbers its groups from zero" + ); } #[tokio::test(start_paused = true)] @@ -2098,7 +2225,6 @@ mod tests { origin: produce_origin(), prefix: PathOwned::from(".stats"), node: None, - sequence: Arc::new(AtomicU64::new(0)), depth, linger: DEFAULT_LINGER, interval: Duration::from_secs(1), diff --git a/rs/moq-stats/src/produce_bench.rs b/rs/moq-stats/src/produce_bench.rs index 14767c39a4d..ab6d9cb3fcc 100644 --- a/rs/moq-stats/src/produce_bench.rs +++ b/rs/moq-stats/src/produce_bench.rs @@ -1,8 +1,6 @@ //! One producer tick, without the publish interval. The benchmark drives this //! over a registry it filled itself. Not a public API. -use std::sync::Arc; -use std::sync::atomic::AtomicU64; use std::time::Duration; use moq_net::{PathOwned, origin, stats::Registry}; @@ -44,7 +42,6 @@ impl Driver { depth: 0, linger: Duration::from_secs(300), interval: Duration::from_secs(1), - sequence: Arc::new(AtomicU64::new(0)), }; Some(Self { drain: Drain::new(task)?, diff --git a/rs/moq-tokio/examples/clock.rs b/rs/moq-tokio/examples/clock.rs index f5e73100ee8..d86f3782730 100644 --- a/rs/moq-tokio/examples/clock.rs +++ b/rs/moq-tokio/examples/clock.rs @@ -112,7 +112,7 @@ async fn main() -> anyhow::Result<()> { Some(event) = announced.next() => match event { announce::Event::Start(update) | announce::Event::Update(update) => { tracing::info!(broadcast = %update.prefix, "broadcast is online, subscribing to track"); - let broadcast = consumer.request_broadcast(&update.prefix).await?; + let broadcast = consumer.request_broadcast(&update.prefix, None).await?; let track = broadcast .track(&track)?.subscribe(None).await?; clock = Some(Subscriber::new(track)); diff --git a/rs/moq-tokio/src/client.rs b/rs/moq-tokio/src/client.rs index db4a86024f1..ca4be5265ad 100644 --- a/rs/moq-tokio/src/client.rs +++ b/rs/moq-tokio/src/client.rs @@ -1703,7 +1703,7 @@ mod tests { }; assert_eq!(update.prefix.as_str(), "test"); let broadcast = sub_consumer - .request_broadcast("test") + .request_broadcast("test", None) .await .expect("broadcast resolves"); let mut track = broadcast diff --git a/rs/moq-tokio/src/server.rs b/rs/moq-tokio/src/server.rs index b76b409b7aa..ecb2e98ff59 100644 --- a/rs/moq-tokio/src/server.rs +++ b/rs/moq-tokio/src/server.rs @@ -2260,7 +2260,7 @@ mod tests { .expect("origin closed"); assert_eq!(update.prefix.as_str(), "test"); assert!(active); - let broadcast = consumer.request_broadcast("test").await.expect("resolve"); + let broadcast = consumer.request_broadcast("test", None).await.expect("resolve"); let mut track = broadcast .track("video") diff --git a/rs/moq-tokio/tests/backend.rs b/rs/moq-tokio/tests/backend.rs index b6aa80178d5..12479d771e2 100644 --- a/rs/moq-tokio/tests/backend.rs +++ b/rs/moq-tokio/tests/backend.rs @@ -182,7 +182,7 @@ async fn connect_test(config: ConnectTest<'_>) { .expect("origin closed"); assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); - let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timed out") .expect("announced broadcast resolves"); @@ -703,7 +703,7 @@ async fn iroh_connect_test(version: Option<&str>) { .expect("origin closed"); assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); - let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timed out") .expect("announced broadcast resolves"); @@ -870,7 +870,7 @@ async fn noq_client_close_drains_finished_track() { .await .expect("announce timed out") .expect("origin closed"); - let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test")) + let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test", None)) .await .expect("request timed out") .expect("announced broadcast resolves"); @@ -989,7 +989,7 @@ async fn noq_client_close_drains_migrated_predecessor() { .await .expect("announce timed out") .expect("origin closed"); - let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test")) + let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test", None)) .await .expect("request timed out") .expect("announced broadcast resolves"); @@ -1124,7 +1124,7 @@ async fn noq_client_close_keeps_predecessor_handover() { .await .expect("announce timed out") .expect("origin closed"); - let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test")) + let broadcast = tokio::time::timeout(TIMEOUT, consumer.request_broadcast("test", None)) .await .expect("request timed out") .expect("announced broadcast resolves"); diff --git a/rs/moq-tokio/tests/broadcast.rs b/rs/moq-tokio/tests/broadcast.rs index 2bf68189091..6db94c3d7ec 100644 --- a/rs/moq-tokio/tests/broadcast.rs +++ b/rs/moq-tokio/tests/broadcast.rs @@ -90,7 +90,7 @@ async fn broadcast_test(scheme: &str, client_version: Option<&str>, server_versi assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); - let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timed out") .expect("announced broadcast resolves"); @@ -203,7 +203,7 @@ async fn lite05_timestamp_roundtrip(scheme: &str) { .expect("origin closed"); assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); - let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timed out") .expect("announced broadcast resolves"); @@ -326,7 +326,7 @@ async fn lite05_fetch_roundtrip(scheme: &str) { .expect("origin closed"); assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); - let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timed out") .expect("announced broadcast resolves"); @@ -436,7 +436,7 @@ async fn transport_fetch_roundtrip(version: &str, served: bool) { .await .expect("announce timed out") .expect("origin closed"); - let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timed out") .expect("announced broadcast resolves"); @@ -593,7 +593,7 @@ async fn lite05_fetch_during_subscribe(scheme: &str) { .expect("origin closed"); assert_eq!(update.prefix.as_str(), "test"); assert!(active, "expected announce, got retraction"); - let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let bc = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timed out") .expect("announced broadcast resolves"); @@ -705,7 +705,7 @@ async fn broadcast_moq_lite_05_default_timescale() { assert!(active, "expected announce"); let bc = tokio::time::timeout( TIMEOUT, - sub_consumer.request_broadcast(moq_net::Path::new(update.prefix.as_str())), + sub_consumer.request_broadcast(moq_net::Path::new(update.prefix.as_str()), None), ) .await .expect("request timed out") @@ -803,7 +803,7 @@ async fn broadcast_moq_transport_20_current_group_join() { assert!(announced_active, "expected an announce"); let remote = tokio::time::timeout( TIMEOUT, - sub_consumer.request_broadcast(moq_net::Path::new(announced.prefix.as_str())), + sub_consumer.request_broadcast(moq_net::Path::new(announced.prefix.as_str()), None), ) .await .expect("request timed out") @@ -1121,7 +1121,7 @@ async fn broadcast_route_migration() { let subscription = moq_net::track::Subscription::default() .with_start(moq_net::track::Position::group(1)) .with_max_delay(Duration::from_secs(10)); - let broadcast = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let broadcast = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timeout") .expect("routed broadcast resolves"); @@ -1243,7 +1243,7 @@ async fn rejoin_skips_a_stale_warm_cache(version: &str) { .expect("connect failed"); assert!(next_announce(&mut announcements).await.1); - let remote = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let remote = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timeout") .expect("broadcast resolves"); @@ -1363,7 +1363,7 @@ async fn rejoin_replays_a_current_warm_cache(version: &str, open: bool) { .expect("connect failed"); assert!(next_announce(&mut announcements).await.1); - let remote = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let remote = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timeout") .expect("broadcast resolves"); @@ -1497,7 +1497,7 @@ async fn route_reannounce_test(version: Option<&str>) { assert!(active, "expected announce"); let initial = update.route; - let broadcast = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + let broadcast = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test", None)) .await .expect("request timeout") .expect("routed broadcast resolves"); @@ -1808,7 +1808,7 @@ async fn max_age_test(version: &str, published: Option) -> Option) -> anyhow::R .ok_or_else(|| anyhow::anyhow!("announcements closed"))?, moq_net::announce::Event::Start(_) ) {} - let front = consumer.request_broadcast("age").await?; + let front = consumer.request_broadcast("age", None).await?; for (i, age) in AGES.into_iter().enumerate() { let track = front.track(&i.to_string())?.subscribe(None).await?; anyhow::ensure!( diff --git a/rs/moq-tokio/tests/subscription_end_integrity.rs b/rs/moq-tokio/tests/subscription_end_integrity.rs index 805be83e339..61978f7a73e 100644 --- a/rs/moq-tokio/tests/subscription_end_integrity.rs +++ b/rs/moq-tokio/tests/subscription_end_integrity.rs @@ -80,7 +80,10 @@ async fn round(drop_session: bool) -> (usize, Option) { .expect("client connect"); consumer.routed("test").await.expect("routed"); - let remote = consumer.request_broadcast("test").await.expect("broadcast resolves"); + let remote = consumer + .request_broadcast("test", None) + .await + .expect("broadcast resolves"); let mut sub = remote.track("video").unwrap().subscribe(None).await.expect("subscribe"); let mut got = 0; diff --git a/rs/moq-tokio/tests/unannounce.rs b/rs/moq-tokio/tests/unannounce.rs index 9e27bae341d..f06bfe88ef3 100644 --- a/rs/moq-tokio/tests/unannounce.rs +++ b/rs/moq-tokio/tests/unannounce.rs @@ -12,7 +12,7 @@ async fn release(unannounce: bool, settle: bool) { let origin = moq_tokio::origin::spawn(); let broadcast = origin.publish("a/b", moq_net::origin::Route::default()).unwrap(); let track = broadcast.create_track("t", None).unwrap(); - let consumer = origin.consume().request_broadcast("a/b").await.unwrap(); + let consumer = origin.consume().request_broadcast("a/b", None).await.unwrap(); let sub = consumer.track("t").unwrap().subscribe(None).await.unwrap(); track.demand().used().await.unwrap(); diff --git a/rs/moq-transcode/examples/transcode.rs b/rs/moq-transcode/examples/transcode.rs index d5431d42e11..7614fd98b09 100644 --- a/rs/moq-transcode/examples/transcode.rs +++ b/rs/moq-transcode/examples/transcode.rs @@ -70,7 +70,7 @@ async fn main() -> anyhow::Result<()> { // Resolve it for real; the session subscribes upstream on demand. let source = consumer - .request_broadcast(&source_path) + .request_broadcast(&source_path, None) .await .context("source broadcast unavailable")?; diff --git a/rs/moq-uring/benches/session_lite.rs b/rs/moq-uring/benches/session_lite.rs index 894b1aacdf5..c83673c2388 100644 --- a/rs/moq-uring/benches/session_lite.rs +++ b/rs/moq-uring/benches/session_lite.rs @@ -159,7 +159,10 @@ mod linux { let bc = { let consumer = sub_origin.consume(); consumer.routed("bench").await.expect("broadcast announced"); - consumer.request_broadcast("bench").await.expect("broadcast resolves") + consumer + .request_broadcast("bench", None) + .await + .expect("broadcast resolves") }; let sub = bc .track("data") diff --git a/rs/moq-uring/tests/session.rs b/rs/moq-uring/tests/session.rs index 6993fee3dc0..6d747b2937e 100644 --- a/rs/moq-uring/tests/session.rs +++ b/rs/moq-uring/tests/session.rs @@ -122,7 +122,10 @@ fn lite_session_over_the_worker() { let bc = { let consumer = sub.consume(); consumer.routed("test").await.expect("broadcast announced"); - consumer.request_broadcast("test").await.expect("broadcast resolves") + consumer + .request_broadcast("test", None) + .await + .expect("broadcast resolves") }; let mut track = bc .track("data") @@ -250,7 +253,10 @@ fn two_lite_sessions_share_the_server_socket() { let bc = { let consumer = sub.consume(); consumer.routed("test").await.expect("broadcast announced"); - consumer.request_broadcast("test").await.expect("broadcast resolves") + consumer + .request_broadcast("test", None) + .await + .expect("broadcast resolves") }; let mut track = bc .track("data") diff --git a/rs/moq-uring/tests/web.rs b/rs/moq-uring/tests/web.rs index 7b0e35f466f..e0dbe6a9001 100644 --- a/rs/moq-uring/tests/web.rs +++ b/rs/moq-uring/tests/web.rs @@ -247,7 +247,10 @@ fn lite_session_over_webtransport() { let bc = { let consumer = sub_origin.consume(); consumer.routed("test").await.expect("broadcast announced"); - consumer.request_broadcast("test").await.expect("broadcast resolves") + consumer + .request_broadcast("test", None) + .await + .expect("broadcast resolves") }; let mut track = bc .track("data") diff --git a/rs/moq-wasm/src/lib.rs b/rs/moq-wasm/src/lib.rs index c4cabe2fdfa..cf6e0c77d4f 100644 --- a/rs/moq-wasm/src/lib.rs +++ b/rs/moq-wasm/src/lib.rs @@ -159,7 +159,7 @@ impl Session { future_to_promise(async move { let request = pin!(async { consumer.routed(path.as_str()).await?; - consumer.request_broadcast(path.as_str()).await.ok() + consumer.request_broadcast(path.as_str(), None).await.ok() }); // The origin outlives the session, so its wait alone never ends on a close. let closed = pin!(session.closed());