diff --git a/doc/lib/rs/moq-net.md b/doc/lib/rs/moq-net.md index b06d663b8f0..7b36df4d727 100644 --- a/doc/lib/rs/moq-net.md +++ b/doc/lib/rs/moq-net.md @@ -18,7 +18,7 @@ What you use it for, beyond what the concept page already describes: - **An origin outlives the session.** Broadcasts are created on the origin, and a reconnect announces them again. Closing the session does not delete them. - **Requests can pin an epoch.** `consumer.request_broadcast(path, Some(epoch))` resolves only through a route announcing that [publisher epoch](/concept/moq-lite#publisher-epochs); `None` takes whichever route wins. The resolved consumer names its epoch in `info().epoch`, pinned or not. - **One track per name.** Concurrent subscriptions share one request and producer. A producer that replaces an ended one continues the name's group and datagram sequences; only a new broadcast restarts them. -- **Publish only while someone is watching.** `demand()` on a track, group, or broadcast says whether a subscriber is attached, which is how capture and transcode skip work nobody asked for. Holding a broadcast consumer is not demand. A shared fetch stays up until its last reader leaves. +- **Publish only while someone is watching.** `demand()` on a track, group, or broadcast says whether a subscriber is attached, which is how capture and transcode skip work nobody asked for. Holding a broadcast consumer is not demand. Holding a track consumer is, so drop one you are not reading. A shared fetch stays up until its last reader leaves. - **You drive the session, or `moq-tokio` does.** `connect` and `accept` return a session plus a driver that never reads the clock itself. `moq_net::time::run` polls it on tokio or in the browser. `moq-tokio` and `moq-wasm` hide that. A custom transport implements `moq_net::transport::poll`. - **A session caps what its peer can hold.** `session::Limits` (via `Client::with_limits` and `Server::with_limits`) bounds announces and subscriptions per session; the defaults suit a relay mesh, so lower them for untrusted peers. Past either, the session closes with `TOO_MANY_REQUESTS`. On moq-transport drafts 14 to 16 they also size the request-ID window, which every request counts against, so very low limits can starve it. Peer-declared lengths are capped before they are buffered. - **A graceful close waits.** `session.close().await` withdraws announcements and gives finished tracks up to one second to deliver. `abort` ends immediately. IETF drafts 14 through 16 send withdrawals without waiting. diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index ae545d0b71f..0ad17479a76 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -500,6 +500,7 @@ A subscriber opens a Track Stream (0x6) to learn a Track's immutable publisher p The subscriber sends a TRACK message containing the broadcast path and track name. The publisher replies with a single TRACK_INFO message and then FINs the stream, or resets the stream on error (e.g. the track does not exist). +The open stream is interest in the Track, which the publisher holds until the subscriber closes its side, and a subscriber that goes on to subscribe keeps it open until its Subscribe Stream's first response or reset, so demand never lapses between the two streams at any hop. The returned properties are fixed for the lifetime of the track, so the subscriber SHOULD cache TRACK_INFO keyed by broadcast path and track name, and reuse it across every SUBSCRIBE and FETCH of the same track over the session that served it. The properties are fixed for one track, not for the path it arrived on: a path outlives the broadcast on it, and a different broadcast reaching the same path brings its own. Anything cached against a path, on either side, is therefore scoped to the announcement that carried it and is discarded when that announcement is retracted. @@ -1397,6 +1398,7 @@ The `Message Length` describes the payload size on the wire. - Added `Largest Group` and `Largest Frame` to SUBSCRIBE_OK: the publisher's largest position when it answers, which a subscriber takes as where the live feed is. A publisher MUST answer at once when the requested start is past it. Earlier versions carry no such position, so a subscriber takes the first frame instead. - Added `Epoch` to ANNOUNCE_START, SUBSCRIBE, TRACK, and FETCH: a UUIDv7 naming the publisher instance, or empty. A path and an Epoch name one Broadcast. A relay MAY move a subscription between routes with the same Epoch, continuing from the first frame the subscriber lacks instead of at a group boundary, and never between routes with different Epochs or none. The newest Epoch wins a path and ends subscriptions to the older one. Replaces the first-hop identity. - An untimed Track's frames and datagrams carry their send time. lite-05 and lite-06 publishers do the same. +- An open Track Stream is interest in the Track until the subscriber closes it, which a subscriber does once its Subscribe Stream has a response. lite-05 and lite-06 implementations do the same. - Assigned `moq-lite-07-wip` as this draft's protocol identifier until it is finalized as `moq-lite-07`. - Switched every variable-length integer, including SETUP parameter values, from QUIC's two-bit length prefix to moq-transport's leading-ones encoding, widening the range to 64 bits. diff --git a/js/net/src/broadcast.test.ts b/js/net/src/broadcast.test.ts index a50fdf4cfd0..39d66f044ad 100644 --- a/js/net/src/broadcast.test.ts +++ b/js/net/src/broadcast.test.ts @@ -159,6 +159,112 @@ test("an info lookup releases a request nobody subscribed to", async () => { broadcast.close(); }); +test("held info lookups keep a request until the last one lets go", async () => { + const broadcast = new BroadcastProducer(); + const demand = broadcast.demand(); + const pulled = wireOf(broadcast).requested(); + const first = new AbortController(); + const opened = wireOf(broadcast).resolveTrackInfo("media", first.signal); + const request = await pulled; + if (!request) throw new Error("expected request"); + const producer = request.accept(); + const second = new AbortController(); + await wireOf(broadcast).resolveTrackInfo("media", second.signal); + await opened; + + // The lookup that opened the request lets go first, under the one still holding it. + first.abort(); + expect(producer.closed.peek()).toBeUndefined(); + expect(demand.used.peek()).toBe(true); + + second.abort(); + expect(producer.closed.peek()).toBeDefined(); + expect(demand.used.peek()).toBe(false); + broadcast.close(); +}); + +test("an abandoned info lookup stops counting as demand, and its request goes once answered", async () => { + const broadcast = new BroadcastProducer(); + const demand = broadcast.demand(); + const pulled = wireOf(broadcast).requested(); + const hold = new AbortController(); + const info = wireOf(broadcast).resolveTrackInfo("media", hold.signal); + const request = await pulled; + if (!request) throw new Error("expected request"); + expect(demand.used.peek()).toBe(true); + + hold.abort(); + await expect(info).rejects.toThrow(); + expect(demand.used.peek()).toBe(false); + + // The handler still gets an open track, and the answer lets it go. + const producer = request.accept({ timescale: Timescale.MILLI }); + producer.writeString("late"); + await producer.closed; + broadcast.close(); +}); + +test("a held info lookup on an inserted track counts as demand until it lets go", async () => { + const broadcast = new BroadcastProducer(); + const demand = broadcast.demand(); + const track = broadcast.createTrack("media", { timescale: Timescale.MILLI }); + const hold = new AbortController(); + await wireOf(broadcast).resolveTrackInfo("media", hold.signal); + expect(demand.used.peek()).toBe(true); + + hold.abort(); + expect(demand.used.peek()).toBe(false); + expect(track.closed.peek()).toBeUndefined(); + broadcast.close(); +}); + +test("a held info lookup ending after removeTrack leaves demand alone", async () => { + const broadcast = new BroadcastProducer(); + const demand = broadcast.demand(); + const track = broadcast.createTrack("media", { timescale: Timescale.MILLI }); + const subscriber = track.subscribe(); + const hold = new AbortController(); + await wireOf(broadcast).resolveTrackInfo("media", hold.signal); + expect(demand.used.peek()).toBe(true); + + broadcast.removeTrack("media"); + expect(demand.used.peek()).toBe(false); + hold.abort(); + expect(demand.used.peek()).toBe(false); + subscriber.close(); + await Promise.resolve(); + expect(demand.used.peek()).toBe(false); + + // Re-inserting the track counts it again. + broadcast.insertTrack(track); + expect(demand.used.peek()).toBe(false); + const resubscriber = track.subscribe(); + await Promise.resolve(); + expect(demand.used.peek()).toBe(true); + + resubscriber.close(); + track.close(); + broadcast.close(); +}); + +test("removeTrack leaves a track that only a request serves", async () => { + const broadcast = new BroadcastProducer(); + const demand = broadcast.demand(); + const pulled = wireOf(broadcast).requested(); + const subscriber = broadcast.track("media").subscribe(); + const request = await pulled; + if (!request) throw new Error("expected request"); + const producer = request.accept(); + + broadcast.removeTrack("media"); + expect(demand.used.peek()).toBe(true); + expect(producer.closed.peek()).toBeUndefined(); + + subscriber.close(); + producer.close(); + broadcast.close(); +}); + test("closing a broadcast rejects a dequeued request", async () => { const broadcast = new BroadcastProducer(); const pulled = wireOf(broadcast).requested(); diff --git a/js/net/src/broadcast.ts b/js/net/src/broadcast.ts index 29b05a6bba7..8526ec6be02 100644 --- a/js/net/src/broadcast.ts +++ b/js/net/src/broadcast.ts @@ -27,10 +27,12 @@ export interface Announcer { let attachAnnouncer: (producer: Producer, announcer: Announcer) => void; let stampProducer: (producer: Producer, path: Path.Valid) => void; -// Info lookups pending on a requested track. Each is demand, as a pending track request is in Rust, +// Info lookups pending or held on a track. Each is demand, as a pending track request is in Rust, // though nobody subscribes. class Lookups { pending = 0; + // Whether a lookup opened the request, so the last lookup to end closes it unless subscribed. + opened = false; // Re-evaluates the track's demand at once; set by watchDemand. changed = () => {}; @@ -92,6 +94,8 @@ function watchDemand(state: BroadcastState, producer: track.Producer, lookups?: const cleanup = () => { disposeUsed(); disposeClosed(); + // A removed track keeps its lookups, so a hold ending later must not count it again. + if (lookups?.changed === update) lookups.changed = () => {}; if (active) { state.active--; active = false; @@ -201,18 +205,45 @@ function subscribe(state: BroadcastState, name: string, options: track.Subscript return logical(state, name).producer.subscribe(options); } -async function resolveTrackInfo(state: BroadcastState, name: string): Promise { +// A pending lookup is demand, and so is a held one until `hold` aborts, which also abandons it before +// the answer. A request a lookup opened is let go once it is answered and nobody holds or subscribes +// to it; closing it sooner would hand its handler a closed producer. +async function resolveTrackInfo(state: BroadcastState, name: string, hold?: AbortSignal): Promise { + hold?.throwIfAborted(); const { producer, requested } = logical(state, name); const lookups = state.lookups.get(producer); - lookups?.add(1); - try { - return await producer.info(); - } finally { - lookups?.add(-1); - // A finished lookup is no demand: let go of a request it opened that nobody subscribed to - // meanwhile. - if (requested && !producer.demand().used.peek()) producer.close(); + if (!lookups) return producer.info(); + if (requested) lookups.opened = true; + lookups.add(1); + + const info = producer.info(); + let answered = false; + let ended = false; + const release = () => { + if (lookups.opened && lookups.pending === 0 && !producer.demand().used.peek()) producer.close(); + }; + info.then( + () => { + answered = true; + if (ended) release(); + }, + () => {}, + ); + const end = () => { + ended = true; + lookups.add(-1); + if (answered) release(); + }; + + if (!hold) { + try { + return await info; + } finally { + end(); + } } + hold.addEventListener("abort", end, { once: true }); + return untilAborted(info, hold); } // Serve a group from the local retained window by subscribing and scanning to the @@ -292,7 +323,7 @@ export class Producer { constructor() { registerWire(this, { subscribe: (name, options) => subscribe(this.#state, name, options), - resolveTrackInfo: (name) => resolveTrackInfo(this.#state, name), + resolveTrackInfo: (name, hold) => resolveTrackInfo(this.#state, name, hold), fetchGroup: (name, sequence, options) => fetchGroup(this.#state, name, sequence, options), requested: () => requested(this.#state), }); @@ -338,7 +369,9 @@ export class Producer { throw new Error(`duplicate track: ${track.name}`); } - watchDemand(this.#state, track); + const lookups = this.#state.lookups.get(track) ?? new Lookups(); + this.#state.lookups.set(track, lookups); + watchDemand(this.#state, track, lookups); this.#state.tracks.set(track.name, track); // A finished track keeps serving its cache, so only an abort evicts it. @@ -453,7 +486,7 @@ export class Consumer { this.#state.consumers++; registerWire(this, { subscribe: (name, options) => subscribe(this.#state, name, options), - resolveTrackInfo: (name) => resolveTrackInfo(this.#state, name), + resolveTrackInfo: (name, hold) => resolveTrackInfo(this.#state, name, hold), fetchGroup: (name, sequence, options) => fetchGroup(this.#state, name, sequence, options), requested: () => requested(this.#state), }); diff --git a/js/net/src/goaway-requests.test.ts b/js/net/src/goaway-requests.test.ts index 49aadb5b7c6..772fb7b5a39 100644 --- a/js/net/src/goaway-requests.test.ts +++ b/js/net/src/goaway-requests.test.ts @@ -3,7 +3,6 @@ import { Once } from "@moq/signals"; import { accept } from "./connection/accept.ts"; import { connect } from "./connection/connect.ts"; import type { Drain } from "./connection/goaway.ts"; -import { StreamCode } from "./error.ts"; import type { Session } from "./ietf/adapter.ts"; import { GoAway } from "./ietf/goaway.ts"; import { Subscriber } from "./ietf/subscriber.ts"; @@ -172,9 +171,9 @@ async function handover(kind: "lite" | "ietf"): Promise { const later = front.track("later").subscribe().ordered(); expect(await later.readString()).toBe("from-draining"); - // The publisher no longer holds group 0, so its answer proves the FETCH reached the wire. + // A fresh consumer caches nothing, so the publisher's answer proves the FETCH reached the wire. const fetched = wireOf(client).consume(Path.from("room")).track("video").fetchGroup(0); - if (kind === "lite") await expect(fetched).rejects.toMatchObject({ code: StreamCode.NotFound }); + if (kind === "lite") expect(await (await fetched).readString()).toBe("one"); else await expect(fetched).rejects.toThrow(/not supported/); // A new announce-interest opens too, and sees the route at the drain cost. diff --git a/js/net/src/integration.test.ts b/js/net/src/integration.test.ts index 8122b17b437..e4781b22ab4 100644 --- a/js/net/src/integration.test.ts +++ b/js/net/src/integration.test.ts @@ -161,7 +161,12 @@ test("integration: lite subscription options and updates reach the publisher", a const request = await wireOf(broadcast).requested(); if (!request) return; const producer = request.accept({ timescale: Timescale.MILLI }); - if (request.subscription.groups?.start?.included === 1) resolveProducer?.(producer); + // The TRACK lookup opens the request and the SUBSCRIBE joins it, so its options land after. + const check = (subscription = producer.subscription.peek()) => { + if (subscription?.groups?.start?.included === 1) resolveProducer?.(producer); + }; + check(); + producer.subscription.subscribe(check); } })(); @@ -219,7 +224,12 @@ test("integration: lite carries a fractional maxDelay as a whole millisecond", a const request = await wireOf(broadcast).requested(); if (!request) return; const producer = request.accept({ timescale: Timescale.MILLI }); - if (request.subscription.groups?.start?.included === 1) resolveProducer?.(producer); + // The TRACK lookup opens the request and the SUBSCRIBE joins it, so its options land after. + const check = (subscription = producer.subscription.peek()) => { + if (subscription?.groups?.start?.included === 1) resolveProducer?.(producer); + }; + check(); + producer.subscription.subscribe(check); } })(); @@ -364,6 +374,54 @@ test("integration: lite draft-06", async () => { await runPublishSubscribeFlow(Lite.ALPN_06); }); +// A subscriber learns a track's properties over a TRACK stream before it subscribes. The +// publisher holds the track for that stream until the subscriber closes it, which it does once +// its SUBSCRIBE is answered, so a publisher starting the track on demand sees one request and one +// demand edge per viewer rather than one for each stream with a gap in between. +test.each([Lite.ALPN_05, Lite.ALPN_06, Lite.ALPN_07_WIP])( + "integration: %s demand holds across a subscription's TRACK and SUBSCRIBE", + async (protocol) => { + const pair = createMockTransportPair(protocol); + const origin = new OriginProducer(); + const [client, server] = await Promise.all([ + connect(url, { transport: pair.client }), + accept(pair.server, url, { publish: origin.consume() }), + ]); + + const broadcast = publish(origin, Path.from("test")); + const demand = broadcast.demand(); + const edges: boolean[] = []; + const dispose = demand.used.subscribe((used) => edges.push(used)); + let requests = 0; + const serving = (async () => { + for (;;) { + const request = await wireOf(broadcast).requested(); + if (!request) return; + requests++; + request.accept({ timescale: Timescale.MILLI }).writeString("hello"); + } + })(); + + const remote = wireOf(client).consume(Path.from("test")); + const track = remote.track("video").subscribe().ordered(); + expect(await track.readString()).toBe("hello"); + await sleep(50); + expect(edges).toEqual([true]); + expect(requests).toBe(1); + + track.close(); + await withTimeout(demand.unused(), 1000, "demand outlived the reader"); + expect(edges).toEqual([true, false]); + + dispose(); + remote.close(); + broadcast.close(); + await serving; + client.abort(); + server.abort(); + }, +); + test("integration: lite draft-06 announce lifecycle", async () => { const pair = createMockTransportPair(Lite.ALPN_06); const origin = new OriginProducer(); diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index d5557cd3cf0..93f7a48348a 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -2,7 +2,7 @@ import { type Dispose, type Getter, race, Signal } from "@moq/signals"; import type * as broadcast from "../broadcast.ts"; import { Withdrawal } from "../connection/withdrawal.ts"; import * as DatagramStream from "../datagram_stream.ts"; -import { error, NotFound, reason, StreamCode, StreamError } from "../error.ts"; +import { error, NotFound, ProtocolViolation, reason, StreamCode, StreamError } from "../error.ts"; import type * as group from "../group.ts"; import { Cost, type Hop, type Route, routesEqual } from "../hop.ts"; import { hiddenBelow, hooks, presented } from "../internal.ts"; @@ -11,6 +11,7 @@ import type * as Path from "../path.ts"; import { type Reader, type Stream, Writer } from "../stream.ts"; import { Milli, Timescale, Timestamp } from "../time.ts"; import type * as track from "../track.ts"; +import { untilAborted } from "../util/abort.ts"; import { type Advertised, type Advertisements, wireOf } from "../wire.ts"; import { AnnounceInit, AnnounceOk, type AnnounceRequest, encodeAnnounceBroadcast } from "./announce.ts"; import { Datagram as DatagramMessage } from "./datagram.ts"; @@ -75,6 +76,17 @@ function acknowledged(writer: Writer): Promise { return writer.closed.catch(() => {}); } +/** A track's cached TRACK_INFO, and the request behind it while requesters hold it. */ +interface TrackInfoEntry { + info: Promise; + /** The requesters holding the request: open TRACK streams, and FETCHes awaiting the answer. */ + holders: number; + /** Whether the application answered, so a FETCH can reuse it after the request is let go. */ + answered: boolean; + /** Lets the request go once its last holder leaves, abandoning it if still unanswered. */ + release: AbortController; +} + /** What {@link Publisher.openGroup} and {@link Publisher.serveGroup} need to serve one group. */ interface RunGroup { /** The subscription ID. */ @@ -386,7 +398,7 @@ export class Publisher { // routing front rather than the path: immutability holds for one broadcast, and a // republish puts a different one on the path, so its entries must not be reused. // A rejected lookup is evicted so a retry can re-probe. - #trackInfo = new WeakMap>>(); + #trackInfo = new WeakMap>(); /** * Creates a new Publisher instance. @@ -735,11 +747,21 @@ export class Publisher { // response here, on the same scale as the group streams it competes with. stream.writer.setPriority(sendOrder({ priority: msg.priority })); + // A FETCH requester leaves early by resetting its stream or losing its session, which + // lets go of a lookup still waiting on the answer. + const hold = new AbortController(); + void stream.reader.closed.catch(() => hold.abort()); + let group: group.Consumer | undefined; try { // The timescale is immutable, so serve exactly what TRACK_INFO advertised. Both // come off the same front, so the metadata and the frames are one generation. - const info = await this.#resolveTrackInfo(front, msg.track); + let info: TrackInfoMessage; + try { + info = await this.#resolveTrackInfo(front, msg.track, hold.signal, true); + } finally { + hold.abort(); + } group = await wireOf(front).fetchGroup(msg.track, msg.group, { priority: msg.priority }); await this.#runFetchGroup(group, stream.writer, { timescale: Timescale(info.timescale), @@ -984,9 +1006,23 @@ export class Publisher { /** * Answers a TRACK stream (0x6) with a single TRACK_INFO, then FINs. * + * The open stream is interest in the track, held until the requester closes its side, + * so demand holds while the requester moves on to SUBSCRIBE. Only the reply is owed. + * * @internal */ async runTrackInfo(msg: TrackMessage, stream: Stream) { + const hold = new AbortController(); + // Watched from the start, so a requester leaving while the reply is still blocked on + // flow control lets go of the track too. + void stream.reader.done().then( + (fin) => { + hold.abort(); + // TRACK is the requester's only message. + if (!fin) stream.abort(new ProtocolViolation("data after TRACK")); + }, + () => hold.abort(), + ); try { const front = this.#publish && @@ -994,12 +1030,13 @@ export class Publisher { (await wireOf(this.#publish).demand(msg.broadcast, msg.epoch))); if (!front) throw new NotFound(`broadcast ${msg.broadcast}`); - const info = await this.#resolveTrackInfo(front, msg.track); + const info = await this.#resolveTrackInfo(front, msg.track, hold.signal); await info.encode(stream.writer, this.version); console.debug(`track info: broadcast=${msg.broadcast} track=${msg.track}`); - stream.close(); + stream.writer.close(); await acknowledged(stream.writer); } catch (err) { + hold.abort(); console.debug(`track unknown: broadcast=${msg.broadcast} track=${msg.track}`); stream.writer.reset(error(err)); } @@ -1008,34 +1045,67 @@ export class Publisher { // Resolve (and cache) a track's immutable TRACK_INFO by asking the application. // `resolveTrackInfo` triggers a TrackRequest the app answers with accept(TrackInfo); // only the immutable properties are needed (not the groups). Cached because they're - // fixed for the track's lifetime. Rejects if the track is unavailable. - #resolveTrackInfo(front: broadcast.Consumer, track: string): Promise { + // fixed for the track's lifetime. Rejects if the track is unavailable, or once `hold` + // aborts. Each requester holds the lookup until `hold` aborts, and the lookup ends with + // its last holder, answered or not; an unanswered one is dropped, so the next requester + // asks again. A TRACK stream is interest, so it always holds a live lookup, while a + // FETCH (`reuse`) only needs the answer and takes one a released lookup left behind. + #resolveTrackInfo( + front: broadcast.Consumer, + track: string, + hold: AbortSignal, + reuse = false, + ): Promise { + hold.throwIfAborted(); let tracks = this.#trackInfo.get(front); if (!tracks) { tracks = new Map(); this.#trackInfo.set(front, tracks); } - const cached = tracks.get(track); - if (cached !== undefined) return cached; - - const pending = (async () => { - const info = await wireOf(front).resolveTrackInfo(track); - return new TrackInfoMessage({ - priority: info.priority, - // Publisher Max Age: the publisher's retention bound, advertised so - // relays re-serve with the same window. - maxAge: info.maxAge, - // Lite05 mandates per-frame timestamps. Advertise the track's timescale; - // `#serveGroup` emits each frame converted to it. - timescale: wireTimescale(info), - }); - })(); + let entry = tracks.get(track); + if (reuse && entry?.answered) return entry.info; + if (!entry || entry.release.signal.aborted) { + const release = new AbortController(); + const info = (async () => { + const info = await wireOf(front).resolveTrackInfo(track, release.signal); + return new TrackInfoMessage({ + priority: info.priority, + // Publisher Max Age: the publisher's retention bound, advertised so + // relays re-serve with the same window. + maxAge: info.maxAge, + // Lite05 mandates per-frame timestamps. Advertise the track's timescale; + // `#serveGroup` emits each frame converted to it. + timescale: wireTimescale(info), + }); + })(); + + const created: TrackInfoEntry = { info, holders: 0, answered: false, release }; + info.then( + () => { + created.answered = true; + }, + // Don't poison the cache on failure: a later request may succeed. + () => { + if (tracks.get(track) === created) tracks.delete(track); + }, + ); + tracks.set(track, created); + entry = created; + } - // Don't poison the cache on failure: a later request may succeed. - pending.catch(() => tracks.delete(track)); - tracks.set(track, pending); - return pending; + const held = entry; + held.holders++; + hold.addEventListener( + "abort", + () => { + if (--held.holders > 0) return; + held.release.abort(); + if (!held.answered && tracks.get(track) === held) tracks.delete(track); + }, + { once: true }, + ); + return untilAborted(held.info, hold); } /** diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index 2eb652e50f1..0caaeb51bea 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -758,6 +758,8 @@ interface FakeStream { // Resolves once the subscriber waits on a read the test has not answered. reading: Promise; aborted: Promise; + // Resolves once the subscriber FINs its side. + finished: Promise; // Every chunk the subscriber wrote. written: Uint8Array[]; // Hands the stream to the subscriber, for an open the session was told to park. @@ -765,17 +767,21 @@ interface FakeStream { } // A session whose streams the test answers by hand and that never fails them on its own, so -// only Subscriber.close() can end a wait. Opens numbered in `park` wait for `release()`. -function fakeSession(park: number[] = []) { +// only Subscriber.close() can end a wait. Opens numbered in `park` wait for `release()`, and +// writes to those numbered in `stall` never get credit. +function fakeSession(park: number[] = [], stall: number[] = []) { const streams: FakeStream[] = []; const quic = { createBidirectionalStream: () => { let inbound!: ReadableStreamDefaultController; let onRead!: () => void; let onAbort!: (reason: unknown) => void; + let onFinish!: () => void; let release!: () => void; const reading = new Promise((resolve) => (onRead = resolve)); const aborted = new Promise((resolve) => (onAbort = resolve)); + const finished = new Promise((resolve) => (onFinish = resolve)); + const stalled = stall.includes(streams.length); // No high water mark, so pull() means the subscriber is blocked on a read. const readable = new ReadableStream( { @@ -788,12 +794,16 @@ function fakeSession(park: number[] = []) { ); const written: Uint8Array[] = []; const writable = new WritableStream({ - write: (chunk) => void written.push(chunk), + write: (chunk) => { + written.push(chunk); + return stalled ? new Promise(() => {}) : undefined; + }, + close: () => onFinish(), abort: (reason) => void onAbort(reason), }); const opened = new Promise((resolve) => (release = () => resolve({ readable, writable }))); if (!park.includes(streams.length)) release(); - streams.push({ inbound, reading, aborted, written, release }); + streams.push({ inbound, reading, aborted, finished, written, release }); return opened; }, } as unknown as WebTransport; @@ -941,6 +951,35 @@ test("a lite subscribe that times out waiting on a stream slot for the SUBSCRIBE } }); +// A SUBSCRIBE write blocked on flow control may never let the setup settle, so the deadline +// lets go of the held TRACK stream without waiting for it. +test("a lite subscribe whose SUBSCRIBE write stalls past the deadline closes its TRACK stream", async () => { + jest.useFakeTimers(); + const warn = spyOn(console, "warn").mockImplementation(() => {}); + const { quic, streams } = fakeSession([], [1]); + const subscriber = new Subscriber(quic, Version.DRAFT_05, HopSchema.parse(1n)); + try { + const track = subscriber.consume(Path.from("room")).track("video").subscribe(); + await drainUntil(() => streams.length === 1); + await streams[0].reading; + await answerTrackInfo(streams[0]); + await drainUntil(() => streams.length === 2 && streams[1].written.length > 0); + + let finished = false; + void streams[0].finished.then(() => { + finished = true; + }); + jest.advanceTimersByTime(SUBSCRIBE_SETUP_TIMEOUT_MS); + await drainUntil(() => track.closed.peek() !== undefined); + await drainUntil(() => finished); + expect(finished).toBe(true); + } finally { + subscriber.close(); + warn.mockRestore(); + jest.useRealTimers(); + } +}); + test("a fetch started after the subscriber closes rejects without opening a stream", async () => { const { quic, streams } = fakeSession(); const subscriber = new Subscriber(quic, Version.DRAFT_05, HopSchema.parse(1n)); diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 2b2f11e4bbe..726c93f0370 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -99,6 +99,10 @@ interface SubscribeEntry { end?: number; // Group streams opened by the publisher, when SUBSCRIBE_END carries the count. streams?: number; + // The TRACK stream that answered (lite-05+), held open as interest in the track until + // the SUBSCRIBE has its first response, so the publisher's demand never lapses between + // the two. + held?: Stream; } /** @@ -575,11 +579,13 @@ export class Subscriber { endGroup: inclusiveGroupEnd(bounds.end), }); - // Open the stream under a timeout. The stream handle flows back via `state` - // so the timeout path can abort it if it finishes opening after the deadline, + // Open the stream under a timeout. The stream handles flow back via `state` + // so the timeout path can close them if they finish opening after the deadline, // and `cancel` ends a setup still running once the deadline passed, resetting // its TRACK stream so a peer that never answers can't hold one per attempt. - const state: { stream?: Stream; cancel: AbortController } = { cancel: new AbortController() }; + const state: { stream?: Stream; track?: Stream; cancel: AbortController } = { + cancel: new AbortController(), + }; const setup = this.#openSubscribe(state, msg, request, id, timescale); let opened: { stream: Stream; entry: SubscribeEntry }; @@ -598,13 +604,16 @@ export class Subscriber { request.reject(e); this.#subscribes.delete(id); console.warn(`subscribe error: id=${id} broadcast=${broadcast} track=${request.name} error=${reason(e)}`); - // If the stream eventually opens after the timeout, abort it so we - // don't leak it. Cover both branches: setup may resolve late, or it - // may reject (e.g. encode/decode failure) after the stream is open. - setup.then( - () => state.stream?.abort(e), - () => state.stream?.abort(e), - ); + // Close the streams open now, since a write blocked on flow control may never let + // setup settle, and any that open after the timeout once it does. Cover both + // branches: setup may resolve late, or it may reject (e.g. encode/decode failure) + // after a stream is open. + const leave = () => { + state.stream?.abort(e); + state.track?.close(); + }; + leave(); + setup.then(leave, leave); return; } @@ -665,21 +674,23 @@ export class Subscriber { console.warn(`subscribe error: id=${id} broadcast=${broadcast} track=${request.name} error=${reason(e)}`); stream.abort(e); } finally { + entry.held?.close(); this.#subscribes.delete(id); } } // Determine the track's immutable properties, accept the request (so the // application's track.Subscriber resolves and incoming groups have a producer to - // write into), register it, then open the subscribe stream. `state.stream` is - // populated as soon as the subscribe stream opens so the caller can clean it up - // on timeout even before this promise settles. + // write into), register it, then open the subscribe stream. `state.stream` and + // `state.track` are populated as soon as each stream opens so the caller can clean + // them up on timeout even before this promise settles. // - // On lite-05+ the properties come from a TRACK stream opened first, and the - // SUBSCRIBE is accepted implicitly (no SUBSCRIBE_OK). Older drafts carry no - // per-track properties, so they resolve to defaults and just drain SUBSCRIBE_OK. + // On lite-05+ the properties come from a TRACK stream opened first, held open until + // the SUBSCRIBE is answered, and the SUBSCRIBE is accepted implicitly (no + // SUBSCRIBE_OK). Older drafts carry no per-track properties, so they resolve to + // defaults and just drain SUBSCRIBE_OK. async #openSubscribe( - state: { stream?: Stream; cancel: AbortController }, + state: { stream?: Stream; track?: Stream; cancel: AbortController }, msg: Subscribe, request: track.Request, id: bigint, @@ -690,7 +701,8 @@ export class Subscriber { if (supportsTrackStream(this.version)) { // Fetch the immutable properties once via the TRACK stream. - const info = await this.#trackInfo(msg.broadcast, msg.epoch, msg.track, state.cancel.signal); + const { info, stream } = await this.#trackInfo(msg.broadcast, msg.epoch, msg.track, state.cancel.signal); + state.track = stream; // The deadline passed as TRACK_INFO landed: the request is already rejected, so don't // register it again or send its SUBSCRIBE. state.cancel.signal.throwIfAborted(); @@ -717,6 +729,7 @@ export class Subscriber { }, }), requested: msg.startGroup, + held: state.track, }; this.#subscribes.set(id, entry); @@ -738,22 +751,21 @@ export class Subscriber { return { stream: state.stream, entry }; } - // Opens a TRACK stream, reads the single TRACK_INFO, and FINs. Lite-05+ only. + // Opens a TRACK stream and reads the single TRACK_INFO. Lite-05+ only. The stream + // stays open, as interest in the track, until the caller closes it. async #trackInfo( broadcast: Path.Valid, epoch: Epoch.Valid | undefined, track: string, signal?: AbortSignal, - ): Promise { + ): Promise<{ info: TrackInfo; stream: Stream }> { return this.#exchange( { version: this.version }, async (stream) => { await stream.writer.u53(StreamId.Track); await new TrackMessage(broadcast, track, epoch).encode(stream.writer, this.version); const info = await TrackInfo.decode(stream.reader, this.version); - // The publisher FINs after TRACK_INFO; FIN our side too. - stream.close(); - return info; + return { info, stream }; }, signal, ); @@ -792,12 +804,22 @@ export class Subscriber { // Resolve a track's immutable model info via a TRACK stream (lite-05+), for the // ConsumeBroadcast backing track.Consumer.query(). On older drafts there's no TRACK - // stream, so this rejects rather than fabricating defaults. - async resolveTrackInfo(broadcast: Path.Valid, track: string, epoch?: Epoch.Valid): Promise { + // stream, so this rejects rather than fabricating defaults. The TRACK stream stays + // open until `hold` aborts, so the publisher keeps the track for whoever asked here; + // aborting it before TRACK_INFO resets the stream. + async resolveTrackInfo( + broadcast: Path.Valid, + track: string, + epoch?: Epoch.Valid, + hold?: AbortSignal, + ): Promise { if (!supportsTrackStream(this.version)) { throw new Error("track info requires moq-lite-05 or newer"); } - return this.#toModelInfo(await this.#trackInfo(broadcast, epoch, track)); + const { info, stream } = await this.#trackInfo(broadcast, epoch, track, hold); + if (hold && !hold.aborted) hold.addEventListener("abort", () => stream.close(), { once: true }); + else stream.close(); + return this.#toModelInfo(info); } // Open a FETCH stream for one group and stream its bare frames into a group, for the @@ -894,7 +916,13 @@ export class Subscriber { priority: number, group: netGroup.Producer, ): Promise<{ stream: Stream; info: TrackInfo }> { - const info = await untilClosed(group, this.#trackInfo(broadcast, epoch, track)); + const answered = this.#trackInfo(broadcast, epoch, track); + // Only the properties are needed, so the TRACK stream closes however the wait ends. + answered.then( + ({ stream }) => stream.close(), + () => {}, + ); + const { info } = await untilClosed(group, answered); return this.#exchange({ sendOrder: sendOrder({ priority }), version: this.version }, async (stream) => { await stream.writer.u53(StreamId.Fetch); await new FetchMessage({ broadcast, epoch, track, priority, group: sequence }).encode( @@ -968,6 +996,9 @@ export class Subscriber { async #runResponses(stream: Stream, entry: SubscribeEntry): Promise { for (;;) { const resp = await decodeSubscribeResponseMaybe(stream.reader, this.version); + // The publisher answered, so its demand stands on the subscription from here. + entry.held?.close(); + entry.held = undefined; if (!resp) { if (entry.end === undefined) throw new ProtocolViolation("subscribe stream ended without SUBSCRIBE_END"); @@ -1369,7 +1400,7 @@ class ConsumeBroadcast extends broadcast.Consumer { constructor(subscriber: Subscriber, path: Path.Valid, epoch: Epoch.Valid | undefined, state?: never) { super(state); overrideBroadcastWire(this, { - resolveTrackInfo: (name) => subscriber.resolveTrackInfo(path, name, epoch), + resolveTrackInfo: (name, hold) => subscriber.resolveTrackInfo(path, name, epoch, hold), fetchGroup: (name, sequence, options) => subscriber.fetchGroup(path, name, sequence, options, epoch), }); this.#subscriber = subscriber; diff --git a/js/net/src/lite/track_stream.test.ts b/js/net/src/lite/track_stream.test.ts new file mode 100644 index 00000000000..c230c2f31e0 --- /dev/null +++ b/js/net/src/lite/track_stream.test.ts @@ -0,0 +1,384 @@ +// An open Track Stream is interest in the track: the subscriber keeps it open until its +// SUBSCRIBE is answered, and the publisher keeps the track wanted until the subscriber closes it. +import { describe, expect, test } from "bun:test"; +import { randomHop } from "../hop.ts"; +import { createMockTransportPair } from "../mock.ts"; +import { Producer as OriginProducer } from "../origin.ts"; +import * as Path from "../path.ts"; +import { Stream } from "../stream.ts"; +import { Milli, Timescale } from "../time.ts"; +import { wireOf } from "../wire.ts"; +import { Fetch } from "./fetch.ts"; +import { Publisher } from "./publisher.ts"; +import { StreamId } from "./stream.ts"; +import { encodeSubscribeResponse, Subscribe, SubscribeStart } from "./subscribe.ts"; +import { Subscriber } from "./subscriber.ts"; +import { TrackInfo, Track as TrackMessage } from "./track.ts"; +import { ALPN_05, Version } from "./version.ts"; + +/** Whether `promise` settles within `ms`. */ +async function settlesWithin(promise: Promise, ms: number): Promise { + let timer: ReturnType | undefined; + const pending = new Promise((resolve) => { + timer = setTimeout(() => resolve(false), ms); + }); + try { + return await Promise.race([promise.then(() => true), pending]); + } finally { + clearTimeout(timer); + } +} + +describe.each([Version.DRAFT_05, Version.DRAFT_06, Version.DRAFT_07])("%s", (version) => { + /** A subscriber whose publisher the test plays by hand, up to the SUBSCRIBE. */ + async function subscribing() { + const pair = createMockTransportPair(ALPN_05); + const subscriber = new Subscriber(pair.client, version, randomHop()); + const reader = subscriber + .consume(Path.from("room")) + .track("video") + .subscribe({ maxDelay: Milli(100) }); + + const track = await Stream.accept(pair.server, version); + if (!track) throw new Error("the subscriber never asked for TRACK_INFO"); + expect(await track.reader.u53()).toBe(StreamId.Track); + await TrackMessage.decode(track.reader, version); + await new TrackInfo({}).encode(track.writer, version); + track.writer.close(); + + const sub = await Stream.accept(pair.server, version); + if (!sub) throw new Error("the subscriber never subscribed"); + expect(await sub.reader.u53()).toBe(StreamId.Subscribe); + await Subscribe.decode(sub.reader, version); + return { subscriber, reader, track, sub }; + } + + test("a subscription holds its TRACK stream until the SUBSCRIBE is answered", async () => { + const { subscriber, reader, track, sub } = await subscribing(); + const fin = track.reader.done(); + expect(await settlesWithin(fin, 50)).toBe(false); + + await encodeSubscribeResponse(sub.writer, { start: new SubscribeStart(0) }, version); + expect(await fin).toBe(true); + + reader.close(); + subscriber.close(); + }); + + test("a subscription that ends unanswered closes its TRACK stream", async () => { + const { subscriber, reader, track } = await subscribing(); + const fin = track.reader.done(); + expect(await settlesWithin(fin, 50)).toBe(false); + + reader.close(); + expect(await fin).toBe(true); + subscriber.close(); + }); + + test("an info query closes its TRACK stream once answered", async () => { + const pair = createMockTransportPair(ALPN_05); + const subscriber = new Subscriber(pair.client, version, randomHop()); + const info = subscriber.consume(Path.from("room")).track("video").info(); + + const track = await Stream.accept(pair.server, version); + if (!track) throw new Error("the subscriber never asked for TRACK_INFO"); + expect(await track.reader.u53()).toBe(StreamId.Track); + await TrackMessage.decode(track.reader, version); + await new TrackInfo({ priority: 3 }).encode(track.writer, version); + track.writer.close(); + + expect((await info).priority).toBe(3); + expect(await track.reader.done()).toBe(true); + subscriber.close(); + }); + + test("a held info lookup abandoned before TRACK_INFO resets its TRACK stream", async () => { + const pair = createMockTransportPair(ALPN_05); + const subscriber = new Subscriber(pair.client, version, randomHop()); + const hold = new AbortController(); + const info = subscriber.resolveTrackInfo(Path.from("room"), "video", undefined, hold.signal); + + const track = await Stream.accept(pair.server, version); + if (!track) throw new Error("the subscriber never asked for TRACK_INFO"); + expect(await track.reader.u53()).toBe(StreamId.Track); + await TrackMessage.decode(track.reader, version); + + hold.abort(); + await expect(info).rejects.toThrow(); + await expect(track.reader.done()).rejects.toThrow(); + subscriber.close(); + }); + + test.each(["FIN", "reset"] as const)( + "an answered TRACK stream keeps the track wanted until the requester's %s", + async (close) => { + const pair = createMockTransportPair(ALPN_05); + const origin = new OriginProducer(); + const publisher = new Publisher(pair.server, version, randomHop(), origin.consume()); + const broadcast = origin.createBroadcast(Path.from("room")); + broadcast.announce(); + const demand = broadcast.demand(); + const serving = (async () => { + for (;;) { + const request = await wireOf(broadcast).requested(); + if (!request) return; + request.accept({ timescale: Timescale.MILLI }); + } + })(); + + const client = await Stream.open(pair.client, { version }); + const server = await Stream.accept(pair.server, version); + if (!server) throw new Error("the publisher never accepted the TRACK stream"); + void publisher.runTrackInfo(new TrackMessage(Path.from("room"), "video"), server); + + await TrackInfo.decode(client.reader, version); + expect(await client.reader.done()).toBe(true); + expect(await settlesWithin(demand.unused(), 50)).toBe(false); + + if (close === "FIN") client.writer.close(); + else client.writer.reset(new Error("gone")); + expect(await settlesWithin(demand.unused(), 1000)).toBe(true); + + publisher.close(); + broadcast.close(); + await serving; + origin.close(); + }, + ); + + test.each(["FIN", "reset"] as const)( + "a requester's %s while TRACK_INFO is blocked lets the track go", + async (close) => { + const pair = createMockTransportPair(ALPN_05); + const origin = new OriginProducer(); + const publisher = new Publisher(pair.server, version, randomHop(), origin.consume()); + const broadcast = origin.createBroadcast(Path.from("room")); + broadcast.announce(); + const demand = broadcast.demand(); + const serving = (async () => { + for (;;) { + const request = await wireOf(broadcast).requested(); + if (!request) return; + request.accept({ timescale: Timescale.MILLI }); + } + })(); + + const client = await Stream.open(pair.client, { version }); + const incoming = pair.server.incomingBidirectionalStreams.getReader(); + const accepted = await incoming.read(); + incoming.releaseLock(); + if (accepted.done) throw new Error("the publisher never saw the TRACK stream"); + // A send side that never gets credit, so the reply stays blocked. + let blocked!: () => void; + const writing = new Promise((resolve) => (blocked = resolve)); + const writable = new WritableStream({ + write: () => { + blocked(); + return new Promise(() => {}); + }, + }); + const server = new Stream({ readable: accepted.value.readable, writable, version }); + void publisher.runTrackInfo(new TrackMessage(Path.from("room"), "video"), server); + + await writing; + expect(demand.used.peek()).toBe(true); + + if (close === "FIN") client.writer.close(); + else client.writer.reset(new Error("gone")); + expect(await settlesWithin(demand.unused(), 1000)).toBe(true); + + publisher.close(); + broadcast.close(); + await serving; + origin.close(); + }, + ); + + test("one TRACK stream closing leaves the track wanted while another holds it", async () => { + const pair = createMockTransportPair(ALPN_05); + const origin = new OriginProducer(); + const publisher = new Publisher(pair.server, version, randomHop(), origin.consume()); + const broadcast = origin.createBroadcast(Path.from("room")); + broadcast.announce(); + const demand = broadcast.demand(); + let requests = 0; + const serving = (async () => { + for (;;) { + const request = await wireOf(broadcast).requested(); + if (!request) return; + requests++; + request.accept({ timescale: Timescale.MILLI }); + } + })(); + + const open = async () => { + const client = await Stream.open(pair.client, { version }); + const server = await Stream.accept(pair.server, version); + if (!server) throw new Error("the publisher never accepted the TRACK stream"); + void publisher.runTrackInfo(new TrackMessage(Path.from("room"), "video"), server); + return client; + }; + const first = await open(); + const second = await open(); + await TrackInfo.decode(first.reader, version); + await TrackInfo.decode(second.reader, version); + expect(requests).toBe(1); + + first.writer.close(); + expect(await settlesWithin(demand.unused(), 50)).toBe(false); + + second.writer.close(); + expect(await settlesWithin(demand.unused(), 1000)).toBe(true); + + publisher.close(); + broadcast.close(); + await serving; + origin.close(); + }); + + test("a TRACK stream after the cached lookup was let go holds the track again", async () => { + const pair = createMockTransportPair(ALPN_05); + const origin = new OriginProducer(); + const publisher = new Publisher(pair.server, version, randomHop(), origin.consume()); + const broadcast = origin.createBroadcast(Path.from("room")); + broadcast.announce(); + const demand = broadcast.demand(); + let requests = 0; + const serving = (async () => { + for (;;) { + const request = await wireOf(broadcast).requested(); + if (!request) return; + requests++; + request.accept({ timescale: Timescale.MILLI }); + } + })(); + + const open = async () => { + const client = await Stream.open(pair.client, { version }); + const server = await Stream.accept(pair.server, version); + if (!server) throw new Error("the publisher never accepted the TRACK stream"); + void publisher.runTrackInfo(new TrackMessage(Path.from("room"), "video"), server); + return client; + }; + const first = await open(); + await TrackInfo.decode(first.reader, version); + first.writer.close(); + expect(await settlesWithin(demand.unused(), 1000)).toBe(true); + + const second = await open(); + await TrackInfo.decode(second.reader, version); + expect(requests).toBe(2); + expect(await settlesWithin(demand.unused(), 50)).toBe(false); + + second.writer.close(); + expect(await settlesWithin(demand.unused(), 1000)).toBe(true); + + publisher.close(); + broadcast.close(); + await serving; + origin.close(); + }); + + /** Opens a TRACK or FETCH stream for room/video, served by `publisher`. */ + async function requester(pair: ReturnType, publisher: Publisher, kind: string) { + const client = await Stream.open(pair.client, { version }); + const server = await Stream.accept(pair.server, version); + if (!server) throw new Error(`the publisher never accepted the ${kind} stream`); + if (kind === "TRACK") void publisher.runTrackInfo(new TrackMessage(Path.from("room"), "video"), server); + else + void publisher.runFetch( + new Fetch({ broadcast: Path.from("room"), track: "video", priority: 0, group: 0 }), + server, + ); + return client; + } + + test.each(["TRACK", "FETCH"])( + "%s requesters leaving before the answer let the track go once the last one leaves", + async (kind) => { + const pair = createMockTransportPair(ALPN_05); + const origin = new OriginProducer(); + const publisher = new Publisher(pair.server, version, randomHop(), origin.consume()); + const broadcast = origin.createBroadcast(Path.from("room")); + broadcast.announce(); + const demand = broadcast.demand(); + // Answered by hand, after every requester left. + const requested = wireOf(broadcast).requested(); + + const first = await requester(pair, publisher, kind); + const request = await requested; + if (!request) throw new Error("the publisher never asked for the track"); + const second = await requester(pair, publisher, kind); + + first.writer.reset(new Error("gone")); + expect(await settlesWithin(demand.unused(), 50)).toBe(false); + + second.writer.reset(new Error("gone")); + expect(await settlesWithin(demand.unused(), 1000)).toBe(true); + + // The handler still gets an open track, and the answer lets it go. + const producer = request.accept({ timescale: Timescale.MILLI }); + expect(producer.closed.peek()).toBeUndefined(); + expect(await settlesWithin(Promise.resolve(producer.closed), 1000)).toBe(true); + + publisher.close(); + broadcast.close(); + origin.close(); + }, + ); + + test.each(["TRACK", "FETCH"])("a lost session lets go of a %s still waiting on the answer", async (kind) => { + const pair = createMockTransportPair(ALPN_05); + const origin = new OriginProducer(); + const publisher = new Publisher(pair.server, version, randomHop(), origin.consume()); + const broadcast = origin.createBroadcast(Path.from("room")); + broadcast.announce(); + const demand = broadcast.demand(); + const requested = wireOf(broadcast).requested(); + + await requester(pair, publisher, kind); + const request = await requested; + if (!request) throw new Error("the publisher never asked for the track"); + expect(demand.used.peek()).toBe(true); + + pair.server.close(); + expect(await settlesWithin(demand.unused(), 1000)).toBe(true); + + request.reject(); + publisher.close(); + broadcast.close(); + origin.close(); + }); + + test("a TRACK stream after every earlier one left an unanswered lookup holds the track again", async () => { + const pair = createMockTransportPair(ALPN_05); + const origin = new OriginProducer(); + const publisher = new Publisher(pair.server, version, randomHop(), origin.consume()); + const broadcast = origin.createBroadcast(Path.from("room")); + broadcast.announce(); + const demand = broadcast.demand(); + // Answered by hand, after the first TRACK stream has left. + const requested = wireOf(broadcast).requested(); + + const first = await requester(pair, publisher, "TRACK"); + const request = await requested; + if (!request) throw new Error("the publisher never asked for the track"); + first.writer.reset(new Error("gone")); + expect(await settlesWithin(demand.unused(), 1000)).toBe(true); + + // The next TRACK stream joins the request still waiting on the application. + const second = await requester(pair, publisher, "TRACK"); + expect(demand.used.peek()).toBe(true); + expect(await settlesWithin(wireOf(broadcast).requested(), 50)).toBe(false); + request.accept({ timescale: Timescale.MILLI }); + await TrackInfo.decode(second.reader, version); + expect(await settlesWithin(demand.unused(), 50)).toBe(false); + + second.writer.close(); + expect(await settlesWithin(demand.unused(), 1000)).toBe(true); + + publisher.close(); + broadcast.close(); + origin.close(); + }); +}); diff --git a/js/net/src/wire.ts b/js/net/src/wire.ts index d94751806ad..75ae92e0a45 100644 --- a/js/net/src/wire.ts +++ b/js/net/src/wire.ts @@ -20,7 +20,8 @@ import type * as track from "./track.ts"; /** The protocol-facing operations behind a broadcast handle. */ export interface Broadcast { subscribe(name: string, options?: track.Subscription): track.Subscriber; - resolveTrackInfo(name: string): Promise; + /** The track's properties; `hold` keeps the track wanted until it aborts, the way a held TRACK stream does, and aborting it before the answer abandons the lookup. */ + resolveTrackInfo(name: string, hold?: AbortSignal): Promise; fetchGroup(name: string, sequence: number, options?: track.FetchGroupOptions): Promise; requested(): Promise; /** diff --git a/quest/m0/README.md b/quest/m0/README.md index c985900ce1f..9428970e7af 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -24,8 +24,7 @@ serve instead of enumerating broadcasts. Serving the relay's ingested-only view (`origin::Consumer::local()`) to localhost workers belongs to moq.pro's edge, which embeds moq-relay; it moved there on 2026-09-28. Pools of claim workers (transcoders) also need the relay to forget a front -nobody reads, and demand that doesn't flap on a viewer's first subscribe -(both found 2026-10-07). +nobody reads (found 2026-10-07). Interop: Fastly's moq-relay-interop report (run of 2026-09-23, build 7ee2b02) was triaged against `main` on 2026-10-07. Its SETUP, UNSUBSCRIBE @@ -62,7 +61,6 @@ remain and no release waits on them. - [Draft-22 media on 0.17](/quest/m0/release-22/README.md) - a 0.17.x with the LOCATION_FILTER and FIRST_OBJECT fixes and moq-noq 1.3.4, before Seattle - [Capped stream END_OF_GROUP](/quest/m0/ietf-end-of-track-location.md) - a stream capped by the subscription's end Location never claims END_OF_GROUP; moving End of Track's Location is deferred - [End of Group status](/quest/m0/ietf-end-of-group-status.md) - an End of Group status on a stream whose header already marks the group's end is accepted, so imquic's last object per group arrives -- [TRACK stream demand](/quest/m0/track-stream-demand.md) - a lite-05+ subscribe shows the publisher one `used` edge, not a flap between TRACK and SUBSCRIBE - [web-transport releases the qmux fixes](/quest/m0/qmux-credit-upstream.md) - waiting on moq-dev/web-transport#412 and #413 to merge and ship, which qmux credit bumps to - [qmux credit](/quest/m0/qmux-credit.md) - qmux returns connection credit for dropped and stopped streams and delivers its close frame, on both lines - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh route epoch, the newest wins a path, and only routes with the same epoch resume a subscription diff --git a/quest/m0/track-stream-demand.md b/quest/m0/track-stream-demand.md deleted file mode 100644 index 70bbc9fe4b9..00000000000 --- a/quest/m0/track-stream-demand.md +++ /dev/null @@ -1,88 +0,0 @@ -# [M] Demand holds across a subscription's TRACK and SUBSCRIBE - -## Goal - -On moq-lite 05 and later, a publisher sees one `used` edge when a viewer -subscribes, not `used`, `unused`, `used`. Today demand drops for one round -trip per hop between the TRACK request and the SUBSCRIBE that follows it. A -publisher that stops work on `broadcast::Demand::unused`, which is what -`Demand` documents itself for, closes its output on every first viewer and is -asked to create it again a few milliseconds later; a transcoder tears down its -encoders and waits for the next keyframe. It happens under a prefix claim and -for an exact announce alike. lite-04 and moq-transport don't do this. Rust, -JS, and the lite draft. - -The other edge holds too: in `@moq/net`, a TRACK or FETCH requester that -resets its stream or loses its session before the answer releases its hold, -and `Broadcast.Demand` drops once no requester is left, as in Rust. - -## Plan - -Facts from `main`: - -- The publisher answers TRACK with a track `query()`, which counts as demand - on the track and its broadcast, and drops it once TRACK_INFO is written - (`lite/publisher.rs`, `TrackInfoServe`). At the publisher's front it - arrives as an `Asked` with no subscription to hold; the `Asked::sub` - comment covers moq-transport's case, not this one. -- The subscriber sends SUBSCRIBE only after reading TRACK_INFO. At a relay, - the upstream SUBSCRIBE waits for the downstream reader's own SUBSCRIBE, so - the gap grows by a round trip per hop. JS subscribers also await TRACK_INFO - before SUBSCRIBE. -- A relay's own copy flaps too, but `track::IDLE_LINGER` hides that from - everything downstream. Only an end publisher sees the raw edge. - -Decisions (2026-10-07): - -- An open TRACK stream counts as interest. The publisher keeps its query - until the requester FINs or resets the TRACK stream. The subscriber FINs it - once its SUBSCRIBE stream has its first response (the start, a group, an - end, or a reset), or once its own demand goes (an info-only query). The hold - chains across hops, so demand stays continuous end to end. It is backward - compatible: an older subscriber FINs at once, as today. -- Holding only until SUBSCRIBE is sent isn't enough: QUIC doesn't order the - two streams, and a relay that handles the TRACK FIN before the SUBSCRIBE - parks the track and drops its copy at once, letting go upstream. Waiting for - the first response costs one stream held for about a round trip. -- A held TRACK stream is demand without a subscription, so it counts against - the per-session subscription cap (`session::Limits`, from #4820); a peer - can't hold more interest than it could by subscribing. A held TRACK and the - SUBSCRIBE for the same track on the same session share one slot: the SUBSCRIBE takes over the - TRACK's reservation instead of needing a second, so a session at the cap - can still turn its held TRACKs into subscriptions (decided 2026-10-08 from - review). -- Not pipelining TRACK and SUBSCRIBE, which the draft already allows: it - still races, and every hop would have to buffer frames until TRACK_INFO. -- Consumers never debounce demand; the docs promise clean edges. -- JS query lifetime (folded in from the JS probe-lifetime quest, 2026-10-08: - the same publisher plumbing). Rust's query is a consumer of the track state - that drops with the serve (`TrackInfoServe`). In JS, `resolveTrackInfo` - (`js/net/src/broadcast.ts`) pins demand for as long as the lookup runs, - whoever still wants it. Give it a cancel signal, and have the lite - publisher (`runTrackInfo`, `runFetch`, and `#resolveTrackInfo` in - `js/net/src/lite/publisher.ts`) release its hold when the TRACK or FETCH - stream is reset; the shared per-front query ends with its last holder. - Also from #4956's review: `removeTrack` on a name cached only by a - `consume()` subscription is outside its documented contract; document or - refuse it. -- Scope: Rust and JS, plus one sentence in the draft's Track Stream section. - Run `just test interop --all`. - -Verification: a `moq-net` integration test on the simulated network (10 ms -latency) asserting exactly one `used` edge and no `unused` while the reader -stays subscribed, on lite-05, 06, and 07, direct and through one relay. It -fails on `main` today. The relay case controls the ordering so the -downstream TRACK FIN is handled before its SUBSCRIBE. A boundary test fills -the per-session subscription cap with held TRACK streams, turns each into a -live SUBSCRIBE with no `unused` edge, and checks that one more TRACK closes -the session with TOO_MANY_REQUESTS. JS counterparts for the JS side, plus a -JS requester (TRACK or FETCH) that disconnects before the answer and drops -broadcast demand, while a second requester keeps it pinned until it leaves -too. - -Public API: none. Wire: semantics only (holding the TRACK stream open), no new -fields. - -## Related - -- [#4225](https://github.com/moq-dev/moq/pull/4225) - holds a lite subscription's demand in the same publisher code; expect a conflict diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 5d0c8b2e2cf..d777c681766 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -66,7 +66,7 @@ struct Shared { goaway: crate::goaway::Protocol, // Control streams still serving the peer data, which a draining close waits for. owed: AtomicUsize, - subscriptions: crate::session::Slots, + interests: Interests, // The send time an untimed track's frames carry, since no lite version can mark them // untimed yet (see `wire_timestamp`). runtime: crate::time::Clock, @@ -173,7 +173,7 @@ impl Publisher { version: config.version, goaway: config.goaway, owed: AtomicUsize::new(0), - subscriptions: config.subscriptions, + interests: Interests::new(config.subscriptions), runtime: config.runtime.clone(), }), runtime: config.runtime, @@ -198,7 +198,6 @@ where shared: self.shared.clone(), runtime: self.runtime.clone(), state: ControlState::Start { stream }, - _slot: None, }); } Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), @@ -243,8 +242,6 @@ struct Control { // Handed to the children that arm timers (PROBE, announce linger). runtime: crate::time::Clock, state: ControlState, - // A subscription's place under the session's cap, held until the stream ends. - _slot: Option, } // A state machine's enum is its storage: one transient instance per stream, so the @@ -267,21 +264,6 @@ enum ControlState { Done, } -impl ControlState { - /// Whether this serve owes the peer data until it ends on its own. - fn owes(&self) -> bool { - matches!(self, Self::Subscribe(_) | Self::Fetch(_) | Self::TrackInfo(_)) - } -} - -impl Drop for Control { - fn drop(&mut self) { - if self.state.owes() { - self.shared.owed.fetch_sub(1, Ordering::Relaxed); - } - } -} - impl kio::Task for Control { type Output = (); @@ -308,20 +290,9 @@ impl Control { lite::ControlType::Announce => { ControlState::Announce(AnnounceServe::new(self.shared.clone(), stream)) } - lite::ControlType::Subscribe => match self.shared.subscriptions.acquire() { - Ok(slot) => { - self._slot = Some(slot); - ControlState::Subscribe(RequestServe::new(self.shared.clone(), stream)) - } - // A peer past its limits loses the session, not just this request. - Err(err) => { - self.shared - .session - .clone() - .close(crate::SessionError::from(&err).to_code(), "too many subscriptions"); - return Poll::Ready(Err(err)); - } - }, + lite::ControlType::Subscribe => { + ControlState::Subscribe(RequestServe::new(self.shared.clone(), stream)) + } // The Track Stream and FETCH are lite-05+ only. lite::ControlType::Fetch | lite::ControlType::Track if !self.shared.version.has_track_stream() => @@ -338,9 +309,6 @@ impl Control { lite::ControlType::Goaway => ControlState::Goaway { stream }, lite::ControlType::Session => return Poll::Ready(Err(Error::UnexpectedStream)), }; - if self.state.owes() { - self.shared.owed.fetch_add(1, Ordering::Relaxed); - } } ControlState::Announce(serve) => return serve.poll(waiter), ControlState::Subscribe(serve) => return serve.poll(waiter), @@ -993,6 +961,11 @@ trait Request: Sized { /// Whether we answer the requester's FIN with our own. Otherwise we reset: the reply /// would end short, and a FIN there would read as a whole one. const GRACEFUL: bool; + /// What the request counts as against the session's subscription cap while it is open. + const CHARGE: Option; + /// Whether the request stays open past its reply, keeping what serving it took, until + /// the requester closes its side. + const HOLDS: bool; /// The message that opens the stream. type Message: Decode + std::fmt::Debug; /// What the requester may send after the request; [`NoUpdate`] for nothing. @@ -1042,6 +1015,10 @@ struct RequestServe> { // Taken to abort. stream: Option>, state: RequestState, + // Counted in `Shared::owed` until the reply is delivered, so a draining close waits for it. + owing: bool, + // The request's place under the session's subscription cap. + _charged: Option, // The latest update to arrive before the reply started. Each one replaces the last. update: Option, // Log context, filled in after the decode. @@ -1058,6 +1035,13 @@ enum RequestState { requesting: Option, }, Serve(R), + /// FIN the reply, and keep the request until the requester closes its side. + Hold { + // Held for what serving it took (TRACK: the query). + _request: R, + finished: bool, + acked: bool, + }, /// FIN and wait for the acknowledgement. Finish { finished: bool, @@ -1066,16 +1050,26 @@ enum RequestState { impl> RequestServe { fn new(shared: Arc>, stream: Stream) -> Self { + shared.owed.fetch_add(1, Ordering::Relaxed); Self { shared, stream: Some(stream), state: RequestState::Decode, + owing: true, + _charged: None, update: None, absolute: Default::default(), track: Default::default(), } } + /// The reply no longer owes the peer anything a draining close should wait for. + fn settle(&mut self) { + if std::mem::take(&mut self.owing) { + self.shared.owed.fetch_sub(1, Ordering::Relaxed); + } + } + fn poll(&mut self, waiter: &kio::Waiter) -> Poll> { match ready!(self.poll_serve(waiter)) { Ok(()) => { @@ -1121,6 +1115,19 @@ impl> RequestServe { let (broadcast, track) = R::target(&msg); self.absolute = self.shared.origin.absolute(broadcast).to_owned(); self.track = track.to_string(); + if let Some(charge) = R::CHARGE { + match self.shared.interests.charge(broadcast, track, charge) { + Ok(charged) => self._charged = Some(charged), + // A peer past its limits loses the session, not just this request. + Err(err) => { + self.shared + .session + .clone() + .close(crate::SessionError::from(&err).to_code(), "too many subscriptions"); + return Poll::Ready(Err(err)); + } + } + } tracing::info!(broadcast = %self.absolute, track = %self.track, request = ?msg, "{} started", R::KIND); self.state = RequestState::Resolve { msg, requesting: None }; } @@ -1149,7 +1156,51 @@ impl> RequestServe { } RequestState::Serve(request) => { if ready!(request.poll(&self.shared, &mut stream.writer, waiter))?.is_break() { - self.state = RequestState::Finish { finished: false }; + let RequestState::Serve(request) = + std::mem::replace(&mut self.state, RequestState::Finish { finished: false }) + else { + unreachable!() + }; + if R::HOLDS { + self.state = RequestState::Hold { + _request: request, + finished: false, + acked: false, + }; + } + } + } + RequestState::Hold { finished, acked, .. } => { + if !*finished { + // A reply blocked on flow control may never unblock, so a requester + // leaving before it is out cancels it and lets go of the request. + if stream.writer.poll_flush(&mut cx)?.is_pending() { + let closed = ready!(self.poll_requester(&mut cx)); + self.state = RequestState::Finish { finished: false }; + closed?; + return Poll::Ready(Err(Error::Cancel)); + } + stream.writer.finish()?; + *finished = true; + } + // The reply is the peer's once acknowledged; the hold itself owes nothing. + let delivered = *acked || stream.writer.poll_close(&mut cx).is_ready(); + *acked = delivered; + if delivered { + self.settle(); + } + let closed = ready!(self.poll_requester(&mut cx)); + // Let go of the request now, not when the serve is dropped. + self.state = RequestState::Finish { finished: true }; + // A FIN or a reset both end the requester's interest; only a stray byte + // is a fault. + if let Err(err) = closed + && !is_cancel(&err) + { + return Poll::Ready(Err(err)); + } + if delivered { + return Poll::Ready(Ok(())); } } RequestState::Finish { finished } => { @@ -1186,7 +1237,112 @@ impl> RequestServe { } } +impl> Drop for RequestServe { + fn drop(&mut self) { + self.settle(); + } +} + +/// What a request counts as against the session's subscription cap. +#[derive(Clone, Copy)] +enum Charge { + /// A TRACK stream, held open as interest in the track. + Track, + /// A subscription. + Subscribe, +} + +/// A session's subscription cap, charged per track. +/// +/// A TRACK stream held open and the SUBSCRIBE that follows it are one interest in the +/// track, so a track costs the larger of its two counts: a peer at the cap can still +/// subscribe to the tracks it holds. +struct Interests { + slots: crate::session::Slots, + tracks: kio::Lock>, +} + +/// The requests open on one track, and the slots they take. +#[derive(Default)] +struct Interest { + tracks: usize, + subscribes: usize, + slots: Vec, +} + +impl Interest { + fn count(&mut self, charge: Charge) -> &mut usize { + match charge { + Charge::Track => &mut self.tracks, + Charge::Subscribe => &mut self.subscribes, + } + } + + fn cost(&self) -> usize { + self.tracks.max(self.subscribes) + } +} + +impl Interests { + fn new(slots: crate::session::Slots) -> Self { + Self { + slots, + tracks: kio::Lock::new(HashMap::new()), + } + } + + /// Charge a request for `broadcast`'s `track` until the returned guard drops, or + /// [`Error::TooManyRequests`] past the cap. + fn charge(&self, broadcast: &crate::Path, track: &str, charge: Charge) -> Result { + let key = (broadcast.to_owned(), track.to_string()); + let mut tracks = self.tracks.lock(); + let interest = tracks.entry(key.clone()).or_default(); + *interest.count(charge) += 1; + if interest.cost() > interest.slots.len() { + match self.slots.acquire() { + Ok(slot) => interest.slots.push(slot), + Err(err) => { + *interest.count(charge) -= 1; + if interest.cost() == 0 { + tracks.remove(&key); + } + return Err(err); + } + } + } + Ok(Charged { + tracks: self.tracks.clone(), + key, + charge, + }) + } +} + +/// One request's charge on its track, released on drop. +struct Charged { + tracks: kio::Lock>, + key: (crate::PathOwned, String), + charge: Charge, +} + +impl Drop for Charged { + fn drop(&mut self) { + let mut tracks = self.tracks.lock(); + let Some(interest) = tracks.get_mut(&self.key) else { + return; + }; + *interest.count(self.charge) -= 1; + interest.slots.truncate(interest.cost()); + if interest.cost() == 0 { + tracks.remove(&self.key); + } + } +} + /// Answers a TRACK stream with the track's TRACK_INFO. +/// +/// The open stream is interest in the track: the query that answered it stays until the +/// requester closes its side, so demand holds while the requester moves on to SUBSCRIBE. struct TrackInfoServe { querying: track::Querying, } @@ -1194,6 +1350,8 @@ struct TrackInfoServe { impl Request for TrackInfoServe { const KIND: &'static str = "track info"; const GRACEFUL: bool = false; + const CHARGE: Option = Some(Charge::Track); + const HOLDS: bool = true; type Message = lite::Track<'static>; type Update = NoUpdate; @@ -1257,6 +1415,8 @@ impl Request for SubscribeServe { const KIND: &'static str = "subscribed"; // Ending a subscription early is still a valid end. const GRACEFUL: bool = true; + const CHARGE: Option = Some(Charge::Subscribe); + const HOLDS: bool = false; type Message = lite::Subscribe<'static>; type Update = lite::SubscribeUpdate; @@ -1419,6 +1579,8 @@ impl Request for FetchServe { const KIND: &'static str = "fetch"; // The response carries no header, so a run cut short would read as whole. const GRACEFUL: bool = false; + const CHARGE: Option = None; + const HOLDS: bool = false; type Message = lite::Fetch<'static>; type Update = NoUpdate; @@ -3750,7 +3912,7 @@ mod serve_group_test { #[cfg(test)] mod tests { use super::*; - use crate::lite::test_transport::{Close, ScriptedSession, SinkSession}; + use crate::lite::test_transport::{Close, ScriptedSession, SinkSend, SinkSession}; use crate::model::ProduceTest; use futures::FutureExt; @@ -4236,6 +4398,86 @@ mod tests { } } + /// An answered TRACK stream holds its track until the requester closes its side, by + /// FIN or reset, and the hold owes nothing a draining close would wait for. A requester + /// that leaves while the reply is still blocked on flow control lets the track go too. + #[moq_net_sim::test] + async fn an_answered_track_stream_holds_the_track_until_the_requester_closes() { + for version in [Version::Lite05, Version::Lite06, Version::Lite07] { + for close in [Close::Fin, Close::Reset] { + for blocked in [false, true] { + let case = format!("{version:?}, {close:?}, blocked {blocked}"); + let origin = crate::origin::Config::new(crate::Hop::new(1).unwrap()).produce(); + let broadcast = origin.create_broadcast("room").unwrap(); + let track = broadcast.create_track("video", None).unwrap(); + broadcast.announce(Default::default()).unwrap(); + + let peer_setup = crate::lite::PeerSetup::default(); + peer_setup.set(crate::lite::Setup::default()); + let (_, goaway) = crate::goaway::Handle::new(true); + let publisher = Publisher::new(PublisherConfig { + runtime: crate::time::Clock::sim(), + session: ScriptedSession::new(Vec::new()), + origin: origin.consume(), + version, + peer_setup, + goaway, + peer_hop: None, + subscriptions: Default::default(), + }); + + let mut script = Vec::new(); + lite::Track { + epoch: None, + broadcast: crate::Path::new("room"), + track: "video".into(), + } + .encode(&mut crate::coding::Encoder::new(&mut script, version.into()), version) + .unwrap(); + let mut session = ScriptedSession::new(script); + let (_, recv) = futures::future::poll_fn(|cx| { + ::poll_open_bi(&mut session, cx) + }) + .await + .unwrap(); + // A closed gate is a reply blocked on flow control, never reopened. + let gate = kio::Producer::new(!blocked); + let stream = Stream:: { + writer: Writer::new(SinkSend::gated(session.log.clone(), gate.consume()), version), + reader: crate::coding::Reader::new(recv, version), + }; + let mut serve = RequestServe::<_, TrackInfoServe>::new(publisher.shared.clone(), stream); + + assert!(!drive(&mut serve).await, "{case}: ended before the requester closed"); + assert!( + matches!(serve.state, RequestState::Hold { finished, .. } if finished != blocked), + "{case}: TRACK_INFO was not answered as expected" + ); + assert!(track.demand().is_used(), "{case}: the track was let go"); + assert_eq!( + publisher.shared.owed.load(Ordering::Relaxed), + usize::from(blocked), + "{case}: only an undelivered reply owes" + ); + + session.close(close); + assert!(drive(&mut serve).await, "{case}: still holding"); + moq_net_sim::timeout(Duration::from_millis(1), track.demand().unused()) + .await + .unwrap_or_else(|_| panic!("{case}: the track is still wanted")) + .unwrap(); + // The incomplete reply is cancelled; a delivered one is not. + assert_eq!( + session.log.resets().is_empty(), + !blocked, + "{case}: {:?}", + session.log.resets() + ); + } + } + } + } + /// SUBSCRIBE_UPDATE is not a close: an update arriving while the subscription /// resolves keeps it waiting, and still applies once it starts. #[moq_net_sim::test] @@ -4313,39 +4555,104 @@ mod tests { assert_eq!(*run.track_priority_tx.read(), 7, "the update was lost"); } - /// A subscription past the session's cap closes the session with TOO_MANY_REQUESTS. + /// A SUBSCRIBE or a TRACK past the session's cap closes the session with + /// TOO_MANY_REQUESTS. #[moq_net_sim::test] - async fn subscriptions_past_the_cap_close_the_session() { + async fn requests_past_the_cap_close_the_session() { use crate::coding::Encode as _; const VERSION: Version = Version::Lite06; - let mut script = Vec::new(); - lite::ControlType::Subscribe - .encode(&mut crate::coding::Encoder::new(&mut script, VERSION.into()), VERSION) - .unwrap(); - let session = crate::lite::test_transport::ScriptedSession::new(Vec::new()).with_incoming_bidis(vec![script]); - let log = session.log.clone(); - - let origin = Hop::random().produce(); - let (_, goaway) = crate::goaway::Handle::new(true); - let mut publisher = Publisher::new(PublisherConfig { - runtime: crate::time::Clock::sim(), - session, - origin: origin.consume(), - version: VERSION, - peer_setup: crate::lite::PeerSetup::default(), - goaway, - peer_hop: None, - subscriptions: crate::session::Slots::new(0), + let encode = |kind: lite::ControlType, msg: &dyn Fn(&mut Vec)| { + let mut script = Vec::new(); + kind.encode(&mut crate::coding::Encoder::new(&mut script, VERSION.into()), VERSION) + .unwrap(); + msg(&mut script); + script + }; + let subscribe = encode(lite::ControlType::Subscribe, &|script| { + lite::Subscribe { + epoch: None, + id: 0, + broadcast: crate::Path::new("room"), + track: "video".into(), + priority: 0, + max_delay: Duration::ZERO, + start_group: None, + end_group: None, + start_frame: 0, + end_frame: None, + } + .encode(&mut crate::coding::Encoder::new(script, VERSION.into()), VERSION) + .unwrap() + }); + let track = encode(lite::ControlType::Track, &|script| { + lite::Track { + epoch: None, + broadcast: crate::Path::new("room"), + track: "video".into(), + } + .encode(&mut crate::coding::Encoder::new(script, VERSION.into()), VERSION) + .unwrap() }); - let _ = publisher.poll(&kio::Waiter::noop()); - assert_eq!( - log.closes(), - vec![( - crate::SessionError::TooManyRequests.to_code(), - "too many subscriptions".to_string() - )] - ); + for script in [subscribe, track] { + let session = + crate::lite::test_transport::ScriptedSession::new(Vec::new()).with_incoming_bidis(vec![script]); + let log = session.log.clone(); + + let origin = Hop::random().produce(); + let (_, goaway) = crate::goaway::Handle::new(true); + let mut publisher = Publisher::new(PublisherConfig { + runtime: crate::time::Clock::sim(), + session, + origin: origin.consume(), + version: VERSION, + peer_setup: crate::lite::PeerSetup::default(), + goaway, + peer_hop: None, + subscriptions: crate::session::Slots::new(0), + }); + + let _ = publisher.poll(&kio::Waiter::noop()); + assert_eq!( + log.closes(), + vec![( + crate::SessionError::TooManyRequests.to_code(), + "too many subscriptions".to_string() + )] + ); + } + } + + /// A TRACK held open and the SUBSCRIBE for the same track are one interest, whichever + /// arrives first, while another track costs a slot of its own. + #[test] + fn a_track_and_its_subscription_share_a_slot() { + let slots = crate::session::Slots::new(1); + let interests = Interests::new(slots.clone()); + let room = crate::Path::new("room"); + + for (first, second) in [(Charge::Track, Charge::Subscribe), (Charge::Subscribe, Charge::Track)] { + let a = interests.charge(&room, "video", first).expect("the first fits"); + let b = interests + .charge(&room, "video", second) + .expect("shares the first's slot"); + assert!(matches!( + interests.charge(&room, "audio", Charge::Track), + Err(Error::TooManyRequests) + )); + // Either leaving keeps the slot for the other. + drop(a); + assert!(slots.acquire().is_err(), "the remaining request lost its slot"); + drop(b); + drop(slots.acquire().expect("the slot was not returned")); + } + + // Two of a kind are two interests. + let _a = interests.charge(&room, "video", Charge::Subscribe).unwrap(); + assert!(matches!( + interests.charge(&room, "video", Charge::Subscribe), + Err(Error::TooManyRequests) + )); } } diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 1771f9e69e4..f7c8ed0470d 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -4144,11 +4144,13 @@ impl kio::Task for TrackServeRun { let res = ready!(info.poll_fetch(&self.serve, waiter)); let request = request.take().expect("request pending"); match res { - Ok(info) => { + Ok((info, held)) => { // Lite05 carries per-frame timestamps on the wire at this scale; // `Some` tells the ingest to decode them. let timescale = info.timescale; - self.state = TrackRunState::Serve(ServeLoop::new(&self.serve, request, info, timescale)); + let mut serve_loop = ServeLoop::new(&self.serve, request, info, timescale); + serve_loop.held = Some(held); + self.state = TrackRunState::Serve(serve_loop); } Err(err) => { tracing::warn!(broadcast = %self.serve.subscriber.log_path(&self.serve.path), track = %self.serve.name, %err, "track info failed"); @@ -4226,7 +4228,8 @@ impl Drop for TrackServeRun { } /// Opens a TRACK stream, reads the single TRACK_INFO, and maps it to the -/// model's [`track::Info`]. Lite05+ only. Bails if the session dies meanwhile. +/// model's [`track::Info`], handing back the stream to hold. Lite05+ only. Bails if the +/// session dies meanwhile. struct TrackInfoFetch { session: S, // A dedicated close-watch handle for the read. @@ -4250,7 +4253,11 @@ impl TrackInfoFetch { } } - fn poll_fetch(&mut self, serve: &TrackServe, waiter: &kio::Waiter) -> Poll> { + fn poll_fetch( + &mut self, + serve: &TrackServe, + waiter: &kio::Waiter, + ) -> Poll), Error>> { let mut cx = waiter.context(); loop { match &mut self.state { @@ -4277,9 +4284,10 @@ impl TrackInfoFetch { return Poll::Ready(Err(Error::from_transport(err))); } let info = ready!(stream.reader.poll_decode::(&mut cx))?; - // The publisher FINs after TRACK_INFO; FIN our side too and let the - // stream drop. - let _ = stream.writer.finish(); + let TrackInfoState::Read { stream } = std::mem::replace(&mut self.state, TrackInfoState::Open) + else { + unreachable!() + }; // Publisher Max Age rides on the wire, so the local retention // window matches what the upstream advertises (relays re-serve with @@ -4289,13 +4297,25 @@ impl TrackInfoFetch { .with_timescale(info.timescale) .with_max_age(info.max_age) .with_priority(info.priority); - return Poll::Ready(Ok(model)); + return Poll::Ready(Ok((model, HeldTrack(stream)))); } } } } } +/// An answered TRACK stream, kept open as interest in the track: the publisher holds the +/// track for us until we FIN it, which dropping this does. Held until the SUBSCRIBE that +/// follows has its first response, or until nobody wants this copy, so demand never +/// lapses between the two streams at any hop. +struct HeldTrack(Stream); + +impl Drop for HeldTrack { + fn drop(&mut self) { + let _ = self.0.writer.finish(); + } +} + /// The serve loop proper: owns this session's copy of the track (pumped into /// the origin's logical track), driving the single upstream subscription /// (opened lazily on the first downstream subscriber, canceled when the last @@ -4323,6 +4343,9 @@ struct ServeLoop { /// Armed while nobody holds the copy: it stays, cache and all, for a returning /// reader until this fires. linger: crate::time::Deadline, + /// The TRACK stream that answered, until a SUBSCRIBE has a response or nobody holds + /// the copy. + held: Option>, } // A state machine's enum is its storage: one transient instance per stream, so the @@ -4370,6 +4393,7 @@ impl ServeLoop { timescale, mode: ServeMode::Select, linger: crate::time::Deadline::new(&serve.subscriber.runtime), + held: None, } } @@ -4478,6 +4502,7 @@ impl ServeLoop { // reader or fetch that asks again soon, then is dropped. In-flight // fetches keep it alive: work already accepted still gets finished. if self.fetches.is_empty() && self.demand.poll_unused(waiter).is_ready() { + self.held = None; if self.linger.deadline().is_none() { let now = serve.subscriber.runtime.now(); self.linger.set(now.checked_add(track::IDLE_LINGER)); @@ -4499,6 +4524,11 @@ impl ServeLoop { .reader .poll_decode_maybe::(&mut cx) { + // The publisher answered the SUBSCRIBE, so its demand stands on the + // subscription now. A bare `track::Consumer` still held after that + // subscription ends keeps local `Demand` used while the publisher sees + // unused: accepted, since nothing holds a handle that way. + self.held = None; match res { Ok(Some(msg)) => { match &msg { diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 782941d2463..f63794fd6a1 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -2734,10 +2734,11 @@ pub(crate) enum DemandState { /// A handle to a single track within a broadcast. /// -/// Obtained from [`broadcast::Consumer::track`]. Holding it sends nothing -/// to the publisher; it just names a track you can [`subscribe`](Self::subscribe) -/// to (a live, ongoing stream of groups) later. The same handle can be subscribed -/// to multiple times, and clones are cheap. +/// Obtained from [`broadcast::Consumer::track`]. Holding it counts toward the +/// track's [`Demand`], but starts no live delivery until you [`subscribe`](Self::subscribe) +/// to it (a live, ongoing stream of groups); [`fetch_group`](Self::fetch_group) takes one +/// group without that. The same handle can be subscribed to multiple times, and clones +/// are cheap. #[derive(Clone)] pub struct Consumer { name: Arc, diff --git a/rs/moq-net/src/session.rs b/rs/moq-net/src/session.rs index 0d79a9c5e9e..a072ca33520 100644 --- a/rs/moq-net/src/session.rs +++ b/rs/moq-net/src/session.rs @@ -658,7 +658,8 @@ impl Drop for Withdrawing { pub struct Limits { /// Broadcasts (moq-lite) or namespaces (moq-transport) the peer may have announced to us. pub announces: usize, - /// Subscriptions the peer may hold on our broadcasts. + /// Subscriptions the peer may hold on our broadcasts. On moq-lite 05 and later an open + /// TRACK stream holds one too, sharing it with a SUBSCRIBE for the same track. pub subscriptions: usize, } diff --git a/rs/moq-net/tests/datagram.rs b/rs/moq-net/tests/datagram.rs index ffcddd02130..907cd647abe 100644 --- a/rs/moq-net/tests/datagram.rs +++ b/rs/moq-net/tests/datagram.rs @@ -50,6 +50,9 @@ async fn connect_datagram_track() -> Fixture { consumer.routed("bench").await.unwrap(); let remote = consumer.request_broadcast("bench", None).await.unwrap(); let subscriber = remote.track("datagrams").unwrap().subscribe(None).await.unwrap(); + // The subscriber is handed over once TRACK_INFO answers, before its SUBSCRIBE reaches + // the publisher, and a datagram only goes to a subscription already there. + support::harness::subscribed(&producer).await; Fixture { producer, diff --git a/rs/moq-net/tests/finished_broadcast_mock.rs b/rs/moq-net/tests/finished_broadcast_mock.rs index bdf7267968b..cf0b33ad0af 100644 --- a/rs/moq-net/tests/finished_broadcast_mock.rs +++ b/rs/moq-net/tests/finished_broadcast_mock.rs @@ -77,10 +77,9 @@ async fn round(finish_broadcast: bool) -> (Vec>, Option) } }); - moq_net_sim::timeout(TIMEOUT, track.demand().used()) + moq_net_sim::timeout(TIMEOUT, support::harness::subscribed(&track)) .await - .expect("no subscriber appeared") - .unwrap(); + .expect("no subscriber appeared"); // One frame and a textbook clean end, innermost first, with no await in between: // nothing is served until the ending is done. diff --git a/rs/moq-net/tests/session_close.rs b/rs/moq-net/tests/session_close.rs index 31d70bc6a8f..389b8f1f403 100644 --- a/rs/moq-net/tests/session_close.rs +++ b/rs/moq-net/tests/session_close.rs @@ -71,10 +71,9 @@ async fn setup(version: &str) -> Setup { } }); - moq_net_sim::timeout(TIMEOUT, track.demand().used()) + moq_net_sim::timeout(TIMEOUT, support::harness::subscribed(&track)) .await - .expect("no subscriber appeared") - .unwrap(); + .expect("no subscriber appeared"); Setup { pair, diff --git a/rs/moq-net/tests/support/harness.rs b/rs/moq-net/tests/support/harness.rs index 59877a1cf75..7ea37f3d393 100644 --- a/rs/moq-net/tests/support/harness.rs +++ b/rs/moq-net/tests/support/harness.rs @@ -30,6 +30,13 @@ pub fn spawn(mut driver: D) { } } +/// Wait for a subscription to reach `track`. Demand alone does not say one has: on +/// moq-lite 05 and later a TRACK stream is demand before its SUBSCRIBE arrives. +pub async fn subscribed(track: &moq_net::track::Producer) { + let mut track = track.clone(); + while track.subscription_changed().await.expect("track open").is_none() {} +} + /// Options for [`connect_mock`]. pub struct MockConnectOptions { /// The MoQ version to negotiate (determines the ALPN protocol string). diff --git a/rs/moq-net/tests/support/mock.rs b/rs/moq-net/tests/support/mock.rs index 0b0a62c0eb9..c8ec82bef58 100644 --- a/rs/moq-net/tests/support/mock.rs +++ b/rs/moq-net/tests/support/mock.rs @@ -446,6 +446,8 @@ struct SessionSide { conn: Arc, /// Uni streams this side opened that the peer has not accepted yet, while held. held: Mutex>>, + /// Bidi streams this side opened that the peer has not accepted yet, while held. + held_bidis: Mutex>>, /// Whether the peer has withheld uni stream credit, parking every open. withheld: Mutex, /// While set, the uni streams this side opens hold back everything after their first @@ -513,6 +515,11 @@ impl poll::Session for MockSession { our_send.withhold_fin = self.side.withhold_bidi_fins.clone(); let (peer_send, our_recv) = new_stream_pair(&self.side.conn); + if let Some(held) = self.side.held_bidis.lock().unwrap().as_mut() { + held.push((peer_send, peer_recv)); + return Poll::Ready(Ok((our_send, our_recv))); + } + // Deliver (peer_send, peer_recv) to the peer's accept_bi. match self.side.peer_bidi.try_push((peer_send, peer_recv)) { Ok(()) => Poll::Ready(Ok((our_send, our_recv))), @@ -655,6 +662,21 @@ impl MockSession { self.side.held.lock().unwrap().take(); } + /// Hold back the bidi streams this side opens from now on, until + /// [`Self::release_bidis`]: QUIC does not order streams, so the peer may see a later + /// stream's data before an earlier stream arrives at all. + pub fn hold_bidis(&self) { + self.side.held_bidis.lock().unwrap().get_or_insert_default(); + } + + /// Deliver the held bidi streams to the peer in the order they were opened, and stop + /// holding. + pub fn release_bidis(&self) { + for stream in self.side.held_bidis.lock().unwrap().take().unwrap_or_default() { + let _ = self.side.peer_bidi.try_push(stream); + } + } + /// Park every uni stream this side opens from now on, like a peer that has granted /// no more stream credit. pub fn withhold_unis(&self) { @@ -724,6 +746,7 @@ pub fn create_mock_session_pair(protocol: Option<&'static str>) -> (MockSession, protocol, conn: conn.clone(), held: Mutex::default(), + held_bidis: Mutex::default(), withheld: Mutex::default(), split: Mutex::default(), lossy: Mutex::default(), @@ -740,6 +763,7 @@ pub fn create_mock_session_pair(protocol: Option<&'static str>) -> (MockSession, protocol, conn, held: Mutex::default(), + held_bidis: Mutex::default(), withheld: Mutex::default(), split: Mutex::default(), lossy: Mutex::default(), diff --git a/rs/moq-net/tests/track_stream_demand.rs b/rs/moq-net/tests/track_stream_demand.rs new file mode 100644 index 00000000000..a0718206c7d --- /dev/null +++ b/rs/moq-net/tests/track_stream_demand.rs @@ -0,0 +1,247 @@ +//! A subscription's demand holds across its TRACK and SUBSCRIBE streams. +//! +//! On moq-lite 05 and later a subscriber learns a track's properties over a TRACK stream +//! before it subscribes. The publisher counts that stream as interest until the subscriber +//! closes it, and the subscriber keeps it open until its SUBSCRIBE is answered, so a +//! publisher watching its demand sees one `used` edge for a viewer, not `used`, `unused`, +//! `used` with a round trip per hop in between. + +mod support; + +use std::{cell::RefCell, rc::Rc, time::Duration}; + +use moq_net::{Hop, Timestamp, Version, origin}; +use support::{ + harness::{MockConnectOptions, MockPair, connect_mock}, + mock::MockSession, +}; + +const TIMEOUT: Duration = Duration::from_secs(10); +const LATENCY: Duration = Duration::from_millis(10); +const VERSIONS: [&str; 3] = ["moq-lite-05", "moq-lite-06", "moq-lite-07-wip"]; +/// How long the publisher takes to start a track once asked, like a transcoder spinning up. +const STARTUP: Duration = Duration::from_millis(5); + +fn produce_origin(hop: u64) -> origin::Producer { + let (producer, driver) = origin::Producer::new(origin::Config::new(Hop::new(hop).unwrap())); + support::harness::spawn(driver); + producer +} + +/// `subscriber` reads what `publisher` serves, over a link with [`LATENCY`] each way. +async fn link(version: Version, publisher: &origin::Producer, subscriber: &origin::Producer) -> MockPair { + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(publisher.consume()); + options.client_subscribe = Some(subscriber.clone()); + options.latency = LATENCY; + connect_mock(options).await +} + +/// Every demand edge a `broadcast::Demand` or `track::Demand` reports from here on, +/// `true` for used. +macro_rules! record { + ($demand:expr) => {{ + let edges = Rc::new(RefCell::new(Vec::new())); + let log = edges.clone(); + let demand = $demand; + drop(moq_net_sim::spawn(async move { + while demand.used().await.is_ok() { + log.borrow_mut().push(true); + if demand.unused().await.is_err() { + break; + } + log.borrow_mut().push(false); + } + })); + edges + }}; +} + +/// A reader subscribes through `relays` relays and stays: the publisher, which starts +/// the track on demand, is asked for it once, and its broadcast and track each see exactly +/// one `used` edge, then `unused` once the reader leaves. +/// +/// With `reorder`, the reader's SUBSCRIBE stream reaches the first hop well after its +/// TRACK stream's data, as QUIC may deliver them, so a hop that let go of the TRACK +/// before its SUBSCRIBE arrived would drop its demand in between. +async fn one_used_edge(version: &str, relays: usize, reorder: bool) { + let version: Version = version.parse().unwrap(); + let label = format!("{version} over {relays} relays, reorder={reorder}"); + + let nodes: Vec<_> = (1..=relays as u64 + 2).map(produce_origin).collect(); + let mut links = Vec::new(); + for pair in nodes.windows(2) { + links.push(link(version, &pair[0], &pair[1]).await); + } + let reader: &MockSession = &links.last().unwrap().client_transport; + + let broadcast = nodes[0].create_broadcast("room").unwrap(); + let mut dynamic = broadcast.dynamic(); + broadcast.announce(Default::default()).unwrap(); + let broadcast_edges = record!(broadcast.demand()); + + // Serve every request for the track, keeping what it started. + let requests = Rc::new(RefCell::new(Vec::new())); + let track_edges = Rc::new(RefCell::new(None)); + let served = (requests.clone(), track_edges.clone()); + let publisher = moq_net_sim::spawn(async move { + while let Ok(request) = dynamic.requested_track().await { + served.1.borrow_mut().get_or_insert_with(|| record!(request.demand())); + moq_net_sim::sleep(STARTUP).await; + let track = request.accept(None); + let mut group = track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, b"key".as_ref()).unwrap(); + served.0.borrow_mut().push((track, group)); + } + }); + + let consumer = nodes.last().unwrap().consume(); + moq_net_sim::timeout(TIMEOUT, consumer.routed("room")) + .await + .unwrap_or_else(|_| panic!("{label}: never announced")) + .unwrap(); + let remote = consumer.request_broadcast("room", None).await.unwrap(); + + let subscribing = remote.track("video").unwrap().subscribe(None); + if reorder { + // The TRACK stream opens at once; the SUBSCRIBE only once TRACK_INFO is back. + moq_net_sim::sleep(Duration::from_millis(1)).await; + reader.hold_bidis(); + } + let mut subscriber = moq_net_sim::timeout(TIMEOUT, subscribing) + .await + .unwrap_or_else(|_| panic!("{label}: subscribe stalled")) + .unwrap_or_else(|err| panic!("{label}: subscribe failed: {err}")); + if reorder { + moq_net_sim::sleep(LATENCY * 20).await; + reader.release_bidis(); + } + + let mut received = moq_net_sim::timeout(TIMEOUT, subscriber.recv_group()) + .await + .unwrap_or_else(|_| panic!("{label}: no group arrived")) + .unwrap() + .expect("the track is live"); + received.read_frame().await.unwrap().expect("the group has a frame"); + + // Stay subscribed while every hop settles. + moq_net_sim::sleep(Duration::from_secs(2)).await; + assert_eq!(requests.borrow().len(), 1, "{label}: asked for the track again"); + let track_edges = track_edges.borrow().clone().expect("the track was requested"); + assert_eq!(*broadcast_edges.borrow(), [true], "{label}: broadcast demand"); + assert_eq!(*track_edges.borrow(), [true], "{label}: track demand"); + + // Holding the TRACK stream must not outlive the reader. + drop(received); + drop(subscriber); + let track = requests.borrow()[0].0.clone(); + moq_net_sim::timeout(TIMEOUT, track.demand().unused()) + .await + .unwrap_or_else(|_| panic!("{label}: demand outlived the reader")) + .unwrap(); + moq_net_sim::sleep(Duration::from_millis(100)).await; + assert_eq!(*broadcast_edges.borrow(), [true, false], "{label}: broadcast demand"); + assert_eq!(*track_edges.borrow(), [true, false], "{label}: track demand"); + publisher.abort(); +} + +#[moq_net_sim::test] +async fn direct() { + for version in VERSIONS { + one_used_edge(version, 0, false).await; + one_used_edge(version, 0, true).await; + } +} + +#[moq_net_sim::test] +async fn through_a_relay() { + for version in VERSIONS { + one_used_edge(version, 1, false).await; + one_used_edge(version, 1, true).await; + } +} + +/// Held TRACK streams count against the server's subscription cap, and each SUBSCRIBE +/// takes over its TRACK's place: a client at the cap still subscribes to every track it +/// holds without a demand gap, and one more TRACK closes the session. +#[moq_net_sim::test] +async fn held_tracks_share_the_cap_with_their_subscriptions() { + const CAP: usize = 3; + + for version in VERSIONS { + let version: Version = version.parse().unwrap(); + let server = produce_origin(1); + let client = produce_origin(2); + + let broadcast = server.create_broadcast("room").unwrap(); + let mut tracks = Vec::new(); + for i in 0..=CAP { + let track = broadcast.create_track(format!("t{i}"), None).unwrap(); + let mut group = track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, b"key".as_ref()).unwrap(); + let edges = record!(track.demand()); + tracks.push((track, group, edges)); + } + broadcast.announce(Default::default()).unwrap(); + + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(server.consume()); + options.client_subscribe = Some(client.clone()); + options.latency = LATENCY; + let mut limits = moq_net::session::Limits::default(); + limits.subscriptions = CAP; + options.server_limits = Some(limits); + let pair = connect_mock(options).await; + + let remote = moq_net_sim::timeout(TIMEOUT, client.consume().routed_broadcast("room")) + .await + .expect("announce timeout") + .unwrap(); + + // A held handle keeps each track wanted, and so its TRACK stream open. + let mut held = Vec::new(); + for i in 0..CAP { + let track = remote.track(&format!("t{i}")).unwrap(); + moq_net_sim::timeout(TIMEOUT, track.query()) + .await + .unwrap_or_else(|_| panic!("{version}: t{i} info stalled")) + .unwrap_or_else(|err| panic!("{version}: t{i} info refused: {err}")); + held.push(track); + } + moq_net_sim::sleep(Duration::from_secs(1)).await; + for (i, (.., edges)) in tracks.iter().enumerate().take(CAP) { + assert_eq!(*edges.borrow(), [true], "{version}: t{i} is not held"); + } + + let mut subscribers = Vec::new(); + for (i, track) in held.iter().enumerate() { + let mut subscriber = moq_net_sim::timeout(TIMEOUT, track.subscribe(None)) + .await + .unwrap_or_else(|_| panic!("{version}: t{i} subscribe stalled")) + .unwrap_or_else(|err| panic!("{version}: t{i} refused at the cap: {err}")); + moq_net_sim::timeout(TIMEOUT, subscriber.recv_group()) + .await + .unwrap_or_else(|_| panic!("{version}: t{i} delivered nothing")) + .unwrap() + .expect("the track is live"); + subscribers.push(subscriber); + } + moq_net_sim::sleep(Duration::from_secs(1)).await; + for (i, (.., edges)) in tracks.iter().enumerate().take(CAP) { + assert_eq!(*edges.borrow(), [true], "{version}: t{i} demand"); + } + + // The subscriptions alone fill the cap now. + let refused = moq_net_sim::timeout(TIMEOUT, remote.track(&format!("t{CAP}")).unwrap().query()) + .await + .expect("info timeout"); + assert!(refused.is_err(), "{version}: one more TRACK was admitted"); + let err = moq_net_sim::timeout(TIMEOUT, pair.client.closed()) + .await + .unwrap_or_else(|_| panic!("{version}: the session stayed open")); + assert!( + matches!(err, moq_net::Error::Session(moq_net::SessionError::TooManyRequests)), + "{version}: closed with {err}" + ); + } +} diff --git a/rs/moq-net/tests/track_tail.rs b/rs/moq-net/tests/track_tail.rs index 5f9409e0906..b938b0430d9 100644 --- a/rs/moq-net/tests/track_tail.rs +++ b/rs/moq-net/tests/track_tail.rs @@ -134,10 +134,9 @@ async fn round(version: &str, late: Late, final_sequence: u64) -> Outcome { (frames, err, moq_net_sim::now()) }); - moq_net_sim::timeout(TIMEOUT, track.demand().used()) + moq_net_sim::timeout(TIMEOUT, support::harness::subscribed(&track)) .await - .expect("no subscriber appeared") - .unwrap(); + .expect("no subscriber appeared"); pair.server_transport.hold_unis(); if final_sequence > 0 { @@ -267,10 +266,9 @@ async fn ietf_leaving_cancels_a_blocked_end_of_track() { .subscribe(subscription) .await .expect("subscribe"); - moq_net_sim::timeout(TIMEOUT, track.demand().used()) + moq_net_sim::timeout(TIMEOUT, support::harness::subscribed(&track)) .await - .expect("no subscriber appeared") - .unwrap(); + .expect("no subscriber appeared"); let mut group = track.append_group().unwrap(); group.write_frame(Timestamp::ZERO, PAYLOAD).unwrap(); @@ -332,10 +330,9 @@ async fn a_lost_datagram_never_delays_the_end() { } (groups, moq_net_sim::now()) }); - moq_net_sim::timeout(TIMEOUT, track.demand().used()) + moq_net_sim::timeout(TIMEOUT, support::harness::subscribed(&track)) .await - .expect("no subscriber appeared") - .unwrap(); + .expect("no subscriber appeared"); for datagram in [false, true, false] { if datagram { diff --git a/rs/moq-transcode/src/feed.rs b/rs/moq-transcode/src/feed.rs index ea4b98dc9b2..2b83f5f736b 100644 --- a/rs/moq-transcode/src/feed.rs +++ b/rs/moq-transcode/src/feed.rs @@ -10,7 +10,8 @@ //! The decode loop runs only while at least one [`Listener`] exists: the first //! listener subscribes to the source and opens the decoder, dropping the last //! one tears both down. An idle broadcast costs nothing, exactly like the -//! per-rung subscriptions this replaces. +//! per-rung subscriptions this replaces. That includes the source track handle +//! itself: holding one is interest in the track all the way to its publisher. use std::sync::{Arc, Mutex}; @@ -51,8 +52,11 @@ pub(crate) struct Feed { } struct Inner { - /// The source media track (subscribed only while listeners exist). - source: moq_net::track::Consumer, + /// The source broadcast, whose media track is looked up only while listeners + /// exist. + broadcast: moq_net::broadcast::Consumer, + /// The source media track's name. + name: String, /// The source rendition's catalog entry (codec + container). config: VideoConfig, /// The decoder: which implementation, and where its frames live. @@ -70,13 +74,15 @@ struct State { impl Feed { pub(crate) fn new( - source: moq_net::track::Consumer, + broadcast: moq_net::broadcast::Consumer, + name: String, config: VideoConfig, decoder: moq_video::decode::Config, ) -> Self { Self { inner: Arc::new(Inner { - source, + broadcast, + name, config, decoder, state: Mutex::new(State::default()), @@ -181,7 +187,7 @@ async fn decode(inner: &Inner, sender: &broadcast::Sender) -> Result<(), E // Arrival order on purpose: group order doesn't matter here, since every group // decodes independently from its own keyframe, so there is nothing for a sequence // cursor to buy. Either one drops whatever falls behind the live edge. - let mut subscriber = inner.source.subscribe(None).await?; + let mut subscriber = inner.broadcast.track(&inner.name)?.subscribe(None).await?; while let Some(mut group) = subscriber.recv_group().await? { // Sends only fail with zero receivers, which is fine: teardown aborts diff --git a/rs/moq-transcode/src/lib.rs b/rs/moq-transcode/src/lib.rs index 18dc7c44264..71958b56941 100644 --- a/rs/moq-transcode/src/lib.rs +++ b/rs/moq-transcode/src/lib.rs @@ -165,7 +165,7 @@ impl Transcoder { request = dynamic.requested_track() => { // Err means the broadcast closed; nothing left to serve. let Ok(request) = request else { break }; - match ladder.rung(request.name())? { + match ladder.rung(request.name()) { Some(rung) => { tasks.spawn(rung::serve(rung, request)); } None => request.reject(moq_net::Error::NotFound), } @@ -1179,6 +1179,58 @@ mod tests { transcoder.abort(); } + /// A ladder nobody watches holds nothing of its source. A held track handle is + /// interest in the track all the way to its publisher, so a transcoder that + /// kept one per idle rung would stop an on-demand source from ever idling. + #[tokio::test] + async fn an_idle_rung_leaves_its_source_unused() { + let source = source_catalog(320, 240); + let demand = source._track.demand(); + + let config = Config { + ladder: Ladder::new([Rung::new(120, moq_net::bandwidth::Rate::from_bps(100_000))]).unwrap(), + encoder: moq_video::encode::Kind::Software, + decoder: moq_video::decode::Kind::Software, + source: None, + ..Default::default() + }; + let output = moq_net::broadcast::Info::default().produce(); + let consumer = output.consume(); + let transcoder = tokio::spawn(run(source.broadcast.consume(), output, config)); + + let catalog = loop { + match consumer.track(hang::Catalog::DEFAULT_NAME) { + Ok(track) => break track, + Err(moq_net::Error::NotFound) => tokio::task::yield_now().await, + Err(err) => panic!("catalog track: {err}"), + } + }; + let mut catalogs = moq_mux::catalog::hang::Consumer::<()>::new(catalog.subscribe(None).await.unwrap()); + await_catalog(&mut catalogs, |snapshot| { + snapshot.video.renditions.contains_key("video/120p") + }) + .await; + assert!(!demand.is_used(), "the published ladder holds its source"); + + // A rung consumer is demand on its own, so the rung starts encoding and + // subscribes to the source. + let rung = consumer.track("video/120p").unwrap(); + rung.query().await.unwrap(); + tokio::time::timeout(std::time::Duration::from_secs(5), demand.used()) + .await + .expect("the rung never asked for its source") + .unwrap(); + + // Its task outlives the viewer, idle until the next one. + drop(rung); + tokio::time::timeout(std::time::Duration::from_secs(5), demand.unused()) + .await + .expect("the idle rung kept its source wanted") + .unwrap(); + + transcoder.abort(); + } + /// The other half of the retirement boundary: a fetch that is still opening /// its decoder has claimed no output group, so retirement must not declare a /// final sequence until it has. Finishing at retirement instead computes the diff --git a/rs/moq-transcode/src/pipeline.rs b/rs/moq-transcode/src/pipeline.rs index ea33202fd3a..1a753fe79e4 100644 --- a/rs/moq-transcode/src/pipeline.rs +++ b/rs/moq-transcode/src/pipeline.rs @@ -70,7 +70,7 @@ impl Pipeline { ) -> Result { // One shared live decode for every rung of this source: N active rungs // share one subscription and one decoder instead of N. - let feed = Feed::new(source.track(&name)?, rendition.clone(), config.feed_decoder()); + let feed = Feed::new(source.clone(), name.clone(), rendition.clone(), config.feed_decoder()); let mut ladder = Self { source, @@ -146,15 +146,13 @@ impl Pipeline { /// The rung to serve a requested track with, or `None` if the ladder has no /// such rung right now. - pub(crate) fn rung(&mut self, name: &str) -> Result, Error> { - let Some(published) = self.rungs.iter().find(|published| published.rung.name == name) else { - return Ok(None); - }; + pub(crate) fn rung(&mut self, name: &str) -> Option { + let published = self.rungs.iter().find(|published| published.rung.name == name)?; let (retired, retire) = rung::Retire::channel(); self.serving.insert(published.rung.name.clone(), retired); - Ok(Some(rung::Rung { - source: self.source.track(&self.name)?, + Some(rung::Rung { + source: self.name.clone(), feed: self.feed.clone(), broadcast: self.source.clone(), config: self.rendition.clone(), @@ -164,7 +162,7 @@ impl Pipeline { active: self.active.clone(), info: published.rung.clone(), retire, - })) + }) } /// Resolve the ladder again against a new source catalog snapshot. @@ -211,7 +209,12 @@ impl Pipeline { let _ = retired.send(true); } self.serving.clear(); - self.feed = Feed::new(self.source.track(&name)?, rendition.clone(), self.config.feed_decoder()); + self.feed = Feed::new( + self.source.clone(), + name.clone(), + rendition.clone(), + self.config.feed_decoder(), + ); } self.name = name; self.rendition = rendition; diff --git a/rs/moq-transcode/src/rung.rs b/rs/moq-transcode/src/rung.rs index ccab02bb6d4..70e66e3bbbd 100644 --- a/rs/moq-transcode/src/rung.rs +++ b/rs/moq-transcode/src/rung.rs @@ -65,11 +65,13 @@ impl Retire { #[derive(Clone)] pub(crate) struct Rung { pub info: Resolved, - /// The source media track, for group fetches (not yet subscribed). - pub source: moq_net::track::Consumer, + /// The source media track's name, looked up per group fetch rather than held: + /// a held track handle is interest in the source all the way to its publisher, + /// and an idle rung has none. + pub source: String, /// The shared live decode of the source, for the live path. pub feed: Feed, - /// The source broadcast, to notice it closing while idle. + /// The source broadcast, to fetch from and to notice it closing while idle. pub broadcast: moq_net::broadcast::Consumer, /// The source rendition's catalog entry (codec + container). pub config: VideoConfig, @@ -506,11 +508,18 @@ async fn fetch(rung: Rung, request: moq_net::group::Request) -> Result<(), Error // this handle going unused is them picking the group up, not abandoning it. let demand = request.demand(); + let track = match rung.broadcast.track(&rung.source) { + Ok(track) => track, + Err(err) => { + request.reject(err.clone()); + return Err(err.into()); + } + }; let options = moq_net::group::Fetch::default().with_priority(request.priority()); let mut source = tokio::select! { biased; _ = demand.unused() => return abandon(request), - source = rung.source.fetch_group(request.sequence(), options) => match source { + source = track.fetch_group(request.sequence(), options) => match source { Ok(source) => source, Err(err) => { request.reject(err.clone());