diff --git a/doc/concept/standard.md b/doc/concept/standard.md index a2431d426a2..477dc62bda5 100644 --- a/doc/concept/standard.md +++ b/doc/concept/standard.md @@ -26,7 +26,7 @@ maps everything else to "not supported" or a harmless equivalent. The subset drops. What a peer actually observes against this implementation: - **Pull.** Subscribers ask. Single-track `PUBLISH` offers are declined, unless they answer our `SUBSCRIBE_TRACKS` (below); announce a namespace and serve the resulting subscriptions. Announcements go out unsolicited. A client asks only for the prefixes its readers watch, and a server or relay link for every prefix it may discover. Rust never holds two overlapping requests at a peer that did not declare solicit: a broader one replaces the narrower ones once they are closed. The solicit `SETUP` option makes us wait to be asked. Without it, a draft 16+ peer also gets each match as a `NAMESPACE` on its `SUBSCRIBE_NAMESPACE` stream, so it hears the namespace twice. We send `PUBLISH` only to answer a `SUBSCRIBE_TRACKS` (below): on drafts 16 and 17 a `SUBSCRIBE_NAMESPACE` asking only for `PUBLISH` is refused, one asking for both gets only `NAMESPACE`, and any other Subscribe Options value closes the session. -- **`SUBSCRIBE_TRACKS` lists, it never pushes.** On draft 18 and later, it lists every track of every broadcast under its prefix, later ones included. With `FORWARD=0`, each track gets a paused `PUBLISH` carrying its Track Properties, and an error `PUBLISH_DONE` once the track is no longer listed, never `TRACK_ENDED`, since the track may go on elsewhere. A `REQUEST_UPDATE` with `FORWARD=1` resumes one, served from the cache like a `SUBSCRIBE` with the update's filter and answered with `LARGEST_OBJECT`; `FORWARD=0` pauses it again. Each paused `PUBLISH` opens only as the peer's stream credit admits it, so a peer that grants none holds none. A listing that omits `FORWARD` or sets it to 1 asks for every track's objects: from draft 19 each track gets `PUBLISH_SKIPPED`, and the subscriber `SUBSCRIBE`s to what it wants; draft 18 has no such skip, so the request is refused with `NOT_SUPPORTED`. A prefix overlapping another listing gets `PREFIX_OVERLAP`, a `TRACK_PROPERTY_FILTER` gets `NOT_SUPPORTED`, and a `REQUEST_UPDATE` on the listing itself ends it. On draft 18, a `SUBSCRIBE` to a track we `PUBLISH`ed gets `DUPLICATE_SUBSCRIPTION`. Cancelling the listing stops new `PUBLISH`es, while those sent live on. A broadcast is listed only while our publish grant covers it, from scratch once it does. Rust sends it upstream for a listing (below); JavaScript does not. +- **`SUBSCRIBE_TRACKS` lists, it never pushes.** On draft 18 and later, it lists every track of every broadcast under its prefix, later ones included. With `FORWARD=0`, each track gets a paused `PUBLISH` carrying its Track Properties, and a `PUBLISH_DONE` once the track is no longer listed: `TRACK_ENDED` if it finished, even just before its broadcast ended, and an error otherwise, since the track may go on elsewhere. A `REQUEST_UPDATE` with `FORWARD=1` resumes one, served from the cache like a `SUBSCRIBE` with the update's filter and answered with `LARGEST_OBJECT`; `FORWARD=0` pauses it again. Each paused `PUBLISH` opens only as the peer's stream credit admits it, so a peer that grants none holds none. A listing that omits `FORWARD` or sets it to 1 asks for every track's objects: from draft 19 each track gets `PUBLISH_SKIPPED`, and the subscriber `SUBSCRIBE`s to what it wants; draft 18 has no such skip, so the request is refused with `NOT_SUPPORTED`. A prefix overlapping another listing gets `PREFIX_OVERLAP`, a `TRACK_PROPERTY_FILTER` gets `NOT_SUPPORTED`, and a `REQUEST_UPDATE` on the listing itself ends it. On draft 18, a `SUBSCRIBE` to a track we `PUBLISH`ed gets `DUPLICATE_SUBSCRIPTION`. Cancelling the listing stops new `PUBLISH`es, while those sent live on. A broadcast is listed only while our publish grant covers it, from scratch once it does. Rust sends it upstream for a listing (below); JavaScript does not. - **A Rust TRACKS listing sends `SUBSCRIBE_TRACKS` upstream, only on demand.** Listing a broadcast from a moq-transport peer on draft 18 and later sends `SUBSCRIBE_TRACKS` with `FORWARD=0` while someone lists it, never on connect, announce, or subscribe, so a peer that refuses, resets, or ignores it affects only that listing. Before draft 18 the listing is refused as unsupported. Each listed track's paused `PUBLISH` is its upstream copy, which a reader resumes rather than subscribing again. Only a `PUBLISH_DONE` with `TRACK_ENDED` removes a listed track, and it lists as finished; any other end, including the error our own answerer sends for a track that left unfinished, fails the listing, and readers list again. - **History is one group.** In Rust, Object ID gaps must be marked by Prior Object ID Gap properties for a group to be fetchable. A `FETCH` returns one group from the cache, or on drafts 14 through 19 the saved prefix of the group a new subscription just joined. A range of groups is refused, as is a draft 20+ `LOCATION_FILTER` that ends at Largest Object rather than an absolute End Location. On drafts 14 through 19, `FETCH_OK` End Object 0 covers the entire End Group, matching the request; Rust also accepts the legacy answer naming the next group with Object 0. On draft 20 and later, `FETCH_OK`'s End Location is the requested end, capped at Largest Object: a range holding no objects gets an empty fetch stream, and only a start past Largest Object is refused, with `INVALID_RANGE`. A relay's copy with no upstream subscription can't know Largest Object short of the track's end, so it echoes the requested end instead. JavaScript publishing refuses every `FETCH`. Datagrams are never fetchable. A Rust reader that only fetches learns the track from `TRACK_STATUS` instead of subscribing, so a finished track stays fetchable; draft 17 still subscribes, since its `TRACK_STATUS` answer cannot say whether the track is timed, and so does a reader whose publisher refuses `TRACK_STATUS` as `NOT_SUPPORTED`. - **Compression.** Rust peers on draft 17 and later exchange compressed tracks when both offer the [Flate extension](/draft/moq-flate). Older peers receive one shared inflated copy. A publisher that seals the compression declaration makes that fallback unavailable: an incompatible peer is refused, preserving the sealed metadata. JavaScript does not offer the IETF extension. diff --git a/js/net/src/broadcast.ts b/js/net/src/broadcast.ts index a9d6e31af4c..65d71072aab 100644 --- a/js/net/src/broadcast.ts +++ b/js/net/src/broadcast.ts @@ -147,7 +147,7 @@ async function requested(state: BroadcastState): Promise { diff --git a/js/net/src/ietf/publisher.ts b/js/net/src/ietf/publisher.ts index 31186b82a53..882537e4bff 100644 --- a/js/net/src/ietf/publisher.ts +++ b/js/net/src/ietf/publisher.ts @@ -2540,7 +2540,19 @@ export class Publisher { reading ??= listing.next().then((event): Woken => ({ kind: "track", event })); waits.push(reading); } - const woken = await untilAborted(race(waits), hold.signal); + let woken: Woken; + try { + woken = await untilAborted(race(waits), hold.signal); + } catch (err) { + // A closed broadcast's listing has already ended, after the tracks that left + // before it, so it is read to that end whatever withdrew the broadcast: a + // PUBLISH whose track finished ends with TRACK_ENDED. Nothing more goes out. + if (source.closed.peek() === undefined) throw err; + pending.clear(); + unstage(); + reading ??= listing.next().then((event): Woken => ({ kind: "track", event })); + woken = await reading; + } if (woken.kind === "changed") continue; if (woken.kind === "failed") { console.debug(`listed broadcast=${namespace}: PUBLISH stream did not open`, woken.err); diff --git a/js/net/src/ietf/subscribe_tracks.test.ts b/js/net/src/ietf/subscribe_tracks.test.ts index 49a2967e053..6786fbd2ea5 100644 --- a/js/net/src/ietf/subscribe_tracks.test.ts +++ b/js/net/src/ietf/subscribe_tracks.test.ts @@ -233,24 +233,30 @@ for (const version of drafts) { const clock = new Manual(); const installed = clock.install(); using _installed = { [Symbol.dispose]: () => installed.close() }; - const fx = fixture(version); - const broadcast = fx.origin.createBroadcast(Path.from("room/live")); - broadcast.announce(); - const track = broadcast.createTrack("video"); - const listing = await fx.list(); - await ok(listing, version); + // Spied before listing, since the PUBLISH may go out before the listing's OK is read. + // Restored only once the test is done: `Message.write` reads `writer.write` before + // awaiting its frame, so restoring mid-write would drop a concurrent write on the stale spy. const resume = Promise.withResolvers(); + let holding = false; const write = Writer.prototype.write; const held = spyOn(Writer.prototype, "write").mockImplementation(async function (this: Writer, chunk) { await write.call(this, chunk); - if (chunk[0] !== Publish.id) return; - held.mockRestore(); + if (holding || chunk[0] !== Publish.id) return; + holding = true; await resume.promise; }); + using _held = { [Symbol.dispose]: () => held.mockRestore() }; + const fx = fixture(version); + const broadcast = fx.origin.createBroadcast(Path.from("room/live")); + broadcast.announce(); + const track = broadcast.createTrack("video"); + const listing = await fx.list(); + await ok(listing, version); const { stream } = await fx.offered(false); track.close(); broadcast.removeTrack("video"); await clock.advance(Nano.zero); + expect(holding).toBe(true); resume.resolve(); await publishOk(stream, version); expect(await stream.reader.u53()).toBe(PublishDone.id); @@ -258,6 +264,53 @@ for (const version of drafts) { expect(done.statusCode).toBe(PublishDoneStatus.TRACK_ENDED); }); + // A broadcast closing right after a track finished ends that track's PUBLISH with TRACK_ENDED, + // removed first or not and answered or not, while a track left unfinished still ends in error: + // PUBLISH_DONE once answered, a reset before. + for (const remove of [false, true]) { + for (const answer of [true, false]) { + const name = `remove=${remove} answer=${answer}`; + test(`a clean end keeps its finishes, ${name} (${version})`, async () => { + const clock = new Manual(); + const installed = clock.install(); + using _installed = { [Symbol.dispose]: () => installed.close() }; + const fx = fixture(version); + const broadcast = fx.origin.createBroadcast(Path.from("room/live")); + broadcast.announce(); + const video = broadcast.createTrack("video"); + broadcast.createTrack("audio"); + const listing = await fx.list(); + await ok(listing, version); + const published = new Map(); + for (let i = 0; i < 2; i++) { + const { stream, publish } = await fx.offered(answer); + published.set(publish.trackName, stream); + } + await clock.advance(Nano.zero); + video.close(); + if (remove) broadcast.removeTrack("video"); + broadcast.close(); + await clock.advance(Nano.zero); + const videoStream = published.get("video"); + const audioStream = published.get("audio"); + if (!videoStream || !audioStream) throw new Error("missing PUBLISH"); + if (!answer) await publishOk(videoStream, version); + expect(await videoStream.reader.u53()).toBe(PublishDone.id); + expect((await PublishDone.decode(videoStream.reader, version)).statusCode).toBe( + PublishDoneStatus.TRACK_ENDED, + ); + if (!answer) { + await expect(audioStream.reader.u53()).rejects.toThrow(); + return; + } + expect(await audioStream.reader.u53()).toBe(PublishDone.id); + expect((await PublishDone.decode(audioStream.reader, version)).statusCode).toBe( + PublishDoneStatus.INTERNAL_ERROR, + ); + }); + } + } + test(`track listing rejects overlapping prefixes (${version})`, async () => { const fx = fixture(version); const broadcast = fx.origin.createBroadcast(Path.from("room/live")); diff --git a/js/net/src/listing.test.ts b/js/net/src/listing.test.ts index 3a4773ea10b..7f47d064fe5 100644 --- a/js/net/src/listing.test.ts +++ b/js/net/src/listing.test.ts @@ -196,6 +196,64 @@ test("the broadcast ending ends the listing", async () => { expect(() => consumer.tracks()).toThrow("broadcast is closed"); }); +// A clean end reports each track that left before it, and how, but not one the reader never +// saw; an error ends at once. +test("a clean end follows the removals before it", async () => { + const broadcast = new BroadcastProducer(); + const a = broadcast.createTrack("a"); + broadcast.createTrack("b"); + const live = broadcast.createTrack("live"); + const tracks = list(broadcast); + expect(await tracks.drain()).toEqual(["+a", "+b", "+live", "synced"]); + a.close(); + broadcast.removeTrack("a"); + broadcast.removeTrack("b"); + broadcast.createTrack("unseen"); + live.close(); + broadcast.close(); + expect(await tracks.drain()).toEqual(["~a", "-b", "~live", "end"]); + + // The producer's finished track is what a handler's answer of its name served. + const both = new BroadcastProducer(); + const handler = wireOf(both).lister(); + handler.add("a", { priority: 0 }); + handler.synced(); + const own = both.createTrack("a"); + const served = list(both); + expect(await served.drain()).toEqual(["+a", "synced"]); + own.close(); + both.close(); + expect(await served.drain()).toEqual(["~a", "end"]); + + // A reader that closes itself reads nothing more, even what was queued before the end. + const closing = new BroadcastProducer(); + const gone = closing.createTrack("gone"); + const reader = list(closing); + expect(await reader.drain()).toEqual(["+gone", "synced"]); + gone.close(); + closing.removeTrack("gone"); + closing.close(); + reader.tracks.close(); + expect(await reader.drain()).toEqual(["end"]); + + for (const abort of [false, true]) { + const answered = new BroadcastProducer(); + const lister = wireOf(answered).lister(); + const reader = list(answered); + lister.add("up", { priority: 0 }); + lister.synced(); + expect(await reader.drain()).toEqual(["+up", "synced"]); + lister.finish("up"); + if (abort) { + lister.abort(new NotFound("broadcast")); + expect(await reader.drain()).toEqual([`error: ${StreamCode.NotFound}`]); + } else { + lister.end(); + expect(await reader.drain()).toEqual(["~up", "end"]); + } + } +}); + test("a closed listing reads nothing more, and leaving a loop early closes it", async () => { const broadcast = new BroadcastProducer(); broadcast.createTrack("a"); diff --git a/js/net/src/listing.ts b/js/net/src/listing.ts index dddec0d06f2..2ca070e1ba7 100644 --- a/js/net/src/listing.ts +++ b/js/net/src/listing.ts @@ -119,8 +119,14 @@ class Queue { } next(): TracksEvent | undefined { - if (this.end !== undefined) { - if (this.end) throw this.end; + if (this.end) throw this.end; + if (this.end === null) { + // A clean end still reports each track that left before it, and how; an add the + // reader never read is dropped. + for (const [name, pending] of this.pending) { + this.pending.delete(name); + if (pending.kind !== "add") return { kind: pending.finished ? "finished" : "removed", name }; + } return undefined; } if (!this.synced && this.bound !== undefined && (this.bound === 0 || this.pending.size === 0)) { @@ -229,8 +235,9 @@ export class Listing { return queue; } - /** A reader left. A replaced or ended one already did. */ + /** A reader left, so it reads nothing more. A replaced or ended one already did. */ release(queue: Queue, owner?: object) { + queue.pending.clear(); if (queue.detached) return; queue.finish(null); this.#readers.delete(queue); @@ -312,6 +319,18 @@ export class Listing { this.#answered = false; } + /** + * The broadcast ended: so does every listing of it, first unlisting each of its producer's tracks + * that finished. A handler's answer of the same name does not keep it live, since the producer's + * track is what served it. + */ + close() { + for (const [name, entry] of [...this.#live]) { + if (entry.own && finished(entry.own)) this.#remove(name, true); + } + this.detach(null); + } + /** End every reader with `end` and detach them. */ detach(end: Error | null) { for (const queue of this.#readers) queue.finish(end); @@ -336,7 +355,8 @@ export function listTracks(listing: Listing, owner?: object): Tracks { * The tracks a broadcast lists, from its consumer's `tracks()`. * * Reports every track listed when it was created, then `synced`, then each change as it - * happens, until the broadcast ends. Listing a broadcast never requests a track: subscribe to + * happens, until the broadcast ends: a clean end comes after each track that left before it, and + * a track still listed then ends unfinished. Listing a broadcast never requests a track: subscribe to * each listed one through the consumer's `track(name)`. * * @public diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 2593bc96c61..001f2ed311d 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -2595,6 +2595,34 @@ test.each([ } }); +// A source that finished a track and closed just before the front concluded ended its listing +// after that finish, so the front's listing ends after it too. +test("a concluded front's listing keeps its closed source's finishes", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const from = a.consume(); + const answer = wireOf(from).lister(); + answer.add("seq", { priority: 0 }); + answer.add("live", { priority: 0 }); + answer.synced(); + front.serve(from); + const tracks = front.consumer.tracks(); + try { + for (const event of [{ kind: "added", name: "seq" }, { kind: "added", name: "live" }, { kind: "synced" }]) + expect(await tracks.next()).toMatchObject(event); + answer.finish("seq"); + a.close(); + front.conclude(); + expect(await tracks.next()).toMatchObject({ kind: "finished", name: "seq" }); + expect(await tracks.next()).toBeUndefined(); + } finally { + tracks.close(); + answer.close(); + front.consumer.close(); + a.close(); + } +}); + test("only delivery codes are a route failing", () => { // Mirrors `only_delivery_codes_are_a_route_failing` in rs/moq-net/src/model/resume.rs. const delivery = [0x0, 0x1, 0x3, 0x4, 0x31]; diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index a6293169afe..3ca3b4e8657 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -1348,6 +1348,22 @@ export class Front { listing?.reader?.close(); listing = undefined; }; + const read = (reader: Tracks): Promise<{ event?: TracksEvent; err?: Error }> => + reader.next().then( + (event) => ({ event }), + (err: unknown) => ({ err: err instanceof Error ? err : new Error(String(err)) }), + ); + // Forwarded as the source reported it, finished or not. + const unlist = ( + listed: Set, + stale: Set, + event: { kind: "removed" | "finished"; name: string }, + ) => { + stale.delete(event.name); + if (!listed.delete(event.name)) return; + if (event.kind === "finished") lister.finish(event.name); + else lister.remove(event.name); + }; try { for (;;) { if (closed.peek() !== undefined) return; @@ -1360,7 +1376,15 @@ export class Front { continue; } if (this.#concluded) { - // No route will serve it again, so nothing more will be listed. + // No route will serve it again, so nothing more will be listed. A source that + // closed ended its listing after the tracks it finished last: those go first. + while (listing?.reader && listing.source?.from.closed.peek() !== undefined) { + const { event } = await (listing.next ?? read(listing.reader)); + listing.next = undefined; + if (!event) break; + if (event.kind === "removed" || event.kind === "finished") + unlist(listing.listed, listing.stale, event); + } stop(); lister.end(); continue; @@ -1390,10 +1414,7 @@ export class Front { await wait(); continue; } - listing.next ??= reader.next().then( - (event) => ({ event }), - (err: unknown) => ({ err: err instanceof Error ? err : new Error(String(err)) }), - ); + listing.next ??= read(reader); const result = await race([listing.next, wait().then(() => undefined)]); if (!result) continue; listing.next = undefined; @@ -1420,11 +1441,7 @@ export class Front { lister.add(event.name, event.info); } } else if (event.kind === "removed" || event.kind === "finished") { - // Forwarded as the source reported it, finished or not. - listing.stale.delete(event.name); - if (!listing.listed.delete(event.name)) continue; - if (event.kind === "finished") lister.finish(event.name); - else lister.remove(event.name); + unlist(listing.listed, listing.stale, event); } else { // What the new source still lacks at its SYNCED is gone. for (const name of listing.stale) { diff --git a/quest/m1/README.md b/quest/m1/README.md index 4ed4f9260f9..4ad17ea81c1 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -117,7 +117,6 @@ Catalog HLS field. - [A widened max age re-offers cached groups](/quest/m1/resolved-demand.md) - a warm relay serves a reader with a wider max age the cached groups a cold relay would, with no wire change - [Drop set](/quest/m1/drop-set.md) - each cursor judges missing groups against its own budget, so a narrow reader drops promptly and a widening re-offers groups already reported dropped on lite-07 - [Broadcast archives](/quest/m1/archive/README.md) - ANNOUNCE, TRACKS, and a SUBSCRIBE per track record every track under a prefix to any object_store, a host serves a stored broadcast back, and the catalog drops its archive pointer for an opt-in `hls` field -- [A listing's clean end keeps its finishes](/quest/m1/listing-clean-end.md) - a broadcast ending right after its tracks finish ends each finished track's IETF PUBLISH with TRACK_ENDED, in Rust and JS - [Compressed tracks](/quest/m1/compress/README.md) - a track flagged `compress` is DEFLATEd per group by moq-net, forwarded compressed, and inflated once on the relay for peers that cannot signal it; `catalog.json.z` and moq-flate go away - [Finalize moq-lite-07](/quest/m1/lite07-finalize.md) - when the maintainer cuts it, lite-07 negotiates as `moq-lite-07` and the next release ships it - [Moxygen minimal interop](/quest/m1/smoke-moxygen.md) - waiting on Moxygen to send a registered filter type, then the smoke lane stops hiding its failure diff --git a/quest/m1/listing-clean-end.md b/quest/m1/listing-clean-end.md deleted file mode 100644 index 1391be79dde..00000000000 --- a/quest/m1/listing-clean-end.md +++ /dev/null @@ -1,75 +0,0 @@ -# [M] A listing's clean end keeps its finishes - -## Goal - -When a broadcast closes right after its tracks finish, each IETF SUBSCRIBE_TRACKS PUBLISH for a finished track -ends with TRACK_ENDED, not INTERNAL_ERROR, in Rust and JS. A track left -unfinished still ends in error. - -## Plan - -Split from [moq-dev/moq#6399](https://github.com/moq-dev/moq/pull/6399) -(Codex finding -[r4357921727](https://github.com/moq-dev/moq/pull/6399#discussion_r4357921727)), -whose fix predates it. A listing reports its end ahead of the removals still -queued, so a finish queued just before the end is never read. - -Why a "finished cleanly" check on the PUBLISH's track handle can't work: - -- The IETF publisher gets every broadcast from `origin.request_broadcast`, so - its handles are lazy lookups on the origin's front broadcast. A track is only - resolved there when something subscribes, queries, or fetches it, so a - paused PUBLISH's handle never sees its track's end. Making it resolve would - add upstream demand for every listed track. -- The origin forwarder drops the finish before the front sees it: on the - source listing's end it drops the reader (`rs/moq-net/src/model/origin.rs`, - the `Step::Listed` `Ok(None)` arm, around lines 3966-3973), and the source's - queued `Finished` is lost. - -Open design (maintainer to decide): - -1. Crate-internal drain, public `Tracks` order unchanged: crate readers get - queued removals before a clean end. Also needed: a broadcast close unlists - its held finished tracks as finished before closing the listing; the origin - forwarder drains a source's removals before acting on its end; the IETF - publisher reads the listing to its end after the route closes instead of - acting on the route first. Mirror in JS (`#listBroadcast`'s `finally` in - `js/net/src/ietf/publisher.ts` marks every entry "removed"). -2. Change the public `Tracks` order so a clean end follows the queued - removals. Fixes the cause at its source, but was ruled out for #6399, and - still needs the forwarder, close, and publisher changes from option 1. - -Starting point: this test in `rs/moq-net/src/ietf/publisher/tracks.rs` fails -on drafts 18-22 (the finished track gets status 0, not 2). - -```rust -#[moq_clock::test] -async fn a_finished_track_ends_cleanly_when_its_listing_ends() { - for version in DRAFTS { - let mut h = harness(version); - let mut request = h.request(&listing(false)).await; - assert_eq!(next(&mut request).await.0, ietf::RequestOk::ID, "{version}"); - let mut published = HashMap::new(); - for _ in 0..2 { - let (mut stream, _, name, _) = h.publish().await; - publish_ok(&mut stream).await; - published.insert(name, stream); - } - moq_clock::sleep(Duration::from_millis(10)).await; - - let video = h.tracks.remove("video").unwrap(); - video.finish().unwrap(); - drop(video); - h.broadcast.close(); - for (name, code) in [ - ("video", ietf::PublishDoneStatus::TrackEnded.code(version)), - ("audio", ietf::PublishDoneStatus::InternalError.code(version)), - ] { - let (id, body) = next(published.get_mut(name).unwrap()).await; - assert_eq!(id, ietf::PublishDone::ID, "{version}: {name}"); - let done: ietf::PublishDone = body.message(version).unwrap(); - assert_eq!(done.status_code, code, "{version}: {name}"); - } - } -} -``` diff --git a/rs/moq-net/src/ietf/publisher/tracks.rs b/rs/moq-net/src/ietf/publisher/tracks.rs index 1ee432ccd7d..cd38ab88291 100644 --- a/rs/moq-net/src/ietf/publisher/tracks.rs +++ b/rs/moq-net/src/ietf/publisher/tracks.rs @@ -160,6 +160,16 @@ struct Request { live: kio::Consumer<()>, } +/// Why a listed broadcast's route closed. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Route { + /// Another instance took the path: the PUBLISHes of this one end now. + Replaced, + /// Nothing serves the path any more, so its broadcast ends soon too: its listing is read + /// to that end, which says which tracks finished. + Withdrawn, +} + /// One listed broadcast's tracks, by how far each got toward the peer. #[derive(Default)] struct Entries { @@ -197,16 +207,21 @@ impl Entries { } } - /// Every track is no longer listed. - fn unlist(&mut self, skips: &Skips, suffix: &crate::PathOwned) { + /// Nothing more is PUBLISHed or skipped, while the PUBLISHes already sent stay listed. + fn halt(&mut self, skips: &Skips, suffix: &crate::PathOwned) { self.pending.clear(); self.staged = None; - self.listed.clear(); let mut skips = skips.lock(); for name in self.skipped.drain() { skips.remove(&(suffix.clone(), name)); } } + + /// Every track is no longer listed. + fn unlist(&mut self, skips: &Skips, suffix: &crate::PathOwned) { + self.halt(skips, suffix); + self.listed.clear(); + } } /// The next track to PUBLISH, ready but for its stream. @@ -349,8 +364,15 @@ impl Publisher { skips: Skips::default(), live: live.consume(), }; - // Each listed broadcast's route: dropping it ends that broadcast's listing. - let mut routes: HashMap> = HashMap::new(); + // Each listed broadcast's route, closed once its announcement goes; see [`Route`]. + let mut routes: HashMap> = HashMap::new(); + let withdraw = |routes: &mut HashMap>, prefix: &crate::PathOwned| { + if let Some(route) = routes.remove(prefix) + && let Ok(mut route) = route.write() + { + *route = Route::Withdrawn; + } + }; let mut broadcasts = TaskSet::owned(); let mut session = self.session.clone(); let mut finished = false; @@ -396,14 +418,14 @@ impl Publisher { crate::announce::Event::Start(announce) | crate::announce::Event::Restart(announce), ) => announce, Event::Announce(crate::announce::Event::End(announce)) => { - routes.remove(&announce.prefix); + withdraw(&mut routes, &announce.prefix); continue; } // A new hop or cost changes nothing listed. Event::Announce(crate::announce::Event::Update(_)) => continue, }; // A restart is a new instance: the old one's listing ends with its route. - let route = kio::Producer::new(()); + let route = kio::Producer::new(Route::Replaced); let task = self.clone().run_listed_broadcast( request.clone(), announce.prefix.clone(), @@ -465,7 +487,8 @@ impl Publisher { }) .await; match event { - Some(crate::announce::Event::End(announce) | crate::announce::Event::Restart(announce)) => { + Some(crate::announce::Event::End(announce)) => withdraw(&mut routes, &announce.prefix), + Some(crate::announce::Event::Restart(announce)) => { routes.remove(&announce.prefix); } Some(_) => {} @@ -577,7 +600,7 @@ impl Publisher { request: Request, suffix: crate::PathOwned, epoch: Option, - route: kio::Consumer<()>, + route: kio::Consumer, ) { let namespace = request.prefix.join(&suffix).to_owned(); let absolute = self.origin.absolute(&namespace).to_owned(); @@ -605,8 +628,9 @@ impl Publisher { let mut entries = Entries::default(); let mut publishes = TaskSet::owned(); let mut session = self.session.clone(); - // The listing ended, or the request did: nothing more to add. + // The listing ended, the request did, or the route was withdrawn: nothing more to add. let mut adding = true; + let mut withdrawn = false; // The broadcast's listing, held only while our publish permit covers it: a track // it does not cover is not revealed at all, and one it comes to cover is listed // from scratch. @@ -628,8 +652,10 @@ impl Publisher { Opened(Result, Error>), /// Our publish permit may have changed. Regrant(crate::auth::Permit), - /// The broadcast ended or its route went: every track it listed is gone. + /// The broadcast ended or another instance replaced it: every track it listed is gone. Ended, + /// The route was withdrawn: nothing more is listed, and the broadcast ends soon. + Withdrawn, /// The request was cancelled. Cancelled, } @@ -680,8 +706,11 @@ impl Publisher { if adding && request.live.poll_closed(waiter).is_ready() { return Poll::Ready(Some(Event::Cancelled)); } - if route.poll_closed(waiter).is_ready() { - return Poll::Ready(Some(Event::Ended)); + if !withdrawn && route.poll_closed(waiter).is_ready() { + return Poll::Ready(Some(match *route.read() { + Route::Replaced => Event::Ended, + Route::Withdrawn => Event::Withdrawn, + })); } if let Poll::Ready(permit) = self.auth @@ -755,6 +784,14 @@ impl Publisher { entries.pending.clear(); entries.staged = None; } + // Its listing ends once the front serving it sees the route go, after the + // tracks its source finished last, so the PUBLISHes already sent end by it. + // The tracks not yet PUBLISHed or skipped are no longer announced. + Event::Withdrawn => { + withdrawn = true; + adding = false; + entries.halt(&request.skips, &suffix); + } Event::Opened(Err(err)) => { tracing::debug!(broadcast = %absolute, %err, "publish stream did not open"); entries.unlist(&request.skips, &suffix); @@ -1559,6 +1596,43 @@ mod tests { } } + /// A broadcast closing right after a track finished ends that track's PUBLISH with + /// TRACK_ENDED, whether its producer is still held or not, while a track left unfinished + /// still ends in error. + #[moq_clock::test] + async fn a_clean_end_keeps_its_finishes() { + for version in DRAFTS { + for held in [false, true] { + let case = format!("{version}: held={held}"); + let mut h = harness(version); + let mut request = h.request(&listing(false)).await; + assert_eq!(next(&mut request).await.0, ietf::RequestOk::ID, "{case}"); + let mut published = HashMap::new(); + for _ in 0..2 { + let (mut stream, _, name, _) = h.publish().await; + publish_ok(&mut stream).await; + published.insert(name, stream); + } + moq_clock::sleep(Duration::from_millis(10)).await; + + let video = h.tracks.remove("video").unwrap(); + video.finish().unwrap(); + let video = held.then_some(video); + h.broadcast.close(); + for (name, status) in [ + ("video", ietf::PublishDoneStatus::TrackEnded), + ("audio", ietf::PublishDoneStatus::InternalError), + ] { + let (id, body) = next(published.get_mut(name).unwrap()).await; + assert_eq!(id, ietf::PublishDone::ID, "{case}: {name}"); + let done: ietf::PublishDone = body.message(version).unwrap(); + assert_eq!(done.status_code, status.code(version), "{case}: {name}"); + } + drop(video); + } + } + } + /// A REQUEST_UPDATE with FORWARD=1 resumes a paused PUBLISH: its answer names where the /// subscription joins, and new groups flow. FORWARD=0 pauses it again. #[moq_clock::test] @@ -1839,7 +1913,7 @@ mod tests { .collect(); let skips = Skips::default(); let live = kio::Producer::new(()); - let route = kio::Producer::new(()); + let route = kio::Producer::new(Route::Replaced); let request = Request { origin: h._origin.consume(), prefix: crate::Path::new("").to_owned(), diff --git a/rs/moq-net/src/model/broadcast.rs b/rs/moq-net/src/model/broadcast.rs index bffd604b5c1..3e0527d629c 100644 --- a/rs/moq-net/src/model/broadcast.rs +++ b/rs/moq-net/src/model/broadcast.rs @@ -9,7 +9,7 @@ //! [Info] is the broadcast's static metadata, fixed for its lifetime. use crate::{cache, stats, track}; use std::{ - collections::HashMap, + collections::{HashMap, HashSet}, sync::Arc, task::{Poll, ready}, }; @@ -162,6 +162,18 @@ impl BroadcastState { } } + /// End every listing, reporting each listed track whose producer declared its end as + /// finished, though that producer still holds it in the table. + fn close_listing(&mut self) { + let finished: HashSet> = self + .tracks + .iter() + .filter(|track| track.is_finished()) + .map(|track| track.name().clone()) + .collect(); + self.listing.close(|name| finished.contains(name)); + } + /// The track serving `name` now, published or requested, read without joining its demand. fn peek(&mut self, name: &str) -> Option> { if let Some(weak) = self.tracks.get(name) { @@ -546,7 +558,7 @@ impl Alive { // and `Consumer::track` answers `Unroutable` for one asked about after this // point. Say the same to whoever asked earlier. state.reject_unserved(Error::Unroutable); - state.listing.close(); + state.close_listing(); } let _ = self.token.close(); self.retract(); @@ -572,7 +584,7 @@ impl Alive { }; state.closing = true; state.reject_unserved(Error::Unroutable); - state.listing.close(); + state.close_listing(); token }; token.close(); diff --git a/rs/moq-net/src/model/listing.rs b/rs/moq-net/src/model/listing.rs index 332040b8ba5..71dd0bc4c5c 100644 --- a/rs/moq-net/src/model/listing.rs +++ b/rs/moq-net/src/model/listing.rs @@ -273,8 +273,19 @@ impl ListingState { self.generation += 1; } - /// The broadcast ended: so does every listing of it. - pub(super) fn close(&mut self) { + /// The broadcast ended: so does every listing of it, first unlisting each track its + /// producer created that `finished`, as its holder dropping later would have. A handler's + /// answer of the same name does not keep it live, since the producer's track is what served it. + pub(super) fn close(&mut self, finished: impl Fn(&str) -> bool) { + let done: Vec> = self + .live + .iter() + .filter(|(name, entry)| entry.own.is_some() && finished(name)) + .map(|(name, _)| name.clone()) + .collect(); + for name in done { + self.remove(&name, true); + } self.detach(Ok(())); } } @@ -421,8 +432,22 @@ impl Queue { } fn next(&mut self) -> Result, Error> { - if let Some(end) = &self.end { - return end.clone().map(|()| None); + match &self.end { + Some(Err(err)) => return Err(err.clone()), + // A clean end still reports each track that left before it, and how; an add the + // reader never read is dropped. + Some(Ok(())) => { + while let Some((name, pending)) = self.pending.pop_front() { + match pending { + Pending::Add(_) => {} + Pending::Remove { finished } | Pending::Readd { finished, .. } => { + return Ok(Some(removal(name, finished))); + } + } + } + return Ok(None); + } + None => {} } if !self.synced && (self.bound == Some(0) || (self.bound.is_some() && self.pending.is_empty())) { self.synced = true; @@ -454,7 +479,8 @@ fn removal(name: Arc, finished: bool) -> TracksEvent { /// The tracks a broadcast lists, from [`Consumer::tracks`]. /// /// Reports every track listed when it was created, then [`TracksEvent::Synced`], then -/// each change as it happens, until the broadcast ends. Listing a broadcast never +/// each change as it happens, until the broadcast ends: a clean end comes after each +/// track that left before it, and a track still listed then ends unfinished. Listing a broadcast never /// requests a track: subscribe to each listed one through [`Consumer::track`]. pub struct Tracks { broadcast: Consumer, @@ -783,6 +809,58 @@ mod tests { assert!(matches!(lister.finish("up"), Err(Error::NotFound))); } + /// A clean end reports each track that left before it, and how, but not one the reader + /// never saw; an error ends at once. + #[test] + fn a_clean_end_follows_the_removals_before_it() { + let broadcast = Info::new().produce(); + let a = broadcast.create_track("a", None).unwrap(); + let b = broadcast.create_track("b", None).unwrap(); + let live = broadcast.create_track("live", None).unwrap(); + let mut tracks = broadcast.consume().tracks().unwrap(); + assert_eq!(drain(&mut tracks), ["+a", "+b", "+live", "synced"]); + a.finish().unwrap(); + drop(a); + drop(b); + let _unseen = broadcast.create_track("unseen", None).unwrap(); + live.finish().unwrap(); + broadcast.close(); + assert_eq!(drain(&mut tracks), ["~a", "-b", "~live", "end"]); + + // The producer's finished track is what a handler's answer of its name served. + let broadcast = Info::new().produce(); + let lister = broadcast.dynamic().lister().unwrap(); + lister.add("a", track::Info::default()).unwrap(); + lister.synced(); + let a = broadcast.create_track("a", None).unwrap(); + let mut tracks = broadcast.consume().tracks().unwrap(); + assert_eq!(drain(&mut tracks), ["+a", "synced"]); + a.finish().unwrap(); + broadcast.close(); + assert_eq!(drain(&mut tracks), ["~a", "end"]); + + // A handler's end is clean too, and its abort is not. + for abort in [false, true] { + let broadcast = Info::new().produce(); + let lister = broadcast.dynamic().lister().unwrap(); + lister.add("up", track::Info::default()).unwrap(); + lister.synced(); + let mut tracks = broadcast.consume().tracks().unwrap(); + assert_eq!(drain(&mut tracks), ["+up", "synced"]); + lister.finish("up").unwrap(); + match abort { + true => { + lister.abort(Error::Cancel); + assert_eq!(drain(&mut tracks), [format!("error: {}", Error::Cancel)]); + } + false => { + lister.end(); + assert_eq!(drain(&mut tracks), ["~up", "end"]); + } + } + } + } + #[test] fn a_recreated_track_is_removed_before_it_is_added_again() { let broadcast = Info::new().produce(); diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index 8a9dab3a04f..e42ad2851cb 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -3660,12 +3660,8 @@ async fn serve_front(task: FrontTask) -> Vec { Err(_closed) => Err(Error::Unroutable), })); } - if let Some(id) = front.serving() - && let Some(source) = sources.get(&id) - && source.poll_closed(waiter).is_ready() - { - return Poll::Ready(Step::SourceClosed(id)); - } + // The listing first: a source ends its listing before it closes, so this forwards + // the tracks it finished last before its close detaches it. match &mut listing { None => { if front.serving().is_some() && matches!(lister.poll_used(waiter), Poll::Ready(Ok(()))) { @@ -3686,6 +3682,12 @@ async fn serve_front(task: FrontTask) -> Vec { } } } + if let Some(id) = front.serving() + && let Some(source) = sources.get(&id) + && source.poll_closed(waiter).is_ready() + { + return Poll::Ready(Step::SourceClosed(id)); + } while let Poll::Ready(Ok(name)) = ready.poll_pop(waiter) { let Some(io) = tracks.get_mut(&name) else { continue }; io.ready.popped(); @@ -8737,6 +8739,33 @@ mod tests { assert_eq!(listed(&mut tracks).await, "end"); } + /// A source closing right after its tracks finished, held or not, ends the front's + /// listing after those finishes, never ahead of them. + #[moq_clock::test] + #[cfg(feature = "async")] + async fn a_front_forwards_the_finishes_before_its_source_closed() { + let producer = origin(1).produce(); + let consumer = producer.consume(); + let source = producer.publish("room/alice", Route::default()).unwrap(); + let dropped = source.create_track("dropped", None).unwrap(); + let held = source.create_track("held", None).unwrap(); + let _live = source.create_track("live", None).unwrap(); + + let front = consumer.request_broadcast("room/alice", None).await.unwrap(); + let mut tracks = front.tracks().unwrap(); + for event in ["+dropped", "+held", "+live", "synced"] { + assert_eq!(listed(&mut tracks).await, event); + } + + dropped.finish().unwrap(); + drop(dropped); + held.finish().unwrap(); + source.close(); + for event in ["~dropped", "~held", "end"] { + assert_eq!(listed(&mut tracks).await, event); + } + } + /// A source's handler ending its listing while the source stays, as a session does on /// its peer's FIN before any retraction, ends the front's listing too. #[moq_clock::test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 359ee44aa54..5103d52e170 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -3808,6 +3808,11 @@ impl TrackWeak { true } + /// Whether its producer declared the track's end, which is how its listing ends too. + pub(super) fn is_finished(&self) -> bool { + self.state.read().end.is_some() + } + /// Whether anyone is consuming the track right now. A closed track doesn't /// count even if consumers linger to drain its cache: no new work is owed. pub(super) fn is_used(&self) -> bool {