diff --git a/doc/concept/hang.md b/doc/concept/hang.md index 570e8ca61f0..09c38ea9590 100644 --- a/doc/concept/hang.md +++ b/doc/concept/hang.md @@ -51,21 +51,20 @@ VP9, AV1, AAC, and Opus are what the tools produce today. Properties shared by every video rendition (display size, rotation, flip) sit at the section root. A track name always means the same content, so a catalog update never -redefines a track. Its codec, description, container, sample rate, and channel -count are fixed. `codedWidth` and `codedHeight` are ceilings: the picture -resizes in band at a keyframe below them, unless a `description` pins it. -`bitrate` and `framerate` are maxima that only rise, like `jitter` and `delay`. -Everything else, such as `enabled` and the section's display size, updates -freely. A -publisher that needs a new identity publishes a new track name, or restarts the -broadcast under a new [epoch](/concept/moq-lite#publisher-epochs). A player can -configure its decoder once per track. +redefines a track. The fields a decoder can't change without reinitializing +are fixed: codec, description, container, display aspect, sample rate, and +channel count. Everything else, such as the coded size, `bitrate`, +`framerate`, and `enabled`, updates in place; the picture resizes in band at a +keyframe unless a `description` pins it. A publisher that needs a new identity +publishes a new track name, or restarts the broadcast under a new +[epoch](/concept/moq-lite#publisher-epochs). A player can configure its +decoder once per track. A few things the catalog can express beyond decoder config: - **Labels.** Any rendition may carry a human-readable `label` for a track picker. The map key stays the rendition's ID, so labels need not be unique and renaming one doesn't rename the track. - **Renditions in another broadcast.** A rendition may point at a relative broadcast path, so a transcoder can publish a ladder that adds low rungs and references the source's original rendition without re-publishing its bytes. The path resolves against where the consumer found the catalog, so a reference that escapes above the root names nothing and the catalog is rejected. -- **Track names.** The map key is a rendition's ID and, by default, its track name. A rendition sets `track` only when the two must differ, such as two renditions of same-named tracks in different broadcasts; readers subscribe to `track` when present. An ID can't name both a video and an audio rendition. +- **Track names.** The map key is a rendition's ID and, by default, its track name. A rendition sets `track` only when the two must differ, such as two renditions of same-named tracks in different broadcasts, or a rendition whose media changed and so needs a new name; readers subscribe to `track` when present. An ID can't name both a video and an audio rendition. - **Jitter.** A rendition can say how far its frames fell behind the media clock before the publisher flushed them, in whole milliseconds rounded up. Encoders report the spread of lateness above each rendition's own recent minimum, so a constant encoder delay is not jitter; container imports estimate batch spans without counting ingest delay. It describes the publisher, never the network, only grows over the life of a stream, and a player sizes its buffer to at least this much. A `0` is read as absent. - **Delay.** A rendition can also say how far its frames reach the transport behind the broadcast's earliest rendition, measured the same way from each rendition's minimum lateness, so a video encoder running 200 ms behind audio advertises `delay: 200` on video. It follows the same rules as jitter. A player holds the largest `delay + jitter` among the renditions it subscribes to, and never subtracts one rendition's `delay` from another's. - **Disabled renditions.** An audio or video rendition with `enabled: false` has no frames coming, and a player must not select it. Pausing a rendition in `@moq/publish` keeps it in the catalog this way rather than removing it, so a mute is a one-field update; a disabled video rendition encodes one black keyframe first, so a player that predates the field shows black instead of a frozen picture. The field is written only when false. The older video `stalled` field is no longer written, and readers ignore it. diff --git a/doc/lib/js/publish.md b/doc/lib/js/publish.md index 7057e1a0a39..02d2cf111a5 100644 --- a/doc/lib/js/publish.md +++ b/doc/lib/js/publish.md @@ -49,7 +49,10 @@ for a permission, device, or settings change instead of prompting again. The video encoder follows its share of the connection's send-rate estimate, so several publishers on one connection do not each target the whole uplink. Codec, resolution, framerate, and bitrate are tunable through -`el.video.config`, and `el.video.cut()` asks for an extra keyframe. For +`el.video.config`, and `el.video.cut()` asks for an extra keyframe. Size, +bitrate, and framerate change in place, while a codec or audio format change +keeps the rendition's catalog ID under a fresh `track` name that viewers +follow through the catalog. For simulcast, drop the element and register several encoders on a `Publish.Broadcast`, as below. @@ -68,8 +71,10 @@ timeline. The catalog's `clock` maps it to wall time. Stamp your own tracks `broadcast.net` is the underlying `Moq.Broadcast.Producer`, so an application can serve its own tracks alongside the media. It is recreated when the origin, name, `enabled`, or `maxAge` changes, so acquire it from an effect and create each -track once per `net`. A track name always means the same content, so a track -re-created on the same `net` throws on an append that would reuse its sequences: +track once per `net`. Names under a rendition's ID, such as `video/1`, are +reserved for that rendition's replacement tracks. A track name always means +the same content, so a track re-created on the same `net` throws on an append +that would reuse its sequences: ```ts import * as Json from "@moq/json"; diff --git a/drafts/draft-lcurley-moq-hang.md b/drafts/draft-lcurley-moq-hang.md index 5e952d26862..e55d9d1c5dc 100644 --- a/drafts/draft-lcurley-moq-hang.md +++ b/drafts/draft-lcurley-moq-hang.md @@ -166,14 +166,13 @@ A discontinuity marker is a delivery event, not a new epoch, and a system-clock ## Track Identity {#identity} A track name always means the same content, so a catalog update never redefines an audio or video track. -Each rendition field falls in one of three classes: +A track's identity is the fields a consumer cannot change without reinitializing its decoder or demuxer: `codec`, `description`, and `container`, plus `displayAspectWidth` and `displayAspectHeight` for video and `sampleRate` and `numberOfChannels` for audio. +A publisher MUST NOT change them for a track once a catalog lists it. -- Identity: `codec`, `description`, and `container`, plus `sampleRate` and `numberOfChannels` for audio. A publisher MUST NOT change them for a track once a catalog lists it. -- Ceilings: video `codedWidth` and `codedHeight` bound every picture the track carries, and `codec` names a profile and level that covers them. The picture size MAY change at any keyframe within both bounds, which a `description` carrying parameter sets rules out. A publisher MAY fill an absent ceiling once, and MUST NOT change it afterward. -- Maxima: `bitrate` and `framerate` are the most the track has carried or is configured to carry. A publisher MAY raise them and MUST NOT lower them. - -Every other rendition field is live state that MAY change in any update within its own rules (`jitter` and `delay` never fall), as MAY the video section's `display`, `rotation`, and `flip`, which describe presentation rather than decoding. -A publisher that changes an identity field or exceeds a ceiling MUST publish the media under a new track name, or as a new broadcast under a new Epoch {{moql}}. +Every other rendition field is live state that MAY change in any update within its own rules (`jitter` and `delay` never fall), as MAY the video section's `display`, `rotation`, and `flip`. +The picture size MAY change at any keyframe, which a `description` carrying parameter sets rules out, and `codec` names a profile and level that covers every picture the track carries. +A publisher that changes an identity field MUST publish the media under a new track name, or as a new broadcast under a new Epoch {{moql}}. +A rendition MAY keep its ID and name the new track with `track` ({{field-track}}), so a consumer follows the new name rather than reusing a decoder configured for the ID. A track name removed from the catalog MUST NOT return with a different identity. A consumer MAY configure a decoder once per track from the catalog. @@ -1123,7 +1122,7 @@ This document has no IANA actions. ## moq-hang-04 {:numbered="false"} -- Fixed a track's identity for its name ({{identity}}): codec, description, container, sample rate, and channel count never change; `codedWidth` and `codedHeight` are ceilings below which the picture resizes in band; `bitrate` and `framerate` are maxima that only rise. A change of identity takes a new track name or a new Epoch. +- Fixed a track's identity for its name ({{identity}}): codec, description, container, display aspect, sample rate, and channel count never change, while every other field, including size, bitrate, and framerate, updates in place. A change of identity takes a new track name, which a rendition may name with `track` while keeping its ID, or a new Epoch. - Specified MSF text metadata preservation and refusal of unrepresentable text packaging during gateway conversion. - Specified conversion between relative broadcast references and MSF track namespaces without changing track identity, reading a namespace aligned with the catalog path's trailing segments as the catalog itself. diff --git a/js/hang/src/catalog/audio.ts b/js/hang/src/catalog/audio.ts index 746618ac405..1d04093571f 100644 --- a/js/hang/src/catalog/audio.ts +++ b/js/hang/src/catalog/audio.ts @@ -43,7 +43,7 @@ export const AudioConfigSchema = z.object({ // The number of channels in the audio numberOfChannels: u53Schema, - // The maximum bitrate of the audio in bits per second. It only rises. + // The maximum bitrate of the audio in bits per second. // TODO: Support up to Number.MAX_SAFE_INTEGER bitrate: z.optional(u53Schema), diff --git a/js/hang/src/catalog/video.ts b/js/hang/src/catalog/video.ts index b19b1fd0c27..11fca3d7465 100644 --- a/js/hang/src/catalog/video.ts +++ b/js/hang/src/catalog/video.ts @@ -44,10 +44,10 @@ export const VideoConfigSchema = z.object({ displayAspectWidth: z.optional(u53Schema), displayAspectHeight: z.optional(u53Schema), - // The maximum frame rate of the video in frames per second. It only rises. + // The maximum frame rate of the video in frames per second. framerate: z.optional(z.number()), - // The maximum bitrate of the video in bits per second. It only rises. + // The maximum bitrate of the video in bits per second. // TODO: Support up to Number.MAX_SAFE_INTEGER bitrate: z.optional(u53Schema), diff --git a/js/publish/src/audio/capture.test.ts b/js/publish/src/audio/capture.test.ts index f711b4caf2e..9c0aa7e928f 100644 --- a/js/publish/src/audio/capture.test.ts +++ b/js/publish/src/audio/capture.test.ts @@ -504,6 +504,10 @@ function installFailableAudioEncoder() { this.state = "configured"; } encode(): void {} + async flush(): Promise { + // The throwaway description probe is not a live capture encoder. + instances.splice(instances.indexOf(this), 1); + } close(): void { this.state = "closed"; } @@ -572,6 +576,7 @@ test("a failed encoder stays failed until the capture source changes", async () try { await render(); + if (!encoder.out.catalog.peek()) await encoder.out.catalog.changed(); expect(codecs.instances).toHaveLength(1); const failure = new Error("audio encoder failed"); codecs.instances[0].fail(failure); diff --git a/js/publish/src/audio/encoder.test.ts b/js/publish/src/audio/encoder.test.ts index 70ddf1c7dcb..4adaaa70a98 100644 --- a/js/publish/src/audio/encoder.test.ts +++ b/js/publish/src/audio/encoder.test.ts @@ -1,5 +1,5 @@ import { describe, expect, mock, spyOn, test } from "bun:test"; -import type * as Catalog from "@moq/hang/catalog"; +import * as Catalog from "@moq/hang/catalog"; import * as Json from "@moq/json"; import * as Moq from "@moq/net"; import { Time } from "@moq/net"; @@ -70,6 +70,8 @@ describe("toEncoderConfig", () => { // input timestamps never reaches the output. class LaggingAudioEncoder { static instances: LaggingAudioEncoder[] = []; + static probes: AudioEncoderConfig[] = []; + static description: ((config: AudioEncoderConfig) => Uint8Array) | undefined; static readonly LAG = 2; // Called on configure; the encoder publishes its pipeline synchronously right after. @@ -80,6 +82,8 @@ class LaggingAudioEncoder { #held: { timestamp: number; duration: number }[] = []; #base: number | undefined; #encoded = 0; + #probe = false; + #config?: AudioEncoderConfig; #error: AudioEncoderInit["error"]; constructor(init: AudioEncoderInit) { @@ -95,7 +99,10 @@ class LaggingAudioEncoder { configure(config: AudioEncoderConfig): void { this.state = "configured"; - LaggingAudioEncoder.onConfigure?.(config); + this.#config = config; + queueMicrotask(() => { + if (!this.#probe) LaggingAudioEncoder.onConfigure?.(config); + }); } encode(data: AudioData): void { @@ -104,18 +111,36 @@ class LaggingAudioEncoder { this.#held.push({ timestamp: this.#base + this.#encoded, duration }); this.#encoded += duration; while (this.#held.length > LaggingAudioEncoder.LAG) { - const { timestamp, duration } = this.#held.shift() as { timestamp: number; duration: number }; - const chunk = { - type: "key", - timestamp, - duration, - byteLength: 1, - copyTo: (dest: Uint8Array) => dest.set([1]), - }; - this.#output(chunk as unknown as EncodedAudioChunk); + this.#emit(this.#held.shift() as { timestamp: number; duration: number }); } } + #emit({ timestamp, duration }: { timestamp: number; duration: number }): void { + const chunk = { type: "key", timestamp, duration, byteLength: 1, copyTo: (dest: Uint8Array) => dest.set([1]) }; + const config = this.#config; + const description = config && LaggingAudioEncoder.description?.(config); + const metadata = + config && description + ? { + decoderConfig: { + codec: config.codec, + sampleRate: config.sampleRate, + numberOfChannels: config.numberOfChannels, + description, + }, + } + : undefined; + this.#output(chunk as unknown as EncodedAudioChunk, metadata); + } + + async flush(): Promise { + this.#probe = true; + if (this.#config) LaggingAudioEncoder.probes.push(structuredClone(this.#config)); + const index = LaggingAudioEncoder.instances.indexOf(this); + if (index !== -1) LaggingAudioEncoder.instances.splice(index, 1); + for (const chunk of this.#held.splice(0)) this.#emit(chunk); + } + reset(): void { this.state = "unconfigured"; this.#held = []; @@ -145,6 +170,8 @@ class FakeAudioData { function installFakeWebCodecs() { LaggingAudioEncoder.instances = []; + LaggingAudioEncoder.probes = []; + LaggingAudioEncoder.description = undefined; const names = ["AudioEncoder", "AudioDecoder", "AudioData"] as const; const originals = names.map((name) => Object.getOwnPropertyDescriptor(globalThis, name)); const fakes = [LaggingAudioEncoder, class {}, FakeAudioData]; @@ -573,6 +600,7 @@ test("a rendition trailing the broadcast's earliest advertises delay", async () // Regression: codec settings that can't resolve left the encoder unsettled, so `` never // announced and withheld every other rendition with it. test("settles when the codec settings can't resolve", async () => { + using _webcodecs = installFakeWebCodecs(); const error = spyOn(console, "error").mockImplementation(() => {}); const capture = { in: { source: new Signal(undefined) }, @@ -595,7 +623,9 @@ test("settles when the codec settings can't resolve", async () => { expect(error).toHaveBeenCalled(); // A valid duration resolves, and the config keeps it settled. + const ready = encoder.out.catalog.changed(); encoder.codec.set({ mime: "opus", frameDuration: Time.Milli(20) }); + await ready; await settle(); expect(encoder.out.catalog.peek()).toBeDefined(); expect(encoder.settled.peek()).toBe(true); @@ -607,6 +637,7 @@ test("settles when the codec settings can't resolve", async () => { // A fade the gain can't ramp over is refused like any other bad setting, rather than ramping wrong. test("refuses the rendition while the fade is invalid", async () => { + using _webcodecs = installFakeWebCodecs(); const error = spyOn(console, "error").mockImplementation(() => {}); const capture = { in: { source: new Signal(undefined) }, @@ -625,8 +656,9 @@ test("refuses the rendition while the fade is invalid", async () => { expect(encoder.settled.peek()).toBe(true); expect(error).toHaveBeenCalled(); + const ready = encoder.out.catalog.changed(); encoder.fade.set(Time.Milli(0)); - await settle(); + await ready; expect(encoder.out.catalog.peek()).toBeDefined(); encoder.fade.set(Time.Milli(Number.NaN)); @@ -767,6 +799,7 @@ test("audio failure stays local through demand changes and capture format re-arm const config = (await catalog.latest())?.value.audio?.renditions.audio; if (!config?.track) throw new Error("expected recovered track name"); expect(config.track).not.toBe("audio"); + expect(config.numberOfChannels).toBe(Catalog.u53(2)); expect(config.enabled).not.toBe(false); recovered = net.track(config.track).subscribe(); await settle(); @@ -865,3 +898,26 @@ test.each(["failure first", "config first"])( } }, ); + +test("Opus description is probed with the exact live config before advertising", async () => { + using _codecs = installFakeWebCodecs(); + LaggingAudioEncoder.description = (config) => new TextEncoder().encode(JSON.stringify(config)); + using env = await setup(new Baseline(), { + mime: "opus", + frameDuration: Time.Milli(2.5), + complexity: 3, + packetlossperc: 7, + useinbandfec: true, + bitrate: 64000, + }); + expect(env.encoder.out.catalog.peek()?.description).toBe(Buffer.from(JSON.stringify(env.config)).toString("hex")); + expect(LaggingAudioEncoder.probes).toEqual([env.config]); + expect(LaggingAudioEncoder.instances).toHaveLength(1); + expect(env.written).toHaveLength(0); + const configuring = configured(); + env.capture.in.source.set({ track: {}, kind: "voice" } as never); + const voice = await configuring; + expect(voice.opus).toMatchObject({ application: "voip", signal: "voice" }); + expect(LaggingAudioEncoder.probes.at(-1)).toEqual(voice); + expect(env.encoder.out.catalog.peek()?.description).toBe(Buffer.from(JSON.stringify(voice)).toString("hex")); +}); diff --git a/js/publish/src/audio/encoder.ts b/js/publish/src/audio/encoder.ts index 7ba0515edd2..7ee7fc56afc 100644 --- a/js/publish/src/audio/encoder.ts +++ b/js/publish/src/audio/encoder.ts @@ -169,10 +169,8 @@ export class Encoder { return this.in.capture.peek(); } - // The encode settings known before encoding starts. Opus adds its decoder description from the - // first encoder output without feeding that catalog-only update back into the encoder. - #config = new Signal(undefined); - #decoderDescription = new Signal<{ config: Catalog.AudioConfig; description: Catalog.Hex } | undefined>(undefined); + // The exact encoder settings and probed decoder description advertised before demand arrives. + #config = new Signal<(Resolved & { encoder: AudioEncoderConfig }) | undefined>(undefined); readonly #out: EncoderOutput = { error: new Signal(undefined), @@ -279,8 +277,8 @@ export class Encoder { try { run.call(this, effect); } catch (err) { - this.#failures.update((n) => n + 1); - effect.cleanup(() => this.#failures.update((n) => n - 1)); + if (run === this.#runConfig) this.#config.set(undefined); + this.#fail(effect); throw err; } }); @@ -394,7 +392,8 @@ export class Encoder { const capture = effect.get(this.in.capture); const format = capture ? effect.get(capture.out.format) : undefined; const fatal = effect.get(this.#out.error); - if (!enabled || !format || fatal) return; + const resolved = effect.get(this.#config); + if (!enabled || !format || fatal || !resolved) return; this.#encode(broadcast.baseline, format, effect); }); @@ -469,15 +468,83 @@ export class Encoder { const capture = effect.get(this.in.capture); const captured = capture ? effect.get(capture.out.format) : undefined; if (!effect.get(this.in.enabled) || !captured) { - effect.set(this.#config, undefined); + this.#config.set(undefined); return; } - effect.set(this.#config, resolve(captured, effect.get(this.codec))); + const resolved = resolve(captured, effect.get(this.codec)); + const source = capture ? effect.get(capture.in.source) : undefined; + const kind = source ? sourceKind(source) : "auto"; + const encoderConfig = toEncoderConfig(resolved, kind, this.#opusOptions(effect)); + if (resolved.catalog.codec !== "opus") { + this.#config.set({ ...resolved, encoder: encoderConfig }); + return; + } + + const superseded = effect.abort; + effect.spawn(async () => { + try { + await Util.Libav.polyfill(); + if (superseded.aborted) return; + let description: Catalog.Hex | undefined; + let failure: Error | undefined; + const encoder = new AudioEncoder({ + output: (_frame, metadata) => { + const source = metadata?.decoderConfig?.description; + if (!source) return; + const bytes = ArrayBuffer.isView(source) + ? new Uint8Array(source.buffer, source.byteOffset, source.byteLength) + : new Uint8Array(source); + description = Util.Hex.fromBytes(bytes); + }, + error: (error) => { + failure = error; + }, + }); + effect.cleanup(() => { + if (encoder.state !== "closed") encoder.close(); + }); + encoder.configure(encoderConfig); + const samples = Math.round( + (encoderConfig.sampleRate * (resolved.frameDuration ?? Time.Micro(20_000))) / 1_000_000, + ); + const frame = new AudioData({ + format: "f32-planar", + sampleRate: encoderConfig.sampleRate, + numberOfChannels: encoderConfig.numberOfChannels, + numberOfFrames: samples, + timestamp: 0, + data: new Float32Array(samples * encoderConfig.numberOfChannels), + }); + try { + encoder.encode(frame); + } finally { + frame.close(); + } + await effect.race(encoder.flush()); + if (superseded.aborted) return; + if (failure) throw failure; + encoder.close(); + this.#config.set({ + ...resolved, + catalog: { ...resolved.catalog, ...(description === undefined ? {} : { description }) }, + encoder: encoderConfig, + }); + } catch (error) { + if (superseded.aborted) return; + this.#config.set(undefined); + this.#fail(effect); + throw error; + } + }); + } + + #fail(effect: Effect): void { + this.#failures.update((n) => n + 1); + effect.cleanup(() => this.#failures.update((n) => n - 1)); } - // Publish the config immediately so a consumer can request the demand-gated track. Once encoding - // starts, republish Opus with the exact decoder description reported for that encoder config. A + // Publish the probed config before a consumer can request the demand-gated track. A // disabled rendition keeps its last config, since muting also releases the capture, and stays // disabled after re-enabling until the reopened capture reports its format. #runCatalog(effect: Effect): void { @@ -498,13 +565,11 @@ export class Encoder { } this.#paused = false; - const decoder = effect.get(this.#decoderDescription); - const catalog = decoder?.config === config ? { ...config, description: decoder.description } : config; // The catalog bitrate is the most this track has been configured for, so lowering the knob // retunes the encoder without lowering what the catalog advertises. - track.bitrate = Math.max(track.bitrate, catalog.bitrate ?? 0); + track.bitrate = Math.max(track.bitrate, config.bitrate ?? 0); const bitrate = track.bitrate > 0 ? Catalog.u53(track.bitrate) : undefined; - track.last = { ...catalog, bitrate, ...estimate }; + track.last = { ...config, bitrate, ...estimate }; effect.set(this.#out.catalog, track.last); } @@ -538,20 +603,18 @@ export class Encoder { // Encode captured audio frames into whichever track producer is live. The broadcast owns the // track's lifetime, so this never closes it; a fatal encoder error aborts only the live track. #encode(baseline: Baseline, format: Format, effect: Effect): void { + // The parent owns the config subscription before the async codec scope is opened. + const resolved = this.#config.peek(); + if (!resolved) return; effect.spawn(async () => { // We're using an async polyfill temporarily for Safari support. await Util.Libav.polyfill(); effect.run((effect: Effect) => { const aborted = effect.abort; - const resolved = effect.get(this.#config); - if (!resolved) return; const config = resolved.catalog; - const capture = effect.get(this.in.capture); - const source = capture ? effect.get(capture.in.source) : undefined; - const kind: Kind = source ? sourceKind(source) : "auto"; - const encoderConfig = toEncoderConfig(resolved, kind, this.#opusOptions(effect)); + const encoderConfig = resolved.encoder; // WebCodecs rejects input whose rate doesn't match the encoder config outright, so // anything arriving at a rate the codec can't carry (a 44.1kHz file as Opus) has to @@ -569,13 +632,11 @@ export class Encoder { const framer = createFramer(resolved, config.sampleRate); const encoder = new AudioEncoder({ - output: (frame, metadata) => { + output: (frame) => { if (frame.type !== "key") { throw new Error("only key frames are supported"); } - this.#setDecoderDescription(config, metadata?.decoderConfig?.description); - this.#out.stats.update((stats) => ({ frames: stats.frames + 1, bytes: stats.bytes + frame.byteLength, @@ -691,19 +752,6 @@ export class Encoder { this.#live?.track.close(cause); } - #setDecoderDescription(config: Catalog.AudioConfig, source: AllowSharedBufferSource | undefined): void { - if (config.codec !== "opus" || !source) return; - - const bytes = ArrayBuffer.isView(source) - ? new Uint8Array(source.buffer, source.byteOffset, source.byteLength) - : new Uint8Array(source); - const description = Util.Hex.fromBytes(bytes); - const current = this.#decoderDescription.peek(); - if (current?.config === config && current.description === description) return; - - this.#decoderDescription.set({ config, description }); - } - close() { this.#signals.close(); } diff --git a/js/publish/src/broadcast.test.ts b/js/publish/src/broadcast.test.ts index a8af4db5bd2..b0d0c5f0be7 100644 --- a/js/publish/src/broadcast.test.ts +++ b/js/publish/src/broadcast.test.ts @@ -398,3 +398,272 @@ test("registering a sibling after an encoder failure keeps the network broadcast broadcast.close(); } }); + +// A name always means the same content, so a served rendition whose decoder identity changes moves to +// a fresh track name under the same catalog ID and epoch, ending the old track on its old identity. +// Fields a decoder needn't reinitialize for update the served track in place. +test("a served decoder identity change moves the rendition to a fresh track name", async () => { + const broadcast = new Broadcast({ origin: new Origin.Producer(), name: Path.from("identity.hang") }); + const video = broadcast.video("video"); + const sized = { + ...videoConfig, + codedWidth: Catalog.u53(1280), + codedHeight: Catalog.u53(720), + bitrate: Catalog.u53(1_000_000), + framerate: 30, + }; + video.config.set(sized); + await settle(); + + const net = broadcast.net.peek(); + if (!net) throw new Error("expected a network producer once connected"); + const epoch = net.route?.epoch; + const catalog = new Json.Snapshot.Consumer({ + track: net.track(Broadcast.CATALOG_TRACK).subscribe(), + }); + const first = net.track("video").subscribe(); + try { + expect((await catalog.latest())?.value.video?.renditions.video?.track).toBeUndefined(); + await settle(); + const track = video.track.peek(); + if (!track) throw new Error("expected the demanded track"); + track.appendGroup().close(); + + // Bitrate, frame rate, and coded size move either way on the served track. + for (const update of [ + { bitrate: Catalog.u53(250_000), framerate: 15 }, + { codedWidth: Catalog.u53(1920), codedHeight: Catalog.u53(1080), bitrate: Catalog.u53(4_000_000) }, + { codedWidth: Catalog.u53(640), codedHeight: Catalog.u53(360) }, + ]) { + video.config.set({ ...sized, ...update }); + await settle(); + const config = (await catalog.latest())?.value.video?.renditions.video; + expect(config).toMatchObject(update); + expect(config?.track).toBeUndefined(); + expect(video.track.peek()).toBe(track); + } + + // A codec change is a new identity. + video.config.set({ ...sized, codec: "vp8" }); + await settle(); + const config = (await catalog.latest())?.value.video?.renditions.video; + expect(config?.track).toBe("video/1"); + expect(config?.codec).toBe("vp8"); + expect(track.closed.peek()).toBeNull(); + expect(video.track.peek()).toBeUndefined(); + expect(broadcast.net.peek()).toBe(net); + expect(net.route?.epoch).toBe(epoch); + + const second = net.track("video/1").subscribe(); + await settle(); + const replacement = video.track.peek(); + expect(replacement?.name).toBe("video/1"); + expect(replacement?.appendGroup().sequence).toBe(0); + + // So is a display aspect change, which moves it again rather than redefining the second name. + video.config.set({ + ...sized, + codec: "vp8", + displayAspectWidth: Catalog.u53(4), + displayAspectHeight: Catalog.u53(3), + }); + await settle(); + expect((await catalog.latest())?.value.video?.renditions.video?.track).toBe("video/2"); + expect(replacement?.closed.peek()).toBeNull(); + second.close(); + } finally { + first.close(); + broadcast.close(); + } +}); + +// The counter skips every name this broadcast already uses, including an application's own sections. +test("a minted track name skips names already in the catalog", async () => { + const broadcast = new Broadcast({ origin: new Origin.Producer(), name: Path.from("collide.hang") }); + broadcast.audio("video/1").config.set(audioConfig); + broadcast.catalog.mutate((catalog) => { + catalog.json = { tracks: { meta: { mode: "stream", track: "video/2" } } }; + }); + const video = broadcast.video("video"); + video.config.set(videoConfig); + await settle(); + + video.config.set({ ...videoConfig, codec: "vp8" }); + await settle(); + expect((await readCatalog(broadcast))?.video?.renditions.video?.track).toBe("video/3"); + + broadcast.close(); +}); + +// A removed rendition keeps its network track, so its name stays taken even if it never had a config. +test("a minted track name skips a removed rendition's track", async () => { + const broadcast = new Broadcast({ origin: new Origin.Producer(), name: Path.from("removed.hang") }); + const removed = broadcast.video("video/1"); + await settle(); + removed.close(); + const video = broadcast.video("video"); + video.config.set(videoConfig); + await settle(); + + video.config.set({ ...videoConfig, codec: "vp8" }); + await settle(); + expect((await readCatalog(broadcast))?.video?.renditions.video?.track).toBe("video/2"); + const net = broadcast.net.peek(); + if (!net) throw new Error("expected a network producer once connected"); + const subscriber = net.track("video/2").subscribe(); + await settle(); + expect(video.track.peek()?.name).toBe("video/2"); + + subscriber.close(); + broadcast.close(); +}); + +// A minted name belongs to its rendition's media, so registering it as an ID fails loud. +test("registering a replacement track's name throws", async () => { + const broadcast = new Broadcast({ origin: new Origin.Producer(), name: Path.from("owned.hang") }); + const video = broadcast.video("video"); + video.config.set(videoConfig); + await settle(); + video.config.set({ ...videoConfig, codec: "vp8" }); + await settle(); + video.config.set({ ...videoConfig, codec: "vp9" }); + await settle(); + expect((await readCatalog(broadcast))?.video?.renditions.video?.track).toBe("video/2"); + + // Both the current replacement and a retired one stay owned. + expect(() => broadcast.video("video/2")).toThrow("replacement track"); + expect(() => broadcast.video("video/1")).toThrow("replacement track"); + // A sibling still registers and serves. + broadcast.audio("audio").config.set(audioConfig); + await settle(); + expect((await readCatalog(broadcast))?.audio?.renditions.audio?.codec).toBe("opus"); + + broadcast.close(); +}); + +// A replacement minted before its rendition ever had a config is never served, but still owns its name. +test("a replacement track name is never reused, even one never served", async () => { + const broadcast = new Broadcast({ enabled: true, origin: new Origin.Producer(), name: Path.from("unserved.hang") }); + let video = broadcast.video("video"); + await settle(); + const net = broadcast.net.peek(); + if (!net) throw new Error("expected a network producer once connected"); + + // Fail the rendition on its current track, then register a replacement for it. + const fail = async (name: string) => { + const sub = net.track(name).subscribe(); + await settle(); + video.track.peek()?.close(new Error("encoder failed")); + video.close(); + await settle(); + sub.close(); + video = broadcast.video("video"); + await settle(); + }; + await fail("video"); + await fail("video/1"); + await fail("video/2"); + expect(net.track("video/3").subscribe()).toBeDefined(); + expect(() => broadcast.video("video/1")).toThrow("replacement track"); + + video.config.set(videoConfig); + await settle(); + expect((await readCatalog(broadcast))?.video?.renditions.video?.track).toBe("video/3"); + + broadcast.close(); +}); + +test("a rendition config pointing at another track is refused", async () => { + const broadcast = new Broadcast({ origin: new Origin.Producer(), name: Path.from("remote.hang") }); + const video = broadcast.video("video"); + video.config.set(videoConfig); + await settle(); + + // A remote reference would send viewers to media this publisher never writes. + video.config.set({ + ...videoConfig, + codec: "vp8", + broadcast: "source" as Catalog.RelativeBroadcast, + track: "video", + }); + await settle(); + expect((await readCatalog(broadcast))?.video?.renditions.video?.codec).toBe(videoConfig.codec); + + video.config.set({ ...videoConfig, codec: "vp8" }); + await settle(); + expect((await readCatalog(broadcast))?.video?.renditions.video?.codec).toBe("vp8"); + + broadcast.close(); +}); + +// The served identity is a snapshot, so a producer mutating a container it shares can't rewrite it. +test("mutating a shared container still takes a fresh track name", async () => { + const broadcast = new Broadcast({ origin: new Origin.Producer(), name: Path.from("shared.hang") }); + const video = broadcast.video("video"); + const container = { kind: "cmaf" as const, init: "AAAA" }; + video.config.set({ ...videoConfig, container }); + await settle(); + expect((await readCatalog(broadcast))?.video?.renditions.video?.track).toBeUndefined(); + + container.init = "BBBB"; + video.config.set({ ...videoConfig, container, bitrate: Catalog.u53(1_000_000) }); + await settle(); + expect((await readCatalog(broadcast))?.video?.renditions.video?.track).toBe("video/1"); + + broadcast.close(); +}); + +test("a name returning with a different identity takes a fresh track name", async () => { + const broadcast = new Broadcast({ origin: new Origin.Producer(), name: Path.from("returning.hang") }); + let audio = broadcast.audio("audio"); + audio.config.set(audioConfig); + await settle(); + expect((await readCatalog(broadcast))?.audio?.renditions.audio?.track).toBeUndefined(); + + // The same identity returns to its track, even at a lower bitrate. + audio.close(); + await settle(); + audio = broadcast.audio("audio"); + audio.config.set({ ...audioConfig, bitrate: Catalog.u53(32_000) }); + await settle(); + expect((await readCatalog(broadcast))?.audio?.renditions.audio?.track).toBeUndefined(); + + audio.close(); + await settle(); + audio = broadcast.audio("audio"); + audio.config.set({ ...audioConfig, numberOfChannels: Catalog.u53(1) }); + await settle(); + expect((await readCatalog(broadcast))?.audio?.renditions.audio?.track).toBe("audio/1"); + + broadcast.close(); +}); + +// Nobody has seen an unserved catalog, so its renditions settle without minting names. +test("identity changes before the first announce keep the catalog ID as the track name", async () => { + const announce = new Signal(false); + const broadcast = new Broadcast({ origin: new Origin.Producer(), name: Path.from("unserved.hang"), announce }); + const video = broadcast.video("video"); + video.config.set({ ...videoConfig, bitrate: Catalog.u53(1_000_000) }); + await settle(); + video.config.set({ ...videoConfig, codec: "vp8", bitrate: Catalog.u53(500_000) }); + await settle(); + + announce.set(true); + await settle(); + const net = broadcast.net.peek(); + if (!net) throw new Error("expected a network producer once connected"); + const catalog = new Json.Snapshot.Consumer({ + track: net.track(Broadcast.CATALOG_TRACK).subscribe(), + }); + const config = (await catalog.latest())?.value.video?.renditions.video; + expect(config?.codec).toBe("vp8"); + expect(config?.bitrate).toBe(Catalog.u53(500_000)); + expect(config?.track).toBeUndefined(); + + // The first snapshot fixed the name's identity. + video.config.set(videoConfig); + await settle(); + expect((await catalog.latest())?.value.video?.renditions.video?.track).toBe("video/1"); + + broadcast.close(); +}); diff --git a/js/publish/src/broadcast.ts b/js/publish/src/broadcast.ts index 8dbe8a58fbf..68f940c8a01 100644 --- a/js/publish/src/broadcast.ts +++ b/js/publish/src/broadcast.ts @@ -1,5 +1,6 @@ import * as Catalog from "@moq/hang/catalog"; import * as Container from "@moq/hang/container"; +import * as Json from "@moq/json"; import * as Moq from "@moq/net"; import { Effect, type Getter, getter, type Inputs, type Readonlys, Signal } from "@moq/signals"; import { CatalogProducer } from "./catalog"; @@ -71,6 +72,7 @@ export class Broadcast { // and is `undefined` in between. It lives in the origin rather than any session, so it spans // reconnects. Exposed so an application can serve its own tracks alongside the built-in // catalog/audio/video, e.g. `net.createTrack("meta.json")` plus a matching `catalog` section. + // Names under a rendition's ID (`video/1`) are reserved for that rendition's replacement tracks. // Reacquire it via an effect, since a rename swaps in a fresh producer. readonly net = new Signal(undefined); @@ -91,9 +93,22 @@ export class Broadcast { // The kind of each rendition track in the current network broadcast. A new one starts over. #kinds = new Map(); - // Replacement tracks keep their catalog IDs but never reuse a closed track name. + // Each catalog ID's track name in the current network broadcast, when it is not the ID itself. + // A replacement for a closed track, or a rendition whose media identity changed, keeps its + // catalog ID under a fresh track name, so a name always means the same content. readonly #names = new Signal>({}); + // Every replacement track name minted in the current network broadcast, even one never served. + #replacements = new Set(); + + // The decoder identity of each track name the current network broadcast's catalog has served, or + // undefined before it serves one. Kept across a rendition's removal, so a name returning with + // different media takes a fresh name too. + #served: Map | undefined; + + // The identity of each track name in the latest catalog fold, recorded when the catalog is first served. + #listed = new Map(); + #signals = new Effect(); constructor(props?: Inputs) { @@ -143,6 +158,10 @@ export class Broadcast { if (previous !== undefined && previous !== kind) { throw new Error(`rendition ${name} was registered as ${previous}, not ${kind}`); } + // A replacement track holds or held the name, so another rendition's media owns it. + if (this.#replacements.has(name)) { + throw new Error(`rendition ${name} names another rendition's replacement track`); + } const track = new Signal(undefined); this.#tracks.set(name, track); @@ -179,12 +198,28 @@ export class Broadcast { const video: Record = {}; const audio: Record = {}; const text: Record = {}; + const listed = new Map(); + const renamed: Record = {}; for (const rendition of Object.values(renditions)) { const source = enabled ? effect.get(rendition.config) : undefined; if (source === undefined) continue; - const track = names[rendition.name]; - const config = track ? { ...(source as object), track } : source; + // The rendition's media is this broadcast's own track, so its config can't point elsewhere. + const target = source as { broadcast?: string; track?: string }; + if (target.broadcast !== undefined || target.track !== undefined) { + throw new Error(`rendition ${rendition.name} config sets broadcast or track, which the broadcast owns`); + } + let track = names[rendition.name] ?? rendition.name; + // Everything else (bitrate, frame rate, coded size, ...) updates the served track in place. + const media = identity(source); + const served = this.#served?.get(track); + if (served && !Json.deepEqual(served, media)) { + track = this.#mint(rendition.name); + renamed[rendition.name] = track; + } + this.#served?.set(track, media); + listed.set(track, media); + const config = track !== rendition.name ? { ...(source as object), track } : source; if (rendition.kind === "video") { video[rendition.name] = config as Catalog.VideoConfig; @@ -223,6 +258,45 @@ export class Broadcast { delete catalog.text; } }); + this.#listed = listed; + + // The network loop ends each renamed rendition's old track and serves the new name. + if (Object.keys(renamed).length > 0) this.#names.update((names) => ({ ...names, ...renamed })); + } + + // The next unused `/` track name in the current network broadcast. This publisher owns the + // namespace and reserves `/` for the rendition (see `net`), so skipping every name it has + // served, registered, or listed in the catalog (including an application's own sections) is + // enough to never reuse one. + #mint(id: string): string { + const taken = new Set([ + ...this.#replacements, + ...(this.#served?.keys() ?? []), + ...Object.keys(this.#renditions.peek()), + // Every ID that held a track here, even one removed before it had a config. + ...this.#kinds.keys(), + ]); + // A no-op edit is the catalog's only read; it publishes nothing. + this.catalog.mutate((catalog) => { + for (const entries of [ + catalog.audio?.renditions, + catalog.video?.renditions, + catalog.text?.renditions, + catalog.json?.tracks, + catalog.binary?.tracks, + ]) { + for (const [key, config] of Object.entries(entries ?? {})) { + taken.add(key); + taken.add(Catalog.track(key, config)); + } + } + }); + for (let n = 1; ; n++) { + const name = `${id}/${n}`; + if (taken.has(name)) continue; + this.#replacements.add(name); + return name; + } } #run(effect: Effect) { @@ -258,6 +332,8 @@ export class Broadcast { // again finds its track; the catalog's stay out of reach of a colliding rendition. const tracks = new Map }>(); this.#names.set({}); + this.#replacements = new Set(); + this.#served = undefined; const kinds = new Map(); this.#kinds = kinds; effect.cleanup(() => { @@ -277,6 +353,8 @@ export class Broadcast { // is whatever the catalog holds at announce time. if (!serving) { serving = true; + // From here a name keeps the identity this first snapshot gives it. + this.#served = new Map(this.#listed); for (const [name, compression] of [ [Broadcast.CATALOG_TRACK, false], [Broadcast.CATALOG_TRACK_COMPRESSED, true], @@ -302,6 +380,7 @@ export class Broadcast { // so capture and encoding still stop when the final subscriber leaves. effect.run((inner) => { const renditions = inner.get(this.#renditions); + const names = inner.get(this.#names); for (const rendition of Object.values(renditions)) { const signal = this.#tracks.get(rendition.name); @@ -310,10 +389,20 @@ export class Broadcast { let entry = tracks.get(rendition.name); const cause = entry && errors.get(entry.owner)?.peek(); if (entry && cause) entry.track.close(cause); - // An error leaves this rendition down. Only a replacement registration may - // recover, with a fresh track name so its group sequence can start over. - if (!entry || (entry.owner !== rendition && entry.track.closed.peek() !== undefined)) { - const name = entry ? `${rendition.name}/${Moq.Epoch.mint()}` : rendition.name; + const name = names[rendition.name] ?? rendition.name; + if (entry && entry.track.name !== name) { + // The catalog moved this rendition to a new name for new media. End the old track + // cleanly on its last identity, so its subscribers follow the catalog. + entry.track.close(); + entry = undefined; + } else if (entry && entry.owner !== rendition && entry.track.closed.peek() !== undefined) { + // An error leaves this rendition down. Only a replacement registration may recover, + // with a fresh track name so its group sequence can start over. + const fresh = this.#mint(rendition.name); + this.#names.update((names) => ({ ...names, [rendition.name]: fresh })); + continue; + } + if (!entry) { const track = broadcast.createTrack( name, Container.trackInfo({ maxAge, priority: Catalog.PRIORITY[rendition.kind] }), @@ -321,7 +410,6 @@ export class Broadcast { entry = { track, owner: rendition }; tracks.set(rendition.name, entry); kinds.set(rendition.name, rendition.kind); - if (name !== rendition.name) this.#names.update((names) => ({ ...names, [rendition.name]: name })); } else { entry.owner = rendition; } @@ -344,3 +432,32 @@ export class Broadcast { this.#signals.close(); } } + +// The fields a consumer can't change without reinitializing its decoder (or demuxer, for +// `container`), so changing any of them takes a new track name. +type Identity = Pick< + Partial, + | "codec" + | "description" + | "container" + | "sampleRate" + | "numberOfChannels" + | "displayAspectWidth" + | "displayAspectHeight" +>; + +// Cloned, so a producer mutating a config it shares (such as its container) can't rewrite what a +// name was served with. +function identity(config: unknown): Identity { + const { codec, description, container, sampleRate, numberOfChannels, displayAspectWidth, displayAspectHeight } = + config as Identity; + return structuredClone({ + codec, + description, + container, + sampleRate, + numberOfChannels, + displayAspectWidth, + displayAspectHeight, + }); +} diff --git a/js/publish/src/catalog.test.ts b/js/publish/src/catalog.test.ts index 073531a1b30..c802272e69e 100644 --- a/js/publish/src/catalog.test.ts +++ b/js/publish/src/catalog.test.ts @@ -239,6 +239,28 @@ for (const field of ["jitter", "delay"] as const) { } } +// Same-named tracks in different broadcasts are different tracks. +test("catalog keeps estimates apart for same-named tracks in different broadcasts", () => { + const catalog = new CatalogProducer(); + const rendition = (broadcast: string, jitter: number) => ({ + codec: "opus", + container: { kind: "legacy" as const }, + sampleRate: Catalog.u53(48000), + numberOfChannels: Catalog.u53(2), + broadcast: broadcast as Catalog.RelativeBroadcast, + track: "audio", + jitter: Catalog.u53(jitter), + }); + catalog.mutate((value) => { + value.audio = { renditions: { en: rendition("../en", 64), es: rendition("../es", 128) } }; + }); + expect(() => + catalog.mutate((value) => { + delete value.audio?.renditions.es; + }), + ).not.toThrow(); +}); + for (const section of ["json", "binary"] as const) { test(`catalog refuses zero or decreasing ${section} jitter without retaining it`, () => { const catalog = new CatalogProducer(); @@ -434,3 +456,43 @@ describe("estimate publication window", () => { expect((await consumer.latest())?.value.video?.renditions.v.jitter).toBe(Catalog.u53(3)); }); }); + +// Each ID's estimate is checked on its own, even when two IDs alias one track. +test("catalog refuses a falling estimate on one alias of a track", () => { + const catalog = new CatalogProducer(); + const alias = (jitter: number) => ({ + codec: "opus", + container: { kind: "legacy" as const }, + sampleRate: Catalog.u53(48000), + numberOfChannels: Catalog.u53(2), + track: "audio", + jitter: Catalog.u53(jitter), + }); + catalog.mutate((value) => { + value.audio = { renditions: { en: alias(100), es: alias(100) } }; + }); + expect(() => + catalog.mutate((value) => { + value.audio = { renditions: { en: alias(10), es: alias(100) } }; + }), + ).toThrow("jitter cannot decrease"); +}); + +// A renamed rendition's new track starts its estimates over. +test("catalog lets a renamed track start its estimates over", () => { + const catalog = new CatalogProducer(); + const config = { codec: "vp8", container: { kind: "legacy" as const } }; + catalog.mutate((value) => { + value.video = { renditions: { video: { ...config, jitter: Catalog.u53(100) } } }; + }); + catalog.mutate((value) => { + value.video = { renditions: { video: { ...config, codec: "vp9", track: "video/1", jitter: Catalog.u53(10) } } }; + }); + expect(() => + catalog.mutate((value) => { + value.video = { + renditions: { video: { ...config, codec: "vp9", track: "video/1", jitter: Catalog.u53(5) } }, + }; + }), + ).toThrow("jitter cannot decrease"); +}); diff --git a/js/publish/src/catalog.ts b/js/publish/src/catalog.ts index 17bd38e6b57..92b523f06ab 100644 --- a/js/publish/src/catalog.ts +++ b/js/publish/src/catalog.ts @@ -31,15 +31,14 @@ export class CatalogProducer { fn(value); // Every consumer refuses a catalog that fails this, so refuse the edit before it reaches one. Catalog.checkRenditions(value); + // Estimates only rise for a track; a renamed rendition starts its new track over. for (const field of ["jitter", "delay"] as const) { - const previous = advertised(this.#value, field); - for (const [section, next] of Object.entries(advertised(value, field))) { - for (const [name, estimate] of Object.entries(next)) { - if (estimate === 0) throw new Error(`omit ${field} rather than advertising 0`); - const before = previous[section]?.[name]; - if (before !== undefined && (estimate === undefined || estimate < before)) { - throw new Error(`${field} cannot decrease for an existing track`); - } + const previous = new Map(advertised(this.#value, field)); + for (const [track, estimate] of advertised(value, field)) { + if (estimate === 0) throw new Error(`omit ${field} rather than advertising 0`); + const before = previous.get(track); + if (before !== undefined && (estimate === undefined || estimate < before)) { + throw new Error(`${field} cannot decrease for an existing track`); } } } @@ -111,20 +110,25 @@ function structure(catalog: Catalog.Root): Catalog.Root { return value; } -/** Every track's advertised `field`, by section and then track name. */ -function advertised( - catalog: Catalog.Root, - field: "jitter" | "delay", -): Record> { - const pick = (tracks: Record | undefined) => - Object.fromEntries(Object.entries(tracks ?? {}).map(([name, config]) => [name, config[field]])); - return { - audio: pick(catalog.audio?.renditions), - video: pick(catalog.video?.renditions), - text: pick(catalog.text?.renditions), - json: pick(catalog.json?.tracks), - binary: pick(catalog.binary?.tracks), +type Advertised = { broadcast?: string; track?: string; jitter?: number; delay?: number }; + +/** Every entry's advertised `field`, keyed by its section, ID, broadcast, and track name. */ +function advertised(catalog: Catalog.Root, field: "jitter" | "delay"): [string, number | undefined][] { + const sections = { + audio: catalog.audio?.renditions, + video: catalog.video?.renditions, + text: catalog.text?.renditions, + json: catalog.json?.tracks, + binary: catalog.binary?.tracks, }; + return Object.entries(sections).flatMap(([section, tracks]) => + Object.entries((tracks ?? {}) as Record).map( + ([id, config]): [string, number | undefined] => [ + JSON.stringify([section, id, config.broadcast, Catalog.track(id, config)]), + config[field], + ], + ), + ); } // An empty root advertising the wall time of `performance.now() === 0`, the zero every js/publish diff --git a/js/publish/src/rendition.ts b/js/publish/src/rendition.ts index d8ffc262ad4..0fe646a6038 100644 --- a/js/publish/src/rendition.ts +++ b/js/publish/src/rendition.ts @@ -19,13 +19,18 @@ export class Rendition { /** * The catalog entry for this rendition, written by its producer (usually an encoder). - * `undefined` omits it from the catalog (e.g. while disabled). + * `undefined` omits it from the catalog (e.g. while disabled). Changing a field a decoder must + * reinitialize for (codec, description, container, sample rate, channel count, or display aspect) + * ends the current track and serves the rendition under a fresh catalog `track` name, so a name + * always means the same content. Every other field updates the current track in place. The + * broadcast owns `broadcast` and `track`, so a config setting either is refused. */ readonly config = new Signal(undefined); /** * The live track producer while a subscriber is attached, `undefined` otherwise. - * Producers should encode only while this is set (the demand gate). + * Producers should encode only while this is set (the demand gate), and only media matching + * the current {@link config}: a new identity swaps in a new track. */ readonly track: Getter; diff --git a/js/publish/src/video/encoder.test.ts b/js/publish/src/video/encoder.test.ts index 93147850df4..4f9378dc929 100644 --- a/js/publish/src/video/encoder.test.ts +++ b/js/publish/src/video/encoder.test.ts @@ -1479,3 +1479,48 @@ test("a refused first ceiling recovers at a smaller size", async () => { error.mockRestore(); } }); + +// A decoder follows a resize in band, so outgrowing the advertised size re-probes for a codec that +// covers it and updates the served track in place while the codec string holds. A shrink stays in band. +test("ceiling growth updates the served track in place", async () => { + using _videoEncoder = installFakeVideoEncoder(); + using _probe = spyOn(FakeVideoEncoder, "isConfigSupported").mockResolvedValue({ supported: true }); + using _now = spyOn(performance, "now").mockReturnValue(0); + const broadcast = new Broadcast({ origin: new Moq.Origin.Producer(), name: Moq.Path.from("ceiling.hang") }); + const capture = { + in: { source: new Signal({ getSettings: () => ({ frameRate: 30 }), getConstraints: () => ({}) }) }, + out: { display: new Signal({ width: 640, height: 480 }), frames: new Signal(undefined) }, + }; + const encoder = new Encoder("video", { broadcast, capture: capture as never, config: { frameRate: 60 } }); + const net = broadcast.net.peek() ?? (await broadcast.net.changed()); + if (!net) throw new Error("expected a network broadcast"); + const subscriber = net.track(Broadcast.CATALOG_TRACK).subscribe(); + const catalog = new Json.Snapshot.Consumer({ track: subscriber }); + const next = async (done: (config: Catalog.VideoConfig) => boolean) => { + for (;;) { + const config = (await catalog.latest())?.value.video?.renditions.video; + if (config && done(config)) return config; + } + }; + try { + const initial = await next(() => true); + expect(initial).toMatchObject({ codedWidth: 640, codedHeight: 480, framerate: 60 }); + expect(initial.track).toBeUndefined(); + + capture.out.display.set({ width: 320, height: 240 }); + encoder.config.set({ frameRate: 15 }); + let live = encoder.out.resolved.peek(); + while (live?.width !== 320 || live.framerate !== 15) live = await encoder.out.resolved.changed(); + expect(encoder.out.catalog.peek()).toMatchObject({ codedWidth: 640, codedHeight: 480, framerate: 60 }); + + capture.out.display.set({ width: 1280, height: 720 }); + const grown = await next((config) => config.codedWidth === 1280); + expect(grown).toMatchObject({ codec: initial.codec, codedHeight: 720, framerate: 60 }); + expect(grown.track).toBeUndefined(); + expect(broadcast.net.peek()).toBe(net); + } finally { + subscriber.close(); + encoder.close(); + broadcast.close(); + } +}); diff --git a/js/publish/src/video/encoder.ts b/js/publish/src/video/encoder.ts index e3da139daf6..70daa7fb84d 100644 --- a/js/publish/src/video/encoder.ts +++ b/js/publish/src/video/encoder.ts @@ -111,7 +111,8 @@ type EncoderOutput = { * attached (the demand gate). Rename by constructing a new encoder; the name is not a signal. * * The catalog advertises the largest size encoded so far as the coded-size ceiling, so a capture that - * shrinks resizes in band at a keyframe. Outgrowing the ceiling re-probes and advertises a larger one. + * shrinks resizes in band at a keyframe. Outgrowing the ceiling re-probes and advertises a larger one, + * in place unless the codec string changes, which the broadcast serves under a new track name. */ export class Encoder { /** The full track name of this rendition, e.g. `"video/hd"`. */ diff --git a/quest/m0/broadcast-epoch/catalog-references.md b/quest/m0/broadcast-epoch/catalog-references.md index 4279774411a..8c6ed71eec9 100644 --- a/quest/m0/broadcast-epoch/catalog-references.md +++ b/quest/m0/broadcast-epoch/catalog-references.md @@ -87,6 +87,5 @@ guarantee and the selected legacy best-effort behavior. - [Source pin](https://github.com/moq-dev/moq/pull/5252) - exporters share their resolved catalog handle - [JS consume identity](https://github.com/moq-dev/moq/pull/5254) - JS resolves the actual serving identity - [Retired requests](https://github.com/moq-dev/moq/pull/5294) - held handles cannot emit unpinned requests after retirement -- [JS identity enforcement](/quest/m1/catalog-identity-js.md) - catalog updates cannot redefine a track's content in `@moq/publish` - [Rust identity enforcement](/quest/m1/catalog-identity-rs.md) - the same for the Rust publishers - [IETF epochs](/quest/m1/ietf-epochs.md) - verified catalog identity over negotiated IETF sessions diff --git a/quest/m1/README.md b/quest/m1/README.md index c40699aa328..b087d03ddc8 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -101,8 +101,7 @@ above them. - [JSON window timestamps](/quest/m1/json-window-timestamps.md) - Rust and JS window consumers return each event's frame timestamp like the snapshot and stream consumers - [JavaScript FETCH](/quest/m1/js-fetch.md) - browser publishers answer IETF FETCH through the JS ranges request surface - [IETF fill timeout](/quest/m1/ietf-fill-timeout.md) - native FILL_TIMEOUT bounds upstream waiting on drafts 18-22, alongside local content-age enforcement -- [JS identity enforcement](/quest/m1/catalog-identity-js.md) - `@moq/publish` republishes under a new epoch instead of redefining a served track, and probes the Opus description before advertising it -- [Rust identity enforcement](/quest/m1/catalog-identity-rs.md) - `moq-mux` refuses a redefined rendition, and every Rust publisher resizes in band below its ceiling or mints a new rendition or epoch +- [Rust identity enforcement](/quest/m1/catalog-identity-rs.md) - `moq-mux` refuses a redefined rendition, and every Rust publisher mints a new rendition name for a decoder identity change and updates everything else in place - [Archive](/quest/m1/archive/README.md) - record selected tracks to any object_store and replay them over FETCH or derived HLS; the catalog entry and format may break in place, since no archives exist - [In-band auth](/quest/m1/auth/README.md) - a session tells its peer what it may publish and subscribe to, unions tokens presented in band, and fails loud on an out-of-scope publish - [IETF epochs](/quest/m1/ietf-epochs.md) - negotiate explicit broadcast identity on drafts 17-22, retaining unchanged paths and epochless clients on updated relays diff --git a/quest/m1/catalog-identity-js.md b/quest/m1/catalog-identity-js.md deleted file mode 100644 index b38a5a4abf4..00000000000 --- a/quest/m1/catalog-identity-js.md +++ /dev/null @@ -1,51 +0,0 @@ -# [M] JS identity enforcement - -## Goal - -`@moq/publish` never redefines a track it has served. Once a catalog is -served, a change to a rendition's identity (`codec`, `description`, -`container`, `sampleRate`, `numberOfChannels`) or growth past its -`codedWidth`/`codedHeight` ceiling republishes the broadcast under a new -epoch rather than mutating the track, per the hang draft's -[Track Identity](/drafts/draft-lcurley-moq-hang.md#identity) section. Live -state and rising maxima stay plain catalog updates, and a picture that shrinks -below its fixed ceiling resizes in band with no update at all. - -## Plan - -- ✅ Decided 2026-10-13 (maintainer, #5459): enforce by republishing under a - new epoch. Once a catalog is served, an edit that redefines a served - rendition stops feeding the old catalog track (so it never carries the new - identity) and `Broadcast` restarts under a new epoch, as it already does - for a closed rendition track. Refusing the change, or minting a new - rendition name, were rejected. -- ✅ Decided 2026-10-13 (maintainer, #5459): probe the Opus `description` - before advertising it. Today the audio encoder adds the encoder-reported - description on the first encode, which would restart the broadcast on - every first subscriber. Probe it with a throwaway encoder, as the video - encoder probes its codec string, using the exact encoder config (kind - defaults and Opus knobs included, since they move pre-skip). -- ✅ Decided 2026-10-13 (maintainer, #5459): the video ceiling stays the - per-side maximum of every size encoded, so outgrowing it is the one - resize that republishes. - -Track identities per network broadcast in `CatalogProducer` (keyed by section -and name, kept across a rendition's removal so a returning name is compared -too), recorded from the first serve and cleared when every served track ends. -An absent ceiling may be filled once. Bitrate and framerate decreases are -refused like jitter and delay. The encoders' ceiling and maxima are keyed by -the `Broadcast` component today, so a restart under a new epoch inherits the -old run's limits; key them per epoch with the identities. - -What then republishes: video ceiling growth, a codec knob or re-probe that -changes the codec string, an audio capture that changes rate or channel count, -and an audio codec change. - -Test a served identity change reaching a new epoch with the old catalog track -ending on the old identity, a returning name, the Opus description present -from the first catalog, and a shrink or maxima rise staying in place. - -## Related - -- [Rust identity enforcement](/quest/m1/catalog-identity-rs.md) - the same contract for the Rust publishers -- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - a restart is a new epoch rather than a changed track diff --git a/quest/m1/catalog-identity-rs.md b/quest/m1/catalog-identity-rs.md index 022de8b6170..5f5eac72145 100644 --- a/quest/m1/catalog-identity-rs.md +++ b/quest/m1/catalog-identity-rs.md @@ -3,41 +3,51 @@ ## Goal No Rust publisher redefines a track it has published. A change to a -rendition's identity (`codec`, `description`, `container`, `sample_rate`, -`channel_count`) or growth past its `coded_width`/`coded_height` ceiling -publishes a new rendition name or a new broadcast epoch, per the hang draft's -[Track Identity](/drafts/draft-lcurley-moq-hang.md#identity) section, and -`bitrate`/`framerate` never fall. A replayed archive catalog then describes -every group of the tracks it lists. +rendition's identity, the fields a consumer cannot change without +reinitializing its decoder or demuxer (`codec`, `description`, `container`, +`display_aspect_width`/`display_aspect_height`, `sample_rate`, +`channel_count`), publishes a new rendition name, per the hang draft's +[Track Identity](/drafts/draft-lcurley-moq-hang.md#identity) section. Every +other field, including coded size, `bitrate`, and `framerate`, updates in +place. A replayed archive catalog then describes every group of the tracks it +lists. ## Plan - ✅ Decided 2026-10-13 (maintainer, #5459): split from the catalog identity - quest. The contract, docs, and the `js/publish` ceiling landed there; - [JS identity enforcement](/quest/m1/catalog-identity-js.md) owns the + quest. The contract and docs landed there; `@moq/publish` enforces the browser side. -- ✅ Decided 2026-10-13 (maintainer, #5459): JS enforces identity by - republishing under a new epoch with the Opus description probed up front, - and the `js/publish` video ceiling keeps growing per side. - -Open: whether each Rust violator mints a new rendition name (as FLV, HLS -import, and transcode rungs already do) or restarts under a new epoch (as JS -will). Recommended: mint a name where the catalog can keep both, and take a -new epoch only for a whole-broadcast restart. +- ✅ Decided 2026-10-13 (maintainer, #5459): the Opus description is probed + up front. +- ✅ Decided 2026-10-15 (maintainer): mint a new rendition name, not a new + epoch; reverses the #5459 decision. Restarting the whole broadcast is an + extreme response to one rendition changing, and a name always means the + same content, so changed content gets a new name. `@moq/publish` keeps the + catalog ID and points its `track` at the next free `/` (#5552); each + Rust violator mints a name likewise, as FLV, HLS import, and transcode rungs + already do. A new epoch stays reserved for a whole-broadcast restart. +- ✅ Decided 2026-10-15 (maintainer): only fields that force a decoder + reinitialization mint a new track; everything else updates in place. New + tracks have user impact, so they are minted only when they must be. This + drops the #5459 coded-size ceilings and the rule that `bitrate`/`framerate` + never fall: a decoder follows a resize in band, and `codec` names a level + covering every picture, so outgrowing the size takes a new name only when + the codec string changes. Guard `moq-mux` `Rendition::set`, `replace`, and `stage` -(`rs/moq-mux/src/catalog/tracks.rs`): refuse a change to identity or a set -ceiling, and keep `resolved`'s `bitrate`/`framerate` from falling. Then fix -each violator the 2026-10-13 audit found: +(`rs/moq-mux/src/catalog/tracks.rs`): refuse a change to identity, and let +every other field update. Then fix each violator the 2026-10-13 audit found, +minting a name where the identity changes and updating in place otherwise: - The h264/h265 importers (an inline SPS, or a new avcC/hvcC through `initialize`), av1 (a new sequence header, including the av1C defaults and the minimal placeholder), and vp8/vp9 (keyframe size; VP9's level follows it) rewrite in place, including MPEG-TS and `moq-rtc` ingest through them. - Below the ceiling, resolution changes in band with no catalog update. + A size change alone updates in place; a new description or codec string + mints a name. - A codec hint published up front is republished with the real - profile/level and size by the first keyframe. Publish once from the stream, - or treat the hint as the ceiling. + profile/level by the first keyframe. Publish once from the stream, or mint + when the codec string differs. - `moq-video` reopening onto another camera mode or backend (`source_resize_updates_the_published_rendition` asserts the rewrite). - `moq-c` re-declaring a config under the same name, and `moq-gst` reusing an @@ -45,23 +55,17 @@ each violator the 2026-10-13 audit found: - TS AAC/MP2/AC-3 and FLV legacy audio fix the config from the first header and silently ignore a later change; refuse or mint instead. A PMT that changes a PID's stream type keeps the old importer. -- `moq-audio`/`moq-ffi` `set_bitrate` raising past the advertised bitrate - should raise the catalog. +- `moq-audio`/`moq-ffi` `set_bitrate` should update the catalog's bitrate. FLV, fMP4, MKV, HLS import, and `moq-transcode` rungs already comply; keep -their tests. Update the tests that assert in-place rewrites (`tracks.rs`, -`moq-video`, `moq-gst`, `moq-c`, and the transcode fixture's same-name -resize). +their tests. Update the tests that assert in-place identity rewrites +(`tracks.rs`, `moq-video`, `moq-gst`, `moq-c`); a same-name resize, such as +the transcode fixture's, is now allowed. Later, at lower priority: watch and HLS export consumer coverage for in-band resolution changes, rendition switches, reconnects, and late joiners against -the contract. Watch's `renditionJitter` fallback -(`js/watch/src/video/playhead.ts`) reads `framerate` as the live cadence, but it -is now a maximum, so a stream running below it gets less than a frame interval -of buffer when no `jitter` is advertised. +the contract. ## Related -- [JS identity enforcement](/quest/m1/catalog-identity-js.md) - the same contract for `@moq/publish` - [Archive](/quest/m1/archive/README.md) - storage and replay consume the identity contract -- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - a restart is a new epoch rather than a changed track diff --git a/rs/hang/src/catalog/audio/mod.rs b/rs/hang/src/catalog/audio/mod.rs index 7f11d192ca3..01382bb0ea6 100644 --- a/rs/hang/src/catalog/audio/mod.rs +++ b/rs/hang/src/catalog/audio/mod.rs @@ -127,7 +127,7 @@ pub struct AudioConfig { #[serde(rename = "numberOfChannels")] pub channel_count: u32, - // The maximum bitrate of the audio track in bits per second. It only rises. + // The maximum bitrate of the audio track in bits per second. #[serde(default)] pub bitrate: Option, diff --git a/rs/hang/src/catalog/video/mod.rs b/rs/hang/src/catalog/video/mod.rs index b503aa8ef3a..581913e3411 100644 --- a/rs/hang/src/catalog/video/mod.rs +++ b/rs/hang/src/catalog/video/mod.rs @@ -227,7 +227,7 @@ pub struct VideoConfig { pub display_aspect_height: Option, // TODO color space - /// The maximum bitrate of the video track, if known. It only rises. + /// The maximum bitrate of the video track, if known. #[serde(default)] pub bitrate: Option, @@ -239,7 +239,7 @@ pub struct VideoConfig { )] pub enabled: bool, - /// The maximum frame rate of the video track, if known. It only rises. + /// The maximum frame rate of the video track, if known. #[serde(default)] pub framerate: Option,