diff --git a/doc/concept/audio-jitter.md b/doc/concept/audio-jitter.md index a7d55b60b45..291435da14d 100644 --- a/doc/concept/audio-jitter.md +++ b/doc/concept/audio-jitter.md @@ -621,8 +621,9 @@ with its accelerate alone. Neither generates replacement audio for a missing packet, so a hole in the media plays as a hole. The estimator above is unchanged. Both share the pitch search, admission, seams, and [PCM corpus](audio-jitter/stretch/corpus.json) below; they differ only in -[what they decide on](#browser-decision). The browser stretches only live -playback: buffered playback plays every sample it holds, as it always has. +[what they decide on](#browser-decision). Buffered browser playback plays +every sample it holds, but accelerates the same way to catch back up to the +shared clock. The source is WebRTC's [`TimeStretch`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/time_stretch.cc), @@ -686,6 +687,21 @@ accelerate after a refill catches up about as fast as that drop did. Arrivals th outrun the target underrun instead: that is the cost of a target short of the spread, and the estimator's to fix, not the stretch's. +Buffered playback holds its lookahead upstream and feeds the ring only up to +its target plus one 30 ms window, so it can always stage a full window to +stretch, and its depth says nothing about the clock. It decides on +`excess = clock - playhead` instead, the shared clock video plays from less the +playhead at the window's start, by the same three rules. A stall, a rise of the +delay, or a publisher clock running fast leaves audio behind the clock; it +plays faster until it is back within a pitch period, and never ahead of it, +since future-dated frames are what buffered playback holds. That holds for the +tail before a declared end too, since the clock, not the depth, measures it. +A delay under one window rests up to the window less the delay behind instead, +since a window stretches only once 30 ms past the playhead has arrived. +The main thread reads the clock with each of the worklet's state reports, +paired with the `AudioContext`'s `currentTime`, and the worklet carries it +forward on its own time until the next. + ### Pitch search and admission Use channel zero as NetEq does. All channels share the selected seam and lag, diff --git a/doc/lib/js/watch.md b/doc/lib/js/watch.md index 7efb5225cb7..8a577959420 100644 --- a/doc/lib/js/watch.md +++ b/doc/lib/js/watch.md @@ -245,7 +245,9 @@ By default the player minimizes latency: it time-stretches received audio that falls behind the delay back onto it, playing it slightly faster, and skips ahead only when media piles up far past it. Content produced faster than real time, such as a TTS response emitted in one burst with future timestamps, wants the opposite. Set -`lookahead` to how far ahead it may run and it plays through at the encoded pace: +`lookahead` to how far ahead it may run and it plays through at the encoded +pace. After a stall, or anything else that leaves audio behind video, it catches +up by playing slightly faster rather than skipping, so lip sync returns: ```html diff --git a/js/watch/src/audio/buffer.test.ts b/js/watch/src/audio/buffer.test.ts index fbf3cb5135f..14cfc913a95 100644 --- a/js/watch/src/audio/buffer.test.ts +++ b/js/watch/src/audio/buffer.test.ts @@ -180,9 +180,10 @@ describe("AudioBuffer", () => { const ahead = buffer.wait(120_000 as Micro, signal); expect(Bun.peek.status(ahead)).toBe("pending"); - // The floor lands and plays past the frame's headroom. + // The floor lands and plays past the frame's headroom, and a playing ring takes one stretch window more. await state(60_000, false, 0); - expect(Bun.peek.status(buffer.wait(170_000 as Micro, signal))).toBe("pending"); + expect(Bun.peek.status(buffer.wait(190_000 as Micro, signal))).toBe("fulfilled"); + expect(Bun.peek.status(buffer.wait(200_000 as Micro, signal))).toBe("pending"); expect(Bun.peek.status(ahead)).toBe("fulfilled"); } finally { close(); @@ -225,8 +226,8 @@ describe("AudioBuffer", () => { // The frame past the gap un-stalls the ring, which plays the gap out up to it. await state(10_000_000, false, 0); expect(Bun.peek.status(next)).toBe("fulfilled"); - expect(Bun.peek.status(buffer.wait(10_090_000 as Micro, signal))).toBe("fulfilled"); - expect(Bun.peek.status(buffer.wait(10_110_000 as Micro, signal))).toBe("pending"); + expect(Bun.peek.status(buffer.wait(10_130_000 as Micro, signal))).toBe("fulfilled"); + expect(Bun.peek.status(buffer.wait(10_150_000 as Micro, signal))).toBe("pending"); } finally { close(); } @@ -277,13 +278,14 @@ describe("Backpressure", () => { expect(Bun.peek.status(gate.wait(ms(10_000), new AbortController().signal))).toBe("fulfilled"); }); - it("holds buffered playback to the floor ahead of the playhead", () => { + // One stretch window past the floor, so the ring can always stage a full window to catch up with. + it("holds buffered playback to the floor and a stretch window ahead of the playhead", () => { const gate = new Backpressure(1000, true, Milli(200)); gate.observe(ms(0), false); const wait = gate.wait(ms(1000), new AbortController().signal); - gate.observe(ms(799), false); + gate.observe(ms(769), false); expect(Bun.peek.status(wait)).toBe("pending"); - gate.observe(ms(800), false); + gate.observe(ms(770), false); expect(Bun.peek.status(wait)).toBe("fulfilled"); }); @@ -396,7 +398,7 @@ describe("Backpressure", () => { expect(Bun.peek.status(wait)).toBe("pending"); gate.decoded(ms(1020), ms(20)); expect(Bun.peek.status(wait)).toBe("fulfilled"); - expect(Bun.peek.status(gate.wait(ms(1520), signal))).toBe("pending"); + expect(Bun.peek.status(gate.wait(ms(1560), signal))).toBe("pending"); }); // A replacement rendition keeps the ring, and the decoder it replaced never answers its frames. @@ -553,3 +555,59 @@ describe("AudioBuffer past its decoded floor", () => { } }); }); + +// Each state report reads the shared clock alongside the context's time, so the worklet maps the clock +// onto its own however late the report or the reply lands. A live ring steers on its own depth. +it("reads the shared clock with the context's time on each current report of a buffered ring", () => { + for (const buffered of [true, false]) { + const sent: Message[] = []; + const port = Object.assign(new EventTarget(), { + postMessage: (msg: Message) => sent.push(msg), + start() {}, + }); + const context = { currentTime: 12.5 }; + let clock: Milli | undefined = Milli(800); + const buffer = new AudioBuffer({ port, context } as unknown as AudioWorkletNode, { + channels: 1, + rate: 1000, + delay: Milli(100), + buffered, + clock: { now: () => clock }, + }); + const state = (epoch: number) => { + const data: State = { + type: "state", + timestamp: ms(700), + stalled: false, + anchored: true, + underruns: 0, + skips: 0, + trimmed: Milli.zero, + accelerated: Milli.zero, + silent: false, + epoch, + }; + port.dispatchEvent(new MessageEvent("message", { data })); + }; + try { + state(0); + clock = undefined; + context.currentTime = 13; + state(0); + // A report from before a reset belongs to the previous timeline. + buffer.reset(); + state(0); + const clocks = sent.filter((msg) => msg.type === "clock"); + expect(clocks).toEqual( + buffered + ? [ + { type: "clock", timestamp: ms(800), time: 12.5 }, + { type: "clock", time: 13 }, + ] + : [], + ); + } finally { + buffer.close(); + } + } +}); diff --git a/js/watch/src/audio/buffer.ts b/js/watch/src/audio/buffer.ts index 9c0b892ed40..688ae60f091 100644 --- a/js/watch/src/audio/buffer.ts +++ b/js/watch/src/audio/buffer.ts @@ -1,7 +1,8 @@ import { Micro, Milli, type Second } from "@moq/clock"; import { Effect, type Getter, Signal } from "@moq/signals"; import { ringHold, ringSamples } from "./latency"; -import type { Data, End, Init, Latency, Playing, Reset, State, Truncate } from "./render"; +import type { Clock, Data, End, Init, Latency, Playing, Reset, State, Truncate } from "./render"; +import { WINDOW } from "./stretch"; /** * Decides when the decode loop may decode a frame, which is how the target delay is split between @@ -17,7 +18,9 @@ import type { Data, End, Init, Latency, Playing, Reset, State, Truncate } from " * plays through and cannot shed what overflows it; * - while playing, once the playhead is within the floor of where its output lands, keeping the * ring on its floor. Live, that is where the output ends: the ring's time stretch measures its - * depth right after each write, and reads a tail past the floor as audio behind the clock; + * depth right after each write, and reads a tail past the floor as audio behind the clock. + * Buffered, it is where the output starts, plus one stretch window, so the ring can always + * stage a full window to catch up to the clock; * - while playing live, also once the media queued behind it covers the hold, so media past the * target reaches the ring, whose skip band catches up to the live edge. Buffered mode plays * through instead. @@ -235,7 +238,9 @@ export class Backpressure { #due(timestamp: Micro): boolean { const playhead = this.#playhead; if (!this.#stalled && playhead !== undefined) { - const tail = this.#buffered ? 0 : this.#frame; + // Buffered, a frame is released a stretch window early, so the ring always holds a full window + // to accelerate back onto the clock, even on a floor shorter than one. + const tail = this.#buffered ? -Micro.fromMilli(Milli(WINDOW)) : this.#frame; if (timestamp + tail - this.#offset - this.#headroom <= playhead) return true; return !this.#buffered && this.#covered(timestamp); } @@ -311,11 +316,13 @@ export class AudioBuffer { delay: Milli; /** Anchor to the first frame and play through, holding the lookahead upstream. */ buffered: boolean; + /** The shared clock a buffered ring plays faster to catch back up to whenever it trails. */ + clock?: { now(): Milli | undefined }; /** Hold consumption until the shared startup floor is ready. */ playing?: Getter; }, ) { - const { channels, rate, buffered, playing } = props; + const { channels, rate, buffered, playing, clock } = props; this.#playing = props.playing; this.#worklet = worklet; this.channels = channels; @@ -351,6 +358,16 @@ export class AudioBuffer { this.#silent.set(data.silent); if (data.epoch === this.#epoch) { this.#backpressure.observe(data.anchored ? data.timestamp : undefined, data.stalled); + // Read the clock and the context's time together, so a late message misplaces neither. + if (buffered && clock) { + const now = clock.now(); + const msg: Clock = { + type: "clock", + ...(now !== undefined && { timestamp: Milli.toMicro(now) }), + time: worklet.context.currentTime, + }; + worklet.port.postMessage(msg); + } } } }); @@ -378,7 +395,7 @@ export class AudioBuffer { /** * Insert audio samples at the given timestamp, transferring their buffers. A write may land out of - * order, but samples behind what live playback already staged, up to 30 ms past the playhead, are dropped. + * order, but samples behind what playback already staged, up to 30 ms past the playhead, are dropped. */ insert(timestamp: Micro, data: Float32Array[]): void { const end = (timestamp + Micro.fromSecond(((data[0]?.length ?? 0) / this.rate) as Second)) as Micro; diff --git a/js/watch/src/audio/decoder.test.ts b/js/watch/src/audio/decoder.test.ts index 7d88a18d2c7..de28e022f1a 100644 --- a/js/watch/src/audio/decoder.test.ts +++ b/js/watch/src/audio/decoder.test.ts @@ -63,6 +63,8 @@ let worklets: FakeWorklet[] = []; // every quantum read reports the ring's state back, the way `render-worklet.ts` does. class FakeWorklet { ring?: AudioRingBuffer; + // Read with each clock message; this fake renders no real time, so it applies no clock. + readonly context = { currentTime: 0 }; #epoch = 0; #playing = true; readonly port = Object.assign(new EventTarget(), { diff --git a/js/watch/src/audio/decoder.ts b/js/watch/src/audio/decoder.ts index 58c9e26d025..7cd7187a1cb 100644 --- a/js/watch/src/audio/decoder.ts +++ b/js/watch/src/audio/decoder.ts @@ -402,6 +402,7 @@ export class Decoder { rate: sampleRate, delay: this.sync.out.delay.peek(), buffered, + clock: this.sync, playing: this.sync.playing, }); effect.cleanup(() => ring.close()); diff --git a/js/watch/src/audio/render-worklet.test.ts b/js/watch/src/audio/render-worklet.test.ts index 658172e8dad..7c806448145 100644 --- a/js/watch/src/audio/render-worklet.test.ts +++ b/js/watch/src/audio/render-worklet.test.ts @@ -228,3 +228,44 @@ it("clears comfort noise when the timeline resets", () => { send({ type: "reset", epoch: 1 }); expect([...read(20)]).toEqual(Array(20).fill(0)); }); + +// The main thread reads the shared clock once per state report, so the worklet carries it forward on +// its own `currentTime` between readings, until a reset ends the timeline it was read on or a delay +// change moves the clock. +it("carries the shared clock forward on its own time for a buffered ring to catch up to", () => { + const render = new Render(); + const send = (data: Message) => render.port.onmessage?.({ data } as MessageEvent); + const rate = 48_000; + const play = (seconds: number) => { + for (let i = 0; i < (seconds * rate) / 128; i++) { + expect(render.process([], [[new Float32Array(128)]], {})).toBe(true); + time += 128 / rate; + } + return render.sent.at(-1); + }; + let time = 10; + const previous = Object.getOwnPropertyDescriptor(globalThis, "currentTime"); + Object.defineProperty(globalThis, "currentTime", { configurable: true, get: () => time }); + try { + send({ type: "init", channels: 1, rate, latency: Milli(100), buffered: true }); + send({ type: "data", timestamp: Micro(1_000_000), data: [new Float32Array(rate * 0.9).fill(0.5)] }); + // Half a second before this first quantum, the clock read 700 ms: it now stands 1.2 s, 200 ms past + // the playhead at 1 s. + send({ type: "clock", timestamp: Micro(700_000), time: 9.5 }); + const caught = play(0.5); + expect(caught?.accelerated).toBeGreaterThan(200 - 2.5); + expect(caught?.accelerated).toBeLessThanOrEqual(200); + + // A delay change moves the clock, so the reading from before it no longer counts. + send({ type: "clock", timestamp: Micro(2_000_000), time }); + send({ type: "latency", latency: Milli(100) }); + expect(play(0.1)?.accelerated).toBe(caught?.accelerated); + + send({ type: "reset", epoch: 1 }); + send({ type: "data", timestamp: Micro(1_000_000), data: [new Float32Array(rate * 0.9).fill(0.5)] }); + expect(play(0.5)?.accelerated).toBe(caught?.accelerated); + } finally { + if (previous) Object.defineProperty(globalThis, "currentTime", previous); + else Reflect.deleteProperty(globalThis, "currentTime"); + } +}); diff --git a/js/watch/src/audio/render-worklet.ts b/js/watch/src/audio/render-worklet.ts index ffcb5421f10..2905d9e6f4d 100644 --- a/js/watch/src/audio/render-worklet.ts +++ b/js/watch/src/audio/render-worklet.ts @@ -1,5 +1,6 @@ +import type { Micro } from "@moq/clock"; import { ComfortNoise } from "./comfort"; -import type { Message, State } from "./render"; +import type { Clock, Message, State } from "./render"; import { AudioRingBuffer } from "./ring-buffer"; // Samples to fade over when playback stops or resumes short of a full quantum. A shortfall @@ -16,6 +17,8 @@ class Render extends AudioWorkletProcessor { #playing = true; // Whether the previous quantum ended short, so the next one fades back in. #short = false; + // The shared clock's latest reading, which a buffered ring converges on. + #clock?: Clock; constructor() { super(); @@ -28,10 +31,13 @@ class Render extends AudioWorkletProcessor { this.#playing = msg.playing ?? true; this.#underflow = 0; this.#short = false; + this.#clock = undefined; } else if (msg.type === "data") { this.#ring?.write(msg.timestamp, msg.data); } else if (msg.type === "latency") { this.#ring?.setLatency(msg.latency); + // The clock moves with the delay, so a reading from before it would pull audio ahead. + this.#clock = undefined; if (msg.park !== undefined) { this.#ring?.park(); this.#epoch = msg.park; @@ -48,8 +54,12 @@ class Render extends AudioWorkletProcessor { } this.#epoch = msg.epoch; this.#playing = msg.playing ?? true; + // A reset ends the timeline the clock was read on. + this.#clock = undefined; } else if (msg.type === "playing" && msg.epoch === this.#epoch) { this.#playing = msg.playing; + } else if (msg.type === "clock") { + this.#clock = msg; } }; } @@ -57,7 +67,12 @@ class Render extends AudioWorkletProcessor { process(_inputs: Float32Array[][], outputs: Float32Array[][], _parameters: Record) { const output = outputs[0]; const ring = this.#ring; - const samplesRead = this.#playing ? (ring?.read(output) ?? 0) : 0; + const clock = this.#clock; + const shared = + clock?.timestamp !== undefined + ? ((clock.timestamp + (currentTime - clock.time) * 1_000_000) as Micro) + : undefined; + const samplesRead = this.#playing ? (ring?.read(output, shared) ?? 0) : 0; // A declared silence between two buffered utterances plays as comfort noise, faded like any other. const quiet = this.#playing ? ring?.quiet : undefined; if (quiet) { diff --git a/js/watch/src/audio/render.ts b/js/watch/src/audio/render.ts index 7760d84bbab..05f733758ea 100644 --- a/js/watch/src/audio/render.ts +++ b/js/watch/src/audio/render.ts @@ -1,7 +1,7 @@ import type { Micro, Milli } from "@moq/clock"; /** Everything the main thread sends the render worklet over its port. */ -export type Message = Init | Data | Latency | Reset | Truncate | Playing | End; +export type Message = Init | Data | Latency | Reset | Truncate | Playing | End | Clock; /** Create the worklet's ring. */ export interface Init { @@ -44,6 +44,18 @@ export interface Data { timestamp: Micro; } +/** + * Where the shared clock stands, for a buffered ring to time-stretch back onto. The clock runs at real + * time, so one reading maps it onto the worklet's own time until the next. + */ +export interface Clock { + type: "clock"; + // The clock's media time at context time `time`, unset while the clock has none. + timestamp?: Micro; + // The AudioContext's `currentTime` when the clock was read, in seconds. + time: number; +} + /** Latency update: the floor held as decoded PCM. */ export interface Latency { type: "latency"; diff --git a/js/watch/src/audio/replay.test.ts b/js/watch/src/audio/replay.test.ts index 6db5da84f7e..0ab140df931 100644 --- a/js/watch/src/audio/replay.test.ts +++ b/js/watch/src/audio/replay.test.ts @@ -2,6 +2,7 @@ import { afterEach, beforeEach, describe, expect, it, type Mock, spyOn } from "b import { Milli } from "@moq/clock"; import type * as Catalog from "@moq/hang/catalog"; import { type Arrival, type Options, QUANTUM, type Quantum, replay } from "./replay"; +import { WINDOW } from "./stretch"; const RATE = 48000; const CONFIG = { codec: "opus", sampleRate: RATE, numberOfChannels: 1 } as Catalog.AudioConfig; @@ -335,3 +336,107 @@ it("plays an advertised flush span past the decoded floor at its delay without a expect(lag).toBeGreaterThanOrEqual(5020); expect(lag).toBeLessThan(5040); }); + +// A buffered player keeps no depth to steer on, so it measures how far its playhead trails the shared +// clock and plays faster until back on it. It never skips: every frame plays, and the container keeps +// every group, since audio on the clock stays within the max delay of the newest arrival. +describe("buffered replay against the shared clock", () => { + let warn: Mock; + beforeEach(() => { + warn = spyOn(console, "warn").mockImplementation(() => {}); + }); + afterEach(() => warn.mockRestore()); + + /** Plays `trace`, returning how far audio trails the clock once settled, and what was dropped. */ + async function play(trace: Arrival[], options: Options, settled: (at: number) => boolean) { + const lags: number[] = []; + let heard = 0; + let last: Quantum | undefined; + for await (const quantum of replay(trace, options)) { + heard += quantum.output.filter((v) => v !== 0).length; + if (settled(quantum.at) && !quantum.stalled && quantum.clock !== undefined) { + lags.push(quantum.clock - quantum.timestamp); + } + last = quantum; + } + if (!last) throw new Error("nothing played"); + return { + min: Math.min(...lags), + max: Math.max(...lags), + skips: last.skips, + trimmed: last.trimmed, + accelerated: last.accelerated, + // Frames the consumer passed over rather than delivered. + lost: trace.length - last.arrivals.filter((a) => a.kind === "frame").length, + heard: (heard * 1000) / RATE, + }; + } + + it("catches up after each stall", async () => { + // Every ten seconds, 400 ms of frames are held and land at once. + const trace = paced(60, (i) => 30 + (i % 500 >= 250 && i % 500 < 270 ? (270 - (i % 500)) * 20 : 0)).sort( + (a, b) => a.at - b.at, + ); + const options = { + rate: RATE, + config: CONFIG, + delay: Milli(100), + lookahead: Milli(500), + duration: observed(60), + }; + // Two seconds after each stall ends. Without the stretch, audio trailed by 317 ms for good and the + // consumer dropped 12 frames. + const result = await play(trace, options, (at) => at >= 2000 && at % 10_000 >= 7500); + expect(result.min).toBeGreaterThanOrEqual(0); + expect(result.max).toBeLessThan(2.5); + expect(result).toMatchObject({ skips: 0, trimmed: 0, lost: 0 }); + expect(warn).not.toHaveBeenCalled(); + }); + + it("catches up after each stall on a floor shorter than a stretch window", async () => { + const trace = paced(60, (i) => 30 + (i % 500 >= 250 && i % 500 < 270 ? (270 - (i % 500)) * 20 : 0)).sort( + (a, b) => a.at - b.at, + ); + const options = { rate: RATE, config: CONFIG, delay: Milli(10), lookahead: Milli(500), duration: observed(60) }; + // Without a full window released ahead of the floor, audio trailed by 397 ms for good. A window + // stretches only once 30 ms past the playhead has arrived, so audio rests within that less the delay. + const result = await play(trace, options, (at) => at >= 2000 && at % 10_000 >= 7500); + expect(result.min).toBeGreaterThanOrEqual(0); + expect(result.max).toBeLessThan(WINDOW - 10); + expect(result).toMatchObject({ skips: 0, trimmed: 0, lost: 0 }); + expect(warn).not.toHaveBeenCalled(); + }); + + it("follows a publisher clock running 1% fast at auto", async () => { + const trace = paced(60, (i) => 30 - (i * 20) / 101); + const options = { + rate: RATE, + config: CONFIG, + delay: "auto" as const, + lookahead: Milli(500), + duration: observed(60), + }; + // Without the stretch, audio trailed by 60 to 290 ms and the consumer dropped 273 frames as slow + // groups. + const result = await play(trace, options, (at) => at >= 5000); + expect(result.min).toBeGreaterThanOrEqual(0); + expect(result.max).toBeLessThan(23); + expect(result).toMatchObject({ skips: 0, trimmed: 0, lost: 0 }); + expect(warn).not.toHaveBeenCalled(); + }); + + it("plays every word of an utterance that stalls mid-burst, catching up after", async () => { + // Twenty seconds of speech generated four times faster than real time, held for 3.5 s after its + // first two seconds, so playback runs dry and resumes about two seconds behind the clock. + const trace = paced(20, (i) => (i < 100 ? 30 : 3500) - i * 15); + const options = { rate: RATE, config: CONFIG, delay: Milli(100), lookahead: Milli(30_000), duration: 22_000 }; + // Without the stretch, audio trailed by 1969 ms to the end. + const result = await play(trace, options, (at) => at >= 15_000); + expect(result.min).toBeGreaterThanOrEqual(0); + expect(result.max).toBeLessThanOrEqual(4); + expect(result).toMatchObject({ skips: 0, trimmed: 0, lost: 0 }); + // Every frame played, less only what the stretch shortened. + expect(result.heard + result.accelerated).toBe(20_000); + expect(warn).not.toHaveBeenCalled(); + }); +}); diff --git a/js/watch/src/audio/replay.ts b/js/watch/src/audio/replay.ts index 21b2d60602d..b250c96905d 100644 --- a/js/watch/src/audio/replay.ts +++ b/js/watch/src/audio/replay.ts @@ -25,7 +25,7 @@ import { type Arrival as Diagnostic, gap } from "../arrivals"; import { type Delay, Sync } from "../sync"; import { Backpressure } from "./buffer"; import { frameDuration } from "./config"; -import { target } from "./latency"; +import { ringHold, target } from "./latency"; import { AudioRingBuffer } from "./ring-buffer"; /** An AudioWorklet render quantum, in frames. */ @@ -60,6 +60,8 @@ export interface Options { rate: number; /** The configured delay, as the element takes it. "instant" plays no audio, so it has no replay. */ delay: Exclude; + /** The configured lookahead: nonzero plays in buffered mode, as the decoder does. */ + lookahead?: Milli; /** The rendition as the catalog described it: its advertised jitter, delay, and codec. */ config: Catalog.AudioConfig; /** @@ -139,7 +141,9 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato ends.set(arrival.group, i); }); - const sync = new Sync({ delay: options.delay }); + const lookahead = options.lookahead ?? Milli.zero; + const buffered = lookahead > 0; + const sync = new Sync({ delay: options.delay, lookahead }); const track = new Track.Producer("audio"); const consumer = new Container.Consumer(track.subscribe({ delay: { max: WIRE_MAX_DELAY } }), { format: new Container.Legacy.Format("audio"), @@ -157,23 +161,37 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato delay: config.delay !== undefined ? Milli(config.delay) : undefined, }), ); - const audio = sync.track({ target: playout }); + // Buffered startup holds playback until the decoded span, plus whatever of the hold is queued + // encoded past it, covers the delay, the way the decoder reports its startup span. + const decoded = new Signal<{ start: Milli; end: Milli } | null>(null); + const newest = new Signal(undefined); + const startup = buffered + ? signals.computed((effect) => { + const span = effect.get(decoded); + if (!span) return null; + const queued = Math.max(0, Milli.fromMicro(effect.get(newest) ?? (0 as Micro)) - span.end); + const hold = Milli.fromMicro(ringHold(effect.get(sync.out.delay))); + return { start: span.start, end: Milli(span.end + Math.min(queued, hold)) }; + }) + : undefined; + const audio = sync.track({ target: playout, startup }); signals.cleanup(() => audio.close()); await settle(); let delay = sync.out.delay.peek(); // The decode gate, and the worklet's ring on the floor it sets, the way `AudioBuffer` drives both. - const gate = new Backpressure(rate, false, delay); - const ring = new AudioRingBuffer({ rate, channels: 1, latency: millis(rate, gate.floor) }); + const gate = new Backpressure(rate, buffered, delay); + const ring = new AudioRingBuffer({ rate, channels: 1, latency: millis(rate, gate.floor), buffered }); const observe = () => gate.observe(ring.anchored ? ring.timestamp : undefined, ring.stalled); // Arrived but not yet decoded media, read or still in the consumer, is what covers the part of the // target held encoded, and what resumes a parked ring. const read = new Signal(undefined); signals.run((effect) => { - let newest = effect.get(read); - for (const { end } of effect.get(consumer.buffered)) newest = Math.max(newest ?? 0, micros(end)) as Micro; - if (gate.queued(newest)) ring.setLatency(millis(rate, gate.floor)); + let queued = effect.get(read); + for (const { end } of effect.get(consumer.buffered)) queued = Math.max(queued ?? 0, micros(end)) as Micro; + newest.set(queued); + if (gate.queued(queued)) ring.setLatency(millis(rate, gate.floor)); }); // The decoder's read loop, with the codec taken out. Traces record no declared ends, so a gap is @@ -206,6 +224,7 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato // The decoder reports each output to the gate the way `AudioBuffer` and `Decoder.#emit` do. const end = (timestamp + micros((pcm.length / rate) * 1000)) as Micro; gate.inserted(timestamp, end); + decoded.update((span) => ({ start: span?.start ?? media, end: Milli.fromMicro(end) })); gate.decoded(end, end); } })().catch((error: unknown) => { @@ -276,8 +295,16 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato if (park) ring.park(); } + // The worklet reads nothing until buffered startup releases it, then the shared clock as of the + // quantum's start, mapped onto its own time. + const start = sync.now(); output.fill(0); - ring.read([output]); + if (sync.playing.peek()) { + ring.read( + [output], + buffered && start !== undefined ? Milli.toMicro(Milli(Math.max(0, start - step))) : undefined, + ); + } observe(); yield { arrivals, diff --git a/js/watch/src/audio/ring-buffer.test.ts b/js/watch/src/audio/ring-buffer.test.ts index 07ca8a3f2bc..c53c0c2323e 100644 --- a/js/watch/src/audio/ring-buffer.test.ts +++ b/js/watch/src/audio/ring-buffer.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from "bun:test"; -import { Micro, Milli } from "@moq/clock"; +import { Micro, Milli, type Second } from "@moq/clock"; import { AudioRingBuffer } from "./ring-buffer"; function read(buffer: AudioRingBuffer, samples: number, channelCount = 2): Float32Array[] { @@ -957,6 +957,66 @@ describe("time stretch", () => { expect(buffer.length).toBeLessThan(frames(480)); }); + describe("buffered on the shared clock", () => { + const buffered = (latency: number) => + new AudioRingBuffer({ rate: RATE, channels: 1, latency: latency as Milli, buffered: true }); + + // Plays `seconds` of quanta with the clock `behind` ms ahead of where the playhead started, + // topping the ring up to its 100 ms floor the way the paced decode loop does. Returns how far + // the playhead trails the clock after each quantum, in frames. + function follow(buffer: AudioRingBuffer, seconds: number, behind: number | undefined, end?: number) { + const start = frames(1000); + let written = start; + const lags: number[] = []; + for (let rendered = 0; rendered < frames(seconds * 1000); rendered += QUANTUM) { + const playhead = buffer.anchored + ? Math.round((Milli.fromMicro(buffer.timestamp) * RATE) / 1000) + : start; + while (written - playhead < frames(100)) { + write(buffer, ((written * 1000) / RATE) as Milli, frames(20), { value: 0.5 }); + written += frames(20); + } + if (end !== undefined && rendered === 0) buffer.end(Micro.fromMilli(end as Milli)); + const clock = behind === undefined ? undefined : start + frames(behind) + rendered; + const output = [new Float32Array(QUANTUM)]; + expect( + buffer.read(output, clock === undefined ? undefined : Micro.fromSecond((clock / RATE) as Second)), + ).toBe(QUANTUM); + if (clock !== undefined) + lags.push(Math.round(clock + QUANTUM - (Milli.fromMicro(buffer.timestamp) * RATE) / 1000)); + } + return lags; + } + + it("accelerates a playhead behind the clock back onto it, never past it, dropping nothing", () => { + const buffer = buffered(100); + const lags = follow(buffer, 3, 300); + // Fast accelerate halves each window until the lag is under 20 ms, and the ordinary one then + // removes a period only while one fits, so the playhead rests within a period behind the clock. + for (const lag of lags) expect(lag).toBeGreaterThanOrEqual(0); + for (const lag of lags.slice(-50)) expect(lag).toBeLessThan(PERIOD); + expect(buffer.accelerated).toBeGreaterThan(300 - 2.5); + expect(buffer.skips).toBe(0); + expect(buffer.underruns).toBe(0); + }); + + it("plays at the encoded pace on or ahead of the clock, or without one", () => { + for (const behind of [0, -200, undefined]) { + const buffer = buffered(100); + follow(buffer, 1, behind); + expect(buffer.accelerated).toBe(Milli.zero); + expect(Milli.fromMicro(buffer.timestamp)).toBe(2000 as Milli); + } + }); + + // Live, a tail's depth says nothing about the clock; buffered, the clock is measured directly. + it("catches up a tail before a declared end", () => { + const buffer = buffered(100); + follow(buffer, 1, 100, 6000); + expect(buffer.accelerated).toBeGreaterThan(100 - 2.5); + }); + }); + it("drops the staged window with the audio a re-anchor skips", () => { const buffer = live(100); write(buffer, 0 as Milli, frames(100), { value: 0.25 }); diff --git a/js/watch/src/audio/ring-buffer.ts b/js/watch/src/audio/ring-buffer.ts index 5fde0e94fab..855d9654a31 100644 --- a/js/watch/src/audio/ring-buffer.ts +++ b/js/watch/src/audio/ring-buffer.ts @@ -45,8 +45,9 @@ export class AudioRingBuffer { // decode would otherwise widen the band for the life of the ring. #chunk = 0; - // Live playback converges on the floor by accelerating each window of received PCM on its way out. - // The read index is past the staged window, which spans `#window` samples of the ring. + // Live playback converges on the floor, and buffered playback on the shared clock, by accelerating + // each window of received PCM on its way out. The read index is past the staged window, which spans + // `#window` samples of the ring. readonly #stretch: Stretch; #window = 0; // The ring's length right after the latest write, less what the stretch has removed since. That @@ -57,7 +58,8 @@ export class AudioRingBuffer { // Whether the staged window lies in a declared silence, which plays unstretched as comfort noise. #windowQuiet = false; - // Buffered mode: play through everything buffered without skipping ahead. + // Buffered mode: play through everything buffered without skipping ahead, catching up to the + // shared clock by playing faster instead. readonly #buffered: boolean; // Un-stall threshold in samples (how much to buffer before playback starts). #latencySamples: number; @@ -427,7 +429,13 @@ export class AudioRingBuffer { this.#peak = 0; } - read(output: Float32Array[]): number { + /** + * Fill `output` from the playhead, returning how many frames it supplied. A buffered ring given + * `clock`, where the shared clock stands at the first output frame, accelerates whenever its + * playhead trails it, until it is back on the clock. It never runs ahead: a live ring steers on its + * own depth instead, and a buffered one without a clock plays at the encoded pace. + */ + read(output: Float32Array[], clock?: Micro): number { if (output.length !== this.channels) throw new Error("wrong number of channels"); for (const dst of output) if (dst.length !== output[0].length) throw new Error("mismatching number of samples"); this.#quiet = undefined; @@ -442,9 +450,13 @@ export class AudioRingBuffer { } const stretch = this.#stretch; + const shared = + this.#buffered && clock !== undefined ? Math.round(Second.fromMicro(clock) * this.rate) : undefined; let samples = 0; while (samples < output[0].length) { - if (stretch.available === 0 && !this.#stage(output[0].length - samples)) break; + // A window stages only once the previous one has played out, so the playhead is the read index. + const behind = shared === undefined ? undefined : shared + samples - this.#readIndex; + if (stretch.available === 0 && !this.#stage(output[0].length - samples, behind)) break; const from = samples; samples += stretch.read(output, samples); // A quantum overlaps at most one declared silence unless an utterance is shorter than it, @@ -469,22 +481,24 @@ export class AudioRingBuffer { return samples; } - // Move the next window off the ring into the stretch, which accelerates a live ring onto its floor. - // A window stops at the edge of a declared silence, which plays unstretched, so the stretch never - // moves where one starts or ends. Whatever cannot stretch stages only the `need` this quantum - // reads, so a write still lands anywhere past the playhead. Returns false once the ring is empty. - #stage(need: number): boolean { + // Move the next window off the ring into the stretch, which accelerates a live ring onto its floor, + // and a buffered ring `behind` the shared clock by that many frames back onto it. A window stops at + // the edge of a declared silence, which plays unstretched, so the stretch never moves where one + // starts or ends. Whatever cannot stretch stages only the `need` this quantum reads, so a write + // still lands anywhere past the playhead. Returns false once the ring is empty. + #stage(need: number, behind?: number): boolean { const buffered = this.#writeIndex - this.#readIndex; if (buffered <= 0) return false; this.#prune(); const pause = this.#pauses[0]; const quiet = pause !== undefined && pause.start <= this.#readIndex; - // Audio before a declared end is a finite tail: it drains as received, since the next utterance - // re-anchors the ring anyway. + // Live, audio before a declared end is a finite tail: it drains as received, since the next + // utterance re-anchors the ring anyway, and its depth says nothing about the clock. A buffered + // ring measures the clock itself, so a tail behind it catches up like any other audio. const end = this.#previous ?? this.#end; const ending = end !== undefined && this.#readIndex < end; - const enabled = !quiet && !ending && !this.#buffered; + const enabled = !quiet && (this.#buffered ? behind !== undefined && behind > 0 : !ending); let frames = Math.min(buffered, this.#stretch.window, enabled ? buffered : need); if (pause) frames = Math.min(frames, (quiet ? pause.end : pause.start) - this.#readIndex); @@ -494,7 +508,7 @@ export class AudioRingBuffer { for (let i = 0; i < frames; i++) dst[i] = src[(this.#readIndex + i) % src.length]; } const stretch = this.#stretch; - stretch.process(frames, this.#peak - this.#latencySamples, enabled); + stretch.process(frames, this.#buffered ? (behind ?? 0) : this.#peak - this.#latencySamples, enabled); this.#peak -= frames - stretch.length; this.#accelerated += Math.max(0, frames - stretch.length); this.#readIndex += frames; diff --git a/js/watch/src/audio/stretch.ts b/js/watch/src/audio/stretch.ts index 31ad99d53f7..8d3fc9da8c1 100644 --- a/js/watch/src/audio/stretch.ts +++ b/js/watch/src/audio/stretch.ts @@ -9,6 +9,9 @@ * @license BSD-3-Clause, with WebRTC's patent grant: see WEBRTC-LICENSE and WEBRTC-PATENTS in this package. */ +/** A full processing window of received PCM, in milliseconds. */ +export const WINDOW = 30; + /** What a window boundary decides, named as the shared corpus names it. */ export type Decision = "Hold" | "Accelerate" | "Fast"; @@ -55,7 +58,7 @@ export class Stretch { constructor(rate: number, channels: number) { this.#rate = rate; - this.window = Math.max(1, Math.floor((rate * 30) / 1000)); + this.window = Math.max(1, Math.floor((rate * WINDOW) / 1000)); this.#middle = Math.floor((rate * 15) / 1000); this.#band = Math.floor((rate * 20) / 1000); this.#rest = Math.floor((rate * 50) / 1000); diff --git a/js/watch/src/element.ts b/js/watch/src/element.ts index b300d237663..f9b2a9f7b45 100644 --- a/js/watch/src/element.ts +++ b/js/watch/src/element.ts @@ -606,8 +606,10 @@ export default class MoqWatch extends HTMLElement { * * Zero (the default) minimizes latency. A larger value enables buffered playback: build up * future-dated frames (e.g. TTS written faster than real-time) and only skip ahead - * once they would sit further than `delay + lookahead` past the playhead. Call `reset()` at each - * utterance boundary. Not a jitter buffer: live streams leave it at zero and tune `delay`. + * once they would sit further than `delay + lookahead` past the playhead. Audio behind the + * clock, such as after a stall, catches up by playing slightly faster rather than skipping. + * Call `reset()` at each utterance boundary. Not a jitter buffer: live streams leave it at zero + * and tune `delay`. */ get lookahead(): Milli { return this.controls.lookahead.peek(); diff --git a/js/watch/src/sync.ts b/js/watch/src/sync.ts index 9529a364f46..61c46549228 100644 --- a/js/watch/src/sync.ts +++ b/js/watch/src/sync.ts @@ -25,8 +25,9 @@ export type SyncInput = { * * Zero minimizes latency, the live default: playback re-anchors as soon as anything arrives * early. A larger value lets faster-than-real-time frames (a TTS response with future - * timestamps) build up instead of being skipped. Always finite, so worst case the audio ring - * drops its oldest samples rather than exhausting memory. With a nonzero lookahead, the first video + * timestamps) build up instead of being skipped. Buffered audio behind the clock, such as after a + * stall, catches up by playing slightly faster rather than skipping. Always finite, so worst case + * the audio ring drops its oldest samples rather than exhausting memory. With a nonzero lookahead, the first video * frame is a poster until every enabled media track holds the resolved delay; `lookahead` is * the most held ahead, not the amount required to start. */ diff --git a/quest/m1/audio-jitter-target/README.md b/quest/m1/audio-jitter-target/README.md index b025813de04..01f77935ee3 100644 --- a/quest/m1/audio-jitter-target/README.md +++ b/quest/m1/audio-jitter-target/README.md @@ -59,7 +59,6 @@ native playback against its budgets. - [Browser ceilings](/quest/m1/audio-jitter-target/browser-ceilings.md) - the chromium rows' ceilings re-measured from seven Nightlies with the browser's time stretch, budgeting `player_skips_per_min` - [Native stretch](/quest/m1/audio-jitter-target/native-stretch.md) - `moq play` copies NetEq's accelerate and preemptive expand, sharing the browser's seams and vectors, so its native rows play with no audible glitch once the target settles -- [Buffered converge](/quest/m1/audio-jitter-target/buffered-converge.md) - a buffered `` time-stretches its audio back onto the shared clock after a stall or drift, so lip sync returns and no audio group is dropped - [Capped budgets](/quest/m1/audio-jitter-target/capped-budgets.md) - the AAC browser budgets are re-measured from nightlies after the TS importer groups each PES - [Native A/V lag](/quest/m1/audio-jitter-target/native-av-lag.md) - `moq play` reports how far audio trails its presentation clock, and native stops expanding into the gap between arrivals if it settles audio late - [Native rendition delay](/quest/m1/audio-jitter-target/native-rendition-delay.md) - `moq play` holds each subscribed rendition's catalog `delay` plus its target, as the browser does diff --git a/quest/m1/audio-jitter-target/buffered-converge.md b/quest/m1/audio-jitter-target/buffered-converge.md deleted file mode 100644 index b418840c46c..00000000000 --- a/quest/m1/audio-jitter-target/buffered-converge.md +++ /dev/null @@ -1,65 +0,0 @@ -# [M] Buffered audio converges on the shared clock - -## Goal - -A `` with a nonzero `lookahead` keeps lip sync in steady state: once -a stall, a rise of the `"auto"` delay, or publisher clock drift has left the -audio playhead behind the shared clock, the ring time-stretches back onto it. -It never skips, so a TTS utterance still plays every word. Today buffered -mode turns off the ring's catch-up (`playing = !this.#stalled && -!this.#buffered` in `js/watch/src/audio/ring-buffer.ts`), so any lag is -permanent. Audio trails video for the rest of the stream, and once the lag -passes `delay + lookahead` the container consumer drops whole audio groups -("skipping slow groups"), which is the cut-out reported in -[moq-dev/moq#8031](https://github.com/moq-dev/moq/issues/8031). - -Boundaries: video already follows the shared clock and is unchanged. The -clock is not driven by audio, and a slow audio track never holds video. Live -mode (`lookahead` of zero) keeps its own catch-up. Native playback has no -buffered mode. - -## Plan - -Planned 2026-11-06 (maintainer, /quest-import of -[moq-dev/moq#8031](https://github.com/moq-dev/moq/issues/8031)). A -simulation on main with the real `Sync`, container `Consumer`, `Backpressure` -and `AudioRingBuffer` (a copy of `js/watch/src/audio/replay.ts` with buffered -mode on) measured audio behind video: about -12 ms live versus +105 ms -buffered under `"auto"` with jittery then calm phases; 0 versus +300 ms after -400 ms stalls with `delay=100ms lookahead=500ms`; and a publisher clock 1% fast -climbing to 1.4 s of lag with 2926 slow-group skips. The reporter's other -claim, that a delay above 500 ms breaks lip sync, did not reproduce (300, 800 -and 1500 ms all held about -22 ms). - -Decisions: - -- Direction: ✅ converge buffered audio by time-stretching / won't fix, - reply and close / sharpen docs and warn only. Steady-state lip sync comes - from each track converging on the shared clock, and buffered mode broke - that even for TTS with video. -- Milestone: ✅ m1, in this epic / m3. - -Recommendations: - -- Reuse the ring's stretch (`js/watch/src/audio/stretch.ts`) rather - than a second mechanism: buffered mode measures the playhead's lag behind - the shared clock and accelerates until it is back within the floor. It - never stretches ahead of the clock, since future-dated frames are - buffered mode's purpose. -- Extend the replay harness with buffered rows (the stall and the drift - scenarios above), budgeting audio-behind-video and slow-group skips, and - keep a TTS row asserting no frame is dropped. -- Update the `lookahead` docs in `js/watch/src/element.ts`, - `js/watch/src/sync.ts` and `doc/lib/js/watch.md` to say that a buffered - player catches up after a stall by playing slightly faster. - -Public API: none expected. Wire: none. - -## Closes - -- [#8031](https://github.com/moq-dev/moq/issues/8031) - buffered playback loses lip sync and cuts audio out on a live stream - -## Related - -- [Browser ceilings](/quest/m1/audio-jitter-target/browser-ceilings.md) - measures on the nightly runner the ring's time stretch this reuses, which landed in [moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058) -- [Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md) - the other buffered-mode change to the audio path