Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions doc/concept/audio-jitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -621,8 +621,9 @@ with its accelerate alone. Neither generates replacement audio for a missing
packet, so a hole in the media plays as a hole. The estimator above is
unchanged. Both share the pitch search, admission, seams, and
[PCM corpus](audio-jitter/stretch/corpus.json) below; they differ only in
[what they decide on](#browser-decision). The browser stretches only live
playback: buffered playback plays every sample it holds, as it always has.
[what they decide on](#browser-decision). Buffered browser playback plays
every sample it holds, but accelerates the same way to catch back up to the
shared clock.

The source is WebRTC's
[`TimeStretch`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/time_stretch.cc),
Expand Down Expand Up @@ -686,6 +687,21 @@ accelerate after a refill catches up about as fast as that drop did. Arrivals th
outrun the target underrun instead: that is the cost of a target short of the
spread, and the estimator's to fix, not the stretch's.

Buffered playback holds its lookahead upstream and feeds the ring only up to
its target plus one 30 ms window, so it can always stage a full window to
stretch, and its depth says nothing about the clock. It decides on
`excess = clock - playhead` instead, the shared clock video plays from less the
playhead at the window's start, by the same three rules. A stall, a rise of the
delay, or a publisher clock running fast leaves audio behind the clock; it
plays faster until it is back within a pitch period, and never ahead of it,
since future-dated frames are what buffered playback holds. That holds for the
tail before a declared end too, since the clock, not the depth, measures it.
A delay under one window rests up to the window less the delay behind instead,
since a window stretches only once 30 ms past the playhead has arrived.
The main thread reads the clock with each of the worklet's state reports,
paired with the `AudioContext`'s `currentTime`, and the worklet carries it
forward on its own time until the next.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

### Pitch search and admission

Use channel zero as NetEq does. All channels share the selected seam and lag,
Expand Down
4 changes: 3 additions & 1 deletion doc/lib/js/watch.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,7 +245,9 @@ By default the player minimizes latency: it time-stretches received audio
that falls behind the delay back onto it, playing it slightly faster, and skips
ahead only when media piles up far past it. Content produced faster than real time, such as a TTS
response emitted in one burst with future timestamps, wants the opposite. Set
`lookahead` to how far ahead it may run and it plays through at the encoded pace:
`lookahead` to how far ahead it may run and it plays through at the encoded
pace. After a stall, or anything else that leaves audio behind video, it catches
up by playing slightly faster rather than skipping, so lip sync returns:

```html
<moq-watch url="..." name="bot/tts.hang" delay="100ms" lookahead="30s"></moq-watch>
Expand Down
74 changes: 66 additions & 8 deletions js/watch/src/audio/buffer.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -180,9 +180,10 @@ describe("AudioBuffer", () => {
const ahead = buffer.wait(120_000 as Micro, signal);
expect(Bun.peek.status(ahead)).toBe("pending");

// The floor lands and plays past the frame's headroom.
// The floor lands and plays past the frame's headroom, and a playing ring takes one stretch window more.
await state(60_000, false, 0);
expect(Bun.peek.status(buffer.wait(170_000 as Micro, signal))).toBe("pending");
expect(Bun.peek.status(buffer.wait(190_000 as Micro, signal))).toBe("fulfilled");
expect(Bun.peek.status(buffer.wait(200_000 as Micro, signal))).toBe("pending");
expect(Bun.peek.status(ahead)).toBe("fulfilled");
} finally {
close();
Expand Down Expand Up @@ -225,8 +226,8 @@ describe("AudioBuffer", () => {
// The frame past the gap un-stalls the ring, which plays the gap out up to it.
await state(10_000_000, false, 0);
expect(Bun.peek.status(next)).toBe("fulfilled");
expect(Bun.peek.status(buffer.wait(10_090_000 as Micro, signal))).toBe("fulfilled");
expect(Bun.peek.status(buffer.wait(10_110_000 as Micro, signal))).toBe("pending");
expect(Bun.peek.status(buffer.wait(10_130_000 as Micro, signal))).toBe("fulfilled");
expect(Bun.peek.status(buffer.wait(10_150_000 as Micro, signal))).toBe("pending");
} finally {
close();
}
Expand Down Expand Up @@ -277,13 +278,14 @@ describe("Backpressure", () => {
expect(Bun.peek.status(gate.wait(ms(10_000), new AbortController().signal))).toBe("fulfilled");
});

it("holds buffered playback to the floor ahead of the playhead", () => {
// One stretch window past the floor, so the ring can always stage a full window to catch up with.
it("holds buffered playback to the floor and a stretch window ahead of the playhead", () => {
const gate = new Backpressure(1000, true, Milli(200));
gate.observe(ms(0), false);
const wait = gate.wait(ms(1000), new AbortController().signal);
gate.observe(ms(799), false);
gate.observe(ms(769), false);
expect(Bun.peek.status(wait)).toBe("pending");
gate.observe(ms(800), false);
gate.observe(ms(770), false);
expect(Bun.peek.status(wait)).toBe("fulfilled");
});

Expand Down Expand Up @@ -396,7 +398,7 @@ describe("Backpressure", () => {
expect(Bun.peek.status(wait)).toBe("pending");
gate.decoded(ms(1020), ms(20));
expect(Bun.peek.status(wait)).toBe("fulfilled");
expect(Bun.peek.status(gate.wait(ms(1520), signal))).toBe("pending");
expect(Bun.peek.status(gate.wait(ms(1560), signal))).toBe("pending");
});

// A replacement rendition keeps the ring, and the decoder it replaced never answers its frames.
Expand Down Expand Up @@ -553,3 +555,59 @@ describe("AudioBuffer past its decoded floor", () => {
}
});
});

// Each state report reads the shared clock alongside the context's time, so the worklet maps the clock
// onto its own however late the report or the reply lands. A live ring steers on its own depth.
it("reads the shared clock with the context's time on each current report of a buffered ring", () => {
for (const buffered of [true, false]) {
const sent: Message[] = [];
const port = Object.assign(new EventTarget(), {
postMessage: (msg: Message) => sent.push(msg),
start() {},
});
const context = { currentTime: 12.5 };
let clock: Milli | undefined = Milli(800);
const buffer = new AudioBuffer({ port, context } as unknown as AudioWorkletNode, {
channels: 1,
rate: 1000,
delay: Milli(100),
buffered,
clock: { now: () => clock },
});
const state = (epoch: number) => {
const data: State = {
type: "state",
timestamp: ms(700),
stalled: false,
anchored: true,
underruns: 0,
skips: 0,
trimmed: Milli.zero,
accelerated: Milli.zero,
silent: false,
epoch,
};
port.dispatchEvent(new MessageEvent("message", { data }));
};
try {
state(0);
clock = undefined;
context.currentTime = 13;
state(0);
// A report from before a reset belongs to the previous timeline.
buffer.reset();
state(0);
const clocks = sent.filter((msg) => msg.type === "clock");
expect(clocks).toEqual(
buffered
? [
{ type: "clock", timestamp: ms(800), time: 12.5 },
{ type: "clock", time: 13 },
]
: [],
);
} finally {
buffer.close();
}
}
});
27 changes: 22 additions & 5 deletions js/watch/src/audio/buffer.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import { Micro, Milli, type Second } from "@moq/clock";
import { Effect, type Getter, Signal } from "@moq/signals";
import { ringHold, ringSamples } from "./latency";
import type { Data, End, Init, Latency, Playing, Reset, State, Truncate } from "./render";
import type { Clock, Data, End, Init, Latency, Playing, Reset, State, Truncate } from "./render";
import { WINDOW } from "./stretch";

/**
* Decides when the decode loop may decode a frame, which is how the target delay is split between
Expand All @@ -17,7 +18,9 @@ import type { Data, End, Init, Latency, Playing, Reset, State, Truncate } from "
* plays through and cannot shed what overflows it;
* - while playing, once the playhead is within the floor of where its output lands, keeping the
* ring on its floor. Live, that is where the output ends: the ring's time stretch measures its
* depth right after each write, and reads a tail past the floor as audio behind the clock;
* depth right after each write, and reads a tail past the floor as audio behind the clock.
* Buffered, it is where the output starts, plus one stretch window, so the ring can always
* stage a full window to catch up to the clock;
* - while playing live, also once the media queued behind it covers the hold, so media past the
* target reaches the ring, whose skip band catches up to the live edge. Buffered mode plays
* through instead.
Expand Down Expand Up @@ -235,7 +238,9 @@ export class Backpressure {
#due(timestamp: Micro): boolean {
const playhead = this.#playhead;
if (!this.#stalled && playhead !== undefined) {
const tail = this.#buffered ? 0 : this.#frame;
// Buffered, a frame is released a stretch window early, so the ring always holds a full window
// to accelerate back onto the clock, even on a floor shorter than one.
const tail = this.#buffered ? -Micro.fromMilli(Milli(WINDOW)) : this.#frame;
if (timestamp + tail - this.#offset - this.#headroom <= playhead) return true;
return !this.#buffered && this.#covered(timestamp);
}
Expand Down Expand Up @@ -311,11 +316,13 @@ export class AudioBuffer {
delay: Milli;
/** Anchor to the first frame and play through, holding the lookahead upstream. */
buffered: boolean;
/** The shared clock a buffered ring plays faster to catch back up to whenever it trails. */
clock?: { now(): Milli | undefined };
/** Hold consumption until the shared startup floor is ready. */
playing?: Getter<boolean>;
},
) {
const { channels, rate, buffered, playing } = props;
const { channels, rate, buffered, playing, clock } = props;
this.#playing = props.playing;
this.#worklet = worklet;
this.channels = channels;
Expand Down Expand Up @@ -351,6 +358,16 @@ export class AudioBuffer {
this.#silent.set(data.silent);
if (data.epoch === this.#epoch) {
this.#backpressure.observe(data.anchored ? data.timestamp : undefined, data.stalled);
// Read the clock and the context's time together, so a late message misplaces neither.
if (buffered && clock) {
const now = clock.now();
const msg: Clock = {
type: "clock",
...(now !== undefined && { timestamp: Milli.toMicro(now) }),
time: worklet.context.currentTime,
};
worklet.port.postMessage(msg);
}
}
}
});
Expand Down Expand Up @@ -378,7 +395,7 @@ export class AudioBuffer {

/**
* Insert audio samples at the given timestamp, transferring their buffers. A write may land out of
* order, but samples behind what live playback already staged, up to 30 ms past the playhead, are dropped.
* order, but samples behind what playback already staged, up to 30 ms past the playhead, are dropped.
*/
insert(timestamp: Micro, data: Float32Array[]): void {
const end = (timestamp + Micro.fromSecond(((data[0]?.length ?? 0) / this.rate) as Second)) as Micro;
Expand Down
2 changes: 2 additions & 0 deletions js/watch/src/audio/decoder.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ let worklets: FakeWorklet[] = [];
// every quantum read reports the ring's state back, the way `render-worklet.ts` does.
class FakeWorklet {
ring?: AudioRingBuffer;
// Read with each clock message; this fake renders no real time, so it applies no clock.
readonly context = { currentTime: 0 };
#epoch = 0;
#playing = true;
readonly port = Object.assign(new EventTarget(), {
Expand Down
1 change: 1 addition & 0 deletions js/watch/src/audio/decoder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,7 @@ export class Decoder {
rate: sampleRate,
delay: this.sync.out.delay.peek(),
buffered,
clock: this.sync,
playing: this.sync.playing,
});
effect.cleanup(() => ring.close());
Expand Down
41 changes: 41 additions & 0 deletions js/watch/src/audio/render-worklet.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -228,3 +228,44 @@ it("clears comfort noise when the timeline resets", () => {
send({ type: "reset", epoch: 1 });
expect([...read(20)]).toEqual(Array(20).fill(0));
});

// The main thread reads the shared clock once per state report, so the worklet carries it forward on
// its own `currentTime` between readings, until a reset ends the timeline it was read on or a delay
// change moves the clock.
it("carries the shared clock forward on its own time for a buffered ring to catch up to", () => {
const render = new Render();
const send = (data: Message) => render.port.onmessage?.({ data } as MessageEvent<Message>);
const rate = 48_000;
const play = (seconds: number) => {
for (let i = 0; i < (seconds * rate) / 128; i++) {
expect(render.process([], [[new Float32Array(128)]], {})).toBe(true);
time += 128 / rate;
}
return render.sent.at(-1);
};
let time = 10;
const previous = Object.getOwnPropertyDescriptor(globalThis, "currentTime");
Object.defineProperty(globalThis, "currentTime", { configurable: true, get: () => time });
try {
send({ type: "init", channels: 1, rate, latency: Milli(100), buffered: true });
send({ type: "data", timestamp: Micro(1_000_000), data: [new Float32Array(rate * 0.9).fill(0.5)] });
// Half a second before this first quantum, the clock read 700 ms: it now stands 1.2 s, 200 ms past
// the playhead at 1 s.
send({ type: "clock", timestamp: Micro(700_000), time: 9.5 });
const caught = play(0.5);
expect(caught?.accelerated).toBeGreaterThan(200 - 2.5);
expect(caught?.accelerated).toBeLessThanOrEqual(200);

// A delay change moves the clock, so the reading from before it no longer counts.
send({ type: "clock", timestamp: Micro(2_000_000), time });
send({ type: "latency", latency: Milli(100) });
expect(play(0.1)?.accelerated).toBe(caught?.accelerated);

send({ type: "reset", epoch: 1 });
send({ type: "data", timestamp: Micro(1_000_000), data: [new Float32Array(rate * 0.9).fill(0.5)] });
expect(play(0.5)?.accelerated).toBe(caught?.accelerated);
} finally {
if (previous) Object.defineProperty(globalThis, "currentTime", previous);
else Reflect.deleteProperty(globalThis, "currentTime");
}
});
19 changes: 17 additions & 2 deletions js/watch/src/audio/render-worklet.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import type { Micro } from "@moq/clock";
import { ComfortNoise } from "./comfort";
import type { Message, State } from "./render";
import type { Clock, Message, State } from "./render";
import { AudioRingBuffer } from "./ring-buffer";

// Samples to fade over when playback stops or resumes short of a full quantum. A shortfall
Expand All @@ -16,6 +17,8 @@ class Render extends AudioWorkletProcessor {
#playing = true;
// Whether the previous quantum ended short, so the next one fades back in.
#short = false;
// The shared clock's latest reading, which a buffered ring converges on.
#clock?: Clock;

constructor() {
super();
Expand All @@ -28,10 +31,13 @@ class Render extends AudioWorkletProcessor {
this.#playing = msg.playing ?? true;
this.#underflow = 0;
this.#short = false;
this.#clock = undefined;
} else if (msg.type === "data") {
this.#ring?.write(msg.timestamp, msg.data);
} else if (msg.type === "latency") {
this.#ring?.setLatency(msg.latency);
// The clock moves with the delay, so a reading from before it would pull audio ahead.
this.#clock = undefined;
if (msg.park !== undefined) {
this.#ring?.park();
this.#epoch = msg.park;
Expand All @@ -48,16 +54,25 @@ class Render extends AudioWorkletProcessor {
}
this.#epoch = msg.epoch;
this.#playing = msg.playing ?? true;
// A reset ends the timeline the clock was read on.
this.#clock = undefined;
} else if (msg.type === "playing" && msg.epoch === this.#epoch) {
this.#playing = msg.playing;
} else if (msg.type === "clock") {
this.#clock = msg;
}
};
}

process(_inputs: Float32Array[][], outputs: Float32Array[][], _parameters: Record<string, Float32Array>) {
const output = outputs[0];
const ring = this.#ring;
const samplesRead = this.#playing ? (ring?.read(output) ?? 0) : 0;
const clock = this.#clock;
const shared =
clock?.timestamp !== undefined
? ((clock.timestamp + (currentTime - clock.time) * 1_000_000) as Micro)
: undefined;
const samplesRead = this.#playing ? (ring?.read(output, shared) ?? 0) : 0;
Comment thread
kixelated marked this conversation as resolved.
Resolved
// A declared silence between two buffered utterances plays as comfort noise, faded like any other.
const quiet = this.#playing ? ring?.quiet : undefined;
if (quiet) {
Expand Down
14 changes: 13 additions & 1 deletion js/watch/src/audio/render.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import type { Micro, Milli } from "@moq/clock";

/** Everything the main thread sends the render worklet over its port. */
export type Message = Init | Data | Latency | Reset | Truncate | Playing | End;
export type Message = Init | Data | Latency | Reset | Truncate | Playing | End | Clock;

/** Create the worklet's ring. */
export interface Init {
Expand Down Expand Up @@ -44,6 +44,18 @@ export interface Data {
timestamp: Micro;
}

/**
* Where the shared clock stands, for a buffered ring to time-stretch back onto. The clock runs at real
* time, so one reading maps it onto the worklet's own time until the next.
*/
export interface Clock {
type: "clock";
// The clock's media time at context time `time`, unset while the clock has none.
timestamp?: Micro;
// The AudioContext's `currentTime` when the clock was read, in seconds.
time: number;
}

/** Latency update: the floor held as decoded PCM. */
export interface Latency {
type: "latency";
Expand Down
Loading