Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
9163a90
chore(quest): claim quest/m2/audio-playout/output-latency.md
kixelated Nov 13, 2026
bbea5f3
feat(moq-audio,moq-cli): place native audio where it is heard and hol…
kixelated Nov 13, 2026
2fb1f81
feat(watch): place browser audio where it is heard and hold the outpu…
kixelated Nov 13, 2026
c8207d6
fix(moq-cli): don't shadow the output with the latency peak
kixelated Nov 13, 2026
1f250a9
test(audio-quality): project the browser's playhead to the instant it…
kixelated Nov 13, 2026
f183be8
test,doc: cover the output latency and document placing audio where i…
kixelated Nov 13, 2026
df19bf5
quest: finish output latency
kixelated Nov 13, 2026
3fd16ba
fix(watch,moq-audio,moq-cli): hold only the post-output depth at star…
kixelated Nov 13, 2026
c1d9877
fix(watch,moq-cli): count the output latency in buffered startup and …
kixelated Nov 13, 2026
e18ff56
fix(watch): start buffered playback with the opening due as it first …
kixelated Nov 13, 2026
f70c29f
fix(watch): carry the output's lead in the startup span, so an openin…
kixelated Nov 13, 2026
1567b48
fix(watch): pair the clock at its signed position before media zero's…
kixelated Nov 13, 2026
275db58
docs(audio-jitter): a short fixed browser delay underruns rather than…
kixelated Nov 14, 2026
5497e5c
fix(moq-audio): count the part of the last read still sounding in Sin…
kixelated Nov 14, 2026
ded421b
test(moq-audio): count the device latency in the device tests' drain …
kixelated Nov 14, 2026
1ca3afe
fix(watch,moq-cli): cap the native target last, and read the output l…
kixelated Nov 14, 2026
b926d52
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Nov 14, 2026
537e929
fix(watch): drop the worklet's target term and Manual import the merg…
kixelated Nov 14, 2026
57deb31
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Nov 14, 2026
8f08697
docs(audio-jitter): the first target formula is the network's share
kixelated Nov 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 32 additions & 1 deletion doc/concept/audio-jitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,8 @@ already inside every number on this page: a publisher writing 100 ms of media at
once produces arrivals whose relative delay reaches
100 ms less one frame, on a perfect network, with no help from the catalog.

So the advertised value is a **floor**, not an addend:
So the advertised value is a **floor**, not an addend, in the network's share of
the target (audio adds its output latency [at the speaker](#at-the-speaker)):

```
target = max(measured, advertised) + frame
Expand Down Expand Up @@ -517,6 +518,30 @@ interval in progress and any open stall, folds into the continued histogram as
if that interval closed at the switch, judged by the outgoing track's join: a switch most often follows an outage,
and would otherwise drop the outage's one observation.

## At the speaker

Audio is placed on the clock where it is heard, not where it is rendered. The
output takes time to sound what the player hands it, so an audio track's target
also counts that **output latency**, and video waits for it too, which lip sync
on that device needs:

```
target = max(measured, advertised) + frame + output
Comment thread
kixelated marked this conversation as resolved.
Resolved
```

The player reads the clock at the moment the output sounds a sample and renders
the media due then; its buffer holds the target less the delay between render
and speaker. The output latency is the most the device reported over the last
5 s, since a host's report moves with where each read lands.

| | Placed at | `output` |
| --- | --- | --- |
| Browser | `getOutputTimestamp()`, the context time sounding at a page instant, against which the clock is paired at the greatest offset, falling 0.1 ms a report | `baseLatency` plus the peak `outputLatency` |
| Native | the callback's playback timestamp | the host's reported delay plus one device period |

A fixed browser delay is still taken literally, and a fixed native delay is
raised to the output latency, as [Fixed targets](#fixed-targets) describes.

## Rise, fall, startup, and the ceiling

There is **no separate rise or fall limiter**. The histogram is the only
Expand Down Expand Up @@ -631,6 +656,12 @@ An impossible fixed target is refused, not silently adjusted. A negative delay,
or one beyond the track's max age, is an error at configuration time.
Supported or refused, not warned about and continued.

The one exception is the [output latency](#at-the-speaker), which no delay can
undercut. Natively, `moq play` raises a fixed delay below its device's output
latency to that latency, since audio cannot sound sooner. The browser keeps the
delay as given, so one shorter than the jitter plus the output latency
underruns rather than letting audio trail video.

In the browser this is the numeric `delay` setting; natively it is
`decode::Options::delay.min` with `decode::Options::estimate` set to `false`.
`estimate` defaults to `true`, which runs the estimator instead and never goes
Expand Down
192 changes: 164 additions & 28 deletions js/watch/src/audio/buffer.test.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,39 @@
import { describe, expect, it } from "bun:test";
import { Duration } from "@moq/clock";
import { Duration, Instant, now } from "@moq/clock";
import { Manual } from "@moq/clock/test";
import { type Getter, Signal } from "@moq/signals";
import { AudioBuffer, Backpressure } from "./buffer";
import type { Init, Message, State } from "./render";
import { AudioRingBuffer } from "./ring-buffer";

const HOUR = 60 * 60 * 1000;

// A manual clock moved past the page's origin, where a performance time of zero means none.
function started(): { close(): void } {
const manual = new Manual();
const installed = manual.install();
manual.jump(Duration.secs(1));
return installed;
}

// A context that reports its output position as sounding `sounding` now, or nothing until it is set.
function silent(): {
currentTime: number;
baseLatency: number;
outputLatency: number;
sounding?: number;
getOutputTimestamp(): AudioTimestamp;
} {
return {
currentTime: 0,
baseLatency: 0,
outputLatency: 0,
getOutputTimestamp() {
if (this.sounding === undefined) return { contextTime: 0, performanceTime: 0 };
return { contextTime: this.sounding, performanceTime: now().since(Instant.ZERO).asMillis() };
},
};
}
const ms = (value: number) => value * 1000;

// A buffered buffer over a real port, with the worklet's end of it left to the test.
Expand All @@ -16,7 +44,7 @@ function open(): {
close: () => void;
} {
const channel = new MessageChannel();
const buffer = new AudioBuffer({ port: channel.port1 } as unknown as AudioWorkletNode, {
const buffer = new AudioBuffer({ port: channel.port1, context: silent() } as unknown as AudioWorkletNode, {
channels: 1,
rate: 48_000,
delay: Duration.millis(100),
Expand All @@ -30,6 +58,7 @@ function open(): {
const msg: State = {
type: "state",
timestamp: timestamp ?? 0,
time: 0,
stalled,
anchored: timestamp !== undefined,
underruns: 0,
Expand All @@ -56,7 +85,7 @@ function open(): {
describe("AudioBuffer", () => {
it("transfers PCM ownership and orders control messages after the data they change", async () => {
const channel = new MessageChannel();
const buffer = new AudioBuffer({ port: channel.port1 } as unknown as AudioWorkletNode, {
const buffer = new AudioBuffer({ port: channel.port1, context: silent() } as unknown as AudioWorkletNode, {
channels: 2,
rate: 48_000,
delay: Duration.millis(100),
Expand Down Expand Up @@ -407,7 +436,7 @@ function fake(
delay: number,
props: {
bound?: number;
clock?: { now(): Duration | undefined; out: { reference: Getter<number | undefined> } };
clock?: { position(): number | undefined; out: { reference: Getter<number | undefined> } };
} = {},
) {
const sent: Message[] = [];
Expand All @@ -416,7 +445,7 @@ function fake(
start() {},
});
// The context's time in seconds, which the test advances.
const context = { currentTime: 0 };
const context = silent();
const buffer = new AudioBuffer({ port, context } as unknown as AudioWorkletNode, {
channels: 1,
rate: 1000,
Expand All @@ -425,10 +454,11 @@ function fake(
...(props.bound !== undefined && { bound: Duration.millis(props.bound) }),
clock: props.clock,
});
const state = (timestamp: number, stalled: boolean, epoch: number) => {
const state = (timestamp: number, stalled: boolean, epoch: number, time = 0) => {
const data: State = {
type: "state",
timestamp,
time,
stalled,
anchored: true,
underruns: 0,
Expand All @@ -452,7 +482,7 @@ describe("AudioBuffer past its decoded floor", () => {
postMessage: (msg: Message) => sent.push(msg),
start() {},
});
new AudioBuffer({ port } as unknown as AudioWorkletNode, {
new AudioBuffer({ port, context: silent() } as unknown as AudioWorkletNode, {
channels: 2,
rate: 48_000,
delay: Duration.millis(delay),
Expand Down Expand Up @@ -517,43 +547,149 @@ describe("AudioBuffer steering", () => {
}
});

// The context's time advances a render quantum at a time, so a reading of the clock lands up to a
// quantum late against it. The offset keeps the earliest, rises only as slowly as drift needs, and
// jumps when the delay moves the clock for real.
it("sends the clock at its earliest offset from the context's time", () => {
// A browser that reads its output position once per render callback reports it up to a callback
// early, so the offset keeps the greatest, falls only as slowly as drift needs, and moves whole when
// the delay or the reference moves the clock for real.
it("sends the clock against the context time sounding at its greatest offset", () => {
let now: Duration | undefined = Duration.millis(1000);
const reference = new Signal<number | undefined>(0);
const { buffer, sent, state, context } = fake(100, { clock: { now: () => now, out: { reference } } });
const report = (time: number, clock: number | undefined) => {
const clock = started();
const { buffer, sent, state, context } = fake(100, {
clock: { position: () => now?.asMillis(), out: { reference } },
});
const report = (time: number, sounding: number, clock: number | undefined) => {
context.currentTime = time;
context.sounding = sounding;
now = clock === undefined ? undefined : Duration.millis(clock);
state(ms(0), false, 0);
};
try {
report(1, 1000);
// Read 2 ms late: the offset rises by only 100 us.
report(1.25, 1252);
report(1.5, 1500);
// The output sounds 100 ms behind the context's time.
report(1, 0.9, 1000);
// Read 2 ms early: the offset falls by only 100 us.
report(1.25, 1.152, 1250);
report(1.5, 1.4, 1500);
// A delay change moved the clock 50 ms.
report(1.75, 1800);
report(1.75, 1.65, 1800);
// The reference moved 5 ms earlier: the offset takes it whole, under the filter's.
reference.set(-5);
report(2, 2055);
// A later reference rises only as fast as the filter does.
report(2, 1.9, 2055);
// A later reference moves it back, though it is no jump.
reference.set(0);
report(2.25, 2300);
report(2.5, undefined);
report(2.25, 2.15, 2300);
report(2.5, 2.4, undefined);
expect(sent.filter((msg) => msg.type === "clock")).toEqual([
{ type: "clock", timestamp: 1_000_000, time: 1 },
{ type: "clock", timestamp: 1_250_100, time: 1.25 },
{ type: "clock", timestamp: 1_500_000, time: 1.5 },
{ type: "clock", timestamp: 1_800_000, time: 1.75 },
{ type: "clock", timestamp: 2_055_000, time: 2 },
{ type: "clock", timestamp: 2_300_000, time: 2.25 },
{ type: "clock", timestamp: 1_100_000, time: 1 },
{ type: "clock", timestamp: 1_349_900, time: 1.25 },
{ type: "clock", timestamp: 1_600_000, time: 1.5 },
{ type: "clock", timestamp: 1_900_000, time: 1.75 },
{ type: "clock", timestamp: 2_155_000, time: 2 },
{ type: "clock", timestamp: 2_400_000, time: 2.25 },
{ type: "clock", time: 2.5 },
]);
} finally {
buffer.close();
clock.close();
}
});

// Before media zero's turn, as while an opening waits for its output to sound it, the clock pairs
// at its negative position rather than the zero it reads as.
it("pairs a clock short of media zero at its negative position", () => {
const clock = started();
const { buffer, sent, state, context } = fake(100, {
clock: { position: () => -120, out: { reference: new Signal<number | undefined>(0) } },
});
try {
context.sounding = 1;
state(ms(0), false, 0);
expect(sent.filter((msg) => msg.type === "clock")).toEqual([
{ type: "clock", timestamp: -1_120_000, time: 0 },
]);
} finally {
buffer.close();
clock.close();
}
});

// Before the output has played anything there is no position to pair the clock with.
it("sends no clock until the output reports a position", () => {
const { buffer, sent, state } = fake(100, {
clock: { position: () => 1000, out: { reference: new Signal<number | undefined>(0) } },
});
try {
state(ms(0), false, 0);
expect(sent.filter((msg) => msg.type === "clock")).toEqual([{ type: "clock", time: 0 }]);
} finally {
buffer.close();
}
});

// The ring renders ahead of the speaker by the output latency, so it holds the delay less that,
// decoded floor included, and the delay the clock holds counts the base latency and the output
// latency's recent peak.
it("holds the delay less the output latency it renders ahead by", () => {
const { buffer, sent, state, context } = fake(300);
try {
context.baseLatency = 0.01;
context.outputLatency = 0.12;
state(ms(0), false, 0);
context.outputLatency = 0.104;
state(ms(0), false, 0);
expect(buffer.output.peek()).toEqual(Duration.millis(130));
buffer.setLatency(Duration.millis(300));
// A delay short of the output latency still holds one callback.
expect(buffer.held(Duration.millis(50))).toEqual(Duration.millis(10));
buffer.setLatency(Duration.millis(50));
expect(sent.filter((msg) => msg.type === "latency")).toEqual([
{ type: "latency", latency: 180, bound: 0 },
{ type: "latency", latency: 128, bound: 0 },
]);
} finally {
buffer.close();
}
});

// Startup holds only what the ring must from the first message, before any state report.
it("holds the delay less the output latency the context reports at construction", () => {
const sent: Message[] = [];
const port = Object.assign(new EventTarget(), { postMessage: (msg: Message) => sent.push(msg), start() {} });
const context = silent();
context.outputLatency = 0.12;
const buffer = new AudioBuffer({ port, context } as unknown as AudioWorkletNode, {
channels: 1,
rate: 1000,
delay: Duration.millis(300),
buffered: false,
});
try {
expect(buffer.held(Duration.millis(300))).toEqual(Duration.millis(180));
expect(sent[0]).toMatchObject({ type: "init", latency: 180 });
} finally {
buffer.close();
}
});

// The speaker sounds what was rendered an output latency ago, so the playback timestamp is where
// the rendered playhead stood then, advancing at real time, and holds through a stall.
it("reports the playhead sounding now", () => {
const clock = started();
const { buffer, state, context } = fake(100);
try {
context.sounding = 1;
state(ms(5000), false, 0, 1.1);
// Nothing rendered at or before what is sounding: the first report, back by the difference.
expect(buffer.timestamp.peek()).toBe(ms(4900));
context.sounding = 1.12;
state(ms(5050), false, 0, 1.15);
expect(buffer.timestamp.peek()).toBe(ms(5020));
// The ring stalled at 5050: what sounds holds there once it has caught up.
context.sounding = 1.17;
state(ms(5050), true, 0, 1.2);
expect(buffer.timestamp.peek()).toBe(ms(5050));
} finally {
buffer.close();
clock.close();
}
});
});
Loading