Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
b66835c
chore(quest): claim quest/m1/audio-jitter-target/watch.md
kixelated Nov 6, 2026
250ad37
feat(watch): time-stretch the audio ring onto its target
kixelated Nov 6, 2026
bcfeade
Merge remote-tracking branch 'origin/main' into feat/audio-jitter-str…
kixelated Nov 6, 2026
4e8f815
Merge remote-tracking branch 'origin/main' into quest/m1/audio-jitter…
kixelated Nov 6, 2026
05c0846
feat(audio): switch expand variants for the listening check
kixelated Nov 6, 2026
354e35d
fix(watch): unstage a staged window on a handover truncate
kixelated Nov 6, 2026
3ccc61a
feat(watch): stretch with NetEq's expand and maximum-delay band
kixelated Nov 7, 2026
a115752
Merge remote-tracking branch 'origin/main' into feat/audio-jitter-str…
kixelated Nov 7, 2026
0805f87
Merge branch 'quest/m1/audio-jitter-target/watch' of https://github.c…
kixelated Nov 7, 2026
a6e0761
fix(watch): drop the staged window with the audio a re-anchor skips
kixelated Nov 7, 2026
6d351f6
Merge remote-tracking branch 'origin/main' into fix/watch-audio-jitter
kixelated Nov 7, 2026
b075e85
quest: record the clean MacBook Air listening run
kixelated Nov 7, 2026
e0af594
quest: judge js/watch's stretch on automated measurements alone
kixelated Nov 7, 2026
d1aebc4
Merge origin/main into quest/m1/audio-jitter-target/watch
kixelated Nov 7, 2026
03f8e9e
js/watch: count a re-anchor that drops only the staged window as a skip
kixelated Nov 7, 2026
e858f4d
js/watch: keep the declared-silence catch-up forward-only
kixelated Nov 7, 2026
c5461c1
Merge origin/main into quest/m1/audio-jitter-target/watch
kixelated Nov 8, 2026
35ec11a
Merge origin/main into quest/m1/audio-jitter-target/watch
kixelated Nov 8, 2026
a01e4c7
fix(watch): the audio ring's stretch never settles audio behind the c…
kixelated Nov 8, 2026
11edd06
Merge origin/main into quest/m1/audio-jitter-target/watch
kixelated Nov 8, 2026
683d964
test(audio-quality): the probe's fake sync reads a clock, and the lag…
kixelated Nov 8, 2026
5b5795c
Merge remote-tracking branch 'origin/main' into quest/m1/audio-jitter…
kixelated Nov 8, 2026
ffd6e03
Merge remote-tracking branch 'origin/main' into quest/m1/audio-jitter…
kixelated Nov 9, 2026
77fce72
quest: plan native A/V lag reporting and its expand decision
kixelated Nov 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 45 additions & 12 deletions doc/concept/audio-jitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -614,20 +614,24 @@ the `bun doc/concept/audio-jitter/corpus.ts` command above. It runs in

## Received-audio time stretch

Native playback adapts received PCM to the target with WebRTC NetEq's
accelerate and preemptive expand. It never generates replacement audio for a
missing packet. The estimator above is unchanged. The browser implementation
must conform to this same algorithm and [PCM corpus](audio-jitter/stretch/corpus.json)
when it adopts stretching; the browser does not yet stretch on this revision.
Both players adapt received PCM to the target with the seams of WebRTC
NetEq's time stretch: native playback in `rs/moq-audio` with its accelerate and
preemptive expand, and the browser in the AudioWorklet's ring in `js/watch`
with its accelerate alone. Neither generates replacement audio for a missing
packet, so a hole in the media plays as a hole. The estimator above is
unchanged. Both share the pitch search, admission, seams, and
[PCM corpus](audio-jitter/stretch/corpus.json) below; they differ only in
[what they decide on](#browser-decision). The browser stretches only live
playback: buffered playback plays every sample it holds, as it always has.

The source is WebRTC's
[`TimeStretch`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/time_stretch.cc),
[`Accelerate`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/accelerate.cc),
[`PreemptiveExpand`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/preemptive_expand.cc),
[`DspHelper`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/dsp_helper.cc),
and [`DecisionLogic`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/decision_logic.cc),
read on 2026-11-04. The native port retains the BSD license and patent grant in
`rs/moq-audio/WEBRTC-LICENSE` and `WEBRTC-PATENTS`.
read on 2026-11-04. Both ports retain the BSD license and patent grant, in
`WEBRTC-LICENSE` and `WEBRTC-PATENTS` beside `rs/moq-audio` and `js/watch`.

### PCM and decision state

Expand All @@ -637,13 +641,16 @@ below are frames at the output device's rate `R`. Duration-to-frame conversion
uses `floor(R * milliseconds / 1000)`, except a measured pitch rounds to the
nearest frame, with half rounded upward. A processing window is 30 ms of
**newly received PCM**. A short window passes unchanged. Startup padding is
never analysed, and finish drains the remaining input without stretching.
never analysed, and finish drains the remaining input without stretching. So
does the tail before a [declared end](#declared-silence): running dry after it
is silence rather than an underrun, so nothing expands toward a target that no
arrival will refill.

`buffered` counts unread frames in the ring plus staged output. `target` is
the current playout target. The state is `cooldown`, initially 60 ms
(NetEq's six startup ticks), reduced
by each frame rendered, saturating at zero. At a window boundary, with
`high = target + 20 ms`, choose in this order:
by each frame rendered, saturating at zero. At a window boundary, native
playback chooses in this order, with `high = target + 20 ms`:

1. `buffered >= 4 * high`: fast accelerate, even during cooldown.
2. `cooldown > 0`: hold.
Expand All @@ -658,6 +665,27 @@ packet-history maximum-delay term is not added again: this controller sees the
already decoded PCM queue and its jitter target rather than an RTP playout
timestamp and packet-arrival history. There is no concealment state.

### Browser decision

The browser's ring starts on its target and plays at the shared clock video
plays from, so slowing it down only ever settles audio behind that clock. It
never expands. It decides on `peak`, the ring's length right after the latest
write less what the stretch has removed since: how far the playhead trails the
newest arrival, which is the playout delay NetEq's `DecisionLogic` compares.
`buffered` drains between arrivals because the next one has not landed, not
because audio is ahead of the clock, so deciding on it expands into every gap
and settles a burst behind. With `excess = peak - target`, choose in this order:

1. `excess >= 20 ms`: fast accelerate, even during cooldown.
2. `cooldown > 0` or `excess <= 0`: hold.
3. Otherwise: accelerate, admitting only a lag `L <= excess`.

So the ring rests within one pitch period of its target right after each
arrival, where dropping back to the target used to leave it, and a fast
accelerate after a refill catches up about as fast as that drop did. Arrivals that
outrun the target underrun instead: that is the cost of a target short of the
spread, and the estimator's to fix, not the stretch's.

### Pitch search and admission

Use channel zero as NetEq does. All channels share the selected seam and lag,
Expand Down Expand Up @@ -720,7 +748,10 @@ ring and staged output hold the target again. Padding never enters the pitch
search. Finishing bypasses that wait so a short tail can drain. Trimming is
emergency recovery at the ring's capacity watermark. The
CLI's incoming-media trim is reserved for a distance at the fast threshold,
not ordinary 20 ms drift. These bounds do not promise concealment of missing
not ordinary 20 ms drift. The browser's ring receives each decoded frame as a
write, so it re-anchors on its target there, at the lower of `4 * high` and
three quarters of its headroom above the target, and counts each re-anchor as
a skip in its stats. These bounds do not promise concealment of missing
media or convergence of a signal whose seams continually fail admission.

### Stretch corpus
Expand All @@ -732,7 +763,9 @@ Compare output within 1e-6 absolute sample amplitude. It includes hold,
cooldown, fast-mode bypass, linked stereo, passive noise, rejected active
noise, and 8/44.1/48 kHz seams. Both implementations consume these exact
vectors; generating expected output from the implementation under test is
not conformance.
not conformance. The vectors decide as native playback does, so the browser
grades its accelerate and fast seams against them, at an excess inside and
past the 20 ms band, and holds wherever native expands.

## Comfort noise

Expand Down
9 changes: 6 additions & 3 deletions doc/lib/js/watch.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,9 @@ player publishes its own broadcast, reporting only what it counted, so a second
player on one connection that resolves the same feedback path, for any catalog,
publishes nothing and warns until the first one leaves; give each player its own
name. The decoders'
`out.stats` and the stats panel read the same counters. The token must allow
`out.stats` and the stats panel read the same counters. Audio's `out.stats`
also counts `skips`, each jump that dropped buffered audio rather than playing
it faster, which the stats panel shows and the feedback track leaves out. The token must allow
publishing under the echo path.

## Binding from a framework
Expand Down Expand Up @@ -239,8 +241,9 @@ them; nothing is retained without an open handle.

## Buffered playback

By default the player minimizes latency: it skips ahead whenever media piles
up past the delay. Content produced faster than real time, such as a TTS
By default the player minimizes latency: it time-stretches received audio
that falls behind the delay back onto it, playing it slightly faster, and skips
ahead only when media piles up far past it. Content produced faster than real time, such as a TTS
response emitted in one burst with future timestamps, wants the opposite. Set
`lookahead` to how far ahead it may run and it plays through at the encoded pace:

Expand Down
9 changes: 7 additions & 2 deletions js/common/package.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// This creates a dist/ folder with the correct paths and dependencies for publishing
// Split from release.ts to allow building packages without publishing

import { copyFileSync, existsSync, readFileSync, writeFileSync } from "node:fs";
import { copyFileSync, existsSync, readdirSync, readFileSync, writeFileSync } from "node:fs";
import { join, resolve } from "node:path";
import { publint } from "publint";
import { formatMessage } from "publint/utils";
Expand Down Expand Up @@ -112,9 +112,14 @@ if (existsSync("src/version.ts")) {
writeFileSync(file, body.replace(placeholder, `VERSION = ${JSON.stringify(pkg.version)};`));
}

// Copy static files
// Copy static files, including the license and patent notices of any third-party code ported in,
// such as js/watch's WEBRTC-LICENSE and WEBRTC-PATENTS.
console.log("📄 Copying README.md...");
copyFileSync("README.md", join("dist", "README.md"));
for (const notice of readdirSync(".").filter((name) => /^[A-Z0-9]+-(LICENSE|PATENTS)$/.test(name))) {
console.log(`📄 Copying ${notice}...`);
copyFileSync(notice, join("dist", notice));
}

// Lint the package to catch publishing issues
console.log("🔍 Running publint...");
Expand Down
29 changes: 29 additions & 0 deletions js/watch/WEBRTC-LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
Copyright (c) 2011, The WebRTC project authors. All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in
the documentation and/or other materials provided with the
distribution.

* Neither the name of Google nor the names of its contributors may
be used to endorse or promote products derived from this software
without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
24 changes: 24 additions & 0 deletions js/watch/WEBRTC-PATENTS
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
Additional IP Rights Grant (Patents)

"This implementation" means the copyrightable works distributed by
Google as part of the WebRTC code package.

Google hereby grants to you a perpetual, worldwide, non-exclusive,
no-charge, irrevocable (except as stated in this section) patent
license to make, have made, use, offer to sell, sell, import,
transfer, and otherwise run, modify and propagate the contents of this
implementation of the WebRTC code package, where such license applies
only to those patent claims, both currently owned by Google and
acquired in the future, licensable by Google that are necessarily
infringed by this implementation of the WebRTC code package. This
grant does not include claims that would be infringed only as a
consequence of further modification of this implementation. If you or
your agent or exclusive licensee institute or order or agree to the
institution of patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that this
implementation of the WebRTC code package or any code incorporated
within this implementation of the WebRTC code package constitutes
direct or contributory patent infringement, or inducement of patent
infringement, then any patent rights granted to you under this License
for this implementation of the WebRTC code package shall terminate as
of the date such litigation is filed.
1 change: 1 addition & 0 deletions js/watch/api/index.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ export namespace Audio {
bytesReceived: number;
errors: number;
late: number;
skips: number;
stalled: Clock.Milli;
stalls: number;
underruns: number;
Expand Down
12 changes: 11 additions & 1 deletion js/watch/src/audio/buffer.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ function open(): {
stalled,
anchored: timestamp !== undefined,
underruns: 0,
skips: 0,
silent: false,
epoch,
};
Expand Down Expand Up @@ -433,7 +434,16 @@ function fake(delay: Milli, buffered = false) {
buffered,
});
const state = (timestamp: Micro, stalled: boolean, epoch: number) => {
const data: State = { type: "state", timestamp, stalled, anchored: true, underruns: 0, silent: false, epoch };
const data: State = {
type: "state",
timestamp,
stalled,
anchored: true,
underruns: 0,
skips: 0,
silent: false,
epoch,
};
port.dispatchEvent(new MessageEvent("message", { data }));
};
return { buffer, sent, state };
Expand Down
21 changes: 18 additions & 3 deletions js/watch/src/audio/buffer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ import type { Data, End, Init, Latency, Playing, Reset, State, Truncate } from "
* stops once the PCM in the ring plus the frames still decoding reach the floor, since the ring
* plays through and cannot shed what overflows it;
* - while playing, once the playhead is within the floor of where its output lands, keeping the
* ring on its floor;
* ring on its floor. Live, that is where the output ends: the ring's time stretch measures its
* depth right after each write, and reads a tail past the floor as audio behind the clock;
* - while playing live, also once the media queued behind it covers the hold, so media past the
* target reaches the ring, whose skip band catches up to the live edge. Buffered mode plays
* through instead.
Expand Down Expand Up @@ -63,6 +64,10 @@ export class Backpressure {
// ring compares a frame to its playhead where the output lands, or an offset past the floor would
// drain it.
#offset = 0;
// The duration of the latest output, which the next frame's is taken to match. A live frame released
// by where its output starts would land with its tail past the floor, and the stretch accelerating
// that away would only draw the next frame out sooner, playing through the hold.
#frame = 0 as Micro;

#waiters: Array<{ timestamp: Micro; resolve: () => void }> = [];

Expand Down Expand Up @@ -141,6 +146,7 @@ export class Backpressure {

/** Note that PCM from `start` to `end` reached the ring. */
inserted(start: Micro, end: Micro): void {
this.#frame = (end - start) as Micro;
const inserted = this.#inserted;
this.#inserted = { start: inserted?.start ?? start, end: inserted && inserted.end > end ? inserted.end : end };
this.#release();
Expand Down Expand Up @@ -229,7 +235,8 @@ export class Backpressure {
#due(timestamp: Micro): boolean {
const playhead = this.#playhead;
if (!this.#stalled && playhead !== undefined) {
if (timestamp - this.#offset - this.#headroom <= playhead) return true;
const tail = this.#buffered ? 0 : this.#frame;
if (timestamp + tail - this.#offset - this.#headroom <= playhead) return true;
return !this.#buffered && this.#covered(timestamp);
}

Expand Down Expand Up @@ -270,6 +277,10 @@ export class AudioBuffer {
/** How many times the ring has run dry mid-playback, cumulative. */
readonly underruns: Getter<number> = this.#underruns;

readonly #skips = new Signal<number>(0);
/** How many times the ring skipped ahead, dropping queued audio, cumulative. */
readonly skips: Getter<number> = this.#skips;

readonly #silent = new Signal<boolean>(false);
/** Whether playback is at a declared end, reached or announced before any audio, and nothing has played since. */
readonly silent: Getter<boolean> = this.#silent;
Expand Down Expand Up @@ -326,6 +337,7 @@ export class AudioBuffer {
this.#timestamp.set(data.timestamp);
this.#stalled.set(data.stalled);
this.#underruns.set(data.underruns);
this.#skips.set(data.skips);
this.#silent.set(data.silent);
if (data.epoch === this.#epoch) {
this.#backpressure.observe(data.anchored ? data.timestamp : undefined, data.stalled);
Expand Down Expand Up @@ -354,7 +366,10 @@ export class AudioBuffer {
this.#backpressure.dropped();
}

/** Insert audio samples at the given timestamp, transferring their buffers. Handles out-of-order writes. */
/**
* Insert audio samples at the given timestamp, transferring their buffers. A write may land out of
* order, but samples behind what live playback already staged, up to 30 ms past the playhead, are dropped.
*/
insert(timestamp: Micro, data: Float32Array[]): void {
const end = (timestamp + Micro.fromSecond(((data[0]?.length ?? 0) / this.rate) as Second)) as Micro;
this.#backpressure.inserted(timestamp, end);
Expand Down
22 changes: 14 additions & 8 deletions js/watch/src/audio/decoder.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,9 @@ class FakeWorklet {
get length(): number {
return this.ring?.length ?? 0;
}
get skips(): number {
return this.ring?.skips ?? 0;
}
// Render one quantum and report the ring's state back to the main thread.
read(output: Float32Array[]): number {
const ring = this.ring;
Expand All @@ -108,6 +111,7 @@ class FakeWorklet {
stalled: ring.stalled,
anchored: ring.anchored,
underruns: ring.underruns,
skips: ring.skips,
silent: ring.silent,
epoch: this.#epoch,
};
Expand Down Expand Up @@ -1357,7 +1361,7 @@ describe("Decoder across a rename", () => {
const delay = playback.sync.out.delay.peek();
expect(delay).toBeGreaterThan(Milli(200));
const ring = playback.ring();
const before = { ...playback.counts, buffered: ring?.length ?? 0 };
const before = { silent: playback.counts.silent, skips: ring?.skips ?? 0 };
let lowest = delay;
const watch = new Effect();
watch.subscribe(playback.sync.out.delay, (value) => {
Expand All @@ -1368,10 +1372,8 @@ describe("Decoder across a rename", () => {
await playback.run(26_000);
watch.close();

const { inserted, played, silent } = playback.counts;
const buffered = playback.ring()?.length ?? 0;
const skipped = inserted - before.inserted - (played - before.played) - (buffered - before.buffered);
expect({ skipped, silent: silent - before.silent }).toEqual({ skipped: 0, silent: 0 });
const skips = (playback.ring()?.skips ?? 0) - before.skips;
expect({ skips, silent: playback.counts.silent - before.silent }).toEqual({ skips: 0, silent: 0 });
expect(playback.ring()).toBe(ring);
expect(lowest).toBeGreaterThanOrEqual(delay);
} finally {
Expand Down Expand Up @@ -1715,13 +1717,17 @@ for (const cmaf of [false, true]) {
(buffered ? [0, 20, 40, 60, 80] : [140, 160]).map((ms) => start + ms * 1000),
);

// A running live ring sheds old audio when it gets ahead; a buffered ring plays it all.
// A running live ring sheds old audio and lands on its floor once it reaches the fast
// accelerate threshold, four times a 100 ms floor plus its 20 ms band; a buffered ring
// plays it all.
const ring = worklets.at(-1)?.ring;
if (!ring) throw new Error("no running audio ring");
ring.reset();
const samples = [new Float32Array(4800), new Float32Array(4800)];
for (const timestamp of [0, 100_000, 200_000]) ring.write(Micro(timestamp), samples);
expect(ring.length).toBe(samples[0].length * (buffered ? 3 : 1));
for (let timestamp = 0; timestamp < 500_000; timestamp += 100_000)
ring.write(Micro(timestamp), samples);
expect(ring.length).toBe(samples[0].length * (buffered ? 5 : 1));
expect(ring.skips).toBe(buffered ? 0 : 1);
} finally {
playback.close();
await microtasks();
Expand Down
Loading
Loading