diff --git a/doc/concept/audio-jitter.md b/doc/concept/audio-jitter.md index fbdc52e32e3..a7d55b60b45 100644 --- a/doc/concept/audio-jitter.md +++ b/doc/concept/audio-jitter.md @@ -614,11 +614,15 @@ the `bun doc/concept/audio-jitter/corpus.ts` command above. It runs in ## Received-audio time stretch -Native playback adapts received PCM to the target with WebRTC NetEq's -accelerate and preemptive expand. It never generates replacement audio for a -missing packet. The estimator above is unchanged. The browser implementation -must conform to this same algorithm and [PCM corpus](audio-jitter/stretch/corpus.json) -when it adopts stretching; the browser does not yet stretch on this revision. +Both players adapt received PCM to the target with the seams of WebRTC +NetEq's time stretch: native playback in `rs/moq-audio` with its accelerate and +preemptive expand, and the browser in the AudioWorklet's ring in `js/watch` +with its accelerate alone. Neither generates replacement audio for a missing +packet, so a hole in the media plays as a hole. The estimator above is +unchanged. Both share the pitch search, admission, seams, and +[PCM corpus](audio-jitter/stretch/corpus.json) below; they differ only in +[what they decide on](#browser-decision). The browser stretches only live +playback: buffered playback plays every sample it holds, as it always has. The source is WebRTC's [`TimeStretch`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/time_stretch.cc), @@ -626,8 +630,8 @@ The source is WebRTC's [`PreemptiveExpand`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/preemptive_expand.cc), [`DspHelper`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/dsp_helper.cc), and [`DecisionLogic`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/decision_logic.cc), -read on 2026-11-04. The native port retains the BSD license and patent grant in -`rs/moq-audio/WEBRTC-LICENSE` and `WEBRTC-PATENTS`. +read on 2026-11-04. Both ports retain the BSD license and patent grant, in +`WEBRTC-LICENSE` and `WEBRTC-PATENTS` beside `rs/moq-audio` and `js/watch`. ### PCM and decision state @@ -637,13 +641,16 @@ below are frames at the output device's rate `R`. Duration-to-frame conversion uses `floor(R * milliseconds / 1000)`, except a measured pitch rounds to the nearest frame, with half rounded upward. A processing window is 30 ms of **newly received PCM**. A short window passes unchanged. Startup padding is -never analysed, and finish drains the remaining input without stretching. +never analysed, and finish drains the remaining input without stretching. So +does the tail before a [declared end](#declared-silence): running dry after it +is silence rather than an underrun, so nothing expands toward a target that no +arrival will refill. `buffered` counts unread frames in the ring plus staged output. `target` is the current playout target. The state is `cooldown`, initially 60 ms (NetEq's six startup ticks), reduced -by each frame rendered, saturating at zero. At a window boundary, with -`high = target + 20 ms`, choose in this order: +by each frame rendered, saturating at zero. At a window boundary, native +playback chooses in this order, with `high = target + 20 ms`: 1. `buffered >= 4 * high`: fast accelerate, even during cooldown. 2. `cooldown > 0`: hold. @@ -658,6 +665,27 @@ packet-history maximum-delay term is not added again: this controller sees the already decoded PCM queue and its jitter target rather than an RTP playout timestamp and packet-arrival history. There is no concealment state. +### Browser decision + +The browser's ring starts on its target and plays at the shared clock video +plays from, so slowing it down only ever settles audio behind that clock. It +never expands. It decides on `peak`, the ring's length right after the latest +write less what the stretch has removed since: how far the playhead trails the +newest arrival, which is the playout delay NetEq's `DecisionLogic` compares. +`buffered` drains between arrivals because the next one has not landed, not +because audio is ahead of the clock, so deciding on it expands into every gap +and settles a burst behind. With `excess = peak - target`, choose in this order: + +1. `excess >= 20 ms`: fast accelerate, even during cooldown. +2. `cooldown > 0` or `excess <= 0`: hold. +3. Otherwise: accelerate, admitting only a lag `L <= excess`. + +So the ring rests within one pitch period of its target right after each +arrival, where dropping back to the target used to leave it, and a fast +accelerate after a refill catches up about as fast as that drop did. Arrivals that +outrun the target underrun instead: that is the cost of a target short of the +spread, and the estimator's to fix, not the stretch's. + ### Pitch search and admission Use channel zero as NetEq does. All channels share the selected seam and lag, @@ -720,7 +748,10 @@ ring and staged output hold the target again. Padding never enters the pitch search. Finishing bypasses that wait so a short tail can drain. Trimming is emergency recovery at the ring's capacity watermark. The CLI's incoming-media trim is reserved for a distance at the fast threshold, -not ordinary 20 ms drift. These bounds do not promise concealment of missing +not ordinary 20 ms drift. The browser's ring receives each decoded frame as a +write, so it re-anchors on its target there, at the lower of `4 * high` and +three quarters of its headroom above the target, and counts each re-anchor as +a skip in its stats. These bounds do not promise concealment of missing media or convergence of a signal whose seams continually fail admission. ### Stretch corpus @@ -732,7 +763,9 @@ Compare output within 1e-6 absolute sample amplitude. It includes hold, cooldown, fast-mode bypass, linked stereo, passive noise, rejected active noise, and 8/44.1/48 kHz seams. Both implementations consume these exact vectors; generating expected output from the implementation under test is -not conformance. +not conformance. The vectors decide as native playback does, so the browser +grades its accelerate and fast seams against them, at an excess inside and +past the 20 ms band, and holds wherever native expands. ## Comfort noise diff --git a/doc/lib/js/watch.md b/doc/lib/js/watch.md index ef103a9bf41..d8a06bb91ea 100644 --- a/doc/lib/js/watch.md +++ b/doc/lib/js/watch.md @@ -107,7 +107,9 @@ player publishes its own broadcast, reporting only what it counted, so a second player on one connection that resolves the same feedback path, for any catalog, publishes nothing and warns until the first one leaves; give each player its own name. The decoders' -`out.stats` and the stats panel read the same counters. The token must allow +`out.stats` and the stats panel read the same counters. Audio's `out.stats` +also counts `skips`, each jump that dropped buffered audio rather than playing +it faster, which the stats panel shows and the feedback track leaves out. The token must allow publishing under the echo path. ## Binding from a framework @@ -239,8 +241,9 @@ them; nothing is retained without an open handle. ## Buffered playback -By default the player minimizes latency: it skips ahead whenever media piles -up past the delay. Content produced faster than real time, such as a TTS +By default the player minimizes latency: it time-stretches received audio +that falls behind the delay back onto it, playing it slightly faster, and skips +ahead only when media piles up far past it. Content produced faster than real time, such as a TTS response emitted in one burst with future timestamps, wants the opposite. Set `lookahead` to how far ahead it may run and it plays through at the encoded pace: diff --git a/js/common/package.ts b/js/common/package.ts index a19ace17e48..7075691fba9 100644 --- a/js/common/package.ts +++ b/js/common/package.ts @@ -2,7 +2,7 @@ // This creates a dist/ folder with the correct paths and dependencies for publishing // Split from release.ts to allow building packages without publishing -import { copyFileSync, existsSync, readFileSync, writeFileSync } from "node:fs"; +import { copyFileSync, existsSync, readdirSync, readFileSync, writeFileSync } from "node:fs"; import { join, resolve } from "node:path"; import { publint } from "publint"; import { formatMessage } from "publint/utils"; @@ -112,9 +112,14 @@ if (existsSync("src/version.ts")) { writeFileSync(file, body.replace(placeholder, `VERSION = ${JSON.stringify(pkg.version)};`)); } -// Copy static files +// Copy static files, including the license and patent notices of any third-party code ported in, +// such as js/watch's WEBRTC-LICENSE and WEBRTC-PATENTS. console.log("📄 Copying README.md..."); copyFileSync("README.md", join("dist", "README.md")); +for (const notice of readdirSync(".").filter((name) => /^[A-Z0-9]+-(LICENSE|PATENTS)$/.test(name))) { + console.log(`📄 Copying ${notice}...`); + copyFileSync(notice, join("dist", notice)); +} // Lint the package to catch publishing issues console.log("🔍 Running publint..."); diff --git a/js/watch/WEBRTC-LICENSE b/js/watch/WEBRTC-LICENSE new file mode 100644 index 00000000000..4c41b7b251e --- /dev/null +++ b/js/watch/WEBRTC-LICENSE @@ -0,0 +1,29 @@ +Copyright (c) 2011, The WebRTC project authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + * Neither the name of Google nor the names of its contributors may + be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/js/watch/WEBRTC-PATENTS b/js/watch/WEBRTC-PATENTS new file mode 100644 index 00000000000..190607ac261 --- /dev/null +++ b/js/watch/WEBRTC-PATENTS @@ -0,0 +1,24 @@ +Additional IP Rights Grant (Patents) + +"This implementation" means the copyrightable works distributed by +Google as part of the WebRTC code package. + +Google hereby grants to you a perpetual, worldwide, non-exclusive, +no-charge, irrevocable (except as stated in this section) patent +license to make, have made, use, offer to sell, sell, import, +transfer, and otherwise run, modify and propagate the contents of this +implementation of the WebRTC code package, where such license applies +only to those patent claims, both currently owned by Google and +acquired in the future, licensable by Google that are necessarily +infringed by this implementation of the WebRTC code package. This +grant does not include claims that would be infringed only as a +consequence of further modification of this implementation. If you or +your agent or exclusive licensee institute or order or agree to the +institution of patent litigation against any entity (including a +cross-claim or counterclaim in a lawsuit) alleging that this +implementation of the WebRTC code package or any code incorporated +within this implementation of the WebRTC code package constitutes +direct or contributory patent infringement, or inducement of patent +infringement, then any patent rights granted to you under this License +for this implementation of the WebRTC code package shall terminate as +of the date such litigation is filed. diff --git a/js/watch/api/index.api.md b/js/watch/api/index.api.md index 4fc9e5c9faf..690197eb262 100644 --- a/js/watch/api/index.api.md +++ b/js/watch/api/index.api.md @@ -92,6 +92,7 @@ export namespace Audio { bytesReceived: number; errors: number; late: number; + skips: number; stalled: Clock.Milli; stalls: number; underruns: number; diff --git a/js/watch/src/audio/buffer.test.ts b/js/watch/src/audio/buffer.test.ts index ab2ac43a6ed..fa8a065764f 100644 --- a/js/watch/src/audio/buffer.test.ts +++ b/js/watch/src/audio/buffer.test.ts @@ -32,6 +32,7 @@ function open(): { stalled, anchored: timestamp !== undefined, underruns: 0, + skips: 0, silent: false, epoch, }; @@ -433,7 +434,16 @@ function fake(delay: Milli, buffered = false) { buffered, }); const state = (timestamp: Micro, stalled: boolean, epoch: number) => { - const data: State = { type: "state", timestamp, stalled, anchored: true, underruns: 0, silent: false, epoch }; + const data: State = { + type: "state", + timestamp, + stalled, + anchored: true, + underruns: 0, + skips: 0, + silent: false, + epoch, + }; port.dispatchEvent(new MessageEvent("message", { data })); }; return { buffer, sent, state }; diff --git a/js/watch/src/audio/buffer.ts b/js/watch/src/audio/buffer.ts index 446e3c31e44..a8b0c69ee8d 100644 --- a/js/watch/src/audio/buffer.ts +++ b/js/watch/src/audio/buffer.ts @@ -16,7 +16,8 @@ import type { Data, End, Init, Latency, Playing, Reset, State, Truncate } from " * stops once the PCM in the ring plus the frames still decoding reach the floor, since the ring * plays through and cannot shed what overflows it; * - while playing, once the playhead is within the floor of where its output lands, keeping the - * ring on its floor; + * ring on its floor. Live, that is where the output ends: the ring's time stretch measures its + * depth right after each write, and reads a tail past the floor as audio behind the clock; * - while playing live, also once the media queued behind it covers the hold, so media past the * target reaches the ring, whose skip band catches up to the live edge. Buffered mode plays * through instead. @@ -63,6 +64,10 @@ export class Backpressure { // ring compares a frame to its playhead where the output lands, or an offset past the floor would // drain it. #offset = 0; + // The duration of the latest output, which the next frame's is taken to match. A live frame released + // by where its output starts would land with its tail past the floor, and the stretch accelerating + // that away would only draw the next frame out sooner, playing through the hold. + #frame = 0 as Micro; #waiters: Array<{ timestamp: Micro; resolve: () => void }> = []; @@ -141,6 +146,7 @@ export class Backpressure { /** Note that PCM from `start` to `end` reached the ring. */ inserted(start: Micro, end: Micro): void { + this.#frame = (end - start) as Micro; const inserted = this.#inserted; this.#inserted = { start: inserted?.start ?? start, end: inserted && inserted.end > end ? inserted.end : end }; this.#release(); @@ -229,7 +235,8 @@ export class Backpressure { #due(timestamp: Micro): boolean { const playhead = this.#playhead; if (!this.#stalled && playhead !== undefined) { - if (timestamp - this.#offset - this.#headroom <= playhead) return true; + const tail = this.#buffered ? 0 : this.#frame; + if (timestamp + tail - this.#offset - this.#headroom <= playhead) return true; return !this.#buffered && this.#covered(timestamp); } @@ -270,6 +277,10 @@ export class AudioBuffer { /** How many times the ring has run dry mid-playback, cumulative. */ readonly underruns: Getter = this.#underruns; + readonly #skips = new Signal(0); + /** How many times the ring skipped ahead, dropping queued audio, cumulative. */ + readonly skips: Getter = this.#skips; + readonly #silent = new Signal(false); /** Whether playback is at a declared end, reached or announced before any audio, and nothing has played since. */ readonly silent: Getter = this.#silent; @@ -326,6 +337,7 @@ export class AudioBuffer { this.#timestamp.set(data.timestamp); this.#stalled.set(data.stalled); this.#underruns.set(data.underruns); + this.#skips.set(data.skips); this.#silent.set(data.silent); if (data.epoch === this.#epoch) { this.#backpressure.observe(data.anchored ? data.timestamp : undefined, data.stalled); @@ -354,7 +366,10 @@ export class AudioBuffer { this.#backpressure.dropped(); } - /** Insert audio samples at the given timestamp, transferring their buffers. Handles out-of-order writes. */ + /** + * Insert audio samples at the given timestamp, transferring their buffers. A write may land out of + * order, but samples behind what live playback already staged, up to 30 ms past the playhead, are dropped. + */ insert(timestamp: Micro, data: Float32Array[]): void { const end = (timestamp + Micro.fromSecond(((data[0]?.length ?? 0) / this.rate) as Second)) as Micro; this.#backpressure.inserted(timestamp, end); diff --git a/js/watch/src/audio/decoder.test.ts b/js/watch/src/audio/decoder.test.ts index 8e82ce475ef..8a48cfe6954 100644 --- a/js/watch/src/audio/decoder.test.ts +++ b/js/watch/src/audio/decoder.test.ts @@ -97,6 +97,9 @@ class FakeWorklet { get length(): number { return this.ring?.length ?? 0; } + get skips(): number { + return this.ring?.skips ?? 0; + } // Render one quantum and report the ring's state back to the main thread. read(output: Float32Array[]): number { const ring = this.ring; @@ -108,6 +111,7 @@ class FakeWorklet { stalled: ring.stalled, anchored: ring.anchored, underruns: ring.underruns, + skips: ring.skips, silent: ring.silent, epoch: this.#epoch, }; @@ -1357,7 +1361,7 @@ describe("Decoder across a rename", () => { const delay = playback.sync.out.delay.peek(); expect(delay).toBeGreaterThan(Milli(200)); const ring = playback.ring(); - const before = { ...playback.counts, buffered: ring?.length ?? 0 }; + const before = { silent: playback.counts.silent, skips: ring?.skips ?? 0 }; let lowest = delay; const watch = new Effect(); watch.subscribe(playback.sync.out.delay, (value) => { @@ -1368,10 +1372,8 @@ describe("Decoder across a rename", () => { await playback.run(26_000); watch.close(); - const { inserted, played, silent } = playback.counts; - const buffered = playback.ring()?.length ?? 0; - const skipped = inserted - before.inserted - (played - before.played) - (buffered - before.buffered); - expect({ skipped, silent: silent - before.silent }).toEqual({ skipped: 0, silent: 0 }); + const skips = (playback.ring()?.skips ?? 0) - before.skips; + expect({ skips, silent: playback.counts.silent - before.silent }).toEqual({ skips: 0, silent: 0 }); expect(playback.ring()).toBe(ring); expect(lowest).toBeGreaterThanOrEqual(delay); } finally { @@ -1715,13 +1717,17 @@ for (const cmaf of [false, true]) { (buffered ? [0, 20, 40, 60, 80] : [140, 160]).map((ms) => start + ms * 1000), ); - // A running live ring sheds old audio when it gets ahead; a buffered ring plays it all. + // A running live ring sheds old audio and lands on its floor once it reaches the fast + // accelerate threshold, four times a 100 ms floor plus its 20 ms band; a buffered ring + // plays it all. const ring = worklets.at(-1)?.ring; if (!ring) throw new Error("no running audio ring"); ring.reset(); const samples = [new Float32Array(4800), new Float32Array(4800)]; - for (const timestamp of [0, 100_000, 200_000]) ring.write(Micro(timestamp), samples); - expect(ring.length).toBe(samples[0].length * (buffered ? 3 : 1)); + for (let timestamp = 0; timestamp < 500_000; timestamp += 100_000) + ring.write(Micro(timestamp), samples); + expect(ring.length).toBe(samples[0].length * (buffered ? 5 : 1)); + expect(ring.skips).toBe(buffered ? 0 : 1); } finally { playback.close(); await microtasks(); diff --git a/js/watch/src/audio/decoder.ts b/js/watch/src/audio/decoder.ts index 748ca269d7c..c32c642412c 100644 --- a/js/watch/src/audio/decoder.ts +++ b/js/watch/src/audio/decoder.ts @@ -98,6 +98,8 @@ export interface Stats { bytesReceived: number; /** How many times playback ran dry mid-stream; a rising count means the delay is too low. */ underruns: number; + /** How many times playback skipped ahead, dropping buffered audio rather than playing it faster. */ + skips: number; /** Frames that arrived after their playout time. */ late: number; /** Renditions that failed to decode. */ @@ -111,7 +113,7 @@ export interface Stats { } // The running totals before anything is counted. -const EMPTY: Stats = { bytesReceived: 0, underruns: 0, late: 0, errors: 0, stalls: 0, stalled: Milli.zero }; +const EMPTY: Stats = { bytesReceived: 0, underruns: 0, skips: 0, late: 0, errors: 0, stalls: 0, stalled: Milli.zero }; // What a decode attempt reads, and where its counts go. type Attempt = { config: DecoderConfig; failure: Signal; path: Moq.Path.Valid; key: Key }; @@ -424,6 +426,14 @@ export class Decoder { const key = phase.kind === "running" ? phase.key : undefined; this.#count(key, { underruns: added }, (stats) => ({ underruns: stats.underruns + added })); }); + let skipped = 0; + effect.run((inner) => { + const skips = inner.get(ring.skips); + const added = skips - skipped; + skipped = skips; + if (added <= 0) return; + this.#count(undefined, {}, (stats) => ({ skips: stats.skips + added })); + }); effect.set(this.#out.root, worklet); }); diff --git a/js/watch/src/audio/playout.test.ts b/js/watch/src/audio/playout.test.ts index 8508b3180ba..6694c6f246a 100644 --- a/js/watch/src/audio/playout.test.ts +++ b/js/watch/src/audio/playout.test.ts @@ -136,8 +136,10 @@ function build(latencyMs: number): AudioRingBuffer { interface Result { /** Quanta rendered short of a full block after the warmup, i.e. audible underruns. */ underruns: number; - /** Samples inserted after the warmup that were never played: skipped or overflow-dropped. */ - skipped: number; + /** Times the ring re-anchored after the warmup, dropping buffered audio rather than playing it. */ + skips: number; + /** Samples time stretch added or removed after the warmup: output against the media it covered. */ + stretched: number; } /** @@ -148,40 +150,40 @@ function replay(ring: AudioRingBuffer, t: Arrival[], warmupMs: number): Result { const step = (QUANTUM / RATE) * 1000; const output = [new Float32Array(QUANTUM)]; const end = t[t.length - 1].arrival; + const position = () => (ring.timestamp * RATE) / 1_000_000; let next = 0; - let played = 0; - let inserted = 0; let short = 0; + let skips = 0; + let stretched = 0; let started = false; - let mark: { played: number; inserted: number; buffered: number; short: number } | undefined; for (let now = 0; now < end; now += step) { + const playhead = position(); + const skipped = ring.skips; while (next < t.length && t[next].arrival <= now) { ring.write(Micro.fromMilli(t[next].media as Milli), [new Float32Array(CHUNK).fill(0.5)]); - inserted += CHUNK; next++; } const count = ring.read(output); - played += count; + if (now < warmupMs) { + if (count > 0) started = true; + continue; + } if (started && count < QUANTUM) short++; if (count > 0) started = true; - - if (now >= warmupMs && !mark) mark = { played, inserted, buffered: ring.length, short }; + if (ring.skips > skipped) skips += ring.skips - skipped; + else stretched += Math.abs(position() - playhead - count); } - const from = mark ?? { played, inserted, buffered: ring.length, short }; - return { - underruns: short - from.short, - skipped: inserted - from.inserted - (played - from.played) - (ring.length - from.buffered), - }; + return { underruns: short, skips, stretched: Math.round(stretched) }; } describe("ring replay", () => { it("plays an evenly paced sender without underruns or skips", async () => { const t = trace(600, 1, 10); - expect(replay(build(await settled(t)), t, 2000)).toEqual({ underruns: 0, skipped: 0 }); + expect(replay(build(await settled(t)), t, 2000)).toMatchObject({ underruns: 0, skips: 0 }); }); it("plays a bursty sender at the measured target", async () => { @@ -189,7 +191,7 @@ describe("ring replay", () => { const t = trace(600, 3, 5); const measured = await settled(t); expect(measured).toBeGreaterThan(2 * CHUNK_MS); - expect(replay(build(measured), t, 2000)).toEqual({ underruns: 0, skipped: 0 }); + expect(replay(build(measured), t, 2000)).toMatchObject({ underruns: 0, skips: 0 }); }); it("keeps a five frame flush span playing", async () => { @@ -197,16 +199,21 @@ describe("ring replay", () => { // slack above it absorbs the peak, so the flush plays instead of being cut down to the // target on arrival. const t = trace(600, 5, 5); - expect(replay(build(await settled(t)), t, 2000)).toEqual({ underruns: 0, skipped: 0 }); + expect(replay(build(await settled(t)), t, 2000)).toMatchObject({ underruns: 0, skips: 0 }); }); - it("underruns constantly when the target ignores the arrival spread", () => { + it("still underruns when the target ignores the arrival spread, but drops nothing", () => { // 46ms is what the round-trip formula produced on the connection this was measured on: it // describes the network and says nothing about a sender that flushes five frames at once. + // Every flush's trough falls below it, and the ring never slows down to cover one, since that + // would only settle audio behind the clock, so it runs dry on most flushes; accelerate brings + // each peak back down rather than skipping it. const t = trace(600, 5, 5); const result = replay(build(46), t, 2000); - expect(result.underruns).toBeGreaterThan(100); - expect(result.skipped).toBeGreaterThan(10 * CHUNK); + const flushes = (t.length - 2000 / CHUNK_MS) / 5; + expect(result.underruns).toBeGreaterThan(flushes / 2); + expect(result.skips).toBe(0); + expect(result.stretched).toBeGreaterThan(0); }); }); diff --git a/js/watch/src/audio/render-worklet.test.ts b/js/watch/src/audio/render-worklet.test.ts index 044e1e7c960..f50a003f05f 100644 --- a/js/watch/src/audio/render-worklet.test.ts +++ b/js/watch/src/audio/render-worklet.test.ts @@ -85,6 +85,7 @@ describe("render dispatch", () => { stalled: false, anchored: true, underruns: 0, + skips: 0, silent: false, }, ]); @@ -113,6 +114,7 @@ describe("render dispatch", () => { stalled: false, anchored: true, underruns: 0, + skips: 0, silent: false, }, ]); diff --git a/js/watch/src/audio/render-worklet.ts b/js/watch/src/audio/render-worklet.ts index 230dba0e782..e7c23e255b4 100644 --- a/js/watch/src/audio/render-worklet.ts +++ b/js/watch/src/audio/render-worklet.ts @@ -100,6 +100,7 @@ class Render extends AudioWorkletProcessor { stalled: ring.stalled, anchored: ring.anchored, underruns: ring.underruns, + skips: ring.skips, silent: ring.silent, epoch: this.#epoch, }; diff --git a/js/watch/src/audio/render.ts b/js/watch/src/audio/render.ts index f60f1612000..b3ac4dde878 100644 --- a/js/watch/src/audio/render.ts +++ b/js/watch/src/audio/render.ts @@ -62,6 +62,8 @@ export interface State { anchored: boolean; // How many times the ring has run dry mid-playback, cumulative. underruns: number; + // How many times the ring skipped ahead, dropping queued audio, cumulative. + skips: number; // Whether playback is at a declared end, reached or announced before any audio, and nothing has played since. silent: boolean; // The epoch of the last reset or park the worklet applied. diff --git a/js/watch/src/audio/replay.test.ts b/js/watch/src/audio/replay.test.ts index 44e110a609f..6db5da84f7e 100644 --- a/js/watch/src/audio/replay.test.ts +++ b/js/watch/src/audio/replay.test.ts @@ -136,6 +136,61 @@ describe("replay", () => { }); }); +/** + * How far the playhead trails the shared clock video plays from, in ms, for each quantum played once + * the first `settle` ms have passed, with the underruns the ring counted by the end. + */ +async function lag(trace: Arrival[], options: Options, settle: number) { + const lags: number[] = []; + let underruns = 0; + for await (const quantum of replay(trace, options)) { + if (quantum.at >= settle && !quantum.stalled && quantum.clock !== undefined) { + lags.push(quantum.clock - quantum.timestamp); + } + underruns = quantum.underruns; + } + lags.sort((a, b) => a - b); + return { lags, median: lags[lags.length >> 1], underruns }; +} + +// Audio and video keep separate clocks that align once the delay covers the arrivals. Settling audio +// later than the clock would trade lip sync for fewer underruns, so these hold the line: at steady +// state the playhead rests on or just ahead of the clock, and a target short of the spread underruns +// rather than drifting behind it. +describe("replay against the shared clock", () => { + let warn: Mock; + beforeEach(() => { + warn = spyOn(console, "warn").mockImplementation(() => {}); + }); + afterEach(() => warn.mockRestore()); + + // The ring rests one frame ahead of the newest arrival's playout, as the trim that preceded the + // time stretch left it, so audio leads the clock by up to that frame and never trails it. + for (const [name, lateness] of [ + ["evenly paced", () => 30], + ["flushed five frames at a time", (i: number) => 30 + (4 - (i % 5)) * 20], + ] as const) { + it(`plays ${name} frames on the clock at auto, never behind it`, async () => { + const options: Options = { rate: RATE, config: CONFIG, delay: "auto", duration: observed(60) }; + const { lags } = await lag(paced(60, lateness), options, 30_000); + expect(lags.length).toBeGreaterThan(0); + expect(lags[0]).toBeGreaterThanOrEqual(-25); + expect(lags.at(-1)).toBeLessThanOrEqual(0); + }); + } + + it("underruns a target short of the spread rather than settling behind the clock", async () => { + // 80 ms of spread against a 40 ms target: every flush runs the ring dry. + const trace = paced(60, (i) => 30 + (4 - (i % 5)) * 20); + const options: Options = { rate: RATE, config: CONFIG, delay: Milli(40), duration: observed(60) }; + const { lags, median, underruns } = await lag(trace, options, 30_000); + expect(underruns).toBeGreaterThanOrEqual(trace.length / 5); + // Each refill catches back up within the frame after it, never by a whole flush. + expect(median).toBeLessThanOrEqual(20); + expect(lags.at(-1)).toBeLessThan(40); + }); +}); + describe("replay at auto", () => { /** The delay the replay resolved once `seconds` of {@link paced} frames played out. */ async function settled(seconds: number, lateness: (i: number) => number): Promise { diff --git a/js/watch/src/audio/replay.ts b/js/watch/src/audio/replay.ts index 8e9cc3f172a..6f7ace49450 100644 --- a/js/watch/src/audio/replay.ts +++ b/js/watch/src/audio/replay.ts @@ -81,10 +81,14 @@ export interface Quantum { stalled: boolean; /** The playhead after this quantum, on the media clock, in ms. */ timestamp: number; + /** Where the shared clock puts video as of this quantum (`Sync.now()`), on the media clock, in ms. */ + clock?: number; /** The delay `Sync` resolved as of this quantum, in ms: what the ring and the decode gate hold. */ delay: number; /** The ring's cumulative underruns, as the player's stats count them: each run-dry and each short quantum. */ underruns: number; + /** The ring's cumulative skips, as the player's stats count them: each jump that dropped queued audio. */ + skips: number; } /** Samples as the milliseconds `AudioBuffer` posts to the worklet. */ @@ -277,8 +281,10 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato output, stalled: ring.stalled, timestamp: Milli.fromMicro(ring.timestamp), + clock: sync.now(), delay, underruns: ring.underruns, + skips: ring.skips, }; } } finally { diff --git a/js/watch/src/audio/ring-buffer.test.ts b/js/watch/src/audio/ring-buffer.test.ts index 238dd6144d5..4bb25240372 100644 --- a/js/watch/src/audio/ring-buffer.test.ts +++ b/js/watch/src/audio/ring-buffer.test.ts @@ -108,8 +108,10 @@ describe("writing data", () => { } }); + // These drain the ring below its floor without refilling it, which a live ring stretches, so they + // run buffered, which plays exactly what the ring holds. it("should handle late-arriving samples (out-of-order writes)", () => { - const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100 as Milli }); + const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100 as Milli, buffered: true }); // Fill buffer to exit stalled mode write(buffer, 0 as Milli, 100, { channels: 1, value: 0.0 }); @@ -146,7 +148,7 @@ describe("writing data", () => { }); it("should discard samples that are too old", () => { - const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100 as Milli }); + const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100 as Milli, buffered: true }); // Exit stalled mode by filling buffer write(buffer, 0 as Milli, 100, { channels: 2, value: 0.0 }); @@ -171,9 +173,10 @@ describe("writing data", () => { }); }); +// Buffered, for the same reason as the writes above. describe("reading data", () => { it("should read available data", () => { - const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100 as Milli }); + const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100 as Milli, buffered: true }); // Exit stalled mode by filling the buffer write(buffer, 0 as Milli, 100, { channels: 2, value: 0.0 }); @@ -211,7 +214,7 @@ describe("reading data", () => { }); it("should handle partial reads", () => { - const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100 as Milli }); + const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100 as Milli, buffered: true }); // Exit stalled mode by filling the buffer write(buffer, 0 as Milli, 100, { channels: 2, value: 0.0 }); @@ -269,47 +272,23 @@ describe("stall behavior", () => { describe("ring buffer wrapping", () => { it("should wrap around when buffer is full", () => { - // 100 sample target, so 200 samples of capacity. - const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100 as Milli }); - - // Fill to the target - write(buffer, 0 as Milli, 100, { channels: 1, value: 1.0 }); - expect(buffer.stalled).toBe(false); - - // Read 50 samples to make room (readIndex at 50) - const output1 = read(buffer, 50, 1); - expect(output1[0].length).toBe(50); - - // Write 50 more samples at timestamp 100 (fills from sample 100-149) - write(buffer, 100 as Milli, 50, { channels: 1, value: 2.0 }); - - // Now we have 100 samples available (50-149) - expect(buffer.length).toBe(100); - - // One chunk above the target is tolerated rather than dropped (50-199). - write(buffer, 150 as Milli, 50, { channels: 1, value: 3.0 }); - expect(buffer.length).toBe(150); - - // Past the band: drop back to the target, wrapping abs 200-249 onto slots 0-49. - write(buffer, 200 as Milli, 50, { channels: 1, value: 4.0 }); - expect(buffer.length).toBe(100); + // A 1000-sample ring, buffered so it plays through everything it holds. + const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100 as Milli, buffered: true }); + write(buffer, 0 as Milli, 600, { channels: 1, value: 1.0 }); + expect(read(buffer, 500, 1)[0].length).toBe(500); - // Read all 100 samples: abs 150-199 = 3.0, abs 200-249 = 4.0. - const output2 = read(buffer, 100, 1); - expect(output2[0].length).toBe(100); + // Abs 600-1199 wraps onto slots 600-999 and 0-199. + write(buffer, 600 as Milli, 600, { channels: 1, value: 2.0 }); + expect(buffer.length).toBe(700); - for (let i = 0; i < 50; i++) { - expect(output2[0][i]).toBe(3.0); - } - for (let i = 50; i < 100; i++) { - expect(output2[0][i]).toBe(4.0); - } + const [output] = read(buffer, 700, 1); + expect([...output]).toEqual([...Array(100).fill(1), ...Array(600).fill(2)]); }); }); describe("multi-channel handling", () => { it("should handle stereo data correctly", () => { - const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100 as Milli }); + const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100 as Milli, buffered: true }); // Exit stalled mode by filling buffer write(buffer, 0 as Milli, 100, { channels: 2, value: 0.5 }); @@ -610,12 +589,13 @@ describe("declared end", () => { expect(read(buffer, 20, 1)[0].length).toBe(20); buffer.next(); - write(buffer, 50 as Milli, 40, { channels: 1, value: 2 }); + // One 20 ms frame, so the ring stays within a frame of its floor and plays it as received. + write(buffer, 50 as Milli, 20, { channels: 1, value: 2 }); // The rest of the tail, the short gap to the next utterance, then the utterance itself. - const [played] = read(buffer, 70, 1); + const [played] = read(buffer, 50, 1); expect([...played.slice(0, 20)]).toEqual(Array(20).fill(1)); expect([...played.slice(20, 30)]).toEqual(Array(10).fill(0)); - expect([...played.slice(30)]).toEqual(Array(40).fill(2)); + expect([...played.slice(30)]).toEqual(Array(20).fill(2)); // The gap is the declared silence, for the reader to fill with comfort noise. expect(buffer.quiet).toEqual([20, 30]); expect(buffer.underruns).toBe(0); @@ -793,7 +773,8 @@ describe("latency increase", () => { // Video holds a deeper floor on its own, so audio has to park for the difference or it runs // ahead. Flushing instead cost the whole floor in silence for every rise, however small. it("parks a playing ring until the deeper floor refills, keeping what it buffered", () => { - const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100 as Milli }); + // Buffered, so it plays its floor down without the expand a live ring stretches it with. + const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100 as Milli, buffered: true }); // Fill to the floor so playback starts, then play half of it. write(buffer, 0 as Milli, 100, { channels: 1, value: 1.0 }); @@ -838,3 +819,218 @@ describe("latency increase", () => { expect(read(buffer, 1, 1)[0][0]).toBe(1.0); }); }); + +describe("time stretch", () => { + it("truncate replaces the unplayed part of a staged window", () => { + // Live, so the first read stages a whole 30 ms window though it plays only 128 samples. + const buffer = new AudioRingBuffer({ rate: 48000, channels: 1, latency: 100 as Milli }); + write(buffer, 0 as Milli, 4800, { channels: 1, value: 0.1 }); + read(buffer, 128, 1); + + buffer.truncate(Micro.fromMilli(10 as Milli)); + write(buffer, 10 as Milli, 960, { channels: 1, value: 0.9 }); + + // The predecessor plays to the truncation, then the successor in full. + const out = read(buffer, 480 - 128 + 960, 1)[0]; + expect(out.length).toBe(480 - 128 + 960); + expect(out[480 - 128 - 1]).toBeCloseTo(0.1, 5); + expect(out[480 - 128]).toBeCloseTo(0.9, 5); + expect(buffer.length).toBe(0); + }); + + // 48 kHz: 30 ms windows of 1440 frames, and a constant signal stretches by its shortest pitch + // period, 2.5 ms or 120 frames. + const RATE = 48000; + const QUANTUM = 128; + const live = (latency: number) => new AudioRingBuffer({ rate: RATE, channels: 1, latency: latency as Milli }); + const frames = (ms: number) => (RATE * ms) / 1000; + + // Slowing down only ever settles audio behind the shared clock, so a ring short of its floor plays + // what it holds at its own pace and runs dry: the underrun is the cost of arrivals running late. + it("never expands a ring short of its floor, so it plays only what it holds before running dry", () => { + const buffer = live(100); + write(buffer, 0 as Milli, frames(100), { value: 0.5 }); + let played = 0; + for (;;) { + const [output] = read(buffer, QUANTUM, 1); + played += output.length; + if (output.length < QUANTUM) break; + } + expect(played).toBe(frames(100)); + expect(buffer.underruns).toBe(1); + }); + + it("drains the tail before a declared end as received, since running dry there is silence", () => { + const buffer = live(100); + write(buffer, 0 as Milli, frames(100), { value: 0.5 }); + buffer.end(Micro.fromMilli(100 as Milli)); + let played = 0; + for (;;) { + const [output] = read(buffer, QUANTUM, 1); + played += output.length; + if (output.length < QUANTUM) break; + } + expect(played).toBe(frames(100)); + expect(buffer.underruns).toBe(0); + expect(buffer.silent).toBe(true); + }); + + // Fed `burst` ms of 20 ms frames at a time, at the real-time rate, so only the stretch can move the + // ring's depth. Returns the depth right after each burst lands, where it reads the clock. + async function converge(buffer: AudioRingBuffer, start: number, seconds: number, burst = 20) { + const timestamps: number[] = []; + const peaks: number[] = []; + let written = start; + for (let rendered = 0; rendered < frames(seconds * 1000); rendered += QUANTUM) { + if (written < start + rendered + QUANTUM) { + for (const end = written + frames(burst); written < end; written += frames(20)) { + write(buffer, ((written * 1000) / RATE) as Milli, frames(20), { value: 0.5 }); + } + peaks.push(buffer.length); + } + const [output] = read(buffer, QUANTUM, 1); + if (burst === 20) expect(output.length).toBe(QUANTUM); + timestamps.push(Milli.fromMicro(buffer.timestamp)); + } + return { timestamps, peaks, written }; + } + + // A constant signal accelerates by one 2.5 ms pitch period at a time, and only by a period that fits + // in the excess, so the ring rests within one period of its floor and never under it. + const PERIOD = frames(2.5); + const settled = (peaks: number[], floor: number) => { + for (const peak of peaks.slice(-20)) { + expect(peak).toBeGreaterThanOrEqual(frames(floor)); + expect(peak).toBeLessThan(frames(floor) + PERIOD); + } + }; + + it("accelerates a ring above its floor back onto it, dropping nothing", async () => { + const buffer = live(100); + write(buffer, 0 as Milli, frames(300), { value: 0.5 }); + const { timestamps, peaks, written } = await converge(buffer, frames(300), 8); + + settled(peaks, 100); + // Every sample written either played or is still buffered, so nothing was skipped. + expect(buffer.skips).toBe(0); + expect(Math.round((Milli.fromMicro(buffer.timestamp) * RATE) / 1000) + buffer.length).toBeGreaterThan( + written - frames(30), + ); + for (let i = 1; i < timestamps.length; i++) { + const step = timestamps[i] - timestamps[i - 1]; + expect(step).toBeGreaterThan(0); + expect(step).toBeLessThanOrEqual((2 * QUANTUM * 1000) / RATE + 1e-9); + } + expect(buffer.underruns).toBe(0); + }); + + // Five 20 ms frames flushed at once, every 100 ms. Between bursts the ring drains a whole burst + // below its floor, which is arrivals running late rather than audio ahead of the clock, so steering + // on that depth would settle a burst behind it. + it("rests on its floor right after each burst, never a burst above it", async () => { + const buffer = live(120); + write(buffer, 0 as Milli, frames(120), { value: 0.5 }); + const { peaks } = await converge(buffer, frames(120), 8, 100); + settled(peaks, 120); + expect(buffer.underruns).toBe(0); + }); + + // A floor short of the arrival spread underruns, and that is the cost of the floor: the ring must + // not grow its depth to stop underrunning, which only plays everything later. + it("underruns every burst that outruns its floor rather than settling later", async () => { + const buffer = live(60); + write(buffer, 0 as Milli, frames(60), { value: 0.5 }); + const { peaks } = await converge(buffer, frames(60), 8, 100); + // Each burst lands on an empty ring, never on audio left from the one before. + for (const peak of peaks.slice(-20)) expect(peak).toBe(frames(100)); + expect(buffer.underruns).toBeGreaterThanOrEqual(70); + }); + + it("fast-accelerates a ring four times past the band, even straight after a stretch", async () => { + const buffer = live(100); + write(buffer, 0 as Milli, frames(600), { value: 0.5 }); + await converge(buffer, frames(600), 1); + // Fast accelerate removes up to 15 ms of every 30 ms window until the ring is under 480 ms. + expect(buffer.length).toBeLessThan(frames(480)); + }); + + it("drops the staged window with the audio a re-anchor skips", () => { + const buffer = live(100); + write(buffer, 0 as Milli, frames(100), { value: 0.25 }); + read(buffer, QUANTUM, 1); + + // 400 ms more crosses the 480 ms fast threshold with a 30 ms window still staged. + write(buffer, 100 as Milli, frames(400), { value: 0.5 }); + expect(buffer.skips).toBe(1); + expect(buffer.length).toBe(frames(100)); + expect(Milli.fromMicro(buffer.timestamp)).toBeCloseTo(400, 6); + const [output] = read(buffer, QUANTUM, 1); + expect(output.length).toBe(QUANTUM); + expect(output.every((sample) => sample === 0.5)).toBe(true); + }); + + it("counts a re-anchor that drops only the staged window as a skip", () => { + const buffer = live(100); + // Four whole 30 ms windows, so once the playhead passes 90 ms the ring is empty and only the + // last window is left, staged. + write(buffer, 0 as Milli, frames(120), { value: 0.25 }); + while (Milli.fromMicro(buffer.timestamp) <= 90) read(buffer, QUANTUM, 1); + expect(buffer.length).toBeGreaterThan(0); + expect(buffer.length).toBeLessThan(frames(30)); + + write(buffer, 120 as Milli, frames(500), { value: 0.5 }); + expect(buffer.skips).toBe(1); + expect(buffer.length).toBe(frames(100)); + }); + + // The ring past the read index holds less than the floor, but the staged window lifts the total a + // chunk above it, so the catch-up over the declared silence must count the staged output too. + it("catches up over a declared silence without rewinding behind a staged window", () => { + const buffer = live(100); + write(buffer, 0 as Milli, frames(120), { value: 0.25 }); + while (Milli.fromMicro(buffer.timestamp) <= 90) read(buffer, QUANTUM, 1); + const staged = buffer.length; + expect(staged).toBeGreaterThan(frames(20)); + + // The next utterance lands 80 ms after the last, 10 ms long: 90 ms of ring behind the window. + buffer.next(); + const playhead = buffer.timestamp; + write(buffer, 200 as Milli, frames(10), { value: 0.5 }); + expect(buffer.timestamp).toBeGreaterThanOrEqual(playhead); + expect(buffer.length).toBe(frames(100)); + expect(buffer.skips).toBe(0); + + // The staged window plays out once, then the rest of the silence, then the utterance. + const [output] = read(buffer, frames(100), 1); + expect(output.length).toBe(frames(100)); + const silence = output.findIndex((sample) => sample === 0); + expect(silence).toBe(staged); + expect(output.slice(silence).every((sample) => sample === 0 || sample === 0.5)).toBe(true); + expect(output.at(-1)).toBe(0.5); + }); + + // Past NetEq's fast threshold, four times the floor plus its band, or three quarters of the + // headroom above the floor, whichever is lower, stretching back would take seconds, so the ring + // re-anchors on its floor as moq-audio does. With 1000 ms of capacity, a 100 ms floor reaches the + // first at 480 ms, and a 300 ms floor the second at 825 ms. + for (const [floor, threshold] of [ + [100, 480], + [300, 825], + ]) { + it(`re-anchors a ${floor} ms floor only once the ring holds ${threshold} ms`, () => { + const buffer = live(floor); + write(buffer, 0 as Milli, frames(floor), { value: 0.5 }); + read(buffer, QUANTUM, 1); + const playhead = buffer.timestamp; + + write(buffer, floor as Milli, frames(threshold - 10 - floor), { value: 0.5 }); + expect(buffer.timestamp).toBe(playhead); + expect(buffer.skips).toBe(0); + + write(buffer, (threshold - 10) as Milli, frames(20), { value: 0.5 }); + expect(buffer.timestamp).toBeGreaterThan(playhead); + expect(buffer.length).toBeLessThan(frames(floor) + frames(30)); + expect(buffer.skips).toBe(1); + }); + } +}); diff --git a/js/watch/src/audio/ring-buffer.ts b/js/watch/src/audio/ring-buffer.ts index 3afd723d588..7638b44ae37 100644 --- a/js/watch/src/audio/ring-buffer.ts +++ b/js/watch/src/audio/ring-buffer.ts @@ -1,5 +1,6 @@ import { Micro, type Milli, Second } from "@moq/clock"; import { ringCapacity } from "./latency"; +import { Stretch } from "./stretch"; export class AudioRingBuffer { #buffer: Float32Array[]; @@ -12,6 +13,7 @@ export class AudioRingBuffer { // Held stalled for the part of the target above the floor, until the next floor update. #parked = false; #underruns = 0; + #skips = 0; // Where the publisher declared its audio ends, as a sample index, until the next reset. #end?: number; @@ -33,12 +35,24 @@ export class AudioRingBuffer { // silence, which is no underrun and survives the reset that re-anchors the next utterance. #silent = false; - // Samples in the most recent write, i.e. one decoded chunk. The overflow band tolerates this - // much above the target before dropping, so a ring sitting exactly on the target doesn't drop - // audio the moment the next chunk lands. The most recent write rather than a running maximum: - // one oversized decode would otherwise widen the band for the life of the ring. + // Samples in the most recent write, i.e. one decoded chunk. Catching up over a declared silence + // tolerates this much above the target, so a ring sitting exactly on the target doesn't skip the + // moment the next chunk lands. The most recent write rather than a running maximum: one oversized + // decode would otherwise widen the band for the life of the ring. #chunk = 0; + // Live playback converges on the floor by accelerating each window of received PCM on its way out. + // The read index is past the staged window, which spans `#window` samples of the ring. + readonly #stretch: Stretch; + #window = 0; + // The ring's length right after the latest write, less what the stretch has removed since. That + // is where depth reads the clock: a frame has just arrived. Between arrivals the ring drains + // toward the next one, which says nothing about how far behind the playhead is, so steering on + // the current length would expand into every gap and settle behind the shared clock. + #peak = 0; + // Whether the staged window lies in a declared silence, which plays unstretched as comfort noise. + #windowQuiet = false; + // Buffered mode: play through everything buffered without skipping ahead. readonly #buffered: boolean; // Un-stall threshold in samples (how much to buffer before playback starts). @@ -74,6 +88,7 @@ export class AudioRingBuffer { for (let i = 0; i < this.channels; i++) { this.#buffer[i] = new Float32Array(capacity); } + this.#stretch = new Stretch(this.rate, this.channels); } get stalled(): boolean { @@ -85,6 +100,11 @@ export class AudioRingBuffer { return this.#underruns; } + /** How many times the ring skipped ahead, dropping queued audio rather than playing it. */ + get skips(): number { + return this.#skips; + } + /** Whether playback is at a declared end, reached or announced before any audio, and nothing has played since. */ get silent(): boolean { return this.#silent; @@ -98,8 +118,11 @@ export class AudioRingBuffer { return this.#quiet; } + /** The playhead, with the staged window's unplayed output mapped back onto the samples it came from. */ get timestamp(): Micro { - return Micro.fromSecond((this.#readIndex / this.rate) as Second); + const stretch = this.#stretch; + const staged = stretch.length > 0 ? (stretch.available * this.#window) / stretch.length : 0; + return Micro.fromSecond(((this.#readIndex - staged) / this.rate) as Second); } /** Whether an insert has anchored the indices since creation or the last reset. */ @@ -107,8 +130,9 @@ export class AudioRingBuffer { return this.#anchored; } + /** Samples left to play: the ring past the read index, plus the stretched output staged from it. */ get length(): number { - return this.#writeIndex - this.#readIndex; + return this.#writeIndex - this.#readIndex + this.#stretch.available; } get capacity(): number { @@ -161,7 +185,8 @@ export class AudioRingBuffer { this.#anchored = true; } - // Ignore samples that are too old (before the read index) + // Ignore samples behind the read index, which runs ahead of the playhead by the staged window: + // that part is already staged to play, so a late write there is dropped too. let offset = this.#readIndex - start; if (offset > samples) { // All samples are too old, ignore them @@ -186,20 +211,37 @@ export class AudioRingBuffer { if (start > this.#writeIndex) this.#pauses.push({ start: this.#writeIndex, end: start }); } - // Bound the ring. While playing, drop the oldest once it holds a whole chunk more than the - // target and land back on the target: frames arrive one chunk at a time, so a ring sitting - // exactly on the target is a chunk above it the moment the next one lands, and dropping on - // that overshoot discards audio on every single write. While stalled the reader is not - // consuming, so only the hard capacity applies; the band would throw away the very audio the - // refill is accumulating. Buffered mode plays through everything, so it is capacity-bound too. + // Bound the ring. While playing live, the read path accelerates the ring back onto its floor, so + // dropping the oldest audio and landing on the floor is an emergency re-anchor, as moq-audio + // does: at NetEq's fast threshold, four times the floor plus 20 ms, or once the ring holds three + // quarters of its headroom above the floor, whichever comes first. While stalled the + // reader is not consuming, so only the hard capacity applies; dropping would throw away the very + // audio the refill is accumulating. Buffered mode plays through everything, so it is + // capacity-bound too. const playing = !this.#stalled && !this.#buffered; - // Until the tail before a declared silence has played, only the capacity bounds the ring; the - // band then catches up by skipping the silence. + // Until the tail before a declared silence has played, only the capacity bounds the ring; then + // a ring a chunk above its floor catches up by skipping the silence, which drops no audio. const band = playing && (this.#tail === undefined || this.#readIndex >= this.#tail); - const limit = band ? Math.min(this.#latencySamples + this.#chunk, this.capacity) : this.capacity; - if (end - this.#readIndex > limit) { - this.#readIndex = end - (band ? Math.min(this.#latencySamples, this.capacity) : this.capacity); + this.#prune(); + const pause = this.#pauses[0]; + const latency = this.#latencySamples; + const queued = end - this.#readIndex + this.#stretch.available; + if (end - this.#readIndex > this.capacity) { + this.#skip(end - this.capacity); + } else if (band && pause && pause.start <= this.#readIndex && queued > latency + this.#chunk) { + // Skip only the silence past the queued floor, which the staged window counts toward, so the + // read index only moves forward and never re-stages PCM the window already holds. + this.#readIndex = Math.min(pause.end, this.#readIndex + queued - latency); this.#prune(); + } else if ( + band && + queued >= + Math.min( + 4 * (latency + Math.floor((this.rate * 20) / 1000)), + latency + Math.floor(((this.capacity - latency) * 3) / 4), + ) + ) { + this.#skip(end - latency); } // Fill gaps with zeros if there's a discontinuity @@ -236,6 +278,7 @@ export class AudioRingBuffer { if (end > this.#writeIndex) { this.#writeIndex = end; } + this.#peak = this.length; // Start playback once we've buffered the latency target. This is the only way out of a // stall but a park, so an underrun mid-playback refills to the target before resuming rather @@ -254,11 +297,23 @@ export class AudioRingBuffer { truncate(timestamp: Micro): void { const target = Math.round(Second.fromMicro(timestamp) * this.rate); if (target >= this.#writeIndex) return; + const length = this.length; + // A staged window runs the read index ahead of the playhead. Unstage it so the successor + // replaces whatever of it is still unplayed, resuming the ring at the sample now playing. + const stretch = this.#stretch; + if (target < this.#readIndex && stretch.available > 0) { + const staged = (stretch.available * this.#window) / stretch.length; + const capacity = this.#buffer[0].length; + this.#readIndex = Math.max(Math.round(this.#readIndex - staged), this.#writeIndex - capacity); + stretch.clear(); + this.#window = 0; + } // Never retreat past the playhead: those samples are already due. this.#writeIndex = Math.max(target, this.#readIndex); this.#pauses = this.#pauses.filter((pause) => pause.start < this.#writeIndex); const last = this.#pauses.at(-1); if (last) last.end = Math.min(last.end, this.#writeIndex); + this.#peak = Math.max(0, this.#peak - (length - this.length)); } /** @@ -273,6 +328,21 @@ export class AudioRingBuffer { if (!this.#anchored || (this.#stalled && this.length <= 0 && this.#ended())) this.#silent = true; } + // Move the read index forward to `index`, dropping any queued audio it passes, staged included. + #skip(index: number): void { + if (index > this.#readIndex && this.length > 0) this.#skips++; + // The staged window lies behind the new index, so it is dropped with the rest, as moq-audio's + // trim discards staged output first. + const length = this.length; + if (index > this.#readIndex) { + this.#stretch.clear(); + this.#window = 0; + } + this.#readIndex = index; + this.#peak = Math.max(0, this.#peak - (length - this.length)); + this.#prune(); + } + // Forget the declared silences the playhead has passed. #prune(): void { const pauses = this.#pauses; @@ -318,15 +388,18 @@ export class AudioRingBuffer { this.#next = false; this.#tail = undefined; this.#pauses = []; + this.#stretch.clear(); + this.#window = 0; + this.#peak = 0; } read(output: Float32Array[]): number { if (output.length !== this.channels) throw new Error("wrong number of channels"); + for (const dst of output) if (dst.length !== output[0].length) throw new Error("mismatching number of samples"); this.#quiet = undefined; if (this.#stalled) return 0; - const samples = Math.min(this.#writeIndex - this.#readIndex, output[0].length); - if (samples <= 0) { + if (this.length <= 0) { // Ran dry mid-playback: re-stall so write() refills to the target before resuming. this.#stalled = true; if (this.#ended()) this.#silent = true; @@ -334,31 +407,16 @@ export class AudioRingBuffer { return 0; } - for (let channel = 0; channel < this.channels; channel++) { - const dst = output[channel]; - const src = this.#buffer[channel]; - - if (dst.length !== output[0].length) throw new Error("mismatching number of samples"); - - for (let i = 0; i < samples; i++) { - const readPos = (this.#readIndex + i) % src.length; - dst[i] = src[readPos]; - } + const stretch = this.#stretch; + let samples = 0; + while (samples < output[0].length) { + if (stretch.available === 0 && !this.#stage(output[0].length - samples)) break; + const from = samples; + samples += stretch.read(output, samples); + // A quantum overlaps at most one declared silence unless an utterance is shorter than it, + // which no codec frame is; the span then covers both silences and the sliver between them. + if (this.#windowQuiet) this.#quiet = [this.#quiet?.[0] ?? from, samples]; } - - // A quantum overlaps at most one declared silence unless an utterance is shorter than it, which - // no codec frame is; the span then covers both silences and the sliver between them. - this.#prune(); - const pauses = this.#pauses; - const after = this.#readIndex + samples; - const first = pauses[0]; - if (first && first.start < after) { - let last = first; - for (const pause of pauses) if (pause.start < after) last = pause; - this.#quiet = [Math.max(0, first.start - this.#readIndex), Math.min(samples, last.end - this.#readIndex)]; - } - - this.#readIndex += samples; this.#silent = false; // A short quantum ends in silence, so it counts as an underrun even if a chunk lands before @@ -376,4 +434,37 @@ export class AudioRingBuffer { return samples; } + + // Move the next window off the ring into the stretch, which accelerates a live ring onto its floor. + // A window stops at the edge of a declared silence, which plays unstretched, so the stretch never + // moves where one starts or ends. Whatever cannot stretch stages only the `need` this quantum + // reads, so a write still lands anywhere past the playhead. Returns false once the ring is empty. + #stage(need: number): boolean { + const buffered = this.#writeIndex - this.#readIndex; + if (buffered <= 0) return false; + + this.#prune(); + const pause = this.#pauses[0]; + const quiet = pause !== undefined && pause.start <= this.#readIndex; + // Audio before a declared end is a finite tail: it drains as received, since the next utterance + // re-anchors the ring anyway. + const end = this.#previous ?? this.#end; + const ending = end !== undefined && this.#readIndex < end; + const enabled = !quiet && !ending && !this.#buffered; + let frames = Math.min(buffered, this.#stretch.window, enabled ? buffered : need); + if (pause) frames = Math.min(frames, (quiet ? pause.end : pause.start) - this.#readIndex); + + for (let channel = 0; channel < this.channels; channel++) { + const src = this.#buffer[channel]; + const dst = this.#stretch.input[channel]; + for (let i = 0; i < frames; i++) dst[i] = src[(this.#readIndex + i) % src.length]; + } + const stretch = this.#stretch; + stretch.process(frames, this.#peak - this.#latencySamples, enabled); + this.#peak -= frames - stretch.length; + this.#readIndex += frames; + this.#window = frames; + this.#windowQuiet = quiet; + return true; + } } diff --git a/js/watch/src/audio/stretch.test.ts b/js/watch/src/audio/stretch.test.ts new file mode 100644 index 00000000000..9d6de528ad0 --- /dev/null +++ b/js/watch/src/audio/stretch.test.ts @@ -0,0 +1,119 @@ +import { expect, it } from "bun:test"; +import { readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { type Decision, Stretch } from "./stretch"; + +interface Vector { + name: string; + rate: number; + channels: number; + input: number[]; + buffered: number; + target: number; + cooldown: number; + /** As native playback decides: it also expands, which this player never does. */ + decision: Decision | "Expand"; + lag: number; + output: number[]; +} + +// Shared with moq-audio, so neither implementation is graded against the other. +const corpus = join( + dirname(fileURLToPath(import.meta.url)), + "../../../../doc/concept/audio-jitter/stretch/corpus.json", +); +const vectors = JSON.parse(readFileSync(corpus, "utf8")) as Vector[]; + +// The corpus is interleaved; the ring is planar. +const planar = (samples: number[], channels: number) => + Array.from({ length: channels }, (_, channel) => samples.filter((_, i) => i % channels === channel)); + +// The corpus decides as NetEq does, on the buffered depth against a band around the target. This +// player decides on the excess above the target alone and never expands, so the corpus grades its +// seams: an excess past the 20 ms band fast-accelerates, one inside it accelerates, and a ring at or +// short of its target, where NetEq holds or expands, plays its window unchanged. +const excess = (vector: Vector) => { + const band = Math.floor((vector.rate * 20) / 1000); + if (vector.decision === "Fast") return vector.buffered - vector.target; + if (vector.decision === "Accelerate") return band - 1; + return Math.min(0, vector.buffered - vector.target); +}; + +for (const vector of vectors) { + it(`stretches the corpus PCM: ${vector.name}`, () => { + const stretch = new Stretch(vector.rate, vector.channels); + const input = planar(vector.input, vector.channels); + input.forEach((samples, channel) => { + stretch.input[channel].set(samples); + }); + stretch.cooldown = vector.cooldown; + const expand = vector.decision === "Expand"; + expect(stretch.decide(excess(vector))).toBe(vector.decision === "Expand" ? "Hold" : vector.decision); + + stretch.process(stretch.window, excess(vector), true); + expect(stretch.window - stretch.available).toBe(expand ? 0 : vector.lag); + + const output = Array.from({ length: vector.channels }, () => new Float32Array(stretch.available)); + expect(stretch.read(output, 0)).toBe(output[0].length); + const expected = expand ? input : planar(vector.output, vector.channels); + for (let channel = 0; channel < vector.channels; channel++) { + expect(output[channel].length).toBe(expected[channel].length); + output[channel].forEach((sample, i) => { + expect(Math.abs(sample - expected[channel][i])).toBeLessThanOrEqual(1e-6); + }); + } + }); +} + +it("decides on the excess above the target, and never to slow down", () => { + const stretch = new Stretch(48000, 1); + stretch.cooldown = 0; + // At or short of the target is audio on the clock or arrivals running late: either way, hold. + expect(stretch.decide(0)).toBe("Hold"); + expect(stretch.decide(-48000)).toBe("Hold"); + // Any excess accelerates, so the ring rests on its target rather than in a band above it. + expect(stretch.decide(1)).toBe("Accelerate"); + expect(stretch.decide(959)).toBe("Accelerate"); + // From NetEq's 20 ms granularity, fast-accelerate, even straight after a stretch. + expect(stretch.decide(960)).toBe("Fast"); + stretch.cooldown = 1; + expect(stretch.decide(959)).toBe("Hold"); + expect(stretch.decide(960)).toBe("Fast"); +}); + +it("passes a short, disabled, exhausted, or late window through unchanged", () => { + for (const rate of [8000, 44100, 48000, 96000]) { + const stretch = new Stretch(rate, 2); + for (const channel of stretch.input) channel.fill(0.25); + stretch.cooldown = 0; + + // A ring well above its target accelerates a full window of received PCM. + stretch.process(stretch.window, stretch.window, true); + expect(stretch.available).toBeLessThan(stretch.window); + const output = [new Float32Array(rate), new Float32Array(rate)]; + expect(stretch.read(output, 0)).toBe(stretch.length); + expect(stretch.read(output, 0)).toBe(0); + + stretch.cooldown = 0; + stretch.process(stretch.window, stretch.window, false); + expect(stretch.available).toBe(stretch.window); + stretch.process(stretch.window >> 1, stretch.window, true); + expect(stretch.available).toBe(stretch.window >> 1); + // A ring a whole window short of its target still plays it as received. + stretch.process(stretch.window, -stretch.window, true); + expect(stretch.available).toBe(stretch.window); + } +}); + +it("accelerates only by a pitch period that fits in the excess", () => { + // A constant signal's shortest period is 2.5 ms: 120 frames at 48 kHz. + const stretch = new Stretch(48000, 1); + stretch.input[0].fill(0.25); + stretch.cooldown = 0; + stretch.process(stretch.window, 119, true); + expect(stretch.available).toBe(stretch.window); + stretch.cooldown = 0; + stretch.process(stretch.window, 120, true); + expect(stretch.available).toBe(stretch.window - 120); +}); diff --git a/js/watch/src/audio/stretch.ts b/js/watch/src/audio/stretch.ts new file mode 100644 index 00000000000..31ad99d53f7 --- /dev/null +++ b/js/watch/src/audio/stretch.ts @@ -0,0 +1,216 @@ +/*! + * Received-PCM adaptation of WebRTC NetEq's accelerate, the shared seams and pitch search + * doc/concept/audio-jitter.md specifies for this player and moq-audio's. This player never expands: + * its ring starts on its target, so slowing down only ever settles audio behind the shared clock. + * + * Copyright (c) 2012 The WebRTC project authors. All Rights Reserved. + * Ported from modules/audio_coding/neteq/{time_stretch,accelerate,dsp_helper, + * audio_vector,decision_logic}. + * @license BSD-3-Clause, with WebRTC's patent grant: see WEBRTC-LICENSE and WEBRTC-PATENTS in this package. + */ + +/** What a window boundary decides, named as the shared corpus names it. */ +export type Decision = "Hold" | "Accelerate" | "Fast"; + +// NetEq's 48 kHz Q12 FIR. Sampling its taps at fractional positions also admits other device rates +// without altering the search domain. +const FILTER = [1019, 390, 427, 440, 427, 390, 1019]; + +// The 4 kHz analysis: 110 downsampled samples, correlated over 50 at lags 10 through 59. +const DOWN = 110; +const LAGS = 50; +const MIN_LAG = 10; + +// NetEq's voice activity bound before it has a background noise model, from signed 16-bit PCM. +const PASSIVE = (8 * 75000) / (32768 * 32768); + +/** + * One window of received PCM and the stretched output staged from it, allocated up front so the + * render callback never allocates. + */ +export class Stretch { + /** The window to analyse, one array per channel: fill the first `frames` of each before {@link process}. */ + readonly input: Float32Array[]; + readonly #output: Float32Array[]; + + /** A full processing window: 30 ms of received PCM, in frames. */ + readonly window: number; + + readonly #rate: number; + // 15 ms: the seam's position, and the longest pitch period. + readonly #middle: number; + // 20 ms: NetEq's adjustment granularity. An excess this large fast-accelerates. + readonly #band: number; + // 50 ms: NetEq's five 10 ms ticks between stretches. + readonly #rest: number; + + #position = 0; + #length = 0; + + /** Frames left to render before an ordinary stretch: NetEq's six startup ticks, then its five. */ + cooldown: number; + + readonly #down = new Float64Array(DOWN); + readonly #correlation = new Float64Array(LAGS); + + constructor(rate: number, channels: number) { + this.#rate = rate; + this.window = Math.max(1, Math.floor((rate * 30) / 1000)); + this.#middle = Math.floor((rate * 15) / 1000); + this.#band = Math.floor((rate * 20) / 1000); + this.#rest = Math.floor((rate * 50) / 1000); + this.cooldown = Math.floor((rate * 60) / 1000); + this.input = Array.from({ length: channels }, () => new Float32Array(this.window)); + this.#output = Array.from({ length: channels }, () => new Float32Array(this.window)); + } + + /** Output frames staged and not yet read. */ + get available(): number { + return this.#length - this.#position; + } + + /** Output frames the last window produced, read or not. */ + get length(): number { + return this.#length; + } + + /** Drop whatever is staged. */ + clear(): void { + this.#position = 0; + this.#length = 0; + } + + /** Copy staged frames into `output` from `offset`, as many as fit, and return how many. */ + read(output: Float32Array[], offset: number): number { + const frames = Math.min(output[0].length - offset, this.available); + // Index loops rather than `set(subarray(...))`, since each `subarray` allocates a view. + for (let channel = 0; channel < output.length; channel++) { + const src = this.#output[channel]; + const dst = output[channel]; + for (let i = 0; i < frames; i++) dst[offset + i] = src[this.#position + i]; + } + this.#position += frames; + this.cooldown = Math.max(0, this.cooldown - frames); + return frames; + } + + /** + * Stage the first `frames` of {@link input}, accelerated by at most `excess` frames: how far the ring + * sits behind its target. Only a full window of received PCM can stretch, so a short block, or one + * that is not `enabled`, passes unchanged and a finite input never becomes concealment. + */ + process(frames: number, excess: number, enabled: boolean): void { + const decision = enabled && frames === this.window ? this.decide(excess) : "Hold"; + const lag = decision === "Hold" ? 0 : this.#pitch(decision === "Fast", excess); + this.#position = 0; + this.#length = frames - lag; + if (lag === 0) { + for (let channel = 0; channel < this.input.length; channel++) { + const input = this.input[channel]; + const output = this.#output[channel]; + for (let i = 0; i < frames; i++) output[i] = input[i]; + } + return; + } + + // Remove `lag` frames after the seam, crossfading the period before the middle into the one after. + const middle = this.#middle; + const seam = middle - lag; + for (let channel = 0; channel < this.input.length; channel++) { + const input = this.input[channel]; + const output = this.#output[channel]; + for (let i = 0; i < seam; i++) output[i] = input[i]; + // NetEq's AudioVector::CrossFade, in floating point. The endpoints stay inside the seam + // rather than duplicating either neighbour. + for (let i = 0; i < lag; i++) { + const weight = (i + 1) / (lag + 1); + output[seam + i] = input[seam + i] * (1 - weight) + input[middle + i] * weight; + } + for (let i = middle + lag; i < frames; i++) output[i - lag] = input[i]; + } + this.cooldown = this.#rest; + } + + /** + * What a full window decides given the ring's `excess` frames above its target, measured where a + * frame has just arrived. Any excess accelerates, so the ring rests on its target rather than + * anywhere in a band above it, and one past NetEq's 20 ms granularity fast-accelerates, which + * catches up after a refill about as fast as dropping the backlog would. A ring at or below its + * target holds: that is either audio on the clock or arrivals running late, and slowing down for + * either adds latency. + */ + decide(excess: number): Decision { + if (excess >= this.#band) return "Fast"; + if (this.cooldown > 0 || excess <= 0) return "Hold"; + return "Accelerate"; + } + + // Search the first channel at 4 kHz, then admit the full-rate seam, returning its length or zero. + // Fast mode admits a weaker match and takes every whole period that fits before the middle. An + // ordinary accelerate removes one period, and only one that fits in the `excess`, so the ring never + // lands ahead of its target. + #pitch(fast: boolean, excess: number): number { + const middle = this.#middle; + if (middle === 0) return 0; + + const rate = this.#rate; + const down = this.#down; + for (let i = 0; i < DOWN; i++) { + let value = 0; + for (let tap = 0; tap < FILTER.length; tap++) { + value += (FILTER[tap] * this.#sample(((10 + 12 * i - tap) * rate) / 48000)) / 4096; + } + down[i] = value; + } + + const correlation = this.#correlation; + for (let index = 0; index < LAGS; index++) { + const lag = index + MIN_LAG; + let value = 0; + for (let t = 60; t < DOWN; t++) value += down[t] * down[t - lag]; + correlation[index] = value; + } + + // The largest sum, the smallest lag on a tie, refined by a parabola through its neighbours. + let peak = 0; + for (let index = 1; index < LAGS; index++) { + if (correlation[index] > correlation[peak]) peak = index; + } + let offset = 0; + if (peak !== 0) { + const left = correlation[peak - 1]; + const center = correlation[peak]; + const right = peak + 1 < LAGS ? correlation[peak + 1] : 0; + const denominator = left - 2 * center + right; + if (denominator !== 0) offset = Math.min(Math.max((0.5 * (left - right)) / denominator, -0.5), 0.5); + } + const lag = Math.min(Math.max(Math.round(((peak + MIN_LAG + offset) * rate) / 4000), 1), middle); + + const input = this.input[0]; + let energy1 = 0; + let energy2 = 0; + let cross = 0; + for (let i = 0; i < lag; i++) { + const a = input[middle - lag + i]; + const b = input[middle + i]; + energy1 += a * a; + energy2 += b * b; + cross += a * b; + } + const passive = (energy1 + energy2) / (2 * lag) <= PASSIVE; + const threshold = (fast ? 8192 : 14746) / 16384; + if (!passive && cross <= threshold * Math.sqrt(energy1 * energy2)) return 0; + if (fast) return Math.floor(middle / lag) * lag; + return lag <= excess ? lag : 0; + } + + // The first channel at a fractional frame, linearly interpolated and clamped to the window. + #sample(position: number): number { + const input = this.input[0]; + const last = this.window - 1; + const clamped = Math.min(Math.max(position, 0), last); + const at = Math.floor(clamped); + const weight = clamped - at; + return input[at] * (1 - weight) + input[Math.min(at + 1, last)] * weight; + } +} diff --git a/js/watch/src/ui/stats.ts b/js/watch/src/ui/stats.ts index c86dcfadfe1..e52b244f29e 100644 --- a/js/watch/src/ui/stats.ts +++ b/js/watch/src/ui/stats.ts @@ -67,6 +67,7 @@ export function statsTab(parent: Effect, watch: MoqWatch): HTMLElement { const aChannels = statLine(audioCard.grid, "Channels"); const aBitrate = statLine(audioCard.grid, "Bitrate"); const aUnderruns = statLine(audioCard.grid, "Underruns"); + const aSkips = statLine(audioCard.grid, "Skips"); const aLate = statLine(audioCard.grid, "Late frames"); track(parent, audioCard, { catalog: watch.audio.source.out.catalog, @@ -125,6 +126,8 @@ export function statsTab(parent: Effect, watch: MoqWatch): HTMLElement { // A non-zero count means the target is below what arrivals actually need. const aUnder = aStats?.underruns ?? 0; aUnderruns.textContent = watch.audio.out.stalled.peek() ? `${aUnder} (buffering)` : `${aUnder}`; + // Each one dropped buffered audio, too far above the target to stretch back onto it. + aSkips.textContent = aStats ? `${aStats.skips}` : "-"; aLate.textContent = aStats ? `${aStats.late}` : "-"; // Network. "Estimated max" is the congestion controller / PROBE estimate; diff --git a/quest/m1/audio-jitter-target/README.md b/quest/m1/audio-jitter-target/README.md index 96cfa558533..df803e1127d 100644 --- a/quest/m1/audio-jitter-target/README.md +++ b/quest/m1/audio-jitter-target/README.md @@ -11,10 +11,12 @@ at once got a target far too shallow to play through the next flush. One written algorithm, cited by both implementations, produces the same target from the same arrival trace. -Boundaries: converging by playing slightly faster or slower is the browser's -[Watch](/quest/m1/audio-jitter-target/watch.md) quest. No packet loss concealment. -Video keeps its own target; the shared clock stays on local time, and -[Watch](/quest/m1/audio-jitter-target/watch.md) converges the audio ring on it. +Boundaries: both players converge on the target with NetEq's time stretch: +native playback by playing slightly faster or slower, and the browser only +faster, since its ring starts on the target and slowing down only settles audio +behind the shared clock. Underruns past the target are accepted, not covered +with latency. No packet loss concealment. Video keeps its own target; the +shared clock stays on local time, and the audio ring's stretch converges on it. ## Plan @@ -22,16 +24,25 @@ Both implementations are on `main` (#4162): `js/hang/src/container/jitter.ts` and `rs/moq-audio`'s `jitter.rs` follow `doc/concept/audio-jitter.md` and pass its conformance corpus, `js/watch` `Sync` holds the deepest track's target in `"auto"`, and `moq play --delay` defaults to `auto`. Both are the default for -every viewer, so `"auto"` shipped ahead of the Chrome and Safari proof the -Watch quest still owes; the maintainer accepted that, and the re-measured +every viewer, so `"auto"` shipped ahead of its browser proof; the maintainer +accepted that, and the re-measured replay budgets, on 2026-10-04. Decided 2026-10-13: retain the remaining proof in m1 because the A/V clock and worker release contracts depend on it. Native quality expansion remains separately scoped in m2. The 2026-11-03 proof run found the 0.95 quantile leaves about six re-stalls a minute on real paths by design; decided -2026-11-03 (maintainer), Watch closes that by time stretch rather than a deeper -target, and `moq play` copies NetEq's stretch in m1 as -[Native stretch](/quest/m1/audio-jitter-target/native-stretch.md), sharing -one algorithm and its vectors with the browser's. +2026-11-03 (maintainer), time stretch closes that rather than a deeper +target, in both players from one algorithm and its vectors: +[moq-dev/moq#7856](https://github.com/moq-dev/moq/pull/7856) natively and +[moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058) in the browser's ring. + +Decided 2026-11-08 (maintainer, on +[moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058)): never add +latency to chase zero underruns. Audio and video keep separate clocks, which +align at steady state once the delay is right, so the browser's stretch only +steers audio back onto the clock and never settles it later to absorb jitter. +Some underruns during congestion are expected. The replay rows budget +`av_lag_ms`, how far audio trails the shared clock, exactly as measured, so a +change that buys underruns with latency fails them. Decided 2026-11-04 (maintainer), for push-to-talk's silence suppression ([moq-dev/moq#7942](https://github.com/moq-dev/moq/pull/7942)): a silence the publisher declares @@ -39,17 +50,18 @@ with an end marker is not idle, so the estimator folds no idle intervals for it and keeps its estimate, where an undeclared gap still relaxes toward the prior. The document's `silence` corpus case holds both implementations to it. -What the line still owes once Watch lands: its recorded trace replayed through +What the line still owes: its recorded trace replayed through the native decode path too, asserting the same target series the browser's `playout.test.ts` does. The audio quality harness's `pulse` rows already grade native playback against its budgets. ## Required -- [Watch](/quest/m1/audio-jitter-target/watch.md) - the browser plays the measured real-microphone rows with zero steady-state underruns by time-stretching the ring -- [Native stretch](/quest/m1/audio-jitter-target/native-stretch.md) - `moq play` copies NetEq's accelerate and preemptive expand, sharing the browser's algorithm and vectors, so its native rows reach zero steady-state underruns +- [Browser ceilings](/quest/m1/audio-jitter-target/browser-ceilings.md) - the chromium rows' ceilings re-measured from seven Nightlies with the browser's time stretch, budgeting `player_skips_per_min` +- [Native stretch](/quest/m1/audio-jitter-target/native-stretch.md) - `moq play` copies NetEq's accelerate and preemptive expand, sharing the browser's seams and vectors, so its native rows play with no audible glitch once the target settles - [Buffered converge](/quest/m1/audio-jitter-target/buffered-converge.md) - a buffered `` time-stretches its audio back onto the shared clock after a stall or drift, so lip sync returns and no audio group is dropped - [Capped budgets](/quest/m1/audio-jitter-target/capped-budgets.md) - the AAC browser budgets are re-measured from nightlies after the TS importer groups each PES +- [Native A/V lag](/quest/m1/audio-jitter-target/native-av-lag.md) - `moq play` reports how far audio trails its presentation clock, and native stops expanding into the gap between arrivals if it settles audio late - [Native rendition delay](/quest/m1/audio-jitter-target/native-rendition-delay.md) - `moq play` holds each subscribed rendition's catalog `delay` plus its target, as the browser does ## Closes diff --git a/quest/m1/audio-jitter-target/browser-ceilings.md b/quest/m1/audio-jitter-target/browser-ceilings.md new file mode 100644 index 00000000000..411b975b72b --- /dev/null +++ b/quest/m1/audio-jitter-target/browser-ceilings.md @@ -0,0 +1,50 @@ +# [S] Re-measure the chromium audio ceilings with the browser's time stretch + +## Goal + +`test/audio-quality/budgets.json` carries every chromium row's ceilings +re-measured, by the rule in its note, from seven scheduled Nightly runs on +`main` that all ran with the browser's NetEq time stretch, and budgets +`player_skips_per_min` and `av_lag_ms` on those rows beside the tap metrics. No chromium row +skips buffered audio in steady state: its `player_skips_per_min` stays at the +startup re-anchors the estimator's hold releases, as on the replay rows. What +still runs dry is what NetEq would conceal, a receive stall longer than the +target, and its underrun ceilings are measured rather than fixed at zero. + +## Waiting + +- moq-dev/moq: seven scheduled Nightly runs on `main` after + [moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058) merges. Find them + with `gh run list --workflow nightly.yml --event schedule` and download each + with `gh run download -n audio-quality-run`; never dispatch the nightly by + hand. The artifacts expire after 7 days, so download each run's as it lands + rather than waiting for the seventh. + +## Plan + +Decided 2026-11-07 (maintainer, /quest-complete on +[moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058)): ✅ land the +stretch, then measure (rejected: hold the PR until the Nightly ceilings are in). +The chromium rows can only be measured once the stretch is on `main`, so this +quest carries that measurement, and its Goal is judged by it and the replay +rows alone, with no maintainer listening run (decided 2026-11-07, +/quest-unblock). + +[moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058) budgeted every +replay row exactly as measured, A/V lag included, and left the browser +ceilings unchanged. The browser's stretch never slows down, so underruns stay +where the trim had them (`relay-mic-rtt40` 10.6 `player_underruns_per_min` +against 8.2) while the audio the trim dropped plays faster instead, and the +A/V lag rests about one frame ahead of the clock. Expect the chromium rows +alike: re-stalls about as before, `player_skips` near zero, and +`skip_aheads` and `discarded_ms`, which infer a skip from a step in the +playhead's lag, reading a fast accelerate after a receive stall as a skip. `player_skips` counts the ring's real +re-anchors, so a rise there is a regression worth reporting before budgeting +it, as is an `av_lag_ms` p50 above zero, which is audio settling behind the +clock. Rewrite the note's browser sentences to the new nights and say the stretch +is in them. + +## Related + +- [Capped budgets](/quest/m1/audio-jitter-target/capped-budgets.md) - re-measures the AAC browser rows from nightlies too; the same seven nights can serve both +- [Native stretch](/quest/m1/audio-jitter-target/native-stretch.md) - the native rows' re-measure after the same stretch diff --git a/quest/m1/audio-jitter-target/buffered-converge.md b/quest/m1/audio-jitter-target/buffered-converge.md index b438cfbfa81..b418840c46c 100644 --- a/quest/m1/audio-jitter-target/buffered-converge.md +++ b/quest/m1/audio-jitter-target/buffered-converge.md @@ -41,7 +41,7 @@ Decisions: Recommendations: -- Reuse [Watch](/quest/m1/audio-jitter-target/watch.md)'s stretch rather +- Reuse the ring's stretch (`js/watch/src/audio/stretch.ts`) rather than a second mechanism: buffered mode measures the playhead's lag behind the shared clock and accelerates until it is back within the floor. It never stretches ahead of the clock, since future-dated frames are @@ -55,14 +55,11 @@ Recommendations: Public API: none expected. Wire: none. -## Required - -- [Watch](/quest/m1/audio-jitter-target/watch.md) - lands the ring's time stretch this reuses - ## Closes - [#8031](https://github.com/moq-dev/moq/issues/8031) - buffered playback loses lip sync and cuts audio out on a live stream ## Related +- [Browser ceilings](/quest/m1/audio-jitter-target/browser-ceilings.md) - measures on the nightly runner the ring's time stretch this reuses, which landed in [moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058) - [Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md) - the other buffered-mode change to the audio path diff --git a/quest/m1/audio-jitter-target/capped-budgets.md b/quest/m1/audio-jitter-target/capped-budgets.md index 586b61f9f80..7d33d9d0bd5 100644 --- a/quest/m1/audio-jitter-target/capped-budgets.md +++ b/quest/m1/audio-jitter-target/capped-budgets.md @@ -42,7 +42,7 @@ arrival slots handed to its frames oldest first, as one group, halved the AAC `bursty` and `wide` discards (`bursty` summed over the seven nights 1156 to 584 ms a minute) and lowered both rows' target by 20 to 60 ms. That shallower target doubled AAC `wide`'s player re-stalls (mean 3.0 to 6.6 a minute), the 0.95 quantile's trade -that [Watch](/quest/m1/audio-jitter-target/watch.md)'s time stretch absorbs. +that the browser's time stretch absorbs. AAC `bursty` still discards about three times what Opus `bursty` does (584 against 174 summed), and parity is not expected: the harness keeps ffmpeg's diff --git a/quest/m1/audio-jitter-target/native-av-lag.md b/quest/m1/audio-jitter-target/native-av-lag.md new file mode 100644 index 00000000000..e7b0af6ea4d --- /dev/null +++ b/quest/m1/audio-jitter-target/native-av-lag.md @@ -0,0 +1,56 @@ +# [M] Native audio never settles behind its presentation clock + +## Goal + +`moq play --metrics` reports how far the native audio playhead trails the +presentation clock video plays from, so every native row in +`test/audio-quality` grades `av_lag_ms` (p50, p95) as the replay rows do. If +native's time stretch settles audio behind that clock, as the browser's did +before [moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058), native +stops spending latency on it: at steady state its p50 lag is at or below zero +on every native row, and its ceilings guard that. + +Boundaries: the jitter estimator and its target stay as +`doc/concept/audio-jitter.md` specifies them. Underruns during congestion are +accepted, never bought down with latency. Native keeps compensating clock +drift between publisher and device, which the browser's ring cannot. + +## Plan + +Decided 2026-11-08 (maintainer, on +[moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058)): never add +latency to chase zero underruns; audio and video keep separate clocks that +align at steady state once the delay is right. The browser's ring now decides +on its depth right after each arrival and never expands, because expanding on +the depth between arrivals settled audio a burst behind the shared clock +(`bursty-aac` lag p50 1 to 289 ms). Native's `Stretch::decide` +(`rs/moq-audio/src/stretch.rs`) still decides on `buffered` at each callback, +so it may do the same. + +1. Report the lag. The metrics writer (`rs/moq-cli/src/play/metrics.rs`) + already holds the presentation `moq_mux::sync::Consumer` and the + `Playhead`. Write each sample's `clock`, the presentation clock's media + time at the instant the playhead is read, with the field and meaning + `test/audio-quality/clients/js/src/schema.ts` gives the browser's, so + `analyze.ts` reduces native samples to `av_lag_ms` unchanged. Leave it out + while nothing is mapped, as other unreadable fields are. +2. Measure. Run the native rows locally against `main`'s decision, then read + the scheduled Nightlies. Lag p50 near or below zero means native already + aligns: budget `av_lag_ms` by the native rule and finish. +3. If p50 sits above zero on any row, decide on the depth right after each + arrival, less what the stretch removed since, as the browser's + `AudioRingBuffer` does, and expand only to follow a publisher clock + slower than the device's, not to fill the gap between arrivals. Hold the + listened drift case (media per render second about 0.998 against a 0.998 + publisher) and the shared stretch corpus, and update the spec's native + decision in the same PR. + +[Native stretch](/quest/m1/audio-jitter-target/native-stretch.md) is +calibrating the native ceilings from seven Nightlies of today's decision. +Step 3 moves those ceilings, so a decision change re-measures them by the +same rule rather than stacking on the old calibration. + +## Related + +- [Native stretch](/quest/m1/audio-jitter-target/native-stretch.md) - the native ceilings this lag joins, calibrated from the current decision +- [Browser ceilings](/quest/m1/audio-jitter-target/browser-ceilings.md) - budgets `av_lag_ms` on the chromium rows the same way diff --git a/quest/m1/audio-jitter-target/native-stretch.md b/quest/m1/audio-jitter-target/native-stretch.md index 69461a8e24b..78e85a1fc9a 100644 --- a/quest/m1/audio-jitter-target/native-stretch.md +++ b/quest/m1/audio-jitter-target/native-stretch.md @@ -1,11 +1,14 @@ -# [L] moq play reaches zero steady-state underruns by time-stretching +# [L] moq play converges inaudibly by time-stretching ## Goal `rs/moq-audio`'s playback converges on its playout target by time-stretching received audio, as WebRTC NetEq does, instead of running its ring dry and padding with silence or trimming buffered audio. `moq play`'s native rows in -`test/audio-quality` reach zero underruns once the target settles. The +`test/audio-quality` play with no audible glitch once the target settles, +with underrun ceilings measured by `budgets.json`'s rule rather than fixed at +zero, as decided 2026-11-06 on +[moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058). The stretch shares one algorithm and one set of test vectors with the browser's. Boundaries: no packet loss concealment, so a frame that never arrives is @@ -23,15 +26,22 @@ received-audio accelerate and preemptive expand into the native playback callback. The estimator is unchanged. `doc/concept/audio-jitter.md` specifies the shared floating-point algorithm; `doc/concept/audio-jitter/stretch/corpus.json` supplies literal PCM vectors -for both languages. The browser half remains -[Watch](/quest/m1/audio-jitter-target/watch.md); neither implementation waits -for the other. +for both languages. The browser half landed in +[moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058). The port retains WebRTC's BSD notice and patent grant, 4 kHz analysis, 50-sample correlation at lags 10 through 59, 14746/16384 normal and 8192/16384 fast admission thresholds, passive-speech VAD, 20 ms decision band, 4x fast threshold, and 60 ms startup / 50 ms successful-stretch -cooldowns. Analysis samples NetEq's 48 kHz FIR on a virtual clock for other +cooldowns. +[moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058) drains a declared +end's tail unstretched, as finishing does. Its browser half decides on the +ring's depth right after each arrival and never expands, since expanding on +the depth between arrivals settled audio a burst behind the shared clock +(decided 2026-11-08). Native playback still decides on `buffered`, so +whether it settles behind its clock the same way is worth measuring once its +rows report A/V lag. +Analysis samples NetEq's 48 kHz FIR on a virtual clock for other device rates and uses floating-point peak refinement and fades. Those arithmetic adaptations are documented, not claimed bit-identical to int16 WebRTC. No threshold was tuned. @@ -44,8 +54,8 @@ capacity watermark and an incoming excess at the fast threshold trim. Remaining verification: -- Native `test/audio-quality` rows must show zero steady-state underruns - once their targets settle. Keep a failed or void row explicit and fix +- Native `test/audio-quality` rows must meet their measured underrun + ceilings once their targets settle. Keep a failed or void row explicit and fix any implementation cause before calling this complete. - Re-measure the provisional native budgets from seven scheduled nightly artifacts using `budgets.json`'s rule: worst clean run times 1.25, plus @@ -110,4 +120,4 @@ bench waits below remain. ## Related -- [Watch](/quest/m1/audio-jitter-target/watch.md) - the browser stretch sharing this algorithm and its vectors +- [Browser ceilings](/quest/m1/audio-jitter-target/browser-ceilings.md) - re-measures the browser rows with the stretch sharing this algorithm and its vectors diff --git a/quest/m1/audio-jitter-target/watch.md b/quest/m1/audio-jitter-target/watch.md deleted file mode 100644 index e2c0409ff1a..00000000000 --- a/quest/m1/audio-jitter-target/watch.md +++ /dev/null @@ -1,178 +0,0 @@ -# [L] js/watch: zero steady-state audio underruns by time-stretching the ring - -## Goal - -On the measured real-microphone rows, the "Real-time" (`"auto"`) preset plays -with zero player underruns after convergence and no skip-aheads in steady -state, with the estimator's `QUANTILE` left at 0.95. Audio holds back or -catches up by playing slightly slower or faster, the way NetEq's preemptive -expand and accelerate do, rather than running dry, discarding buffered audio, -or rendering silence. Convergence is inaudible at ordinary drift and burst -sizes. - -The measured rows are the `relay-mic-rtt40` replay row, whose -`player_underruns_per_min` must reach 0, and a live re-run of the 2026-11-03 -rows below with a real microphone, Chrome's with 40 ms or more of added RTT -and Safari's on the direct path, read from the stats panel's audio Underruns. - -Boundaries: no packet loss concealment, so a frame that never arrives is still -missing audio; only received audio is stretched. The estimator stays as -`doc/concept/audio-jitter.md` specifies it. Native playback is out of scope -here. - -## Plan - -Decided 2026-11-03 (maintainer, on -[moq-dev/moq#7679](https://github.com/moq-dev/moq/pull/7679)): pull time -stretch into m1 as this quest's fix, keeping `QUANTILE` at 0.95. The options -were restating the Goal to the measured rate, or raising the quantile. The m2 -Time stretch quest was folded in here. - -Decided 2026-11-06 (maintainer, /quest-unblock): drop the Safari row with -40 ms or more of added RTT. Safari can't be redirected through the RTT relay, -and shaping it needs Network Link Conditioner or `dnctl`/`pfctl` on the -laptop, so its rows run on the direct path. Not chosen: a hands-on Safari run -with shaped RTT. - -### Baseline, 2026-11-03 - -An agent ran the maintainer's MacBook Air (macOS 26.6.2) against -`cdn.moq.dev/anon`. The built-in microphone was published by `` in -Chrome 156 and played by the harness page in a second Chrome 156 profile and in -Safari 26.6.2. The added RTT was a userspace UDP relay on the laptop holding -each packet 20 ms each way, reached through Chrome's `--host-resolver-rules`. -Safari cannot be redirected that way and negotiated WebSocket, so its rows ran -on the direct path. Underruns are the stats panel's counter after the first -30 s. - -| Row | Transport | RTT min | Underruns after 30 s | Skip-aheads after 30 s | Auto delay p50 / max | -| --- | --- | --- | --- | --- | --- | -| Chrome, direct, 20 ms groups | WebTransport | 5 ms | 9 in 88 s (6.1/min) | 0 | 40 / 60 ms | -| Chrome, +40 ms, 20 ms groups | WebTransport | 47 ms | 26 in 149 s (10.5/min) | 1 (43 ms) | 100 / 120 ms | -| Chrome, +40 ms, 100 ms groups | WebTransport | 47 ms | 9 in 149 s (3.6/min) | 0 | 100 / 120 ms | -| Safari, direct, 20 ms groups | WebSocket | 5 ms | 5 in 118 s (2.5/min) | 0 | 120 / 140 ms | -| Safari, direct, 100 ms groups | WebSocket | 5 ms | 12 in 119 s (6.1/min) | 0 | 120 / 140 ms | - -The Chrome +40 ms, 20 ms-group capture is checked in as the `relay-mic-rtt40` -replay row, budgeted exactly as measured: 7.9 player underruns a minute. Every -replay row now also budgets `player_underruns_per_min`, the player's own -counter, which counts the run-dry re-stalls the output tap files as stalled -time. - -Replayed, each row's arrivals reproduce its live re-stalls and short quanta -almost exactly, so the browser's decode and render path adds nothing. The ring -sits at the target less one frame, and each run-dry is one frame arriving later -than that cushion. The target is the 95th percentile of each 500 ms -interval's worst delay, so about 5% of intervals, some six a minute, hold such -a frame. Raising `QUANTILE` in a replay sweep cut the rate but not to zero: -0.99 gave 1.0 to 3.5 a minute at a 60 to 200 ms p50 target, against 5.6 to -13.2 at 40 to 120 ms. - -### How deep the stretch must go - -Against the cushion each frame met, the five rows' late frames (12 to 27 in -each after 30 s) exceed it by a median of 10 to 16 ms and at most 44 to 81 ms. -Stretching the cushion across that wait means playing at `1 - k` of real time -from the moment the ring dips, with `k = excess / delay`. That is 11 to 31% at -the median, 30 to 47% at p90, and up to 38 to 71%. A bound of a few percent -per quantum cannot meet the Goal. - -### Implementation: copy NetEq - -Decided 2026-11-03 (maintainer): copy what WebRTC's NetEq does. The stretch -removes or repeats whole pitch periods of received audio, found by -autocorrelation, and cross-fades across the seam. It is not a generic -resampler or a phase vocoder. The mirrored files are in WebRTC's -`modules/audio_coding/neteq/`, read at `a057d00e46` (2026-11-03): - -- `time_stretch.cc` and `time_stretch.h`, the shared search. Downsample to - 4 kHz and autocorrelate over lags `kMinLag` 10 to `kMaxLag` 60 (2.5 to 15 ms - pitch periods), over `kCorrelationLen` 50. Then refine the best peak at full - rate (`DspHelper::PeakDetection`), and normalise the cross-correlation by - the two segments' energies. `SpeechDetection` treats a segment as passive - when its energy per pitch sample is at most 8 times the background noise's - (a fixed 75000 before noise is estimated). Passive audio may stretch - regardless of correlation. -- `accelerate.cc`. With at least 30 ms of input, keep the first 15 ms, - cross-fade one pitch period onto it, and drop that period. It stretches only - when the correlation is over `kCorrelationThreshold` (0.9 in Q14) or the - audio is passive. Fast accelerate lowers the bar to 0.5 and removes as many - whole periods as fit in 15 ms. -- `preemptive_expand.cc`. Under the same 0.9 test, with at most 15 ms of old - data ahead of the seam, repeat one pitch period after the unmodified part - and cross-fade it in. -- `decision_logic.cc` and `decision_logic.h`, when to stretch. The low limit - is the target and the high limit is the target plus the arrival history's - maximum delay plus `kDelayAdjustmentGranularityMs` (20 ms). - - Preemptive expand when the playout delay is under the low limit. - - Accelerate at or above the high limit. - - Fast accelerate at four times the high limit. - - At most one stretch every `kMinTimescaleInterval` (5) ticks of 10 ms. - - Here the target is `Sync`'s resolved delay, and the arrival history's - maximum is the estimator's. - -NetEq decides once per 10 ms of output, and our worklet renders 128-sample -quanta. So the stretch runs on blocks of decoded PCM before they reach the -render quantum, where at least 30 ms are buffered. - -One boundary differs from NetEq. NetEq preemptive-expands only while the next -packet is already buffered (`ExpectedPacketAvailable`). Once it is missing, -NetEq runs Expand, which is concealment and outside this quest. At NetEq's -cadence, one period of at most 15 ms every 50 ms, preemptive expand slows -playback by about a quarter at most. That covers the median late frame above -but not the 38 to 71% tail. So the work is: start at NetEq's thresholds, -measure `relay-mic-rtt40`, and only then decide how to close the tail. Options -are expanding sooner or more often as the ring nears empty, or accepting the -remaining re-stalls. Each departure from NetEq's constants is recorded with -the replay numbers that justify it. - -- Port the shared specification rather than NetEq independently: - [moq-dev/moq#7856](https://github.com/moq-dev/moq/pull/7856) landed the - native stretch with `doc/concept/audio-jitter.md` specifying the shared - floating-point adaptation and `doc/concept/audio-jitter/stretch/corpus.json` - holding literal PCM vectors for both languages. Implement that spec and - pass those vectors, so both players stretch alike; record any departure - from it in the spec, not only here. -- Put the stretch where `js/watch/src/audio/replay.ts` exercises it, such as - `AudioRingBuffer`'s read path or a module the render worklet and the replay - share, so the replay rows measure the stretch and not a model of it. -- Accelerate replaces the skip-ahead when the ring sits above the high - limit. A re-stall remains for a ring that empties despite the stretch, and a - skip-ahead remains only above what fast accelerate can close. -- Verification: `relay-mic-rtt40` reaches 0 `player_underruns_per_min` and its - budget is tightened to the new measurement. No other replay row regresses, - and the nightly browser rows are re-measured by the budgets note's rule. A - listening check confirms that a 2 ms/s drift and the measured expansions are - inaudible or close to it. The 2026-11-03 rows are re-run on real hardware: - Chrome's at +40 ms and Safari's on the direct path. - -### Publisher, measured in the same run - -`AudioEncoder` input-to-output lag on live input is p50 0.9 ms and p99 1.1 ms, -with drift under 0.04 ms/s at 20 ms and 100 ms groups. The 7.35 s lag and 88 -to 275 ms/s drift of -[moq-dev/moq#3477](https://github.com/moq-dev/moq/issues/3477) did not recur. -Each demand session opens with up to 30 s of retained history -(`js/publish/src/audio/backlog.ts`), which the watcher discards. 100 ms groups -halved Chrome's rate at +40 ms but more than doubled Safari's on the direct -path, so the 20 ms default stays. - -Decided 2026-10-15: keep this proof in m1 with the existing playback and -A/V clock work. - -Decided 2026-11-05 (maintainer, /quest-plan): steady-state lip sync comes -from this stretch converging the ring's playhead on `Sync.now()`, the shared -clock on local time anchored at the earliest arrival. The clock never follows -audio: one slow track must not hold the others, so audio and video stay in sync -in the steady state and drift apart only while a track is congested. -[moq-dev/moq#7941](https://github.com/moq-dev/moq/pull/7941) dropped an -audio-driven clock for this. Recommendation: measure the gap against the sample -reaching the output, through `AudioContext.getOutputTimestamp()`, so output -latency, Bluetooth included, is part of the lip sync. - -## Related - -- [Capped budgets](/quest/m1/audio-jitter-target/capped-budgets.md) - re-measures the AAC browser budgets after the publisher change, no gate on this proof -- [Native stretch](/quest/m1/audio-jitter-target/native-stretch.md) - the native time-stretch sharing one NetEq algorithm and its test vectors with this one -- [Watch worker](/quest/m3/watch-worker.md) - moves the worklet this stretch changes diff --git a/quest/m1/push-to-talk-latency.md b/quest/m1/push-to-talk-latency.md index da675e8ed15..0e08084622d 100644 --- a/quest/m1/push-to-talk-latency.md +++ b/quest/m1/push-to-talk-latency.md @@ -30,7 +30,7 @@ Decided 2026-11-06 (maintainer, on `just test push-to-talk` run until Watch lands, since its no-underrun check fails on the defaults by design? -- ✅ (a) Require [Watch](/quest/m1/audio-jitter-target/watch.md) for this +- ✅ (a) Require Watch's time stretch ([moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058)) for this quest. The run holds a fixed 300 ms delay until then, and the defaults are budgeted after Watch lands, on the jitter buffer users will get. - (b) Run the defaults now and stop failing on underruns while `"auto"` is @@ -91,10 +91,7 @@ measured term (`js/watch/src/audio/latency.ts`); device and render follow the watch AudioContext's `latencyHint` (`"interactive"` in `js/watch/src/audio/decoder.ts`), otherwise the platform. -## Required - -- [Watch](/quest/m1/audio-jitter-target/watch.md) - `"auto"` stops underrunning by time-stretching the ring, so the run can hold the default delay - ## Related +- [Browser ceilings](/quest/m1/audio-jitter-target/browser-ceilings.md) - `"auto"` underruns less by time-stretching the ring, so the run can hold the default delay; the stretch landed in [moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058), and its nightly ceilings are re-measured there - [Audio jitter target](/quest/m1/audio-jitter-target/README.md) - the `"auto"` target this latency feeds diff --git a/quest/m3/watch-worker.md b/quest/m3/watch-worker.md index fb621ca3ef1..e18e1983bf8 100644 --- a/quest/m3/watch-worker.md +++ b/quest/m3/watch-worker.md @@ -33,7 +33,3 @@ Changed 2026-11-07 (maintainer, /quest-plan from the API sweep of open PRs and r ## Required - [Plan: watch worker](/quest/m3/plan-watch-worker.md) - picks the worker model and rewrites this quest - -## Related - -- [Watch](/quest/m1/audio-jitter-target/watch.md) - time-stretches in the worklet this feeds diff --git a/quest/m3/webcodecs-priming.md b/quest/m3/webcodecs-priming.md index 75ba8569738..3c74ec04175 100644 --- a/quest/m3/webcodecs-priming.md +++ b/quest/m3/webcodecs-priming.md @@ -18,7 +18,7 @@ gap ([w3c/webcodecs#626](https://github.com/w3c/webcodecs/issues/626)). Out of scope: Rust, where `moq-audio` already declares Opus lookahead as pre-skip and folds the AAC delay into timestamps (`folded_delay` in `rs/moq-audio/src/encode/encoder.rs`); and playback-side sync, which -[Watch](/quest/m1/audio-jitter-target/watch.md) converges by time-stretching. +the audio ring converges by time-stretching. ## Plan diff --git a/quest/m4/audio-bufferbloat-av.md b/quest/m4/audio-bufferbloat-av.md index 7a0dbcf66aa..e246bfc7635 100644 --- a/quest/m4/audio-bufferbloat-av.md +++ b/quest/m4/audio-bufferbloat-av.md @@ -24,7 +24,7 @@ Decided in planning (2026-10-20): audio. Video freezes, skips, and lag are reported, not gated: the point is that audio does not pay for them. Since 2026-11-05 the shared clock never follows audio, so congested video lags it by design, and steady-state lip - sync belongs to [the Watch stretch](/quest/m1/audio-jitter-target/watch.md). + sync belongs to the audio ring's time stretch. - Folded in: the native `pulse` rows become enforced. They report only because an intermittent two-second device stall voids about one AAC native row in seven (see `test/audio-quality/README.md`). Reproduce and fix that diff --git a/rs/moq-audio/src/ring.rs b/rs/moq-audio/src/ring.rs index 5789da4a6fb..aac9342b4f2 100644 --- a/rs/moq-audio/src/ring.rs +++ b/rs/moq-audio/src/ring.rs @@ -620,6 +620,9 @@ impl Consumer { } frames = frames.min(ahead); } + // A declared end's tail drains as received: running dry after it is + // silence, not an underrun, so nothing should expand toward the target. + let declared = self.declared(); let Some(stretch) = self.stretch.as_mut() else { let frames = frames.min((output.len() - done) / channels); if frames == 0 { @@ -640,7 +643,7 @@ impl Consumer { break; } // Never transform startup padding or a finite stream's tail. - let enabled = self.prime == 0 && !self.shared.finished.load(Ordering::Relaxed); + let enabled = self.prime == 0 && !self.shared.finished.load(Ordering::Relaxed) && !declared; let frames = if self.prime > 0 { frames.min(self.prime) } else { frames }; self.staged_prime = self.prime > 0; self.prime -= frames.min(self.prime); @@ -980,6 +983,34 @@ mod tests { assert_eq!(cons.read(&mut out), Read::Underflow { frames: 0 }); } + #[test] + fn a_declared_end_drains_its_tail_unstretched() { + let (mut prod, mut cons) = channel( + 2, + 48_000, + 48_000, + Config { + latency: Duration::from_millis(50), + capacity: Duration::from_secs(1), + pad: false, + trim: false, + }, + ); + cons.stretch(48_000); + cons.start(); + // Twice the target, so the tail drains through it: without the end it + // would expand there, toward a target nothing will refill. + assert_eq!(prod.push(&vec![0.25; 4800 * 2]), Push::Ok); + prod.end(); + let mut out = [0.0; 128 * 2]; + let mut reads = 0; + while cons.available() > 0 { + cons.read(&mut out); + reads += 1; + } + assert_eq!(reads, 7200usize.div_ceil(128), "a declared end never extends the tail"); + } + #[test] fn recovery_waits_for_received_pcm_without_stretching_padding() { let (mut prod, mut cons) = channel( @@ -1035,12 +1066,18 @@ mod tests { for (i, frame) in pcm.as_chunks_mut::<2>().0.iter_mut().enumerate() { frame.fill((i as f32 * std::f32::consts::TAU * 200.0 / 48_000.0).sin() * 0.25); } - let (first, second) = pcm.split_at(143_000 * 2); - // Play out the 50 ms prime and the 60 ms startup cooldown below the - // trim watermark, ending on a window boundary. + let (first, rest) = pcm.split_at(143_000 * 2); + let (cushion, second) = rest.split_at(1_000 * 2); + // Play out the 50 ms prime below the trim watermark, then top the queue + // up to the target so its first window, inside the 60 ms startup + // cooldown, holds rather than expands. End on a window boundary. assert_eq!(prod.push(first), Push::Ok); let mut out = [0.0; 480 * 2]; - for _ in 0..8 { + for _ in 0..5 { + assert_eq!(cons.read(&mut out), Read::Ok); + } + assert_eq!(prod.push(cushion), Push::Ok); + for _ in 0..3 { assert_eq!(cons.read(&mut out), Read::Ok); } // Above NetEq's high limit (target + 20 ms) but under the trim diff --git a/test/audio-quality/README.md b/test/audio-quality/README.md index fcd82fc55c0..3e347e172cb 100644 --- a/test/audio-quality/README.md +++ b/test/audio-quality/README.md @@ -223,6 +223,12 @@ writes the same samples a page does, so `analyze.ts` reduces both alike, and the and clock checks have nothing to void. Decoding is taken as instant. A replay is deterministic, so its budgets are exactly what it measured. +The ring time-stretches live playback as the page's does, but the replay's audio is a constant, so +every seam admits at the shortest pitch period, 2.5 ms. Accelerate therefore moves less per stretch +than it would from speech; only fast accelerate, which takes every whole period that fits, moves its +most. `skip_aheads` also reads a sustained accelerate as a skip; `player_skips` counts only the ring's +own re-anchors. + ```bash just test audio-quality-record # the three recorded lanes, 35 s each, needs the network just test audio-quality-record --lanes relay-bbb @@ -282,6 +288,9 @@ player's own count, which its stats panel shows, is read alongside: - `player_underruns`: `audio.out.stats.underruns`, each time the ring ran dry mid-playback and re-stalled to refill, plus each quantum it filled only part-way. The tap files the quanta after a run-dry as stalled time, so `underruns` leaves those out and this counts them. Unset natively. +- `player_skips`: `audio.out.stats.skips`, each jump that dropped buffered audio rather than playing + it faster. `skip_aheads` below infers the same events from the playhead, and also reads a time + stretch that sustains 40 ms of acceleration as one, so this is the exact count. Unset natively. - `underruns`: quanta the ring filled only partly, or not at all, while playing. - `short_quanta`: the underruns that were partly filled: the ring ran dry part-way through. - `underrun_episodes`: maximal runs of consecutive underruns, one audible gap each. @@ -295,6 +304,10 @@ player's own count, which its stats panel shows, is read alongside: that starts within 60 ms of a stall is that stall, and is not an underrun. - `silence`: the share of rendered quanta under -60 dBFS at the output, whatever the cause. - `target_ms`: the resolved playout delay (`sync.out.delay`). +- `av_lag_ms`: how far the audio playhead trails the shared clock video plays from (`sync.now()`), + sampled while playing. Audio and video keep separate clocks that align once the delay covers the + arrivals, so this is the latency audio adds beyond the target: positive is audio late, negative + early. Its replay ceilings hold audio from settling later to buy fewer underruns. Unset natively. - `converge_ms`: from first audio to the last time the target moved more than one 20 ms bucket from its final value. - `render_load`: Chromium's render capacity, the share of each quantum's budget the graph used. diff --git a/test/audio-quality/budgets.json b/test/audio-quality/budgets.json index da8764b3a1b..916e5d9bfcc 100644 --- a/test/audio-quality/budgets.json +++ b/test/audio-quality/budgets.json @@ -1,5 +1,5 @@ { - "note": "The browser rows are measured on the nightly runner (ubuntu-24.04-arm, headless Chromium, seed 7, 60 s a row), from the audio-quality-run artifacts of the seven scheduled Nightly runs on main from 2026-10-29 to 2026-11-04 (40885347517, 41065371094, 41204380716, 41313466339, 41483654426, 41656613559, 41852517544), the first after a fresh subscriber stopped receiving the relay's cached history. The AAC rows predate the TS importer publishing each PES as one group. Each ceiling is the worst of its seven nights times 1.25, plus one runner hiccup, rounded up to two significant figures. The hiccup is sized from the single gaps the 2026-10-08 night put on otherwise clean rows, the fixed-250 control among them: 250 underruns/min, 700 gap ms/min, 3.3 skips/min, 280 discarded ms/min, 0.01 stalled, 0.015 silence, and one 20 ms bucket of target. Budgeted this way, every night passes when it is held out of its row's measurement except one, and that row's own ceiling includes it: the 2026-11-03 Opus bursty row raised its target to 300 ms 2.6 s in and held it for one 20 s half-life before settling at 180. Such startup holds, Opus near-zero's 100 ms on 2026-10-30 and 10-31 among them, are budgeted as measured. These ceilings record today's player as the runner hears it, not target audio quality. To re-measure, download the same artifact from the latest scheduled nightlies (`gh run download -n audio-quality-run`), which keep every run, and apply the same rule. The replay rows are deterministic, so their ceilings are exactly what they measured, through the end of each recording, with no headroom: any regression fails them, and an improvement should be tightened in. They were last re-measured on 2026-10-22 using version 3 traces with encoded frame durations: AAC-LC uses its fixed 1024 samples, and relay-mic retains Opus packet TOC durations. relay-mic was re-recorded on 2026-11-03 from a real microphone, Chrome 156 on a MacBook Air publishing its built-in microphone through the public relay, recorded by headless Chromium on a Linux host (35 s, minimum RTT 2 ms), which raised its target ceiling from 60 to 80 ms and its silence from 0 to 0.001 as measured. The recording path itself has no added RTT. relay-mic-rtt40 is the 2026-11-03 Chrome row at 40 ms of added RTT on that laptop (180 s, minimum RTT 47 ms), kept as the baseline time stretch must bring to zero. player_underruns_per_min, the player's own counter that its stats panel shows, is budgeted on the replay rows only, exactly as measured on 2026-11-03; it counts the run-dry re-stalls the tap files as stalled time. The local-aac replay's stalled ceiling rises from 0.008 to 0.017 and silence from 0.015 to 0.020 because replay no longer stretches each frame to nearby timestamp gaps; the production player is unchanged. All other changed replay ceilings tighten to the exact measurement. That replay calibration leaves browser and native ceilings unchanged. On 2026-11-07, after the ordered read stopped delivering groups below its cursor (#8071) and took Rust's hole and age bounds (#8218), relay-bbb and relay-mic-rtt40 were re-measured to exactly what they play: relay-bbb's player underruns rise from 12.1 to 16.1/min, stalled from 0.017 to 0.025, and silence from 0.016 to 0.02; relay-mic-rtt40's player underruns rise from 7.9 to 8.2/min while its discarded audio tightens from 229.3 to 70.9 ms/min and its target p95 from 780 to 320 ms. Neither loses audio the previous player heard. With the frames it received but never played removed from the trace, stale backlog that had raised its startup target to 780 ms, the previous player measured relay-mic-rtt40's same 8.2. relay-bbb's extra episode is the ring trimming one AAC frame (23 ms) deeper after the recorder's 10 s backlog burst, which the previous player's out-of-order backfill happened to leave higher; the current read plays every frame there that the previous one did, and more. relay-mic measures its ceilings exactly, so it is unchanged, although the previous player had played it at 4.0 underruns/min for the same reason as relay-bbb. bursty-aac is the 2026-10-18 AAC bursty isolated capture, recorded with the observer subscribed at the decoder's 2 s budget, so it holds every group the player saw. The native rows are provisional: measured on a loaded developer host (x86_64 Linux, the private PulseAudio null sink, seed 7, 60 s a row) on 2026-10-17, three runs of every row from the debug build nightly uses plus three more of the AAC rows from a release build, which played alike. Each ceiling is the worst clean run times 1.25, plus one native hiccup sized from the unimpaired Opus row's worst run (20 underruns/min, 50 gap ms/min, 2.2 skips/min, 110 discarded ms/min, 0.01 stalled, 0.005 silence, one 20 ms bucket of target), rounded up to two significant figures. On that host the null sink's clock ran about 150 ppm slow against the monotonic clock, read within 5 ms at p95 per sample, and its output latency sat between 14 and 28 ms. Five of the 36 AAC rows (none of the 18 Opus rows) voided on the clock: the device stopped pulling for about two seconds mid-row, sometimes after a backend error and a stream restart, which is a native playback defect rather than noise, and the ceilings exclude those runs. Re-measure from the nightly runner's artifacts once it has a week of native rows.", + "note": "The browser rows are measured on the nightly runner (ubuntu-24.04-arm, headless Chromium, seed 7, 60 s a row), from the audio-quality-run artifacts of the seven scheduled Nightly runs on main from 2026-10-29 to 2026-11-04 (40885347517, 41065371094, 41204380716, 41313466339, 41483654426, 41656613559, 41852517544), the first after a fresh subscriber stopped receiving the relay's cached history. The AAC rows predate the TS importer publishing each PES as one group. Each ceiling is the worst of its seven nights times 1.25, plus one runner hiccup, rounded up to two significant figures. The hiccup is sized from the single gaps the 2026-10-08 night put on otherwise clean rows, the fixed-250 control among them: 250 underruns/min, 700 gap ms/min, 3.3 skips/min, 280 discarded ms/min, 0.01 stalled, 0.015 silence, and one 20 ms bucket of target. Budgeted this way, every night passes when it is held out of its row's measurement except one, and that row's own ceiling includes it: the 2026-11-03 Opus bursty row raised its target to 300 ms 2.6 s in and held it for one 20 s half-life before settling at 180. Such startup holds, Opus near-zero's 100 ms on 2026-10-30 and 10-31 among them, are budgeted as measured. These ceilings record today's player as the runner hears it, not target audio quality. To re-measure, download the same artifact from the latest scheduled nightlies (`gh run download -n audio-quality-run`), which keep every run, and apply the same rule. The replay rows are deterministic, so their ceilings are exactly what they measured, through the end of each recording, with no headroom: any regression fails them, and an improvement should be tightened in. They were last re-measured on 2026-10-22 using version 3 traces with encoded frame durations: AAC-LC uses its fixed 1024 samples, and relay-mic retains Opus packet TOC durations. relay-mic was re-recorded on 2026-11-03 from a real microphone, Chrome 156 on a MacBook Air publishing its built-in microphone through the public relay, recorded by headless Chromium on a Linux host (35 s, minimum RTT 2 ms), which raised its target ceiling from 60 to 80 ms and its silence from 0 to 0.001 as measured. The recording path itself has no added RTT. relay-mic-rtt40 is the 2026-11-03 Chrome row at 40 ms of added RTT on that laptop (180 s, minimum RTT 47 ms), kept as the baseline time stretch improves on; like every replay row, its player_underruns_per_min ceiling is what it measures, not a fixed zero. player_underruns_per_min, the player's own counter that its stats panel shows, is budgeted on the replay rows only; it counts the run-dry re-stalls the tap files as stalled time. On 2026-11-07, after the ordered read stopped delivering groups below its cursor (#8071) and took Rust's hole and age bounds (#8218), relay-bbb was re-measured at 16.1 player underruns/min, 0.025 stalled and 0.02 silence, and relay-mic-rtt40 at 8.2, its discarded audio at 70.9 ms/min and its target p95 at 320 ms rather than 780, since the stale backlog it never played no longer reaches the estimator; neither row lost audio the previous player heard. On 2026-11-08 the player began time-stretching live playback with NetEq's accelerate, deciding on the ring's depth right after each arrival and never expanding, and every replay row was re-measured with it, the browser rows left to the nightly runner. Each row's av_lag_ms ceilings, how far the playhead trails the shared clock video plays from, are what it measured: the ring rests one frame ahead of the newest arrival's playout, as the trim it replaces left it, so audio leads the clock by 19 to 23 ms at p50, and any change that settles audio later to buy fewer underruns fails them. Against the trim, the A/V lag p50 fell from 0.7, -17.5, -21.5, -18.2 and -15.6 ms to -22.4 on bursty-aac, -22.2 on local-aac, -22.7 on relay-bbb, -19 on relay-mic and -20.1 on relay-mic-rtt40, and its p95 fell on every row, most on relay-bbb (10.5 to -4.2) and relay-mic-rtt40 (-0.9 to -13.4). Underruns stayed where the trim had them, as the latency they would cost is not spent: player_underruns_per_min held at 4.4, 12.1, 16.1 and 6.1 and rose from 8.2 to 10.6 on relay-mic-rtt40, and relay-mic's tap now files 2 short quanta a minute, 2 ms of missing audio, where it filed none. What the stretch buys is the audio the trim dropped: discarded_ms_per_min fell from 87.8 to 0 on relay-bbb and from 127.5 to 100.6 on local-aac, whose remainder drops nothing: two receive stalls 14 and 15.3 s in leave audio 159 and 132 ms behind the clock, and the fast accelerate that brings it back within half a second steps the playhead's lag the way skip_aheads reads as a skip. player_skips_per_min, the ring's own count of re-anchors that dropped buffered audio, is budgeted beside it, at 0 on every row. Neither replay calibration changed the browser or native ceilings. bursty-aac is the 2026-10-18 AAC bursty isolated capture, recorded with the observer subscribed at the decoder's 2 s budget, so it holds every group the player saw. The native rows are provisional: measured on a loaded developer host (x86_64 Linux, the private PulseAudio null sink, seed 7, 60 s a row) on 2026-10-17, three runs of every row from the debug build nightly uses plus three more of the AAC rows from a release build, which played alike. Each ceiling is the worst clean run times 1.25, plus one native hiccup sized from the unimpaired Opus row's worst run (20 underruns/min, 50 gap ms/min, 2.2 skips/min, 110 discarded ms/min, 0.01 stalled, 0.005 silence, one 20 ms bucket of target), rounded up to two significant figures. On that host the null sink's clock ran about 150 ppm slow against the monotonic clock, read within 5 ms at p95 per sample, and its output latency sat between 14 and 28 ms. Five of the 36 AAC rows (none of the 18 Opus rows) voided on the clock: the device stopped pulling for about two seconds mid-row, sometimes after a backend error and a stream restart, which is a native playback defect rather than noise, and the ceilings exclude those runs. Re-measure from the nightly runner's artifacts once it has a week of native rows.", "rows": [ { "runtime": "chromium", @@ -319,13 +319,16 @@ "rate": 44100, "profile": "bursty-aac", "player_underruns_per_min": 4.4, + "player_skips_per_min": 0, "underruns_per_min": 0, "underrun_ms_per_min": 0, "skip_aheads_per_min": 0, "discarded_ms_per_min": 0, "stalled_share": 0.009, - "silence_share": 0.01, - "target_ms_p95": 343.2 + "silence_share": 0.012, + "target_ms_p95": 343.2, + "av_lag_ms_p50": -22.4, + "av_lag_ms_p95": -22.4 }, { "runtime": "replay", @@ -333,13 +336,16 @@ "rate": 44100, "profile": "local-aac", "player_underruns_per_min": 12.1, + "player_skips_per_min": 0, "underruns_per_min": 0, "underrun_ms_per_min": 0, "skip_aheads_per_min": 2, - "discarded_ms_per_min": 127.5, + "discarded_ms_per_min": 100.6, "stalled_share": 0.017, - "silence_share": 0.02, - "target_ms_p95": 203.2 + "silence_share": 0.021, + "target_ms_p95": 203.2, + "av_lag_ms_p50": -22.2, + "av_lag_ms_p95": -4.2 }, { "runtime": "replay", @@ -347,13 +353,16 @@ "rate": 44100, "profile": "relay-bbb", "player_underruns_per_min": 16.1, + "player_skips_per_min": 0, "underruns_per_min": 0, "underrun_ms_per_min": 0, - "skip_aheads_per_min": 2, - "discarded_ms_per_min": 87.8, + "skip_aheads_per_min": 0, + "discarded_ms_per_min": 0, "stalled_share": 0.025, "silence_share": 0.02, - "target_ms_p95": 223.2 + "target_ms_p95": 223.2, + "av_lag_ms_p50": -22.7, + "av_lag_ms_p95": -4.2 }, { "runtime": "replay", @@ -361,27 +370,33 @@ "rate": 48000, "profile": "relay-mic", "player_underruns_per_min": 6.1, - "underruns_per_min": 0, - "underrun_ms_per_min": 0, + "player_skips_per_min": 0, + "underruns_per_min": 2, + "underrun_ms_per_min": 2, "skip_aheads_per_min": 0, "discarded_ms_per_min": 0, "stalled_share": 0, - "silence_share": 0.001, - "target_ms_p95": 80 + "silence_share": 0.002, + "target_ms_p95": 80, + "av_lag_ms_p50": -19, + "av_lag_ms_p95": -18.5 }, { "runtime": "replay", "codec": "opus", "rate": 48000, "profile": "relay-mic-rtt40", - "player_underruns_per_min": 8.2, - "underruns_per_min": 0.3, + "player_underruns_per_min": 10.6, + "player_skips_per_min": 0, + "underruns_per_min": 0.7, "underrun_ms_per_min": 0.5, "skip_aheads_per_min": 0.3, - "discarded_ms_per_min": 70.9, + "discarded_ms_per_min": 72, "stalled_share": 0.003, - "silence_share": 0.003, - "target_ms_p95": 320 + "silence_share": 0.004, + "target_ms_p95": 320, + "av_lag_ms_p50": -20.1, + "av_lag_ms_p95": -13.4 } ] } diff --git a/test/audio-quality/clients/js/grade.ts b/test/audio-quality/clients/js/grade.ts index 8386ae4bf0e..da5fcde903d 100644 --- a/test/audio-quality/clients/js/grade.ts +++ b/test/audio-quality/clients/js/grade.ts @@ -69,14 +69,17 @@ for (const summary of summaries) { /** The columns a reader scans first: what the listener would have noticed, and the delay paid for it. */ const COLUMNS: [string, string][] = [ ["player underruns/min", "player_underruns_per_min"], + ["player skips/min", "player_skips_per_min"], ["underruns/min", "underruns_per_min"], ["short/min", "short_quanta_per_min"], ["episodes/min", "underrun_episodes_per_min"], ["gap ms/min", "underrun_ms_per_min"], - ["skips/min", "skip_aheads_per_min"], + ["skip-aheads/min", "skip_aheads_per_min"], ["discarded ms/min", "discarded_ms_per_min"], ["stalled", "stalled_share"], ["silence", "silence_share"], + ["A/V lag p50", "av_lag_ms_p50"], + ["A/V lag p95", "av_lag_ms_p95"], ["target p95", "target_ms_p95"], ["converge ms", "converge_ms_last"], ]; diff --git a/test/audio-quality/clients/js/replay.ts b/test/audio-quality/clients/js/replay.ts index fb7d0355480..16cf26cd019 100644 --- a/test/audio-quality/clients/js/replay.ts +++ b/test/audio-quality/clients/js/replay.ts @@ -84,6 +84,17 @@ const out = values.out as string; // Loaded only to play: listing runs before `run.sh` has installed the workspace the player needs. const { replay } = await import("../../../../js/watch/src/audio/replay.ts"); +/** What a sample reads from the latest quantum. */ +interface Reading { + at: number; + timestamp: number; + clock?: number; + stalled: boolean; + delay: number; + underruns: number; + skips: number; +} + for (const { row, trace } of rows) { const tag = rowKey(row); const rate = row.rate; @@ -103,15 +114,17 @@ for (const { row, trace } of rows) { }; const arrivals = trace.arrivals.map(([at, timestamp, group, duration]) => ({ at, timestamp, group, duration })); - const sample = (quantum: { at: number; timestamp: number; stalled: boolean; delay: number; underruns: number }) => { + const sample = (quantum: Reading) => { const gaps = ledger.take(); samples.push({ at: quantum.at, render: quantum.at, timestamp: quantum.timestamp, + clock: quantum.clock, stalled: quantum.stalled, delay: quantum.delay, playerUnderruns: quantum.underruns, + playerSkips: quantum.skips, quanta: ledger.counts.quanta, quiet: ledger.counts.quiet, gaps: gaps.length > 0 ? gaps : undefined, @@ -126,7 +139,7 @@ for (const { row, trace } of rows) { config: trace.config as unknown as Catalog.AudioConfig, duration: trace.duration, }); - let last: { at: number; timestamp: number; stalled: boolean; delay: number; underruns: number } | undefined; + let last: Reading | undefined; for await (const quantum of quanta) { // The render clock is the simulated one: the trace's `at`, which starts at zero. ledger.add(frame, quantum.output.length, classify([quantum.output])); @@ -138,9 +151,11 @@ for (const { row, trace } of rows) { last = { at: quantum.at, timestamp: quantum.timestamp, + clock: quantum.clock, stalled: quantum.stalled, delay: quantum.delay, underruns: quantum.underruns, + skips: quantum.skips, }; if (quantum.at >= next) { diff --git a/test/audio-quality/clients/js/src/analyze.test.ts b/test/audio-quality/clients/js/src/analyze.test.ts index 596e8c6852e..05b3683eb87 100644 --- a/test/audio-quality/clients/js/src/analyze.test.ts +++ b/test/audio-quality/clients/js/src/analyze.test.ts @@ -104,6 +104,24 @@ describe("analyze", () => { expect(analyze(input(clean())).metrics.player_underruns_total).toBeNull(); }); + test("the player's own skips are its counter's rise across the window", () => { + const samples = clean().map((s, i) => ({ ...s, playerSkips: i < 100 ? 1 : 3 })); + expect(analyze(input(samples)).metrics).toMatchObject({ player_skips_total: 2 }); + expect(analyze(input(clean())).metrics.player_skips_total).toBeNull(); + }); + + test("the A/V lag is how far the playhead trails the shared clock while playing", () => { + // Three in four samples 20 ms ahead of the clock, the rest 10 ms; a stalled sample's playhead is + // frozen, so it is left out however far the clock has run. + const samples = clean().map((s, i) => { + const timestamp = s.timestamp ?? 0; + if (i === 50) return { ...s, stalled: true, clock: timestamp + 500 }; + return { ...s, clock: timestamp - 20 + (i % 4 === 3 ? 10 : 0) }; + }); + expect(analyze(input(samples)).metrics).toMatchObject({ av_lag_ms_p50: -20, av_lag_ms_p95: -10 }); + expect(analyze(input(clean())).metrics.av_lag_ms_p50).toBeNull(); + }); + test("a gap that is the ring stalling is not an underrun", () => { const samples = clean(); samples[100] = { diff --git a/test/audio-quality/clients/js/src/analyze.ts b/test/audio-quality/clients/js/src/analyze.ts index b3cc7b3ac9d..6d97a51aa71 100644 --- a/test/audio-quality/clients/js/src/analyze.ts +++ b/test/audio-quality/clients/js/src/analyze.ts @@ -202,10 +202,13 @@ export function analyze(input: Input): Summary { const silence = quanta > 0 ? ((lastTap?.quiet ?? 0) - (firstTap?.quiet ?? 0)) / quanta : null; if (!lastTap) notes.push("tap: never reported, so underruns and silence are unmeasured"); - // The player's own counter is cumulative, so the window's count is its rise across the window. - const counted = window.filter((s) => s.playerUnderruns !== undefined); - const playerUnderruns = - counted.length > 0 ? (counted.at(-1)?.playerUnderruns ?? 0) - (counted[0]?.playerUnderruns ?? 0) : null; + // The player's own counters are cumulative, so the window's count is each one's rise across it. + const rise = (counter: (s: Sample) => number | undefined) => { + const counted = window.map(counter).filter((n) => n !== undefined); + return counted.length > 0 ? (counted.at(-1) ?? 0) - (counted[0] ?? 0) : null; + }; + const playerUnderruns = rise((s) => s.playerUnderruns); + const playerSkips = rise((s) => s.playerSkips); // ── the playhead ──────────────────────────────────────────────────────── // The lag drifts when the media timeline does not run at wall rate, a property of the source @@ -226,6 +229,8 @@ export function analyze(input: Input): Summary { s.timestamp === undefined || s.stalled ? undefined : s.at - s.timestamp - (mediaRate ?? 0) * (s.at - base), ); const skips = skipAheads(lags); + // Both on the media clock, read at one instant, so no reduction to the viewer's is needed. + const behind = playing.flatMap((s) => (s.clock === undefined ? [] : [s.clock - (s.timestamp as Ms)])); const stalledShare = window.length > 0 ? window.filter((s) => s.stalled).length / window.length : null; // ── the target ────────────────────────────────────────────────────────── @@ -258,12 +263,16 @@ export function analyze(input: Input): Summary { underrun_ms_max: lastTap ? round(Math.max(0, ...gaps.map((g) => g.ms))) : null, player_underruns_total: playerUnderruns, player_underruns_per_min: playerUnderruns === null ? null : perMin(playerUnderruns), + player_skips_total: playerSkips, + player_skips_per_min: playerSkips === null ? null : perMin(playerSkips), skip_aheads_total: skips.count, skip_aheads_per_min: perMin(skips.count), discarded_ms_total: round(skips.ms), discarded_ms_per_min: perMin(skips.ms), stalled_share: round(stalledShare, 3), silence_share: round(silence, 3), + av_lag_ms_p50: round(percentile(behind, 50)), + av_lag_ms_p95: round(percentile(behind, 95)), target_ms_p50: round(percentile(targets, 50)), target_ms_p95: round(percentile(targets, 95)), target_ms_max: round(percentile(targets, 100)), diff --git a/test/audio-quality/clients/js/src/probe.test.ts b/test/audio-quality/clients/js/src/probe.test.ts index 0cac89a67ef..0b642f88e63 100644 --- a/test/audio-quality/clients/js/src/probe.test.ts +++ b/test/audio-quality/clients/js/src/probe.test.ts @@ -23,7 +23,7 @@ test("audio-only sampling holds probe demand until its final RTT sample", async out: { stalled: new Signal(false), context: empty, root: empty, timestamp: empty, stats: empty }, source: { out: { config: empty } }, }, - sync: { out: { delay: empty, jitter: empty } }, + sync: { out: { delay: empty, jitter: empty }, now: () => undefined }, broadcast: { out: { catalog: empty } }, }, } as unknown as MoqWatch; diff --git a/test/audio-quality/clients/js/src/probe.ts b/test/audio-quality/clients/js/src/probe.ts index c8ae79c7e24..412fde6ecc7 100644 --- a/test/audio-quality/clients/js/src/probe.ts +++ b/test/audio-quality/clients/js/src/probe.ts @@ -126,8 +126,10 @@ export function probe(watch: MoqWatch, capture = false): Probe { render: context ? context.currentTime * 1000 : undefined, timestamp: audio.out.timestamp.peek(), stalled: audio.out.stalled.peek(), + clock: sync.now(), delay: sync.out.delay.peek(), playerUnderruns: audio.out.stats.peek()?.underruns, + playerSkips: audio.out.stats.peek()?.skips, rtt: watch.connection.probe.peek()?.rtt, jitter: audio.source.out.config.peek()?.jitter, renditionDelay: audio.source.out.config.peek()?.delay, diff --git a/test/audio-quality/clients/js/src/schema.ts b/test/audio-quality/clients/js/src/schema.ts index 0ce5d6631fb..83fa67dd1e7 100644 --- a/test/audio-quality/clients/js/src/schema.ts +++ b/test/audio-quality/clients/js/src/schema.ts @@ -104,6 +104,13 @@ export const METRICS = { description: "The player's own count, which its stats panel shows: each time the ring ran dry mid-playback and re-stalled to refill, plus each quantum it filled only part-way. `underruns` leaves out the run-dry ones, since the quanta after them render while stalled.", }, + player_skips: { + unit: "count", + clock: "render", + aggregations: ["total", "per_min"], + description: + "The player's own count of skips: each jump that dropped buffered audio rather than playing it faster. `skip_aheads` infers them from the playhead, which reads a sustained time stretch as one too.", + }, skip_aheads: { unit: "count", clock: "viewer", @@ -132,6 +139,13 @@ export const METRICS = { description: "Share of rendered quanta under the silence floor at the player's output, whatever the cause. The published tone is never quiet, so any of it is a gap or a stall.", }, + av_lag_ms: { + unit: "ms", + clock: "media", + aggregations: ["p50", "p95"], + description: + "How far the audio playhead trails the shared clock video plays from (`sync.now()`), sampled while playing: latency audio adds beyond the resolved target. Negative is audio ahead. Unset natively.", + }, target_ms: { unit: "ms", clock: "viewer", @@ -383,6 +397,8 @@ export type Sample = { * is only a sink with nothing written yet, or none at all. */ stalled?: boolean; + /** `sync.now()`: where the shared clock puts video, on the `media` clock. Unset natively. */ + clock?: Ms; /** `sync.out.delay`: the resolved playout target. Natively, the target the speaker is held on. */ delay?: Ms; /** @@ -390,6 +406,11 @@ export type Sample = { * Unset natively. */ playerUnderruns?: number; + /** + * `audio.out.stats.skips`, cumulative: the player's own count of jumps that dropped buffered audio + * rather than playing it faster. Unset natively. + */ + playerSkips?: number; /** The current connection PROBE round trip. */ rtt?: Ms; /** The selected rendition's advertised jitter at this sample. */