From 9308ad3661a228f57ca225bd5994c8d434caeb4b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 09:26:57 -0800 Subject: [PATCH 1/8] chore(quest): claim quest/m1/audio-jitter-target/native-stretch.md Quest-Claim: a8377dede0887977 From f155190999e946dbbd1f00cda43b34b30d184abf Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 09:34:26 -0800 Subject: [PATCH 2/8] chore(quest): grade the first four native nightly nights of the stretch Co-Authored-By: Claude Opus 5.5 --- .../m1/audio-jitter-target/native-stretch.md | 69 ++++++++++++++----- test/audio-quality/README.md | 4 +- 2 files changed, 56 insertions(+), 17 deletions(-) diff --git a/quest/m1/audio-jitter-target/native-stretch.md b/quest/m1/audio-jitter-target/native-stretch.md index 69461a8e24b..955f4fa4ddd 100644 --- a/quest/m1/audio-jitter-target/native-stretch.md +++ b/quest/m1/audio-jitter-target/native-stretch.md @@ -45,19 +45,45 @@ capacity watermark and an incoming excess at the fast threshold trim. Remaining verification: - Native `test/audio-quality` rows must show zero steady-state underruns - once their targets settle. Keep a failed or void row explicit and fix - any implementation cause before calling this complete. + once their targets settle on the remaining nights too. Keep a failed or + void row explicit and fix any implementation cause. - Re-measure the provisional native budgets from seven scheduled nightly artifacts using `budgets.json`'s rule: worst clean run times 1.25, plus its documented native hiccup, rounded to two significant figures. A - developer run is not that calibration. -- Run `just rs bench-playback` on a quiet host with no other agents. It - now sweeps both sink count and callback size with and without stretch, - and the nightly `rust-test` benchmark job (`sh/gh/nightly.sh`) already - runs it. A loaded-host smoke run is not a timing acceptance claim. -- Listen to real speech with 2 ms/s drift and the measured expansions on - a real output device (below). Deterministic PCM and allocation checks do - not establish perceptual acceptance. + developer run is not that calibration. The first four nights are graded + below; the claim PR's body keeps every native row's metrics, so an + expired artifact loses nothing. +- Settle whether the nightly runner's `just rs bench-playback` is the + quiet-host timing run (Waiting). A loaded-host smoke run is not a timing + acceptance claim. + +Graded 2026-11-08: the scheduled Nightlies of 2026-11-05 to 11-08 on +`main` (42050683248, 42236107954, 42377867523, 42487375545) are the first +four with this implementation; 11-04's ran at 14:56 UTC, before it merged +at 17:31. Every night has all 12 native rows and none voided. Eleven rows +read zero underruns every night. Opus `step` reads one 22 to 44 ms gap on +three nights, at 29.6 s, when the shaper's one-way delay steps from 5 to +60 ms while the settled target is 60 ms: a late arrival no stretch can +cover without concealment, played clean once the target rises to 100 ms. +The scheduled Nightlies of 11-02 to 11-04, before this implementation, +read up to 23.9 underruns a minute on AAC `wide` and 8.6 on Opus `wide`. + +The stretch adds skip-aheads the harness did not see before: up to 3.3 a +minute and 232 ms discarded on Opus `bursty`, 2.2 and 181 ms on AAC +`near-zero` and `step` on 11-08. Each is the accelerate shedding a lowered +target or a startup overfill at about 8% speed for 1 to 4 s, with no single +sample's media advancing more than 33 ms past real time. The harness counts +any shed of more than 40 ms across nine samples as a skip, which its README +now says. Two rows failed their provisional ceilings that way: Opus +`bursty` on 11-07 (232 > 170 ms discarded, and a 380 ms startup target hold +> 350) and AAC `step` on 11-08 (157.5 > 110). The rest of the job's red is +outside this work: `raw-audio-bench` on 11-06, `qmux-bench` on 11-08, and +browser and replay rows. + +The nightly `rust-test` benchmark job ran `just rs bench-playback` and +passed on all four nights, alone on a GitHub `ubuntu-24.04-arm` runner. The +stretch costs 16 to 43% per callback, stable across nights; its worst cell, +64 sinks at 2048 frames, is 1.02 ms of a 42.7 ms period. Decided 2026-11-04 (maintainer, on [moq-dev/moq#7856](https://github.com/moq-dev/moq/pull/7856)): accept the @@ -101,12 +127,23 @@ bench waits below remain. ## Waiting -- moq-dev/moq: collect seven scheduled native nightly rows for this - implementation, then apply the existing calibration rule. Keep the - current provisional budgets until those artifacts exist. -- moq-dev/moq: run `just rs bench-playback` on a quiet host. On - 2026-11-04 `nixos`, `nixos2`, and `kixtop` all ran agents and `desktop` - refused SSH. +- moq-dev/moq: three more scheduled Nightlies with this implementation, due + 2026-11-09 to 11-11. Find them with `gh run list --workflow nightly.yml + --event schedule`, download each with `gh run download -n + audio-quality-run` within its 7 days, add its native rows to the claim PR, + and apply the calibration rule once seven exist. Never dispatch the + nightly by hand. +- Maintainer: should the calibration budget the accelerate's skip-aheads? + Only a maintainer can say whether an inaudible accelerate is the + discarded audio these rows guard. Options: keep the metric and let the + seven nights size it (recommended: it is literal discarded media, and the + ceiling still catches a regression that sheds more); or have `moq play` + report its own trimmed and accelerated audio and budget trims alone, a + new metric needing seven more nights. +- Maintainer: is the nightly runner's `bench-playback` (above) the + quiet-host run? Only a maintainer can judge that host. Recommended yes: it + is the same command, run alone on a dedicated runner, stable across four + nights. Otherwise name a quiet host to run it on. ## Related diff --git a/test/audio-quality/README.md b/test/audio-quality/README.md index c1b91d7553e..2f75abfadbf 100644 --- a/test/audio-quality/README.md +++ b/test/audio-quality/README.md @@ -288,7 +288,9 @@ player's own count, which its stats panel shows, is read alongside: - `underrun_ms`: the missing audio, summed, and the longest episode as `max`. - `skip_aheads`: re-anchors that discarded buffered audio, seen as a forward step of more than 40 ms in the lag between the viewer clock and the playhead (judged on the median of four samples either - side, after removing the media clock's drift). + side, after removing the media clock's drift). A time stretch that accelerates onto a lower target + by more than 40 ms across those nine samples counts too, though it re-anchors nothing: `moq play` + accelerates about 8%, so shedding more than about 40 ms reads as one skip or more. - `discarded_ms`: the media those skip-aheads jumped over. Audio dropped before it reached the ring moves no playhead, so it is not counted. - `stalled`: the share of the window the ring spent stalled, refilling rather than playing. A gap From 005469779f241ad78d7977b75e1665bf6bd2f9f6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 18:35:41 -0800 Subject: [PATCH 3/8] feat(moq-audio,moq-cli,watch): players report their own trimmed and accelerated audio moq play --metrics and @moq/watch's audio stats now count received audio they discarded to bound their delay, and moq play counts what its time stretch shed by accelerating, so the audio-quality calibration can budget trims alone instead of the analyzer's discarded_ms, which counts the inaudible accelerate. Co-Authored-By: Claude Opus 5.5 --- doc/bin/cli.md | 5 +- js/watch/src/audio/buffer.ts | 5 ++ js/watch/src/audio/decoder.ts | 20 +++++++- js/watch/src/audio/render-worklet.test.ts | 2 + js/watch/src/audio/render-worklet.ts | 1 + js/watch/src/audio/render.ts | 2 + js/watch/src/audio/replay.ts | 3 ++ js/watch/src/audio/ring-buffer.test.ts | 1 + js/watch/src/audio/ring-buffer.ts | 19 ++++++- .../m1/audio-jitter-target/native-stretch.md | 51 ++++++++++--------- quest/m1/audio-jitter-target/watch.md | 7 +++ rs/moq-audio/src/playback/mixer.rs | 38 ++++++++++++++ rs/moq-audio/src/playback/observe.rs | 28 ++++++++++ rs/moq-audio/src/ring.rs | 21 ++++++++ rs/moq-cli/src/play/args.rs | 7 +-- rs/moq-cli/src/play/headless.rs | 5 ++ rs/moq-cli/src/play/media.rs | 6 ++- rs/moq-cli/src/play/metrics.rs | 48 +++++++++++++---- rs/moq-cli/src/play/window.rs | 5 +- test/audio-quality/README.md | 8 +++ test/audio-quality/clients/js/grade.ts | 2 + test/audio-quality/clients/js/replay.ts | 10 +++- test/audio-quality/clients/js/src/analyze.ts | 17 +++++-- test/audio-quality/clients/js/src/probe.ts | 1 + test/audio-quality/clients/js/src/schema.ts | 23 ++++++++- 25 files changed, 285 insertions(+), 50 deletions(-) diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 5a46af26322..3892f850cf1 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -189,10 +189,11 @@ default, `auto`, sizes the buffer to how unevenly audio arrives, with the same delay and is also how long a stalled group is waited on before it is skipped. Each role follows the catalog, switching rendition when the publisher retires the one playing. With `--video off`, `--metrics ` writes a JSON line every -250 ms with the playhead, the delay, and every gap and stall the speaker hit, in +250 ms with the playhead, the delay, every gap and stall the speaker hit, and +the received audio it trimmed to bound its delay or shed by playing faster, in the schema the [audio quality harness](https://github.com/moq-dev/moq/tree/main/test/audio-quality) -reduces to underruns, skips, and silence. A silence the publisher declares, such +reduces to underruns, trims, and silence. A silence the publisher declares, such as a push-to-talk mute, plays as comfort noise at the level of the quietest audio heard before it, and counts as no gap or stall. The metrics total that comfort noise and name the media timestamp each utterance after it resumes from. diff --git a/js/watch/src/audio/buffer.ts b/js/watch/src/audio/buffer.ts index 446e3c31e44..c654ca56ebd 100644 --- a/js/watch/src/audio/buffer.ts +++ b/js/watch/src/audio/buffer.ts @@ -270,6 +270,10 @@ export class AudioBuffer { /** How many times the ring has run dry mid-playback, cumulative. */ readonly underruns: Getter = this.#underruns; + readonly #trimmed = new Signal(Milli.zero); + /** Buffered audio the ring dropped unplayed to stay within its bound, cumulative. */ + readonly trimmed: Getter = this.#trimmed; + readonly #silent = new Signal(false); /** Whether playback is at a declared end, reached or announced before any audio, and nothing has played since. */ readonly silent: Getter = this.#silent; @@ -326,6 +330,7 @@ export class AudioBuffer { this.#timestamp.set(data.timestamp); this.#stalled.set(data.stalled); this.#underruns.set(data.underruns); + this.#trimmed.set(data.trimmed); this.#silent.set(data.silent); if (data.epoch === this.#epoch) { this.#backpressure.observe(data.anchored ? data.timestamp : undefined, data.stalled); diff --git a/js/watch/src/audio/decoder.ts b/js/watch/src/audio/decoder.ts index 748ca269d7c..7534d371e72 100644 --- a/js/watch/src/audio/decoder.ts +++ b/js/watch/src/audio/decoder.ts @@ -98,6 +98,8 @@ export interface Stats { bytesReceived: number; /** How many times playback ran dry mid-stream; a rising count means the delay is too low. */ underruns: number; + /** Buffered audio dropped unplayed to keep playback at its delay. */ + trimmed: Milli; /** Frames that arrived after their playout time. */ late: number; /** Renditions that failed to decode. */ @@ -111,7 +113,15 @@ export interface Stats { } // The running totals before anything is counted. -const EMPTY: Stats = { bytesReceived: 0, underruns: 0, late: 0, errors: 0, stalls: 0, stalled: Milli.zero }; +const EMPTY: Stats = { + bytesReceived: 0, + underruns: 0, + trimmed: Milli.zero, + late: 0, + errors: 0, + stalls: 0, + stalled: Milli.zero, +}; // What a decode attempt reads, and where its counts go. type Attempt = { config: DecoderConfig; failure: Signal; path: Moq.Path.Valid; key: Key }; @@ -424,6 +434,14 @@ export class Decoder { const key = phase.kind === "running" ? phase.key : undefined; this.#count(key, { underruns: added }, (stats) => ({ underruns: stats.underruns + added })); }); + let trimmed = Milli.zero; + effect.run((inner) => { + const total = inner.get(ring.trimmed); + const added = total - trimmed; + trimmed = total; + if (added <= 0) return; + this.#count(undefined, {}, (stats) => ({ trimmed: Milli(stats.trimmed + added) })); + }); effect.set(this.#out.root, worklet); }); diff --git a/js/watch/src/audio/render-worklet.test.ts b/js/watch/src/audio/render-worklet.test.ts index 044e1e7c960..c512d7559d2 100644 --- a/js/watch/src/audio/render-worklet.test.ts +++ b/js/watch/src/audio/render-worklet.test.ts @@ -85,6 +85,7 @@ describe("render dispatch", () => { stalled: false, anchored: true, underruns: 0, + trimmed: 0, silent: false, }, ]); @@ -113,6 +114,7 @@ describe("render dispatch", () => { stalled: false, anchored: true, underruns: 0, + trimmed: 0, silent: false, }, ]); diff --git a/js/watch/src/audio/render-worklet.ts b/js/watch/src/audio/render-worklet.ts index 230dba0e782..c0f786e166f 100644 --- a/js/watch/src/audio/render-worklet.ts +++ b/js/watch/src/audio/render-worklet.ts @@ -100,6 +100,7 @@ class Render extends AudioWorkletProcessor { stalled: ring.stalled, anchored: ring.anchored, underruns: ring.underruns, + trimmed: ring.trimmed, silent: ring.silent, epoch: this.#epoch, }; diff --git a/js/watch/src/audio/render.ts b/js/watch/src/audio/render.ts index f60f1612000..80db2eb25fe 100644 --- a/js/watch/src/audio/render.ts +++ b/js/watch/src/audio/render.ts @@ -62,6 +62,8 @@ export interface State { anchored: boolean; // How many times the ring has run dry mid-playback, cumulative. underruns: number; + // Buffered audio the ring dropped unplayed to stay within its bound, cumulative. + trimmed: Milli; // Whether playback is at a declared end, reached or announced before any audio, and nothing has played since. silent: boolean; // The epoch of the last reset or park the worklet applied. diff --git a/js/watch/src/audio/replay.ts b/js/watch/src/audio/replay.ts index 8e9cc3f172a..591cd23f5c8 100644 --- a/js/watch/src/audio/replay.ts +++ b/js/watch/src/audio/replay.ts @@ -85,6 +85,8 @@ export interface Quantum { delay: number; /** The ring's cumulative underruns, as the player's stats count them: each run-dry and each short quantum. */ underruns: number; + /** The buffered audio the ring has dropped to stay within its bound, cumulative, in ms. */ + trimmed: number; } /** Samples as the milliseconds `AudioBuffer` posts to the worklet. */ @@ -279,6 +281,7 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato timestamp: Milli.fromMicro(ring.timestamp), delay, underruns: ring.underruns, + trimmed: ring.trimmed, }; } } finally { diff --git a/js/watch/src/audio/ring-buffer.test.ts b/js/watch/src/audio/ring-buffer.test.ts index 238dd6144d5..f9a8f11a3ae 100644 --- a/js/watch/src/audio/ring-buffer.test.ts +++ b/js/watch/src/audio/ring-buffer.test.ts @@ -293,6 +293,7 @@ describe("ring buffer wrapping", () => { // Past the band: drop back to the target, wrapping abs 200-249 onto slots 0-49. write(buffer, 200 as Milli, 50, { channels: 1, value: 4.0 }); expect(buffer.length).toBe(100); + expect(buffer.trimmed).toBe(100); // Read all 100 samples: abs 150-199 = 3.0, abs 200-249 = 4.0. const output2 = read(buffer, 100, 1); diff --git a/js/watch/src/audio/ring-buffer.ts b/js/watch/src/audio/ring-buffer.ts index 3afd723d588..8edd736f12c 100644 --- a/js/watch/src/audio/ring-buffer.ts +++ b/js/watch/src/audio/ring-buffer.ts @@ -1,4 +1,4 @@ -import { Micro, type Milli, Second } from "@moq/clock"; +import { Micro, Milli, Second } from "@moq/clock"; import { ringCapacity } from "./latency"; export class AudioRingBuffer { @@ -12,6 +12,8 @@ export class AudioRingBuffer { // Held stalled for the part of the target above the floor, until the next floor update. #parked = false; #underruns = 0; + // Samples of buffered audio dropped to bound the ring, cumulative. + #trimmed = 0; // Where the publisher declared its audio ends, as a sample index, until the next reset. #end?: number; @@ -85,6 +87,11 @@ export class AudioRingBuffer { return this.#underruns; } + /** Buffered audio the ring dropped unplayed to stay within its bound, cumulative. */ + get trimmed(): Milli { + return Milli.fromSecond((this.#trimmed / this.rate) as Second); + } + /** Whether playback is at a declared end, reached or announced before any audio, and nothing has played since. */ get silent(): boolean { return this.#silent; @@ -198,7 +205,15 @@ export class AudioRingBuffer { const band = playing && (this.#tail === undefined || this.#readIndex >= this.#tail); const limit = band ? Math.min(this.#latencySamples + this.#chunk, this.capacity) : this.capacity; if (end - this.#readIndex > limit) { - this.#readIndex = end - (band ? Math.min(this.#latencySamples, this.capacity) : this.capacity); + const readIndex = end - (band ? Math.min(this.#latencySamples, this.capacity) : this.capacity); + // Only written audio counts: not a discontinuity's gap, nor a declared silence's zeros. + const dropped = Math.min(readIndex, this.#writeIndex); + let trimmed = Math.max(0, dropped - this.#readIndex); + for (const pause of this.#pauses) { + trimmed -= Math.max(0, Math.min(pause.end, dropped) - Math.max(pause.start, this.#readIndex)); + } + this.#trimmed += trimmed; + this.#readIndex = readIndex; this.#prune(); } diff --git a/quest/m1/audio-jitter-target/native-stretch.md b/quest/m1/audio-jitter-target/native-stretch.md index 955f4fa4ddd..bd7033be718 100644 --- a/quest/m1/audio-jitter-target/native-stretch.md +++ b/quest/m1/audio-jitter-target/native-stretch.md @@ -48,14 +48,14 @@ Remaining verification: once their targets settle on the remaining nights too. Keep a failed or void row explicit and fix any implementation cause. - Re-measure the provisional native budgets from seven scheduled nightly - artifacts using `budgets.json`'s rule: worst clean run times 1.25, plus + artifacts that carry `moq play`'s own `trimmed` and `accelerated` + counters, using `budgets.json`'s rule: worst clean run times 1.25, plus its documented native hiccup, rounded to two significant figures. A - developer run is not that calibration. The first four nights are graded - below; the claim PR's body keeps every native row's metrics, so an - expired artifact loses nothing. -- Settle whether the nightly runner's `just rs bench-playback` is the - quiet-host timing run (Waiting). A loaded-host smoke run is not a timing - acceptance claim. + developer run is not that calibration. Native rows then budget + `trimmed_ms_per_min` in place of `skip_aheads_per_min` and + `discarded_ms_per_min`, which count the accelerate too; the browser rows + keep theirs until [Watch](/quest/m1/audio-jitter-target/watch.md) + stretches too. Graded 2026-11-08: the scheduled Nightlies of 2026-11-05 to 11-08 on `main` (42050683248, 42236107954, 42377867523, 42487375545) are the first @@ -84,6 +84,19 @@ The nightly `rust-test` benchmark job ran `just rs bench-playback` and passed on all four nights, alone on a GitHub `ubuntu-24.04-arm` runner. The stretch costs 16 to 43% per callback, stable across nights; its worst cell, 64 sinks at 2048 frames, is 1.02 ms of a 42.7 ms period. +[moq-dev/moq#8308](https://github.com/moq-dev/moq/pull/8308)'s body keeps +every graded native row's metrics. + +Decided 2026-11-09 (maintainer, on +[moq-dev/moq#8308](https://github.com/moq-dev/moq/pull/8308#issuecomment-6730322286)): + +- ✅ `moq play --metrics` reports its own trimmed and accelerated audio, + and the calibration budgets trims alone (rejected: keep budgeting + accelerates as `discarded_ms`). An accelerate is inaudible, so it is not + the discarded audio these rows guard. #8308 adds the counters, and the + browser player's `audio.out.stats.trimmed` for parity. +- ✅ The nightly runner's `bench-playback` counts as the quiet-host run + (rejected: naming another host). That settles the timing check. Decided 2026-11-04 (maintainer, on [moq-dev/moq#7856](https://github.com/moq-dev/moq/pull/7856)): accept the @@ -127,23 +140,13 @@ bench waits below remain. ## Waiting -- moq-dev/moq: three more scheduled Nightlies with this implementation, due - 2026-11-09 to 11-11. Find them with `gh run list --workflow nightly.yml - --event schedule`, download each with `gh run download -n - audio-quality-run` within its 7 days, add its native rows to the claim PR, - and apply the calibration rule once seven exist. Never dispatch the - nightly by hand. -- Maintainer: should the calibration budget the accelerate's skip-aheads? - Only a maintainer can say whether an inaudible accelerate is the - discarded audio these rows guard. Options: keep the metric and let the - seven nights size it (recommended: it is literal discarded media, and the - ceiling still catches a regression that sheds more); or have `moq play` - report its own trimmed and accelerated audio and budget trims alone, a - new metric needing seven more nights. -- Maintainer: is the nightly runner's `bench-playback` (above) the - quiet-host run? Only a maintainer can judge that host. Recommended yes: it - is the same command, run alone on a dedicated runner, stable across four - nights. Otherwise name a quiet host to run it on. +- moq-dev/moq: seven scheduled Nightlies on a `main` that includes + [moq-dev/moq#8308](https://github.com/moq-dev/moq/pull/8308), whose + native rows report `trimmed_ms`. Find them with `gh run list --workflow + nightly.yml --event schedule`, download each with `gh run download + -n audio-quality-run` within its 7 days, keep its native rows in the + calibration PR's body, and apply the calibration rule once seven exist. + Never dispatch the nightly by hand. ## Related diff --git a/quest/m1/audio-jitter-target/watch.md b/quest/m1/audio-jitter-target/watch.md index e2c0409ff1a..cb4d7218e32 100644 --- a/quest/m1/audio-jitter-target/watch.md +++ b/quest/m1/audio-jitter-target/watch.md @@ -140,6 +140,13 @@ the replay numbers that justify it. - Accelerate replaces the skip-ahead when the ring sits above the high limit. A re-stall remains for a ring that empties despite the stretch, and a skip-ahead remains only above what fast accelerate can close. +- Report what the stretch sheds as `audio.out.stats.accelerated`, beside the + `trimmed` the ring already reports, and emit it as the harness `Sample`'s + `accelerated`, as `moq play --metrics` does. The analyzer's `skip_aheads` + counts an accelerate past 40 ms, so the browser rows then budget + `trimmed_ms_per_min` in place of `skip_aheads_per_min` and + `discarded_ms_per_min`, as the native rows will (decided 2026-11-09 on + [moq-dev/moq#8308](https://github.com/moq-dev/moq/pull/8308#issuecomment-6730322286)). - Verification: `relay-mic-rtt40` reaches 0 `player_underruns_per_min` and its budget is tightened to the new measurement. No other replay row regresses, and the nightly browser rows are re-measured by the budgets note's rule. A diff --git a/rs/moq-audio/src/playback/mixer.rs b/rs/moq-audio/src/playback/mixer.rs index 8b75b8abd95..80066732b1e 100644 --- a/rs/moq-audio/src/playback/mixer.rs +++ b/rs/moq-audio/src/playback/mixer.rs @@ -126,6 +126,10 @@ pub(super) struct Entry { applied: f32, /// [`ring::Consumer::comfort`] as of the last read. comfort: u64, + /// [`ring::Consumer::trimmed`] as of the last read. + trimmed: u64, + /// [`ring::Consumer::accelerated`] as of the last read. + accelerated: u64, } /// Something the audio thread is done with and must not free itself. @@ -256,6 +260,8 @@ impl Mixer { let entry = Entry { id, comfort: cons.comfort(), + trimmed: cons.trimmed(), + accelerated: cons.accelerated(), cons, gain, completion, @@ -327,6 +333,8 @@ impl Mixer { let mut tail = 0; // The most comfort noise any sink still being written played. let mut comfort = 0; + // What those sinks discarded, summed. + let (mut trimmed, mut accelerated) = (0, 0); for entry in entries.iter_mut() { // Read even at zero volume: this is live audio, so a muted sink @@ -345,6 +353,14 @@ impl Mixer { }; let comforted = entry.cons.comfort(); let filled = comforted - std::mem::replace(&mut entry.comfort, comforted); + let cut = entry.cons.trimmed(); + let cut = cut - std::mem::replace(&mut entry.trimmed, cut); + let sped = entry.cons.accelerated(); + let sped = sped - std::mem::replace(&mut entry.accelerated, sped); + if entry.completion.writing() { + trimmed += cut as usize; + accelerated += sped as usize; + } match read { Some(read) if entry.completion.writing() => { playing = true; @@ -383,6 +399,7 @@ impl Mixer { (playing, supplied) = (true, tail); } ledger.add(&bus[..samples], channels, supplied, playing, comfort); + ledger.shed(trimmed, accelerated); // Tap the mix on its way out. Overflow means the microphone stopped // draining, which the canceller notices on its own; there is nothing @@ -918,4 +935,25 @@ mod tests { observation.comfort ); } + + /// A queue trimmed back to its latency is reported as discarded audio, + /// distinct from what the stretch sheds. + #[test] + fn a_trim_is_timed_as_discarded_audio() { + let mut harness = Harness::new(Layout::Stereo); + let mut observer = observe::Observer::new(harness.observe.clone()).unwrap(); + let mut out = vec![0.0f32; 512 * STEREO]; + + let (_, mut prod) = harness.add(Arc::new(Gain::new())); + harness.fill(&mut out[..STEREO]); + // Past three quarters of the second of headroom above the 10 ms latency. + push(&mut prod, 0.5, 40_000); + harness.fill(&mut out); + + let observation = observer.read().unwrap(); + // The 10 ms prime and the push, less the buffer read and the latency kept. + let trimmed = 480 + 40_000 - 512 - 480; + assert_eq!(observation.trimmed, Duration::from_secs_f64(trimmed as f64 / RATE as f64)); + assert_eq!(observation.accelerated, Duration::ZERO); + } } diff --git a/rs/moq-audio/src/playback/observe.rs b/rs/moq-audio/src/playback/observe.rs index fe02ecf4578..ec3aecae84a 100644 --- a/rs/moq-audio/src/playback/observe.rs +++ b/rs/moq-audio/src/playback/observe.rs @@ -71,6 +71,12 @@ pub struct Observation { /// Device time filled with comfort noise for a silence the publisher /// declared, since the first audible quantum. pub comfort: Duration, + /// Received audio discarded unplayed to bring a sink's queue back down to + /// its latency, summed over the sinks being written. + pub trimmed: Duration, + /// Received audio a sink's time stretch removed by playing faster, summed + /// over the sinks being written. Inaudible by design, unlike a trim. + pub accelerated: Duration, /// Whether nothing is playing: no sink is being written to, or every one /// still waits for its first samples. pub stalled: bool, @@ -96,6 +102,10 @@ pub(super) struct Shared { quiet: AtomicU64, /// [`Observation::comfort`] in nanoseconds. comfort: AtomicU64, + /// [`Observation::trimmed`] in nanoseconds. + trimmed: AtomicU64, + /// [`Observation::accelerated`] in nanoseconds. + accelerated: AtomicU64, stalled: AtomicBool, /// The reading [`Observer`]'s generation, or 0 with none: events only queue /// while one reads. Each event carries it, so a send racing one observer's @@ -132,6 +142,8 @@ impl Default for Shared { quanta: AtomicU64::new(0), quiet: AtomicU64::new(0), comfort: AtomicU64::new(0), + trimmed: AtomicU64::new(0), + accelerated: AtomicU64::new(0), stalled: AtomicBool::new(true), observer: AtomicU64::new(0), generation: AtomicU64::new(0), @@ -199,6 +211,8 @@ impl Observer { quanta: shared.quanta.load(Ordering::Relaxed), quiet: shared.quiet.load(Ordering::Relaxed), comfort: Duration::from_nanos(shared.comfort.load(Ordering::Relaxed)), + trimmed: Duration::from_nanos(shared.trimmed.load(Ordering::Relaxed)), + accelerated: Duration::from_nanos(shared.accelerated.load(Ordering::Relaxed)), stalled: shared.stalled.load(Ordering::Relaxed), ..Default::default() }; @@ -391,6 +405,20 @@ impl Ledger { self.shared.clock.store(self.nanos(self.frames), Ordering::Relaxed); } + /// Count frames the sinks being written discarded: `trimmed` to bound their + /// queues, `accelerated` by stretching. Nothing counts once the measurement + /// is over. + pub(super) fn shed(&self, trimmed: usize, accelerated: usize) { + if self.finished().is_some() { + return; + } + for (frames, total) in [(trimmed, &self.shared.trimmed), (accelerated, &self.shared.accelerated)] { + if frames > 0 { + total.fetch_add((frames as f64 * 1e9 / self.rate).round() as u64, Ordering::Relaxed); + } + } + } + /// Classify the quantum just completed. fn quantum(&mut self, channels: usize) { let missing = std::mem::take(&mut self.missing); diff --git a/rs/moq-audio/src/ring.rs b/rs/moq-audio/src/ring.rs index 5789da4a6fb..12a1702df89 100644 --- a/rs/moq-audio/src/ring.rs +++ b/rs/moq-audio/src/ring.rs @@ -158,6 +158,8 @@ pub(crate) fn channel(channels: usize, input_rate: u32, output_rate: u32, config cushioned: 0, comfort: crate::comfort::Comfort::new(output_rate, channels), trim: config.trim, + trimmed: 0, + accelerated: 0, shared, }; (producer, consumer) @@ -494,6 +496,10 @@ pub(crate) struct Consumer { comfort: crate::comfort::Comfort, /// Trim only after filling three quarters of the current target's headroom. trim: bool, + /// Frames [`Read::Trimmed`] discarded, ever. + trimmed: u64, + /// Frames the stretch removed by accelerating, ever. + accelerated: u64, shared: Arc, } @@ -542,6 +548,7 @@ impl Consumer { if queued >= trim && queued > latency { let frames = queued - latency; self.discard(frames); + self.trimmed += frames as u64; return Read::Trimmed { frames }; } } @@ -648,6 +655,7 @@ impl Consumer { debug_assert_eq!(read, frames * channels); self.position = self.position.wrapping_add(frames); stretch.process(frames, buffered, self.shared.target.load(Ordering::Relaxed), enabled); + self.accelerated += frames.saturating_sub(stretch.available()) as u64; // The writer still counts the input while it is being analysed. // Publish only the resulting duration change, then the rendered // frames. Moving PCM into staging never makes it disappear. @@ -662,6 +670,16 @@ impl Consumer { self.comfort.filled() } + /// Frames discarded to trim the queue back to its latency, ever. + pub(crate) fn trimmed(&self) -> u64 { + self.trimmed + } + + /// Frames of received audio the stretch removed by accelerating, ever. + pub(crate) fn accelerated(&self) -> u64 { + self.accelerated + } + /// Accept pushes without reading first, as a read would. pub(crate) fn start(&self) { self.shared.read.store(true, Ordering::Relaxed); @@ -867,6 +885,7 @@ mod tests { assert_eq!(prod.push(&vec![0.5; 3800 * STEREO]), Push::Ok); assert_eq!(cons.read(&mut out), Read::Trimmed { frames: 3880 - 480 }); + assert_eq!(cons.trimmed(), 3880 - 480); assert_eq!(cons.available(), 480); } @@ -1059,6 +1078,8 @@ mod tests { (120..=720).contains(&removed), "accelerate removes one pitch period, removed {removed}" ); + assert_eq!(cons.accelerated(), removed as u64, "the shed is counted"); + assert_eq!(cons.trimmed(), 0, "and is no trim"); assert_eq!( prod.buffered(), Duration::from_secs_f64(cons.available() as f64 / 48_000.0) diff --git a/rs/moq-cli/src/play/args.rs b/rs/moq-cli/src/play/args.rs index c142cbb1cc8..78b2a8f2c37 100644 --- a/rs/moq-cli/src/play/args.rs +++ b/rs/moq-cli/src/play/args.rs @@ -84,9 +84,10 @@ pub struct Args { /// Write what the speaker played to this file, one JSON sample per line every 250ms. /// /// Each line has the playhead, the playout target, the render quanta the speaker played - /// and how many were quiet, and the gaps and stalls since the line before, in the metric - /// schema of `test/audio-quality`, whose analyzer reduces it to underruns, skips, and - /// silence. Audio only, so it needs `--video off`. + /// and how many were quiet, the received audio trimmed and accelerated away, and the gaps + /// and stalls since the line before, in the metric schema of `test/audio-quality`, whose + /// analyzer reduces it to underruns, trims, and silence. Audio only, so it needs + /// `--video off`. #[usage(long, value_name = "PATH", value_hint = usage_rs::ValueHint::FilePath, extensions("ndjson"))] pub metrics: Option, } diff --git a/rs/moq-cli/src/play/headless.rs b/rs/moq-cli/src/play/headless.rs index b8bd1c2e9c5..9243cb86960 100644 --- a/rs/moq-cli/src/play/headless.rs +++ b/rs/moq-cli/src/play/headless.rs @@ -88,6 +88,11 @@ pub(crate) async fn run( metrics.utterance(timestamp); } } + Some(Event::Trimmed(trimmed)) => { + if let Some(metrics) = &mut metrics { + metrics.trimmed(trimmed); + } + } None => break Err(anyhow::anyhow!("audio playback event channel closed")), }, result = &mut media, if !completed => { diff --git a/rs/moq-cli/src/play/media.rs b/rs/moq-cli/src/play/media.rs index 402a8bd2eca..d321d00f306 100644 --- a/rs/moq-cli/src/play/media.rs +++ b/rs/moq-cli/src/play/media.rs @@ -830,6 +830,10 @@ async fn play_audio( } let skip = fit.skip as usize * stride; let written = sink.write(&frame.data[skip.min(frame.data.len())..])?; + let trimmed = fit.skip + written.dropped_sample_frames as u64; + if trimmed > 0 { + output.send(Event::Trimmed(Duration::from_secs_f64(trimmed as f64 / sample_rate as f64))); + } // A dropped write leaves the utterance to start on the next frame the sink accepts. if utterance && written.accepted_sample_frames > 0 { utterance = false; @@ -1870,7 +1874,7 @@ mod tests { match event { Event::Failed(err) => panic!("playback failed: {err}"), Event::Ended | Event::Finished => panic!("the player stopped following the path"), - Event::Wake | Event::Utterance(_) => {} + Event::Wake | Event::Utterance(_) | Event::Trimmed(_) => {} } } assert!(!player.is_finished(), "the player stopped following the path"); diff --git a/rs/moq-cli/src/play/metrics.rs b/rs/moq-cli/src/play/metrics.rs index d1b159c5a95..5d468c47a43 100644 --- a/rs/moq-cli/src/play/metrics.rs +++ b/rs/moq-cli/src/play/metrics.rs @@ -8,6 +8,7 @@ //! `timestamp` the media clock; fields this player cannot read are left out //! rather than written as zero. `comfort` and `utterances` have no browser //! counterpart: they show how a declared silence played, for push-to-talk. +//! `accelerated` has none until the browser stretches its audio too. use std::fs::File; use std::io::{BufWriter, Write}; @@ -48,6 +49,9 @@ struct Sample { quiet: Option, #[serde(skip_serializing_if = "Option::is_none")] comfort: Option, + trimmed: f64, + #[serde(skip_serializing_if = "Option::is_none")] + accelerated: Option, #[serde(skip_serializing_if = "Vec::is_empty")] gaps: Vec, #[serde(skip_serializing_if = "Vec::is_empty")] @@ -90,6 +94,8 @@ struct Totals { quanta: u64, quiet: u64, comfort: Duration, + trimmed: Duration, + accelerated: Duration, } impl Totals { @@ -99,6 +105,19 @@ impl Totals { quanta: observation.quanta, quiet: observation.quiet, comfort: observation.comfort, + trimmed: observation.trimmed, + accelerated: observation.accelerated, + } + } + + fn add(self, other: Self) -> Self { + Self { + clock: self.clock + other.clock, + quanta: self.quanta + other.quanta, + quiet: self.quiet + other.quiet, + comfort: self.comfort + other.comfort, + trimmed: self.trimmed + other.trimmed, + accelerated: self.accelerated + other.accelerated, } } } @@ -113,6 +132,8 @@ pub(super) struct Metrics { base: Totals, /// Utterances since the last sample. utterances: Vec, + /// Received audio the player discarded before it reached a speaker's queue. + trimmed: Duration, start: Instant, interval: kio::clock::Interval, } @@ -134,6 +155,7 @@ impl Metrics { current: None, base: Totals::default(), utterances: Vec::new(), + trimmed: Duration::ZERO, start: Instant::now(), interval, }; @@ -154,6 +176,11 @@ impl Metrics { }); } + /// Count received audio the player discarded rather than handed to the speaker. + pub(super) fn trimmed(&mut self, trimmed: Duration) { + self.trimmed += trimmed; + } + /// End the measurement: close any gap still open and write one last sample. pub(super) async fn finish(mut self) -> anyhow::Result<()> { // A speaker reopened since the last sample is the one playing now: adopt it @@ -182,13 +209,7 @@ impl Metrics { if let Some((mut observer, base)) = self.current.take() { let last = observer.read().context("speaker metrics")?; collect(&last, base, &mut gaps, &mut stalls); - let totals = Totals::of(&last); - self.base = Totals { - clock: base.clock + totals.clock, - quanta: base.quanta + totals.quanta, - quiet: base.quiet + totals.quiet, - comfort: base.comfort + totals.comfort, - }; + self.base = base.add(Totals::of(&last)); } self.current = Some((next, self.base)); } @@ -201,6 +222,9 @@ impl Metrics { } None => None, }; + let totals = observation + .as_ref() + .map_or(self.base, |(o, base)| base.add(Totals::of(o))); // Read straight after the device clock, so the two pair up. let now = Instant::now(); let (playhead, target) = { @@ -210,14 +234,16 @@ impl Metrics { let sample = Sample { at: ms(now.duration_since(self.start)), - render: observation.as_ref().map(|(o, base)| ms(base.clock + o.clock)), + render: observation.as_ref().map(|_| ms(totals.clock)), timestamp: playhead.map(ms), stalled: observation.as_ref().map(|(o, _)| o.stalled), delay: target.map(ms), output_latency: observation.as_ref().and_then(|(o, _)| o.latency).map(ms), - quanta: observation.as_ref().map(|(o, base)| base.quanta + o.quanta), - quiet: observation.as_ref().map(|(o, base)| base.quiet + o.quiet), - comfort: observation.as_ref().map(|(o, base)| ms(base.comfort + o.comfort)), + quanta: observation.as_ref().map(|_| totals.quanta), + quiet: observation.as_ref().map(|_| totals.quiet), + comfort: observation.as_ref().map(|_| ms(totals.comfort)), + trimmed: ms(self.trimmed + totals.trimmed), + accelerated: observation.as_ref().map(|_| ms(totals.accelerated)), gaps, stalls, utterances: std::mem::take(&mut self.utterances), diff --git a/rs/moq-cli/src/play/window.rs b/rs/moq-cli/src/play/window.rs index 4e99d266ea8..76f8cef3820 100644 --- a/rs/moq-cli/src/play/window.rs +++ b/rs/moq-cli/src/play/window.rs @@ -45,6 +45,9 @@ pub(super) enum Event { Failed(String), /// Audio resumed after a declared silence, starting at this media timestamp. Utterance(Duration), + /// Received audio the player discarded rather than played to bound its + /// delay: skipped at the fast threshold, or refused by a full sink. + Trimmed(Duration), } /// Run native playback on the calling thread until the window closes. @@ -286,7 +289,7 @@ impl ApplicationHandler for App { } } Event::Failed(err) => self.fail(event_loop, err), - Event::Utterance(_) => {} + Event::Utterance(_) | Event::Trimmed(_) => {} } } diff --git a/test/audio-quality/README.md b/test/audio-quality/README.md index 2f75abfadbf..04b87efb9b7 100644 --- a/test/audio-quality/README.md +++ b/test/audio-quality/README.md @@ -293,6 +293,14 @@ player's own count, which its stats panel shows, is read alongside: accelerates about 8%, so shedding more than about 40 ms reads as one skip or more. - `discarded_ms`: the media those skip-aheads jumped over. Audio dropped before it reached the ring moves no playhead, so it is not counted. +- `trimmed_ms`: the player's own count of received audio it discarded unplayed to bound its delay + (`audio.out.stats.trimmed`): the browser ring dropping its oldest past the target band, and + natively the ring's capacity trim, the incoming excess skipped at the stretch's fast threshold, + and writes a full ring refused. Unlike `discarded_ms`, it never counts an accelerate. Media the + subscription skipped before decoding, a group past its age budget, is a gap in the timeline, not + a trim, in either player. +- `accelerated_ms`: native only, the received audio `moq play`'s time stretch removed by playing + faster. Inaudible by design, so it is reported rather than budgeted. - `stalled`: the share of the window the ring spent stalled, refilling rather than playing. A gap that starts within 60 ms of a stall is that stall, and is not an underrun. - `silence`: the share of rendered quanta under -60 dBFS at the output, whatever the cause. diff --git a/test/audio-quality/clients/js/grade.ts b/test/audio-quality/clients/js/grade.ts index 9d2f26c2361..864ab20bd99 100644 --- a/test/audio-quality/clients/js/grade.ts +++ b/test/audio-quality/clients/js/grade.ts @@ -71,6 +71,8 @@ const COLUMNS: [string, string][] = [ ["gap ms/min", "underrun_ms_per_min"], ["skips/min", "skip_aheads_per_min"], ["discarded ms/min", "discarded_ms_per_min"], + ["trimmed ms/min", "trimmed_ms_per_min"], + ["accelerated ms/min", "accelerated_ms_per_min"], ["stalled", "stalled_share"], ["silence", "silence_share"], ["target p95", "target_ms_p95"], diff --git a/test/audio-quality/clients/js/replay.ts b/test/audio-quality/clients/js/replay.ts index fb7d0355480..2e3320282fa 100644 --- a/test/audio-quality/clients/js/replay.ts +++ b/test/audio-quality/clients/js/replay.ts @@ -103,7 +103,14 @@ for (const { row, trace } of rows) { }; const arrivals = trace.arrivals.map(([at, timestamp, group, duration]) => ({ at, timestamp, group, duration })); - const sample = (quantum: { at: number; timestamp: number; stalled: boolean; delay: number; underruns: number }) => { + const sample = (quantum: { + at: number; + timestamp: number; + stalled: boolean; + delay: number; + underruns: number; + trimmed: number; + }) => { const gaps = ledger.take(); samples.push({ at: quantum.at, @@ -112,6 +119,7 @@ for (const { row, trace } of rows) { stalled: quantum.stalled, delay: quantum.delay, playerUnderruns: quantum.underruns, + trimmed: quantum.trimmed, quanta: ledger.counts.quanta, quiet: ledger.counts.quiet, gaps: gaps.length > 0 ? gaps : undefined, diff --git a/test/audio-quality/clients/js/src/analyze.ts b/test/audio-quality/clients/js/src/analyze.ts index b3cc7b3ac9d..200d5c0d723 100644 --- a/test/audio-quality/clients/js/src/analyze.ts +++ b/test/audio-quality/clients/js/src/analyze.ts @@ -203,9 +203,16 @@ export function analyze(input: Input): Summary { if (!lastTap) notes.push("tap: never reported, so underruns and silence are unmeasured"); // The player's own counter is cumulative, so the window's count is its rise across the window. - const counted = window.filter((s) => s.playerUnderruns !== undefined); - const playerUnderruns = - counted.length > 0 ? (counted.at(-1)?.playerUnderruns ?? 0) - (counted[0]?.playerUnderruns ?? 0) : null; + const rise = (read: (s: Sample) => number | undefined) => { + const counted = window.flatMap((s) => { + const value = read(s); + return value === undefined ? [] : [value]; + }); + return counted.length > 0 ? (counted.at(-1) ?? 0) - (counted[0] ?? 0) : null; + }; + const playerUnderruns = rise((s) => s.playerUnderruns); + const trimmed = rise((s) => s.trimmed); + const accelerated = rise((s) => s.accelerated); // ── the playhead ──────────────────────────────────────────────────────── // The lag drifts when the media timeline does not run at wall rate, a property of the source @@ -262,6 +269,10 @@ export function analyze(input: Input): Summary { skip_aheads_per_min: perMin(skips.count), discarded_ms_total: round(skips.ms), discarded_ms_per_min: perMin(skips.ms), + trimmed_ms_total: round(trimmed), + trimmed_ms_per_min: trimmed === null ? null : perMin(trimmed), + accelerated_ms_total: round(accelerated), + accelerated_ms_per_min: accelerated === null ? null : perMin(accelerated), stalled_share: round(stalledShare, 3), silence_share: round(silence, 3), target_ms_p50: round(percentile(targets, 50)), diff --git a/test/audio-quality/clients/js/src/probe.ts b/test/audio-quality/clients/js/src/probe.ts index c8ae79c7e24..c9dd1280143 100644 --- a/test/audio-quality/clients/js/src/probe.ts +++ b/test/audio-quality/clients/js/src/probe.ts @@ -128,6 +128,7 @@ export function probe(watch: MoqWatch, capture = false): Probe { stalled: audio.out.stalled.peek(), delay: sync.out.delay.peek(), playerUnderruns: audio.out.stats.peek()?.underruns, + trimmed: audio.out.stats.peek()?.trimmed, rtt: watch.connection.probe.peek()?.rtt, jitter: audio.source.out.config.peek()?.jitter, renditionDelay: audio.source.out.config.peek()?.delay, diff --git a/test/audio-quality/clients/js/src/schema.ts b/test/audio-quality/clients/js/src/schema.ts index 0ce5d6631fb..0d3e8733ed5 100644 --- a/test/audio-quality/clients/js/src/schema.ts +++ b/test/audio-quality/clients/js/src/schema.ts @@ -109,7 +109,7 @@ export const METRICS = { clock: "viewer", aggregations: ["total", "per_min"], description: - "Re-anchors that discarded buffered audio: a forward step in the lag between the viewer clock and the playhead.", + "Re-anchors that discarded buffered audio: a forward step in the lag between the viewer clock and the playhead. A time stretch that accelerates far enough counts too.", }, discarded_ms: { unit: "ms", @@ -118,6 +118,19 @@ export const METRICS = { description: "Media time the skip-aheads jumped over without playing. Audio dropped before it reached the ring moves no playhead and is not counted.", }, + trimmed_ms: { + unit: "ms", + clock: "media", + aggregations: ["total", "per_min"], + description: + "The player's own count of received audio it discarded unplayed to bound its delay. Unlike `discarded_ms`, a time stretch's accelerate is not in it.", + }, + accelerated_ms: { + unit: "ms", + clock: "media", + aggregations: ["total", "per_min"], + description: "The player's own count of received audio its time stretch removed by playing faster. Native only.", + }, stalled: { unit: "share", clock: "viewer", @@ -390,6 +403,14 @@ export type Sample = { * Unset natively. */ playerUnderruns?: number; + /** + * `audio.out.stats.trimmed`, cumulative: buffered audio the player dropped unplayed to bound its + * delay. Natively, the ring's trims, the incoming excess skipped at the stretch's fast threshold, + * and writes a full ring refused. + */ + trimmed?: Ms; + /** Native only: cumulative received audio the time stretch removed by playing faster. */ + accelerated?: Ms; /** The current connection PROBE round trip. */ rtt?: Ms; /** The selected rendition's advertised jitter at this sample. */ From f131686a408c3953084e5a3fb0640c88e52d4eee Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 20:00:53 -0800 Subject: [PATCH 4/8] fix: carry trimmed through the test fixtures, replay, and the API report Co-Authored-By: Claude Opus 5.5 --- js/watch/api/index.api.md | 1 + js/watch/src/audio/buffer.test.ts | 12 +++++++++++- js/watch/src/audio/decoder.test.ts | 1 + js/watch/src/audio/render-worklet.test.ts | 4 ++-- js/watch/src/audio/ring-buffer.test.ts | 2 +- rs/moq-audio/src/playback/mixer.rs | 5 ++++- rs/moq-cli/src/play/media.rs | 4 +++- test/audio-quality/clients/js/replay.ts | 3 ++- test/audio-quality/clients/js/src/schema.ts | 3 ++- 9 files changed, 27 insertions(+), 8 deletions(-) diff --git a/js/watch/api/index.api.md b/js/watch/api/index.api.md index 4fc9e5c9faf..a7d4dc96832 100644 --- a/js/watch/api/index.api.md +++ b/js/watch/api/index.api.md @@ -94,6 +94,7 @@ export namespace Audio { late: number; stalled: Clock.Milli; stalls: number; + trimmed: Clock.Milli; underruns: number; } export type Supported = (config: Hang.Catalog.AudioConfig) => Promise; diff --git a/js/watch/src/audio/buffer.test.ts b/js/watch/src/audio/buffer.test.ts index ab2ac43a6ed..446914ad20c 100644 --- a/js/watch/src/audio/buffer.test.ts +++ b/js/watch/src/audio/buffer.test.ts @@ -32,6 +32,7 @@ function open(): { stalled, anchored: timestamp !== undefined, underruns: 0, + trimmed: Milli.zero, silent: false, epoch, }; @@ -433,7 +434,16 @@ function fake(delay: Milli, buffered = false) { buffered, }); const state = (timestamp: Micro, stalled: boolean, epoch: number) => { - const data: State = { type: "state", timestamp, stalled, anchored: true, underruns: 0, silent: false, epoch }; + const data: State = { + type: "state", + timestamp, + stalled, + anchored: true, + underruns: 0, + trimmed: Milli.zero, + silent: false, + epoch, + }; port.dispatchEvent(new MessageEvent("message", { data })); }; return { buffer, sent, state }; diff --git a/js/watch/src/audio/decoder.test.ts b/js/watch/src/audio/decoder.test.ts index 8e82ce475ef..375fb811746 100644 --- a/js/watch/src/audio/decoder.test.ts +++ b/js/watch/src/audio/decoder.test.ts @@ -108,6 +108,7 @@ class FakeWorklet { stalled: ring.stalled, anchored: ring.anchored, underruns: ring.underruns, + trimmed: ring.trimmed, silent: ring.silent, epoch: this.#epoch, }; diff --git a/js/watch/src/audio/render-worklet.test.ts b/js/watch/src/audio/render-worklet.test.ts index c512d7559d2..fc2ab832122 100644 --- a/js/watch/src/audio/render-worklet.test.ts +++ b/js/watch/src/audio/render-worklet.test.ts @@ -85,7 +85,7 @@ describe("render dispatch", () => { stalled: false, anchored: true, underruns: 0, - trimmed: 0, + trimmed: Milli.zero, silent: false, }, ]); @@ -114,7 +114,7 @@ describe("render dispatch", () => { stalled: false, anchored: true, underruns: 0, - trimmed: 0, + trimmed: Milli.zero, silent: false, }, ]); diff --git a/js/watch/src/audio/ring-buffer.test.ts b/js/watch/src/audio/ring-buffer.test.ts index f9a8f11a3ae..1d966c5e051 100644 --- a/js/watch/src/audio/ring-buffer.test.ts +++ b/js/watch/src/audio/ring-buffer.test.ts @@ -293,7 +293,7 @@ describe("ring buffer wrapping", () => { // Past the band: drop back to the target, wrapping abs 200-249 onto slots 0-49. write(buffer, 200 as Milli, 50, { channels: 1, value: 4.0 }); expect(buffer.length).toBe(100); - expect(buffer.trimmed).toBe(100); + expect(buffer.trimmed).toBe(Milli(100)); // Read all 100 samples: abs 150-199 = 3.0, abs 200-249 = 4.0. const output2 = read(buffer, 100, 1); diff --git a/rs/moq-audio/src/playback/mixer.rs b/rs/moq-audio/src/playback/mixer.rs index 80066732b1e..8075bea885e 100644 --- a/rs/moq-audio/src/playback/mixer.rs +++ b/rs/moq-audio/src/playback/mixer.rs @@ -953,7 +953,10 @@ mod tests { let observation = observer.read().unwrap(); // The 10 ms prime and the push, less the buffer read and the latency kept. let trimmed = 480 + 40_000 - 512 - 480; - assert_eq!(observation.trimmed, Duration::from_secs_f64(trimmed as f64 / RATE as f64)); + assert_eq!( + observation.trimmed, + Duration::from_secs_f64(trimmed as f64 / RATE as f64) + ); assert_eq!(observation.accelerated, Duration::ZERO); } } diff --git a/rs/moq-cli/src/play/media.rs b/rs/moq-cli/src/play/media.rs index d321d00f306..5fa3ec471a7 100644 --- a/rs/moq-cli/src/play/media.rs +++ b/rs/moq-cli/src/play/media.rs @@ -832,7 +832,9 @@ async fn play_audio( let written = sink.write(&frame.data[skip.min(frame.data.len())..])?; let trimmed = fit.skip + written.dropped_sample_frames as u64; if trimmed > 0 { - output.send(Event::Trimmed(Duration::from_secs_f64(trimmed as f64 / sample_rate as f64))); + output.send(Event::Trimmed(Duration::from_secs_f64( + trimmed as f64 / sample_rate as f64, + ))); } // A dropped write leaves the utterance to start on the next frame the sink accepts. if utterance && written.accepted_sample_frames > 0 { diff --git a/test/audio-quality/clients/js/replay.ts b/test/audio-quality/clients/js/replay.ts index 2e3320282fa..2e5f6ecebf2 100644 --- a/test/audio-quality/clients/js/replay.ts +++ b/test/audio-quality/clients/js/replay.ts @@ -134,7 +134,7 @@ for (const { row, trace } of rows) { config: trace.config as unknown as Catalog.AudioConfig, duration: trace.duration, }); - let last: { at: number; timestamp: number; stalled: boolean; delay: number; underruns: number } | undefined; + let last: Parameters[0] | undefined; for await (const quantum of quanta) { // The render clock is the simulated one: the trace's `at`, which starts at zero. ledger.add(frame, quantum.output.length, classify([quantum.output])); @@ -149,6 +149,7 @@ for (const { row, trace } of rows) { stalled: quantum.stalled, delay: quantum.delay, underruns: quantum.underruns, + trimmed: quantum.trimmed, }; if (quantum.at >= next) { diff --git a/test/audio-quality/clients/js/src/schema.ts b/test/audio-quality/clients/js/src/schema.ts index 0d3e8733ed5..2da1c80e022 100644 --- a/test/audio-quality/clients/js/src/schema.ts +++ b/test/audio-quality/clients/js/src/schema.ts @@ -129,7 +129,8 @@ export const METRICS = { unit: "ms", clock: "media", aggregations: ["total", "per_min"], - description: "The player's own count of received audio its time stretch removed by playing faster. Native only.", + description: + "The player's own count of received audio its time stretch removed by playing faster. Native only.", }, stalled: { unit: "share", From a66ac9238c1bf79208c7c47256b2fd82560c0794 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 20:05:39 -0800 Subject: [PATCH 5/8] fix(watch): count only received audio a ring trim discards The trim count missed the front of an incoming chunk the playhead skipped and counted a gap's zeros as received audio. Each slot now records whether it holds received audio. Co-Authored-By: Claude Opus 5.5 --- js/watch/src/audio/ring-buffer.test.ts | 30 ++++++++++++++++++++++++++ js/watch/src/audio/ring-buffer.ts | 18 ++++++++++------ 2 files changed, 42 insertions(+), 6 deletions(-) diff --git a/js/watch/src/audio/ring-buffer.test.ts b/js/watch/src/audio/ring-buffer.test.ts index 1d966c5e051..b789448d9cf 100644 --- a/js/watch/src/audio/ring-buffer.test.ts +++ b/js/watch/src/audio/ring-buffer.test.ts @@ -308,6 +308,36 @@ describe("ring buffer wrapping", () => { }); }); +describe("trimmed", () => { + // A 10 ms target at 1 kHz tolerates one chunk above it before dropping back to it. + const playing = () => { + const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 10 as Milli }); + write(buffer, 0 as Milli, 10, { channels: 1, value: 1 }); + expect(buffer.stalled).toBe(false); + return buffer; + }; + + it("counts the front of the incoming chunk the playhead skips", () => { + const buffer = playing(); + write(buffer, 10 as Milli, 20, { channels: 1, value: 2 }); + expect(buffer.trimmed).toBe(Milli(0)); + // The playhead jumps from 0 to 40: the 30 buffered and the chunk's first 10. + write(buffer, 30 as Milli, 20, { channels: 1, value: 3 }); + expect(buffer.trimmed).toBe(Milli(40)); + }); + + it("leaves out a gap's zeros", () => { + const buffer = playing(); + write(buffer, 10 as Milli, 5, { channels: 1, value: 2 }); + // The playhead jumps from 0 to 25 over the 15 buffered; 15 to 29 is a gap of zeros. + write(buffer, 30 as Milli, 5, { channels: 1, value: 3 }); + expect(buffer.trimmed).toBe(Milli(15)); + // From 25 to 70: 25 to 29 gap, 30 to 34 received, 35 to 59 gap, 60 to 69 the chunk's front. + write(buffer, 60 as Milli, 20, { channels: 1, value: 4 }); + expect(buffer.trimmed).toBe(Milli(15 + 5 + 10)); + }); +}); + describe("multi-channel handling", () => { it("should handle stereo data correctly", () => { const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100 as Milli }); diff --git a/js/watch/src/audio/ring-buffer.ts b/js/watch/src/audio/ring-buffer.ts index 8edd736f12c..6d84b4a2023 100644 --- a/js/watch/src/audio/ring-buffer.ts +++ b/js/watch/src/audio/ring-buffer.ts @@ -3,6 +3,9 @@ import { ringCapacity } from "./latency"; export class AudioRingBuffer { #buffer: Float32Array[]; + // Per slot, whether it holds received audio rather than the zeros filling a gap or a declared + // silence, so a trim counts only what it discards of the former. + #received: Uint8Array; #writeIndex = 0; #readIndex = 0; @@ -72,6 +75,7 @@ export class AudioRingBuffer { const capacity = ringCapacity(this.rate); if (this.#latencySamples > capacity) throw new Error("latency exceeds capacity"); + this.#received = new Uint8Array(capacity); this.#buffer = []; for (let i = 0; i < this.channels; i++) { this.#buffer[i] = new Float32Array(capacity); @@ -206,13 +210,13 @@ export class AudioRingBuffer { const limit = band ? Math.min(this.#latencySamples + this.#chunk, this.capacity) : this.capacity; if (end - this.#readIndex > limit) { const readIndex = end - (band ? Math.min(this.#latencySamples, this.capacity) : this.capacity); - // Only written audio counts: not a discontinuity's gap, nor a declared silence's zeros. - const dropped = Math.min(readIndex, this.#writeIndex); - let trimmed = Math.max(0, dropped - this.#readIndex); - for (const pause of this.#pauses) { - trimmed -= Math.max(0, Math.min(pause.end, dropped) - Math.max(pause.start, this.#readIndex)); + // Received audio the new playhead passes: what was buffered, less any gap or declared + // silence in it, and the front of this chunk when the playhead lands inside it. + const buffered = Math.min(readIndex, this.#writeIndex); + for (let i = this.#readIndex; i < buffered; i++) { + if (i >= start || this.#received[i % this.capacity]) this.#trimmed++; } - this.#trimmed += trimmed; + this.#trimmed += Math.max(0, Math.min(readIndex, end) - Math.max(start, this.#writeIndex)); this.#readIndex = readIndex; this.#prune(); } @@ -231,6 +235,7 @@ export class AudioRingBuffer { dst[writePos] = 0; } } + for (let i = 0; i < gapSize; i++) this.#received[(this.#writeIndex + i) % this.capacity] = 0; } // Write the actual samples @@ -246,6 +251,7 @@ export class AudioRingBuffer { dst[writePos] = src[i]; } } + for (let i = 0; i < samples; i++) this.#received[(start + i) % this.capacity] = 1; // Update write index, but only if we're moving forward if (end > this.#writeIndex) { From 3e68e25cce50a801a8b648d337df92d7fa89ee6c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 20:16:33 -0800 Subject: [PATCH 6/8] fix(moq-audio): a ring trim counts no startup silence or comfort noise Co-Authored-By: Claude Opus 5.5 --- rs/moq-audio/src/playback/observe.rs | 5 ++-- rs/moq-audio/src/ring.rs | 42 +++++++++++++++++++++++----- test/audio-quality/README.md | 4 ++- 3 files changed, 41 insertions(+), 10 deletions(-) diff --git a/rs/moq-audio/src/playback/observe.rs b/rs/moq-audio/src/playback/observe.rs index ec3aecae84a..51cba28b2c9 100644 --- a/rs/moq-audio/src/playback/observe.rs +++ b/rs/moq-audio/src/playback/observe.rs @@ -71,8 +71,9 @@ pub struct Observation { /// Device time filled with comfort noise for a silence the publisher /// declared, since the first audible quantum. pub comfort: Duration, - /// Received audio discarded unplayed to bring a sink's queue back down to - /// its latency, summed over the sinks being written. + /// Written audio discarded unplayed to bring a sink's queue back down to + /// its latency, summed over the sinks being written. A sink's opening + /// silence and comfort noise never count; silence the writer wrote does. pub trimmed: Duration, /// Received audio a sink's time stretch removed by playing faster, summed /// over the sinks being written. Inaudible by design, unlike a trim. diff --git a/rs/moq-audio/src/ring.rs b/rs/moq-audio/src/ring.rs index 12a1702df89..610046bfa94 100644 --- a/rs/moq-audio/src/ring.rs +++ b/rs/moq-audio/src/ring.rs @@ -496,7 +496,7 @@ pub(crate) struct Consumer { comfort: crate::comfort::Comfort, /// Trim only after filling three quarters of the current target's headroom. trim: bool, - /// Frames [`Read::Trimmed`] discarded, ever. + /// Received frames [`Read::Trimmed`] discarded, ever. trimmed: u64, /// Frames the stretch removed by accelerating, ever. accelerated: u64, @@ -547,8 +547,7 @@ impl Consumer { let trim = latency + (capacity - latency) * 3 / 4; if queued >= trim && queued > latency { let frames = queued - latency; - self.discard(frames); - self.trimmed += frames as u64; + self.trimmed += self.discard(frames) as u64; return Read::Trimmed { frames }; } } @@ -670,7 +669,9 @@ impl Consumer { self.comfort.filled() } - /// Frames discarded to trim the queue back to its latency, ever. + /// Pushed frames discarded to trim the queue back to its latency, ever: not + /// the startup silence or a resume's comfort noise, though silence padding + /// a writer pushed itself, or [`Push::Padded`] added, counts. pub(crate) fn trimmed(&self) -> u64 { self.trimmed } @@ -695,10 +696,12 @@ impl Consumer { buffered + cushion + self.stretch.as_ref().map_or(0, |stretch| stretch.available()) } - /// Drop up to `frames` queued frames unread. - pub(crate) fn discard(&mut self, frames: usize) { + /// Drop up to `frames` queued frames unread, returning how many of them were + /// pushed rather than the startup silence or a resume's comfort noise. + pub(crate) fn discard(&mut self, frames: usize) -> usize { let staged = self.stretch.as_mut().map_or(0, |stretch| stretch.discard(frames)); self.consumed = self.consumed.wrapping_add(staged); + let mut pushed = if self.staged_prime { 0 } else { staged }; let mut left = frames - staged; while left > 0 { let mut take = left; @@ -713,12 +716,15 @@ impl Consumer { if skipped == 0 { break; } - self.prime = self.prime.saturating_sub(skipped); + let prime = skipped.min(self.prime); + self.prime -= prime; + pushed += skipped - prime; self.position = self.position.wrapping_add(skipped); self.consumed = self.consumed.wrapping_add(skipped); left -= skipped; } self.shared.consumed.store(self.consumed, Ordering::Relaxed); + pushed } } @@ -889,6 +895,28 @@ mod tests { assert_eq!(cons.available(), 480); } + /// A trim that reaches into the startup silence counts only what was pushed. + #[test] + fn a_trim_counts_no_startup_silence() { + let (mut prod, mut cons) = channel( + STEREO, + 48_000, + 48_000, + Config { + latency: Duration::from_millis(10), + capacity: Duration::from_millis(100), + pad: false, + trim: true, + }, + ); + cons.start(); + assert_eq!(prod.push(&vec![0.5; 4000 * STEREO]), Push::Ok); + let mut out = vec![0.0; 128 * STEREO]; + // 352 of the 480 startup frames are left behind the 128 read. + assert_eq!(cons.read(&mut out), Read::Trimmed { frames: 4352 - 480 }); + assert_eq!(cons.trimmed(), 4352 - 480 - 352); + } + /// Construction allocates for the reader's rate and the queue's channels, /// never for a declared input rate. #[test] diff --git a/test/audio-quality/README.md b/test/audio-quality/README.md index 04b87efb9b7..ac74f6c5324 100644 --- a/test/audio-quality/README.md +++ b/test/audio-quality/README.md @@ -296,7 +296,9 @@ player's own count, which its stats panel shows, is read alongside: - `trimmed_ms`: the player's own count of received audio it discarded unplayed to bound its delay (`audio.out.stats.trimmed`): the browser ring dropping its oldest past the target band, and natively the ring's capacity trim, the incoming excess skipped at the stretch's fast threshold, - and writes a full ring refused. Unlike `discarded_ms`, it never counts an accelerate. Media the + and writes a full ring refused. The native ring cannot tell silence `moq play` wrote for a short + hole from received audio, so a trim through one counts it; neither player counts its opening + silence or comfort noise. Unlike `discarded_ms`, it never counts an accelerate. Media the subscription skipped before decoding, a group past its age budget, is a gap in the timeline, not a trim, in either player. - `accelerated_ms`: native only, the received audio `moq play`'s time stretch removed by playing From 1d265408bc746a2403981c01a1eaea8baf6a45e2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 21:58:39 -0800 Subject: [PATCH 7/8] fix(watch): count the browser's trimmed and accelerated audio as media time The maintainer chose media time over sample provenance at the round-3 design checkpoint: a skip counts everything it passes, gap zeros included, less declared silence, as moq-audio already does. That drops the ring's per-slot received marks and the staged window's received share. Co-Authored-By: Claude Opus 5.5 --- doc/bin/cli.md | 2 +- js/watch/src/audio/buffer.ts | 4 +- js/watch/src/audio/decoder.ts | 4 +- js/watch/src/audio/render.ts | 4 +- js/watch/src/audio/ring-buffer.test.ts | 18 +++++++- js/watch/src/audio/ring-buffer.ts | 46 ++++++++------------- rs/moq-audio/src/playback/observe.rs | 2 +- test/audio-quality/README.md | 9 ++-- test/audio-quality/clients/js/src/schema.ts | 8 ++-- 9 files changed, 49 insertions(+), 48 deletions(-) diff --git a/doc/bin/cli.md b/doc/bin/cli.md index ff348d0e386..e8b271e7efe 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -195,7 +195,7 @@ holding the other one back. Each role follows the catalog, switching rendition when the publisher retires the one playing. With `--video off`, `--metrics ` writes a JSON line every 250 ms with the playhead, the delay, every gap and stall the speaker hit, and -the received audio it trimmed to bound its delay or shed by playing faster, in +the media it trimmed to bound its delay or shed by playing faster, in the schema the [audio quality harness](https://github.com/moq-dev/moq/tree/main/test/audio-quality) reduces to underruns, trims, and silence. A silence the publisher declares, such diff --git a/js/watch/src/audio/buffer.ts b/js/watch/src/audio/buffer.ts index 4a634c67a3c..9c0b892ed40 100644 --- a/js/watch/src/audio/buffer.ts +++ b/js/watch/src/audio/buffer.ts @@ -282,11 +282,11 @@ export class AudioBuffer { readonly skips: Getter = this.#skips; readonly #trimmed = new Signal(Milli.zero); - /** Received audio the ring's skips dropped unplayed, cumulative. */ + /** Media the ring's skips dropped unplayed, less declared silence, cumulative. */ readonly trimmed: Getter = this.#trimmed; readonly #accelerated = new Signal(Milli.zero); - /** Received audio the stretch removed by playing faster, cumulative. */ + /** Media the stretch removed by playing faster, cumulative. */ readonly accelerated: Getter = this.#accelerated; readonly #silent = new Signal(false); diff --git a/js/watch/src/audio/decoder.ts b/js/watch/src/audio/decoder.ts index e403310396c..58c9e26d025 100644 --- a/js/watch/src/audio/decoder.ts +++ b/js/watch/src/audio/decoder.ts @@ -93,9 +93,9 @@ export interface Stats { underruns: number; /** How many times playback skipped ahead, dropping buffered audio rather than playing it faster. */ skips: number; - /** Received audio those skips dropped unplayed. */ + /** Media those skips dropped unplayed, less declared silence. */ trimmed: Milli; - /** Received audio the time stretch removed by playing faster. */ + /** Media the time stretch removed by playing faster. */ accelerated: Milli; /** Frames that arrived after their playout time. */ late: number; diff --git a/js/watch/src/audio/render.ts b/js/watch/src/audio/render.ts index 2df38cf36bc..7760d84bbab 100644 --- a/js/watch/src/audio/render.ts +++ b/js/watch/src/audio/render.ts @@ -64,9 +64,9 @@ export interface State { underruns: number; // How many times the ring skipped ahead, dropping queued audio, cumulative. skips: number; - // Received audio those skips dropped unplayed, cumulative. + // Media those skips dropped unplayed, less declared silence, cumulative. trimmed: Milli; - // Received audio the stretch removed by playing faster, cumulative. + // Media the stretch removed by playing faster, cumulative. accelerated: Milli; // Whether playback is at a declared end, reached or announced before any audio, and nothing has played since. silent: boolean; diff --git a/js/watch/src/audio/ring-buffer.test.ts b/js/watch/src/audio/ring-buffer.test.ts index 91b9d5d770c..ace61d6d6ed 100644 --- a/js/watch/src/audio/ring-buffer.test.ts +++ b/js/watch/src/audio/ring-buffer.test.ts @@ -975,7 +975,7 @@ describe("time stretch", () => { expect(output.every((sample) => sample === 0.5)).toBe(true); }); - it("trims no gap's zeros", () => { + it("trims a gap's zeros as media", () => { const buffer = live(100); write(buffer, 0 as Milli, frames(100), { value: 0.25 }); read(buffer, QUANTUM, 1); @@ -984,7 +984,21 @@ describe("time stretch", () => { write(buffer, 210 as Milli, frames(300), { value: 0.5 }); expect(buffer.skips).toBe(1); expect(Milli.fromMicro(buffer.timestamp)).toBeCloseTo(410, 6); - expect(buffer.trimmed).toBeCloseTo(410 - 100 - (QUANTUM * 1000) / RATE, 6); + expect(buffer.trimmed).toBeCloseTo(410 - (QUANTUM * 1000) / RATE, 6); + }); + + it("trims no declared silence", () => { + const buffer = live(100); + write(buffer, 0 as Milli, frames(100), { value: 0.25 }); + read(buffer, QUANTUM, 1); + // A declared silence over [100, 200) ms, then more than the ring's 1 s capacity, which skips to + // 250 ms through the silence and the incoming chunk's first 50 ms. + buffer.end(Micro(100_000)); + buffer.next(); + write(buffer, 200 as Milli, frames(1050), { value: 0.5 }); + expect(buffer.skips).toBe(1); + expect(Milli.fromMicro(buffer.timestamp)).toBeCloseTo(250, 6); + expect(buffer.trimmed).toBeCloseTo(250 - 100 - (QUANTUM * 1000) / RATE, 6); }); it("counts a re-anchor that drops only the staged window as a skip", () => { diff --git a/js/watch/src/audio/ring-buffer.ts b/js/watch/src/audio/ring-buffer.ts index 387aaf3ab2d..5faf44cd11d 100644 --- a/js/watch/src/audio/ring-buffer.ts +++ b/js/watch/src/audio/ring-buffer.ts @@ -4,9 +4,6 @@ import { Stretch } from "./stretch"; export class AudioRingBuffer { #buffer: Float32Array[]; - // Per slot, whether it holds received audio rather than the zeros filling a gap or a declared - // silence, so a skip counts only what it discards of the former. - #received: Uint8Array; #writeIndex = 0; #readIndex = 0; @@ -17,7 +14,8 @@ export class AudioRingBuffer { #parked = false; #underruns = 0; #skips = 0; - // Samples of received audio a skip dropped, and that the stretch removed by accelerating. + // Samples of media a skip dropped, and that the stretch removed by accelerating, gap zeros + // included and declared silence excluded, as moq-audio counts them. #trimmed = 0; #accelerated = 0; @@ -90,7 +88,6 @@ export class AudioRingBuffer { const capacity = ringCapacity(this.rate); if (this.#latencySamples > capacity) throw new Error("latency exceeds capacity"); - this.#received = new Uint8Array(capacity); this.#buffer = []; for (let i = 0; i < this.channels; i++) { this.#buffer[i] = new Float32Array(capacity); @@ -112,12 +109,12 @@ export class AudioRingBuffer { return this.#skips; } - /** Received audio the ring's skips dropped unplayed, cumulative. */ + /** Media the ring's skips dropped unplayed, less declared silence, cumulative. */ get trimmed(): Milli { return Milli.fromSecond((this.#trimmed / this.rate) as Second); } - /** Received audio the stretch removed by playing faster, cumulative. */ + /** Media the stretch removed by playing faster, cumulative. */ get accelerated(): Milli { return Milli.fromSecond((this.#accelerated / this.rate) as Second); } @@ -244,7 +241,7 @@ export class AudioRingBuffer { const latency = this.#latencySamples; const queued = end - this.#readIndex + this.#stretch.available; if (end - this.#readIndex > this.capacity) { - this.#skip(end - this.capacity, start, end); + this.#skip(end - this.capacity); } else if (band && pause && pause.start <= this.#readIndex && queued > latency + this.#chunk) { // Skip only the silence past the queued floor, which the staged window counts toward, so the // read index only moves forward and never re-stages PCM the window already holds. @@ -258,7 +255,7 @@ export class AudioRingBuffer { latency + Math.floor(((this.capacity - latency) * 3) / 4), ) ) { - this.#skip(end - latency, start, end); + this.#skip(end - latency); } // Fill gaps with zeros if there's a discontinuity @@ -275,7 +272,6 @@ export class AudioRingBuffer { dst[writePos] = 0; } } - for (let i = 0; i < gapSize; i++) this.#received[(this.#writeIndex + i) % this.capacity] = 0; } // Write the actual samples @@ -291,7 +287,6 @@ export class AudioRingBuffer { dst[writePos] = src[i]; } } - for (let i = 0; i < samples; i++) this.#received[(start + i) % this.capacity] = 1; // Update write index, but only if we're moving forward if (end > this.#writeIndex) { @@ -348,10 +343,10 @@ export class AudioRingBuffer { } // Move the read index forward to `index`, dropping any queued audio it passes, staged included, - // ahead of writing the chunk `[start, end)`, whose front the new index may also pass. - #skip(index: number, start: number, end: number): void { + // and the front of an incoming chunk when `index` lies past the write index. + #skip(index: number): void { if (index > this.#readIndex && this.length > 0) this.#skips++; - if (index > this.#readIndex) this.#trimmed += this.#discarded(index, start, end); + if (index > this.#readIndex) this.#trimmed += this.#discarded(index); // The staged window lies behind the new index, so it is dropped with the rest, as moq-audio's // trim discards staged output first. const length = this.length; @@ -364,23 +359,16 @@ export class AudioRingBuffer { this.#prune(); } - // Received samples a skip to `index` drops: the staged window's, what is buffered less any gap or - // declared silence in it, and the front of the incoming chunk `[start, end)`. - #discarded(index: number, start: number, end: number): number { - let dropped = 0; + // Media a skip to `index` drops: the unplayed share of the staged window and everything up to + // `index`, less declared silence. + #discarded(index: number): number { const stretch = this.#stretch; - if (stretch.available > 0 && !this.#windowQuiet) { - let received = 0; - for (let i = this.#readIndex - this.#window; i < this.#readIndex; i++) { - received += this.#received[i % this.capacity]; - } - dropped += (stretch.available * received) / stretch.length; - } - const buffered = Math.min(index, this.#writeIndex); - for (let i = this.#readIndex; i < buffered; i++) { - if (i >= start || this.#received[i % this.capacity]) dropped++; + let dropped = index - this.#readIndex; + if (stretch.available > 0 && !this.#windowQuiet) dropped += (stretch.available * this.#window) / stretch.length; + for (const pause of this.#pauses) { + dropped -= Math.max(0, Math.min(index, pause.end) - Math.max(this.#readIndex, pause.start)); } - return dropped + Math.max(0, Math.min(index, end) - Math.max(start, this.#writeIndex)); + return dropped; } // Forget the declared silences the playhead has passed. diff --git a/rs/moq-audio/src/playback/observe.rs b/rs/moq-audio/src/playback/observe.rs index ec4853e970c..ffc3e1e832d 100644 --- a/rs/moq-audio/src/playback/observe.rs +++ b/rs/moq-audio/src/playback/observe.rs @@ -77,7 +77,7 @@ pub struct Observation { /// its latency, summed over the sinks being written. A sink's opening /// silence and comfort noise never count; silence the writer wrote does. pub trimmed: Duration, - /// Received audio a sink's time stretch removed by playing faster, summed + /// Media a sink's time stretch removed by playing faster, summed /// over the sinks being written. Inaudible by design, unlike a trim. pub accelerated: Duration, /// Whether nothing is playing: no sink is being written to, or every one diff --git a/test/audio-quality/README.md b/test/audio-quality/README.md index 0aab3000845..57a8072ad54 100644 --- a/test/audio-quality/README.md +++ b/test/audio-quality/README.md @@ -304,13 +304,12 @@ player's own count, which its stats panel shows, is read alongside: accelerates about 8%, so shedding more than about 40 ms reads as one skip or more. - `discarded_ms`: the media those skip-aheads jumped over. Audio dropped before it reached the ring moves no playhead, so it is not counted. -- `trimmed_ms`: `audio.out.stats.trimmed`, the received audio those `player_skips` dropped - unplayed. The native ring cannot tell silence `moq play` wrote for a short - hole from received audio, so a trim through one counts it; neither player counts its opening - silence or comfort noise. Unlike `discarded_ms`, it never counts an accelerate. Media the +- `trimmed_ms`: `audio.out.stats.trimmed`, the media those `player_skips` dropped unplayed, + including the zeros filling a network gap; neither player counts its opening silence, declared + silence, or comfort noise. Unlike `discarded_ms`, it never counts an accelerate. Media the subscription skipped before decoding, a group past its age budget, is a gap in the timeline, not a trim, in either player. -- `accelerated_ms`: `audio.out.stats.accelerated`, the received audio the time stretch removed by +- `accelerated_ms`: `audio.out.stats.accelerated`, the media the time stretch removed by playing faster. Inaudible by design, so it is reported rather than budgeted. - `stalled`: the share of the window the ring spent stalled, refilling rather than playing. A gap that starts within 60 ms of a stall is that stall, and is not an underrun. diff --git a/test/audio-quality/clients/js/src/schema.ts b/test/audio-quality/clients/js/src/schema.ts index 8ac8083e643..fece2bea085 100644 --- a/test/audio-quality/clients/js/src/schema.ts +++ b/test/audio-quality/clients/js/src/schema.ts @@ -130,13 +130,13 @@ export const METRICS = { clock: "media", aggregations: ["total", "per_min"], description: - "The player's own count of received audio its skips dropped unplayed. Unlike `discarded_ms`, a time stretch's accelerate is not in it.", + "The player's own count of media its skips dropped unplayed. Unlike `discarded_ms`, a time stretch's accelerate is not in it.", }, accelerated_ms: { unit: "ms", clock: "media", aggregations: ["total", "per_min"], - description: "The player's own count of received audio its time stretch removed by playing faster.", + description: "The player's own count of media its time stretch removed by playing faster.", }, stalled: { unit: "share", @@ -425,9 +425,9 @@ export type Sample = { * threshold, or write a full ring refused. */ playerSkips?: number; - /** `audio.out.stats.trimmed`, cumulative: the received audio those skips dropped unplayed. */ + /** `audio.out.stats.trimmed`, cumulative: the media those skips dropped unplayed. */ trimmed?: Ms; - /** `audio.out.stats.accelerated`, cumulative: received audio the time stretch removed by playing faster. */ + /** `audio.out.stats.accelerated`, cumulative: media the time stretch removed by playing faster. */ accelerated?: Ms; /** The current connection PROBE round trip. */ rtt?: Ms; From 117144380beb68ad56b0ebab8c7bd9140b6ecc77 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 22:36:00 -0800 Subject: [PATCH 8/8] fix(js/watch): trim only what the ring held, gated like the skip count A re-anchor over a gap the ring never held counted the gap as trimmed, and on an empty ring counted it with no skip. Both counters now move only when the ring held audio, and the trim counts the held span and the incoming chunk's front, not the jump between them, as moq-audio does. Co-Authored-By: Claude Opus 5.5 --- js/watch/src/audio/ring-buffer.test.ts | 19 +++++++++++++++++ js/watch/src/audio/ring-buffer.ts | 28 ++++++++++++++++---------- 2 files changed, 36 insertions(+), 11 deletions(-) diff --git a/js/watch/src/audio/ring-buffer.test.ts b/js/watch/src/audio/ring-buffer.test.ts index ace61d6d6ed..07ca8a3f2bc 100644 --- a/js/watch/src/audio/ring-buffer.test.ts +++ b/js/watch/src/audio/ring-buffer.test.ts @@ -1001,6 +1001,25 @@ describe("time stretch", () => { expect(buffer.trimmed).toBeCloseTo(250 - 100 - (QUANTUM * 1000) / RATE, 6); }); + it("trims only what the ring held, never the gap a re-anchor jumps", () => { + const buffer = live(100); + write(buffer, 0 as Milli, frames(100), { value: 0.25 }); + read(buffer, QUANTUM, 1); + // A group past its age budget: the next audio lands 2 s on, past the ring's capacity. + write(buffer, 2000 as Milli, frames(20), { value: 0.5 }); + expect(buffer.skips).toBe(1); + expect(buffer.trimmed).toBeCloseTo(100 - (QUANTUM * 1000) / RATE, 6); + }); + + it("counts nothing when an empty ring re-anchors", () => { + const buffer = live(100); + write(buffer, 0 as Milli, frames(100), { value: 0.25 }); + while (buffer.length > 0) read(buffer, QUANTUM, 1); + write(buffer, 3000 as Milli, frames(20), { value: 0.5 }); + expect(buffer.skips).toBe(0); + expect(buffer.trimmed).toBe(Milli.zero); + }); + it("counts a re-anchor that drops only the staged window as a skip", () => { const buffer = live(100); // Four whole 30 ms windows, so once the playhead passes 90 ms the ring is empty and only the diff --git a/js/watch/src/audio/ring-buffer.ts b/js/watch/src/audio/ring-buffer.ts index 5faf44cd11d..5fde0e94fab 100644 --- a/js/watch/src/audio/ring-buffer.ts +++ b/js/watch/src/audio/ring-buffer.ts @@ -241,7 +241,7 @@ export class AudioRingBuffer { const latency = this.#latencySamples; const queued = end - this.#readIndex + this.#stretch.available; if (end - this.#readIndex > this.capacity) { - this.#skip(end - this.capacity); + this.#skip(end - this.capacity, start); } else if (band && pause && pause.start <= this.#readIndex && queued > latency + this.#chunk) { // Skip only the silence past the queued floor, which the staged window counts toward, so the // read index only moves forward and never re-stages PCM the window already holds. @@ -255,7 +255,7 @@ export class AudioRingBuffer { latency + Math.floor(((this.capacity - latency) * 3) / 4), ) ) { - this.#skip(end - latency); + this.#skip(end - latency, start); } // Fill gaps with zeros if there's a discontinuity @@ -343,10 +343,13 @@ export class AudioRingBuffer { } // Move the read index forward to `index`, dropping any queued audio it passes, staged included, - // and the front of an incoming chunk when `index` lies past the write index. - #skip(index: number): void { - if (index > this.#readIndex && this.length > 0) this.#skips++; - if (index > this.#readIndex) this.#trimmed += this.#discarded(index); + // and the front of the incoming chunk at `start` when `index` lies past it. An empty ring only + // re-anchors over a gap it never held: neither counter moves, as in moq-audio. + #skip(index: number, start: number): void { + if (index > this.#readIndex && this.length > 0) { + this.#skips++; + this.#trimmed += this.#discarded(index, start); + } // The staged window lies behind the new index, so it is dropped with the rest, as moq-audio's // trim discards staged output first. const length = this.length; @@ -359,14 +362,17 @@ export class AudioRingBuffer { this.#prune(); } - // Media a skip to `index` drops: the unplayed share of the staged window and everything up to - // `index`, less declared silence. - #discarded(index: number): number { + // Media a skip to `index` drops: the unplayed share of the staged window, what the ring holds up + // to `index`, and the incoming chunk's front below it, less declared silence. The gap between the + // write index and the chunk at `start` was never held, so it is no trim. + #discarded(index: number, start: number): number { const stretch = this.#stretch; - let dropped = index - this.#readIndex; + const held = Math.min(index, this.#writeIndex); + const front = Math.max(start, this.#writeIndex); + let dropped = held - this.#readIndex + Math.max(0, index - front); if (stretch.available > 0 && !this.#windowQuiet) dropped += (stretch.available * this.#window) / stretch.length; for (const pause of this.#pauses) { - dropped -= Math.max(0, Math.min(index, pause.end) - Math.max(this.#readIndex, pause.start)); + dropped -= Math.max(0, Math.min(held, pause.end) - Math.max(this.#readIndex, pause.start)); } return dropped; }