diff --git a/quest/m1/push-to-talk-latency.md b/quest/m1/push-to-talk-latency.md index 1a7b2f50cb7..a25d6e5208c 100644 --- a/quest/m1/push-to-talk-latency.md +++ b/quest/m1/push-to-talk-latency.md @@ -10,77 +10,13 @@ playback is out of scope. ## Plan -Measured 2026-11-04 by -[moq-dev/moq#7760](https://github.com/moq-dev/moq/pull/7760), for -[Push-to-talk](/quest/m1/push-to-talk/README.md). The setup was `just test -push-to-talk` through a local relay, publisher and listener on separate -sessions, with no shaping or loss, 60 ms Opus at 128 kbit/s, and a fixed 300 -ms `delay`. Chromium 156 ran headless on Linux. There were five runs of 22 -presses each, 15 of them separated, so 75 separated presses in all; one press -lost its head (see Push-to-talk) and has no breakdown, leaving 74. -`summary.json` `path` splits each press at page-clock boundaries that sum -exactly to press → projected output. - -| Part | Boundaries | Median ms | Max ms | -| --- | --- | --- | --- | -| capture | worklet delivery and framing | 29 | 110 | -| packetization | media the head frame still needed after the press | 39 | 60 | -| encode | encoder input to container write | 0.8 | 14 | -| network | container write to arrival, through the relay | 3.2 | 15 | -| decode | arrival to decoded PCM | 0.7 | 13 | -| jitter buffer | head's PCM to the first PCM that filled the delay | 252 | 268 | -| render | that PCM to the render onset | 21 | 54 | -| device | render onset to projected output | 115 | 119 | -| total | press to projected output | 469 | 576 | - -The capture and packetization split is approximate, but their 66 ms median sum -is exact. Playout on top of one-way delivery was 261 ms median and 294 ms max -over all 75, inside the 300 ms the maintainer set. - -Biggest contributors and their knobs: - -- **Playout delay, about 300 ms.** The ring starts once it holds the delay in - decoded PCM, counted from the head's first sample. That is why the jitter - buffer plus packetization sum to about 291 ms. Knob: `` `delay`. - This test fixes it at 300 ms. In `"auto"`, the default, the target is - `max(measured, advertised) + frame + catalog delay` - (`js/watch/src/audio/latency.ts`). -- **Device output, about 115 ms.** This matches Chromium's declared - `baseLatency + outputLatency` on this headless Linux host. Knob: the watch - AudioContext's `latencyHint` (`"interactive"` in - `js/watch/src/audio/decoder.ts`), otherwise the platform. A real device, - such as `kixtop`'s macOS output, should be measured before acting on it. -- **Capture and render, about 52 ms together.** Both followed the - AudioContext callback cadence on this host: PipeWire gives Chromium 2048 - samples (43 ms) per callback even at `"interactive"`. Capture no longer - does, since its context renders to no device - ([moq-dev/moq#7881](https://github.com/moq-dev/moq/pull/7881)): on this host - its median dropped from 29 ms to 1-9 ms, and the median per-press arrival - spread from 45 ms to 12 ms. Render still does. Knob: the watch AudioContext's - `latencyHint`, otherwise the platform. -- **Opus frame, 60 ms.** This test chooses it; the publisher defaults to - 20 ms. Under a fixed delay it is absorbed by the delay. In `"auto"` it adds - one frame to the target. Knob: the encoder's `frameDuration`. - -No part looked like a bug. On an ideal path, encode and decode take about 1 ms -and the relay hop about 3 ms, and the head is not trimmed as decoder warmup. -The one lost head is an ordering defect, owned by -[Push-to-talk](/quest/m1/push-to-talk/README.md), not a latency part. The -fixed delay bypasses the jitter estimator. In `"auto"`, the 46 ms median -per-press arrival-lateness spread, from capture bursts rather than the -network, would make the measured term nonzero; that is the estimator working -as specified in [Audio jitter -target](/quest/m1/audio-jitter-target/README.md). With capture fixed, the -spread left on Chromium 155+ is its 10 ms capture chunks; a release no longer -flushes the codec's lookahead chunk out ahead of its media. -Older Chromium still paces capture at the device's callback size. - Decided 2026-11-04 (maintainer, planning in [moq-dev/moq#7816](https://github.com/moq-dev/moq/pull/7816)): - ✅ Budget the default path with realistic settings, not the fixed 300 ms / - 60 ms setup above; drop the fixed-delay row from `just test push-to-talk`. - The maintainer is fine assuming the constants are the problem. + 60 ms setup [moq-dev/moq#7760](https://github.com/moq-dev/moq/pull/7760) + measured; drop the fixed-delay row from `just test push-to-talk` (once Watch + lands, per the 2026-11-06 decision below). The maintainer is fine assuming the constants are the problem. - ✅ Measure capture and output latency on real macOS hardware (`kixtop`) first; set the budget from those numbers, and keep output-side knobs until then. @@ -89,6 +25,74 @@ Decided 2026-11-04 (maintainer, planning in budget is measured on silence suppression, which [moq-dev/moq#7942](https://github.com/moq-dev/moq/pull/7942) landed. +Decided 2026-11-06 (maintainer, on +[moq-dev/moq#8009](https://github.com/moq-dev/moq/pull/8009)): how should +`just test push-to-talk` run until Watch lands, since its no-underrun check +fails on the defaults by design? + +- ✅ (a) Require [Watch](/quest/m1/audio-jitter-target/watch.md) for this + quest. The run holds a fixed 300 ms delay until then, and the defaults are + budgeted after Watch lands, on the jitter buffer users will get. +- (b) Run the defaults now and stop failing on underruns while `"auto"` is + on. +- (c) Run both rows, roughly doubling the nine-minute nightly step. + +[moq-dev/moq#8009](https://github.com/moq-dev/moq/pull/8009) landed the +analyzer's delay timeline: the page records the resolved delay over time, so +each press's `jitter_buffer` ends at the first PCM that reaches the target the +ring held when that PCM landed. The run also keeps 60 ms / 128 kbit/s Opus, +the setup the nightly passes on. On the loaded agent host, 2026-11-06 runs at +the fixed 300 ms delay had no underrun or stall but lost heads ("utterance +head was not decoded"): 2 of 22 on the default 20 ms Opus and 1 on 60 ms. +Whether 20 ms frames hit that ordering defect, which +[Push-to-talk](/quest/m1/push-to-talk/README.md) owns, more often is not +settled. What remains is to switch the page to the defaults, measure on `kixtop` again, +and set and enforce the budget. + +Measured 2026-11-05 on the defaults, before Watch, one run of 22 presses +each, Chromium 156 headless through a local relay. `kixtop` is a MacBook Air +on macOS 26 with its built-in speakers as the default output. Linux is the +agent host under heavy load, with PipeWire output. The table gives the median +and max in ms over the separated presses with a breakdown: 15 on `kixtop`, 13 +on Linux, which lost two heads. + +| Part | `kixtop` | Linux | +| --- | --- | --- | +| capture | 9 / 12 | 10 / 11 | +| packetization | 16 / 20 | 16 / 20 | +| encode | 0.5 / 0.6 | 0.5 / 8 | +| network | 2 / 12 | 3 / 5 | +| decode | 0.5 / 3 | 0.6 / 17 | +| jitter buffer | 21 / 59 | 40 / 43 | +| render | 23 / 34 | -1 / 24 | +| press to render onset | 73 / 110 | 71 / 95 | +| device | 54 / 57 | 105 / 113 | +| `"auto"` target | 40 (max 80) | 60 | +| underruns / stalls | 10 / 9 | 64 / 33 | + +Both runs failed the no-underrun check. `"auto"` targets the 0.95 quantile of +arrival lateness and accepts the re-stalls past it by design until Watch +time-stretches the ring. Push-to-talk's capture spread on this ideal path +reached 60 ms on `kixtop` and 78 ms on Linux, past a 40 or 60 ms target. +Linux's much higher count is not separated from its host load (load average +above 20) or its 43 ms render callbacks draining a 60 ms target. + +Measurement limits: the harness's headless Chromium mutes its output and reads +a fake microphone, so `device` is Chromium's projected output for the default +device and `capture` excludes the microphone's own input latency. Linux's +render callbacks are 43 ms, so its `render` can be negative by up to one +callback. The lost heads are the ordering defect +[Push-to-talk](/quest/m1/push-to-talk/README.md) owns. + +Knobs: the target's frame term (`frameDuration`, 20 ms by default) and its +measured term (`js/watch/src/audio/latency.ts`); device and render follow the +watch AudioContext's `latencyHint` (`"interactive"` in +`js/watch/src/audio/decoder.ts`), otherwise the platform. + +## Required + +- [Watch](/quest/m1/audio-jitter-target/watch.md) - `"auto"` stops underrunning by time-stretching the ring, so the run can hold the default delay + ## Related - [Audio jitter target](/quest/m1/audio-jitter-target/README.md) - the `"auto"` target this latency feeds diff --git a/quest/m1/push-to-talk/README.md b/quest/m1/push-to-talk/README.md index 1ff239eec43..d9dcc1915a6 100644 --- a/quest/m1/push-to-talk/README.md +++ b/quest/m1/push-to-talk/README.md @@ -59,8 +59,9 @@ Decided with the maintainer on (2026-11-03): "We can't ever expect 300ms latency if it takes more than 300ms to deliver a packet." No clocked live start or other startup machinery. Five ideal runs on 2026-11-04 held it on all 75 separated presses - (median 261 ms, max 294 ms). The breakdown and its knobs are in - [Push-to-talk latency](/quest/m1/push-to-talk-latency.md). + (median 261 ms, max 294 ms) at a fixed 300 ms delay, broken down on + [moq-dev/moq#7760](https://github.com/moq-dev/moq/pull/7760). The defaults' + breakdown is in [Push-to-talk latency](/quest/m1/push-to-talk-latency.md). - ✅ **No encoder recovery on re-enable** (2026-11-03). Asked first why a fatal encoder error is expected at all: it is not. No push-to-talk run reported one, and no issue reports a publisher encoder failure; the latch diff --git a/test/audio-quality/README.md b/test/audio-quality/README.md index 5a8cc36489e..b4a4a2f6f58 100644 --- a/test/audio-quality/README.md +++ b/test/audio-quality/README.md @@ -165,13 +165,15 @@ it: `capture` (worklet delivery and framing, after the media the frame still nee `packetization` (that media, placed on the media clock by the newest chunk captured before the press), `encode` (codec and container write), `network` (write to arrival, through the relay), `decode` (arrival to decoded PCM), `jitter_buffer` (the head's PCM to the first PCM that fills the -playout delay), `render` (that PCM to the render onset: the worklet hand-off and the context's -render callback), and `device`, as the stages below name them. `spread` is how far an utterance's frames +playout delay in effect as it lands, which each press reports as `delay`), `render` (that PCM to the render onset: the worklet hand-off and the context's +render callback), and `device`, as the stages below name them. The onset is placed on the render +clock, which a context rendering in large callbacks runs up to one callback ahead of the page, so +there `render` can be negative and `device` larger by as much. `spread` is how far an utterance's frames stray from a steady arrival: the latest arrival relative to its media timestamp minus the earliest, which on this ideal path is the capture graph's cadence and what `"auto"`'s jitter estimate measures. -`latency` and `path` give each one's min, median and max. +`latency`, `path` and `delay` give each one's min, median and max. -The command prints `latency`, `path`, the completeness counts, and `playback`'s underruns and +The command prints `latency`, `path`, `delay`, the completeness counts, and `playback`'s underruns and stalls, and keeps its run directory even when it passes (set `HARNESS_KEEP=0` to drop it). Nightly runs it and keeps the raw evidence alongside the audio quality rows. diff --git a/test/audio-quality/clients/js/ptt-analyze.test.ts b/test/audio-quality/clients/js/ptt-analyze.test.ts index 6e8fa9bf4ef..70602ad1a15 100644 --- a/test/audio-quality/clients/js/ptt-analyze.test.ts +++ b/test/audio-quality/clients/js/ptt-analyze.test.ts @@ -25,7 +25,7 @@ function trace() { playback: { underruns: 0, stalls: 0 }, errors: [] as string[], arrivals: [] as [number, number, number, number][], - state: { delay: 300 }, + delays: [{ at: 0, delay: 300 }], }; for (let i = 0; i < 22; i++) { const press = 10_000 + i * 10_000; @@ -85,6 +85,20 @@ test("a complete run passes, reports its latency, and splits each press exactly" }); const parts = Object.values(press.path as Record).reduce((sum, part) => sum + part, 0); expect(parts).toBe(press.audible); + expect(summary.delay).toEqual({ min: 300, median: 300, max: 300 }); +}); + +test("a target that rises before the delay fills is filled by the frame that reaches it", async () => { + // "auto" raises the target a frame as the press starts, so the sixth frame fills it. + const result = trace(); + result.delays.push({ at: result.presses[0].at, delay: 300 + FRAME }); + result.audible[0].rendered += FRAME; + result.audible[0].at += FRAME; + const { exitCode, summary } = await analyze(result); + expect(exitCode).toBe(0); + expect(summary.presses[0].delay).toBe(300 + FRAME); + expect(summary.presses[0].path.jitter_buffer).toBe(5 * FRAME); + expect(summary.presses[0].path.render).toBe(10); }); test("a frame held back by a capture burst widens its press's spread", async () => { diff --git a/test/audio-quality/clients/js/ptt-analyze.ts b/test/audio-quality/clients/js/ptt-analyze.ts index 967254b1fe2..e130a23deac 100644 --- a/test/audio-quality/clients/js/ptt-analyze.ts +++ b/test/audio-quality/clients/js/ptt-analyze.ts @@ -14,6 +14,9 @@ if (result.captureError || result.errors.length) throw new Error("invalid captur const encoded = new Map(result.encoded.map((frame) => [frame.timestamp, frame])); const sent = new Map(result.sent.map((frame) => [frame.timestamp, frame])); const arrivals = new Map(result.arrivals.map((frame) => [Math.round(frame[1] * 1000), frame])); +// The playout target the ring held at `at`: it checks each write against the delay of that moment, +// which "auto" moves with its estimate. +const delayAt = (at: number) => result.delays.findLast((entry) => entry.at <= at)?.delay; // The first decode of each timestamp; a later one would be a re-decode, not the frame's first. const decoded = new Map(); for (const frame of result.decoded) if (!decoded.has(frame.timestamp)) decoded.set(frame.timestamp, frame); @@ -52,26 +55,25 @@ const presses = result.presses // from their media timestamps, slowest minus fastest. An ideal path leaves only capture cadence. const offsets = received.map(([at, media]) => at - media); const spread = offsets.length ? Math.max(...offsets) - Math.min(...offsets) : null; + const first = input[0]; + // The first PCM that reached the delay from the head: the ring leaves its stall on it, and a + // later frame decoded before the onset only adds to a ring already playing. + const filled = input + .flatMap((frame) => decoded.get(frame.timestamp) ?? []) + .toSorted((a, b) => a.at - b.at) + .find((frame) => { + const delay = delayAt(frame.at); + return delay !== undefined && frame.timestamp + (frame.duration ?? 0) - first.timestamp >= delay * 1000; + }); // The head frame's path from the press to projected output, one boundary after another, so // the parts sum to `audible`. Capture and packetization split at the head frame's end on the // media clock, placed against the newest captured chunk delivered before the press. - const first = input[0]; const path = (() => { - if (!audible || !first || !press.capture) return null; + if (!audible || !first || !press.capture || !filled) return null; const write = sent.get(first.timestamp); const arrival = arrivals.get(first.timestamp); const output = decoded.get(first.timestamp); - // The first PCM that reached the delay from the head: the ring leaves its stall on it, and a - // later frame decoded before the onset only adds to a ring already playing. - const delay = result.state?.delay; - const filled = - delay === undefined - ? undefined - : input - .flatMap((frame) => decoded.get(frame.timestamp) ?? []) - .toSorted((a, b) => a.at - b.at) - .find((frame) => frame.timestamp + (frame.duration ?? 0) - first.timestamp >= delay * 1000); - if (!write || !arrival || !output || !filled) return null; + if (!write || !arrival || !output) return null; const pressed = press.capture.end / 1000 + (press.at - press.capture.at); const packetization = (first.timestamp + (first.duration ?? 0)) / 1000 - pressed; return { @@ -100,6 +102,7 @@ const presses = result.presses audible: audible ? audible.at - press.at : null, delivery, spread, + delay: filled ? (delayAt(filled.at) ?? null) : null, playout: audible && delivery !== null ? audible.rendered - press.at - delivery : null, output: audible ? audible.at - audible.rendered : null, path, @@ -124,7 +127,8 @@ const summary = { path: Object.fromEntries( parts.map((part) => [part, summarize(presses.map((press) => press.path?.[part] ?? null))]), ), - delay: result.state?.delay, + // The playout target each head filled. + delay: summarize(presses.map((press) => press.delay)), presses, captures: result.captures, encoders: result.encoders, diff --git a/test/audio-quality/clients/js/src/ptt.ts b/test/audio-quality/clients/js/src/ptt.ts index 3741759998f..a664c7ff499 100644 --- a/test/audio-quality/clients/js/src/ptt.ts +++ b/test/audio-quality/clients/js/src/ptt.ts @@ -27,6 +27,8 @@ const records = { encoders: [] as number[], // Each time the published rendition became unselectable (disabled or dropped) or selectable again. catalog: [] as { at: number; selectable: boolean }[], + // The resolved playout delay since the measurement's start: "auto" moves it with the estimate. + delays: [] as { at: number; delay: number }[], errors: [] as string[], }; @@ -208,6 +210,7 @@ publish.audio.out.catalog.subscribe((config) => { if (now !== selectable) records.catalog.push({ at: performance.now(), selectable: now }); selectable = now; }); +watch.player.sync.out.delay.subscribe((delay) => records.delays.push({ at: performance.now(), delay })); // Playback counts from the measurement's start, so the warmup's tune-in is not charged to it. let baseline = { underruns: 0, stalls: 0 }; const playback = () => { @@ -241,6 +244,7 @@ export const measurement = { }, reset() { for (const list of Object.values(records)) list.length = 0; + records.delays.push({ at: performance.now(), delay: watch.player.sync.out.delay.peek() }); capture.drain(); arrivals.length = 0; const stats = audio.out.stats.peek();