diff --git a/quest/m1/README.md b/quest/m1/README.md index 9136e7589c9..3f94db8b1a8 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -165,8 +165,8 @@ their end report, so an embedder's drain alert pages on real cuts. - [A/V clock](/quest/m1/av-clock.md) - the audio playhead drives Sync.reference while audio plays, through per-track sync handles - [Container reads drop backfill](/quest/m1/read-backfill.md) - the JS container consumer drops groups below its read cursor like Rust - [Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md) - a buffered `` asks for the oldest group first, so a voice reply's opening arrives first, and audio's head hold shrinks to the floor -- [Push-to-talk loses no speech](/quest/m1/push-to-talk/README.md) - consumers hold a disabled rendition, and an unmute encodes at once with the mic still open; a reordered head after a long mute is still lost at the listener -- [Push-to-talk's press-to-audible latency is chosen, not inherited](/quest/m1/push-to-talk-latency.md) - each part of an ideal-path press's 469 ms, chiefly the 300 ms delay and 115 ms device output, is reduced through its knob or kept with a reason +- [Push-to-talk loses no speech](/quest/m1/push-to-talk/README.md) - an unmute is heard from its first frame with no catalog round trip: mute moves from holding a disabled rendition to suppressed silence with comfort noise, after a route spike; capture delivers frames on a steady cadence +- [Push-to-talk's press-to-audible latency fits a budget on the defaults](/quest/m1/push-to-talk-latency.md) - press to audible on the browser defaults, excluding device output, stays under a kixtop-measured budget that `just test push-to-talk` enforces - [TS fidelity and recovery](/quest/m1/ts-reliability/README.md) - both modes have explicit profiles, automated validation, loss tests, and a dedicated guide - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS - [CI housekeeping](/quest/m1/ci-housekeeping.md) - remove the actionlint queue ignore once a release after 1.7.12 reaches the dev shell diff --git a/quest/m1/push-to-talk-latency.md b/quest/m1/push-to-talk-latency.md index c46e3b1edad..7b8a1554137 100644 --- a/quest/m1/push-to-talk-latency.md +++ b/quest/m1/push-to-talk-latency.md @@ -1,11 +1,12 @@ -# [S] Push-to-talk's press-to-audible latency is chosen, not inherited +# [M] Push-to-talk's press-to-audible latency fits a budget on the defaults ## Goal -Each part of a push-to-talk press's latency on an ideal path, from capture to -device output, is either reduced through the knob that controls it or kept -with a recorded reason. `just test push-to-talk` reports the result with the -same breakdown. +On an ideal path with the browser defaults (`` `delay: "auto"`, +20 ms Opus), push-to-talk's press-to-audible latency, excluding device output, +stays under a budget set from a real-device measurement on `kixtop`, and +`just test push-to-talk` enforces it with the per-part breakdown. Native +playback is out of scope. ## Plan @@ -67,8 +68,24 @@ network, would make the measured term nonzero; that is the estimator working as specified in [Audio jitter target](/quest/m1/audio-jitter-target/README.md). -Open: decide which knob to turn first, and whether to measure on a real -device before deciding. +Decided 2026-11-04 (maintainer, planning in +[moq-dev/moq#7816](https://github.com/moq-dev/moq/pull/7816)): + +- ✅ Budget the default path with realistic settings, not the fixed 300 ms / + 60 ms setup above; drop the fixed-delay row from `just test push-to-talk`. + The maintainer is fine assuming the constants are the problem. +- ✅ Measure capture and output latency on real macOS hardware (`kixtop`) + first; set the budget from those numbers, and keep output-side knobs until + then. +- ✅ Browser only. +- The catalog toggle forces at least a round trip of jitter buffer, so the + budget is measured once push-to-talk runs on DTX (Required below), and once + capture delivers frames steadily. + +## Required + +- [Push-to-talk suppresses silence instead of toggling the catalog](/quest/m1/push-to-talk/dtx.md) - removes the catalog round trip the budget would otherwise carry +- [Browser capture delivers audio frames on a steady cadence](/quest/m1/push-to-talk/capture-smoothing.md) - so `"auto"` measures the network, not capture bursts ## Related diff --git a/quest/m1/push-to-talk/README.md b/quest/m1/push-to-talk/README.md index 917c1a84e7c..b3a3038af32 100644 --- a/quest/m1/push-to-talk/README.md +++ b/quest/m1/push-to-talk/README.md @@ -22,6 +22,11 @@ Decided 2026-10-28 (maintainer): - **Hold, not DTX.** DTX is for silence the app can't predict. Push-to-talk knows when it is silent, and a held subscription costs no media bytes and no encoder CPU. DTX is off on main anyway ([#4919](https://github.com/moq-dev/moq/pull/4919)). + **Reversed 2026-11-04 (maintainer):** a catalog toggle makes every listener + wait out the `enabled` flip, so the jitter buffer must hold at least a round + trip. Push-to-talk moves to DTX ([Push-to-talk suppresses + silence](/quest/m1/push-to-talk/dtx.md)), and hold retires when that lands; + until then hold stays on main. - **Players hold a disabled rendition**, audio and video, by default, in every first-party player. Exports keep dropping it, since a held track sends nothing and an interleaving muxer or HLS player may wait on it. The @@ -87,6 +92,12 @@ in [Container reads drop backfill](/quest/m1/read-backfill.md). The runner fails on a head that is received but never decoded, so the proof catches this. A complete set of runs finishes this epic. +## Required + +- [Pick how the browser publisher suppresses silence](/quest/m1/push-to-talk/dtx-route.md) - measures the DTX routes +- [Push-to-talk suppresses silence instead of toggling the catalog](/quest/m1/push-to-talk/dtx.md) - replaces hold with DTX +- [Browser capture delivers audio frames on a steady cadence](/quest/m1/push-to-talk/capture-smoothing.md) - so `"auto"` measures the network + ## Closes - [#7101](https://github.com/moq-dev/moq/issues/7101) - a disabled audio rendition is deselected, so the start of the next utterance is lost diff --git a/quest/m1/push-to-talk/capture-smoothing.md b/quest/m1/push-to-talk/capture-smoothing.md new file mode 100644 index 00000000000..162d3925396 --- /dev/null +++ b/quest/m1/push-to-talk/capture-smoothing.md @@ -0,0 +1,27 @@ +# [S] Browser capture delivers audio frames on a steady cadence + +## Goal + +`@moq/publish`'s capture graph hands encoded audio frames to the network at +the frame rate, not in bursts, so the listener's `"auto"` jitter target +measures the network instead of the capture graph. On an ideal path the +per-press arrival spread drops well below the ~46 ms measured on +[moq-dev/moq#7760](https://github.com/moq-dev/moq/pull/7760); the PR records +before and after on Chromium and on kixtop. + +## Plan + +Planned 2026-11-04 with the maintainer. Measured on #7760: the capture graph +delivered 60 ms frames 25-109 ms apart, and capture plus framing took 29 ms +median, 110 ms max. In `"auto"` that spread became about 46 ms of measured +jitter per press, padding the target. + +Decided 2026-11-04 (maintainer): reduce it, not just report it. Start by +finding where the bursts form: the AudioWorklet's posting cadence, the capture +AudioContext's callback size (already `latencyHint: "interactive"` in +`js/publish/src/audio/capture.ts`), or batching between the worklet and the +encoder. Fix at the source; don't re-pace on a timer. + +## Related + +- [Audio jitter target](/quest/m1/audio-jitter-target/README.md) - the estimator that reads this spread diff --git a/quest/m1/push-to-talk/dtx-route.md b/quest/m1/push-to-talk/dtx-route.md new file mode 100644 index 00000000000..7120d97583c --- /dev/null +++ b/quest/m1/push-to-talk/dtx-route.md @@ -0,0 +1,65 @@ +# [S] Pick how the browser publisher suppresses silence + +## Goal + +A measured comparison picks the route `@moq/publish` uses to stop sending +silent Opus frames while keeping every chunk on the capture timeline, and the +maintainer approves the pick in the PR. Done when the PR records each route's +numbers and the chosen route with ✅. + +## Plan + +Planned 2026-11-04 with the maintainer, from the +[moq-dev/moq#7760](https://github.com/moq-dev/moq/pull/7760) latency +breakdown. Push-to-talk moves from the catalog toggle to DTX (see +[Push-to-talk uses DTX](/quest/m1/push-to-talk/dtx.md)), so the publisher needs +a silence suppression that works in every browser. + +Why WebCodecs DTX alone can't be used today: with `usedtx: true`, Chromium +stamps each Opus chunk as the first input timestamp plus the samples it +emitted, so every suppressed frame pulls later audio earlier (14 s of input +ended at 8.92 s), and no WebCodecs output maps a chunk back to its input +([moq-dev/moq#4783](https://github.com/moq-dev/moq/issues/4783), +[moq-dev/moq#4902](https://github.com/moq-dev/moq/pull/4902)). Firefox kept +the timeline; Safari wasn't measured until the 2026-11-04 repro below. +`usedtx` was removed in +[moq-dev/moq#4919](https://github.com/moq-dev/moq/pull/4919). + +Measured 2026-11-04 (repro in +[chromium#581458361](https://issues.chromium.org/issues/581458361), tracked by +[Chromium's Opus encoder stamps chunks from their +input](/quest/m2/chromium-opus-timestamps.md)): with +`usedtx: true`, Chrome 156 (Linux and macOS) ends 14 s of input at 8.92 s, and it +also ignores a jump in input timestamps. Firefox 157 keeps the timeline under +DTX but ignores input jumps. Safari 26.6.2 follows input jumps but never +applies `usedtx`. WebCodecs §5.6 already requires the input's timestamp, so +this is a Chromium bug, not a spec gap. + +Decided 2026-11-04 (maintainer): measure before choosing. The routes: + +1. **Gate before the encoder, stamp from our own inputs.** Confirmed + 2026-11-04 as a workaround for the Chromium bug; it is measured like the + others and is not yet the pick. The publisher decides silence itself + (energy threshold or VAD) and neither encodes nor sends silent frames. Inputs are already cut to one codec frame + each (`js/publish/src/audio/encoder.ts`, "Align those inputs to codec + frames"), and without DTX every browser emits one chunk per input in order, + so each chunk is stamped from a queue of the input timestamps fed, not from + `chunk.timestamp`. That sidesteps both Chromium bugs with no encoder + restart (today's code restarts the encoder on any input gap to re-base, + which also works but costs a re-prime per utterance). Works in every + browser. +2. **WASM libopus.** Ship an Opus encoder with real DTX and exact timestamps. + Overlaps the m2 [Opus backend](/quest/m2/audio-opus-backend.md) quest. +3. **Native WebCodecs DTX** where it keeps the timeline: only Firefox today, + so at best a later optimization once Chromium is fixed, with route 1 or 2 + as the fallback. + +Measure each on Chromium, Firefox and Safari (kixtop): timeline drift over a +tone/silence/noise pattern, bytes sent during silence, onset latency of the +first word after silence (no clipped head), CPU, bundle size, and a listening +check. Record the numbers in the PR and put the pick to the maintainer. + +## Related + +- [Push-to-talk uses DTX](/quest/m1/push-to-talk/dtx.md) - implements the route this picks +- [Opus backend](/quest/m2/audio-opus-backend.md) - the native side's Opus and DTX measurements diff --git a/quest/m1/push-to-talk/dtx.md b/quest/m1/push-to-talk/dtx.md new file mode 100644 index 00000000000..ca235a62ff2 --- /dev/null +++ b/quest/m1/push-to-talk/dtx.md @@ -0,0 +1,43 @@ +# [M] Push-to-talk suppresses silence instead of toggling the catalog + +## Goal + +Muting a push-to-talk publisher keeps its audio rendition enabled and its +subscription live, and sends no media while silent. The next utterance reaches +a listener with no catalog flip and no round trip, and listeners play comfort +noise across the silence without counting it as an underrun. The hold design, +listeners staying subscribed to a disabled rendition, is retired in the same +change. `just test push-to-talk` proves it. + +## Plan + +Planned 2026-11-04 with the maintainer, from the +[moq-dev/moq#7760](https://github.com/moq-dev/moq/pull/7760) latency +breakdown. + +Decided 2026-11-04 (maintainer), reversing the epic's 2026-10-28 "Hold, not +DTX": a catalog toggle makes every listener wait out the `enabled` flip, so +the jitter buffer must hold at least a round trip for push-to-talk to work, +and the maintainer would rather fix DTX and avoid the catalog entirely. + +- **Route:** whichever [Pick how the browser publisher suppresses + silence](/quest/m1/push-to-talk/dtx-route.md) chooses. +- **Hold retires when this lands ✅**, not before, so push-to-talk keeps + working on main meanwhile. Remove the players' default hold of a disabled + rendition and restore the hang spec's "must not select" if nothing else + relies on "should not select"; check video and exports before restoring. +- **Gaps play comfort noise ✅**, not plain silence. A suppressed span is + intentional silence: it counts as no underrun, does not grow the buffer, and + the jitter estimator ignores it + ([Audio jitter target](/quest/m1/audio-jitter-target/README.md) specifies + the estimator; record the exception there too). +- The publisher's mute becomes "suppress", not `enabled: false`. Keep + `enabled` for a real stop (leaving the call), where a catalog change is + correct. + +Update the epic's acceptance and `doc/` pages that describe hold. + +## Required + +- [Pick how the browser publisher suppresses silence](/quest/m1/push-to-talk/dtx-route.md) - the route this implements + diff --git a/quest/m2/README.md b/quest/m2/README.md index f42c9265759..484c1ddca4c 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -299,3 +299,4 @@ lite-07; bitrate enforcement is not part of this release. - [The dev shell builds Kotlin with Gradle 9](/quest/m2/kt-gradle-9.md) - `gradle_9` in the dev shell, so the Android target builds there - [The publish support panel lists only codecs publishing accepts](/quest/m2/publish-support-probe.md) - the settings panel runs the publisher's Annex B encoder probe - [A hung Python test fails instead of hanging the suite](/quest/m2/py-test-timeouts.md) - one pytest-timeout bound covers every `py/` test +- [Chromium's Opus encoder stamps chunks from their input](/quest/m2/chromium-opus-timestamps.md) - waiting on Chromium; unlocks native `usedtx` for push-to-talk diff --git a/quest/m2/chromium-opus-timestamps.md b/quest/m2/chromium-opus-timestamps.md new file mode 100644 index 00000000000..75ad61d0555 --- /dev/null +++ b/quest/m2/chromium-opus-timestamps.md @@ -0,0 +1,42 @@ +# [XS] Chromium's Opus encoder stamps chunks from their input + +## Goal + +Chromium's WebCodecs `AudioEncoder` stamps each Opus chunk with the timestamp +of the input audio it encodes, both across `usedtx` silence and across a jump +in input timestamps. Check by running the repro attached to +[chromium#581458361](https://issues.chromium.org/issues/581458361) in a released +Chrome: scenario A ends near 13.98 s with steps across +suppressed spans, and scenario C follows the +1 s jump. Delete this quest once a stable Chrome +passes, and tell [Pick how the browser publisher suppresses +silence](/quest/m1/push-to-talk/dtx-route.md)'s successor that native +`usedtx` (its route 3) is usable there. + +## Plan + +Measured 2026-11-04: Chrome 156 ends 14 s of DTX input +at 8.92 s and ignores input timestamp jumps. In +`media/audio/audio_opus_encoder.cc`, the timestamp tracker is seeded once from +the first input and advances only when a packet is emitted, so DTX's +non-transmitted frames and input gaps are both lost. WebCodecs §5.6 already +requires the input's timestamp, so this is a Chromium bug; a spec issue +([w3c/webcodecs#986](https://github.com/w3c/webcodecs/issues/986)) was opened +and closed by the maintainer for that reason. + +Decided 2026-11-04 (maintainer): file the Chromium bug +([chromium#581458361](https://issues.chromium.org/issues/581458361), Blink>Media>WebCodecs, +submitted by the maintainer) and no spec issue; track it here in m2, since +push-to-talk works around it by gating silence before the encoder and stamping +chunks from its own inputs, so nothing depends on this. + +## Waiting + +- Chromium: fix [chromium#581458361](https://issues.chromium.org/issues/581458361) + so Opus chunks carry their input's timestamp across `usedtx` silence and + input jumps. Check by running the attached repro in the latest stable Chrome + (scenario A ends near 13.98 s, scenario C follows the +1 s jump); advance it + by commenting on the bug with each stable release that still fails. + +## Related + +- [Pick how the browser publisher suppresses silence](/quest/m1/push-to-talk/dtx-route.md) - the workaround and the native-DTX route this unlocks