Skip to content
Merged
4 changes: 2 additions & 2 deletions quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,8 +165,8 @@ their end report, so an embedder's drain alert pages on real cuts.
- [A/V clock](/quest/m1/av-clock.md) - the audio playhead drives Sync.reference while audio plays, through per-track sync handles
- [Container reads drop backfill](/quest/m1/read-backfill.md) - the JS container consumer drops groups below its read cursor like Rust
- [Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md) - a buffered `<moq-watch>` asks for the oldest group first, so a voice reply's opening arrives first, and audio's head hold shrinks to the floor
- [Push-to-talk loses no speech](/quest/m1/push-to-talk/README.md) - consumers hold a disabled rendition, and an unmute encodes at once with the mic still open; a reordered head after a long mute is still lost at the listener
- [Push-to-talk's press-to-audible latency is chosen, not inherited](/quest/m1/push-to-talk-latency.md) - each part of an ideal-path press's 469 ms, chiefly the 300 ms delay and 115 ms device output, is reduced through its knob or kept with a reason
- [Push-to-talk loses no speech](/quest/m1/push-to-talk/README.md) - an unmute is heard from its first frame with no catalog round trip: mute moves from holding a disabled rendition to suppressed silence with comfort noise, after a route spike; capture delivers frames on a steady cadence
- [Push-to-talk's press-to-audible latency fits a budget on the defaults](/quest/m1/push-to-talk-latency.md) - press to audible on the browser defaults, excluding device output, stays under a kixtop-measured budget that `just test push-to-talk` enforces
- [TS fidelity and recovery](/quest/m1/ts-reliability/README.md) - both modes have explicit profiles, automated validation, loss tests, and a dedicated guide
- [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS
- [CI housekeeping](/quest/m1/ci-housekeeping.md) - remove the actionlint queue ignore once a release after 1.7.12 reaches the dev shell
Expand Down
31 changes: 24 additions & 7 deletions quest/m1/push-to-talk-latency.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
# [S] Push-to-talk's press-to-audible latency is chosen, not inherited
# [M] Push-to-talk's press-to-audible latency fits a budget on the defaults

## Goal

Each part of a push-to-talk press's latency on an ideal path, from capture to
device output, is either reduced through the knob that controls it or kept
with a recorded reason. `just test push-to-talk` reports the result with the
same breakdown.
On an ideal path with the browser defaults (`<moq-watch>` `delay: "auto"`,
20 ms Opus), push-to-talk's press-to-audible latency, excluding device output,
stays under a budget set from a real-device measurement on `kixtop`, and
`just test push-to-talk` enforces it with the per-part breakdown. Native
playback is out of scope.

## Plan

Expand Down Expand Up @@ -67,8 +68,24 @@ network, would make the measured term nonzero; that is the estimator working
as specified in [Audio jitter
target](/quest/m1/audio-jitter-target/README.md).

Open: decide which knob to turn first, and whether to measure on a real
device before deciding.
Decided 2026-11-04 (maintainer, planning in
[moq-dev/moq#7816](https://github.com/moq-dev/moq/pull/7816)):

- ✅ Budget the default path with realistic settings, not the fixed 300 ms /
60 ms setup above; drop the fixed-delay row from `just test push-to-talk`.
The maintainer is fine assuming the constants are the problem.
- ✅ Measure capture and output latency on real macOS hardware (`kixtop`)
first; set the budget from those numbers, and keep output-side knobs until
then.
- ✅ Browser only.
- The catalog toggle forces at least a round trip of jitter buffer, so the
budget is measured once push-to-talk runs on DTX (Required below), and once
capture delivers frames steadily.

## Required

- [Push-to-talk suppresses silence instead of toggling the catalog](/quest/m1/push-to-talk/dtx.md) - removes the catalog round trip the budget would otherwise carry
- [Browser capture delivers audio frames on a steady cadence](/quest/m1/push-to-talk/capture-smoothing.md) - so `"auto"` measures the network, not capture bursts

## Related

Expand Down
11 changes: 11 additions & 0 deletions quest/m1/push-to-talk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ Decided 2026-10-28 (maintainer):
- **Hold, not DTX.** DTX is for silence the app can't predict. Push-to-talk
knows when it is silent, and a held subscription costs no media bytes and no
encoder CPU. DTX is off on main anyway ([#4919](https://github.com/moq-dev/moq/pull/4919)).
**Reversed 2026-11-04 (maintainer):** a catalog toggle makes every listener
wait out the `enabled` flip, so the jitter buffer must hold at least a round
trip. Push-to-talk moves to DTX ([Push-to-talk suppresses
silence](/quest/m1/push-to-talk/dtx.md)), and hold retires when that lands;
until then hold stays on main.
- **Players hold a disabled rendition**, audio and video, by default, in
every first-party player. Exports keep dropping it, since a held track
sends nothing and an interleaving muxer or HLS player may wait on it. The
Expand Down Expand Up @@ -87,6 +92,12 @@ in [Container reads drop backfill](/quest/m1/read-backfill.md).
The runner fails on a head that is received but never decoded, so the proof
catches this. A complete set of runs finishes this epic.

## Required

- [Pick how the browser publisher suppresses silence](/quest/m1/push-to-talk/dtx-route.md) - measures the DTX routes
- [Push-to-talk suppresses silence instead of toggling the catalog](/quest/m1/push-to-talk/dtx.md) - replaces hold with DTX
- [Browser capture delivers audio frames on a steady cadence](/quest/m1/push-to-talk/capture-smoothing.md) - so `"auto"` measures the network

## Closes

- [#7101](https://github.com/moq-dev/moq/issues/7101) - a disabled audio rendition is deselected, so the start of the next utterance is lost
Expand Down
27 changes: 27 additions & 0 deletions quest/m1/push-to-talk/capture-smoothing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# [S] Browser capture delivers audio frames on a steady cadence

## Goal

`@moq/publish`'s capture graph hands encoded audio frames to the network at
the frame rate, not in bursts, so the listener's `"auto"` jitter target
measures the network instead of the capture graph. On an ideal path the
per-press arrival spread drops well below the ~46 ms measured on
[moq-dev/moq#7760](https://github.com/moq-dev/moq/pull/7760); the PR records
before and after on Chromium and on kixtop.

## Plan

Planned 2026-11-04 with the maintainer. Measured on #7760: the capture graph
delivered 60 ms frames 25-109 ms apart, and capture plus framing took 29 ms
median, 110 ms max. In `"auto"` that spread became about 46 ms of measured
jitter per press, padding the target.

Decided 2026-11-04 (maintainer): reduce it, not just report it. Start by
finding where the bursts form: the AudioWorklet's posting cadence, the capture
AudioContext's callback size (already `latencyHint: "interactive"` in
`js/publish/src/audio/capture.ts`), or batching between the worklet and the
encoder. Fix at the source; don't re-pace on a timer.

## Related

- [Audio jitter target](/quest/m1/audio-jitter-target/README.md) - the estimator that reads this spread
65 changes: 65 additions & 0 deletions quest/m1/push-to-talk/dtx-route.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# [S] Pick how the browser publisher suppresses silence

## Goal

A measured comparison picks the route `@moq/publish` uses to stop sending
silent Opus frames while keeping every chunk on the capture timeline, and the
maintainer approves the pick in the PR. Done when the PR records each route's
numbers and the chosen route with ✅.

## Plan

Planned 2026-11-04 with the maintainer, from the
[moq-dev/moq#7760](https://github.com/moq-dev/moq/pull/7760) latency
breakdown. Push-to-talk moves from the catalog toggle to DTX (see
[Push-to-talk uses DTX](/quest/m1/push-to-talk/dtx.md)), so the publisher needs
a silence suppression that works in every browser.

Why WebCodecs DTX alone can't be used today: with `usedtx: true`, Chromium
stamps each Opus chunk as the first input timestamp plus the samples it
emitted, so every suppressed frame pulls later audio earlier (14 s of input
ended at 8.92 s), and no WebCodecs output maps a chunk back to its input
([moq-dev/moq#4783](https://github.com/moq-dev/moq/issues/4783),
[moq-dev/moq#4902](https://github.com/moq-dev/moq/pull/4902)). Firefox kept
the timeline; Safari wasn't measured until the 2026-11-04 repro below.
`usedtx` was removed in
[moq-dev/moq#4919](https://github.com/moq-dev/moq/pull/4919).

Measured 2026-11-04 (repro in
[chromium#581458361](https://issues.chromium.org/issues/581458361), tracked by
[Chromium's Opus encoder stamps chunks from their
input](/quest/m2/chromium-opus-timestamps.md)): with
`usedtx: true`, Chrome 156 (Linux and macOS) ends 14 s of input at 8.92 s, and it
also ignores a jump in input timestamps. Firefox 157 keeps the timeline under
DTX but ignores input jumps. Safari 26.6.2 follows input jumps but never
applies `usedtx`. WebCodecs §5.6 already requires the input's timestamp, so
this is a Chromium bug, not a spec gap.

Decided 2026-11-04 (maintainer): measure before choosing. The routes:

1. **Gate before the encoder, stamp from our own inputs.** Confirmed
2026-11-04 as a workaround for the Chromium bug; it is measured like the
others and is not yet the pick. The publisher decides silence itself
(energy threshold or VAD) and neither encodes nor sends silent frames. Inputs are already cut to one codec frame
each (`js/publish/src/audio/encoder.ts`, "Align those inputs to codec
frames"), and without DTX every browser emits one chunk per input in order,
so each chunk is stamped from a queue of the input timestamps fed, not from
`chunk.timestamp`. That sidesteps both Chromium bugs with no encoder
restart (today's code restarts the encoder on any input gap to re-base,
which also works but costs a re-prime per utterance). Works in every
browser.
2. **WASM libopus.** Ship an Opus encoder with real DTX and exact timestamps.
Overlaps the m2 [Opus backend](/quest/m2/audio-opus-backend.md) quest.
3. **Native WebCodecs DTX** where it keeps the timeline: only Firefox today,
so at best a later optimization once Chromium is fixed, with route 1 or 2
as the fallback.

Measure each on Chromium, Firefox and Safari (kixtop): timeline drift over a
tone/silence/noise pattern, bytes sent during silence, onset latency of the
first word after silence (no clipped head), CPU, bundle size, and a listening
check. Record the numbers in the PR and put the pick to the maintainer.

## Related

- [Push-to-talk uses DTX](/quest/m1/push-to-talk/dtx.md) - implements the route this picks
- [Opus backend](/quest/m2/audio-opus-backend.md) - the native side's Opus and DTX measurements
43 changes: 43 additions & 0 deletions quest/m1/push-to-talk/dtx.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# [M] Push-to-talk suppresses silence instead of toggling the catalog

## Goal

Muting a push-to-talk publisher keeps its audio rendition enabled and its
subscription live, and sends no media while silent. The next utterance reaches
a listener with no catalog flip and no round trip, and listeners play comfort
noise across the silence without counting it as an underrun. The hold design,
listeners staying subscribed to a disabled rendition, is retired in the same
change. `just test push-to-talk` proves it.

## Plan

Planned 2026-11-04 with the maintainer, from the
[moq-dev/moq#7760](https://github.com/moq-dev/moq/pull/7760) latency
breakdown.

Decided 2026-11-04 (maintainer), reversing the epic's 2026-10-28 "Hold, not
DTX": a catalog toggle makes every listener wait out the `enabled` flip, so
the jitter buffer must hold at least a round trip for push-to-talk to work,
and the maintainer would rather fix DTX and avoid the catalog entirely.

- **Route:** whichever [Pick how the browser publisher suppresses
silence](/quest/m1/push-to-talk/dtx-route.md) chooses.
- **Hold retires when this lands ✅**, not before, so push-to-talk keeps
working on main meanwhile. Remove the players' default hold of a disabled
rendition and restore the hang spec's "must not select" if nothing else
relies on "should not select"; check video and exports before restoring.
- **Gaps play comfort noise ✅**, not plain silence. A suppressed span is
intentional silence: it counts as no underrun, does not grow the buffer, and
the jitter estimator ignores it
([Audio jitter target](/quest/m1/audio-jitter-target/README.md) specifies
the estimator; record the exception there too).
- The publisher's mute becomes "suppress", not `enabled: false`. Keep
`enabled` for a real stop (leaving the call), where a catalog change is
correct.

Update the epic's acceptance and `doc/` pages that describe hold.

## Required

- [Pick how the browser publisher suppresses silence](/quest/m1/push-to-talk/dtx-route.md) - the route this implements

1 change: 1 addition & 0 deletions quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,3 +299,4 @@ lite-07; bitrate enforcement is not part of this release.
- [The dev shell builds Kotlin with Gradle 9](/quest/m2/kt-gradle-9.md) - `gradle_9` in the dev shell, so the Android target builds there
- [The publish support panel lists only codecs publishing accepts](/quest/m2/publish-support-probe.md) - the settings panel runs the publisher's Annex B encoder probe
- [A hung Python test fails instead of hanging the suite](/quest/m2/py-test-timeouts.md) - one pytest-timeout bound covers every `py/` test
- [Chromium's Opus encoder stamps chunks from their input](/quest/m2/chromium-opus-timestamps.md) - waiting on Chromium; unlocks native `usedtx` for push-to-talk
42 changes: 42 additions & 0 deletions quest/m2/chromium-opus-timestamps.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# [XS] Chromium's Opus encoder stamps chunks from their input

## Goal

Chromium's WebCodecs `AudioEncoder` stamps each Opus chunk with the timestamp
of the input audio it encodes, both across `usedtx` silence and across a jump
in input timestamps. Check by running the repro attached to
[chromium#581458361](https://issues.chromium.org/issues/581458361) in a released
Chrome: scenario A ends near 13.98 s with steps across
suppressed spans, and scenario C follows the +1 s jump. Delete this quest once a stable Chrome
passes, and tell [Pick how the browser publisher suppresses
silence](/quest/m1/push-to-talk/dtx-route.md)'s successor that native
`usedtx` (its route 3) is usable there.

## Plan

Measured 2026-11-04: Chrome 156 ends 14 s of DTX input
at 8.92 s and ignores input timestamp jumps. In
`media/audio/audio_opus_encoder.cc`, the timestamp tracker is seeded once from
the first input and advances only when a packet is emitted, so DTX's
non-transmitted frames and input gaps are both lost. WebCodecs §5.6 already
requires the input's timestamp, so this is a Chromium bug; a spec issue
([w3c/webcodecs#986](https://github.com/w3c/webcodecs/issues/986)) was opened
and closed by the maintainer for that reason.

Decided 2026-11-04 (maintainer): file the Chromium bug
([chromium#581458361](https://issues.chromium.org/issues/581458361), Blink>Media>WebCodecs,
submitted by the maintainer) and no spec issue; track it here in m2, since
push-to-talk works around it by gating silence before the encoder and stamping
chunks from its own inputs, so nothing depends on this.

## Waiting

- Chromium: fix [chromium#581458361](https://issues.chromium.org/issues/581458361)
so Opus chunks carry their input's timestamp across `usedtx` silence and
input jumps. Check by running the attached repro in the latest stable Chrome
(scenario A ends near 13.98 s, scenario C follows the +1 s jump); advance it
by commenting on the bug with each stable release that still fails.

## Related

- [Pick how the browser publisher suppresses silence](/quest/m1/push-to-talk/dtx-route.md) - the workaround and the native-DTX route this unlocks
Loading