diff --git a/quest/m0/3477-watch-auto-latency.md b/quest/m0/3477-watch-auto-latency.md deleted file mode 100644 index 5580ded276b..00000000000 --- a/quest/m0/3477-watch-auto-latency.md +++ /dev/null @@ -1,54 +0,0 @@ -# [S] js/watch: the measured auto latency is proven clean in a browser against a real relay - -## Goal - -The "Real-time" / auto preset plays clean audio on a LAN and against the -public relay: zero underruns after convergence and no skip-aheads in steady -state, proven by the browser harness and by a manual run on Chrome and -Safari, not only by the unit replay. The publisher-side audio encoder lag the -same report observed is measured in the same manual run and either confirmed -fixed or turned into a root-cause quest. - -Boundaries: the estimator and the ring re-buffer themselves are PR #3517 on -`dev`; this quest only verifies them where the PR could not. Convergence -still uses skip-ahead and silence; time-stretching is -[Time stretch](/quest/m2/watch-audio-time-stretch.md). - -## Plan - -Branch from dev. #3517 lands the measured arrival `spread` per track, the -`max(advertised, measured)` target, one chunk of ring slack, re-stall on -underrun, and an underrun counter in the stats panel, verified with synthetic -replay traces. It leaves three things undone, which are this quest: - -- Browser assertion: the `test/smoke` JS driver plays `bbb` from a local - `moq import ts` with ffmpeg's default PES packing at auto and asserts the - underrun counter stays at zero after convergence, on the isolated and the - postMessage ring paths. -- Recorded traces: the arrival traces from #3477 live on the reporter's fork. - Attach a trimmed copy under `js/watch` and replay them through both rings - in `replay.test.ts` in place of the synthetic traces of the same shape. -- Manual run against the public relay on Chrome and Safari, the two rows the - issue measured, with a real microphone and 40 ms or more of added RTT. - Measure the audio encoder's input-to-output lag in the same run using the - reporter's instrumented harness (fork branch `debug/rt-audio`); #3518 fixed - the known cause (the encoder was rebuilt on every subscriber churn), so the - 7.35 s lag and the 88 to 275 ms/s drift the issue reported stand - unconfirmed. If drift survives, the suspects are `writeFrame` opening a - group per audio frame under WebTransport stream credit and the main-thread - task queue that delivers encoder output; turn that into its own quest - rather than fixing it here. - -## Required - -- PR #3517 has merged to `dev` - -## Closes - -- [#3477](https://github.com/moq-dev/moq/issues/3477) - close this issue when the quest finishes -- [#2812](https://github.com/moq-dev/moq/issues/2812) - the iOS stutter report this reproduces on desktop; close it with this - -## Related - -- [Plan: A/V clock](/quest/m1/plan-av-clock.md) - the audio-master clock that follows the estimator -- [Time stretch](/quest/m2/watch-audio-time-stretch.md) - convergence without skips or silence, on top of this diff --git a/quest/m0/3479-mux-jitter-flush-span.md b/quest/m0/3479-mux-jitter-flush-span.md index 934b32f05c9..0f6cb0c5d90 100644 --- a/quest/m0/3479-mux-jitter-flush-span.md +++ b/quest/m0/3479-mux-jitter-flush-span.md @@ -61,5 +61,5 @@ The work: ## Related -- [Auto latency](/quest/m0/3477-watch-auto-latency.md) - reads this field as its codec floor +- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - reads this field as its codec floor - [#3208](/quest/m1/3208-make-2-5-ms-opus-frame-durations-work-across-bindings.md) - the same ceiling rule for Opus frame durations diff --git a/quest/m0/README.md b/quest/m0/README.md index dd38c78d83c..ffbcdec6d73 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -15,7 +15,7 @@ regression test per Root Cause First. ## Quests -- [Auto latency](/quest/m0/3477-watch-auto-latency.md) - js/watch: the measured auto latency (#3517 on dev) is proven clean in a browser and against the public relay +- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the audio playout target is a measured, ported estimate of arrival timing in both languages, not a round-trip guess - [Jitter estimator](/quest/m0/3479-mux-jitter-flush-span.md) - moq-mux: catalog jitter is the publisher's maximum flush span, never a running minimum - [WebKit gate](/quest/m0/webkit-webtransport-gate.md) - js/net: every WebKit engine takes the WebSocket path, not just the Safari brand, so iOS Chrome and Firefox stop freezing after two minutes - [IETF error codes](/quest/m0/ietf-error-codes.md) - every code on a moq-transport wire is a registered value for the negotiated draft, requests and stream resets alike diff --git a/quest/m0/audio-jitter-target/README.md b/quest/m0/audio-jitter-target/README.md new file mode 100644 index 00000000000..03c950236d7 --- /dev/null +++ b/quest/m0/audio-jitter-target/README.md @@ -0,0 +1,58 @@ +# Audio jitter target + +## Goal + +The audio playout target is a measured estimate of arrival timing, ported from +a known-good implementation, in the browser and natively alike. The round-trip +formula is gone. `max(20ms, 1.25 x minRtt)` sized the buffer for a single +retransmit, which describes how long the network takes to recover a loss and +not how unevenly a publisher emits frames, so a sender flushing 100 ms of media +at once got a target far too shallow to play through the next flush. One +written algorithm, cited by both implementations, produces the same target from +the same arrival trace. + +Boundaries: convergence still uses skip-ahead and silence, so playing slightly +faster or slower to converge stays [Time +stretch](/quest/m2/watch-audio-time-stretch.md). No packet loss concealment. +Video keeps its own target; making the audio playhead the clock is [Plan: A/V +clock](/quest/m1/plan-av-clock.md). + +## Plan + +Three quests: the survey and the written algorithm first, then one +implementation per language against it. The two implementations are +independent once the spec lands and may run in parallel. + +PR #3517 stays open and is fixed in place by the watch quest rather than +restarted. It already deleted the RTT term and plumbed a per-track arrival +`spread` through `Container.Consumer`, measured at container frame arrival and +before the age budget can skip a group, which is the right observation point. +What it got wrong is the estimator on top. A viewer on that branch reads a +14.56 s jitter buffer in auto, and the cause is reproduced in +[Spec](/quest/m0/audio-jitter-target/spec.md): the "plus one frame" term is +learned from the first gap between observed timestamps, so a tune-in across a +stale group sets it to seconds, and nothing clamps the target or bounds how +fast it rises. + +Native has no jitter buffer at all. `rs/moq-audio`'s decode `Config` carries +only `latency_max`, an upper bound before skipping a stalled group, and its own +doc comment promises "a companion `latency_min` for jitter-buffer padding will +land in a follow-up". This questline is that follow-up. + +## Quests + +- [Spec](/quest/m0/audio-jitter-target/spec.md) - survey what already exists, then write the algorithm down once +- [Watch](/quest/m0/audio-jitter-target/watch.md) - js/watch and js/hang implement it, fixing #3517 in place +- [Native](/quest/m0/audio-jitter-target/native.md) - rs/moq-audio grows `latency_min` from the same algorithm + +## Closes + +- [#3477](https://github.com/moq-dev/moq/issues/3477) - close this issue when the questline finishes +- [#2812](https://github.com/moq-dev/moq/issues/2812) - the iOS stutter report the same estimator fixes + +## Related + +- [Jitter estimator](/quest/m0/3479-mux-jitter-flush-span.md) - the advertised flush span, whose relationship to the measured target the spec settles +- [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - the automated proof, built on its own schedule +- [Time stretch](/quest/m2/watch-audio-time-stretch.md) - inaudible convergence, on top of this +- [Plan: A/V clock](/quest/m1/plan-av-clock.md) - the clock this target eventually feeds diff --git a/quest/m0/audio-jitter-target/native.md b/quest/m0/audio-jitter-target/native.md new file mode 100644 index 00000000000..1c05512a2b3 --- /dev/null +++ b/quest/m0/audio-jitter-target/native.md @@ -0,0 +1,62 @@ +# [M] rs/moq-audio: a measured jitter buffer, not just an upper bound + +## Goal + +Native playback holds a jitter buffer sized by the algorithm from +[Spec](/quest/m0/audio-jitter-target/spec.md), and passes the same conformance +vector as the browser on the same trace. `moq play` on a path with uneven +arrivals plays through a flush without underrunning, where today it has nothing +to absorb one. + +Boundaries: no time-stretching and no loss concealment, matching the browser. +Video keeps its own path. + +## Plan + +`rs/moq-audio` has no jitter buffer at all. `decode::Config` carries only +`latency_max`, an upper bound before skipping a stalled group, and its doc +comment already promises the companion: + +> The `_max` suffix is a reminder that we never *add* latency here: the consumer +> skips only when newer data is already this far ahead. A companion +> `latency_min` for jitter-buffer padding will land in a follow-up. + +- Measure arrivals at the same point the browser does, on the container + consumer before the age budget can skip a group, so both languages estimate + from the same observation. +- Add `latency_min` to `decode::Config`, additive on a `#[non_exhaustive]` + struct, so it targets `main`. Its exact contract is a decision this quest + makes and records, not something to leave to the reader, because "floor" and + "fixed target" behave differently under an adaptive estimator: + + 1. **Floor.** `Some(d)` is a lower bound and the estimator may still raise + the target above it. Recommended: it matches how the browser treats a + rendition's advertised delay, and a viewer asking for more buffer rarely + means "and never more than that". + 2. **Exact target.** `Some(d)` pins the target and disables adaptation, + mirroring a fixed browser preset exactly. Choose this if preserving a + fixed-latency preset byte for byte matters more than the parallel above. + + Either way, state the precedence against the estimator and against + `latency_max`, and what happens when the requested floor exceeds the + effective `latency_max` that `Consumer` already clamps to publisher + retention. Refusing that combination loudly beats silently clamping it. +- The estimator itself is internal. Only the config field and whatever the + playback path needs to report its current target become public, and each one + is argued for rather than exposed by default. +- `rs/moq-cli` `play` gets the matching knob, and `doc/bin/cli.md` follows. Check + the examples against `--help`. +- The playback sink holds the target the way the browser's rings do: slack above + it, land on it when skipping, re-stall when dry so the next insert refills to + the target rather than playing on an empty cushion. +- Tests: the conformance vector from the spec, plus a decode-path test that + replays the same trimmed arrival trace the browser replays and asserts the + same target series. + +`latency_min` is the natural name only because `latency_max` already exists. If +it reads wrong next to a `Duration` that is a floor rather than a budget, +propose alternatives with a recommendation rather than shipping it. + +## Required + +- [Spec](/quest/m0/audio-jitter-target/spec.md) - the algorithm this implements diff --git a/quest/m0/audio-jitter-target/spec.md b/quest/m0/audio-jitter-target/spec.md new file mode 100644 index 00000000000..c4667f206c7 --- /dev/null +++ b/quest/m0/audio-jitter-target/spec.md @@ -0,0 +1,207 @@ +# [S] The audio playout target algorithm is written down once + +## Goal + +One document under `doc/concept` describes how a receiver estimates its audio +playout target from arrival timing, in enough detail that two implementations +written from it agree on the same trace. Every constant has a stated origin. A +conformance corpus, arrival traces and the target series each must produce, +lives beside it as checked-in JSON so both languages test against the same +numbers rather than against each other. + +Boundaries: no production code. The implementations are the sibling quests, and +either may send a correction back here. + +## Plan + +The survey is done; it is recorded here so the implementations do not repeat +it. What remains is the writing and the corpus. + +### What to implement against + +WebRTC's NetEq, but not the file the name suggests. `delay_manager.cc` on +current main is a 103-line composition; the algorithm lives in +`underrun_optimizer.{h,cc}`, `reorder_optimizer.{h,cc}`, `histogram.{h,cc}`, +and `packet_arrival_history.{h,cc}` under +`modules/audio_coding/neteq/`. In priority order: + +1. **Relative arrival delay**, the highest-value idea and the cheapest. + `delay(p) = max(0, (arrival(p) - arrival(ref)) - (rtp(p) - rtp(ref)))` where + `ref` is the packet in a 2 s window minimising `arrival - rtp`, that is, the + fastest recent packet, tracked with a monotonic deque. Delay against the + best-case path, not against the previous packet. An inter-arrival estimator + is structurally blind to a slow build-up: a hundred packets each 2 ms late + read as a hundred tiny observations, where this reads as a delay climbing to + 200 ms. Frame timestamps stand in for RTP timestamps and there are no + sequence numbers to unwrap. +2. **Reordered arrivals are excluded, not measured.** When a newer group + arrives before an older one, `arrival(p) - arrival(ref)` is positive while + the timestamp difference is negative, so the media-time gap between them is + *added* to the measured delay. That is the same failure this questline + exists to fix, a media-time distance leaking into a delay measurement, and it + arrives through the front door: `Container.Consumer` takes groups off + `recvGroup()` in delivery order and sorts afterwards, so out-of-order + observation is ordinary input rather than an edge case. WebRTC pairs the + formula with exactly this guard, which the description above omitted: + `DelayManager::Update` carries a `reordered` flag, `PacketArrivalHistory` + keeps reordered packets out of its min and max deques, and only the + `ReorderOptimizer` sees them, costed as delay against loss. Say what counts + as reordered here, keep those arrivals out of the reference and the + histogram, and put a reordering case in the conformance corpus. +3. **An empirical histogram with an exponential forget factor, read at a + quantile.** 20 ms buckets, 100 of them. Seed the cold start with the + decaying `0.5^(i+1)` prior. A quantile of an empirical distribution, not a + mean plus k deviations: network delay is heavy-tailed and one-sided, so a + Gaussian assumption sizes the tail wrong. +4. **Max-over-interval resampling**, 500 ms. Easy to miss and behaviourally + significant: the histogram takes one observation per interval, the maximum + in it. That decorrelates observations and turns the forget factor into a + wall-clock time constant, which is the only form worth documenting. +5. **A cold-start forget ramp**, WebRTC's `start_forget_weight`, so the first + seconds converge instead of crawling. +6. **Explicit delay-versus-loss cost minimisation** for the late tail. WebRTC's + `ReorderOptimizer` and SpeexDSP's `compute_opt_delay` reached the same shape + independently, which is decent evidence it is right. Defer it; ship the + underrun estimator first. + +Current defaults on main, for reference and not for copying blind: quantile +`0.95`, underrun forget factor `0.983`, resample interval `500ms`, start +forget weight `2`, reorder forget factor `0.9993`, `ms_per_loss_percent` `20`. +The widely repeated "0.97" is a transitional value, not head. + +Say explicitly that the forget factor decays once per *resampled observation*, +not once per arrival, and give the resulting wall-clock memory +(`500ms / (1 - 0.983)`, about 29 s). Applying a per-observation factor per +arrival is the mistake #3517 made, and a number that reads plausible either way +is the kind that survives review. + +Then settle what an interval with no arrival does. An empty interval produces no +observation and therefore no decay step, so the wall-clock memory above does not +survive a pause unless elapsed empty intervals decay the histogram too. Either +decay them and say so, or drop the wall-clock claim; the conformance corpus needs +a case with a gap in it either way. + +Define units once, for both the algorithm and the corpus: what clock arrival +times are on, what timescale frame timestamps use and how they convert, and +what a timestamp discontinuity does to the reference. Two implementations that +disagree on any of these produce different targets from the same trace while +both look correct. + +**What actually breaks #3517, reproduced.** A viewer on that branch reads a +14.56 s jitter buffer in auto. The estimator cannot produce that from its +histogram: the percentile is clamped to `BUCKETS * BUCKET`, one second. The +unbounded term is the "plus one frame" on top of it. + +`Jitter` learns that frame duration as the running minimum of positive gaps +between consecutive observed media timestamps, so the *first* gap sets it with +nothing to validate against. A tune-in that observes a frame from a stale group +and then the live edge sets it to the distance between them, seconds rather +than milliseconds. `#publish` then raises the target instantly and +unconditionally to `percentile() + step`, and lowers it by one *corrected* +step per second afterwards. Inflation is immediate; deflation is 20 ms/s. +`reanchor()` clears `#latest` but not the spacing, so the bad value outlives +the discontinuity that caused it. + +Replaying two frames 14.5 s apart in media time, then twenty minutes of +perfectly paced audio with zero real jitter: + +``` +after 2 frames: spacing=14500ms target=14.50s + + 1s live, zero jitter: target=14.48s + + 60s live, zero jitter: target=13.30s + + 300s live, zero jitter: target=8.50s + + 600s live, zero jitter: target=2.50s + + 900s live, zero jitter: target=0.02s +``` + +Two lessons for the document, both structural rather than a constant to retune: + +- **Frame duration is a codec property, not an observation.** NetEq's + packet-time units come from the codec for the same reason. Deriving it from + arbitrary timestamp arithmetic makes a tune-in artifact indistinguishable + from a real frame duration. + + Where it comes from is an open question this quest answers, because the + catalog does not carry it: `AudioConfig` has `sampleRate` and `bitrate`, and + its own comment concedes the frame duration "depends on the codec, sample + rate, etc." Opus makes this sharp, since the encoder picks 2.5, 5, 10, 20, + 40, or 60 ms independently of both. Candidates, in order of how little they + cost: the Opus TOC byte, which self-describes the frame duration in band and + needs no schema change; the decoded frame's own sample count, exact but only + known below the container layer where the estimator currently sits; or a new + catalog field, which is the expensive option and would need the full + cross-package sync (`rs/hang`, `js/hang`, `doc/concept`, and the hang draft). + Pick one and say why; do not leave it to each implementation. + + There is prior art to reconcile rather than ignore. `js/watch/src/audio/config.ts` + already derives a per-codec frame duration and then adds `WORKLET_QUANTUM`, the + AudioWorklet's fixed 128-sample render block, so 48 kHz stereo Opus reads as + 23 ms rather than 20 ms. Native has no worklet and its device buffer is a + property of the backend, so if the shared step carries the browser's quantum + and native's does not, the two produce different target series from the same + trace and the conformance corpus cannot hold. Decide whether the render block + belongs in the estimator's step at all or in the ring's slack above it, and + give native's equivalent a name. Note also that this code assumes 20 ms Opus + frames, which is the same guess the catalog cannot settle. +- **The target needs a clamp and a bounded rise.** WebRTC clamps in + `DelayConstraints` (`delay_constraints.{h,cc}`), applied by `DecisionLogic` + rather than by the delay manager, with a hard ceiling at 75% of buffer + capacity. #3517 ported neither the clamp nor a rise limit, so any single bad + observation is permanent on the timescale a viewer will wait. + +The forget factor is a separate and smaller divergence: `0.9993` is NetEq's +*reorder* factor where the underrun path uses `0.983`, applied per arrival with +no 500 ms resampling. Worth fixing, but it is not what produces 14.56 s. + +### Licensing + +WebRTC is BSD-3 plus a PATENTS grant. Copying source means carrying a third +license and its disclaimer inside a `MIT OR Apache-2.0` repository. Implement +from the described algorithm instead, in `f64`, citing WebRTC in a comment as +the design's source. The Q15/Q30 fixed point exists for 2010-era DSPs; floating +point is simpler here and removes a whole class of cross-language parity bug. + +Do not read Asterisk's `main/jitterbuf.c` (GPLv2) or FreeSWITCH's +`switch_jitterbuffer.c` (MPL 1.1) while writing this. SpeexDSP's `jitter.c` is +BSD-3 and safe to read for the cost-minimisation idea. + +### Why not a dependency + +The `neteq` crate (`security-union/videocall-rs`, MIT OR Apache-2.0, actively +released, ships a WASM build) is the only viable candidate and was rejected: +its relative-arrival-delay step takes a percentile over per-packet deltas each +clamped at zero, which cannot see a build-up, the one property the design +exists for; its `ReorderOptimizer` is configured but never implemented; and it +is a whole NetEq, sample-rate-locked ring buffers and WSOLA and PLC, where this +needs a few hundred lines of estimator. Its shape would leak into our API. +Nothing else is close: `webrtc-rs` and `str0m` deliberately leave playout to +the application, GStreamer's `rtpjitterbuffer` solves clock skew rather than +target sizing and is LGPL, and no browser-usable JS port exists. + +### The document + +Write `doc/concept/`, covering the observation point, the estimator, rise and +fall, startup and re-anchoring, and the clamp. Specifically: + +- Measure at container frame arrival and before the age budget can skip a + group. #3517 has this right, and it matters: a target derived from what + survives the budget only ever confirms the budget it was cut to. +- Rise and fall asymmetry is the "too aggressive or not aggressive enough" + dial, so it gets a justification rather than a number. +- The clamp reads the rendition's advertised flush span from [Jitter + estimator](/quest/m0/3479-mux-jitter-flush-span.md). That field is a + publisher-declared floor and the estimate is a measurement of the network: + say plainly which is which, because the shared word "jitter" invites + confusion. #3517 combines them with a max on the grounds they describe the + same quantity; that reasoning does not survive the distinction, so revisit it. +- The document is normative, not either implementation. Parity is held by the + conformance corpus, one JSON fixture set run by both test suites, the way + wire behaviour is held aligned today. A wasm core shared into the browser + audio path is a larger architectural commitment than this warrants. + +Not an IETF draft: nothing here is negotiated on the wire, and a receiver that +chooses differently is still conformant. + +## Related + +- [Jitter estimator](/quest/m0/3479-mux-jitter-flush-span.md) - supplies the advertised flush span the clamp reads diff --git a/quest/m0/audio-jitter-target/watch.md b/quest/m0/audio-jitter-target/watch.md new file mode 100644 index 00000000000..77441ca232f --- /dev/null +++ b/quest/m0/audio-jitter-target/watch.md @@ -0,0 +1,62 @@ +# [M] js/watch: the playout target follows the spec, proven in a real browser + +## Goal + +`js/watch` computes its auto target with the algorithm from +[Spec](/quest/m0/audio-jitter-target/spec.md) and passes the conformance +vector. The "Real-time" preset plays clean audio on a LAN and against the +public relay: zero underruns after convergence and no skip-aheads in steady +state, on both the isolated and the postMessage ring paths, confirmed by a +manual run on Chrome and Safari with a real microphone and 40 ms or more of +added RTT. + +Boundaries: the ring's slack, re-stall and underrun counter are mechanical and +already correct in #3517; this quest changes the estimator above them. + +## Plan + +Fix PR #3517 in place on `dev` rather than reopening it. It is a published +`@moq/watch` break (the `Sync` input `probe` is removed) and `dev` already +carries the `delay`/`buffer` split it is expressed in. Resolve its conflicts +with `dev` first, since it currently does not merge. + +- Replace the estimator in `js/hang/src/container/jitter.ts` with the spec's, + and run the conformance corpus in `jitter.test.ts`. Keep the arrival + plumbing, the `spread` getter, and the measurement point: those were right. +- The reproduced 14.56 s runaway is the regression test to write first, before + any replacement: two frames whose media timestamps are far apart, then paced + audio with zero real jitter, asserting the target stays near the frame + duration. On today's branch it reads 14.5 s and needs about fifteen minutes + of clean audio to unwind. Take the frame duration from the source the spec + settles on rather than from observed timestamps, clamp the target, and bound + how fast it rises; [Spec](/quest/m0/audio-jitter-target/spec.md) has the + detail and the candidates. +- Check what the viewer's saved preset does on load. The element defaults + `delay` to `"auto"` and nothing in `demo/` overrides it, yet a fresh session + came up on the 100 ms chip, so something is restoring or overriding it. Pin + that down: a stored preference silently winning over the default is its own + bug, and it also means auto gets far less real exposure than it looks like. +- Keep the ring work as it stands: one chunk of slack above the target, landing + back on the target when skipping, re-stalling on empty, and the underrun + counter reaching the stats panel and the buffering indicator. +- Replay the recorded traces from #3477 rather than synthetic ones of the same + shape. They are on the reporter's fork + (`fperex/moq`, branch `debug/rt-audio`) with the raw ndjson attached to + release `rt-audio-traces-2026-09-06`. Trim a copy into the repository and + replay it through both rings in `replay.test.ts`. +- Manual run against the public relay on Chrome and Safari, the two rows the + issue measured. Measure the publisher's audio encoder input-to-output lag in + the same run using the reporter's instrumented harness; #3518 fixed the known + cause, so the 7.35 s lag and the 88 to 275 ms/s drift the issue reported + stand unconfirmed. If drift survives, the suspects are `writeFrame` opening a + group per audio frame under WebTransport stream credit and the main-thread + task queue delivering encoder output. Turn that into its own quest rather + than fixing it here. + +Two sibling quests on `main` touch the same audio path +(`3479-watch-audio-identity`, `publish-audio-encoder-lag`), for conflict triage +when `main` next merges into `dev`. + +## Required + +- [Spec](/quest/m0/audio-jitter-target/spec.md) - the algorithm this implements diff --git a/quest/m1/plan-av-clock.md b/quest/m1/plan-av-clock.md index 2e7750a2cb2..ecdb43edbb1 100644 --- a/quest/m1/plan-av-clock.md +++ b/quest/m1/plan-av-clock.md @@ -40,5 +40,5 @@ implementation quest, and re-estimate it. ## Related -- [Auto latency](/quest/m0/3477-watch-auto-latency.md) - the estimator this sits on +- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the estimator this sits on - [Time stretch](/quest/m2/watch-audio-time-stretch.md) - stretching needs a clock to converge toward diff --git a/quest/m2/README.md b/quest/m2/README.md index 93b59a10138..a2b6e98ef1b 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -32,6 +32,8 @@ can act on. Each still carries its own plan and regression test. - [Bandwidth estimate release](/quest/m2/web-transport-bandwidth-estimate.md) - web-transport-quinn reports quinn's BBR bandwidth estimate and ships a release carrying it - [#2847](/quest/m2/2847-the-quinn-backends-send-bandwidth-estimate-is-cwnd-rtt.md) - quinn backend: bump to the releases that report the controller bandwidth estimate instead of cwnd/rtt - [Safari WebTransport](/quest/m2/safari-webtransport.md) - WebKit browsers return to WebTransport once WebKit 319818 ships fixed +- [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - a playout latency regression fails a run instead of arriving as a bug report +- [Latency ledger](/quest/m2/latency-ledger.md) - a session reports where its end-to-end audio delay went, stage by stage - [Benchmark comparisons](/quest/m2/performance-comparisons.md) - retained evidence, repeated paired runs, and uncertainty for performance claims - [Relay profiling](/quest/m2/performance-profiles.md) - reproducible CPU and allocation captures under the existing workloads - [Browser benchmarks](/quest/m2/browser-benchmarks.md) - measure JS transport, container, decode, and render costs in an identified browser diff --git a/quest/m2/audio-quality-harness/README.md b/quest/m2/audio-quality-harness/README.md new file mode 100644 index 00000000000..bad61704479 --- /dev/null +++ b/quest/m2/audio-quality-harness/README.md @@ -0,0 +1,56 @@ +# Audio quality harness + +## Goal + +A regression in audio playout latency fails a run instead of arriving as a bug +report. The harness plays a broadcast over an impaired path, counts what the +listener would actually have heard (underruns, short quanta, discarded samples, +skip-aheads) and what each stage of the pipeline contributed to the delay, and +grades the result against a checked-in budget. It runs in the browser and +natively, on the same jitter profiles, reporting the same numbers. + +Boundaries: audio only. The stage breakdown is defined generically so video can +adopt it later, but no video assertion ships here. No perceptual scoring: the +grade is glitches and latency, not an opinion about how it sounds. + +## Plan + +Two quests, browser first, because that is where the traces and the reported +bug both are. The native lane follows against the same budgets and the same +metric schema, so the two implementations can be compared rather than merely +both passing. + +The starting point is not a blank page. The reporter on #3477 already built a +working browser harness on their fork (`fperex/moq`, branch `debug/rt-audio`): +a CDP driver, a beacon sink, a trace analyzer, a ring replay, a five-scenario +`bench.sh`, and a `compare.mjs` that prints before-and-after tables, with 130 +raw ndjson traces attached to release `rt-audio-traces-2026-09-06`. Upstream +that rather than reinventing it. + +Jitter comes from the seeded userspace UDP shaper in [Impaired +path](/quest/m0/transport-impairment-profile.md), not from a fake arrival clock, +so the transport's own contribution is measured rather than assumed. That quest +builds the shaper inside the relay drills' support code; it has to come out into +something a JS harness process can also put in front of a relay. Extracting it +is part of the browser quest. + +Loss, reorder and rate limiting stay switched off in the profiles here. The +buffer's job is absorbing arrival spread, and mixing congestion response into an +audio quality number makes a failure hard to attribute. + +Nightly, not per-PR: the matrix is jitter profiles by runtime by codec and +sample rate, which is more than a merge gate should carry, and `nightly.yml` +already exists for exactly this trade. Budgets are keyed by the full row, since +each of those dimensions moves the expected floor. + +## Quests + +- [Browser](/quest/m2/audio-quality-harness/browser.md) - upstream the fork's harness, grade it against a budget, run it nightly +- [Native](/quest/m2/audio-quality-harness/native.md) - the same profiles and budgets through `moq play` on a dummy device + +## Related + +- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the estimator this exists to keep honest +- [Latency ledger](/quest/m2/latency-ledger.md) - promotes this harness's probes into a public API +- [Failure artifacts](/quest/m0/qa-failure-artifacts.md) - keeps the run directory and trace of a failing run +- [Time stretch](/quest/m2/watch-audio-time-stretch.md) - graded by this harness once it lands diff --git a/quest/m2/audio-quality-harness/browser.md b/quest/m2/audio-quality-harness/browser.md new file mode 100644 index 00000000000..ef6d63928c9 --- /dev/null +++ b/quest/m2/audio-quality-harness/browser.md @@ -0,0 +1,76 @@ +# [L] The browser audio lane is graded against a budget, nightly + +## Goal + +`just test audio-quality` plays a broadcast in a headless browser over a seeded, +impaired path and prints a graded table: underruns, short quanta, discarded +samples, skip-aheads, and where the end-to-end delay went. It fails when a +profile exceeds its budget. A nightly job runs the full matrix and keeps the +run directory of a failure. + +The same run works on both ring paths, isolated and postMessage, since the +production path is the one without cross-origin isolation. + +## Plan + +Upstream the reporter's harness from `fperex/moq` branch `debug/rt-audio` +rather than rebuilding it, keeping the attribution. It already has the CDP +driver, the beacon sink, the analyzer, the ring replay, `bench.sh`'s five +scenarios, and `compare.mjs`. What it does not have is a home in `test/`, a +budget, or a schedule. + +- Land the driver and analyzer under `test/`, alongside the existing `smoke` + and `drill` lanes, wired into the `justfile` the way they are. Playwright is + already in the tree for the harness quests, so prefer it over a bespoke CDP + driver if the switch is cheap; if it is not, say so and keep CDP. +- Keep the instrumentation ad-hoc for now. The probes stay a debug surface, not + public API; promoting them is [Latency + ledger](/quest/m2/latency-ledger.md), which nothing here waits on. +- The metric schema is the deliverable that outlives this quest, because the + native lane and the ledger both have to emit the same thing. It is a + contract, so write it as one: + - Every counter defined, not just the two that are obvious. An underrun is a + quantum the ring could only partly fill; a skip-ahead is a re-anchor that + discarded buffered audio; short quanta and discarded samples need the same + treatment rather than being left to the reader. + - Units and clock domain stated once. Durations in one unit, and every + timestamp on a named clock, so a browser value and a native value are the + same measurement. Naming the clock is not enough on its own: the stages span + the publisher, the relay and the viewer, which are separate processes and + often separate machines, so either reduce every timestamp to one common + monotonic reference or define the calibration and drift correction that maps + between them. The ledger's sum-to-end-to-end identity is arithmetic across + these values, and it is meaningless if they sit on unaligned clocks. + - Aggregation stated per metric: a count over the run, a max, or a + percentile. "Underruns: 3" and "underruns: 3/s" grade differently. + - Stages as exclusive, non-overlapping spans (capture, encode, publish flush, + network, jitter buffer, decode, render). The ledger's sum-to-end-to-end + identity is unimplementable if two stages can claim the same milliseconds, + and an unaccounted remainder is the finding, so give it a name too. +- Extract the seeded shaper from [Impaired + path](/quest/m0/transport-impairment-profile.md) into something that runs as + its own process in front of a relay, so this harness and the drills share one + impairment implementation. Assert the shaper actually treated traffic: a + profile that silently did nothing turns an impaired run into an unimpaired + pass. +- Profiles: near-zero, mild, bursty (the flush-span shape from #3477), and a + step change that forces the target to move mid-run. Fixed seeds, recorded + with the results. +- One profile runs a fixed delay preset rather than auto, as the control. Every + other profile exercises adaptation, so without this the whole matrix can pass + while a fixed preset regresses, and fixed presets are what a viewer lands on + today. +- Budgets in one checked-in file, keyed by the whole matrix row and not by + profile alone: runtime, codec, sample rate, profile, and the ring path, since + each of those moves the expected floor. A profile-only key silently grades + one row against another's threshold. Tightening a budget is then a visible + diff and loosening one needs a reason in review. +- Trim the released ndjson traces from `rt-audio-traces-2026-09-06` into a + fixture and replay them too, so a real recorded arrival pattern is graded + next to the synthetic profiles. +- Add the lane to `nightly.yml`, and extend its header comment with why this + one is not a PR gate. + +## Required + +- [Impaired path](/quest/m0/transport-impairment-profile.md) - the seeded shaper this puts in front of the relay diff --git a/quest/m2/audio-quality-harness/native.md b/quest/m2/audio-quality-harness/native.md new file mode 100644 index 00000000000..8f16e0a8284 --- /dev/null +++ b/quest/m2/audio-quality-harness/native.md @@ -0,0 +1,46 @@ +# [M] The native audio lane runs the same profiles as the browser + +## Goal + +The same jitter profiles, the same budget file, and the same metric schema run +against `moq play` on a dummy audio device, so a divergence between the native +and browser playout paths shows up as a failing row rather than as a bug report +from whichever platform a user happened to be on. + +## Plan + +- Drive the shipped binary, not a purpose-built harness: `moq play` against a + local relay behind the same seeded shaper, on a cpal null or dummy backend so + CI needs no audio hardware. The speaker is only half of it: `play::run` is the + winit event loop (`rs/moq-cli/src/play/mod.rs` re-exports `window::run`), and + `resumed` builds a `Display` with a real window and a presentable wgpu surface + even for an audio-only broadcast, so the lane dies on a headless runner before + it measures anything. Settle which way out: a virtual display and software + rendering in CI, or an audio-only mode in `moq play` that never opens a + window. The second is worth having on its own, and it is the smaller + dependency to keep working. A real device callback is part of what is being + measured, so keep the real backend rather than substituting a fake clock, and + accept that the timing noise it adds sets the floor for the native budgets. + That floor is worth measuring on its own before the budgets are written. +- `moq play` needs to emit the run's counters and stage timings as JSON for the + analyzer. Add that output, and keep it useful outside the test: a user + debugging their own latency wants the same dump. +- Reuse the browser lane's analyzer and budget file, with the JSON field names + and meanings matching its schema exactly. Native rows get their own budget + values in the same file, keyed the same way, since the device floor differs; + they do not get their own schema. +- Add the lane to the nightly matrix beside the browser one. + +Nothing here is a second estimator. Compare the two runtimes at the estimator +first: the target series each produces from the same arrival trace, which is +the conformance corpus's own comparison and carries no device timing in it. A +disagreement there is a finding against [Audio jitter +target](/quest/m0/audio-jitter-target/README.md) and never a reason to widen a +budget. Only then compare end-to-end totals, with the backend-dependent stages +isolated: this lane deliberately accepts real device callback noise, so a +difference in totals alone proves nothing about the estimator. + +## Required + +- [Browser](/quest/m2/audio-quality-harness/browser.md) - defines the metric schema, the budget file, and the extracted shaper +- [Native jitter target](/quest/m0/audio-jitter-target/native.md) - the estimator this lane grades and compares against the browser; without it there is no target series and the budgets would be set against a playout path that holds nothing diff --git a/quest/m2/latency-ledger.md b/quest/m2/latency-ledger.md new file mode 100644 index 00000000000..7c9f8e246f3 --- /dev/null +++ b/quest/m2/latency-ledger.md @@ -0,0 +1,37 @@ +# [M] Every source of playback latency is reportable, not just measurable in a test + +## Goal + +A running session can report where its end-to-end audio delay went, stage by +stage, through the public API in both languages. The numbers a user reads when +debugging their own latency are the numbers the harness grades, because they +come from the same place. + +## Plan + +The audio quality harness lands on ad-hoc debug probes, which is the right +trade to get it running. This quest promotes them. + +- Take the stage schema the harness already defines (capture, encode, publish + flush, network, jitter buffer, decode, render) and expose it the way + `moq-stats` exposes relay traffic: an observable readout, not a callback. +- Both languages, matching names, per the repo's cross-language rule. Scrutinise + each exported item: a stage nobody outside can act on stays internal. +- Switch the harness over, deleting the probes it replaces. A ledger with no + consumer is how this drifts from reality. +- The stages must sum to the measured end-to-end delay within a stated + tolerance, and that identity is itself a test. It only holds on the harness's + terms: one duration unit, every timestamp on a named clock, and stages as + exclusive spans that cannot both claim the same milliseconds. Inherit those + from the schema rather than restating them. An unaccounted remainder is the + bug this whole line exists to find, so it gets a name and a number rather + than being absorbed into a neighbouring stage. + +## Required + +- [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - defines the stage schema and lands the probes this promotes + +## Related + +- [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - defines the stages and is the first consumer +- [QoS](/quest/m2/qos/README.md) - relay-side health, the same idea from the other end diff --git a/quest/m2/watch-audio-time-stretch.md b/quest/m2/watch-audio-time-stretch.md index 93b140d273c..19897aaa3e4 100644 --- a/quest/m2/watch-audio-time-stretch.md +++ b/quest/m2/watch-audio-time-stretch.md @@ -31,4 +31,4 @@ Branch from dev. - [Plan: A/V clock](/quest/m1/plan-av-clock.md) - stretching against a free-running ring only moves the drift -- [Auto latency](/quest/m0/3477-watch-auto-latency.md) - the recorded traces this replays +- [Watch](/quest/m0/audio-jitter-target/watch.md) - the recorded traces this replays, and the JS target it converges toward