From 020533b4f1c55d2846046d8c97a1e15496eb32f2 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 00:45:47 +0000 Subject: [PATCH 1/7] docs(quest): plan the audio jitter target and its quality harness Two independent lines for the real-time audio latency problem. `m0/audio-jitter-target/` replaces the round-trip formula with a measured estimate of arrival timing in both languages: a spec quest that surveys the prior art and writes the algorithm down once, then one implementation quest per language against it. It subsumes `m0/3477-watch-auto-latency.md`, which is deleted, and carries its issues. `m2/audio-quality-harness/` grades playout against a checked-in budget over seeded jitter profiles, browser lane first and native second, nightly. `m2/latency-ledger.md` records the deferred promotion of its debug probes into a public per-stage API. The survey behind the spec quest found a likely cause for PR #3517 testing broken: it decays its histogram with NetEq's reorder forget factor rather than the underrun one, and adds an observation per arrival instead of one per resampling interval. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XxFS48hsXmRqoxYWBV7StN --- quest/m0/3477-watch-auto-latency.md | 54 ---------- quest/m0/3479-mux-jitter-flush-span.md | 2 +- quest/m0/README.md | 2 +- quest/m0/audio-jitter-target/README.md | 57 +++++++++++ quest/m0/audio-jitter-target/native.md | 48 +++++++++ quest/m0/audio-jitter-target/spec.md | 116 ++++++++++++++++++++++ quest/m0/audio-jitter-target/watch.md | 50 ++++++++++ quest/m1/plan-av-clock.md | 2 +- quest/m2/README.md | 2 + quest/m2/audio-quality-harness/README.md | 55 ++++++++++ quest/m2/audio-quality-harness/browser.md | 53 ++++++++++ quest/m2/audio-quality-harness/native.md | 32 ++++++ quest/m2/latency-ledger.md | 30 ++++++ quest/m2/watch-audio-time-stretch.md | 2 +- 14 files changed, 447 insertions(+), 58 deletions(-) delete mode 100644 quest/m0/3477-watch-auto-latency.md create mode 100644 quest/m0/audio-jitter-target/README.md create mode 100644 quest/m0/audio-jitter-target/native.md create mode 100644 quest/m0/audio-jitter-target/spec.md create mode 100644 quest/m0/audio-jitter-target/watch.md create mode 100644 quest/m2/audio-quality-harness/README.md create mode 100644 quest/m2/audio-quality-harness/browser.md create mode 100644 quest/m2/audio-quality-harness/native.md create mode 100644 quest/m2/latency-ledger.md diff --git a/quest/m0/3477-watch-auto-latency.md b/quest/m0/3477-watch-auto-latency.md deleted file mode 100644 index 5580ded276b..00000000000 --- a/quest/m0/3477-watch-auto-latency.md +++ /dev/null @@ -1,54 +0,0 @@ -# [S] js/watch: the measured auto latency is proven clean in a browser against a real relay - -## Goal - -The "Real-time" / auto preset plays clean audio on a LAN and against the -public relay: zero underruns after convergence and no skip-aheads in steady -state, proven by the browser harness and by a manual run on Chrome and -Safari, not only by the unit replay. The publisher-side audio encoder lag the -same report observed is measured in the same manual run and either confirmed -fixed or turned into a root-cause quest. - -Boundaries: the estimator and the ring re-buffer themselves are PR #3517 on -`dev`; this quest only verifies them where the PR could not. Convergence -still uses skip-ahead and silence; time-stretching is -[Time stretch](/quest/m2/watch-audio-time-stretch.md). - -## Plan - -Branch from dev. #3517 lands the measured arrival `spread` per track, the -`max(advertised, measured)` target, one chunk of ring slack, re-stall on -underrun, and an underrun counter in the stats panel, verified with synthetic -replay traces. It leaves three things undone, which are this quest: - -- Browser assertion: the `test/smoke` JS driver plays `bbb` from a local - `moq import ts` with ffmpeg's default PES packing at auto and asserts the - underrun counter stays at zero after convergence, on the isolated and the - postMessage ring paths. -- Recorded traces: the arrival traces from #3477 live on the reporter's fork. - Attach a trimmed copy under `js/watch` and replay them through both rings - in `replay.test.ts` in place of the synthetic traces of the same shape. -- Manual run against the public relay on Chrome and Safari, the two rows the - issue measured, with a real microphone and 40 ms or more of added RTT. - Measure the audio encoder's input-to-output lag in the same run using the - reporter's instrumented harness (fork branch `debug/rt-audio`); #3518 fixed - the known cause (the encoder was rebuilt on every subscriber churn), so the - 7.35 s lag and the 88 to 275 ms/s drift the issue reported stand - unconfirmed. If drift survives, the suspects are `writeFrame` opening a - group per audio frame under WebTransport stream credit and the main-thread - task queue that delivers encoder output; turn that into its own quest - rather than fixing it here. - -## Required - -- PR #3517 has merged to `dev` - -## Closes - -- [#3477](https://github.com/moq-dev/moq/issues/3477) - close this issue when the quest finishes -- [#2812](https://github.com/moq-dev/moq/issues/2812) - the iOS stutter report this reproduces on desktop; close it with this - -## Related - -- [Plan: A/V clock](/quest/m1/plan-av-clock.md) - the audio-master clock that follows the estimator -- [Time stretch](/quest/m2/watch-audio-time-stretch.md) - convergence without skips or silence, on top of this diff --git a/quest/m0/3479-mux-jitter-flush-span.md b/quest/m0/3479-mux-jitter-flush-span.md index 934b32f05c9..0f6cb0c5d90 100644 --- a/quest/m0/3479-mux-jitter-flush-span.md +++ b/quest/m0/3479-mux-jitter-flush-span.md @@ -61,5 +61,5 @@ The work: ## Related -- [Auto latency](/quest/m0/3477-watch-auto-latency.md) - reads this field as its codec floor +- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - reads this field as its codec floor - [#3208](/quest/m1/3208-make-2-5-ms-opus-frame-durations-work-across-bindings.md) - the same ceiling rule for Opus frame durations diff --git a/quest/m0/README.md b/quest/m0/README.md index dd38c78d83c..ffbcdec6d73 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -15,7 +15,7 @@ regression test per Root Cause First. ## Quests -- [Auto latency](/quest/m0/3477-watch-auto-latency.md) - js/watch: the measured auto latency (#3517 on dev) is proven clean in a browser and against the public relay +- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the audio playout target is a measured, ported estimate of arrival timing in both languages, not a round-trip guess - [Jitter estimator](/quest/m0/3479-mux-jitter-flush-span.md) - moq-mux: catalog jitter is the publisher's maximum flush span, never a running minimum - [WebKit gate](/quest/m0/webkit-webtransport-gate.md) - js/net: every WebKit engine takes the WebSocket path, not just the Safari brand, so iOS Chrome and Firefox stop freezing after two minutes - [IETF error codes](/quest/m0/ietf-error-codes.md) - every code on a moq-transport wire is a registered value for the negotiated draft, requests and stream resets alike diff --git a/quest/m0/audio-jitter-target/README.md b/quest/m0/audio-jitter-target/README.md new file mode 100644 index 00000000000..86c0d85d328 --- /dev/null +++ b/quest/m0/audio-jitter-target/README.md @@ -0,0 +1,57 @@ +# Audio jitter target + +## Goal + +The audio playout target is a measured estimate of arrival timing, ported from +a known-good implementation, in the browser and natively alike. The round-trip +formula is gone. `max(20ms, 1.25 x minRtt)` sized the buffer for a single +retransmit, which describes how long the network takes to recover a loss and +not how unevenly a publisher emits frames, so a sender flushing 100 ms of media +at once got a target far too shallow to play through the next flush. One +written algorithm, cited by both implementations, produces the same target from +the same arrival trace. + +Boundaries: convergence still uses skip-ahead and silence, so playing slightly +faster or slower to converge stays [Time +stretch](/quest/m2/watch-audio-time-stretch.md). No packet loss concealment. +Video keeps its own target; making the audio playhead the clock is [Plan: A/V +clock](/quest/m1/plan-av-clock.md). + +## Plan + +Three quests: the survey and the written algorithm first, then one +implementation per language against it. The two implementations are +independent once the spec lands and may run in parallel. + +PR #3517 stays open and is fixed in place by the watch quest rather than +restarted. It already deleted the RTT term and plumbed a per-track arrival +`spread` through `Container.Consumer`, measured at container frame arrival and +before the age budget can skip a group, which is the right observation point. +What it got wrong is the estimator on top, which tested broken in a real +browser. The likely reason is in [Spec](/quest/m0/audio-jitter-target/spec.md): +it decays its histogram with NetEq's *reorder* forget factor rather than the +underrun one, and adds an observation per arrival instead of one per resampling +interval, so its target barely moves. + +Native has no jitter buffer at all. `rs/moq-audio`'s decode `Config` carries +only `latency_max`, an upper bound before skipping a stalled group, and its own +doc comment promises "a companion `latency_min` for jitter-buffer padding will +land in a follow-up". This questline is that follow-up. + +## Quests + +- [Spec](/quest/m0/audio-jitter-target/spec.md) - survey what already exists, then write the algorithm down once +- [Watch](/quest/m0/audio-jitter-target/watch.md) - js/watch and js/hang implement it, fixing #3517 in place +- [Native](/quest/m0/audio-jitter-target/native.md) - rs/moq-audio grows `latency_min` from the same algorithm + +## Closes + +- [#3477](https://github.com/moq-dev/moq/issues/3477) - close this issue when the questline finishes +- [#2812](https://github.com/moq-dev/moq/issues/2812) - the iOS stutter report the same estimator fixes + +## Related + +- [Jitter estimator](/quest/m0/3479-mux-jitter-flush-span.md) - the advertised flush span the measured target is held above +- [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - the automated proof, built on its own schedule +- [Time stretch](/quest/m2/watch-audio-time-stretch.md) - inaudible convergence, on top of this +- [Plan: A/V clock](/quest/m1/plan-av-clock.md) - the clock this target eventually feeds diff --git a/quest/m0/audio-jitter-target/native.md b/quest/m0/audio-jitter-target/native.md new file mode 100644 index 00000000000..b0e65a22bf5 --- /dev/null +++ b/quest/m0/audio-jitter-target/native.md @@ -0,0 +1,48 @@ +# [M] rs/moq-audio: a measured jitter buffer, not just an upper bound + +## Goal + +Native playback holds a jitter buffer sized by the algorithm from +[Spec](/quest/m0/audio-jitter-target/spec.md), and passes the same conformance +vector as the browser on the same trace. `moq play` on a path with uneven +arrivals plays through a flush without underrunning, where today it has nothing +to absorb one. + +Boundaries: no time-stretching and no loss concealment, matching the browser. +Video keeps its own path. + +## Plan + +`rs/moq-audio` has no jitter buffer at all. `decode::Config` carries only +`latency_max`, an upper bound before skipping a stalled group, and its doc +comment already promises the companion: + +> The `_max` suffix is a reminder that we never *add* latency here: the consumer +> skips only when newer data is already this far ahead. A companion +> `latency_min` for jitter-buffer padding will land in a follow-up. + +- Measure arrivals at the same point the browser does, on the container + consumer before the age budget can skip a group, so both languages estimate + from the same observation. +- Add `latency_min` to `decode::Config`: `None` measures and adapts, `Some` + pins a floor the way a fixed delay does in the browser. This is additive on a + `#[non_exhaustive]` struct, so it targets `main`. +- The estimator itself is internal. Only the config field and whatever the + playback path needs to report its current target become public, and each one + is argued for rather than exposed by default. +- `rs/moq-cli` `play` gets the matching knob, and `doc/bin/cli.md` follows. Check + the examples against `--help`. +- The playback sink holds the target the way the browser's rings do: slack above + it, land on it when skipping, re-stall when dry so the next insert refills to + the target rather than playing on an empty cushion. +- Tests: the conformance vector from the spec, plus a decode-path test that + replays the same trimmed arrival trace the browser replays and asserts the + same target series. + +`latency_min` is the natural name only because `latency_max` already exists. If +it reads wrong next to a `Duration` that is a floor rather than a budget, +propose alternatives with a recommendation rather than shipping it. + +## Required + +- [Spec](/quest/m0/audio-jitter-target/spec.md) - the algorithm this implements diff --git a/quest/m0/audio-jitter-target/spec.md b/quest/m0/audio-jitter-target/spec.md new file mode 100644 index 00000000000..dd101991acc --- /dev/null +++ b/quest/m0/audio-jitter-target/spec.md @@ -0,0 +1,116 @@ +# [S] The audio playout target algorithm is written down once + +## Goal + +One document under `doc/concept` describes how a receiver estimates its audio +playout target from arrival timing, in enough detail that two implementations +written from it agree on the same trace. Every constant has a stated origin. A +conformance corpus, arrival traces and the target series each must produce, +lives beside it as checked-in JSON so both languages test against the same +numbers rather than against each other. + +Boundaries: no production code. The implementations are the sibling quests, and +either may send a correction back here. + +## Plan + +The survey is done; it is recorded here so the implementations do not repeat +it. What remains is the writing and the corpus. + +### What to implement against + +WebRTC's NetEq, but not the file the name suggests. `delay_manager.cc` on +current main is a 103-line composition; the algorithm lives in +`underrun_optimizer.{h,cc}`, `reorder_optimizer.{h,cc}`, `histogram.{h,cc}`, +and `packet_arrival_history.{h,cc}` under +`modules/audio_coding/neteq/`. In priority order: + +1. **Relative arrival delay**, the highest-value idea and the cheapest. + `delay(p) = max(0, (arrival(p) - arrival(ref)) - (rtp(p) - rtp(ref)))` where + `ref` is the packet in a 2 s window minimising `arrival - rtp`, that is, the + fastest recent packet, tracked with a monotonic deque. Delay against the + best-case path, not against the previous packet. An inter-arrival estimator + is structurally blind to a slow build-up: a hundred packets each 2 ms late + read as a hundred tiny observations, where this reads as a delay climbing to + 200 ms. Frame timestamps stand in for RTP timestamps and there are no + sequence numbers to unwrap. +2. **An empirical histogram with an exponential forget factor, read at a + quantile.** 20 ms buckets, 100 of them. Seed the cold start with the + decaying `0.5^(i+1)` prior. A quantile of an empirical distribution, not a + mean plus k deviations: network delay is heavy-tailed and one-sided, so a + Gaussian assumption sizes the tail wrong. +3. **Max-over-interval resampling**, 500 ms. Easy to miss and behaviourally + significant: the histogram takes one observation per interval, the maximum + in it. That decorrelates observations and turns the forget factor into a + wall-clock time constant, which is the only form worth documenting. +4. **A cold-start forget ramp**, WebRTC's `start_forget_weight`, so the first + seconds converge instead of crawling. +5. **Explicit delay-versus-loss cost minimisation** for the late tail. WebRTC's + `ReorderOptimizer` and SpeexDSP's `compute_opt_delay` reached the same shape + independently, which is decent evidence it is right. Defer it; ship the + underrun estimator first. + +Current defaults on main, for reference and not for copying blind: quantile +`0.95`, underrun forget factor `0.983`, resample interval `500ms`, start +forget weight `2`, reorder forget factor `0.9993`, `ms_per_loss_percent` `20`. +The widely repeated "0.97" is a transitional value, not head. + +**This probably explains #3517.** It used `FORGET = 0.9993`, which is NetEq's +*reorder* forget factor, on the equivalent of the underrun histogram, where +head uses `0.983`. Applied per arrival with no 500 ms resampling, that is a +memory far longer than intended, so the target barely moves. Confirm or refute +this before writing, because it decides how much of the rest was actually +wrong. + +### Licensing + +WebRTC is BSD-3 plus a PATENTS grant. Copying source means carrying a third +license and its disclaimer inside a `MIT OR Apache-2.0` repository. Implement +from the described algorithm instead, in `f64`, citing WebRTC in a comment as +the design's source. The Q15/Q30 fixed point exists for 2010-era DSPs; floating +point is simpler here and removes a whole class of cross-language parity bug. + +Do not read Asterisk's `main/jitterbuf.c` (GPLv2) or FreeSWITCH's +`switch_jitterbuffer.c` (MPL 1.1) while writing this. SpeexDSP's `jitter.c` is +BSD-3 and safe to read for the cost-minimisation idea. + +### Why not a dependency + +The `neteq` crate (`security-union/videocall-rs`, MIT OR Apache-2.0, actively +released, ships a WASM build) is the only viable candidate and was rejected: +its relative-arrival-delay step takes a percentile over per-packet deltas each +clamped at zero, which cannot see a build-up, the one property the design +exists for; its `ReorderOptimizer` is configured but never implemented; and it +is a whole NetEq, sample-rate-locked ring buffers and WSOLA and PLC, where this +needs a few hundred lines of estimator. Its shape would leak into our API. +Nothing else is close: `webrtc-rs` and `str0m` deliberately leave playout to +the application, GStreamer's `rtpjitterbuffer` solves clock skew rather than +target sizing and is LGPL, and no browser-usable JS port exists. + +### The document + +Write `doc/concept/`, covering the observation point, the estimator, rise and +fall, startup and re-anchoring, and the clamp. Specifically: + +- Measure at container frame arrival and before the age budget can skip a + group. #3517 has this right, and it matters: a target derived from what + survives the budget only ever confirms the budget it was cut to. +- Rise and fall asymmetry is the "too aggressive or not aggressive enough" + dial, so it gets a justification rather than a number. +- The clamp reads the rendition's advertised flush span from [Jitter + estimator](/quest/m0/3479-mux-jitter-flush-span.md). That field is a + publisher-declared floor and the estimate is a measurement of the network: + say plainly which is which, because the shared word "jitter" invites + confusion. #3517 combines them with a max on the grounds they describe the + same quantity; that reasoning does not survive the distinction, so revisit it. +- The document is normative, not either implementation. Parity is held by the + conformance corpus, one JSON fixture set run by both test suites, the way + wire behaviour is held aligned today. A wasm core shared into the browser + audio path is a larger architectural commitment than this warrants. + +Not an IETF draft: nothing here is negotiated on the wire, and a receiver that +chooses differently is still conformant. + +## Related + +- [Jitter estimator](/quest/m0/3479-mux-jitter-flush-span.md) - supplies the advertised flush span the clamp reads diff --git a/quest/m0/audio-jitter-target/watch.md b/quest/m0/audio-jitter-target/watch.md new file mode 100644 index 00000000000..8fdb362ce13 --- /dev/null +++ b/quest/m0/audio-jitter-target/watch.md @@ -0,0 +1,50 @@ +# [M] js/watch: the playout target follows the spec, proven in a real browser + +## Goal + +`js/watch` computes its auto target with the algorithm from +[Spec](/quest/m0/audio-jitter-target/spec.md) and passes the conformance +vector. The "Real-time" preset plays clean audio on a LAN and against the +public relay: zero underruns after convergence and no skip-aheads in steady +state, on both the isolated and the postMessage ring paths, confirmed by a +manual run on Chrome and Safari with a real microphone and 40 ms or more of +added RTT. + +Boundaries: the ring's slack, re-stall and underrun counter are mechanical and +already correct in #3517; this quest changes the estimator above them. + +## Plan + +Fix PR #3517 in place on `dev` rather than reopening it. It is a published +`@moq/watch` break (the `Sync` input `probe` is removed) and `dev` already +carries the `delay`/`buffer` split it is expressed in. Resolve its conflicts +with `dev` first, since it currently does not merge. + +- Replace the 95th-percentile-over-decaying-histogram estimator in + `js/hang/src/container/jitter.ts` with the spec's, and run the conformance + vector in `jitter.test.ts`. Keep the arrival plumbing, the `spread` getter, + and the measurement point: those were right. +- Keep the ring work as it stands: one chunk of slack above the target, landing + back on the target when skipping, re-stalling on empty, and the underrun + counter reaching the stats panel and the buffering indicator. +- Replay the recorded traces from #3477 rather than synthetic ones of the same + shape. They are on the reporter's fork + (`fperex/moq`, branch `debug/rt-audio`) with the raw ndjson attached to + release `rt-audio-traces-2026-09-06`. Trim a copy into the repository and + replay it through both rings in `replay.test.ts`. +- Manual run against the public relay on Chrome and Safari, the two rows the + issue measured. Measure the publisher's audio encoder input-to-output lag in + the same run using the reporter's instrumented harness; #3518 fixed the known + cause, so the 7.35 s lag and the 88 to 275 ms/s drift the issue reported + stand unconfirmed. If drift survives, the suspects are `writeFrame` opening a + group per audio frame under WebTransport stream credit and the main-thread + task queue delivering encoder output. Turn that into its own quest rather + than fixing it here. + +Two sibling quests on `main` touch the same audio path +(`3479-watch-audio-identity`, `publish-audio-encoder-lag`), for conflict triage +when `main` next merges into `dev`. + +## Required + +- [Spec](/quest/m0/audio-jitter-target/spec.md) - the algorithm this implements diff --git a/quest/m1/plan-av-clock.md b/quest/m1/plan-av-clock.md index 2e7750a2cb2..ecdb43edbb1 100644 --- a/quest/m1/plan-av-clock.md +++ b/quest/m1/plan-av-clock.md @@ -40,5 +40,5 @@ implementation quest, and re-estimate it. ## Related -- [Auto latency](/quest/m0/3477-watch-auto-latency.md) - the estimator this sits on +- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the estimator this sits on - [Time stretch](/quest/m2/watch-audio-time-stretch.md) - stretching needs a clock to converge toward diff --git a/quest/m2/README.md b/quest/m2/README.md index 93b59a10138..a2b6e98ef1b 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -32,6 +32,8 @@ can act on. Each still carries its own plan and regression test. - [Bandwidth estimate release](/quest/m2/web-transport-bandwidth-estimate.md) - web-transport-quinn reports quinn's BBR bandwidth estimate and ships a release carrying it - [#2847](/quest/m2/2847-the-quinn-backends-send-bandwidth-estimate-is-cwnd-rtt.md) - quinn backend: bump to the releases that report the controller bandwidth estimate instead of cwnd/rtt - [Safari WebTransport](/quest/m2/safari-webtransport.md) - WebKit browsers return to WebTransport once WebKit 319818 ships fixed +- [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - a playout latency regression fails a run instead of arriving as a bug report +- [Latency ledger](/quest/m2/latency-ledger.md) - a session reports where its end-to-end audio delay went, stage by stage - [Benchmark comparisons](/quest/m2/performance-comparisons.md) - retained evidence, repeated paired runs, and uncertainty for performance claims - [Relay profiling](/quest/m2/performance-profiles.md) - reproducible CPU and allocation captures under the existing workloads - [Browser benchmarks](/quest/m2/browser-benchmarks.md) - measure JS transport, container, decode, and render costs in an identified browser diff --git a/quest/m2/audio-quality-harness/README.md b/quest/m2/audio-quality-harness/README.md new file mode 100644 index 00000000000..0ab6831ca65 --- /dev/null +++ b/quest/m2/audio-quality-harness/README.md @@ -0,0 +1,55 @@ +# Audio quality harness + +## Goal + +A regression in audio playout latency fails a run instead of arriving as a bug +report. The harness plays a broadcast over an impaired path, counts what the +listener would actually have heard (underruns, short quanta, discarded samples, +skip-aheads) and what each stage of the pipeline contributed to the delay, and +grades the result against a checked-in budget. It runs in the browser and +natively, on the same jitter profiles, reporting the same numbers. + +Boundaries: audio only. The stage breakdown is defined generically so video can +adopt it later, but no video assertion ships here. No perceptual scoring: the +grade is glitches and latency, not an opinion about how it sounds. + +## Plan + +Two quests, browser first, because that is where the traces and the reported +bug both are. The native lane follows against the same budgets and the same +metric schema, so the two implementations can be compared rather than merely +both passing. + +The starting point is not a blank page. The reporter on #3477 already built a +working browser harness on their fork (`fperex/moq`, branch `debug/rt-audio`): +a CDP driver, a beacon sink, a trace analyzer, a ring replay, a five-scenario +`bench.sh`, and a `compare.mjs` that prints before-and-after tables, with 130 +raw ndjson traces attached to release `rt-audio-traces-2026-09-06`. Upstream +that rather than reinventing it. + +Jitter comes from the seeded userspace UDP shaper in [Impaired +path](/quest/m0/transport-impairment-profile.md), not from a fake arrival clock, +so the transport's own contribution is measured rather than assumed. That quest +builds the shaper inside the relay drills' support code; it has to come out into +something a JS harness process can also put in front of a relay. Extracting it +is part of the browser quest. + +Loss, reorder and rate limiting stay switched off in the profiles here. The +buffer's job is absorbing arrival spread, and mixing congestion response into an +audio quality number makes a failure hard to attribute. + +Nightly, not per-PR: the matrix is jitter profiles by runtime by codec and +sample rate, which is more than a merge gate should carry, and `nightly.yml` +already exists for exactly this trade. + +## Quests + +- [Browser](/quest/m2/audio-quality-harness/browser.md) - upstream the fork's harness, grade it against a budget, run it nightly +- [Native](/quest/m2/audio-quality-harness/native.md) - the same profiles and budgets through `moq play` on a dummy device + +## Related + +- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the estimator this exists to keep honest +- [Latency ledger](/quest/m2/latency-ledger.md) - promotes this harness's probes into a public API +- [Failure artifacts](/quest/m0/qa-failure-artifacts.md) - keeps the run directory and trace of a failing run +- [Time stretch](/quest/m2/watch-audio-time-stretch.md) - graded by this harness once it lands diff --git a/quest/m2/audio-quality-harness/browser.md b/quest/m2/audio-quality-harness/browser.md new file mode 100644 index 00000000000..90f953a386a --- /dev/null +++ b/quest/m2/audio-quality-harness/browser.md @@ -0,0 +1,53 @@ +# [L] The browser audio lane is graded against a budget, nightly + +## Goal + +`just test audio-quality` plays a broadcast in a headless browser over a seeded, +impaired path and prints a graded table: underruns, short quanta, discarded +samples, skip-aheads, and where the end-to-end delay went. It fails when a +profile exceeds its budget. A nightly job runs the full matrix and keeps the +run directory of a failure. + +The same run works on both ring paths, isolated and postMessage, since the +production path is the one without cross-origin isolation. + +## Plan + +Upstream the reporter's harness from `fperex/moq` branch `debug/rt-audio` +rather than rebuilding it, keeping the attribution. It already has the CDP +driver, the beacon sink, the analyzer, the ring replay, `bench.sh`'s five +scenarios, and `compare.mjs`. What it does not have is a home in `test/`, a +budget, or a schedule. + +- Land the driver and analyzer under `test/`, alongside the existing `smoke` + and `drill` lanes, wired into the `justfile` the way they are. Playwright is + already in the tree for the harness quests, so prefer it over a bespoke CDP + driver if the switch is cheap; if it is not, say so and keep CDP. +- Keep the instrumentation ad-hoc for now. The probes stay a debug surface, not + public API; promoting them is [Latency + ledger](/quest/m2/latency-ledger.md), which nothing here waits on. +- The metric schema is the deliverable that outlives this quest, because the + native lane has to emit the same thing. Name each stage once (capture, + encode, publish flush, network, jitter buffer, decode, render) and define the + glitch counters exactly: an underrun is a quantum the ring could only partly + fill, a skip-ahead is a re-anchor that discarded buffered audio. +- Extract the seeded shaper from [Impaired + path](/quest/m0/transport-impairment-profile.md) into something that runs as + its own process in front of a relay, so this harness and the drills share one + impairment implementation. Assert the shaper actually treated traffic: a + profile that silently did nothing turns an impaired run into an unimpaired + pass. +- Profiles: near-zero, mild, bursty (the flush-span shape from #3477), and a + step change that forces the target to move mid-run. Fixed seeds, recorded + with the results. +- Budgets in one checked-in file keyed by profile, so tightening one is a + visible diff and loosening one needs a reason in review. +- Trim the released ndjson traces from `rt-audio-traces-2026-09-06` into a + fixture and replay them too, so a real recorded arrival pattern is graded + next to the synthetic profiles. +- Add the lane to `nightly.yml`, and extend its header comment with why this + one is not a PR gate. + +## Required + +- [Impaired path](/quest/m0/transport-impairment-profile.md) - the seeded shaper this puts in front of the relay diff --git a/quest/m2/audio-quality-harness/native.md b/quest/m2/audio-quality-harness/native.md new file mode 100644 index 00000000000..8124e8e1558 --- /dev/null +++ b/quest/m2/audio-quality-harness/native.md @@ -0,0 +1,32 @@ +# [M] The native audio lane runs the same profiles as the browser + +## Goal + +The same jitter profiles, the same budget file, and the same metric schema run +against `moq play` on a dummy audio device, so a divergence between the native +and browser playout paths shows up as a failing row rather than as a bug report +from whichever platform a user happened to be on. + +## Plan + +- Drive the shipped binary, not a purpose-built harness: `moq play` against a + local relay behind the same seeded shaper, on a cpal null or dummy backend so + CI needs no audio hardware. A real device callback is part of what is being + measured, so keep the real backend rather than substituting a fake clock, and + accept that the timing noise it adds sets the floor for the native budgets. + That floor is worth measuring on its own before the budgets are written. +- `moq play` needs to emit the run's counters and stage timings as JSON for the + analyzer. Add that output, and keep it useful outside the test: a user + debugging their own latency wants the same dump. +- Reuse the browser lane's analyzer and budget file. Native rows get their own + budgets in the same file, since the floor differs, but not their own schema. +- Add the lane to the nightly matrix beside the browser one. + +Nothing here is a second estimator. If the native and browser numbers disagree +beyond the profiles' tolerance, that is a finding against +[Audio jitter target](/quest/m0/audio-jitter-target/README.md), not a reason to +widen a budget. + +## Required + +- [Browser](/quest/m2/audio-quality-harness/browser.md) - defines the metric schema, the budget file, and the extracted shaper diff --git a/quest/m2/latency-ledger.md b/quest/m2/latency-ledger.md new file mode 100644 index 00000000000..6448afa2a1c --- /dev/null +++ b/quest/m2/latency-ledger.md @@ -0,0 +1,30 @@ +# [M] Every source of playback latency is reportable, not just measurable in a test + +## Goal + +A running session can report where its end-to-end audio delay went, stage by +stage, through the public API in both languages. The numbers a user reads when +debugging their own latency are the numbers the harness grades, because they +come from the same place. + +## Plan + +The audio quality harness lands on ad-hoc debug probes, which is the right +trade to get it running. This quest promotes them. + +- Take the stage schema the harness already defines (capture, encode, publish + flush, network, jitter buffer, decode, render) and expose it the way + `moq-stats` exposes relay traffic: an observable readout, not a callback. +- Both languages, matching names, per the repo's cross-language rule. Scrutinise + each exported item: a stage nobody outside can act on stays internal. +- Switch the harness over, deleting the probes it replaces. A ledger with no + consumer is how this drifts from reality. +- The stages must sum to the measured end-to-end delay within a stated + tolerance, and that identity is itself a test. An unaccounted remainder is the + bug this whole line exists to find, so it gets a name and a number rather + than being absorbed into a neighbouring stage. + +## Related + +- [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - defines the stages and is the first consumer +- [QoS](/quest/m2/qos/README.md) - relay-side health, the same idea from the other end diff --git a/quest/m2/watch-audio-time-stretch.md b/quest/m2/watch-audio-time-stretch.md index 93b140d273c..4297b1d4a90 100644 --- a/quest/m2/watch-audio-time-stretch.md +++ b/quest/m2/watch-audio-time-stretch.md @@ -31,4 +31,4 @@ Branch from dev. - [Plan: A/V clock](/quest/m1/plan-av-clock.md) - stretching against a free-running ring only moves the drift -- [Auto latency](/quest/m0/3477-watch-auto-latency.md) - the recorded traces this replays +- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the recorded traces this replays, and the target it converges toward From 88e3562d37926838b4a3626d57eb9de644139025 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 00:52:59 +0000 Subject: [PATCH 2/7] docs(quest): record the reproduced cause of the 14.56s auto target Testing #3517 in a browser reads a 14.56 s jitter buffer in auto. The estimator cannot produce that from its histogram, which clamps at one second; the unbounded term is the "plus one frame" added on top. That frame duration is learned as the running minimum of positive gaps between observed media timestamps, so the first gap sets it. A tune-in across a stale group sets it to seconds. The target then rises instantly and unconditionally and falls by one corrected step per second, so it takes about fifteen minutes of clean audio to unwind. Replaces the earlier forget-factor hypothesis, which is a real but much smaller divergence and not what produces the number. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XxFS48hsXmRqoxYWBV7StN --- quest/m0/audio-jitter-target/README.md | 11 +++--- quest/m0/audio-jitter-target/spec.md | 48 ++++++++++++++++++++++---- quest/m0/audio-jitter-target/watch.md | 19 +++++++--- 3 files changed, 63 insertions(+), 15 deletions(-) diff --git a/quest/m0/audio-jitter-target/README.md b/quest/m0/audio-jitter-target/README.md index 86c0d85d328..a40eb2729d7 100644 --- a/quest/m0/audio-jitter-target/README.md +++ b/quest/m0/audio-jitter-target/README.md @@ -27,11 +27,12 @@ PR #3517 stays open and is fixed in place by the watch quest rather than restarted. It already deleted the RTT term and plumbed a per-track arrival `spread` through `Container.Consumer`, measured at container frame arrival and before the age budget can skip a group, which is the right observation point. -What it got wrong is the estimator on top, which tested broken in a real -browser. The likely reason is in [Spec](/quest/m0/audio-jitter-target/spec.md): -it decays its histogram with NetEq's *reorder* forget factor rather than the -underrun one, and adds an observation per arrival instead of one per resampling -interval, so its target barely moves. +What it got wrong is the estimator on top. A viewer on that branch reads a +14.56 s jitter buffer in auto, and the cause is reproduced in +[Spec](/quest/m0/audio-jitter-target/spec.md): the "plus one frame" term is +learned from the first gap between observed timestamps, so a tune-in across a +stale group sets it to seconds, and nothing clamps the target or bounds how +fast it rises. Native has no jitter buffer at all. `rs/moq-audio`'s decode `Config` carries only `latency_max`, an upper bound before skipping a stalled group, and its own diff --git a/quest/m0/audio-jitter-target/spec.md b/quest/m0/audio-jitter-target/spec.md index dd101991acc..d882f8407f4 100644 --- a/quest/m0/audio-jitter-target/spec.md +++ b/quest/m0/audio-jitter-target/spec.md @@ -55,12 +55,48 @@ Current defaults on main, for reference and not for copying blind: quantile forget weight `2`, reorder forget factor `0.9993`, `ms_per_loss_percent` `20`. The widely repeated "0.97" is a transitional value, not head. -**This probably explains #3517.** It used `FORGET = 0.9993`, which is NetEq's -*reorder* forget factor, on the equivalent of the underrun histogram, where -head uses `0.983`. Applied per arrival with no 500 ms resampling, that is a -memory far longer than intended, so the target barely moves. Confirm or refute -this before writing, because it decides how much of the rest was actually -wrong. +**What actually breaks #3517, reproduced.** A viewer on that branch reads a +14.56 s jitter buffer in auto. The estimator cannot produce that from its +histogram: the percentile is clamped to `BUCKETS * BUCKET`, one second. The +unbounded term is the "plus one frame" on top of it. + +`Jitter` learns that frame duration as the running minimum of positive gaps +between consecutive observed media timestamps, so the *first* gap sets it with +nothing to validate against. A tune-in that observes a frame from a stale group +and then the live edge sets it to the distance between them, seconds rather +than milliseconds. `#publish` then raises the target instantly and +unconditionally to `percentile() + step`, and lowers it by one *corrected* +step per second afterwards. Inflation is immediate; deflation is 20 ms/s. +`reanchor()` clears `#latest` but not the spacing, so the bad value outlives +the discontinuity that caused it. + +Replaying two frames 14.5 s apart in media time, then twenty minutes of +perfectly paced audio with zero real jitter: + +``` +after 2 frames: spacing=14500ms target=14.50s + + 1s live, zero jitter: target=14.48s + + 60s live, zero jitter: target=13.30s + + 300s live, zero jitter: target=8.50s + + 600s live, zero jitter: target=2.50s + + 900s live, zero jitter: target=0.02s +``` + +Two lessons for the document, both structural rather than a constant to retune: + +- **Frame duration is a codec property, not an observation.** Take it from the + rendition config. NetEq's packet-time units come from the codec for the same + reason. Deriving it from arbitrary timestamp arithmetic makes a tune-in + artifact indistinguishable from a real frame duration. +- **The target needs a clamp and a bounded rise.** WebRTC clamps in + `DelayConstraints` (`delay_constraints.{h,cc}`), applied by `DecisionLogic` + rather than by the delay manager, with a hard ceiling at 75% of buffer + capacity. #3517 ported neither the clamp nor a rise limit, so any single bad + observation is permanent on the timescale a viewer will wait. + +The forget factor is a separate and smaller divergence: `0.9993` is NetEq's +*reorder* factor where the underrun path uses `0.983`, applied per arrival with +no 500 ms resampling. Worth fixing, but it is not what produces 14.56 s. ### Licensing diff --git a/quest/m0/audio-jitter-target/watch.md b/quest/m0/audio-jitter-target/watch.md index 8fdb362ce13..c8eb74a0c2e 100644 --- a/quest/m0/audio-jitter-target/watch.md +++ b/quest/m0/audio-jitter-target/watch.md @@ -20,10 +20,21 @@ Fix PR #3517 in place on `dev` rather than reopening it. It is a published carries the `delay`/`buffer` split it is expressed in. Resolve its conflicts with `dev` first, since it currently does not merge. -- Replace the 95th-percentile-over-decaying-histogram estimator in - `js/hang/src/container/jitter.ts` with the spec's, and run the conformance - vector in `jitter.test.ts`. Keep the arrival plumbing, the `spread` getter, - and the measurement point: those were right. +- Replace the estimator in `js/hang/src/container/jitter.ts` with the spec's, + and run the conformance corpus in `jitter.test.ts`. Keep the arrival + plumbing, the `spread` getter, and the measurement point: those were right. +- The reproduced 14.56 s runaway is the regression test to write first, before + any replacement: two frames whose media timestamps are far apart, then paced + audio with zero real jitter, asserting the target stays near the frame + duration. On today's branch it reads 14.5 s and needs about fifteen minutes + of clean audio to unwind. Take the frame duration from the rendition config + rather than from observed timestamps, clamp the target, and bound how fast it + rises; [Spec](/quest/m0/audio-jitter-target/spec.md) has the detail. +- Check what the viewer's saved preset does on load. The element defaults + `delay` to `"auto"` and nothing in `demo/` overrides it, yet a fresh session + came up on the 100 ms chip, so something is restoring or overriding it. Pin + that down: a stored preference silently winning over the default is its own + bug, and it also means auto gets far less real exposure than it looks like. - Keep the ring work as it stands: one chunk of slack above the target, landing back on the target when skipping, re-stalling on empty, and the underrun counter reaching the stats panel and the buffering indicator. From b804793b9a30ea7dd98d70c26e929eb038577a49 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 01:07:34 +0000 Subject: [PATCH 3/7] docs(quest): fix the dependency edges and the frame duration source Three findings from review, all correct: The ledger promotes the harness's stage schema and probes and switches that harness over, so it cannot start before the harness exists. It now requires it; without the edge `quest ready` handed out a quest with nothing to promote. The frame duration is not in the catalog. `AudioConfig` carries `sampleRate` and `bitrate`, and its own comment concedes the duration depends on the codec and rate; Opus picks from 2.5 to 60 ms independently of both. The spec quest now names that as the open question and lists the candidates (the Opus TOC byte, the decoded sample count, or a new catalog field with its cross-package sync) rather than pointing at a config field that does not exist. Time stretching is js/watch only, so requiring the whole jitter-target questline blocked it on the unrelated Rust port. It requires the watch quest instead. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XxFS48hsXmRqoxYWBV7StN --- quest/m0/audio-jitter-target/spec.md | 20 ++++++++++++++++---- quest/m0/audio-jitter-target/watch.md | 7 ++++--- quest/m2/latency-ledger.md | 4 ++++ quest/m2/watch-audio-time-stretch.md | 2 +- 4 files changed, 25 insertions(+), 8 deletions(-) diff --git a/quest/m0/audio-jitter-target/spec.md b/quest/m0/audio-jitter-target/spec.md index d882f8407f4..ab474a8db16 100644 --- a/quest/m0/audio-jitter-target/spec.md +++ b/quest/m0/audio-jitter-target/spec.md @@ -84,10 +84,22 @@ after 2 frames: spacing=14500ms target=14.50s Two lessons for the document, both structural rather than a constant to retune: -- **Frame duration is a codec property, not an observation.** Take it from the - rendition config. NetEq's packet-time units come from the codec for the same - reason. Deriving it from arbitrary timestamp arithmetic makes a tune-in - artifact indistinguishable from a real frame duration. +- **Frame duration is a codec property, not an observation.** NetEq's + packet-time units come from the codec for the same reason. Deriving it from + arbitrary timestamp arithmetic makes a tune-in artifact indistinguishable + from a real frame duration. + + Where it comes from is an open question this quest answers, because the + catalog does not carry it: `AudioConfig` has `sampleRate` and `bitrate`, and + its own comment concedes the frame duration "depends on the codec, sample + rate, etc." Opus makes this sharp, since the encoder picks 2.5, 5, 10, 20, + 40, or 60 ms independently of both. Candidates, in order of how little they + cost: the Opus TOC byte, which self-describes the frame duration in band and + needs no schema change; the decoded frame's own sample count, exact but only + known below the container layer where the estimator currently sits; or a new + catalog field, which is the expensive option and would need the full + cross-package sync (`rs/hang`, `js/hang`, `doc/concept`, and the hang draft). + Pick one and say why; do not leave it to each implementation. - **The target needs a clamp and a bounded rise.** WebRTC clamps in `DelayConstraints` (`delay_constraints.{h,cc}`), applied by `DecisionLogic` rather than by the delay manager, with a hard ceiling at 75% of buffer diff --git a/quest/m0/audio-jitter-target/watch.md b/quest/m0/audio-jitter-target/watch.md index c8eb74a0c2e..77441ca232f 100644 --- a/quest/m0/audio-jitter-target/watch.md +++ b/quest/m0/audio-jitter-target/watch.md @@ -27,9 +27,10 @@ with `dev` first, since it currently does not merge. any replacement: two frames whose media timestamps are far apart, then paced audio with zero real jitter, asserting the target stays near the frame duration. On today's branch it reads 14.5 s and needs about fifteen minutes - of clean audio to unwind. Take the frame duration from the rendition config - rather than from observed timestamps, clamp the target, and bound how fast it - rises; [Spec](/quest/m0/audio-jitter-target/spec.md) has the detail. + of clean audio to unwind. Take the frame duration from the source the spec + settles on rather than from observed timestamps, clamp the target, and bound + how fast it rises; [Spec](/quest/m0/audio-jitter-target/spec.md) has the + detail and the candidates. - Check what the viewer's saved preset does on load. The element defaults `delay` to `"auto"` and nothing in `demo/` overrides it, yet a fresh session came up on the 100 ms chip, so something is restoring or overriding it. Pin diff --git a/quest/m2/latency-ledger.md b/quest/m2/latency-ledger.md index 6448afa2a1c..0bc54e49134 100644 --- a/quest/m2/latency-ledger.md +++ b/quest/m2/latency-ledger.md @@ -24,6 +24,10 @@ trade to get it running. This quest promotes them. bug this whole line exists to find, so it gets a name and a number rather than being absorbed into a neighbouring stage. +## Required + +- [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - defines the stage schema and lands the probes this promotes + ## Related - [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - defines the stages and is the first consumer diff --git a/quest/m2/watch-audio-time-stretch.md b/quest/m2/watch-audio-time-stretch.md index 4297b1d4a90..19897aaa3e4 100644 --- a/quest/m2/watch-audio-time-stretch.md +++ b/quest/m2/watch-audio-time-stretch.md @@ -31,4 +31,4 @@ Branch from dev. - [Plan: A/V clock](/quest/m1/plan-av-clock.md) - stretching against a free-running ring only moves the drift -- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the recorded traces this replays, and the target it converges toward +- [Watch](/quest/m0/audio-jitter-target/watch.md) - the recorded traces this replays, and the JS target it converges toward From aefa2c3b1972987dba6a932f6cca5a51d1971c95 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 01:10:25 +0000 Subject: [PATCH 4/7] docs(quest): pin down the contracts the plans left to the reader Review findings, all making the same point: a plan that leaves a shared contract implicit lets two implementations diverge while both look right. - The estimator spec now defines units, timescale conversion, and what a discontinuity does to the reference, and states that the forget factor decays once per resampled observation rather than once per arrival. That distinction is the mistake #3517 made and it reads plausible either way. - `latency_min` is a floor or an exact target, not both. The native quest names the decision, recommends the floor, and requires stating precedence against the estimator and clamping against `latency_max`. - The questline README no longer asserts the flush-span rule that the spec quest exists to settle. - The harness metric schema is written as a contract: every counter, units, clock domain, per-metric aggregation, and stages as exclusive spans. The ledger's sum-to-end-to-end identity is unimplementable without the last one, so it inherits them. - Budgets are keyed by the whole matrix row, not by profile alone, so one row cannot be graded against another's floor. - A fixed-delay control profile joins the matrix; every other profile exercises adaptation, so a fixed preset could regress unnoticed. - The native lane compares estimator target series before end-to-end totals, since it deliberately accepts device callback noise. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XxFS48hsXmRqoxYWBV7StN --- quest/m0/audio-jitter-target/README.md | 2 +- quest/m0/audio-jitter-target/native.md | 20 ++++++++++++--- quest/m0/audio-jitter-target/spec.md | 12 +++++++++ quest/m2/audio-quality-harness/README.md | 3 ++- quest/m2/audio-quality-harness/browser.md | 30 ++++++++++++++++++----- quest/m2/audio-quality-harness/native.md | 18 +++++++++----- quest/m2/latency-ledger.md | 5 +++- 7 files changed, 72 insertions(+), 18 deletions(-) diff --git a/quest/m0/audio-jitter-target/README.md b/quest/m0/audio-jitter-target/README.md index a40eb2729d7..03c950236d7 100644 --- a/quest/m0/audio-jitter-target/README.md +++ b/quest/m0/audio-jitter-target/README.md @@ -52,7 +52,7 @@ land in a follow-up". This questline is that follow-up. ## Related -- [Jitter estimator](/quest/m0/3479-mux-jitter-flush-span.md) - the advertised flush span the measured target is held above +- [Jitter estimator](/quest/m0/3479-mux-jitter-flush-span.md) - the advertised flush span, whose relationship to the measured target the spec settles - [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - the automated proof, built on its own schedule - [Time stretch](/quest/m2/watch-audio-time-stretch.md) - inaudible convergence, on top of this - [Plan: A/V clock](/quest/m1/plan-av-clock.md) - the clock this target eventually feeds diff --git a/quest/m0/audio-jitter-target/native.md b/quest/m0/audio-jitter-target/native.md index b0e65a22bf5..1c05512a2b3 100644 --- a/quest/m0/audio-jitter-target/native.md +++ b/quest/m0/audio-jitter-target/native.md @@ -24,9 +24,23 @@ comment already promises the companion: - Measure arrivals at the same point the browser does, on the container consumer before the age budget can skip a group, so both languages estimate from the same observation. -- Add `latency_min` to `decode::Config`: `None` measures and adapts, `Some` - pins a floor the way a fixed delay does in the browser. This is additive on a - `#[non_exhaustive]` struct, so it targets `main`. +- Add `latency_min` to `decode::Config`, additive on a `#[non_exhaustive]` + struct, so it targets `main`. Its exact contract is a decision this quest + makes and records, not something to leave to the reader, because "floor" and + "fixed target" behave differently under an adaptive estimator: + + 1. **Floor.** `Some(d)` is a lower bound and the estimator may still raise + the target above it. Recommended: it matches how the browser treats a + rendition's advertised delay, and a viewer asking for more buffer rarely + means "and never more than that". + 2. **Exact target.** `Some(d)` pins the target and disables adaptation, + mirroring a fixed browser preset exactly. Choose this if preserving a + fixed-latency preset byte for byte matters more than the parallel above. + + Either way, state the precedence against the estimator and against + `latency_max`, and what happens when the requested floor exceeds the + effective `latency_max` that `Consumer` already clamps to publisher + retention. Refusing that combination loudly beats silently clamping it. - The estimator itself is internal. Only the config field and whatever the playback path needs to report its current target become public, and each one is argued for rather than exposed by default. diff --git a/quest/m0/audio-jitter-target/spec.md b/quest/m0/audio-jitter-target/spec.md index ab474a8db16..72cb45b81a6 100644 --- a/quest/m0/audio-jitter-target/spec.md +++ b/quest/m0/audio-jitter-target/spec.md @@ -55,6 +55,18 @@ Current defaults on main, for reference and not for copying blind: quantile forget weight `2`, reorder forget factor `0.9993`, `ms_per_loss_percent` `20`. The widely repeated "0.97" is a transitional value, not head. +Say explicitly that the forget factor decays once per *resampled observation*, +not once per arrival, and give the resulting wall-clock memory +(`500ms / (1 - 0.983)`, about 29 s). Applying a per-observation factor per +arrival is precisely the mistake #3517 made, and a number that reads plausible +either way is exactly the kind that survives review. + +Define units once, for both the algorithm and the corpus: what clock arrival +times are on, what timescale frame timestamps use and how they convert, and +what a timestamp discontinuity does to the reference. Two implementations that +disagree on any of these produce different targets from the same trace while +both look correct. + **What actually breaks #3517, reproduced.** A viewer on that branch reads a 14.56 s jitter buffer in auto. The estimator cannot produce that from its histogram: the percentile is clamped to `BUCKETS * BUCKET`, one second. The diff --git a/quest/m2/audio-quality-harness/README.md b/quest/m2/audio-quality-harness/README.md index 0ab6831ca65..bad61704479 100644 --- a/quest/m2/audio-quality-harness/README.md +++ b/quest/m2/audio-quality-harness/README.md @@ -40,7 +40,8 @@ audio quality number makes a failure hard to attribute. Nightly, not per-PR: the matrix is jitter profiles by runtime by codec and sample rate, which is more than a merge gate should carry, and `nightly.yml` -already exists for exactly this trade. +already exists for exactly this trade. Budgets are keyed by the full row, since +each of those dimensions moves the expected floor. ## Quests diff --git a/quest/m2/audio-quality-harness/browser.md b/quest/m2/audio-quality-harness/browser.md index 90f953a386a..4dc4b3469ec 100644 --- a/quest/m2/audio-quality-harness/browser.md +++ b/quest/m2/audio-quality-harness/browser.md @@ -27,10 +27,21 @@ budget, or a schedule. public API; promoting them is [Latency ledger](/quest/m2/latency-ledger.md), which nothing here waits on. - The metric schema is the deliverable that outlives this quest, because the - native lane has to emit the same thing. Name each stage once (capture, - encode, publish flush, network, jitter buffer, decode, render) and define the - glitch counters exactly: an underrun is a quantum the ring could only partly - fill, a skip-ahead is a re-anchor that discarded buffered audio. + native lane and the ledger both have to emit the same thing. It is a + contract, so write it as one: + - Every counter defined, not just the two that are obvious. An underrun is a + quantum the ring could only partly fill; a skip-ahead is a re-anchor that + discarded buffered audio; short quanta and discarded samples need the same + treatment rather than being left to the reader. + - Units and clock domain stated once. Durations in one unit, and every + timestamp on a named clock, so a browser value and a native value are the + same measurement. + - Aggregation stated per metric: a count over the run, a max, or a + percentile. "Underruns: 3" and "underruns: 3/s" grade differently. + - Stages as exclusive, non-overlapping spans (capture, encode, publish flush, + network, jitter buffer, decode, render). The ledger's sum-to-end-to-end + identity is unimplementable if two stages can claim the same milliseconds, + and an unaccounted remainder is the finding, so give it a name too. - Extract the seeded shaper from [Impaired path](/quest/m0/transport-impairment-profile.md) into something that runs as its own process in front of a relay, so this harness and the drills share one @@ -40,8 +51,15 @@ budget, or a schedule. - Profiles: near-zero, mild, bursty (the flush-span shape from #3477), and a step change that forces the target to move mid-run. Fixed seeds, recorded with the results. -- Budgets in one checked-in file keyed by profile, so tightening one is a - visible diff and loosening one needs a reason in review. +- One profile runs a fixed delay preset rather than auto, as the control. Every + other profile exercises adaptation, so without this the whole matrix can pass + while a fixed preset regresses, and fixed presets are what a viewer lands on + today. +- Budgets in one checked-in file, keyed by the whole matrix row and not by + profile alone: runtime, codec, sample rate, profile, and the ring path, since + each of those moves the expected floor. A profile-only key silently grades + one row against another's threshold. Tightening a budget is then a visible + diff and loosening one needs a reason in review. - Trim the released ndjson traces from `rt-audio-traces-2026-09-06` into a fixture and replay them too, so a real recorded arrival pattern is graded next to the synthetic profiles. diff --git a/quest/m2/audio-quality-harness/native.md b/quest/m2/audio-quality-harness/native.md index 8124e8e1558..8e6400c3b2a 100644 --- a/quest/m2/audio-quality-harness/native.md +++ b/quest/m2/audio-quality-harness/native.md @@ -18,14 +18,20 @@ from whichever platform a user happened to be on. - `moq play` needs to emit the run's counters and stage timings as JSON for the analyzer. Add that output, and keep it useful outside the test: a user debugging their own latency wants the same dump. -- Reuse the browser lane's analyzer and budget file. Native rows get their own - budgets in the same file, since the floor differs, but not their own schema. +- Reuse the browser lane's analyzer and budget file, with the JSON field names + and meanings matching its schema exactly. Native rows get their own budget + values in the same file, keyed the same way, since the device floor differs; + they do not get their own schema. - Add the lane to the nightly matrix beside the browser one. -Nothing here is a second estimator. If the native and browser numbers disagree -beyond the profiles' tolerance, that is a finding against -[Audio jitter target](/quest/m0/audio-jitter-target/README.md), not a reason to -widen a budget. +Nothing here is a second estimator. Compare the two runtimes at the estimator +first: the target series each produces from the same arrival trace, which is +the conformance corpus's own comparison and carries no device timing in it. A +disagreement there is a finding against [Audio jitter +target](/quest/m0/audio-jitter-target/README.md) and never a reason to widen a +budget. Only then compare end-to-end totals, with the backend-dependent stages +isolated: this lane deliberately accepts real device callback noise, so a +difference in totals alone proves nothing about the estimator. ## Required diff --git a/quest/m2/latency-ledger.md b/quest/m2/latency-ledger.md index 0bc54e49134..7c9f8e246f3 100644 --- a/quest/m2/latency-ledger.md +++ b/quest/m2/latency-ledger.md @@ -20,7 +20,10 @@ trade to get it running. This quest promotes them. - Switch the harness over, deleting the probes it replaces. A ledger with no consumer is how this drifts from reality. - The stages must sum to the measured end-to-end delay within a stated - tolerance, and that identity is itself a test. An unaccounted remainder is the + tolerance, and that identity is itself a test. It only holds on the harness's + terms: one duration unit, every timestamp on a named clock, and stages as + exclusive spans that cannot both claim the same milliseconds. Inherit those + from the schema rather than restating them. An unaccounted remainder is the bug this whole line exists to find, so it gets a name and a number rather than being absorbed into a neighbouring stage. From 0a3c5775c6c91ca71ebccf7dd2cb0c2fddf88365 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 01:17:55 +0000 Subject: [PATCH 5/7] docs(quest): the native harness lane needs the native estimator Consequence of the previous commit. Requiring the native lane to compare estimator target series across runtimes gave it a dependency it did not declare: native has no jitter buffer at all today, so the lane would have had no target series to compare and would have set its budgets against a playout path that holds nothing. The browser lane keeps its single blocker, so the harness line still starts without waiting on the estimator line. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XxFS48hsXmRqoxYWBV7StN --- quest/m2/audio-quality-harness/native.md | 1 + 1 file changed, 1 insertion(+) diff --git a/quest/m2/audio-quality-harness/native.md b/quest/m2/audio-quality-harness/native.md index 8e6400c3b2a..2efafe188b7 100644 --- a/quest/m2/audio-quality-harness/native.md +++ b/quest/m2/audio-quality-harness/native.md @@ -36,3 +36,4 @@ difference in totals alone proves nothing about the estimator. ## Required - [Browser](/quest/m2/audio-quality-harness/browser.md) - defines the metric schema, the budget file, and the extracted shaper +- [Native jitter target](/quest/m0/audio-jitter-target/native.md) - the estimator this lane grades and compares against the browser; without it there is no target series and the budgets would be set against a playout path that holds nothing From eb1efbf116ffae48e330afb019cc50baa0748305 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 01:19:15 +0000 Subject: [PATCH 6/7] docs(quest): close three more gaps in the estimator and metric contracts - An empty resampling interval produces no observation and so no decay step, which breaks the stated wall-clock memory across a pause. The spec quest now has to settle it, with a gap case in the corpus. - `js/watch/src/audio/config.ts` already adds the AudioWorklet's fixed 128-sample render block to the codec frame duration, so 48 kHz stereo Opus reads as 23 ms and not 20 ms. Native has no worklet, so if the shared step carries that quantum the two runtimes produce different target series from one trace. Named, along with the fact that the same code assumes 20 ms Opus frames. - Naming a clock per timestamp does not align clocks across the publisher, relay and viewer. The metric contract now requires one common monotonic reference or a stated calibration, since the ledger's sum identity is arithmetic over those values. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XxFS48hsXmRqoxYWBV7StN --- quest/m0/audio-jitter-target/spec.md | 21 +++++++++++++++++++-- quest/m2/audio-quality-harness/browser.md | 7 ++++++- 2 files changed, 25 insertions(+), 3 deletions(-) diff --git a/quest/m0/audio-jitter-target/spec.md b/quest/m0/audio-jitter-target/spec.md index 72cb45b81a6..6a395ccc168 100644 --- a/quest/m0/audio-jitter-target/spec.md +++ b/quest/m0/audio-jitter-target/spec.md @@ -58,8 +58,14 @@ The widely repeated "0.97" is a transitional value, not head. Say explicitly that the forget factor decays once per *resampled observation*, not once per arrival, and give the resulting wall-clock memory (`500ms / (1 - 0.983)`, about 29 s). Applying a per-observation factor per -arrival is precisely the mistake #3517 made, and a number that reads plausible -either way is exactly the kind that survives review. +arrival is the mistake #3517 made, and a number that reads plausible either way +is the kind that survives review. + +Then settle what an interval with no arrival does. An empty interval produces no +observation and therefore no decay step, so the wall-clock memory above does not +survive a pause unless elapsed empty intervals decay the histogram too. Either +decay them and say so, or drop the wall-clock claim; the conformance corpus needs +a case with a gap in it either way. Define units once, for both the algorithm and the corpus: what clock arrival times are on, what timescale frame timestamps use and how they convert, and @@ -112,6 +118,17 @@ Two lessons for the document, both structural rather than a constant to retune: catalog field, which is the expensive option and would need the full cross-package sync (`rs/hang`, `js/hang`, `doc/concept`, and the hang draft). Pick one and say why; do not leave it to each implementation. + + There is prior art to reconcile rather than ignore. `js/watch/src/audio/config.ts` + already derives a per-codec frame duration and then adds `WORKLET_QUANTUM`, the + AudioWorklet's fixed 128-sample render block, so 48 kHz stereo Opus reads as + 23 ms rather than 20 ms. Native has no worklet and its device buffer is a + property of the backend, so if the shared step carries the browser's quantum + and native's does not, the two produce different target series from the same + trace and the conformance corpus cannot hold. Decide whether the render block + belongs in the estimator's step at all or in the ring's slack above it, and + give native's equivalent a name. Note also that this code assumes 20 ms Opus + frames, which is the same guess the catalog cannot settle. - **The target needs a clamp and a bounded rise.** WebRTC clamps in `DelayConstraints` (`delay_constraints.{h,cc}`), applied by `DecisionLogic` rather than by the delay manager, with a hard ceiling at 75% of buffer diff --git a/quest/m2/audio-quality-harness/browser.md b/quest/m2/audio-quality-harness/browser.md index 4dc4b3469ec..ef6d63928c9 100644 --- a/quest/m2/audio-quality-harness/browser.md +++ b/quest/m2/audio-quality-harness/browser.md @@ -35,7 +35,12 @@ budget, or a schedule. treatment rather than being left to the reader. - Units and clock domain stated once. Durations in one unit, and every timestamp on a named clock, so a browser value and a native value are the - same measurement. + same measurement. Naming the clock is not enough on its own: the stages span + the publisher, the relay and the viewer, which are separate processes and + often separate machines, so either reduce every timestamp to one common + monotonic reference or define the calibration and drift correction that maps + between them. The ledger's sum-to-end-to-end identity is arithmetic across + these values, and it is meaningless if they sit on unaligned clocks. - Aggregation stated per metric: a count over the run, a max, or a percentile. "Underruns: 3" and "underruns: 3/s" grade differently. - Stages as exclusive, non-overlapping spans (capture, encode, publish flush, From 08cbbbb60a5f2777b0de4cb08b9a1709ae5a0c22 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 01:26:15 +0000 Subject: [PATCH 7/7] docs(quest): exclude reordered arrivals, and get native playback headless Two findings worth breaking the review stop for, both verified against the tree. Reordering is the same failure as the one this questline exists to fix. A newer group arriving before an older one makes the arrival difference positive while the timestamp difference is negative, so the media-time gap between them is added to the measured delay. It is ordinary input, not an edge case: `Container.Consumer` takes groups off `recvGroup()` in delivery order and sorts afterwards. WebRTC pairs the formula with the guard the spec quest had omitted, a `reordered` flag that keeps those arrivals out of the reference deques and routes them to the reorder optimiser instead. The corpus gets a reordering case. `moq play` cannot run headless today. A dummy cpal device covers the speaker, but `play::run` is the winit event loop and `resumed` builds a window and a presentable wgpu surface even for audio-only, so the native lane would exit before measuring anything. Either a virtual display in CI or an audio-only mode in the binary; the quest now has to pick. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XxFS48hsXmRqoxYWBV7StN --- quest/m0/audio-jitter-target/spec.md | 22 ++++++++++++++++++---- quest/m2/audio-quality-harness/native.md | 9 ++++++++- 2 files changed, 26 insertions(+), 5 deletions(-) diff --git a/quest/m0/audio-jitter-target/spec.md b/quest/m0/audio-jitter-target/spec.md index 6a395ccc168..c4667f206c7 100644 --- a/quest/m0/audio-jitter-target/spec.md +++ b/quest/m0/audio-jitter-target/spec.md @@ -34,18 +34,32 @@ and `packet_arrival_history.{h,cc}` under read as a hundred tiny observations, where this reads as a delay climbing to 200 ms. Frame timestamps stand in for RTP timestamps and there are no sequence numbers to unwrap. -2. **An empirical histogram with an exponential forget factor, read at a +2. **Reordered arrivals are excluded, not measured.** When a newer group + arrives before an older one, `arrival(p) - arrival(ref)` is positive while + the timestamp difference is negative, so the media-time gap between them is + *added* to the measured delay. That is the same failure this questline + exists to fix, a media-time distance leaking into a delay measurement, and it + arrives through the front door: `Container.Consumer` takes groups off + `recvGroup()` in delivery order and sorts afterwards, so out-of-order + observation is ordinary input rather than an edge case. WebRTC pairs the + formula with exactly this guard, which the description above omitted: + `DelayManager::Update` carries a `reordered` flag, `PacketArrivalHistory` + keeps reordered packets out of its min and max deques, and only the + `ReorderOptimizer` sees them, costed as delay against loss. Say what counts + as reordered here, keep those arrivals out of the reference and the + histogram, and put a reordering case in the conformance corpus. +3. **An empirical histogram with an exponential forget factor, read at a quantile.** 20 ms buckets, 100 of them. Seed the cold start with the decaying `0.5^(i+1)` prior. A quantile of an empirical distribution, not a mean plus k deviations: network delay is heavy-tailed and one-sided, so a Gaussian assumption sizes the tail wrong. -3. **Max-over-interval resampling**, 500 ms. Easy to miss and behaviourally +4. **Max-over-interval resampling**, 500 ms. Easy to miss and behaviourally significant: the histogram takes one observation per interval, the maximum in it. That decorrelates observations and turns the forget factor into a wall-clock time constant, which is the only form worth documenting. -4. **A cold-start forget ramp**, WebRTC's `start_forget_weight`, so the first +5. **A cold-start forget ramp**, WebRTC's `start_forget_weight`, so the first seconds converge instead of crawling. -5. **Explicit delay-versus-loss cost minimisation** for the late tail. WebRTC's +6. **Explicit delay-versus-loss cost minimisation** for the late tail. WebRTC's `ReorderOptimizer` and SpeexDSP's `compute_opt_delay` reached the same shape independently, which is decent evidence it is right. Defer it; ship the underrun estimator first. diff --git a/quest/m2/audio-quality-harness/native.md b/quest/m2/audio-quality-harness/native.md index 2efafe188b7..8f16e0a8284 100644 --- a/quest/m2/audio-quality-harness/native.md +++ b/quest/m2/audio-quality-harness/native.md @@ -11,7 +11,14 @@ from whichever platform a user happened to be on. - Drive the shipped binary, not a purpose-built harness: `moq play` against a local relay behind the same seeded shaper, on a cpal null or dummy backend so - CI needs no audio hardware. A real device callback is part of what is being + CI needs no audio hardware. The speaker is only half of it: `play::run` is the + winit event loop (`rs/moq-cli/src/play/mod.rs` re-exports `window::run`), and + `resumed` builds a `Display` with a real window and a presentable wgpu surface + even for an audio-only broadcast, so the lane dies on a headless runner before + it measures anything. Settle which way out: a virtual display and software + rendering in CI, or an audio-only mode in `moq play` that never opens a + window. The second is worth having on its own, and it is the smaller + dependency to keep working. A real device callback is part of what is being measured, so keep the real backend rather than substituting a fake clock, and accept that the timing noise it adds sets the floor for the native budgets. That floor is worth measuring on its own before the budgets are written.