Repository navigation
docs(quest): plan the audio jitter target and its quality harness #3576
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
020533b
docs(quest): plan the audio jitter target and its quality harness
claude 88e3562
docs(quest): record the reproduced cause of the 14.56s auto target
claude b804793
docs(quest): fix the dependency edges and the frame duration source
claude aefa2c3
docs(quest): pin down the contracts the plans left to the reader
claude 0a3c577
docs(quest): the native harness lane needs the native estimator
claude eb1efbf
docs(quest): close three more gaps in the estimator and metric contracts
claude 08cbbbb
docs(quest): exclude reordered arrivals, and get native playback head…
claude File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # Audio jitter target | ||
|
|
||
| ## Goal | ||
|
|
||
| The audio playout target is a measured estimate of arrival timing, ported from | ||
| a known-good implementation, in the browser and natively alike. The round-trip | ||
| formula is gone. `max(20ms, 1.25 x minRtt)` sized the buffer for a single | ||
| retransmit, which describes how long the network takes to recover a loss and | ||
| not how unevenly a publisher emits frames, so a sender flushing 100 ms of media | ||
| at once got a target far too shallow to play through the next flush. One | ||
| written algorithm, cited by both implementations, produces the same target from | ||
| the same arrival trace. | ||
|
|
||
| Boundaries: convergence still uses skip-ahead and silence, so playing slightly | ||
| faster or slower to converge stays [Time | ||
| stretch](/quest/m2/watch-audio-time-stretch.md). No packet loss concealment. | ||
| Video keeps its own target; making the audio playhead the clock is [Plan: A/V | ||
| clock](/quest/m1/plan-av-clock.md). | ||
|
|
||
| ## Plan | ||
|
|
||
| Three quests: the survey and the written algorithm first, then one | ||
| implementation per language against it. The two implementations are | ||
| independent once the spec lands and may run in parallel. | ||
|
|
||
| PR #3517 stays open and is fixed in place by the watch quest rather than | ||
| restarted. It already deleted the RTT term and plumbed a per-track arrival | ||
| `spread` through `Container.Consumer`, measured at container frame arrival and | ||
| before the age budget can skip a group, which is the right observation point. | ||
| What it got wrong is the estimator on top. A viewer on that branch reads a | ||
| 14.56 s jitter buffer in auto, and the cause is reproduced in | ||
| [Spec](/quest/m0/audio-jitter-target/spec.md): the "plus one frame" term is | ||
| learned from the first gap between observed timestamps, so a tune-in across a | ||
| stale group sets it to seconds, and nothing clamps the target or bounds how | ||
| fast it rises. | ||
|
|
||
| Native has no jitter buffer at all. `rs/moq-audio`'s decode `Config` carries | ||
| only `latency_max`, an upper bound before skipping a stalled group, and its own | ||
| doc comment promises "a companion `latency_min` for jitter-buffer padding will | ||
| land in a follow-up". This questline is that follow-up. | ||
|
|
||
| ## Quests | ||
|
|
||
| - [Spec](/quest/m0/audio-jitter-target/spec.md) - survey what already exists, then write the algorithm down once | ||
| - [Watch](/quest/m0/audio-jitter-target/watch.md) - js/watch and js/hang implement it, fixing #3517 in place | ||
| - [Native](/quest/m0/audio-jitter-target/native.md) - rs/moq-audio grows `latency_min` from the same algorithm | ||
|
|
||
| ## Closes | ||
|
|
||
| - [#3477](https://github.com/moq-dev/moq/issues/3477) - close this issue when the questline finishes | ||
| - [#2812](https://github.com/moq-dev/moq/issues/2812) - the iOS stutter report the same estimator fixes | ||
|
|
||
| ## Related | ||
|
|
||
| - [Jitter estimator](/quest/m0/3479-mux-jitter-flush-span.md) - the advertised flush span, whose relationship to the measured target the spec settles | ||
| - [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - the automated proof, built on its own schedule | ||
| - [Time stretch](/quest/m2/watch-audio-time-stretch.md) - inaudible convergence, on top of this | ||
| - [Plan: A/V clock](/quest/m1/plan-av-clock.md) - the clock this target eventually feeds | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| # [M] rs/moq-audio: a measured jitter buffer, not just an upper bound | ||
|
|
||
| ## Goal | ||
|
|
||
| Native playback holds a jitter buffer sized by the algorithm from | ||
| [Spec](/quest/m0/audio-jitter-target/spec.md), and passes the same conformance | ||
| vector as the browser on the same trace. `moq play` on a path with uneven | ||
| arrivals plays through a flush without underrunning, where today it has nothing | ||
| to absorb one. | ||
|
|
||
| Boundaries: no time-stretching and no loss concealment, matching the browser. | ||
| Video keeps its own path. | ||
|
|
||
| ## Plan | ||
|
|
||
| `rs/moq-audio` has no jitter buffer at all. `decode::Config` carries only | ||
| `latency_max`, an upper bound before skipping a stalled group, and its doc | ||
| comment already promises the companion: | ||
|
|
||
| > The `_max` suffix is a reminder that we never *add* latency here: the consumer | ||
| > skips only when newer data is already this far ahead. A companion | ||
| > `latency_min` for jitter-buffer padding will land in a follow-up. | ||
|
|
||
| - Measure arrivals at the same point the browser does, on the container | ||
| consumer before the age budget can skip a group, so both languages estimate | ||
| from the same observation. | ||
| - Add `latency_min` to `decode::Config`, additive on a `#[non_exhaustive]` | ||
| struct, so it targets `main`. Its exact contract is a decision this quest | ||
| makes and records, not something to leave to the reader, because "floor" and | ||
| "fixed target" behave differently under an adaptive estimator: | ||
|
|
||
| 1. **Floor.** `Some(d)` is a lower bound and the estimator may still raise | ||
| the target above it. Recommended: it matches how the browser treats a | ||
| rendition's advertised delay, and a viewer asking for more buffer rarely | ||
| means "and never more than that". | ||
| 2. **Exact target.** `Some(d)` pins the target and disables adaptation, | ||
| mirroring a fixed browser preset exactly. Choose this if preserving a | ||
| fixed-latency preset byte for byte matters more than the parallel above. | ||
|
|
||
| Either way, state the precedence against the estimator and against | ||
| `latency_max`, and what happens when the requested floor exceeds the | ||
| effective `latency_max` that `Consumer` already clamps to publisher | ||
| retention. Refusing that combination loudly beats silently clamping it. | ||
| - The estimator itself is internal. Only the config field and whatever the | ||
| playback path needs to report its current target become public, and each one | ||
| is argued for rather than exposed by default. | ||
| - `rs/moq-cli` `play` gets the matching knob, and `doc/bin/cli.md` follows. Check | ||
| the examples against `--help`. | ||
| - The playback sink holds the target the way the browser's rings do: slack above | ||
| it, land on it when skipping, re-stall when dry so the next insert refills to | ||
| the target rather than playing on an empty cushion. | ||
| - Tests: the conformance vector from the spec, plus a decode-path test that | ||
| replays the same trimmed arrival trace the browser replays and asserts the | ||
| same target series. | ||
|
|
||
| `latency_min` is the natural name only because `latency_max` already exists. If | ||
| it reads wrong next to a `Duration` that is a floor rather than a budget, | ||
| propose alternatives with a recommendation rather than shipping it. | ||
|
|
||
| ## Required | ||
|
|
||
| - [Spec](/quest/m0/audio-jitter-target/spec.md) - the algorithm this implements |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.