Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions quest/m2/audio-playout/browser-audio-hole-skip.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# [M] Browser audio keeps a hole a late burst still fills

## Goal

`@moq/watch`'s audio no longer skips a missing group that lands a moment
after the group past it: the Chromium `bursty` and `congestion` rows of
`just test audio-quality` grade no worse than before
[moq-dev/moq#8799](https://github.com/moq-dev/moq/pull/8799) gave the audio
consumer the player's clock, while the Firefox `after resume` window that PR
fixed stays audible.

## Plan

Native evidence, from the PR that gave the Rust consumer the same clock
(`just test audio-quality --runtimes native --codecs opus --duration 40
--seed 7`): ending an audio hole once the clock reaches the group past it
took `bursty` from 3 to 155 underruns a minute. A burst's groups land late
together, so the group past a hole is already due when the missing one lands
a moment later in the same burst. Ending it at the clock plus the 200 ms
stretch held `bursty` level but left `congestion` worse than the native
estimate wait (A/V lag p95 808 to 946 ms against 288 to 631 ms over three
runs each). `moq play` gives the clock to video only for that reason.

`js/watch/src/audio/decoder.ts` passes `playhead: this.sync` with a
`wait` of `sync.out.maxDelay`, so the browser has the rule that measured
worst natively. First measure: the Chromium `bursty` and `congestion` rows
with and without that `playhead`, a few runs each, since `congestion` is
noisy.

If the browser regressed, recommended: audio waits a hole out for its
playout estimate from the arrival of the group past it, as native audio does
(`rs/moq-audio/src/decode/consumer.rs`, `set_wait(self.delay())`), instead of
the playhead. That still ends the resume hole within one playout delay.
Confirm with #8799's repro, `taskset -c 4-5 nix develop --command just test
media --media-engines firefox --media-cases rename,pause`. Video keeps the
playhead.

## Related

- [A hole is waited for by the subscription's delay](/quest/m2/delay-hole-wait.md) - removes the separate wait this would keep for browser audio, so the two settle the same rule
23 changes: 0 additions & 23 deletions quest/m2/audio-playout/rust-playhead-holes.md

This file was deleted.

30 changes: 20 additions & 10 deletions quest/m2/delay-hole-wait.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,15 +39,28 @@ Facts found by the audit:
reaches the group past it. The clock already trails live by the estimate,
so it carries the moving value the wait was added for.

- The Rust consumer has the clock too
(`moq_mux::container::Consumer::set_playhead`), and `moq play` gives it to
video only. Given to native audio it measured worse than the estimate wait
(`just test audio-quality --runtimes native --codecs opus --duration 40
--seed 7`): ending a hole at the clock cost `bursty` 155 underruns a minute
against 3, since a late burst's successor is already due when the missing
group lands a moment later; ending it at the clock plus the 200 ms stretch
held `bursty` and `mild` level, but over three runs each `congestion` ran
1714 to 1925 underruns a minute against 1307 to 1767 without it, and an A/V
lag p95 of 808 to 946 ms against 288 to 631 ms. The estimate wait gives
every hole the estimate from the arrival of the group past it, which a late
burst needs.

For the PR, settle the home of native audio's live estimate first. Two ways:

1. Recommended: the playhead is the home. Once
[native players skip a hole their clock passed](/quest/m2/audio-playout/rust-playhead-holes.md),
the clock ends a hole at the estimate, and the static rule (`min`, else
`max`) is only the fallback for a reader with no clock. Prove it by
deleting the `set_wait` call and running #8531's regression tests
1. The playhead is the home: the clock ends a hole, and the static rule
(`min`, else `max`) is only the fallback for a reader with no clock. The
measurements above count against it for audio unless a variant beats the
estimate wait on `bursty` and `congestion`. Prove any variant by deleting
the `set_wait` call and running #8531's regression tests
(`rs/moq-mux/src/container/consumer.rs:2161`, `:2308`, `:2369` and
moq-audio's reorder tests) with the playhead in place.
moq-audio's reorder tests) with the playhead in place, plus those rows.
2. moq-audio writes its estimate into the subscription's `Delay::min` as it
moves. `min` is Subscriber Min Age on the lite-07 wire, so each change is
a SUBSCRIBE_UPDATE, and since [a cold join reaches back by it](/quest/m2/min-age-join.md)
Expand All @@ -63,11 +76,8 @@ knob: the maintainer rejected that end state.
Public API: `moq-mux` loses `container::Consumer::set_wait`; `@moq/hang`
loses `ConsumerProps.wait`. Wire: none.

## Required

- [Native players stop waiting on a hole their clock passed](/quest/m2/audio-playout/rust-playhead-holes.md) - gives the Rust consumer the clock that replaces the live wait

## Related

- [A cold join reaches back by Min Age](/quest/m2/min-age-join.md) - gives `Delay::min` a reader on the wire, which rules out using it as a fast-moving estimate
- [Native audio playout is a library](/quest/m3/audio-playout-library.md) - moves the placement this wait feeds into moq-audio
- [Browser audio keeps a hole a late burst still fills](/quest/m2/audio-playout/browser-audio-hole-skip.md) - measures whether the browser's audio playhead over-skips, which decides the rule this quest applies to both languages
76 changes: 76 additions & 0 deletions rs/moq-cli/src/play/video.rs
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,9 @@ impl<O: Output> Video<O> {
// Signalled when a segment behind the front becomes the front.
let advanced = tokio::sync::Notify::new();
let role = self.presentation.track(moq_mux::container::Kind::Video);
// A missing group stops being waited for once the clock reaches the group after it,
// whose pictures would replace any late one it held.
track.set_playhead(self.presentation.consume());
let receive = async {
// Video has no jitter estimate, so its target is the encoder delay alone,
// which holds audio back with it.
Expand Down Expand Up @@ -1359,4 +1362,77 @@ mod tests {
async fn a_flush_larger_than_the_queue_keeps_its_tail() {
schedule(&(0..10).collect::<Vec<_>>(), 5).await;
}

/// A lost group behind media that arrives late: once the clock, which audio holds, reaches the
/// group past the hole, video shows it rather than waiting a whole playout delay from its
/// arrival, which would leave the picture frozen that much longer.
#[tokio::test]
async fn a_hole_ends_once_the_clock_reaches_the_group_past_it() {
tokio::time::pause();
let delay = Duration::from_millis(100);
let media = media(delay);
let start = Instant::now();
// Audio arrives on time throughout, so its arrivals hold the clock.
let audio = media.presentation.track(moq_mux::container::Kind::Audio);
audio.received(moq_net::Timestamp::ZERO, start);
let broadcast = moq_net::broadcast::Info::new().produce();
let published = broadcast
.create_track("video", hang::container::track_info(hang::catalog::PRIORITY.video))
.unwrap();
// The budgets `moq play` gives video: the playout delay, as the min and the max.
let subscriber = published
.consume()
.subscribe(
moq_net::track::Subscription::default()
.with_delay(moq_net::track::Delay::default().with_min(delay).with_max(delay)),
)
.await
.unwrap();
let format = || moq_mux::catalog::hang::Container::Legacy(moq_mux::container::Kind::Video);
let track = moq_mux::container::Consumer::new(subscriber, format());
let task = tokio::spawn(playback(&media, delay).run(track, Buffered::new(0)));
// One keyframe group per 100 ms of media.
let write = |sequence: u64| {
let mut group = published
.create_group(moq_net::group::Info { start: sequence })
.unwrap();
format().write(&mut group, &[frame(sequence * 100, true)]).unwrap();
group.close().unwrap();
};
let mut shown = Vec::new();
// Groups 0 to 4 arrive live. Group 5 is lost, and from 6 on the path holds video 300 ms.
for at in 0..1_200u64 {
if at % 100 == 0 {
let sequence = match at < 500 {
true => Some(at / 100),
false => (at >= 900).then(|| (at - 300) / 100),
};
if let Some(sequence) = sequence {
write(sequence);
}
}
tokio::task::yield_now().await;
if let Some(timestamp) = media.output.present(&media) {
shown.push((timestamp.as_millis().unwrap(), start.elapsed()));
}
tokio::time::advance(Duration::from_millis(1)).await;
}
task.abort();
// Group 6 lands at 0.9 s, past its 0.7 s due time, so it shows then. Waiting the playout
// delay from its arrival instead would skip at 1 s, when group 7 is due too and replaces it.
let after = shown
.iter()
.filter(|(timestamp, _)| *timestamp >= 500)
.copied()
.collect::<Vec<_>>();
assert_eq!(
after,
[
(600, Duration::from_millis(900)),
(700, Duration::from_millis(1_000)),
(800, Duration::from_millis(1_100)),
],
"shown: {shown:?}"
);
}
}
Loading