diff --git a/quest/m2/audio-playout/browser-audio-hole-skip.md b/quest/m2/audio-playout/browser-audio-hole-skip.md new file mode 100644 index 00000000000..075ac73b448 --- /dev/null +++ b/quest/m2/audio-playout/browser-audio-hole-skip.md @@ -0,0 +1,40 @@ +# [M] Browser audio keeps a hole a late burst still fills + +## Goal + +`@moq/watch`'s audio no longer skips a missing group that lands a moment +after the group past it: the Chromium `bursty` and `congestion` rows of +`just test audio-quality` grade no worse than before +[moq-dev/moq#8799](https://github.com/moq-dev/moq/pull/8799) gave the audio +consumer the player's clock, while the Firefox `after resume` window that PR +fixed stays audible. + +## Plan + +Native evidence, from the PR that gave the Rust consumer the same clock +(`just test audio-quality --runtimes native --codecs opus --duration 40 +--seed 7`): ending an audio hole once the clock reaches the group past it +took `bursty` from 3 to 155 underruns a minute. A burst's groups land late +together, so the group past a hole is already due when the missing one lands +a moment later in the same burst. Ending it at the clock plus the 200 ms +stretch held `bursty` level but left `congestion` worse than the native +estimate wait (A/V lag p95 808 to 946 ms against 288 to 631 ms over three +runs each). `moq play` gives the clock to video only for that reason. + +`js/watch/src/audio/decoder.ts` passes `playhead: this.sync` with a +`wait` of `sync.out.maxDelay`, so the browser has the rule that measured +worst natively. First measure: the Chromium `bursty` and `congestion` rows +with and without that `playhead`, a few runs each, since `congestion` is +noisy. + +If the browser regressed, recommended: audio waits a hole out for its +playout estimate from the arrival of the group past it, as native audio does +(`rs/moq-audio/src/decode/consumer.rs`, `set_wait(self.delay())`), instead of +the playhead. That still ends the resume hole within one playout delay. +Confirm with #8799's repro, `taskset -c 4-5 nix develop --command just test +media --media-engines firefox --media-cases rename,pause`. Video keeps the +playhead. + +## Related + +- [A hole is waited for by the subscription's delay](/quest/m2/delay-hole-wait.md) - removes the separate wait this would keep for browser audio, so the two settle the same rule diff --git a/quest/m2/audio-playout/rust-playhead-holes.md b/quest/m2/audio-playout/rust-playhead-holes.md deleted file mode 100644 index a17023ed1dd..00000000000 --- a/quest/m2/audio-playout/rust-playhead-holes.md +++ /dev/null @@ -1,23 +0,0 @@ -# [M] Native players stop waiting on a hole their clock passed - -## Goal - -`moq play` skips a missing group once its presentation clock reaches the -group past it, as the browser player does since -[moq-dev/moq#8799](https://github.com/moq-dev/moq/pull/8799), instead of -holding the frames behind the hole for the whole hole wait. - -## Plan - -The browser's container consumer takes the player's clock as -`ConsumerProps.playhead` (`js/hang/src/container/consumer.ts`): a hole whose -successor starts at or before the media time presenting now ends then, and the -hole's timer also wakes when the clock gets there. Mirror it in -`rs/moq-mux/src/container/consumer.rs` (`poll_event_once`'s missing-group -branch and its `hole_timer`), reading `moq_mux::sync::Consumer::position`, and -pass the clock from `moq play`'s presentation through -`moq_audio::decode::Consumer` and the video reader in -`rs/moq-cli/src/play/media.rs`. - -Exercise it with the same shape the browser hit: a reader that delivered a -pre-pause tail, then a replayed backlog arriving newest first behind a hole. diff --git a/quest/m2/delay-hole-wait.md b/quest/m2/delay-hole-wait.md index d577b32691c..e3017ca1486 100644 --- a/quest/m2/delay-hole-wait.md +++ b/quest/m2/delay-hole-wait.md @@ -39,15 +39,28 @@ Facts found by the audit: reaches the group past it. The clock already trails live by the estimate, so it carries the moving value the wait was added for. +- The Rust consumer has the clock too + (`moq_mux::container::Consumer::set_playhead`), and `moq play` gives it to + video only. Given to native audio it measured worse than the estimate wait + (`just test audio-quality --runtimes native --codecs opus --duration 40 + --seed 7`): ending a hole at the clock cost `bursty` 155 underruns a minute + against 3, since a late burst's successor is already due when the missing + group lands a moment later; ending it at the clock plus the 200 ms stretch + held `bursty` and `mild` level, but over three runs each `congestion` ran + 1714 to 1925 underruns a minute against 1307 to 1767 without it, and an A/V + lag p95 of 808 to 946 ms against 288 to 631 ms. The estimate wait gives + every hole the estimate from the arrival of the group past it, which a late + burst needs. + For the PR, settle the home of native audio's live estimate first. Two ways: -1. Recommended: the playhead is the home. Once - [native players skip a hole their clock passed](/quest/m2/audio-playout/rust-playhead-holes.md), - the clock ends a hole at the estimate, and the static rule (`min`, else - `max`) is only the fallback for a reader with no clock. Prove it by - deleting the `set_wait` call and running #8531's regression tests +1. The playhead is the home: the clock ends a hole, and the static rule + (`min`, else `max`) is only the fallback for a reader with no clock. The + measurements above count against it for audio unless a variant beats the + estimate wait on `bursty` and `congestion`. Prove any variant by deleting + the `set_wait` call and running #8531's regression tests (`rs/moq-mux/src/container/consumer.rs:2161`, `:2308`, `:2369` and - moq-audio's reorder tests) with the playhead in place. + moq-audio's reorder tests) with the playhead in place, plus those rows. 2. moq-audio writes its estimate into the subscription's `Delay::min` as it moves. `min` is Subscriber Min Age on the lite-07 wire, so each change is a SUBSCRIBE_UPDATE, and since [a cold join reaches back by it](/quest/m2/min-age-join.md) @@ -63,11 +76,8 @@ knob: the maintainer rejected that end state. Public API: `moq-mux` loses `container::Consumer::set_wait`; `@moq/hang` loses `ConsumerProps.wait`. Wire: none. -## Required - -- [Native players stop waiting on a hole their clock passed](/quest/m2/audio-playout/rust-playhead-holes.md) - gives the Rust consumer the clock that replaces the live wait - ## Related - [A cold join reaches back by Min Age](/quest/m2/min-age-join.md) - gives `Delay::min` a reader on the wire, which rules out using it as a fast-moving estimate - [Native audio playout is a library](/quest/m3/audio-playout-library.md) - moves the placement this wait feeds into moq-audio +- [Browser audio keeps a hole a late burst still fills](/quest/m2/audio-playout/browser-audio-hole-skip.md) - measures whether the browser's audio playhead over-skips, which decides the rule this quest applies to both languages diff --git a/rs/moq-cli/src/play/video.rs b/rs/moq-cli/src/play/video.rs index 43fea509254..9cbb47203c3 100644 --- a/rs/moq-cli/src/play/video.rs +++ b/rs/moq-cli/src/play/video.rs @@ -157,6 +157,9 @@ impl Video { // Signalled when a segment behind the front becomes the front. let advanced = tokio::sync::Notify::new(); let role = self.presentation.track(moq_mux::container::Kind::Video); + // A missing group stops being waited for once the clock reaches the group after it, + // whose pictures would replace any late one it held. + track.set_playhead(self.presentation.consume()); let receive = async { // Video has no jitter estimate, so its target is the encoder delay alone, // which holds audio back with it. @@ -1359,4 +1362,77 @@ mod tests { async fn a_flush_larger_than_the_queue_keeps_its_tail() { schedule(&(0..10).collect::>(), 5).await; } + + /// A lost group behind media that arrives late: once the clock, which audio holds, reaches the + /// group past the hole, video shows it rather than waiting a whole playout delay from its + /// arrival, which would leave the picture frozen that much longer. + #[tokio::test] + async fn a_hole_ends_once_the_clock_reaches_the_group_past_it() { + tokio::time::pause(); + let delay = Duration::from_millis(100); + let media = media(delay); + let start = Instant::now(); + // Audio arrives on time throughout, so its arrivals hold the clock. + let audio = media.presentation.track(moq_mux::container::Kind::Audio); + audio.received(moq_net::Timestamp::ZERO, start); + let broadcast = moq_net::broadcast::Info::new().produce(); + let published = broadcast + .create_track("video", hang::container::track_info(hang::catalog::PRIORITY.video)) + .unwrap(); + // The budgets `moq play` gives video: the playout delay, as the min and the max. + let subscriber = published + .consume() + .subscribe( + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_min(delay).with_max(delay)), + ) + .await + .unwrap(); + let format = || moq_mux::catalog::hang::Container::Legacy(moq_mux::container::Kind::Video); + let track = moq_mux::container::Consumer::new(subscriber, format()); + let task = tokio::spawn(playback(&media, delay).run(track, Buffered::new(0))); + // One keyframe group per 100 ms of media. + let write = |sequence: u64| { + let mut group = published + .create_group(moq_net::group::Info { start: sequence }) + .unwrap(); + format().write(&mut group, &[frame(sequence * 100, true)]).unwrap(); + group.close().unwrap(); + }; + let mut shown = Vec::new(); + // Groups 0 to 4 arrive live. Group 5 is lost, and from 6 on the path holds video 300 ms. + for at in 0..1_200u64 { + if at % 100 == 0 { + let sequence = match at < 500 { + true => Some(at / 100), + false => (at >= 900).then(|| (at - 300) / 100), + }; + if let Some(sequence) = sequence { + write(sequence); + } + } + tokio::task::yield_now().await; + if let Some(timestamp) = media.output.present(&media) { + shown.push((timestamp.as_millis().unwrap(), start.elapsed())); + } + tokio::time::advance(Duration::from_millis(1)).await; + } + task.abort(); + // Group 6 lands at 0.9 s, past its 0.7 s due time, so it shows then. Waiting the playout + // delay from its arrival instead would skip at 1 s, when group 7 is due too and replaces it. + let after = shown + .iter() + .filter(|(timestamp, _)| *timestamp >= 500) + .copied() + .collect::>(); + assert_eq!( + after, + [ + (600, Duration::from_millis(900)), + (700, Duration::from_millis(1_000)), + (800, Duration::from_millis(1_100)), + ], + "shown: {shown:?}" + ); + } } diff --git a/rs/moq-mux/src/container/consumer.rs b/rs/moq-mux/src/container/consumer.rs index 805ae842346..116e797b72c 100644 --- a/rs/moq-mux/src/container/consumer.rs +++ b/rs/moq-mux/src/container/consumer.rs @@ -74,7 +74,9 @@ pub(crate) enum Item { /// group in the same burst as the ones past it. Its wait, [`set_wait`](Self::set_wait) or the /// max delay, also expires that much monotonic time after the first group past it arrived, so a /// marker behind a hole is delivered even when the track is paused, and the holes a stalled -/// source left behind expire together rather than one wait each. +/// source left behind expire together rather than one wait each. A reader feeding a playout clock +/// ([`set_playhead`](Self::set_playhead)) also skips a hole once that clock reaches where the +/// first group past it begins. /// /// Delivery starts at the floor the /// [`Subscription::ranges`](moq_net::track::Subscription::ranges) name, raised to where the @@ -180,6 +182,9 @@ pub struct Consumer { // Wakes the reader once the hole at the cursor, or the start's wait, expires. hole_timer: Option>>, + // The playout clock this reader feeds, which ends a hole once it reaches the group past it. + playhead: Option, + // The furthest presentation point delivered so far: max(timestamp + duration), endpoints // included. `None` until the first item, so a loss before it is no gap. edge: Option, @@ -425,6 +430,7 @@ impl Consumer { max_delay, wait: None, hole_timer: None, + playhead: None, edge: None, start: None, floor: None, @@ -847,6 +853,13 @@ impl Consumer { let wait = self.wait.map_or(self.max_delay, |wait| wait.min(self.max_delay)); let elapsed = since.elapsed() >= wait; let aged = next_group.is_some_and(|(_, start)| max_timestamp.saturating_sub(start) >= self.max_delay); + // Once the playout clock reaches where the first group past it begins, that group's + // media is due, and it replaces whatever the missing one held. + let schedule = self.playhead.as_ref().map(|playhead| playhead.schedule()); + let reached = schedule + .zip(next_group) + .and_then(|(schedule, (_, start))| schedule.due(Timestamp::try_from(start).ok()?)); + let passed = reached.is_some_and(|at| at <= kio::clock::now()); let excluded = match &self.reader { Reader::Reading(track) | Reader::Failed(track, _) => { track.next_in_range(self.current).is_none_or(|next| next >= sequence) @@ -861,7 +874,7 @@ impl Consumer { // skipped, still waits for the group past it to show where it starts: stepping // before that frame lands would call a contiguous timeline a loss. let excluded = excluded && first.is_ready(); - if finished || elapsed || aged || excluded || full || continues { + if finished || elapsed || aged || passed || excluded || full || continues { let _ = self.pending.front_mut().unwrap().look_all(waiter, &self.format); let front = self.pending.front_mut().unwrap(); let empty = front.is_empty() && matches!(front.group.poll_finished(waiter), Poll::Ready(Ok(_))); @@ -878,6 +891,7 @@ impl Consumer { finished, elapsed, aged, + passed, excluded, full, continues, @@ -887,13 +901,22 @@ impl Consumer { self.note_group_edge(); continue; } - if let Some(wake) = since.checked_add(wait) { + // The wait expires, or the clock reaches the group past the hole, whichever is first. + let expires = since.checked_add(wait); + if let Some(wake) = expires.into_iter().chain(reached).min() { let timer = self .hole_timer .get_or_insert_with(|| Box::pin(kio::clock::sleep_until(wake))); timer.as_mut().reset(wake); let _ = waiter.poll_future(timer.as_mut()); } + // A clock that jumps, re-anchoring or changing its delay, moves when it gets there. + if let Some((playhead, schedule)) = self.playhead.as_ref().zip(schedule) + && next_group.is_some() + && playhead.poll_changed(waiter, schedule).is_ready() + { + continue; + } } // The current group is blocking. Everything it could still present ends @@ -1395,6 +1418,15 @@ impl Consumer { self.wait = Some(wait); } + /// Skip a missing group once `playhead`, the playout clock this reader feeds, reaches where the + /// first group past it begins (default: no clock), rather than waiting out the whole + /// [`set_wait`](Self::set_wait). Suits media where a late item is worthless once the next is due, + /// such as video, whose next due picture replaces it. Audio that still plays late, catching up + /// by time stretch, keeps the wait. + pub fn set_playhead(&mut self, playhead: crate::sync::Consumer) { + self.playhead = Some(playhead); + } + /// Cap the bytes of decoded frames held ahead of the reader, counting the wire payload they share /// and each frame's and group's own overhead (default [`DEFAULT_MAX_BUFFER`]). Past it the consumer stops decoding ahead /// and pulling new groups; see the type docs. @@ -2435,6 +2467,140 @@ mod tests { assert_eq!(drain(&mut reader), ["gap", "frame 8", "frame 9"]); } + /// Everything `reader` hands over now, polled with `waiter`. + fn drain_with(reader: &mut Consumer, waiter: &kio::Waiter) -> Vec { + let mut read = Vec::new(); + while let Poll::Ready(Ok(Some(item))) = reader.poll_read(waiter) { + read.push(match item { + Read::Frame { group, .. } => format!("frame {group}"), + Read::Gap { .. } => "gap".to_string(), + Read::End { .. } => "end".to_string(), + }); + } + read + } + + /// A waker that records whether it was woken. + #[derive(Default)] + struct Woken(std::sync::atomic::AtomicBool); + + impl std::task::Wake for Woken { + fn wake(self: Arc) { + self.0.store(true, Ordering::SeqCst); + } + } + + impl Woken { + fn waiter(self: &Arc) -> kio::Waiter { + kio::Waiter::new(std::task::Waker::from(self.clone())) + } + + fn take(&self) -> bool { + self.0.swap(false, Ordering::SeqCst) + } + } + + /// A reader that delivered group 0, waiting 500 ms on a hole within a 2 s max delay, and fed + /// to `playhead` when given. + fn holed(track: &mut moq_net::track::Producer, playhead: Option) -> Consumer { + let subscriber = track.subscribe( + moq_net::track::Subscription::default().with_delay(Delay::default().with_max(Duration::from_secs(2))), + ); + let mut reader = Consumer::new(subscriber, Container::Legacy(crate::container::Kind::Data)); + reader.set_wait(Duration::from_millis(500)); + if let Some(playhead) = playhead { + reader.set_playhead(playhead); + } + write_group(track, 0, &[ts(0)]); + assert_eq!(drain_with(&mut reader, &kio::Waiter::noop()), ["frame 0"]); + reader + } + + /// A hole ends once the playout clock reaches where the group past it begins, rather than at + /// the wait, which still ends one the clock trails or a reader without a clock. + #[tokio::test(start_paused = true)] + async fn a_hole_ends_once_the_playout_clock_reaches_the_group_past_it() { + // The media presenting as the group past the hole arrives, and how long until the hole ends. + for (presenting, ends) in [(Some(900), 100), (Some(0), 500), (None, 500)] { + let mut track = track_producer("data", hang::container::track_info(hang::catalog::PRIORITY.audio)); + let clock = crate::sync::Producer::new(Duration::ZERO); + // Another track drives the clock, which presents `presenting` ms of media now. + let video = clock.track(crate::container::Kind::Video); + if let Some(millis) = presenting { + video.received(ts(millis * 1_000), kio::clock::now()); + } + let mut reader = holed(&mut track, presenting.map(|_| clock.consume())); + // A resume: the group past the hole starts a second of media later, inside the max delay. + write_group(&mut track, 50, &[ts(1_000_000)]); + let woken = Arc::new(Woken::default()); + let waiter = woken.waiter(); + assert!(drain_with(&mut reader, &waiter).is_empty()); + tokio::time::advance(Duration::from_millis(ends - 1)).await; + assert!(drain_with(&mut reader, &waiter).is_empty(), "{presenting:?}"); + woken.take(); + tokio::time::advance(Duration::from_millis(1)).await; + assert!(woken.take(), "{presenting:?}: the hole's end did not wake the reader"); + assert_eq!(drain_with(&mut reader, &waiter), ["gap", "frame 50"], "{presenting:?}"); + } + } + + /// A hole the playout clock trails ends as soon as the clock jumps past the group after it, which + /// wakes the reader: a re-anchor here, as a raised or lowered delay would. + #[tokio::test(start_paused = true)] + async fn a_hole_ends_once_the_playout_clock_jumps_past_the_group_past_it() { + let mut track = track_producer("data", hang::container::track_info(hang::catalog::PRIORITY.audio)); + let clock = crate::sync::Producer::new(Duration::ZERO); + let video = clock.track(crate::container::Kind::Video); + video.received(ts(0), kio::clock::now()); + let mut reader = holed(&mut track, Some(clock.consume())); + write_group(&mut track, 50, &[ts(1_000_000)]); + let woken = Arc::new(Woken::default()); + let waiter = woken.waiter(); + assert!(drain_with(&mut reader, &waiter).is_empty()); + tokio::time::advance(Duration::from_millis(10)).await; + assert!(drain_with(&mut reader, &waiter).is_empty()); + woken.take(); + // Video arrives a second of media ahead, which re-pins the clock past the group. + assert!(video.received(ts(1_000_000), kio::clock::now())); + assert!(woken.take(), "the clock's jump did not wake the reader"); + assert_eq!(drain_with(&mut reader, &waiter), ["gap", "frame 50"]); + } + + /// The browser's resume: a reader that delivered a tail before a pause, then a replayed backlog + /// arriving newest first behind a hole. The clock kept running through the pause, so the reader + /// moves on to the oldest group that arrived as soon as the clock passes it, rather than + /// waiting for the backlog's oldest group, which lands last. + #[tokio::test(start_paused = true)] + async fn a_backlog_replayed_newest_first_plays_from_where_the_clock_passed() { + let mut track = track_producer("data", hang::container::track_info(hang::catalog::PRIORITY.audio)); + let clock = crate::sync::Producer::new(Duration::from_millis(100)); + let audio = clock.track(crate::container::Kind::Audio); + let subscriber = track.subscribe( + moq_net::track::Subscription::default().with_delay(Delay::default().with_max(Duration::from_secs(2))), + ); + let mut reader = Consumer::new(subscriber, Container::Legacy(crate::container::Kind::Data)); + reader.set_wait(Duration::from_millis(500)); + reader.set_playhead(clock.consume()); + let waiter = kio::Waiter::noop(); + // The tail before the pause, a group per 100 ms of media and of wall time. + for sequence in 0..5 { + write_group(&mut track, sequence, &[ts(sequence * 100_000)]); + assert_eq!(drain_with(&mut reader, &waiter), [format!("frame {sequence}")]); + audio.received(ts(sequence * 100_000), kio::clock::now()); + tokio::time::advance(Duration::from_millis(100)).await; + } + // Paused until 1.4 s, when the publisher replays groups 14 down to 5, one every 50 ms. + tokio::time::advance(Duration::from_millis(900)).await; + let mut read = Vec::new(); + for sequence in (5..15).rev() { + write_group(&mut track, sequence, &[ts(sequence * 100_000)]); + read.extend(drain_with(&mut reader, &waiter)); + tokio::time::advance(Duration::from_millis(50)).await; + } + // At 1.45 s the clock presents 1.35 s, past group 13, which landed then. + assert_eq!(read, ["gap", "frame 13", "frame 14"]); + } + /// A malformed data marker group found while looking ahead fails the reader, rather than /// reading as a gap when a latency skip passes over it. #[tokio::test(start_paused = true)]