diff --git a/doc/bin/gstreamer.md b/doc/bin/gstreamer.md index c25cc046005..92a8482c3db 100644 --- a/doc/bin/gstreamer.md +++ b/doc/bin/gstreamer.md @@ -118,6 +118,11 @@ restarts, it switches to the new broadcast on the same pads, kept by rendition name, so a pipeline linked by name keeps playing. A format change keeps its pad too, with new caps downstream must accept. The new run's first catalog must list every returning rendition; one it omits gets EOS instead of keeping its old pad. +Within each run, the first media frame fixes one timestamp reference and segment +base for every pad. Audio and video keep their relative broadcast timing, even +when their tracks use different timescales or arrive at different times. Frames +before that reference keep an earlier PTS for downstream to clip. A new broadcast +chooses a fresh reference on the same pads; playback does not require a wall clock. A rendition that leaves the catalog gets EOS once its track ends. A broadcast that ends or loses its publisher holds its pads without EOS until it is announced again, so the pipeline does not end diff --git a/quest/m1/README.md b/quest/m1/README.md index 2acee7fc5cd..903d6bd4ee9 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -142,7 +142,6 @@ above them. - [Open-GOP leading pictures](/quest/m1/open-gop-leading-pictures.md) - a viewer joining at a recovery point drops the leading pictures it cannot decode; continuous viewers keep them - [Audio warmup](/quest/m1/audio-warmup.md) - a viewer joining an Opus rendition mid-stream never hears the unconverged first 80 ms - [#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) - moq-gst picks the broadcast wall epoch; a restarted source is a new epoch, not a forward re-anchor -- [moqsrc A/V alignment](/quest/m1/gst-src-av-sync.md) - every `moqsrc` pad of a run shares one timestamp reference and segment base - [CLI stdin shutdown on release](/quest/m1/cli-stdin-release.md) - backport only the runtime-teardown fix so a stopped CLI exits while its stdin writer stays open - [Draft 14-16 updates](/quest/m1/ietf-legacy-updates.md) - Rust and JS apply and answer moq-transport 14-16 request updates without ending or leaking the request - [Tail arrivals](/quest/m1/tail-arrivals.md) - a track judges its pending tail from recorded arrivals, not a cache scan, and errors a reader whose tail ends short diff --git a/quest/m1/gst-src-av-sync.md b/quest/m1/gst-src-av-sync.md deleted file mode 100644 index a739984957a..00000000000 --- a/quest/m1/gst-src-av-sync.md +++ /dev/null @@ -1,60 +0,0 @@ -# [M] moqsrc keeps audio and video aligned - -## Goal - -All of a `moqsrc` run's pads share one timestamp reference and one segment -base, so audio and video keep the relative timing the broadcast gives them, -at the first start and after every switch. Today each pump starts its PTS at -its own first frame (`reference_ts` in `rs/moq-gst/src/source/imp.rs`) and -bases the segment it pushes with its first buffer on the running time that -buffer arrived, so two tracks line up only as well as their first frames' -arrivals happen to match their timestamps. A video track joining at a -keyframe and an audio track joining at its latest group start apart. - -Non-goals: aligning to wall time across hosts, which -[#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) -settles on the sink side; jitter buffering. - -## Plan - -Decided 2026-10-10, planning the follow-ups of moqsrc's restart work -([#5181](https://github.com/moq-dev/moq/pull/5181), superseded by -[#5191](https://github.com/moq-dev/moq/pull/5191)): the tracks of one -broadcast share one timeline, which the catalog's one `clock` maps to wall -time, so equal times on different tracks present together. They do not -share one scale: CMAF keeps each track's `mdhd` timescale, and LOC can carry -a per-frame one (`moq_mux::container::Frame::timestamp`). `moqsink` already -publishes aligned tracks (`two_pads_keep_av_aligned_through_real_segments`); -only `moqsrc` loses the alignment. - -Decided 2026-10-11 by the maintainer, in #5189: - -- Per run, the first frame any pump reads fixes the run's reference - timestamp and segment base, and every pad of the run maps through both. - Mapping through the catalog's wall `clock` was rejected: the clock is - optional, assumes synchronized hosts, and only means something for - `moqsink` once #3021 lands. -- Compare times converted to one scale, such as nanoseconds: - `moq_net::Timestamp::checked_sub` refuses mismatched scales, and - `relative_pts` turns that refusal into zero. -- A frame before the run's reference must land before the segment start, - where the segment clips it. Today `relative_pts` clamps it to zero instead. - For example, use the converted time as the PTS and the reference as the - segment start. -- With one base, a pump whose first frame arrives late in wall time but - early in media time reaches a clock-synced sink late. That is synced - playback working as intended, not something to compensate for here. -- m1, ranked after #3021. -- Docs stay inline: the implementing PR updates the `moqsrc` paragraph of - `doc/bin/gstreamer.md`. No separate docs quest. - -Test: an audio and a video frame with the same broadcast time reach -`moqsrc`'s two pads at the same running time, at the first start and after a -restart, whichever pump reads first and whichever track's first frame is -earlier. A frame before the run's reference reaches its pad with a PTS before -the segment start, for downstream to clip. One case is a CMAF broadcast whose -tracks use different timescales. - -## Related - -- [#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) - the wall epoch on the sink side diff --git a/rs/moq-gst/src/source/imp.rs b/rs/moq-gst/src/source/imp.rs index 19c9b657c11..961ce206ae9 100644 --- a/rs/moq-gst/src/source/imp.rs +++ b/rs/moq-gst/src/source/imp.rs @@ -1,6 +1,6 @@ use std::collections::{HashMap, HashSet}; use std::sync::atomic::{AtomicU8, AtomicU64, Ordering}; -use std::sync::{Arc, LazyLock, Mutex}; +use std::sync::{Arc, LazyLock, Mutex, OnceLock}; use std::time::Duration; use anyhow::{Context, Result, bail}; @@ -724,6 +724,10 @@ async fn follow_catalog( // (the browser via @moq/hang) that announce an empty catalog before their encoder // configures, then add renditions a beat later, as well as renditions appearing, // disappearing, or changing codec/resolution mid-stream. + let playback = Playback { + pads: pads.clone(), + timeline: Arc::default(), + }; let mut active: HashMap = HashMap::new(); let mut pumps: tokio::task::JoinSet> = tokio::task::JoinSet::new(); // The renditions the latest catalog lists. @@ -761,7 +765,7 @@ async fn follow_catalog( next = catalog_consumer.next(), if !catalog_closed => { match next { Ok(Some(catalog)) => { - listed = reconcile(&catalog, &mut active, &mut pumps, &source, pads, &element, shutdown); + listed = reconcile(&catalog, &mut active, &mut pumps, &source, &playback, &element, shutdown); } // Catalog track closed. Don't cancel the pumps: let each reach its natural end, // and just stop reconciling while they drain. @@ -813,10 +817,11 @@ fn reconcile( active: &mut HashMap, pumps: &mut tokio::task::JoinSet>, source: &moq_mux::Source, - pads: &Pads, + playback: &Playback, element: &glib::WeakRef, shutdown: &watch::Receiver, ) -> HashSet { + let pads = &playback.pads; struct Desired { kind: TrackKind, shape: Shape, @@ -956,6 +961,7 @@ fn reconcile( track: d.shape.track.clone(), container, pads: pads.clone(), + timeline: playback.timeline.clone(), state: state.clone(), cancel: cancel_rx, shutdown: shutdown.clone(), @@ -1006,6 +1012,28 @@ fn plan_reconcile(desired: &HashMap, active: &HashMap, +} + +/// The first decoded frame fixes one mapping for every rendition of this broadcast run. +#[derive(Default)] +struct Timeline(OnceLock>); + +impl Timeline { + fn segment(&self, pts: gst::ClockTime, base: gst::ClockTime) -> &gst::FormattedSegment { + self.0.get_or_init(|| { + let mut segment = gst::FormattedSegment::new(); + segment.set_start(pts); + segment.set_position(pts); + segment.set_base(base); + segment + }) + } +} + /// One rendition's pump: everything [`reconcile`] hands a task it spawns. struct Pump { element: glib::WeakRef, @@ -1018,6 +1046,7 @@ struct Pump { track: String, container: moq_mux::catalog::hang::Container, pads: Pads, + timeline: Arc, /// Shared with this rendition's [`ActiveTrack::state`]. state: Arc, cancel: watch::Receiver, @@ -1040,6 +1069,7 @@ impl Pump { track, container, pads, + timeline, state, mut cancel, shutdown, @@ -1102,7 +1132,7 @@ impl Pump { let _ = pad.set_active(false); } - let mut reference_ts = None; + let mut sent_segment = false; loop { tokio::select! { // This rendition is being torn down (its run ended, or a catalog update replaced it). @@ -1110,19 +1140,20 @@ impl Pump { frame = track.read() => match frame { Ok(Some(moq_mux::container::Read::Gap { .. } | moq_mux::container::Read::End { .. })) => {} Ok(Some(moq_mux::container::Read::Frame { frame, .. })) => { - // PTS restarts at zero with every pump, so the segment places that zero at the - // current running time, or a synced sink would drop a restarted run as late. - let segment = reference_ts.is_none().then(|| { - let mut segment = gst::FormattedSegment::::new(); - segment.set_base( - element - .upgrade() - .and_then(|obj| obj.current_running_time()) - .unwrap_or(gst::ClockTime::ZERO), - ); - gst::event::Segment::new(&segment) + let buffer = match build_buffer(frame, kind) { + Ok(buffer) => buffer, + Err(err) => { + if let Some(obj) = element.upgrade() { + gst::element_error!(obj, gst::StreamError::Format, ("track {name} has an invalid timestamp"), ["{err:#}"]); + } + return Some((kind, name)); + } + }; + let segment = (!sent_segment).then(|| { + let base = element.upgrade().and_then(|obj| obj.current_running_time()).unwrap_or(gst::ClockTime::ZERO); + sent_segment = true; + gst::event::Segment::new(timeline.segment(buffer.pts().unwrap(), base)) }); - let buffer = build_buffer(frame, &mut reference_ts, kind); // pad.push() blocks until downstream accepts the buffer (full queues, a // clock-synced sink). block_in_place hands our sibling tasks to another // worker so a stalled downstream can't pin a runtime thread and starve @@ -1156,7 +1187,7 @@ impl Pump { } /// Start a new stream on the pad: a fresh stream-start and the rendition's caps. The segment -/// waits for the first buffer, which fixes its running time. +/// waits for this pad's first buffer, then uses the run's shared mapping. fn begin(pad: &gst::Pad, name: &str, caps: &gst::Caps) { pad.push_event( gst::event::StreamStart::builder(name) @@ -1166,22 +1197,13 @@ fn begin(pad: &gst::Pad, name: &str, caps: &gst::Caps) { pad.push_event(gst::event::Caps::new(caps)); } -/// Wrap a decoded frame in a gst buffer, assigning a pts relative to the track's first frame. -fn build_buffer( - frame: moq_mux::container::Frame, - reference_ts: &mut Option, - kind: TrackKind, -) -> gst::Buffer { +/// Preserve the broadcast timestamp at the common nanosecond scale used by GStreamer. +fn build_buffer(frame: moq_mux::container::Frame, kind: TrackKind) -> Result { let mut buffer = gst::Buffer::from_slice(frame.payload); let buffer_mut = buffer.get_mut().unwrap(); - let pts = match *reference_ts { - Some(reference) => relative_pts(frame.timestamp, reference), - None => { - *reference_ts = Some(frame.timestamp); - gst::ClockTime::ZERO - } - }; + let pts = + gst::ClockTime::try_from(Duration::from(frame.timestamp)).context("timestamp exceeds GStreamer clock range")?; buffer_mut.set_pts(Some(pts)); let mut flags = buffer_mut.flags(); @@ -1193,19 +1215,7 @@ fn build_buffer( } buffer_mut.set_flags(flags); - buffer -} - -/// PTS of `timestamp` relative to the track's first frame (`reference`). -/// -/// Frames arrive in decode order, so a B-frame's presentation timestamp can fall before -/// the reference. `Timestamp` subtraction panics on underflow, so clamp to zero rather -/// than crash the pump (which would leak its pad). -fn relative_pts(timestamp: moq_net::Timestamp, reference: moq_net::Timestamp) -> gst::ClockTime { - match timestamp.checked_sub(reference) { - Ok(delta) => gst::ClockTime::from_nseconds(Duration::from(delta).as_nanos() as u64), - Err(_) => gst::ClockTime::ZERO, - } + Ok(buffer) } fn video_caps(config: &hang::catalog::VideoConfig) -> Result { @@ -1294,7 +1304,7 @@ fn audio_caps(config: &hang::catalog::AudioConfig) -> Result { #[cfg(test)] mod tests { - use super::{PumpState, plan_reconcile, relative_pts}; + use super::{PumpState, Timeline, plan_reconcile}; use moq_net::Timestamp; use std::collections::HashMap; @@ -1369,24 +1379,42 @@ mod tests { } #[test] - fn relative_pts_clamps_backwards_timestamps() { - let reference = Timestamp::from_millis(2000).unwrap(); - - // A frame presenting before the reference (a decode-order B-frame) must clamp to - // zero, not underflow and panic. + fn the_first_frame_fixes_the_segment_start_and_base() { + gst::init().unwrap(); + let timeline = Timeline::default(); + let start = gst::ClockTime::from_seconds(10); + let base = gst::ClockTime::from_seconds(3); + let first = timeline.segment(start, base); + assert_eq!(first.start(), Some(start)); + assert_eq!(first.position(), Some(start)); + assert_eq!(first.base(), Some(base)); assert_eq!( - relative_pts(Timestamp::from_millis(1000).unwrap(), reference), - gst::ClockTime::ZERO + timeline.segment(gst::ClockTime::from_seconds(9), gst::ClockTime::from_seconds(8)), + first ); - assert_eq!(relative_pts(reference, reference), gst::ClockTime::ZERO); - - // A forward timestamp yields the delta. assert_eq!( - relative_pts(Timestamp::from_millis(2500).unwrap(), reference), - gst::ClockTime::from_mseconds(500) + first.to_running_time(gst::ClockTime::from_seconds(12)), + Some(gst::ClockTime::from_seconds(5)) + ); + assert_eq!(first.to_running_time(gst::ClockTime::from_seconds(9)), None); + assert_eq!( + Timeline::default().segment(start, gst::ClockTime::ZERO).base(), + Some(gst::ClockTime::ZERO) ); } + #[test] + fn an_unrepresentable_timestamp_is_refused() { + gst::init().unwrap(); + let frame = moq_mux::container::Frame { + timestamp: Timestamp::from_scale(1 << 60, 1).unwrap(), + payload: bytes::Bytes::new(), + keyframe: true, + duration: None, + }; + assert!(super::build_buffer(frame, super::TrackKind::Video).is_err()); + } + /// The pad claim and the teardown race for every pump, and only one may win: a subscription /// resolving after a teardown must not publish a rendition the session has finished with and /// then yank it without an EOS. The cancel watch alone cannot say which happened, so this is @@ -1472,6 +1500,137 @@ mod session_tests { config } + /// Real wire consumers and source pads, with each write acknowledged by a pad probe. + /// No arrival-time guesses: the first acknowledgement fixes which pump won the run. + #[test] + fn audio_and_video_share_a_run_timeline_across_scales_and_restarts() { + let _pad_ids = pad_ids(); + let element = element(); + let pads = Pads::default(); + let (tx, rx) = std::sync::mpsc::channel(); + element.connect_pad_added(move |_, pad| { + let name = pad.name().to_string(); + let tx = tx.clone(); + let segment = Mutex::new(gst::FormattedSegment::::new()); + pad.add_probe( + gst::PadProbeType::BUFFER | gst::PadProbeType::EVENT_DOWNSTREAM, + move |_, info| { + if let Some(event) = info.event() + && let gst::EventView::Segment(event) = event.view() + { + *segment.lock().unwrap() = event.segment().downcast_ref::().unwrap().clone(); + } + if let Some(buffer) = info.buffer() { + tx.send((name.clone(), buffer.pts().unwrap(), segment.lock().unwrap().clone())) + .unwrap(); + return gst::PadProbeReturn::Drop; + } + gst::PadProbeReturn::Ok + }, + ); + }); + + for cmaf in [false, true] { + for audio_first in [false, true] { + for other_earlier in [false, true] { + // A second broadcast uses the held pads but must choose a new reference. + for reference in [10, 100] { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let mut catalog = moq_mux::catalog::Producer::new(&mut broadcast, Default::default()).unwrap(); + let mut video = VideoConfig::new(VideoCodec::VP8); + video.coded_width = Some(16); + video.coded_height = Some(16); + let mut audio = audio_rendition(); + let scales = if cmaf { [90_000, 48_000] } else { [1_000_000; 2] }; + if cmaf { + use moq_mux::container::fmp4::Muxer; + video.container = Container::Cmaf { + init: Muxer::video(&video) + .unwrap() + .with_timescale(moq_net::Timescale::new(scales[0]).unwrap()) + .unwrap() + .init() + .unwrap() + .unwrap(), + }; + audio.container = Container::Cmaf { + init: Muxer::audio(&audio).unwrap().init().unwrap().unwrap(), + }; + } + let containers = [ + (&video.container, moq_mux::container::Kind::Video), + (&audio.container, moq_mux::container::Kind::Audio), + ]; + let mut tracks: Vec<_> = ["video", "audio"] + .into_iter() + .zip(containers) + .zip(scales) + .map(|((name, (container, kind)), scale)| { + let track = broadcast + .create_track( + name, + hang::container::track_info(0) + .with_timescale(moq_net::Timescale::new(scale).unwrap()), + ) + .unwrap(); + moq_mux::container::Producer::new( + track, + moq_mux::catalog::hang::Container::new(container, kind).unwrap(), + ) + }) + .collect(); + { + let mut guard = catalog.modify().unwrap(); + guard.video.renditions.insert("video".into(), video); + guard.audio.renditions.insert("audio".into(), audio); + } + let (_shutdown, shutdown) = watch::channel(false); + let (cancel, mut cancelled) = watch::channel(false); + let weak = element.downgrade(); + let pads = pads.clone(); + let source = moq_mux::Source::new(origin().consume(), broadcast.consume()); + let task = super::RUNTIME + .spawn(async move { follow_catalog(source, &pads, weak, &shutdown, &mut cancelled).await }); + let first = usize::from(audio_first); + let other = 1 - first; + let mut write = |index: usize, seconds: u64| { + tracks[index] + .write(moq_mux::container::Frame { + timestamp: moq_net::Timestamp::from_scale(seconds * scales[index], scales[index]) + .unwrap(), + payload: bytes::Bytes::from_static(&[0xf8]), + keyframe: true, + duration: Some( + moq_net::Timestamp::from_scale(scales[index] / 50, scales[index]).unwrap(), + ), + }) + .unwrap(); + rx.recv_timeout(Duration::from_secs(10)) + .expect("pump did not deliver frame") + }; + let (name, pts, segment) = write(first, reference); + assert!(name.starts_with(if audio_first { "audio" } else { "video" })); + assert_eq!(pts, gst::ClockTime::from_seconds(reference)); + assert_eq!(segment.start(), Some(pts)); + let other_time = if other_earlier { reference - 1 } else { reference + 1 }; + let (_, other_pts, other_segment) = write(other, other_time); + assert_eq!(other_pts, gst::ClockTime::from_seconds(other_time)); + assert_eq!(other_segment, segment, "pads chose different run timing"); + if other_earlier { + assert!(other_pts < other_segment.start().unwrap()); + assert_eq!(other_segment.to_running_time(other_pts), None); + } + let (_, a, a_segment) = write(0, reference + 2); + let (_, b, b_segment) = write(1, reference + 2); + assert_eq!(a_segment.to_running_time(a), b_segment.to_running_time(b)); + cancel.send(true).unwrap(); + super::RUNTIME.block_on(task).unwrap().unwrap(); + } + } + } + } + } + /// The element's pads of one kind. Matched by prefix because the `%u` suffix comes from a /// process-global counter, so a pad's number depends on what else the test binary has run. fn pads(element: &super::super::MoqSrc, kind: &str) -> Vec { @@ -2298,6 +2457,7 @@ mod session_tests { track: "video".into(), container: moq_mux::catalog::hang::Container::Legacy(moq_mux::container::Kind::Video), pads: Pads::default(), + timeline: std::sync::Arc::default(), state: std::sync::Arc::new(super::PumpState::new()), cancel, shutdown,