diff --git a/doc/bin/gstreamer.md b/doc/bin/gstreamer.md index 92a8482c3db..05d95014bf7 100644 --- a/doc/bin/gstreamer.md +++ b/doc/bin/gstreamer.md @@ -69,13 +69,19 @@ reports `failed` on an answer redialing can't change, such as a rejected token. Each run from `READY` publishes under a fresh [publisher epoch](/concept/moq-lite#publisher-epochs), so on moq-lite 07 (opt-in) viewers switch to a restarted pipeline at once instead of stalling on the old -one; reconnects within a run keep it. The first catalog waits for every requested -pad to negotiate its rendition, including any codec config carried in its first -buffers. A linked pad that never negotiates holds back the whole catalog until +one; reconnects within a run keep it. The first catalog waits for a timed buffer +to place the shared wall clock and for every requested pad to negotiate its +rendition, including any codec config carried in its first buffers. A linked pad +that never negotiates holds back the whole catalog until it is released, reaches EOS, or fails; a flush or new stream after that EOS holds it back again. Opaque data pads are not listed. Later catalog updates do not wait for newly requested pads. +All pads share one media timeline on one wall clock. The first timed buffer places +it using UNIX or NTP reference timestamp metadata; other clock domains use the +pipeline running time and local wall clock. Delayed buffers keep their media +time. Pauses and flushes keep this mapping; starting a new run chooses a new one. + `connection-stats` is null while disconnected. While connected it is a structure of the transport counters the active backend has (RTT, send and receive estimates, bytes, packets, loss). Missing ones are omitted rather than @@ -96,8 +102,8 @@ default, for file, demuxed, and network media: their arrival reflects the disk or the network, not the original encoder, and a GStreamer segment cannot tell the two apart. Text and opaque pads refuse it. -After a pause, flush, or changed TIME segment, the next media buffer starts a new -timeline epoch and resets the handoff baseline. The pause does not inflate +After a pause, flush, or changed TIME segment, the next media buffer resets +the handoff measurement baseline. The pause does not inflate advertised jitter, and previously measured maxima remain. Resumed timestamps must continue forward on the broadcast media clock. diff --git a/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md b/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md deleted file mode 100644 index 9798c8c593b..00000000000 --- a/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md +++ /dev/null @@ -1,35 +0,0 @@ -# [S] GStreamer picks the broadcast wall epoch - -## Goal - -`moqsink` exposes one fixed wall epoch for the broadcast through the shared -Hang clock contract, so every pad's timestamps relate to UTC the same way. - -## Plan - -The PTS mapping already exists: `rs/moq-gst/src/sink/pad.rs` maps each -buffer PTS through its TIME segment into the shared running-time domain -(`rs/moq-gst/src/sink/timeline.rs`), and #4480 made rewinds drop frames. Only -the wall-epoch choice remains. - -Choose the wall epoch once per broadcast. Prefer GstReferenceTimestampMeta -only when it names a recognized absolute clock domain; otherwise relate the -pipeline clock, base time, running time, and local SystemTime. An unidentified -reference clock is not UTC. Every pad uses the same epoch rather than sampling -its own. Do not define a GStreamer-specific catalog shape. Set the epoch -through moq-mux's catalog clock (`Config::with_clock`, or a `Timebase` placed -with `place`): the clock is final once taken or published, and a -`with_clock` catalog shifts every importer onto it. - -Decided in the 2026-09-30 audit: a restart is a new broadcast epoch, not a -forward re-anchor on the old clock (per remove-live; `moqsink` mints a -publisher epoch per run), and -[#3115](/quest/m2/3115-moqsink-the-publication-has-no-generation-so-a-flush.md) -handles the sink side. - -Test recognized reference metadata, the deterministic local-clock fallback, -delayed first buffers, and multiple pads sharing one epoch. - -## Closes - -- [#3021](https://github.com/moq-dev/moq/issues/3021) - close this issue when the quest finishes diff --git a/quest/m1/README.md b/quest/m1/README.md index c40699aa328..75b3fb3acc9 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -143,7 +143,6 @@ above them. - [T-STD TS export](/quest/m1/tstd/README.md) - `moq export ts` is a proper remux that passes the supported T-STD buffer models, starting with a fixed `--delay` - [Open-GOP leading pictures](/quest/m1/open-gop-leading-pictures.md) - a viewer joining at a recovery point drops the leading pictures it cannot decode; continuous viewers keep them - [Audio warmup](/quest/m1/audio-warmup.md) - a viewer joining an Opus rendition mid-stream never hears the unconverged first 80 ms -- [#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) - moq-gst picks the broadcast wall epoch; a restarted source is a new epoch, not a forward re-anchor - [CLI stdin shutdown on release](/quest/m1/cli-stdin-release.md) - backport only the runtime-teardown fix so a stopped CLI exits while its stdin writer stays open - [Draft 14-16 updates](/quest/m1/ietf-legacy-updates.md) - Rust and JS apply and answer moq-transport 14-16 request updates without ending or leaking the request - [Tail arrivals](/quest/m1/tail-arrivals.md) - a track judges its pending tail from recorded arrivals, not a cache scan, and errors a reader whose tail ends short diff --git a/rs/moq-gst/src/sink/imp.rs b/rs/moq-gst/src/sink/imp.rs index 96ee79a806a..d697bc455c2 100644 --- a/rs/moq-gst/src/sink/imp.rs +++ b/rs/moq-gst/src/sink/imp.rs @@ -63,6 +63,7 @@ struct State { session: Session, broadcast: moq_net::broadcast::Producer, catalog: Option, + wall: std::sync::Arc, /// Whether this publication's single EOS message already went out. One per publication, not per /// entry to PLAYING: a finished publication is terminal here, so there is nothing to announce twice. eos_delivered: bool, @@ -442,8 +443,10 @@ impl ElementImpl for MoqSink { gst::warning!(CAT, "refusing a pad: the publication has ended"); return None; } - if let Some(catalog) = control.live.as_ref().and_then(|state| state.catalog.as_ref()) { - pad.lifecycle().media.reserve(catalog); + if let Some(state) = control.live.as_ref() + && let Some(catalog) = state.catalog.as_ref() + { + pad.lifecycle().media.reserve(catalog, state.wall.clone()); } control.admissions.insert(pad.clone()); } @@ -580,16 +583,20 @@ impl MoqSink { gst::StateChangeError })?; let completion = session.completion(); + let wall = super::wall::Wall::new(&catalog); let updates = { let mut control = self.control.lock().unwrap(); control.live = Some(State { session, broadcast, catalog: Some(catalog), + wall, eos_delivered: false, }); let pads = self.pads_locked(&control); - let catalog = control.live.as_ref().unwrap().catalog.as_ref().unwrap(); + let state = control.live.as_ref().unwrap(); + let catalog = state.catalog.as_ref().unwrap(); + let wall = &state.wall; pads.into_iter() .map(|pad| { let changes = { @@ -597,7 +604,7 @@ impl MoqSink { let changes = lifecycle.reset(); lifecycle.completion = Some(completion.clone()); if !lifecycle.releasing { - lifecycle.media.reserve(catalog); + lifecycle.media.reserve(catalog, wall.clone()); } changes }; @@ -654,6 +661,7 @@ impl MoqSink { ], ); let _rt = RUNTIME.enter(); + state.wall.finish(); if let Some(mut catalog) = state.catalog.take() && let Err(err) = catalog.finish().context("finalize catalog") && failure.is_none() @@ -681,6 +689,16 @@ impl MoqSink { // Only subtitles use this: a cue needs an explicit end, unlike a media frame. let duration = buffer.duration(); let current_running_time = self.obj().current_running_time(); + // Preroll has a frozen running time before the pipeline starts its clock. + let reference_running_time = current_running_time.or_else(|| { + (self.obj().current_state() != gst::State::Playing) + .then(|| self.obj().start_time()) + .flatten() + }); + let sample = super::wall::Sample { + running: reference_running_time, + wall: std::time::SystemTime::now(), + }; let map = buffer.map_readable().map_err(|_| { gst::error!(CAT, "failed to map buffer on pad {}", pad.name()); gst::FlowError::Error @@ -707,9 +725,12 @@ impl MoqSink { if lifecycle.media.is_failed() { return Ok(gst::FlowSuccess::Ok); } - let outcome = lifecycle - .media - .push_buffer(data, pts, duration, current_running_time, Instant::now()); + let outcome = lifecycle.media.place_wall(&buffer, sample).and_then(|()| { + lifecycle + .media + .push_buffer(data, pts, duration, current_running_time, Instant::now()) + .map_err(anyhow::Error::msg) + }); let changes = match &outcome { Ok(PushOutcome::Failed(reason)) => Some(lifecycle.fail(reason.clone())), _ => None, @@ -867,7 +888,7 @@ impl MoqSink { if !lifecycle.releasing { lifecycle.ended = false; if let Some(catalog) = &state.catalog { - lifecycle.media.reserve(catalog); + lifecycle.media.reserve(catalog, state.wall.clone()); } } } @@ -893,7 +914,7 @@ impl MoqSink { lifecycle.media.flush(); let catalog = state.catalog.as_ref(); if let Some(catalog) = catalog { - lifecycle.media.reserve(catalog); + lifecycle.media.reserve(catalog, state.wall.clone()); } let caps = lifecycle.media.resumable_caps(); caps.zip(catalog).and_then(|(caps, catalog)| { @@ -1010,6 +1031,7 @@ impl MoqSink { updates.push(PadUpdate { pad, changes }); } } + state.wall.finish(); if let Some(mut catalog) = state.catalog.take() && let Err(err) = catalog.finish().context("finalize catalog") && failure.is_none() @@ -1131,6 +1153,7 @@ fn observe_caps( lifecycle: &mut PadLifecycle, caps: &gst::Caps, ) -> (Option, bool) { + lifecycle.media.reserve(catalog, state.wall.clone()); let requested = lifecycle.requested().map(str::to_owned); let mut options = ProducerOptions::new(caps) .with_container(lifecycle.container().into()) @@ -1178,6 +1201,26 @@ mod tests { } fn start(&self) -> moq_mux::catalog::Consumer { + let consumer = self.start_unplaced(); + let control = self.0.imp().control.lock().unwrap(); + control + .live + .as_ref() + .unwrap() + .wall + .place( + &gst::Buffer::new(), + Duration::ZERO, + super::super::wall::Sample { + running: Some(gst::ClockTime::ZERO), + wall: std::time::SystemTime::UNIX_EPOCH + Duration::from_secs(1_735_689_600), + }, + ) + .unwrap(); + consumer + } + + fn start_unplaced(&self) -> moq_mux::catalog::Consumer { self.0.set_state(gst::State::Paused).unwrap(); let broadcast = self .0 @@ -1237,6 +1280,94 @@ mod tests { catalog.poll_next(&moq_net::kio::Waiter::noop()) } + #[test] + fn reference_metadata_places_the_broadcast_wall_clock() { + let run = CatalogRun::new(); + let pad = run.pad("audio"); + let mut catalog = run.start_unplaced(); + assert!(negotiate(&pad, &opus_caps())); + assert!( + next_catalog(&mut catalog).is_pending(), + "CAPS must wait for the wall clock" + ); + assert!(pad.send_event(gst::event::Segment::new(&gst::FormattedSegment::::new()))); + let mut buffer = gst::Buffer::from_slice([0xf8, 0xff, 0xfe]); + let buffer_mut = buffer.get_mut().unwrap(); + buffer_mut.set_pts(gst::ClockTime::from_seconds(3)); + gst::ReferenceTimestampMeta::add( + buffer_mut, + &gst::Caps::builder("timestamp/x-unix").build(), + gst::ClockTime::from_seconds(1_735_689_603), + None, + ); + assert_eq!(pad.chain(buffer), Ok(gst::FlowSuccess::Ok)); + let std::task::Poll::Ready(Ok(Some(first))) = next_catalog(&mut catalog) else { + panic!("first catalog") + }; + assert_eq!( + first.clock.unwrap().wall.as_micros(), + ((1_735_689_600 - hang::catalog::MOQ_EPOCH_UNIX_MILLIS / 1000) * 1_000_000) as u128, + ); + } + + #[test] + fn pads_keep_one_wall_clock_and_verbatim_running_times_each_run() { + let run = CatalogRun::new(); + let video = run.pad("video"); + let audio = run.pad("audio"); + for zero in [1_735_689_600u64, 1_735_689_660] { + let _catalog = run.start_unplaced(); + assert!(negotiate(&video, &h264_caps())); + assert!(negotiate(&audio, &opus_caps())); + for (pad, origin, pts, payload, reference) in [ + (&video, 10, 13, super::super::pad::h264_keyframe_au(), zero + 3), + (&audio, 100, 105, Bytes::from_static(&[0xf8, 0xff, 0xfe]), zero + 500), + ] { + let mut segment = gst::FormattedSegment::::new(); + segment.set_start(gst::ClockTime::from_seconds(origin)); + assert!(pad.send_event(gst::event::Segment::new(&segment))); + let mut buffer = gst::Buffer::from_slice(payload); + let b = buffer.get_mut().unwrap(); + b.set_pts(gst::ClockTime::from_seconds(pts)); + gst::ReferenceTimestampMeta::add( + b, + &gst::Caps::builder("timestamp/x-unix").build(), + gst::ClockTime::from_seconds(reference), + None, + ); + assert_eq!(pad.chain(buffer), Ok(gst::FlowSuccess::Ok)); + } + let (broadcast, clock) = { + let control = run.0.imp().control.lock().unwrap(); + let state = control.live.as_ref().unwrap(); + ( + state.broadcast.consume(), + state.catalog.as_ref().unwrap().snapshot().clock.unwrap(), + ) + }; + assert_eq!( + clock.wall.as_micros(), + ((zero - hang::catalog::MOQ_EPOCH_UNIX_MILLIS / 1000) * 1_000_000) as u128 + ); + for (name, kind, expected) in [ + ("video", moq_mux::container::Kind::Video, 3), + ("audio", moq_mux::container::Kind::Audio, 5), + ] { + let subscriber = futures::executor::block_on(broadcast.track(name).unwrap().subscribe(None)).unwrap(); + let mut consumer = + moq_mux::container::Consumer::new(subscriber, moq_mux::catalog::hang::Container::Legacy(kind)); + let std::task::Poll::Ready(Ok(Some(read))) = consumer.poll_read(&moq_net::kio::Waiter::noop()) else { + panic!("published media frame") + }; + let moq_mux::container::Read::Frame { frame, .. } = read else { + panic!("media frame") + }; + assert_eq!(frame.timestamp.as_micros(), expected * 1_000_000, "{name}"); + } + run.0.set_state(gst::State::Null).unwrap(); + } + } + #[test] fn first_catalog_waits_for_late_audio_caps_each_run() { let run = CatalogRun::new(); diff --git a/rs/moq-gst/src/sink/mod.rs b/rs/moq-gst/src/sink/mod.rs index bd10a8f0c14..24f2f07536a 100644 --- a/rs/moq-gst/src/sink/mod.rs +++ b/rs/moq-gst/src/sink/mod.rs @@ -6,6 +6,7 @@ mod pad; mod request_pad; mod session; mod timeline; +mod wall; #[cfg(test)] pub(crate) use pad::h264_keyframe_au; diff --git a/rs/moq-gst/src/sink/pad.rs b/rs/moq-gst/src/sink/pad.rs index 09a096effc0..b15dff8cd2a 100644 --- a/rs/moq-gst/src/sink/pad.rs +++ b/rs/moq-gst/src/sink/pad.rs @@ -250,6 +250,7 @@ pub enum PushOutcome { pub struct Pad { /// The requested pad gates the first catalog even before CAPS can name its rendition. reserved: Option, + wall: Option>, track: Option, caps: Option, /// Set once a producer build rejects this pad's caps or bitstream; further buffers are dropped and @@ -270,6 +271,7 @@ impl Pad { pub fn new() -> Self { Self { reserved: None, + wall: None, track: None, caps: None, failed: false, @@ -281,7 +283,8 @@ impl Pad { } /// Fill a run's slot once, including a pad visible before its admission confirms. - pub(super) fn reserve(&mut self, catalog: &moq_mux::catalog::Producer) { + pub(super) fn reserve(&mut self, catalog: &moq_mux::catalog::Producer, wall: std::sync::Arc) { + self.wall = Some(wall); if self.reserved.is_none() && self.track.is_none() && !self.failed { self.reserved = Some(catalog.reserve()); } @@ -717,6 +720,29 @@ impl Pad { } } + /// Place the run on the wall clock before its first valid timestamp can reach an importer. + pub(super) fn place_wall(&self, buffer: &gst::BufferRef, sample: super::wall::Sample) -> Result<()> { + let Some(wall) = &self.wall else { return Ok(()) }; + if self.failed || self.track.is_none() { + return Ok(()); + } + let timestamp = if matches!(self.track, Some(Sink::Opaque(_))) + && buffer.pts().is_none() + && self.state == PadState::Active + { + sample + .running + .map(|time| time.useconds()) + .ok_or("no current running time") + } else { + self.frame_timestamp(buffer.pts()) + }; + if let Ok(micros) = timestamp { + wall.place(buffer, Duration::from_micros(micros), sample)?; + } + Ok(()) + } + /// Import one buffer into the producer. A failed or producer-less pad drops the buffer; a timeline /// drop is logged. Unstamped opaque data on an active timeline uses the element's current running /// time. An encoder pad records `now` as the frame's handoff once it is published. A bad bitstream diff --git a/rs/moq-gst/src/sink/wall.rs b/rs/moq-gst/src/sink/wall.rs new file mode 100644 index 00000000000..38bf525b771 --- /dev/null +++ b/rs/moq-gst/src/sink/wall.rs @@ -0,0 +1,156 @@ +//! The wall-clock time of running-time zero, shared by every pad in a publication. + +use std::sync::{ + Arc, Mutex, + atomic::{AtomicBool, Ordering}, +}; +use std::time::{Duration, SystemTime}; + +use anyhow::{Context, Result}; + +/// The pipeline running time and local wall time, read together. +#[derive(Clone, Copy)] +pub(super) struct Sample { + pub running: Option, + pub wall: SystemTime, +} + +/// Places running-time zero on the wall clock once per run, holding the first catalog until then. +pub(super) struct Wall { + timebase: moq_mux::catalog::Timebase, + // CAPS can resolve every rendition before the first buffer supplies its reference timestamp. + pending: Mutex>, + placed: AtomicBool, +} + +impl Wall { + pub(super) fn new(catalog: &moq_mux::catalog::Producer) -> Arc { + let timebase = catalog.timebase(); + Arc::new(Self { + pending: Mutex::new(Some(timebase.reserve())), + timebase, + placed: AtomicBool::new(false), + }) + } + + pub(super) fn place(&self, buffer: &gst::BufferRef, pts: Duration, sample: Sample) -> Result<()> { + if self.placed.load(Ordering::Acquire) { + return Ok(()); + } + let mut pending = self.pending.lock().unwrap(); + if pending.is_none() { + return Ok(()); + } + let wall = wall_zero(buffer, pts, sample)?; + self.timebase.place(hang::container::Timestamp::from_micros(0)?, wall)?; + self.placed.store(true, Ordering::Release); + pending.take(); + Ok(()) + } + + /// A run without buffers still closes its catalog, without leaving a reservation outstanding. + pub(super) fn finish(&self) { + self.pending.lock().unwrap().take(); + } +} + +fn wall_zero(buffer: &gst::BufferRef, pts: Duration, sample: Sample) -> Result { + for meta in buffer.iter_meta::() { + let Some(reference) = meta.reference().structure(0) else { + continue; + }; + let timestamp = Duration::from_nanos(meta.timestamp().nseconds()); + let unix = match reference.name().as_str() { + "timestamp/x-unix" => timestamp, + "timestamp/x-ntp" => timestamp + .checked_sub(Duration::from_secs(2_208_988_800)) + .context("NTP reference timestamp is before the UNIX epoch")?, + // PTP and device clocks need an explicit UTC relation; their names do not provide one. + _ => continue, + }; + return SystemTime::UNIX_EPOCH + .checked_add(unix) + .and_then(|wall| wall.checked_sub(pts)) + .context("reference timestamp cannot place running-time zero"); + } + // Use the pipeline's current running time, not this buffer's PTS: delayed first buffers + // must retain their capture time instead of becoming live on arrival. + sample + .wall + .checked_sub(Duration::from_nanos( + sample + .running + .context("pipeline has no running-time reference")? + .nseconds(), + )) + .context("pipeline running time cannot place wall-clock zero") +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn delayed_buffers_and_unknown_domains_use_the_pipeline_clock() { + gst::init().unwrap(); + let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_735_689_620); + let mut buffer = gst::Buffer::new(); + gst::ReferenceTimestampMeta::add( + buffer.get_mut().unwrap(), + &gst::Caps::builder("timestamp/x-ptp").build(), + gst::ClockTime::from_seconds(17), + None, + ); + for pts in [0, 3_000_000, 19_000_000] { + assert_eq!( + wall_zero( + &buffer, + Duration::from_micros(pts), + Sample { + running: Some(gst::ClockTime::from_seconds(20)), + wall: now + } + ) + .unwrap(), + now - Duration::from_secs(20) + ); + } + assert!( + wall_zero( + &buffer, + Duration::ZERO, + Sample { + running: None, + wall: now + } + ) + .is_err() + ); + } + + #[test] + fn unix_and_ntp_references_place_the_same_wall_time() { + gst::init().unwrap(); + for (domain, seconds) in [("timestamp/x-unix", 1_735_689_603), ("timestamp/x-ntp", 3_944_678_403)] { + let mut buffer = gst::Buffer::new(); + gst::ReferenceTimestampMeta::add( + buffer.get_mut().unwrap(), + &gst::Caps::builder(domain).build(), + gst::ClockTime::from_seconds(seconds), + None, + ); + assert_eq!( + wall_zero( + &buffer, + Duration::from_secs(3), + Sample { + running: None, + wall: SystemTime::UNIX_EPOCH + } + ) + .unwrap(), + SystemTime::UNIX_EPOCH + Duration::from_secs(1_735_689_600) + ); + } + } +}