diff --git a/quest/m2/README.md b/quest/m2/README.md index 0f156eb979f..081f6ce4226 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -49,8 +49,7 @@ removal, a rename, or a wire, catalog, or format change that is free until `mpegts`, one strict rule); [declared ends](/quest/m2/time-addressing/consumers.md), back from m3. - **Native media and gateways.** [The video producer](/quest/m2/video-producer.md), - [the hole wait](/quest/m2/delay-hole-wait.md), - [a chained WHEP pull's shutdown](/quest/m2/whep-chain-shutdown.md), and + [the hole wait](/quest/m2/delay-hole-wait.md), and [upstream str0m](https://github.com/moq-dev/moq/pull/8963), which reverses the 2026-10-30 vendoring. - **Bindings.** [The C API as `libmoq`](/quest/m2/libmoq-name.md), reversing 2026-10-25. diff --git a/quest/m2/whep-chain-shutdown.md b/quest/m2/whep-chain-shutdown.md deleted file mode 100644 index d18fdd8964f..00000000000 --- a/quest/m2/whep-chain-shutdown.md +++ /dev/null @@ -1,39 +0,0 @@ -# [S] A WHEP pull chained into an export stops cleanly - -## Goal - -Stopping `moq import webrtc --connect -- export rtmp --connect ` -with SIGTERM exits 0 and logs no warning about its own tracks. The RTMP export -does not fail with `the source's clock runs +59.8 ppm off ours, past the 30 ppm -the output's may follow`. The WHEP ingest's tracks (`0.opus`, `1.avc3`, -`catalog.json`, `catalog`) end with `finish()` or `abort()` instead of logging -`track::Producer dropped without finish() or abort()`. Non-goal: orderly -broadcast ends while the process keeps running, which -[export-close](/quest/m3/export-close.md) owns. - -## Plan - -Found 2026-11-17 while validating -[moq-dev/moq#8997](https://github.com/moq-dev/moq/pull/8997) on the built -binary. Its PR body's Validation section has the three-process loopback setup -(an ffmpeg source through `export webrtc --listen`, the chain, and an RTMP -sink), and the stop shows both symptoms in some runs. The maintainer picked -this follow-up when picking #8997 to merge. - -- **Clock drift on stop.** The error comes from the drift estimate in - `rs/moq-mux/src/jitter.rs` (`Error` in `rs/moq-mux/src/error.rs`), judged over - only the few seconds the chain ran, while the ingest's timeline was still - settling from arrival time onto RTCP sender reports (`IngestClock` in - `rs/moq-webrtc/src/session.rs`). First find out whether the stop causes it - (for example, a timestamp jump while draining) or whether a short run would - trip it anyway. Then fix whichever is at fault: the ingest timeline or an - estimate that lacks enough samples to judge drift. -- **Unfinished tracks.** `Session::run` aborts the sink only when its loop - returns. A SIGTERM drops the pull's future instead, which drops the - `IngestSink` and its bridges unfinished. Finish or abort them on drop, the - way the rtsp import closes its catalog after its select. - -## Related - -- [Container export ends cleanly](/quest/m3/export-close.md) - an export's orderly end when its broadcast closes -- [Interop stops publishers cleanly](/quest/m3/interop-shutdown-warnings.md) - the same unfinished-producer warning in the CLI FETCH round diff --git a/quest/m3/flv-export-arrival.md b/quest/m3/flv-export-arrival.md new file mode 100644 index 00000000000..637b51fcd54 --- /dev/null +++ b/quest/m3/flv-export-arrival.md @@ -0,0 +1,26 @@ +# [XS] An FLV export judges a frame on when it landed + +## Goal + +`container::flv::Export` hands the jitter buffer each frame's real arrival, as the TS and MKV +exports do, so a track the buffer holds back (its reads gated by `Buffer::full`) is not judged as +sent further and further ahead of its decode time. + +## Plan + +Found 2026-11-17 while chasing a spurious `the source's clock runs ... ppm off ours` from +`moq import webrtc --connect -- export rtmp --connect ` (the quest that fixed it +deleted its file). `FlvTrack::empty` is when the source last returned `Pending`, and +`Export::fill` uses it as every frame's `arrived` (`rs/moq-mux/src/container/flv/export.rs`), +ignoring `Event::Frame { arrived }`, which carries the time the frame landed in the cache. A track +whose reads `full` gates never returns `Pending`, so `empty` stops moving, and each frame read +looks older than the last by the time it waited: replaying a recorded trace, one audio track's +`decode - arrived` grew 0.5 s to 1.6 s over 1.5 s. Use `arrived.max(track.empty)` (the MKV export +takes `arrived.max(track.subscribed)`), and add a test that gates one track and checks the +buffer sees no lead growing. + +The maintainer kept this quest (2026-11-17, [on the PR that found it](https://github.com/moq-dev/moq/pull/9032#issuecomment-6962042151)). + +## Related + +- [Container export ends cleanly](/quest/m3/export-close.md) - the same exports' shutdown diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index ad5ea8079e0..43614a20b34 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -4472,6 +4472,23 @@ impl FrontServeRun { } } +/// A front dropped before it ended, as when its origin is torn down, ends the logical tracks it +/// was serving as closed, rather than leave them to warn as producers dropped unfinished. +impl Drop for FrontServeRun { + fn drop(&mut self) { + let close = |logical: &mut Logical| { + if let Logical::Accepted(accepted) = std::mem::replace(logical, Logical::Ended) { + let _ = accepted.abort(Error::Closed); + } + }; + match &mut self.state { + FrontRunState::Start(_) => {} + FrontRunState::Serve(serve) => serve.tracks.values_mut().for_each(|io| close(&mut io.logical)), + FrontRunState::InFlight(in_flight) => in_flight.iter_mut().for_each(|io| close(&mut io.logical)), + } + } +} + impl kio::Task for FrontServeRun { type Output = (); @@ -9140,6 +9157,41 @@ mod tests { assert_eq!(warns, 0, "a clean logical end must not drop an unfinished producer"); } + /// A torn-down origin ends the logical tracks its fronts were serving, rather than warn about + /// producers dropped unfinished. + #[kio::test] + #[cfg(feature = "async")] + async fn teardown_ends_the_logical_tracks() { + let (producer, mut driver) = Producer::new(Config::from(origin(1))); + let consumer = producer.consume(); + let source = producer.route("room", RouteOptions::default()).unwrap(); + let _source_track = source.broadcast().create_track("video", None).unwrap(); + let waiter = kio::Waiter::noop(); + + // The driver runs the front that answers each of these. + let mut request = Box::pin(consumer.request("room", None)); + let front = loop { + driver.poll(kio::clock::now(), &waiter).unwrap(); + if let Some(front) = request.as_mut().now_or_never() { + break front.unwrap(); + } + kio::clock::sim::yield_now().await; + }; + let track = front.track("video").unwrap(); + let mut subscribing = Box::pin(track.subscribe(None)); + let subscriber = loop { + driver.poll(kio::clock::now(), &waiter).unwrap(); + if let Some(subscriber) = subscribing.as_mut().now_or_never() { + break subscriber.unwrap(); + } + kio::clock::sim::yield_now().await; + }; + + let warns = count_drop_warnings("track::Producer dropped without finish", || drop(driver)); + assert_eq!(warns, 0, "a torn-down front must not drop an unfinished track"); + drop(subscriber); + } + #[kio::test] async fn default_config_mints_a_real_hop() { let config = Config::default(); diff --git a/rs/moq-webrtc/src/codec/av1.rs b/rs/moq-webrtc/src/codec/av1.rs index 1730e9b04ea..aa90aa4e369 100644 --- a/rs/moq-webrtc/src/codec/av1.rs +++ b/rs/moq-webrtc/src/codec/av1.rs @@ -32,6 +32,11 @@ impl codec::Bridge for Bridge { Ok(()) } + fn discontinuity(&mut self) -> Result<()> { + self.import.discontinuity()?; + Ok(()) + } + fn finish(self: Box) -> Result { Ok(Box::new(self.import.finish()?)) } diff --git a/rs/moq-webrtc/src/codec/h264.rs b/rs/moq-webrtc/src/codec/h264.rs index 37dad1d1d2b..5c660ee3e6f 100644 --- a/rs/moq-webrtc/src/codec/h264.rs +++ b/rs/moq-webrtc/src/codec/h264.rs @@ -31,6 +31,11 @@ impl codec::Bridge for Bridge { Ok(()) } + fn discontinuity(&mut self) -> Result<()> { + self.import.discontinuity()?; + Ok(()) + } + fn finish(self: Box) -> Result { Ok(Box::new(self.import.finish()?)) } diff --git a/rs/moq-webrtc/src/codec/h265.rs b/rs/moq-webrtc/src/codec/h265.rs index 9beb5dea258..2dd84798786 100644 --- a/rs/moq-webrtc/src/codec/h265.rs +++ b/rs/moq-webrtc/src/codec/h265.rs @@ -33,6 +33,11 @@ impl codec::Bridge for Bridge { Ok(()) } + fn discontinuity(&mut self) -> Result<()> { + self.import.discontinuity()?; + Ok(()) + } + fn finish(self: Box) -> Result { Ok(Box::new(self.import.finish()?)) } diff --git a/rs/moq-webrtc/src/codec/mod.rs b/rs/moq-webrtc/src/codec/mod.rs index 0af85dd4c95..bfa5bf11908 100644 --- a/rs/moq-webrtc/src/codec/mod.rs +++ b/rs/moq-webrtc/src/codec/mod.rs @@ -44,6 +44,10 @@ pub struct Frame { pub trait Bridge: Send { fn push(&mut self, frame: Frame) -> Result<()>; + /// Mark a break in the published track's timeline before the next frame, which opens a group, + /// as the session's clock stepped. + fn discontinuity(&mut self) -> Result<()>; + /// Finish the published track after its final frame, returning what keeps serving it while /// the finished broadcast is held. fn finish(self: Box) -> Result; @@ -66,6 +70,9 @@ pub(crate) trait DeferredImport: Send + Sized { /// Decode one complete codec frame. fn decode(&mut self, frame: Bytes, pts: moq_net::Timestamp) -> moq_mux::Result<()>; + /// Mark a break in the active media track's timeline. + fn discontinuity(&mut self) -> moq_mux::Result<()>; + /// Finish the active media track. fn finish(self) -> moq_mux::Result>; @@ -86,6 +93,10 @@ impl DeferredImport for moq_mux::codec::vp8::Import { moq_mux::codec::vp8::Import::finish(self) } + fn discontinuity(&mut self) -> moq_mux::Result<()> { + moq_mux::codec::vp8::Import::discontinuity(self) + } + fn abort(self, err: moq_net::Error) { moq_mux::codec::vp8::Import::abort(self, err); } @@ -104,6 +115,10 @@ impl DeferredImport for moq_mux::codec::vp9::Import { moq_mux::codec::vp9::Import::finish(self) } + fn discontinuity(&mut self) -> moq_mux::Result<()> { + moq_mux::codec::vp9::Import::discontinuity(self) + } + fn abort(self, err: moq_net::Error) { moq_mux::codec::vp9::Import::abort(self, err); } @@ -164,6 +179,14 @@ impl DeferredVideo { import.decode(frame, pts).map_err(Into::into) } + /// Mark a break in the media track's timeline, which a track no frame has created yet has none of. + pub fn discontinuity(&mut self) -> Result<()> { + match &mut self.state { + DeferredState::Active(import) => import.discontinuity().map_err(Into::into), + _ => Ok(()), + } + } + /// Finish the media track, whether or not a frame ever created its importer. pub fn finish(self) -> Result { Ok(match self.state { diff --git a/rs/moq-webrtc/src/codec/opus.rs b/rs/moq-webrtc/src/codec/opus.rs index 825f6c8c3fa..d119fcf0301 100644 --- a/rs/moq-webrtc/src/codec/opus.rs +++ b/rs/moq-webrtc/src/codec/opus.rs @@ -33,6 +33,11 @@ impl codec::Bridge for Bridge { Ok(()) } + fn discontinuity(&mut self) -> Result<()> { + self.import.discontinuity()?; + Ok(()) + } + fn finish(self: Box) -> Result { Ok(Box::new(self.import.finish()?)) } diff --git a/rs/moq-webrtc/src/codec/vp8.rs b/rs/moq-webrtc/src/codec/vp8.rs index d56df7ab349..3042b817fb9 100644 --- a/rs/moq-webrtc/src/codec/vp8.rs +++ b/rs/moq-webrtc/src/codec/vp8.rs @@ -25,6 +25,10 @@ impl codec::Bridge for Bridge { self.import.decode(frame.payload, pts) } + fn discontinuity(&mut self) -> Result<()> { + self.import.discontinuity() + } + fn finish(self: Box) -> Result { self.import.finish() } diff --git a/rs/moq-webrtc/src/codec/vp9.rs b/rs/moq-webrtc/src/codec/vp9.rs index dfde3c1bf4f..dc08add788c 100644 --- a/rs/moq-webrtc/src/codec/vp9.rs +++ b/rs/moq-webrtc/src/codec/vp9.rs @@ -25,6 +25,10 @@ impl codec::Bridge for Bridge { self.import.decode(frame.payload, pts) } + fn discontinuity(&mut self) -> Result<()> { + self.import.discontinuity() + } + fn finish(self: Box) -> Result { self.import.finish() } diff --git a/rs/moq-webrtc/src/ingest.rs b/rs/moq-webrtc/src/ingest.rs index 4de41862f59..f26dfe3000c 100644 --- a/rs/moq-webrtc/src/ingest.rs +++ b/rs/moq-webrtc/src/ingest.rs @@ -33,10 +33,17 @@ impl IngestSink { pub fn routed(route: moq_net::origin::Route, config: moq_mux::catalog::Config) -> Result { let mut broadcast = route.broadcast().clone(); let catalog = moq_mux::catalog::Producer::new(&mut broadcast, config)?; - Ok(Self { - _route: Some(route), - ..Self::new(broadcast, catalog) - }) + let mut sink = Self::new(broadcast, catalog); + sink._route = Some(route); + Ok(sink) + } +} + +/// Dropped before the session ends, as when the pull's future is cancelled at shutdown, the sink +/// aborts its tracks instead of leaving them unfinished. A finished or aborted sink has nothing left. +impl Drop for IngestSink { + fn drop(&mut self) { + session::MediaSink::abort(self, moq_net::Error::Cancel); } } @@ -79,6 +86,10 @@ impl session::MediaSink for IngestSink { Ok(()) } + fn on_discontinuity(&mut self, mid: str0m::media::Mid) -> Result<()> { + self.bridges.discontinuity(mid) + } + fn on_frame(&mut self, mid: str0m::media::Mid, frame: codec::Frame) -> Result<()> { self.bridges.push(mid, frame) } @@ -158,4 +169,55 @@ mod tests { drop(linger); assert!(consumer.is_closed()); } + + /// A step in the session timeline breaks the track's timeline with a group of its own, which + /// is what lets a reader's clock re-anchor instead of read the step as drift. + #[tokio::test(start_paused = true)] + async fn a_step_publishes_a_marker_group() { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, Default::default()).unwrap(); + let mut sink = IngestSink::new(broadcast, catalog); + let audio = str0m::media::MediaKind::Audio; + sink.on_track("0".into(), audio, str0m::format::Codec::Opus, Some((48_000, 2))) + .unwrap(); + let frame = |timestamp_us| codec::Frame { + timestamp_us, + payload: bytes::Bytes::from_static(&[0xfc, 0xff, 0xfe]), + }; + let mut track = consumer.track("0.opus").unwrap().subscribe(None).await.unwrap(); + let mut groups = 0; + let mut read = async |track: &mut moq_net::track::Subscriber| { + while let Ok(Ok(Some(_))) = kio::clock::timeout(Duration::from_millis(100), track.recv_group()).await { + groups += 1; + } + }; + sink.on_frame("0".into(), frame(20_000)).unwrap(); + read(&mut track).await; + sink.on_frame("0".into(), frame(40_000)).unwrap(); + read(&mut track).await; + sink.on_discontinuity("0".into()).unwrap(); + read(&mut track).await; + sink.on_frame("0".into(), frame(400_000)).unwrap(); + read(&mut track).await; + assert_eq!(groups, 4, "two frames, the marker, and the frame after it"); + } + + /// Dropping a sink before its session ends, as when shutdown cancels a pull, aborts its + /// tracks and the catalog rather than leaving them to warn as dropped unfinished. + #[tokio::test] + async fn a_dropped_sink_aborts_its_tracks() { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, Default::default()).unwrap(); + let mut sink = IngestSink::new(broadcast, catalog.clone()); + let audio = str0m::media::MediaKind::Audio; + sink.on_track("0".into(), audio, str0m::format::Codec::Opus, Some((48_000, 2))) + .unwrap(); + let mut track = consumer.track("0.opus").unwrap().subscribe(None).await.unwrap(); + + drop(sink); + drop(catalog); + assert!(matches!(track.recv_group().await, Err(moq_net::Error::Cancel))); + } } diff --git a/rs/moq-webrtc/src/session.rs b/rs/moq-webrtc/src/session.rs index 62269969d04..af0e9816dbc 100644 --- a/rs/moq-webrtc/src/session.rs +++ b/rs/moq-webrtc/src/session.rs @@ -51,6 +51,10 @@ pub trait MediaSink: Send { audio_params: Option<(u32, u32)>, ) -> Result<()>; + /// Called before the [`on_frame`](Self::on_frame) of a frame that opens a group, when the + /// session timeline stepped to it: the track's timeline breaks there. + fn on_discontinuity(&mut self, mid: str0m::media::Mid) -> Result<()>; + /// Called on each [`MediaData`](str0m::media::MediaData) event. The session /// loop has already converted the timestamp to microseconds. fn on_frame(&mut self, mid: str0m::media::Mid, frame: codec::Frame) -> Result<()>; @@ -296,15 +300,21 @@ impl Session { Admit::Forward => {} Admit::Drop => return Ok(()), Admit::Request => { - if let Some(mut writer) = self.agent.rtc_mut().writer(data.mid) { - writer - .request_keyframe(data.rid, str0m::media::KeyframeRequestKind::Pli) - .map_err(Error::rtc)?; - } + request_keyframe(&mut self.agent, data.mid, data.rid)?; return Ok(()); } } - let timestamp_us = self.ingest_clock.normalize(data.mid, arrival, media_us); + let opens_group = data.is_keyframe() || self.ingest_order.is_audio(data.mid); + let mapped = self.ingest_clock.normalize(data.mid, arrival, media_us, opens_group); + // A video step waits for a keyframe, which a publisher that sends one only + // when asked never volunteers. + if mapped.deferred && self.ingest_order.keyframe_due(data.mid, Instant::now()) { + request_keyframe(&mut self.agent, data.mid, data.rid)?; + } + if mapped.stepped { + sink.on_discontinuity(data.mid)?; + } + let timestamp_us = mapped.timestamp_us; sink.on_frame( data.mid, codec::Frame { @@ -460,6 +470,20 @@ impl IngestRate { } } +/// Ask the publisher of `mid` for a keyframe with PLI. +fn request_keyframe( + agent: &mut crate::ice::Agent, + mid: str0m::media::Mid, + rid: Option, +) -> Result<()> { + if let Some(mut writer) = agent.rtc_mut().writer(mid) { + writer + .request_keyframe(rid, str0m::media::KeyframeRequestKind::Pli) + .map_err(Error::rtc)?; + } + Ok(()) +} + /// Which ingested frames may reach the sink, line by line. /// /// RTP can join a publisher mid-group, and deliver a packet late, as a retransmission does. A @@ -477,8 +501,9 @@ struct IngestOrder { enum IngestLine { /// Video yet to see a keyframe, with when it may next ask for one: `None` without PLI. Keyless { request_at: Option }, - /// Video forwarding, refusing anything before its last keyframe at `keyframe`. - Video { keyframe: u64 }, + /// Video forwarding, refusing anything before its last keyframe at `keyframe`, and with when + /// it may next ask for one (`None` without PLI) to take a deferred timeline step. + Video { keyframe: u64, request_at: Option }, /// Audio forwarding, refusing anything not after its last frame at `last`. Audio { last: Option }, } @@ -506,6 +531,25 @@ impl IngestOrder { self.lines.insert(mid, line); } + /// Whether `mid` is an audio line, whose every frame opens a group. + fn is_audio(&self, mid: str0m::media::Mid) -> bool { + matches!(self.lines.get(&mid), Some(IngestLine::Audio { .. })) + } + + /// Whether to ask `mid`, a video line that deferred a timeline step, for a keyframe now: at + /// most once an interval, and never without PLI. + fn keyframe_due(&mut self, mid: str0m::media::Mid, now: Instant) -> bool { + match self.lines.get_mut(&mid) { + Some(IngestLine::Video { + request_at: Some(at), .. + }) if now >= *at => { + *at = now.saturating_add(KEYFRAME_REQUEST_INTERVAL); + true + } + _ => false, + } + } + /// Whether a frame on `mid` at its line's RTP time `timestamp_us` reaches the sink, and whether /// to ask for a keyframe. fn admit(&mut self, mid: str0m::media::Mid, keyframe: bool, timestamp_us: u64, now: Instant) -> Admit { @@ -520,12 +564,15 @@ impl IngestOrder { } _ => Admit::Drop, }, - IngestLine::Keyless { .. } => { - *line = IngestLine::Video { keyframe: timestamp_us }; + IngestLine::Keyless { request_at } => { + *line = IngestLine::Video { + keyframe: timestamp_us, + request_at: *request_at, + }; Admit::Forward } - IngestLine::Video { keyframe: floor } if timestamp_us < *floor => Admit::Drop, - IngestLine::Video { keyframe: floor } => { + IngestLine::Video { keyframe: floor, .. } if timestamp_us < *floor => Admit::Drop, + IngestLine::Video { keyframe: floor, .. } => { if keyframe { *floor = timestamp_us; } @@ -579,7 +626,13 @@ impl IngestClock { /// Map a raw RTP-derived microsecond timestamp onto the session timeline. /// `arrival` is the packet's network time /// ([`MediaData::network_time`](str0m::media::MediaData::network_time)). - fn normalize(&mut self, mid: str0m::media::Mid, arrival: Instant, media_us: u64) -> u64 { + /// + /// A track's timeline steps when the sender reports replace arrival time, by as much as the + /// arrival skew between tracks was. A step is only taken by a frame that `opens_group`, which a + /// reader can be told is a break ([`Mapped::stepped`]): the readers' clocks follow a track + /// that continues, and would read the step as drift. A frame that continues a group keeps the + /// track's timeline until the next one that opens a group. + fn normalize(&mut self, mid: str0m::media::Mid, arrival: Instant, media_us: u64, opens_group: bool) -> Mapped { let epoch = *self.arrival_epoch.get_or_insert(arrival); let track = self.tracks.entry(mid).or_default(); let offset = *track.arrival_offset_us.get_or_insert_with(|| { @@ -596,6 +649,12 @@ impl IngestClock { }); let fallback = to_u64(media_us as i128 + offset); let previous = track.last_output_us; + // Where the track's last frame puts this one if its timeline just continues, never + // behind it: a reordered picture decodes before the last, as its media time says. + let continued = previous.zip(track.last_media_us).map(|(output, media)| { + let next = output as i128 + media_us as i128 - media as i128; + to_u64(next.max(output as i128 + 1)) + }); track.last_media_us = Some(media_us); track.last_output_us = Some(fallback); @@ -611,8 +670,17 @@ impl IngestClock { Some(mapped) => mapped.max(previous.map_or(fallback, |last| last.saturating_add(1))), None => fallback, }; + let step = continued.is_some_and(|continued| output > continued.saturating_add(STEP_US)); + let (output, stepped, deferred) = match continued { + Some(continued) if step && !opens_group => (continued, false, true), + _ => (output, step, false), + }; self.tracks.get_mut(&mid).expect("track was inserted").last_output_us = Some(output); - output + Mapped { + timestamp_us: output, + stepped, + deferred, + } } /// Switch to the sender's common clock once every negotiated track has both @@ -641,6 +709,20 @@ impl IngestClock { } } +/// How far a frame may land ahead of where its track's timeline continues before it is a step: a +/// sender report's own correction, which readers absorb, stays under it. +const STEP_US: u64 = 20_000; + +/// A frame's place on the session timeline. +struct Mapped { + timestamp_us: u64, + /// The frame opens a group the track's timeline stepped to, which no earlier frame led to. + stepped: bool, + /// The track's timeline should step but this frame continues a group, so it waits for the + /// next keyframe. + deferred: bool, +} + #[derive(Default)] struct IngestTrackClock { arrival_offset_us: Option, @@ -726,6 +808,14 @@ impl Bridges { Ok(()) } + /// Mark a break in `mid`'s track before its next frame. + pub fn discontinuity(&mut self, mid: str0m::media::Mid) -> Result<()> { + if let Some(bridge) = self.inner.get_mut(&mid) { + bridge.discontinuity()?; + } + Ok(()) + } + /// Finish every bridge's track after the final frame. pub fn finish(&mut self) -> Result<()> { for bridge in std::mem::take(&mut self.inner).into_values() { @@ -907,9 +997,9 @@ mod tests { let audio = Mid::from("0"); let video = Mid::from("1"); let t0 = Instant::now(); - assert_eq!(clock.normalize(audio, t0, 0), 0); + assert_eq!(normalize(&mut clock, audio, t0, 0), 0); assert_eq!( - clock.normalize(video, t0.saturating_add(Duration::from_millis(50)), 0), + normalize(&mut clock, video, t0.saturating_add(Duration::from_millis(50)), 0), 50_000 ); @@ -924,11 +1014,16 @@ mod tests { // Both tracks stay on arrival time. assert_eq!( - clock.normalize(audio, t0.saturating_add(Duration::from_secs(1)), 1_000_000), + normalize(&mut clock, audio, t0.saturating_add(Duration::from_secs(1)), 1_000_000), 1_000_000 ); assert_eq!( - clock.normalize(video, t0.saturating_add(Duration::from_millis(1_050)), 1_000_000), + normalize( + &mut clock, + video, + t0.saturating_add(Duration::from_millis(1_050)), + 1_000_000 + ), 1_050_000 ); assert!(SenderAnchor::new(sender_info(1, UNIX_EPOCH, 0)).is_ok()); @@ -982,10 +1077,10 @@ mod tests { let mid = Mid::from("0"); let t0 = Instant::now(); // Raw RTP base is a large random value; the first frame must map to 0. - assert_eq!(clock.normalize(mid, t0, 5_000_000_000), 0); + assert_eq!(normalize(&mut clock, mid, t0, 5_000_000_000), 0); // A later frame advances by the RTP delta, not by arrival jitter. let arrival = t0.saturating_add(Duration::from_millis(17)); // jittered arrival, ignored after anchor - assert_eq!(clock.normalize(mid, arrival, 5_000_020_000), 20_000); + assert_eq!(normalize(&mut clock, mid, arrival, 5_000_020_000), 20_000); } #[test] @@ -995,14 +1090,15 @@ mod tests { let video = Mid::from("1"); let t0 = Instant::now(); // Audio anchors the session at 0 with its own random RTP base. - assert_eq!(clock.normalize(audio, t0, 1_000_000_000), 0); + assert_eq!(normalize(&mut clock, audio, t0, 1_000_000_000), 0); // Video's first frame arrives 5 ms later with an unrelated RTP base; it // must land at +5 ms on the shared timeline, not at video's raw base. let video_arrival = t0.saturating_add(Duration::from_millis(5)); - assert_eq!(clock.normalize(video, video_arrival, 8_000_000_000), 5_000); + assert_eq!(normalize(&mut clock, video, video_arrival, 8_000_000_000), 5_000); // And then track its own RTP delta. assert_eq!( - clock.normalize( + normalize( + &mut clock, video, video_arrival.saturating_add(Duration::from_millis(33)), 8_000_033_000 @@ -1018,14 +1114,15 @@ mod tests { let video = Mid::from("1"); let t0 = Instant::now(); // Audio's MediaData is dequeued first and sets the epoch at t0. - assert_eq!(clock.normalize(audio, t0, 1_000_000), 0); + assert_eq!(normalize(&mut clock, audio, t0, 1_000_000), 0); // Video's first frame actually arrived 5 ms *before* the epoch. Its lead // pulls the start below zero (clamped to 0), and a frame 33 ms into video // lands 28 ms onto the shared timeline (33 ms - the 5 ms head start). let video_arrival = t0.saturating_sub(Duration::from_millis(5)); - assert_eq!(clock.normalize(video, video_arrival, 8_000_000), 0); + assert_eq!(normalize(&mut clock, video, video_arrival, 8_000_000), 0); assert_eq!( - clock.normalize( + normalize( + &mut clock, video, video_arrival.saturating_add(Duration::from_millis(33)), 8_033_000 @@ -1043,17 +1140,28 @@ mod tests { let audio_base = 1_000_000_000; let video_base = 8_000_000_000; - assert_eq!(clock.normalize(audio, t0, audio_base), 0); + assert_eq!(normalize(&mut clock, audio, t0, audio_base), 0); assert_eq!( - clock.normalize(video, t0.saturating_add(Duration::from_millis(50)), video_base), + normalize( + &mut clock, + video, + t0.saturating_add(Duration::from_millis(50)), + video_base + ), 50_000 ); assert_eq!( - clock.normalize(audio, t0.saturating_add(Duration::from_secs(1)), audio_base + 1_000_000), + normalize( + &mut clock, + audio, + t0.saturating_add(Duration::from_secs(1)), + audio_base + 1_000_000 + ), 1_000_000 ); assert_eq!( - clock.normalize( + normalize( + &mut clock, video, t0.saturating_add(Duration::from_millis(1_050)), video_base + 1_000_000, @@ -1067,12 +1175,14 @@ mod tests { clock.observe(audio, sender_info(1, report_time, audio_base + 1_000_000)); clock.observe(video, sender_info(2, report_time, video_base + 1_000_000)); - let audio_time = clock.normalize( + let audio_time = normalize( + &mut clock, audio, t0.saturating_add(Duration::from_millis(1_020)), audio_base + 1_020_000, ); - let video_time = clock.normalize( + let video_time = normalize( + &mut clock, video, t0.saturating_add(Duration::from_millis(1_070)), video_base + 1_020_000, @@ -1081,6 +1191,137 @@ mod tests { assert_eq!(audio_time, 1_070_000); } + /// `mid`'s frame as the timeline maps it, every frame opening a group. + fn normalize(clock: &mut IngestClock, mid: Mid, arrival: Instant, media_us: u64) -> u64 { + clock.normalize(mid, arrival, media_us, true).timestamp_us + } + + /// The sender reports step the track that arrived earlier than the other forward, once, by the + /// arrival skew they replace: audio, whose every frame opens a group, takes the step and says + /// so, and video keeps its timeline through its deltas and takes it at its next keyframe. + #[test] + fn the_sender_reports_step_a_track_where_a_group_opens() { + let mut clock = IngestClock::default(); + let audio = Mid::from("0"); + let video = Mid::from("1"); + let t0 = Instant::now(); + let (audio_base, video_base) = (1_000_000_000, 8_000_000_000); + // A frame of media time `ms` that arrived `late` ms after it was captured. + let frame = |clock: &mut IngestClock, mid, ms: u64, late: u64, base: u64, opens_group| { + let arrival = t0.saturating_add(Duration::from_millis(ms + late)); + clock.normalize(mid, arrival, base + ms * 1_000, opens_group) + }; + + // Video arrives 300 ms behind audio, though both were captured at the same instant. + for ms in (0..1_000).step_by(20) { + assert!(!frame(&mut clock, audio, ms, 0, audio_base, true).stepped); + assert!(!frame(&mut clock, video, ms, 300, video_base, ms == 0).stepped); + } + + // Both reports name one capture instant for media time 1 s. + let report_time = UNIX_EPOCH + Duration::from_secs(1_700_000_001); + clock.observe(audio, sender_info(1, report_time, audio_base + 1_000_000)); + clock.observe(video, sender_info(2, report_time, video_base + 1_000_000)); + + // Audio was placed 300 ms ahead of where it belongs next to video, and no track may + // rewind, so audio moves forward and video stays. + let audio_next = frame(&mut clock, audio, 1_000, 0, audio_base, true); + assert!(audio_next.stepped, "audio takes the step"); + assert_eq!(audio_next.timestamp_us, 1_300_000); + let delta = frame(&mut clock, video, 1_000, 300, video_base, false); + assert!(!delta.stepped, "a video delta continues its group"); + assert_eq!(delta.timestamp_us, 1_300_000); + // Neither steps again. + for ms in (1_020..2_000).step_by(20) { + assert!(!frame(&mut clock, audio, ms, 0, audio_base, true).stepped); + assert!(!frame(&mut clock, video, ms, 300, video_base, false).stepped); + } + } + + /// A video line with its timeline stepping forward keeps it through the deltas of its group and + /// takes it, saying so, at the keyframe that opens the next. + #[test] + fn video_takes_a_step_at_its_next_keyframe() { + let mut clock = IngestClock::default(); + let audio = Mid::from("0"); + let video = Mid::from("1"); + let t0 = Instant::now(); + let (audio_base, video_base) = (1_000_000_000, 8_000_000_000); + let frame = |clock: &mut IngestClock, mid, ms: u64, late: u64, base: u64, opens_group| { + let arrival = t0.saturating_add(Duration::from_millis(ms + late)); + clock.normalize(mid, arrival, base + ms * 1_000, opens_group) + }; + + // Audio arrives 300 ms behind video, so video steps forward once the reports arrive. + for ms in (0..1_000).step_by(40) { + assert!(!frame(&mut clock, video, ms, 0, video_base, ms == 0).stepped); + assert!(!frame(&mut clock, audio, ms, 300, audio_base, true).stepped); + } + let report_time = UNIX_EPOCH + Duration::from_secs(1_700_000_001); + clock.observe(audio, sender_info(1, report_time, audio_base + 1_000_000)); + clock.observe(video, sender_info(2, report_time, video_base + 1_000_000)); + + for ms in (1_000..1_200).step_by(40) { + let delta = frame(&mut clock, video, ms, 0, video_base, false); + assert!(!delta.stepped, "{ms}"); + assert!(delta.deferred, "{ms}: a deferred step asks for a keyframe"); + assert_eq!(delta.timestamp_us, ms * 1_000, "a delta continues its group"); + } + let key = frame(&mut clock, video, 1_200, 0, video_base, true); + assert!(key.stepped, "the keyframe takes the step"); + assert!(!key.deferred); + assert_eq!(key.timestamp_us, 1_500_000); + let after = frame(&mut clock, video, 1_240, 0, video_base, false); + assert!(!after.stepped && !after.deferred); + } + + /// A video line that deferred a step asks for a keyframe at once and then at most once an + /// interval, as long as it negotiated PLI. + #[test] + fn a_deferred_step_asks_for_a_keyframe() { + let mut order = IngestOrder::default(); + let (video, silent, audio) = (Mid::from("1"), Mid::from("2"), Mid::from("0")); + let t0 = Instant::now(); + order.open(video, true, true, t0); + order.open(silent, true, false, t0); + order.open(audio, false, true, t0); + for mid in [video, silent] { + assert_eq!(order.admit(mid, true, 0, t0), Admit::Forward); + } + + assert!(order.keyframe_due(video, t0)); + let early = t0.saturating_add(KEYFRAME_REQUEST_INTERVAL - Duration::from_millis(1)); + assert!(!order.keyframe_due(video, early)); + assert!(order.keyframe_due(video, t0.saturating_add(KEYFRAME_REQUEST_INTERVAL))); + + let late = t0.saturating_add(KEYFRAME_REQUEST_INTERVAL * 5); + assert!(!order.keyframe_due(silent, late), "no PLI, no request"); + assert!(!order.keyframe_due(audio, late), "audio opens a group on every frame"); + } + + /// A reordered picture, which decodes before the one sent ahead of it, lands where its own media + /// time puts it before the reports, and is no step after them. + #[test] + fn a_reordered_picture_is_no_step() { + let mut clock = IngestClock::default(); + let video = Mid::from("1"); + let t0 = Instant::now(); + let at = |ms: u64| t0.saturating_add(Duration::from_millis(ms)); + let base = 8_000_000_000; + let times = |clock: &mut IngestClock, frames: [u64; 3]| { + frames.map(|ms| { + let mapped = clock.normalize(video, at(ms), base + ms * 1_000, ms == 100); + assert!(!mapped.stepped); + mapped.timestamp_us + }) + }; + assert_eq!(times(&mut clock, [100, 166, 133]), [0, 66_000, 33_000]); + let report_time = UNIX_EPOCH + Duration::from_secs(1_700_000_001); + clock.observe(video, sender_info(2, report_time, base + 133_000)); + // With one track the reports change nothing, and a delta keeps its place. + assert_eq!(times(&mut clock, [200, 266, 233]), [100_000, 166_000, 133_000]); + } + fn sender_info(ssrc: u32, ntp_time: SystemTime, rtp_us: u64) -> SenderInfo { SenderInfo { ssrc: Ssrc::from(ssrc),