diff --git a/quest/m1/README.md b/quest/m1/README.md index c4d2e4f177e..3843e1eefa9 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -124,7 +124,6 @@ tracks, so per-group stats stop costing cluster announces. - [Rust resume checks compression](/quest/m1/resume-compress.md) - a same-epoch route copy with a different `compress` is refused instead of corrupting framing - [Rust resume backs single-route groups](/quest/m1/resume-handed-out.md) - a group first read from one route keeps the later copy that holds it, with a passthrough benchmark - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS -- [Exporters follow a rendition to a new track](/quest/m1/export-rendition-rebind.md) - the fMP4, TS, MKV and FLV exporters rebuild a rendition whose resolved `track` or `broadcast` changes under the same ID - [Track live signal](/quest/m1/track-live.md) - a reader can wait for its track's feed to be live, so a player skips old cached groups while a fresh upstream subscription starts - [Producers drop the phantom catalog type](/quest/m1/catalog-phantom.md) - moq-video, moq-audio and moq-mux producers lose an `E: CatalogExt` parameter that only `PhantomData` holds - [JSON encoder options without a downcast](/quest/m1/json-encoder-options.md) - moq-mux's `json_snapshot` takes its `deltas` switch through a typed path any custom catalog entry can use, not an `Any` downcast diff --git a/quest/m1/annexb-export.md b/quest/m1/annexb-export.md index fbea6c71221..ca18f855822 100644 --- a/quest/m1/annexb-export.md +++ b/quest/m1/annexb-export.md @@ -26,7 +26,3 @@ Tests cover configuration changes, rendition switching, parameter-set injection, keyframe joining, completion, and the rebind. Public API: none. Wire: none. - -## Required - -- [Exporters follow a rendition to a new track](/quest/m1/export-rendition-rebind.md) - reworks `ExportSource` and sets the rebind pattern this follows diff --git a/quest/m1/export-rendition-rebind.md b/quest/m1/export-rendition-rebind.md deleted file mode 100644 index 0ef6263b186..00000000000 --- a/quest/m1/export-rendition-rebind.md +++ /dev/null @@ -1,47 +0,0 @@ -# [M] Exporters follow a rendition to a new track - -## Goal - -When a catalog update keeps a rendition's ID but changes its resolved -`track` or `broadcast`, the fMP4, TS, MKV and FLV exporters rebuild that -rendition from the new track, as a remove plus an add. Today each exporter -resolves the track once, when it builds the rendition's `ExportSource`, and -keys it by ID afterwards, so it keeps muxing the old track (or stops when -that track ends) while the catalog names the new one. - -## Plan - -- Keep each source's resolved identity (broadcast reference and track name) - beside it, and compare it on every catalog update. A change drops the old - source and builds a new one, through the same path a newly added - rendition takes in that muxer: fMP4 `Fmp4Track::reconfigure`, the TS - `old.remove(name)` reuse, and the MKV and FLV bound-ID checks. -- Separate source state from output state. Source state (the subscription, - group and frame cursors, the codec description) resets, because the new - track is new content. Output state already emitted continues: TS - continuity counters live per PID outside each `Track` for this reason, - and fMP4's fragment sequence, declared track ID and written timestamp - frontier belong to the ongoing output. Each muxer either signals a - format-specific discontinuity on the continuing output, or ends the export - loudly where its format cannot express the switch; it never resets output - state mid-stream. -- Revalidate declarations already emitted. A replacement track can carry - different initialization data (an AVC or HEVC SPS/PPS, geometry), which - the fMP4 `moov`, MKV `Tracks` and FLV sequence header already declared for - the old track. Each rebind resolves the replacement's declaration and - compares it: if it differs, emit a new header where the format permits, or - fail before writing any replacement payload. The tests include a - replacement with changed initialization data. -- Tests per exporter: a rendition `main` moved from `video-a` to - `video-b` exports `video-b`'s frames after the update and none of - `video-a`'s, and a `broadcast`-only change does the same. Each test also - asserts the output across the switch: TS continuity counters keep - counting on the reused PID, fMP4 fragment sequence numbers and the - timestamp frontier never step back, and the format's discontinuity signal - is present, or else the export ends with an error. - -Raised by a Codex review of #5367, which added the per-rendition `track` -field; the `broadcast` gap predates it. Deferred from that PR by the -maintainer's recommended option. - -Public API: none. Wire: none. diff --git a/rs/moq-mux/src/container/flv/export.rs b/rs/moq-mux/src/container/flv/export.rs index ae203aa69fe..797b2656f74 100644 --- a/rs/moq-mux/src/container/flv/export.rs +++ b/rs/moq-mux/src/container/flv/export.rs @@ -14,7 +14,9 @@ //! stream, so only the best video rendition (see [`Video::ranked`](hang::catalog::Video::ranked)) //! and the best audio rendition (see [`Audio::ranked`](hang::catalog::Audio::ranked)) //! the stream yields are muxed and the rest are ignored. Each pick follows the -//! catalog until the stream header goes out, then stays. With +//! catalog until the stream header goes out, then stays, following that rendition to another +//! track when the catalog moves it (under a new sequence-header tag when its codec config +//! record changes; a move back in time fails the export). With //! [`with_multitrack`](Export::with_multitrack) every rendition the stream yields //! is muxed instead, each as an enhanced-RTMP multitrack track addressed by its own //! track id (use this only for a player that advertised the `Multitrack` capability). @@ -23,7 +25,7 @@ use std::task::Poll; use anyhow::Context; use bytes::{BufMut, Bytes, BytesMut}; -use hang::catalog::{AV1, AudioCodec, Catalog, Container, VideoCodec}; +use hang::catalog::{AV1, AudioCodec, Catalog, Container, Reference, VideoCodec}; use super::{ AAC_AUDIO_TAG_HEADER, AAC_RAW, AAC_SEQUENCE_HEADER, AUDIO_FORMAT_EX, AUDIO_PACKET_CODED_FRAMES, @@ -34,14 +36,26 @@ use super::{ use crate::catalog::Stream; use crate::container::{ExportSource, Frame}; -/// Before the header, unbind every track the catalog no longer lists (removed, or disabled), -/// so the header declares only what the snapshot offers. A single-track stream also follows the -/// best rendition, so a snapshot that lacks it doesn't fix the pick. Unbinding unsubscribes the -/// track; nothing was emitted from it yet. -fn unbind(tracks: &mut Vec, renditions: &[(&String, &C)], multitrack: bool) { +/// Before the header, unbind every track the catalog no longer lists (removed, or disabled) or +/// moved to another track, so the header declares only what the snapshot offers. A single-track +/// stream also follows the best rendition, so a snapshot that lacks it doesn't fix the pick. +/// Unbinding unsubscribes the track; nothing was emitted from it yet. +fn unbind( + tracks: &mut Vec, + renditions: &[(&String, &C)], + multitrack: bool, + source: &crate::Source, +) { + let bound = |track: &FlvTrack, (name, config): &(&String, &C)| { + **name == track.name && track.source.reads(source, name, *config) + }; if multitrack { - tracks.retain(|track| renditions.iter().any(|(name, _)| **name == track.name)); - } else if tracks.first().map(|track| &track.name) != renditions.first().map(|(name, _)| *name) { + tracks.retain(|track| renditions.iter().any(|rendition| bound(track, rendition))); + } else if !tracks + .first() + .zip(renditions.first()) + .is_some_and(|(track, rendition)| bound(track, rendition)) + { tracks.clear(); } // Ids follow bind order, so close the gaps an unbind left. @@ -92,6 +106,10 @@ impl Flavor { } } + fn is_video(self) -> bool { + matches!(self, Flavor::Avc | Flavor::Hevc | Flavor::Av1 | Flavor::Vp9) + } + fn has_composition_time(self) -> bool { matches!(self, Flavor::Avc | Flavor::Hevc) } @@ -125,8 +143,8 @@ pub struct Export { source: crate::Source, catalog: Option, delay: std::time::Duration, - /// Frames with their tag timestamp, encoded only once released. - jitter: crate::jitter::Buffer<(u8, usize), (Frame, u32)>, + /// Frames as tags, encoded only once released. + jitter: crate::jitter::Buffer<(u8, usize), Tag>, /// Emit every rendition as an enhanced-RTMP multitrack track, rather than only /// the best video and audio rendition. multitrack: bool, @@ -138,6 +156,16 @@ pub struct Export { header_emitted: bool, } +/// A frame waiting to go out as a tag. +struct Tag { + frame: Frame, + /// The tag timestamp, the authored DTS for a codec with composition time. + timestamp: u32, + /// The codec config record the frame was read under, which a sequence-header tag must + /// have announced before it. + record: Option, +} + /// A subscribed rendition feeding the muxer. struct FlvTrack { name: String, @@ -160,9 +188,18 @@ struct FlvTrack { dts_reserve: u32, /// Last authored video DTS, in FLV milliseconds. last_dts: Option, + /// The config record the last sequence-header tag announced. + announced: Option, } impl FlvTrack { + /// Read `source`, the track the catalog moved this rendition to. + fn rebind(&mut self, source: ExportSource) { + self.source = source.continuing(&self.source); + self.finished = false; + self.empty = moq_clock::now(); + } + fn tag_timestamp(&self, frame: &Frame) -> anyhow::Result { let pts = frame_timestamp_ms(frame)?; if self.flavor.has_composition_time() { @@ -174,7 +211,7 @@ impl FlvTrack { /// The out-of-band config record for this track's sequence-header tag, or /// `None` for codecs that carry their config in band. - fn config_record(&self) -> anyhow::Result> { + fn config_record(&self) -> anyhow::Result> { if !self.flavor.has_sequence_header() { return Ok(None); } @@ -183,7 +220,7 @@ impl FlvTrack { .description() .or(self.fallback_description.as_ref()) .with_context(|| format!("FLV track '{}' missing its codec config record", self.name))?; - Ok(Some(record.as_ref())) + Ok(Some(record.clone())) } } @@ -274,8 +311,7 @@ impl Export { if let Poll::Ready(ready) = self.jitter.poll_next(waiter) { let (kind, index) = ready.track; - let (frame, timestamp) = ready.item; - return Poll::Ready(Ok(Some(self.encode_frame(kind == 0, index, frame, timestamp)?))); + return Poll::Ready(Ok(Some(self.encode_frame(kind == 0, index, ready.item)?))); } if !self.jitter.is_empty() { return Poll::Pending; @@ -353,7 +389,8 @@ impl Export { break; } }; - if !self.header_emitted && !track.source.header_ready() { + // Until its codec config resolves, a frame has no sequence header to follow. + if !track.source.header_ready() { continue; } crate::jitter::Arrival { @@ -376,6 +413,7 @@ impl Export { if track.flavor.has_composition_time() { track.last_dts = Some(timestamp); } + let record = track.config_record()?; let crate::jitter::Arrival { arrived, read, @@ -398,7 +436,11 @@ impl Export { cut, sync, leading, - item: (item, timestamp), + item: Tag { + frame: item, + timestamp, + record, + }, }, )? == crate::jitter::Push::Late { @@ -421,18 +463,22 @@ impl Export { // A single-track FLV stream binds one rendition of each kind; multitrack // binds them all. // - // Only bind before the header is emitted: the sequence-header (config) tags - // go out with the header, and there's no in-band way to introduce a new - // track's config mid-stream, so a rendition first seen afterward is left - // unmuxed rather than emitted as undecodable config-less frames. Until then a - // single-track stream rebinds to a better-ranked rendition. + // Only bind before the header is emitted: the header announces the stream's + // tracks, so a rendition first seen afterward is left unmuxed. Until then a + // single-track stream rebinds to a better-ranked rendition. A bound rendition the + // catalog moves to another track follows it, under a new sequence-header tag when + // its config record changes. if !self.header_emitted { self.bind_video(&catalog)?; self.bind_audio(&catalog)?; - } else if self.multitrack - && (catalog.video.renditions.len() > self.video.len() || catalog.audio.renditions.len() > self.audio.len()) - { - tracing::warn!("ignoring FLV rendition that appeared after the stream header"); + } else { + self.rebind(&catalog)?; + if self.multitrack + && (catalog.video.renditions.len() > self.video.len() + || catalog.audio.renditions.len() > self.audio.len()) + { + tracing::warn!("ignoring FLV rendition that appeared after the stream header"); + } } // A bound track leaving the catalog is read to its own end rather than refused. A @@ -442,6 +488,48 @@ impl Export { Ok(()) } + /// Move each bound track the catalog moved to another track or broadcast onto it. The tag + /// shape is fixed by the header, so the new track must carry the same codec. + fn rebind(&mut self, catalog: &Catalog) -> anyhow::Result<()> { + for track in &mut self.video { + let Some(config) = catalog.video.renditions.get(&track.name) else { + continue; + }; + if track.source.reads(&self.source, &track.name, config) { + continue; + } + anyhow::ensure!( + video_flavor(config)? == track.flavor, + "FLV track '{}' moved to another codec after the stream header", + track.name + ); + ensure_legacy(&config.container, "video", &track.name)?; + if let Some(source) = ExportSource::for_video(&self.source, &track.name, config, self.delay)? { + track.fallback_description = fallback_description(config); + track.dts_reserve = dts_reserve(config); + track.rebind(source); + } + } + for track in &mut self.audio { + let Some(config) = catalog.audio.renditions.get(&track.name) else { + continue; + }; + if track.source.reads(&self.source, &track.name, config) { + continue; + } + anyhow::ensure!( + audio_flavor(config)? == track.flavor, + "FLV track '{}' moved to another codec after the stream header", + track.name + ); + ensure_legacy(&config.container, "audio", &track.name)?; + if let Some(source) = ExportSource::for_audio(&self.source, &track.name, config, self.delay)? { + track.rebind(source); + } + } + Ok(()) + } + fn bind_video(&mut self, catalog: &Catalog) -> anyhow::Result<()> { let renditions: Vec<_> = if self.multitrack { // Name order (the catalog is a BTreeMap), so each keeps a stable track id. @@ -456,7 +544,7 @@ impl Export { ranked }; - unbind(&mut self.video, &renditions, self.multitrack); + unbind(&mut self.video, &renditions, self.multitrack, &self.source); for (name, config) in renditions { if !self.multitrack && !self.video.is_empty() { @@ -467,12 +555,7 @@ impl Export { } let flavor = video_flavor(config)?; ensure_legacy(&config.container, "video", name)?; - // AV1's av1C is optional in the catalog; synthesize one from the codec - // struct so the enhanced SequenceStart tag always has a config record. - let fallback_description = match (&config.codec, config.description.as_ref()) { - (VideoCodec::AV1(av1), None) => Some(Bytes::copy_from_slice(&av1c_bytes(av1))), - _ => None, - }; + let fallback_description = fallback_description(config); let Some(source) = ExportSource::for_video(&self.source, name, config, self.delay)? else { continue; }; @@ -488,6 +571,7 @@ impl Export { fallback_description, dts_reserve: dts_reserve(config), last_dts: None, + announced: None, }); } Ok(()) @@ -507,7 +591,7 @@ impl Export { ranked }; - unbind(&mut self.audio, &renditions, self.multitrack); + unbind(&mut self.audio, &renditions, self.multitrack, &self.source); for (name, config) in renditions { if !self.multitrack && !self.audio.is_empty() { @@ -534,6 +618,7 @@ impl Export { fallback_description: None, dts_reserve: 0, last_dts: None, + announced: None, }); } Ok(()) @@ -547,7 +632,7 @@ impl Export { } /// Build the FLV file header plus every bound track's sequence-header tag. - fn build_header(&self) -> anyhow::Result { + fn build_header(&mut self) -> anyhow::Result { let mut out = BytesMut::new(); // FLV file header: signature, version 1, type flags, data offset = 9. @@ -565,23 +650,41 @@ impl Export { // PreviousTagSize0 is always zero. out.put_u32(0); - for track in &self.video { - if let Some(body) = self.video_sequence_header(track)? { - write_tag(&mut out, TAG_VIDEO, 0, &body)?; + for track in self.video.iter().chain(&self.audio) { + if let Some(config) = track.config_record()? { + let (tag_type, body) = self.sequence_header(track, &config); + write_tag(&mut out, tag_type, 0, &body)?; } } - for track in &self.audio { - if let Some(body) = self.audio_sequence_header(track)? { - write_tag(&mut out, TAG_AUDIO, 0, &body)?; - } + for track in self.video.iter_mut().chain(&mut self.audio) { + track.announced = track.config_record()?; } Ok(out.freeze()) } - /// Encode one frame as a single FLV tag at its authored `timestamp_ms`. - fn encode_frame(&self, is_video: bool, index: usize, frame: Frame, timestamp_ms: u32) -> anyhow::Result { + /// Encode one frame as a single FLV tag, after the sequence-header tag announcing its + /// config record when the last one announced another. + fn encode_frame(&mut self, is_video: bool, index: usize, tag: Tag) -> anyhow::Result { + let Tag { + frame, + timestamp: timestamp_ms, + record, + } = tag; let mut out = BytesMut::with_capacity(TAG_HEADER_LEN + frame.payload.len() + 8); + let track = match is_video { + true => &self.video[index], + false => &self.audio[index], + }; + if let Some(record) = record.filter(|record| track.announced.as_ref() != Some(record)) { + let (tag_type, body) = self.sequence_header(track, &record); + write_tag(&mut out, tag_type, timestamp_ms, &body)?; + let tracks = match is_video { + true => &mut self.video, + false => &mut self.audio, + }; + tracks[index].announced = Some(record); + } let multitrack = self.multitrack; if is_video { let track = &self.video[index]; @@ -663,22 +766,22 @@ impl Export { Ok(out.freeze()) } - /// Build the FLV video sequence-header tag body for `track`, or `None` for - /// codecs that carry their config in band (VP9). - fn video_sequence_header(&self, track: &FlvTrack) -> anyhow::Result> { - let Some(config) = track.config_record()? else { - return Ok(None); + /// The tag type and body of the sequence-header tag announcing `config` for `track`. + fn sequence_header(&self, track: &FlvTrack, config: &[u8]) -> (u8, BytesMut) { + let tag_type = match track.flavor.is_video() { + true => TAG_VIDEO, + false => TAG_AUDIO, }; let mut body = BytesMut::new(); if self.multitrack { ex_multitrack_sequence_start( &mut body, - TAG_VIDEO, + tag_type, &track.flavor.multitrack_fourcc(), track.track_id, config, ); - return Ok(Some(body)); + return (tag_type, body); } match track.flavor { Flavor::Avc => { @@ -689,32 +792,6 @@ impl Export { } Flavor::Hevc => ex_video_sequence_start(&mut body, b"hvc1", config), Flavor::Av1 => ex_video_sequence_start(&mut body, b"av01", config), - // Codecs with no out-of-band record are filtered out by `config_record`. - Flavor::Vp9 | Flavor::Aac | Flavor::Mp3 | Flavor::Opus | Flavor::Ac3 | Flavor::Eac3 => { - unreachable!("no video sequence header for {:?}", track.name) - } - } - Ok(Some(body)) - } - - /// Build the FLV audio sequence-header tag body for `track`, or `None` for - /// codecs that carry their config in band (MP3 / AC-3 / E-AC-3). - fn audio_sequence_header(&self, track: &FlvTrack) -> anyhow::Result> { - let Some(config) = track.config_record()? else { - return Ok(None); - }; - let mut body = BytesMut::new(); - if self.multitrack { - ex_multitrack_sequence_start( - &mut body, - TAG_AUDIO, - &track.flavor.multitrack_fourcc(), - track.track_id, - config, - ); - return Ok(Some(body)); - } - match track.flavor { Flavor::Aac => { body.put_u8(AAC_AUDIO_TAG_HEADER); body.put_u8(AAC_SEQUENCE_HEADER); @@ -726,11 +803,11 @@ impl Export { body.put_slice(config); } // Codecs with no out-of-band record are filtered out by `config_record`. - Flavor::Mp3 | Flavor::Ac3 | Flavor::Eac3 | Flavor::Avc | Flavor::Hevc | Flavor::Av1 | Flavor::Vp9 => { - unreachable!("no audio sequence header for {:?}", track.name) + Flavor::Vp9 | Flavor::Mp3 | Flavor::Ac3 | Flavor::Eac3 => { + unreachable!("no sequence header for {:?}", track.name) } } - Ok(Some(body)) + (tag_type, body) } } @@ -766,6 +843,15 @@ fn ensure_legacy(container: &Container, kind: &str, name: &str) -> anyhow::Resul } } +/// AV1's av1C is optional in the catalog, so synthesize one from the codec struct and the +/// enhanced SequenceStart tag always has a config record. +fn fallback_description(config: &hang::catalog::VideoConfig) -> Option { + match (&config.codec, config.description.as_ref()) { + (VideoCodec::AV1(av1), None) => Some(Bytes::copy_from_slice(&av1c_bytes(av1))), + _ => None, + } +} + fn video_flavor(config: &hang::catalog::VideoConfig) -> anyhow::Result { match &config.codec { VideoCodec::H264(_) => Ok(Flavor::Avc), diff --git a/rs/moq-mux/src/container/flv/export_test.rs b/rs/moq-mux/src/container/flv/export_test.rs index 8e45f1c50ff..00da99ecda0 100644 --- a/rs/moq-mux/src/container/flv/export_test.rs +++ b/rs/moq-mux/src/container/flv/export_test.rs @@ -1360,6 +1360,62 @@ async fn drain_until_idle( } } +/// A rendition the catalog moves to another track, in its broadcast or a sibling one, goes on +/// as the same FLV video with that track's media and none of the one it left. A new track +/// described by another SPS is announced by a new sequence-header tag ahead of its first frame. +#[tokio::test(start_paused = true)] +async fn a_moved_rendition_continues_on_its_new_track() { + use crate::container::test_util::{Moving, SPS, marked_frame, marks}; + + for (track, broadcast, sps) in [("b", None, Moving::MAIN_SPS), ("a", Some("other"), SPS)] { + let mut moving = Moving::new(); + let mut exporter = Export::new(moving.source().await, moving.catalog_stream().await); + let mut out = Vec::new(); + for i in 0..45 { + moving.a.write(marked_frame(i, 0xa1)).unwrap(); + assert!(drain_until_idle(&mut exporter, &mut out).await.is_none()); + } + + moving.point(track, broadcast, sps); + assert!(drain_until_idle(&mut exporter, &mut out).await.is_none()); + let next = match broadcast { + None => &mut moving.b, + Some(_) => &mut moving.other, + }; + for i in 45..120 { + moving.a.write(marked_frame(i, 0xa2)).unwrap(); + if i >= 60 { + next.write(marked_frame(i, 0xb1)).unwrap(); + } + assert!(drain_until_idle(&mut exporter, &mut out).await.is_none()); + } + + let expected: Vec = [vec![0xa1; 45], vec![0xb1; 60]].concat(); + assert_eq!(marks(&out), expected, "moved to {track} in {broadcast:?}"); + // Each sequence header's record, and the mark of the frame after it. + let tags = parse_tags(&out); + let headers: Vec<(bool, Option)> = tags + .iter() + .enumerate() + .filter(|(_, tag)| tag.tag_type == 9 && tag.body[1] == 0) + .map(|(index, tag)| { + let main = tag.body.windows(4).any(|w| w == &Moving::MAIN_SPS[..4]); + ( + main, + tags.get(index + 1).and_then(|tag| marks(&tag.body).first().copied()), + ) + }) + .collect(); + let mut announced = vec![(false, Some(0xa1))]; + if sps == Moving::MAIN_SPS { + announced.push((true, Some(0xb1))); + } + assert_eq!(headers, announced, "moved to {track} in {broadcast:?}"); + let timestamps: Vec = tags.iter().map(|tag| tag.timestamp).collect(); + assert!(timestamps.is_sorted(), "{timestamps:?}"); + } +} + /// A bound track that leaves the catalog is read to its own end, not refused as removed: /// a publisher retires a rendition as its track ends, and the catalog update can land first. #[tokio::test(start_paused = true)] diff --git a/rs/moq-mux/src/container/fmp4/export.rs b/rs/moq-mux/src/container/fmp4/export.rs index 0279346be1c..9e0de7e2a85 100644 --- a/rs/moq-mux/src/container/fmp4/export.rs +++ b/rs/moq-mux/src/container/fmp4/export.rs @@ -8,10 +8,10 @@ use mp4_atom::DecodeMaybe; use crate::Result; use crate::catalog::Stream; -use crate::container::ExportSource; use crate::container::Frame; use crate::container::consumer::Event; use crate::container::fmp4::Error; +use crate::container::{ExportSource, Resume}; use moq_net::Timestamp; /// How much media a track may queue while another track waits for its codec configuration. @@ -50,8 +50,10 @@ const INIT_QUEUE: Duration = Duration::from_secs(30); /// (`enabled: false`) does, is unsubscribed and idles. On its return it is resubscribed after /// the last group read, under the track it was declared as, when its sample entry is /// unchanged. That comparison covers the catalog-derived record, so an Annex-B encoder that -/// restarts with a new parameter set can return. A new rendition, a changed sample entry, or -/// a return that replays media already written fails the export. +/// restarts with a new parameter set can return. A rendition the catalog moves to another +/// track (a new `track` or `broadcast`) continues under its declared track the same way, from +/// the new track's first sync sample past what was read. A new rendition, a changed sample +/// entry, or a return that replays media already written fails the export. /// /// [`next_chunk`](Self::next_chunk) returns the same bytes as a [`Chunk`], which /// separates the init segment from a [`Fragment`] carrying whether it begins at a @@ -168,6 +170,14 @@ impl Rendition { } } + /// Whether `export` still reads the track this entry names. + fn read_by(&self, export: &ExportSource, source: &crate::Source, name: &str) -> bool { + match self { + Rendition::Video(config) => export.reads(source, name, config), + Rendition::Audio(config) => export.reads(source, name, config), + } + } + fn is_video(&self) -> bool { matches!(self, Rendition::Video(_)) } @@ -232,8 +242,9 @@ struct Fmp4Track { /// Whether the source has signalled end-of-track, or the rendition left the catalog. finished: bool, - /// The group of the last frame read, where a resubscription resumes after. - group: Option, + /// Where the subscription stopped when the rendition left the catalog, which a + /// resubscription resumes from. + resume: Option, /// The source group closed, so its buffered fragment can be emitted. group_finished: bool, @@ -269,7 +280,7 @@ impl Fmp4Track { buffer: Vec::new(), buffer_independent: false, finished: false, - group: None, + resume: None, group_finished: false, written: None, backlog: None, @@ -306,7 +317,15 @@ impl Fmp4Track { self.describe(name) } - /// Resubscribe a rendition that left the catalog after the init and came back. + /// End what the rendition read from the track it moved off: the frame read ahead is + /// dropped, and the buffered run goes out as a fragment of its own, before the new track's. + fn moved(&mut self) { + self.pending = None; + self.group_finished = !self.buffer.is_empty(); + } + + /// Resubscribe a rendition that left the catalog after the init and came back, or moved + /// to another track. fn rejoin(&mut self, name: &str, config: Rendition, source: ExportSource) -> Result<()> { self.source = Some(source); self.finished = false; @@ -559,7 +578,6 @@ impl Export { while let Some(source) = track.source.as_mut() { match source.poll_event(waiter)? { Poll::Ready(Some(Event::Frame { frame, .. })) => { - track.group = source.group(); // A keyframe may carry new parameter sets, so check it still matches. if !track.described || frame.keyframe { track.describe(name)?; @@ -807,17 +825,35 @@ impl Export { if active.contains(&name) { continue; } + let moved = self + .tracks + .get(&name) + .and_then(|track| track.source.as_ref()) + .is_some_and(|source| !config.read_by(source, &self.source, &name)); + // Before the init nothing of a moved rendition went out, so it starts over as a new one. + if moved && !self.init_emitted { + self.tracks.remove(&name); + self.queued.retain(|(queued, _)| *queued != name); + } match self.tracks.get_mut(&name) { - Some(track) if track.source.is_some() => track.reconfigure(&name, config)?, + Some(track) if track.source.is_some() && !moved => track.reconfigure(&name, config)?, Some(track) => { - let Some(mut source) = config.subscribe(&self.source, &name, self.max_delay)? else { + let Some(source) = config.subscribe(&self.source, &name, self.max_delay)? else { continue; }; - // The publisher still caches the group the rendition left on, which a - // fresh subscription would start at, so resume after the last group read. - if let Some(group) = track.group { - source = source.after(group); - } + // The publisher still caches the group the rendition left on, which a fresh + // subscription would start at, so resume after what was read. A move is new + // content, which continues past what was read under the declared track. + let source = match track.source.take() { + Some(previous) => { + track.moved(); + source.continuing(&previous) + } + None => match &track.resume { + Some(resume) => source.resuming(resume), + None => source, + }, + }; track.rejoin(&name, config, source)?; } None => { @@ -842,7 +878,10 @@ impl Export { // The init declared these tracks, so keep them for a rendition that returns. // Ending the subscription writes out what was already read. for (name, track) in &mut self.tracks { - if !active.contains(name) && track.source.take().is_some() { + if !active.contains(name) + && let Some(source) = track.source.take() + { + track.resume = Some(source.resume()); track.finished = true; track.described = false; } diff --git a/rs/moq-mux/src/container/fmp4/export_test.rs b/rs/moq-mux/src/container/fmp4/export_test.rs index 277f72226a5..58bf7593a87 100644 --- a/rs/moq-mux/src/container/fmp4/export_test.rs +++ b/rs/moq-mux/src/container/fmp4/export_test.rs @@ -2517,3 +2517,72 @@ async fn a_rendition_disabled_after_the_init_idles_and_resumes() { "the re-enabled video resumes: {starts:?}" ); } + +/// A rendition the catalog moves to another track, in its broadcast or a sibling one, writes +/// that track's media from then on under the track id the moov declared, and none of the +/// track it left. The track it left goes out whole first, and neither the timeline nor the +/// sequence numbers step back. +#[tokio::test(start_paused = true)] +async fn a_moved_rendition_continues_on_its_new_track() { + use crate::container::test_util::{Moving, marked_frame, marks}; + + for (track, broadcast) in [("b", None), ("a", Some("other"))] { + let mut moving = Moving::new(); + for i in 0..45 { + moving.a.write(marked_frame(i, 0xa1)).unwrap(); + } + let mut exporter = crate::container::fmp4::Export::new(moving.source().await, moving.catalog_stream().await) + .with_max_delay(RECORDING_MAX_DELAY); + let init = chunk_now(&mut exporter).await.init().expect("init"); + let mut fragments = drain_now(&mut exporter).await; + + moving.point(track, broadcast, SPS); + fragments.extend(drain_now(&mut exporter).await); + let next = match broadcast { + None => &mut moving.b, + Some(_) => &mut moving.other, + }; + // Live, so the new track is read from the group it is writing when the move lands. + for i in 45..120 { + moving.a.write(marked_frame(i, 0xa2)).unwrap(); + if i >= 60 { + next.write(marked_frame(i, 0xb1)).unwrap(); + } + fragments.extend(drain_now(&mut exporter).await); + } + + let written: Vec = fragments.iter().flat_map(|fragment| marks(&fragment.data)).collect(); + let expected: Vec = [vec![0xa1; 45], vec![0xb1; 30]].concat(); + assert_eq!(written, expected, "moved to {track} in {broadcast:?}"); + assert_each_track_ascends(&init, &fragments); + assert_ascending_sequence_numbers(&fragments); + let ids: std::collections::HashSet = + fragment_starts(&init, &fragments).iter().map(|(id, _)| *id).collect(); + assert_eq!(ids.len(), 1, "every fragment is written under the declared track"); + } +} + +/// The moov declared the rendition with its first track's parameter sets, which a move to a +/// track described by another SPS cannot change mid-file. +#[tokio::test(start_paused = true)] +async fn a_rendition_moved_to_another_sample_entry_fails() { + use crate::container::test_util::{Moving, marked_frame}; + + let mut moving = Moving::new(); + for i in 0..30 { + moving.a.write(marked_frame(i, 0xa1)).unwrap(); + } + let mut exporter = crate::container::fmp4::Export::new(moving.source().await, moving.catalog_stream().await) + .with_max_delay(RECORDING_MAX_DELAY); + chunk_now(&mut exporter).await.init().expect("init"); + + moving.point("b", None, Moving::MAIN_SPS); + for i in 30..60 { + moving.b.write(marked_frame(i, 0xb1)).unwrap(); + } + let err = error_now(&mut exporter).await; + assert!( + matches!(&err, crate::Error::Cmaf(crate::container::fmp4::Error::TrackChanged(name)) if name == Moving::RENDITION), + "{err:?}" + ); +} diff --git a/rs/moq-mux/src/container/mkv/export.rs b/rs/moq-mux/src/container/mkv/export.rs index 2b129d434d1..aac96760e74 100644 --- a/rs/moq-mux/src/container/mkv/export.rs +++ b/rs/moq-mux/src/container/mkv/export.rs @@ -45,6 +45,11 @@ const TIMESTAMP_SCALE_NS: u64 = 1_000_000; /// /// Only Legacy-container tracks (raw codec payloads) are supported. CMAF tracks /// (moof+mdat passthrough) are rejected with a clear error. +/// +/// A rendition the catalog moves to another track (a new `track` or `broadcast`) goes on +/// under its track number from the new track's first sync frame past what was read, when +/// the header's TrackEntry still describes it; otherwise, or when it goes back in time, the +/// export fails. pub struct Export { source: crate::Source, catalog: Option, @@ -80,9 +85,74 @@ struct MkvTrack { finished: bool, track_number: u64, kind: TrackKind, + /// The TrackEntry the header declared. + entry: Option, + /// The source replaced the one the header was built from, so its frames wait until it + /// can be checked against `entry`. + moved: bool, } impl MkvTrack { + /// A track numbered `track_number` reading `source`, with the video `config` it was added under. + fn new(source: ExportSource, kind: TrackKind, config: Option, track_number: u64) -> Self { + Self { + clock: Default::default(), + config, + restart: 0, + reorder: None, + source, + pending: None, + subscribed: moq_clock::now(), + latest: moq_clock::Instant::ZERO, + finished: false, + track_number, + kind, + entry: None, + moved: false, + } + } + + /// Read `source`, the track the catalog moved this rendition to, after releasing the + /// frames the old one left held for their DTS. + fn rebind( + &mut self, + name: &str, + source: ExportSource, + jitter: &mut crate::jitter::Buffer, + ) -> Result<()> { + self.release(name, jitter, true)?; + self.clock = Default::default(); + self.reorder = None; + self.source = source.continuing(&self.source); + self.finished = false; + self.subscribed = moq_clock::now(); + self.moved = true; + Ok(()) + } + + /// The TrackEntry declaring this track under its catalog entry in `catalog`. + fn track_entry(&self, catalog: &Catalog, name: &str) -> Result { + match self.kind { + TrackKind::Video => { + let config = catalog + .video + .renditions + .get(name) + .ok_or_else(|| Error::MissingVideoTrack(name.to_string()))?; + let config = self.source.video_config(config).unwrap_or_else(|| config.clone()); + build_video_track_entry(self.track_number, &config, self.source.description()) + } + TrackKind::Audio => { + let config = catalog + .audio + .renditions + .get(name) + .ok_or_else(|| Error::MissingAudioTrack(name.to_string()))?; + build_audio_track_entry(self.track_number, config) + } + } + } + fn release(&mut self, name: &str, jitter: &mut crate::jitter::Buffer, flush: bool) -> Result<()> { let Some(config) = &self.config else { return Ok(()) }; let (floor, lookahead) = *self.reorder.get_or_insert_with(|| { @@ -321,9 +391,8 @@ impl Export { return Poll::Ready(Ok(None)); } - // 7. Drop finished, drained tracks so the next catalog update can re-add them. - self.tracks.retain(|_, t| !(t.finished && t.pending.is_none())); - + // A finished track keeps its place: the header declared it, and the catalog may still + // list it or move it to another track. Poll::Pending } @@ -376,9 +445,20 @@ impl Export { .as_ref() .and_then(|catalog| catalog.video.renditions.get(name)) .is_some_and(|config| track.source.video_geometry_ready(config)); - if !self.header_emitted && (!track.source.header_ready() || !geometry_ready) { + let declaring = !self.header_emitted || track.moved; + if declaring && (!track.source.header_ready() || !geometry_ready) { continue; } + // A moved rendition's track must be the one the header declared, before any of + // its frames go out. + if track.moved + && let Some(catalog) = self.catalog_snapshot.as_ref() + { + if Some(track.track_entry(catalog, name)?) != track.entry { + return Err(Error::TrackChanged(name.clone()).into()); + } + track.moved = false; + } crate::jitter::Arrival { arrived, read: moq_clock::now(), @@ -418,11 +498,14 @@ impl Export { fn update_catalog(&mut self, mut catalog: Catalog) -> Result<()> { self.source.retain_selectable(&mut catalog); + // Checked on every listing, so a rendition moved to a CMAF track is refused too. let mut active: HashMap = HashMap::new(); - for name in catalog.video.renditions.keys() { + for (name, config) in &catalog.video.renditions { + ensure_legacy(&config.container, "video", name)?; active.insert(name.clone(), ()); } - for name in catalog.audio.renditions.keys() { + for (name, config) in &catalog.audio.renditions { + ensure_legacy(&config.container, "audio", name)?; active.insert(name.clone(), ()); } @@ -438,8 +521,21 @@ impl Export { return Err(Error::HeaderAddedTrack(name.clone()).into()); } } + for (name, config) in &catalog.audio.renditions { + if let Some(track) = self.tracks.get_mut(name) + && !track.source.reads(&self.source, name, config) + && let Some(source) = ExportSource::for_audio(&self.source, name, config, self.delay)? + { + track.rebind(name, source, &mut self.jitter)?; + } + } for (name, config) in &catalog.video.renditions { if let Some(track) = self.tracks.get_mut(name) { + if !track.source.reads(&self.source, name, config) + && let Some(source) = ExportSource::for_video(&self.source, name, config, self.delay)? + { + track.rebind(name, source, &mut self.jitter)?; + } if track .config .as_ref() @@ -454,32 +550,26 @@ impl Export { return Ok(()); } + // Nothing went out yet, so a rendition that left the catalog is dropped, and one the + // catalog moved to another track starts over. + self.tracks.retain(|name, track| { + match (catalog.video.renditions.get(name), catalog.audio.renditions.get(name)) { + (Some(config), _) => track.source.reads(&self.source, name, config), + (None, Some(config)) => track.source.reads(&self.source, name, config), + (None, None) => false, + } + }); let mut next_track_number: u64 = self.tracks.values().map(|t| t.track_number).max().unwrap_or(0) + 1; for (name, config) in catalog.video.renditions.iter() { if self.tracks.contains_key(name) { continue; } - ensure_legacy(&config.container, "video", name)?; let Some(source) = ExportSource::for_video(&self.source, name, config, self.delay)? else { continue; }; - self.tracks.insert( - name.clone(), - MkvTrack { - clock: Default::default(), - config: Some(config.clone()), - restart: 0, - reorder: None, - source, - pending: None, - subscribed: moq_clock::now(), - latest: moq_clock::Instant::ZERO, - finished: false, - track_number: next_track_number, - kind: TrackKind::Video, - }, - ); + let track = MkvTrack::new(source, TrackKind::Video, Some(config.clone()), next_track_number); + self.tracks.insert(name.clone(), track); next_track_number += 1; } @@ -487,30 +577,14 @@ impl Export { if self.tracks.contains_key(name) { continue; } - ensure_legacy(&config.container, "audio", name)?; let Some(source) = ExportSource::for_audio(&self.source, name, config, self.delay)? else { continue; }; - self.tracks.insert( - name.clone(), - MkvTrack { - clock: Default::default(), - config: None, - restart: 0, - reorder: None, - source, - pending: None, - subscribed: moq_clock::now(), - latest: moq_clock::Instant::ZERO, - finished: false, - track_number: next_track_number, - kind: TrackKind::Audio, - }, - ); + let track = MkvTrack::new(source, TrackKind::Audio, None, next_track_number); + self.tracks.insert(name.clone(), track); next_track_number += 1; } - self.tracks.retain(|name, _| active.contains_key(name)); self.catalog_snapshot = Some(catalog); Ok(()) } @@ -535,7 +609,7 @@ impl Export { }) } - fn build_header(&self) -> Result { + fn build_header(&mut self) -> Result { let catalog = self.catalog_snapshot.as_ref().ok_or(Error::NoCatalogSnapshot)?; // Decide DocType: webm only if every codec is WebM-allowed. @@ -552,24 +626,16 @@ impl Export { let doc_type = if webm_only { "webm" } else { "matroska" }; let mut entries: Vec = Vec::new(); - for (name, config) in catalog.video.renditions.iter() { - let track = self - .tracks - .get(name) - .ok_or_else(|| Error::MissingVideoTrack(name.clone()))?; - let config = track.source.video_config(config).unwrap_or_else(|| config.clone()); - entries.push(build_video_track_entry( - track.track_number, - &config, - track.source.description(), - )?); - } - for (name, config) in catalog.audio.renditions.iter() { - let track = self - .tracks - .get(name) - .ok_or_else(|| Error::MissingAudioTrack(name.clone()))?; - entries.push(build_audio_track_entry(track.track_number, config)?); + let video = catalog.video.renditions.keys().map(|name| (name, TrackKind::Video)); + let audio = catalog.audio.renditions.keys().map(|name| (name, TrackKind::Audio)); + for (name, kind) in video.chain(audio) { + let track = self.tracks.get_mut(name).ok_or_else(|| match kind { + TrackKind::Video => Error::MissingVideoTrack(name.clone()), + TrackKind::Audio => Error::MissingAudioTrack(name.clone()), + })?; + let entry = track.track_entry(catalog, name)?; + track.entry = Some(entry.clone()); + entries.push(entry); } let mut dest = Cursor::new(Vec::new()); diff --git a/rs/moq-mux/src/container/mkv/export_test.rs b/rs/moq-mux/src/container/mkv/export_test.rs index e1d418735ac..68497b2dec3 100644 --- a/rs/moq-mux/src/container/mkv/export_test.rs +++ b/rs/moq-mux/src/container/mkv/export_test.rs @@ -756,6 +756,115 @@ async fn chunk_now(exporter: &mut crate::container::mkv::Export) -> crate::Result> { + let mut out = Vec::new(); + while let Ok(next) = tokio::time::timeout(std::time::Duration::from_millis(1), exporter.next()).await { + match next? { + Some(chunk) => out.extend_from_slice(&chunk), + None => break, + } + } + Ok(out) +} + +/// A rendition the catalog moves to another track, in its broadcast or a sibling one, writes +/// that track's media from then on under the track number the header declared, and none of +/// the track it left. +#[tokio::test(start_paused = true)] +async fn a_moved_rendition_continues_on_its_new_track() { + use crate::container::test_util::{Moving, marked_frame, marks}; + + for (track, broadcast) in [("b", None), ("a", Some("other"))] { + let mut moving = Moving::new(); + let mut exporter = crate::container::mkv::Export::new(moving.source().await, moving.catalog_stream().await); + let mut out = Vec::new(); + for i in 0..45 { + moving.a.write(marked_frame(i, 0xa1)).unwrap(); + out.extend(drain_now(&mut exporter).await.unwrap()); + } + + moving.point(track, broadcast, SPS); + out.extend(drain_now(&mut exporter).await.unwrap()); + let next = match broadcast { + None => &mut moving.b, + Some(_) => &mut moving.other, + }; + for i in 45..120 { + moving.a.write(marked_frame(i, 0xa2)).unwrap(); + if i >= 60 { + next.write(marked_frame(i, 0xb1)).unwrap(); + } + out.extend(drain_now(&mut exporter).await.unwrap()); + } + + let expected: Vec = [vec![0xa1; 45], vec![0xb1; 30]].concat(); + assert_eq!(marks(&out), expected, "moved to {track} in {broadcast:?}"); + let numbers: std::collections::HashSet = WebmIterator::new(&mut Cursor::new(out), &[]) + .filter_map(|tag| match tag { + Ok(MatroskaSpec::SimpleBlock(data)) => Some(SimpleBlock::try_from(&data).unwrap().track), + _ => None, + }) + .collect(); + assert_eq!(numbers.len(), 1, "every block is written under the declared track"); + } +} + +/// The header's TrackEntry carries the rendition's first avcC, which a move to a track +/// described by another SPS cannot change mid-file. It fails before writing any of that track. +#[tokio::test(start_paused = true)] +async fn a_rendition_moved_to_another_codec_configuration_fails() { + use crate::container::test_util::{Moving, marked_frame, marks}; + + let mut moving = Moving::new(); + let mut exporter = crate::container::mkv::Export::new(moving.source().await, moving.catalog_stream().await); + let mut out = Vec::new(); + for i in 0..30 { + moving.a.write(marked_frame(i, 0xa1)).unwrap(); + out.extend(drain_now(&mut exporter).await.unwrap()); + } + + moving.point("b", None, crate::container::test_util::Moving::MAIN_SPS); + out.extend(drain_now(&mut exporter).await.unwrap()); + moving.b.write(marked_frame(30, 0xb1)).unwrap(); + let err = drain_now(&mut exporter).await.unwrap_err(); + assert!( + matches!(&err, crate::Error::Mkv(crate::container::mkv::Error::TrackChanged(name)) if name == Moving::RENDITION), + "{err:?}" + ); + assert!(!marks(&out).contains(&0xb1)); +} + +/// A move to a CMAF track is refused like a CMAF rendition listed from the start. +#[tokio::test(start_paused = true)] +async fn a_rendition_moved_to_a_cmaf_track_fails() { + use crate::container::test_util::{Moving, marked_frame}; + + let mut moving = Moving::new(); + let mut exporter = crate::container::mkv::Export::new(moving.source().await, moving.catalog_stream().await); + for i in 0..30 { + moving.a.write(marked_frame(i, 0xa1)).unwrap(); + drain_now(&mut exporter).await.unwrap(); + } + + let mut config = Moving::config("b", None, SPS); + config.container = hang::catalog::Container::Cmaf { + init: Bytes::from(vec![0u8; 32]), + }; + moving + .catalog + .modify() + .unwrap() + .video + .renditions + .insert(Moving::RENDITION.to_string(), config); + let err = drain_now(&mut exporter).await.unwrap_err(); + assert!( + matches!(&err, crate::Error::Mkv(crate::container::mkv::Error::UnsupportedCmafTrack { name, .. }) if name == Moving::RENDITION), + "{err:?}" + ); +} + /// SimpleBlock count of one cluster, parsed behind the file header. fn count_blocks(header: &[u8], cluster: &[u8]) -> usize { let mut cursor = Cursor::new([header, cluster].concat()); diff --git a/rs/moq-mux/src/container/mkv/mod.rs b/rs/moq-mux/src/container/mkv/mod.rs index ba7e49bf070..349948639f7 100644 --- a/rs/moq-mux/src/container/mkv/mod.rs +++ b/rs/moq-mux/src/container/mkv/mod.rs @@ -100,6 +100,10 @@ pub enum Error { #[error("MKV track layout changed after header was emitted: track '{0}' added")] HeaderAddedTrack(String), + /// The named rendition moved to a track the header's TrackEntry does not describe. + #[error("MKV track '{0}' moved to a track whose codec configuration differs from the header's")] + TrackChanged(String), + /// The named CMAF track cannot be exported as MKV. #[error("MKV export does not support CMAF {kind} track '{name}'")] UnsupportedCmafTrack { diff --git a/rs/moq-mux/src/container/mod.rs b/rs/moq-mux/src/container/mod.rs index 09d7bf4dc26..b51bb2b71e5 100644 --- a/rs/moq-mux/src/container/mod.rs +++ b/rs/moq-mux/src/container/mod.rs @@ -32,7 +32,7 @@ pub mod ts; pub use consumer::Consumer; pub use group::GroupConsumer; pub use producer::Producer; -pub(crate) use source::ExportSource; +pub(crate) use source::{ExportSource, Resume}; /// The media role that determines how a container represents frame durations. #[derive(Clone, Copy, Debug, PartialEq, Eq)] diff --git a/rs/moq-mux/src/container/source.rs b/rs/moq-mux/src/container/source.rs index eb6d0f3d274..62214536d9d 100644 --- a/rs/moq-mux/src/container/source.rs +++ b/rs/moq-mux/src/container/source.rs @@ -14,8 +14,11 @@ //! a record derived from the codec string, or an avcC/hvcC synthesized from //! in-band parameter sets. +use std::ops::Bound; use std::task::{Poll, ready}; +use std::time::Duration; + use bytes::Bytes; use hang::catalog::{AudioConfig, Container, Reference, VideoCodec, VideoConfig}; @@ -66,18 +69,63 @@ enum SourceState { Ended, } +/// The track a source reads: its broadcast's path, the publisher instance that broadcast must +/// be, and the track's name there. +#[derive(Clone, Debug, PartialEq, Eq)] +struct Target { + path: moq_net::PathOwned, + epoch: Option, + track: String, +} + +impl Target { + /// Where rendition `name` reads, or `None` when its reference escapes the root. + fn of(source: &crate::Source, name: &str, config: &impl Reference) -> Option { + Some(Self { + path: source.resolve_reference(config.broadcast())?, + epoch: config.epoch().cloned(), + track: config.track(name).to_string(), + }) + } + + fn request(&self, source: &crate::Source) -> SourceState { + let binding = source.request_path(&self.path, self.epoch.as_ref()); + SourceState::Requesting(binding, self.track.clone()) + } +} + +/// Where a rendition's source stopped reading ([`ExportSource::resume`]). +pub(crate) struct Resume { + target: Target, + /// The group of the last frame delivered. + group: Option, + /// The latest presentation time delivered. + latest: Option, +} + +/// What a source continuing another rendition's track skips: media up to what that track delivered. +struct Splice { + /// The latest presentation time the previous track delivered. + read: Duration, + /// The group of the first frame read, the only one that may overlap `read`. + group: Option, +} + /// A per-rendition source that normalizes frame shape (Annex-B → /// length-prefixed for H.264/H.265) and exposes the resolved codec config /// record alongside the frame stream. pub(crate) struct ExportSource { + /// The rendition's ID in the catalog. + name: String, + target: Target, state: SourceState, /// Wire format, consumed when the subscription resolves into a consumer. media: Option, max_delay: std::time::Duration, /// Start at the newest group, rather than the oldest the max delay still reaches. live: bool, - /// The first group to read, so a resubscription skips groups already read. - start: Option, + /// Read only groups after this one, so a resubscription skips groups already read. + after: Option, transform: Option, /// Resolved codec configuration record (avcC / hvcC / AudioSpecificConfig / /// OpusHead). Some once the codec config is available — from the catalog @@ -99,6 +147,12 @@ pub(crate) struct ExportSource { ends: Option<(u64, u64)>, /// The counters the export source reports for this rendition, as received, before any transform. received: std::sync::Arc, + /// The latest presentation time delivered, here or by the source this one resumes. + latest: Option, + /// The group of the last frame delivered. + delivered: Option, + /// Set while continuing another source, until the first sync frame past what it delivered. + splice: Option, /// The sibling broadcast the rendition reads, once resolved: its end ends the rendition. instance: Option, } @@ -166,32 +220,17 @@ impl ExportSource { transform: Option, ) -> Result, crate::Error> { let media: HangContainer = config.try_into()?; - let description = config.description.as_ref().filter(|b| !b.is_empty()).cloned(); - let Some(request) = source.try_request(config) else { + let Some(target) = Target::of(source, name, config) else { return Ok(None); }; - let mut source = Self { - state: SourceState::Requesting(request, config.track(name).to_string()), - media: Some(media), - max_delay, - live: false, - start: None, - transform, - description, - video_codec: Some(config.codec.clone()), - video_dimensions: catalog_dimensions(config), - absorbed_keyframe: false, - skips: 0, - markers: 0, - open: false, - cuts: 0, - ends: None, - received: source.received(name), - instance: None, - }; - source.resolve_video_dimensions(&[])?; - Ok(Some(source)) + let mut export = Self::new(source, name, target, media, max_delay); + export.transform = transform; + export.description = config.description.as_ref().filter(|b| !b.is_empty()).cloned(); + export.video_codec = Some(config.codec.clone()); + export.video_dimensions = catalog_dimensions(config); + export.resolve_video_dimensions(&[])?; + Ok(Some(export)) } /// Subscribe to an audio rendition. Audio has no codec-shape transform; @@ -205,30 +244,13 @@ impl ExportSource { max_delay: std::time::Duration, ) -> Result, crate::Error> { let media: HangContainer = config.try_into()?; - let description = config.description.as_ref().filter(|b| !b.is_empty()).cloned(); - let Some(request) = source.try_request(config) else { + let Some(target) = Target::of(source, name, config) else { return Ok(None); }; - Ok(Some(Self { - state: SourceState::Requesting(request, config.track(name).to_string()), - media: Some(media), - max_delay, - live: false, - start: None, - transform: None, - description, - video_codec: None, - video_dimensions: None, - absorbed_keyframe: false, - skips: 0, - markers: 0, - open: false, - cuts: 0, - ends: None, - received: source.received(name), - instance: None, - })) + let mut export = Self::new(source, name, target, media, max_delay); + export.description = config.description.as_ref().filter(|b| !b.is_empty()).cloned(); + Ok(Some(export)) } /// Subscribe to a verbatim `mpegts` stream rendition (SCTE-35, private PES, ...). @@ -242,12 +264,25 @@ impl ExportSource { name: &str, max_delay: std::time::Duration, ) -> Result { - Ok(Self { - state: SourceState::Requesting(source.request_catalog(), name.to_string()), - media: Some(HangContainer::Legacy(crate::container::Kind::Data)), + let target = Target { + path: source.path().clone(), + epoch: None, + track: name.to_string(), + }; + let media = HangContainer::Legacy(crate::container::Kind::Data); + Ok(Self::new(source, name, target, media, max_delay)) + } + + /// A source of rendition `name` reading `target`, with no codec configuration yet. + fn new(source: &crate::Source, name: &str, target: Target, media: HangContainer, max_delay: Duration) -> Self { + Self { + name: name.to_string(), + state: target.request(source), + target, + media: Some(media), max_delay, live: false, - start: None, + after: None, transform: None, description: None, video_codec: None, @@ -259,8 +294,55 @@ impl ExportSource { cuts: 0, ends: None, received: source.received(name), + latest: None, + delivered: None, + splice: None, instance: None, - }) + } + } + + /// Whether this source reads the track `config` names for rendition `name`. A catalog update + /// that moves the rendition to another track or broadcast needs a new source. + pub(crate) fn reads(&self, source: &crate::Source, name: &str, config: &impl Reference) -> bool { + Target::of(source, name, config).is_some_and(|target| target == self.target) + } + + /// Carry on from `previous`, the source this rendition read before the catalog moved it to + /// this source's track, as [`Self::resuming`] does. Its counters run on, the move counting as + /// a skip that cuts the group `previous` was reading, if any. + pub(crate) fn continuing(mut self, previous: &ExportSource) -> Self { + let ends = previous.ends.map_or(0, |(_, count)| count); + self.skips = previous.skips + ends + 1; + self.markers = previous.markers + ends; + self.cuts = previous.cuts + u64::from(previous.open); + self.resuming(&previous.resume()) + } + + /// Where this source stopped reading, for a later source of the same rendition. + pub(crate) fn resume(&self) -> Resume { + Resume { + target: self.target.clone(), + group: self.delivered, + latest: self.latest, + } + } + + /// Pick up where an earlier source of this rendition stopped. + /// + /// On the same track, start after the last group it delivered, so the publisher's cached + /// group is not replayed. Another track is new content, read from its live edge, and the + /// rendition never goes back in time: frames are dropped until a sync frame past the latest + /// one delivered. On a shared clock the first group read may overlap that, but a later group + /// that still does fails with [`Error::RenditionRewound`](crate::Error::RenditionRewound). + pub(crate) fn resuming(mut self, resume: &Resume) -> Self { + if resume.target == self.target { + self.after = resume.group; + } else { + self.live = true; + self.splice = resume.latest.map(|read| Splice { read, group: None }); + } + self.latest = resume.latest; + self } /// Start at the newest group, the live edge, rather than the oldest one the max delay still @@ -272,13 +354,6 @@ impl ExportSource { self } - /// Start after `group`, so a resubscription never replays a group already read, such as - /// the one the publisher still caches from before the rendition left. - pub(crate) fn after(mut self, group: u64) -> Self { - self.start = Some(group + 1); - self - } - /// The group sequence of the last frame read, or None until the subscription resolves. /// Meaningful only once a frame was read. pub(crate) fn group(&self) -> Option { @@ -371,6 +446,51 @@ impl ExportSource { /// Read normalized media or a clean group boundary. pub fn poll_event(&mut self, waiter: &kio::Waiter) -> Poll>> { + loop { + let event = ready!(self.poll_track(waiter))?; + match &event { + Some(Event::Frame { frame, .. }) => { + if !self.spliced(frame)? { + continue; + } + let at = Duration::from(frame.timestamp); + self.latest = Some(self.latest.map_or(at, |latest| latest.max(at))); + self.delivered = self.group(); + } + // The boundaries of frames the splice dropped. + Some(_) if self.splice.is_some() => continue, + _ => {} + } + return Poll::Ready(Ok(event)); + } + } + + /// Whether `frame` goes out, rather than being dropped as media the previous track of a + /// [continued](Self::continuing) rendition already covered. + fn spliced(&mut self, frame: &Frame) -> crate::Result { + let group = self.group(); + let sync = frame.keyframe || self.video_codec.is_none(); + let Some(splice) = self.splice.as_mut() else { + return Ok(true); + }; + let at = Duration::from(frame.timestamp); + if sync && at > splice.read { + self.splice = None; + return Ok(true); + } + let first = *splice.group.get_or_insert_with(|| group.unwrap_or_default()); + if frame.keyframe && group != Some(first) { + return Err(crate::Error::RenditionRewound { + rendition: self.name.clone(), + start: at, + read: splice.read, + }); + } + Ok(false) + } + + /// Read the track's normalized media or a clean group boundary. + fn poll_track(&mut self, waiter: &kio::Waiter) -> Poll>> { if matches!(self.state, SourceState::Ended) { return Poll::Ready(Ok(None)); } @@ -387,8 +507,8 @@ impl ExportSource { self.instance = Some(broadcast.clone()); } let mut subscription = moq_net::track::Subscription::default().with_max_delay(self.max_delay); - if let Some(start) = self.start { - subscription = subscription.with_groups(start..); + if let Some(after) = self.after { + subscription = subscription.with_groups((Bound::Excluded(after), Bound::Unbounded)); } let track = match broadcast.track(&name) { Ok(track) => track, @@ -1089,6 +1209,59 @@ mod tests { } } + /// A source continuing a rendition moved off a track that delivered up to `read` starts at + /// the first keyframe past it. Only the first group read may overlap: a later group still + /// behind it is a track that goes back in time. + #[tokio::test(start_paused = true)] + async fn a_continued_source_starts_past_what_its_previous_track_delivered() { + use crate::container::test_util::{Moving, SPS, marked_frame}; + + for (read, first) in [(900, Some(1_000_000)), (1_500, None)] { + let mut moving = Moving::new(); + let source = moving.source().await; + let max_delay = Duration::from_secs(10); + let mut previous = + ExportSource::for_video(&source, Moving::RENDITION, &Moving::config("a", None, SPS), max_delay) + .unwrap() + .unwrap(); + previous.skips = 2; + previous.latest = Some(Duration::from_millis(read)); + let mut next = + ExportSource::for_video(&source, Moving::RENDITION, &Moving::config("b", None, SPS), max_delay) + .unwrap() + .unwrap() + .continuing(&previous); + let mut poll = async || { + tokio::time::timeout(Duration::from_millis(100), kio::wait(|waiter| next.poll_read(waiter))).await + }; + + // A keyframe at zero then deltas up to 967 ms, past 900 ms but no place to start. + for i in 0..30 { + moving.b.write(marked_frame(i, 0xb1)).unwrap(); + } + assert!(poll().await.is_err(), "the first group overlaps: read {read}"); + for i in 30..60 { + moving.b.write(marked_frame(i, 0xb1)).unwrap(); + } + match (poll().await.unwrap(), first) { + (Ok(Some(frame)), Some(first)) => assert_eq!(frame.timestamp.as_micros(), first), + ( + Err(crate::Error::RenditionRewound { + rendition, + start, + read: behind, + }), + None, + ) => { + assert_eq!(rendition, Moving::RENDITION); + assert_eq!((start, behind), (Duration::from_secs(1), Duration::from_millis(read))); + } + (result, _) => panic!("read {read}: {result:?}"), + } + assert_eq!(next.skips(), 3, "the move counts as a skip"); + } + } + async fn audio_export() -> (moq_net::track::Producer, ExportSource) { use std::time::Duration; let live = Live::audio(audio(None)); diff --git a/rs/moq-mux/src/container/test_util.rs b/rs/moq-mux/src/container/test_util.rs index 48155d9f4e7..db47274611a 100644 --- a/rs/moq-mux/src/container/test_util.rs +++ b/rs/moq-mux/src/container/test_util.rs @@ -97,6 +97,110 @@ impl Live { } } +/// A catalog broadcast `live` whose H.264 rendition [`Moving::RENDITION`] the catalog can +/// move between tracks: `a` and `b` in `live`, and `a` in the sibling broadcast `other`. +/// Every frame carries a mark byte, so an export shows which track it read ([`marks`]). +pub(crate) struct Moving { + pub(crate) catalog: crate::catalog::Producer, + /// `live`'s track `a`, the one the rendition starts on. + pub(crate) a: crate::container::Producer, + /// `live`'s track `b`. + pub(crate) b: crate::container::Producer, + /// `other`'s track `a`. + pub(crate) other: crate::container::Producer, + origin: moq_net::origin::Producer, + _broadcasts: [moq_net::broadcast::Producer; 2], +} + +impl Moving { + pub(crate) const RENDITION: &str = "main"; + /// A second SPS, a Main profile one, whose avcC differs from [`SPS`]'s. + pub(crate) const MAIN_SPS: &[u8] = &[0x67, 0x4d, 0x40, 0x1f, 0xde, 0xad, 0xbe, 0xef]; + + pub(crate) fn new() -> Self { + let origin = crate::source::produce_origin(); + let mut live = origin.publish("live", Default::default()).unwrap(); + let mut other = origin.publish("other", Default::default()).unwrap(); + let catalog = crate::catalog::Producer::new(&mut live, crate::catalog::Config::default()).unwrap(); + let track = |broadcast: &mut moq_net::broadcast::Producer, name: &str| { + let info = hang::container::track_info(hang::catalog::PRIORITY.video); + let track = broadcast.create_track(name, info).unwrap(); + let wire = crate::catalog::hang::Container::Legacy(crate::container::Kind::Video); + crate::container::Producer::new(track, wire) + }; + let mut moving = Self { + a: track(&mut live, "a"), + b: track(&mut live, "b"), + other: track(&mut other, "a"), + catalog, + origin, + _broadcasts: [live, other], + }; + moving.point("a", None, SPS); + moving + } + + pub(crate) async fn source(&self) -> crate::Source { + crate::source::resolved(&self.origin, "live").await + } + + pub(crate) async fn catalog_stream(&self) -> crate::catalog::Consumer { + let source = self.source().await; + source.catalog::<()>(crate::catalog::CatalogFormat::Hang).await.unwrap() + } + + /// Point the rendition at `track` in `broadcast` (`live` when `None`), described by `sps`. + pub(crate) fn point(&mut self, track: &str, broadcast: Option<&str>, sps: &'static [u8]) { + let config = Self::config(track, broadcast, sps); + let mut catalog = self.catalog.modify().unwrap(); + catalog.video.renditions.insert(Self::RENDITION.to_string(), config); + } + + /// The rendition's catalog entry reading `track` in `broadcast`, described by `sps`. + pub(crate) fn config(track: &str, broadcast: Option<&str>, sps: &'static [u8]) -> VideoConfig { + let mut config = VideoConfig::new(H264 { + profile: sps[1], + constraints: sps[2], + level: sps[3], + inline: false, + }); + config.coded_width = Some(320); + config.coded_height = Some(240); + config.framerate = Some(30.0); + config.container = Container::Legacy; + config.description = + Some(crate::codec::h264::build_avcc(&[Bytes::from_static(sps)], &[Bytes::from_static(PPS)]).unwrap()); + config.track = Some(track.to_string()); + config.broadcast = broadcast.map(|broadcast| moq_net::path::Relative::new(broadcast).into_owned()); + config + } +} + +/// One 30 fps length-prefixed H.264 frame `index` frames in, a keyframe opening each second, +/// whose slice carries `mark`. +pub(crate) fn marked_frame(index: u64, mark: u8) -> crate::container::Frame { + let keyframe = index.is_multiple_of(30); + let nal = [if keyframe { 0x65 } else { 0x41 }, 0x88, mark, mark, mark, mark, 0x80]; + let mut payload = BytesMut::new(); + payload.extend_from_slice(&(nal.len() as u32).to_be_bytes()); + payload.extend_from_slice(&nal); + crate::container::Frame { + timestamp: Timestamp::from_micros(index * 1_000_000 / 30).unwrap(), + payload: payload.freeze(), + keyframe, + duration: None, + } +} + +/// The mark of each [`marked_frame`] slice in `bytes`, in order. +pub(crate) fn marks(bytes: &[u8]) -> Vec { + bytes + .windows(6) + .filter(|w| matches!(w[0], 0x65 | 0x41) && w[1] == 0x88 && w[2] != 0 && w[2..].iter().all(|b| *b == w[2])) + .map(|w| w[2]) + .collect() +} + /// H.264 NALs used by [`video_frame`], exposed for tests that assert on them. pub(crate) const SPS: &[u8] = &[0x67, 0x42, 0xc0, 0x1f, 0xde, 0xad, 0xbe, 0xef]; pub(crate) const PPS: &[u8] = &[0x68, 0xce, 0x3c, 0x80]; diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index b61bc65f93c..5a731581ed3 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -110,6 +110,10 @@ fn sanitize_mux_rate(rate: u64) -> Option { /// leading PAT/PMT rides on the first frame (so it inherits a real timestamp), and /// is re-emitted at video keyframes and periodically for mid-stream tune-in. /// Returns `None` when the broadcast ends. +/// +/// A rendition the catalog moves to another track (a new `track` or `broadcast`) goes on out +/// of its PID from the new track's first sync frame past what was read, when the PMT can +/// announce it unchanged; one that goes back in time fails the export. pub struct Export where S::Ext: catalog::Catalog, @@ -1031,7 +1035,7 @@ impl Timing { } } -#[derive(Clone)] +#[derive(Clone, PartialEq)] enum Kind { /// Video carries its TS stream type (H.264 = 0x1B, H.265 = 0x24). Video(StreamType), @@ -1060,6 +1064,37 @@ enum Kind { } impl Kind { + /// The `stream_type` the PMT announces it as. + fn stream_type(&self) -> u8 { + match self { + Kind::Video(stream_type) => *stream_type as u8, + Kind::Aac { .. } => StreamType::AdtsAac as u8, + // Opus rides private-data PES; the registration + extension descriptors tell the + // demuxer it's Opus. + Kind::Opus { .. } => 0x06, + // Half-rate MPEG-2 BC audio (< 32 kHz) re-announces as 0x04; the full rates are + // MPEG-1 (0x03). The catalog sample rate came from the frame header, so the mapping + // is faithful. + Kind::Mp2 { sample_rate } if *sample_rate < 32000 => StreamType::Mpeg2HalvedSampleRateAudio as u8, + Kind::Mp2 { .. } => StreamType::Mpeg1Audio as u8, + Kind::Ac3 => StreamType::DolbyDigitalUpToSixChannelAudio as u8, + Kind::Eac3 => StreamType::DolbyDigitalPlusUpTo16ChannelAudioForAtsc as u8, + Kind::Verbatim { stream_type, .. } | Kind::PesPacket { stream_type } => *stream_type, + } + } + + /// The PMT ES descriptors announcing it when import captured none: the ATSC Dolby + /// registration, so a fresh (non-TS) AC-3/E-AC-3 track is announced the way the import + /// path expects, and Opus's registration and channel configuration. + fn descriptors(&self) -> Vec { + match self { + Kind::Ac3 => vec![registration(b"AC-3")], + Kind::Eac3 => vec![registration(b"EAC3")], + Kind::Opus { channel_config_code } => opus_descriptors(*channel_config_code), + _ => Vec::new(), + } + } + /// Its timestamp is arrival on the program clock, rather than a decode deadline. fn placed(&self) -> bool { matches!(self, Self::PesPacket { .. } | Self::Verbatim { .. }) @@ -1951,10 +1986,11 @@ where if !self.stale.is_empty() { self.resubscribe(&catalog, &mpegts)?; } + self.rebind(&catalog)?; // A listed track keeps its PID, kind and descriptors, which its packets went out - // under: a stream the source revises comes under a new name. The timing is not - // fixed: a `jitter` or `framerate` published after the tables still sizes the - // decode clock. + // under, even when the catalog moves it to another track: a stream the source + // revises comes under a new name. The timing is not fixed: a `jitter` or + // `framerate` published after the tables still sizes the decode clock. for (name, config) in catalog.video.renditions.iter() { if let Some(track) = self.tracks.get_mut(name).filter(|track| track.listed) { track.timing.configure(config, name); @@ -2029,9 +2065,9 @@ where reject_colliding_si_pids(&mpegts, pmt_pid, &es_pids)?; // Reuse each joining track's source (and any pending frame) by name; refresh its PID, - // kind, and descriptors from this snapshot. Before the tables, drop tracks no longer - // present; after them, a track the catalog dropped stays until what it holds is gone - // ([`Self::settle`]). + // kind, and descriptors from this snapshot, and start over on one the catalog moved to + // another track. Before the tables, drop tracks no longer present; after them, a track + // the catalog dropped stays until what it holds is gone ([`Self::settle`]). let mut old: HashMap = self .tracks .extract_if(|name, track| !track.listed && active.contains_key(name)) @@ -2044,7 +2080,10 @@ where continue; }; let kind = video_kind(config, name)?; - let mut track = match old.remove(name) { + let reused = old + .remove(name) + .filter(|track| track.source.reads(&self.source, name, config)); + let mut track = match reused { Some(track) => track, None => match ExportSource::for_video(&self.source, name, config, self.budget())? .map(|s| at_edge(s, self.replay)) @@ -2062,7 +2101,10 @@ where continue; }; let kind = audio_kind(config, name)?; - let mut track = match old.remove(name) { + let reused = old + .remove(name) + .filter(|track| track.source.reads(&self.source, name, config)); + let mut track = match reused { Some(track) => track, None => match ExportSource::for_audio(&self.source, name, config, self.budget())? .map(|s| at_edge(s, self.replay)) @@ -2160,6 +2202,57 @@ where Ok(()) } + /// Move each listed track whose rendition the catalog moved to another track or broadcast + /// onto it. One the PMT does not list yet starts over on the new track instead. + /// + /// The PID, PMT and lane stay as announced, and so does the kind the track's frames are + /// framed under as they go out, so the new track must have the same kind: an AAC track keeps + /// its configuration. Video parameter sets ride with each frame and may change. The new + /// source continues the old one ([`ExportSource::continuing`]), and the frames the old one + /// left held for their DTS go out first. + fn rebind(&mut self, catalog: &Catalog) -> anyhow::Result<()> { + let budget = self.budget(); + for (name, track) in self.tracks.iter_mut().filter(|(_, track)| track.listed) { + let (source, kind) = if let Some(config) = catalog.video.renditions.get(name) { + if track.source.reads(&self.source, name, config) { + continue; + } + let source = ExportSource::for_video(&self.source, name, config, budget)?; + (source, video_kind(config, name)?) + } else if let Some(config) = catalog.audio.renditions.get(name) { + if track.source.reads(&self.source, name, config) { + continue; + } + let source = ExportSource::for_audio(&self.source, name, config, budget)?; + (source, audio_kind(config, name)?) + } else { + continue; + }; + let Some(source) = source else { + continue; + }; + let same = match (&kind, &track.kind) { + (Kind::Aac { config, .. }, Kind::Aac { config: old, .. }) => config == old, + (kind, old) => kind == old, + }; + anyhow::ensure!( + same, + "TS track layout changed after PAT/PMT was emitted: '{name}' moved to a track announced differently" + ); + track.release(name, &mut self.jitter, true)?; + track.clock = DecodeClock::default(); + track.source = source.continuing(&track.source); + track.finished = false; + track.subscribed = moq_clock::now(); + // Group sequences are the old track's own, so the new track's first read opens one. + track.group = None; + if let Some(sync) = &mut track.sync { + sync.lane.forget(); + } + } + Ok(()) + } + /// How many frames were dropped for missing their deadline. #[cfg(test)] pub(super) fn dropped(&self) -> u64 { @@ -2599,36 +2692,13 @@ where let streams = tracks .iter() .map(|t| { - let stream_type = match &t.kind { - Kind::Video(stream_type) => *stream_type as u8, - Kind::Aac { .. } => StreamType::AdtsAac as u8, - // Opus rides private-data PES; the registration + extension descriptors - // below tell the demuxer it's Opus. - Kind::Opus { .. } => 0x06, - // Half-rate MPEG-2 BC audio (< 32 kHz) re-announces as 0x04; the full - // rates are MPEG-1 (0x03). The catalog sample rate came from the frame - // header, so the mapping is faithful. - Kind::Mp2 { sample_rate } if *sample_rate < 32000 => StreamType::Mpeg2HalvedSampleRateAudio as u8, - Kind::Mp2 { .. } => StreamType::Mpeg1Audio as u8, - Kind::Ac3 => StreamType::DolbyDigitalUpToSixChannelAudio as u8, - Kind::Eac3 => StreamType::DolbyDigitalPlusUpTo16ChannelAudioForAtsc as u8, - Kind::Verbatim { stream_type, .. } | Kind::PesPacket { stream_type } => *stream_type, - }; - // Prefer the descriptors captured verbatim on import; otherwise synthesize - // the ATSC Dolby registration so a fresh (non-TS) AC-3/E-AC-3 track is - // still announced the way the import path expects. - let descriptors = if !t.descriptors.is_empty() { - t.descriptors.clone() - } else { - match &t.kind { - Kind::Ac3 => vec![registration(b"AC-3")], - Kind::Eac3 => vec![registration(b"EAC3")], - Kind::Opus { channel_config_code } => opus_descriptors(*channel_config_code), - _ => Vec::new(), - } + // Prefer the descriptors captured verbatim on import. + let descriptors = match t.descriptors.is_empty() { + true => t.kind.descriptors(), + false => t.descriptors.clone(), }; Ok(super::psi::Es { - stream_type, + stream_type: t.kind.stream_type(), pid: Pid::new(t.pid)?, descriptors, }) diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 3243a9db0dc..46a0fa8bda9 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -8016,6 +8016,220 @@ where } } +/// Packets whose continuity counter does not follow the last one on their PID, other than +/// those flagging a discontinuity. +fn continuity_errors(frames: &[Frame]) -> usize { + let mut last = std::collections::HashMap::new(); + let mut errors = 0; + for packet in frames.iter().flat_map(|f| f.payload.as_chunks::<188>().0.iter()) { + let pid = u16::from(packet[1] & 0x1f) << 8 | u16::from(packet[2]); + // Only a packet carrying payload advances the counter. + if pid == 0x1fff || packet[3] & 0x10 == 0 { + continue; + } + let counter = packet[3] & 0x0f; + let flagged = packet[3] & 0x20 != 0 && packet[4] > 0 && packet[5] & 0x80 != 0; + if let Some(previous) = last.insert(pid, counter) + && !flagged + && counter != (previous + 1) & 0x0f + { + errors += 1; + } + } + errors +} + +/// The output of an export reading the [`Moving`](crate::container::test_util::Moving) +/// rendition's track `a` up to frame `read`, before the catalog moves the rendition to `track` +/// in `broadcast` described by `sps`. Both tracks run on to frame 120 in step, the new one from +/// frame `start`, marked 0xb0 before frame 60, a keyframe two seconds in, and 0xb1 from it. +async fn moved_export( + (start, read): (u64, u64), + (track, broadcast, sps): (&str, Option<&str>, &'static [u8]), + sync: bool, +) -> Vec { + use crate::container::test_util::{Moving, marked_frame}; + + let mut moving = Moving::new(); + let mut export = open_export(moving.source().await) + .await + .unwrap() + .with_delay(RECORDING_MAX_AGE) + .with_replay(); + if sync { + export = export.with_mux_rate(2_000_000).with_sync(); + } + let write = |moving: &mut Moving, i: u64, mark: u8| { + moving.a.write(marked_frame(i, mark)).unwrap(); + let next = match broadcast { + None => &mut moving.b, + Some(_) => &mut moving.other, + }; + if i >= start { + let mark = if i < 60 { 0xb0 } else { 0xb1 }; + next.write(marked_frame(i, mark)).unwrap(); + } + }; + for i in 0..read { + write(&mut moving, i, 0xa1); + } + let mut frames = drain_frames(&mut export).await; + + moving.point(track, broadcast, sps); + frames.extend(drain_frames(&mut export).await); + for i in read..120 { + write(&mut moving, i, 0xa2); + frames.extend(drain_frames(&mut export).await); + } + match broadcast { + None => moving.b.finish().unwrap(), + Some(_) => moving.other.finish().unwrap(), + } + moving.catalog.finish().unwrap(); + let (rest, end) = drain_to_end(&mut export).await; + frames.extend(rest); + end.unwrap(); + frames +} + +/// A rendition the catalog moves to another track, in its broadcast or a sibling one, goes on +/// out of the PID the PMT announced, with that track's media and none of the one it left. The +/// continuity counters run on, and the move onto the same clock is no break. Its parameter sets +/// ride in-band, so the new track may carry another SPS. +#[tokio::test(start_paused = true)] +async fn a_moved_rendition_continues_on_its_pid() { + use crate::container::test_util::{Moving, marks}; + + for (track, broadcast, sps) in [("b", None, Moving::MAIN_SPS), ("a", Some("other"), SPS)] { + let frames = moved_export((60, 45), (track, broadcast, sps), false).await; + let ts: Vec = frames.iter().flat_map(|f| f.payload.iter().copied()).collect(); + let video = reassemble_video(&ts, StreamType::H264); + let expected: Vec = [vec![0xa1; 45], vec![0xb1; 60]].concat(); + assert_eq!(marks(&video), expected, "moved to {track} in {broadcast:?}"); + assert_eq!( + video.windows(4).any(|w| w == &Moving::MAIN_SPS[..4]), + sps == Moving::MAIN_SPS, + "the new track's parameter sets go out in-band" + ); + assert_eq!(continuity_errors(&frames), 0); + assert_eq!( + count_discontinuity(&frames), + 0, + "a move onto the same clock is no break" + ); + assert!( + pmts(&frames).iter().all(|(version, _)| *version == 0), + "the PMT stays as announced" + ); + } +} + +/// Under `--sync`, legs the catalog moves at different points of the old track render the new +/// track identically: its units take their lane positions from its own groups, whatever the +/// old track left on each leg. The second leg last read the old track's group 1, the sequence +/// the new track's first group it reads has too. +#[tokio::test(start_paused = true)] +async fn synced_legs_agree_after_a_move() { + use crate::container::test_util::{Moving, marks}; + + let mut legs = Vec::new(); + for (read, expected) in [ + (20, [vec![0xa1; 20], vec![0xb0; 30]]), + (50, [vec![0xa1; 50], Vec::new()]), + ] { + let frames = moved_export((30, read), ("b", None, Moving::MAIN_SPS), true).await; + // The video PID's payload. The PCR rides a PID the PMT lists no stream on, which + // [`reassemble_video`]'s reader refuses. + let mut video = None; + let mut es = Vec::new(); + for packet in frames.iter().flat_map(|f| f.payload.as_chunks::<188>().0.iter()) { + let pid = u16::from(packet[1] & 0x1f) << 8 | u16::from(packet[2]); + if packet[3] & 0x10 == 0 { + continue; + } + let payload = match packet[3] & 0x20 { + 0 => &packet[4..], + _ => &packet[5 + usize::from(packet[4])..], + }; + if video.is_none() && payload.starts_with(&[0, 0, 1, 0xe0]) { + video = Some(pid); + } + if video == Some(pid) { + es.extend_from_slice(payload); + } + } + let expected: Vec = [expected[0].clone(), expected[1].clone(), vec![0xb1; 60]].concat(); + assert_eq!(marks(&es), expected, "read {read}"); + assert_eq!(continuity_errors(&frames), 0); + legs.push(frames); + } + // From the slot that carries the new track's frame both legs read first, they carry the + // same packets. + let from = legs[1] + .iter() + .find(|frame| marks(&frame.payload).contains(&0xb1)) + .expect("the new track went out") + .timestamp; + let after = |frames: &[Frame]| -> Vec<(Timestamp, Bytes)> { + frames + .iter() + .filter(|frame| frame.timestamp >= from) + .map(|frame| (frame.timestamp, frame.payload.clone())) + .collect() + }; + let (first, second) = (after(&legs[0]), after(&legs[1])); + let carried: Vec = second.iter().flat_map(|(_, payload)| payload.iter().copied()).collect(); + assert_eq!( + marks(&carried), + vec![0xb1; 60], + "the compared slots carry the new track" + ); + assert!(first == second, "the legs disagree after the move"); +} + +/// AAC's ADTS headers are written from the announced configuration as its frames go out, so +/// a move to an AAC track configured differently is refused rather than framing the old +/// track's queued frames as the new one's. +#[tokio::test(start_paused = true)] +async fn a_rendition_moved_to_another_aac_configuration_fails() { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let mut catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let mut old = aac_rendition(&mut broadcast, &mut catalog, "main"); + let track = broadcast + .create_track("next", hang::container::track_info(hang::catalog::PRIORITY.audio)) + .unwrap(); + let mut next = Producer::new(track, HangContainer::Legacy(Kind::Data)); + + let mut export = open_export(crate::source::announced(&consumer).await) + .await + .unwrap() + .with_delay(RECORDING_MAX_AGE) + .with_replay(); + for ms in (0..200).step_by(20) { + write_aac(&mut old, ms); + } + assert!(!drain_frames(&mut export).await.is_empty(), "the PMT went out"); + + { + let mut guard = catalog.modify().unwrap(); + let config = guard.audio.renditions.get_mut("main").unwrap(); + config.track = Some("next".to_string()); + config.sample_rate = 44_100; + } + write_aac(&mut next, 200); + let err = tokio::time::timeout(DRAIN, async { + loop { + if let Err(err) = export.next().await { + break err; + } + } + }) + .await + .expect("the export reads the catalog update"); + assert!(err.to_string().contains("'main' moved to a track"), "{err}"); +} + /// A track that leaves the catalog after the PMT is read to its own end rather than /// refused as a layout change. A publisher retires a rendition as its track ends, and /// that catalog update can land before the track's last frames and its finish do, so diff --git a/rs/moq-mux/src/container/ts/lane.rs b/rs/moq-mux/src/container/ts/lane.rs index a70ca46f61d..97692319d33 100644 --- a/rs/moq-mux/src/container/ts/lane.rs +++ b/rs/moq-mux/src/container/ts/lane.rs @@ -315,10 +315,16 @@ impl Lane { /// Forget every group, for a timeline that starts again. pub fn reset(&mut self) { - self.group = None; + self.forget(); self.end = 0; } + /// Forget the group being laid, for a source whose group sequences are its own. The + /// timeline runs on, so the next group still lays after the positions taken. + pub fn forget(&mut self) { + self.group = None; + } + /// Whether a unit of group `sequence`, its first when `opens`, would be the first its group /// places, so its first packet flags the break in the continuity counter. pub fn opening(&self, (sequence, opens): (u64, bool)) -> bool { diff --git a/rs/moq-mux/src/error.rs b/rs/moq-mux/src/error.rs index 3358abdf0a7..60c42e7e49b 100644 --- a/rs/moq-mux/src/error.rs +++ b/rs/moq-mux/src/error.rs @@ -280,6 +280,19 @@ pub enum Error { /// Another publisher instance replaced a followed broadcast, and stitching was off. #[error("another publisher instance replaced broadcast `{0}`; enable stitch to follow it as a program switch")] Replaced(String), + + /// A rendition moved to a track whose media goes back before what the export already read for it. + #[error( + "rendition {rendition} moved to a track that goes back in time: a group starts at {start:?}, not past {read:?}" + )] + RenditionRewound { + /// The rendition's ID in the catalog. + rendition: String, + /// Where the new track's group starts. + start: std::time::Duration, + /// The latest media the export read from the rendition's previous track. + read: std::time::Duration, + }, } impl Error { diff --git a/rs/moq-mux/src/source.rs b/rs/moq-mux/src/source.rs index 2a760037abf..665c70a441e 100644 --- a/rs/moq-mux/src/source.rs +++ b/rs/moq-mux/src/source.rs @@ -139,7 +139,7 @@ impl Source { /// The broadcast `path` is pinned to, or a lookup that waits for the path's first /// announcement and pins its answer, either checked against `epoch` on every read. - fn request_path(&self, path: &moq_net::PathOwned, epoch: Option<&moq_net::Epoch>) -> Binding { + pub(crate) fn request_path(&self, path: &moq_net::PathOwned, epoch: Option<&moq_net::Epoch>) -> Binding { let bound = match self.pins.lock().expect("pins poisoned").get(path) { Some(pinned) => Bound::Ready(pinned.clone()), // Unpinned even with an epoch: an epochless route still answers, best effort, and @@ -186,17 +186,6 @@ impl Source { }) } - /// The skipping counterpart to [`Self::bind`] for a catalog entry, returning `None` when its - /// broadcast walks above the origin root. - /// - /// The exporters use it to drop a single bad rendition rather than fail the whole export. - /// [`Self::retain_selectable`] already removes those renditions, so this is the second half of - /// the same policy rather than a different one. - pub(crate) fn try_request(&self, reference: &impl hang::catalog::Reference) -> Option { - let path = self.resolve_reference(reference.broadcast())?; - Some(self.request_path(&path, reference.epoch())) - } - /// Remove the renditions an export must not select: a disabled one (catalog /// `enabled: false`), or one whose broadcast reference escapes above the origin root. ///