From 3ee7680c60e758df3efb20aef9114b2cef093898 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 22 Oct 2026 20:16:03 -0700 Subject: [PATCH 01/11] chore(quest): claim quest/m1/ts-reliability/decode-order.md Quest-Claim: 8afdbb190a23e507 From 9a2f2a65c47bf5a7749c57725ffbdd6223583be4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 22 Oct 2026 21:21:44 -0700 Subject: [PATCH 02/11] fix(mux): bound declared AVC decode slots in MKV Checkpoint signed group-local scheduling and bounded payload retention for progressive AVC with a declared SPS depth. Preserve the published-catalog compatibility decision and the unfinished full exporter/producer rollout. Co-Authored-By: GPT-6 --- quest/m1/ts-reliability/decode-order.md | 39 +++- rs/moq-mux/MIGRATE.md | 10 + rs/moq-mux/benches/export.rs | 119 +++++++++- rs/moq-mux/benches/jitter.rs | 4 +- rs/moq-mux/src/container/decode.rs | 242 +++++++++++++++++--- rs/moq-mux/src/container/flv/export.rs | 4 +- rs/moq-mux/src/container/mkv/export.rs | 116 +++++++--- rs/moq-mux/src/container/mkv/export_test.rs | 112 +++++++++ rs/moq-mux/src/container/ts/export.rs | 12 +- rs/moq-mux/src/container/ts/release.rs | 2 +- rs/moq-mux/src/jitter.rs | 101 ++++++-- 11 files changed, 668 insertions(+), 93 deletions(-) diff --git a/quest/m1/ts-reliability/decode-order.md b/quest/m1/ts-reliability/decode-order.md index 6dc58667284..cc55b839a39 100644 --- a/quest/m1/ts-reliability/decode-order.md +++ b/quest/m1/ts-reliability/decode-order.md @@ -12,6 +12,43 @@ presentation-time only: no decode time on the wire. ## Plan +### Checkpoint + +The Goal remains incomplete. This branch adds signed jitter admission and a +fixed-depth mode to the existing `DecodeClock`, consumed by MKV export when a +progressive AVC SPS declares the depth. It keeps at most that many payloads +after each drain, shifts by presentation slots, and resets at actual group +boundaries. Tests cover timeline zero, audio interleaving, independent groups, +a cut in an earlier group, distant references, zero depth, and malformed +flat/backward PTS swept over frame count and payload size. The exporter benchmark +sweeps those axes through the same MKV consumer. + +TS and FLV still use their existing clock authors, as do MKV's undeclared, +field-coded AVC and HEVC paths. Signed jitter admission reaches all three +exporters, but only MKV's declared progressive AVC path currently authors signed +slots. No new catalog field, producer declaration, JS behavior or wire format is +implemented. The remaining rollout includes codec-level depth validation, +TS/MKV/FLV parity, every producer/importer, Rust/JS catalogs, matching drafts and +migrations, and the full join/loss and T-STD controls described below. + +### Decision needed: published absence + +The decided `absent = 0` rule conflicts with the published-wire compatibility +rule. Published `hang-v0.21.16` has no `reorder` field, and the existing fMP4 +import control `unwrapped_samples_present_at_their_composition_time` publishes +decode-order PTS 1,041,666; 1,125,000; 1,083,333 microseconds without one. Treating +that absence as zero makes a new exporter refuse supported published media. +The current catalog version cannot distinguish that legacy absence from an +intentional zero declaration. + +Recommend a versioned or explicit-declaration contract that preserves published +catalog behavior, then complete the fixed-depth producer/export rollout. The +alternative is deliberate migration/refusal of legacy reordering catalogs, +which changes the published compatibility requirement and needs approval. +Do not silently choose either contract or remove this Goal while unresolved. + +### Settled design + Why: a DTS that depends on what the exporter read before has been patched six times ([moq-dev/moq#1843](https://github.com/moq-dev/moq/pull/1843), [#2967](https://github.com/moq-dev/moq/pull/2967), @@ -19,7 +56,7 @@ six times ([moq-dev/moq#1843](https://github.com/moq-dev/moq/pull/1843), [#4574](https://github.com/moq-dev/moq/pull/4574), [#4645](https://github.com/moq-dev/moq/pull/4645), [#5949](https://github.com/moq-dev/moq/pull/5949)). Today there are three -authors: `DecodeClock::pop` (`rs/moq-mux/src/container/decode.rs`, TS without +authors: timestamp-settled `DecodeClock::settled` (`rs/moq-mux/src/container/decode.rs`, TS without `--sync` and MKV) settles a slot by the highest PTS read and a delay that only grows; `pop_counted` (TS `--sync`) depends on the held frames alone; FLV export authors `PTS - jitter` bumped past the last DTS, which passes the PTS diff --git a/rs/moq-mux/MIGRATE.md b/rs/moq-mux/MIGRATE.md index a3ff4833685..e14e2ab12fe 100644 --- a/rs/moq-mux/MIGRATE.md +++ b/rs/moq-mux/MIGRATE.md @@ -4,6 +4,16 @@ How to update your code for each breaking change, newest first. ## Unreleased +### MKV schedules declared progressive AVC by presentation slots + +A progressive AVC SPS that declares a reorder depth now fixes MKV export's decode +clock for each group. Its first decode slots can precede timestamp zero; block +presentation timestamps remain unchanged. Catalog `jitter` and `framerate` no +longer change that rendition's decode slots. A source that exceeds its declared +depth or cannot produce increasing decode slots now ends the export instead of +moving its timestamps. Correct the source timestamps or declare a new rendition +with the actual reorder depth. + ### JSON `delta_ratio` is `deltas` `json::Config::delta_ratio` and `with_delta_ratio` are replaced by `deltas: bool` and `with_deltas` (default `true`). diff --git a/rs/moq-mux/benches/export.rs b/rs/moq-mux/benches/export.rs index da97d3bb42d..040636fd098 100644 --- a/rs/moq-mux/benches/export.rs +++ b/rs/moq-mux/benches/export.rs @@ -133,5 +133,122 @@ fn bench_export(c: &mut Criterion) { group.finish(); } -criterion_group!(benches, bench_export); +criterion_group!(benches, bench_export, video_reorder); criterion_main!(benches); + +/// A progressive AVC SPS with a declared reorder depth of two (25 fps). +const REORDER_SPS: &[u8] = &[ + 0x67, 0x64, 0x00, 0x0a, 0xac, 0xd9, 0x44, 0x26, 0xc0, 0x44, 0x00, 0x00, 0x03, 0x00, 0x04, 0x00, 0x00, 0x03, 0x00, + 0xca, 0x3c, 0x48, 0x96, 0x58, +]; + +async fn prepare_video( + count: usize, + size: usize, + pattern: &str, +) -> (moq_mux::Source, moq_mux::catalog::Consumer, Fixture) { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let mut catalog = moq_mux::catalog::Producer::new(&mut broadcast, Default::default()).unwrap(); + let stream = moq_mux::catalog::Consumer::from(catalog.consume().unwrap()); + let track = broadcast + .create_track("video", catalog.track_info(hang::catalog::PRIORITY.video)) + .unwrap(); + let mut config = hang::catalog::VideoConfig::new(hang::catalog::H264 { + profile: 100, + constraints: 0, + level: 10, + inline: false, + }); + config.coded_width = Some(64); + config.coded_height = Some(64); + let mut description = vec![1, 100, 0, 10, 0xff, 0xe1]; + description.extend_from_slice(&(REORDER_SPS.len() as u16).to_be_bytes()); + description.extend_from_slice(REORDER_SPS); + description.extend_from_slice(&[1, 0, 4, 0x68, 0xce, 0x3c, 0x80]); + config.description = Some(description.into()); + config.container = hang::catalog::Container::Legacy; + let mut video = catalog.video(track, Container::Legacy(Kind::Video), config).unwrap(); + for index in 0..count { + let pts = match pattern { + "pyramid" => ((index / 9 * 9 + [0, 4, 2, 1, 3, 8, 6, 5, 7][index % 9]) * 3_600) as u64, + "flat" => 90_000, + "backward" => 90_000 - index as u64, + _ => unreachable!(), + }; + let mut payload = vec![0; size]; + payload[..4].copy_from_slice(&((size - 4) as u32).to_be_bytes()); + payload[4] = if index == 0 { 0x65 } else { 0x41 }; + video + .write(Frame { + timestamp: Timestamp::from_scale(pts, 90_000).unwrap(), + duration: None, + payload: payload.into(), + keyframe: index == 0, + }) + .unwrap(); + } + video.finish().unwrap(); + catalog.finish().unwrap(); + let (origin, driver) = moq_net::origin::Producer::new(Default::default()); + let driver = tokio::spawn(moq_net::time::run(driver)); + let dynamic = origin.dynamic_route("", Default::default()).unwrap(); + let requests = tokio::spawn(async move { + while let Ok(request) = dynamic.requested_broadcast().await { + request.accept(consumer.clone()); + } + }); + let resolved = origin.consume().request_broadcast("bench", None).await.unwrap(); + let source = moq_mux::Source::new(origin.consume(), resolved); + ( + source, + stream, + Fixture { + _published: Box::new((broadcast, catalog, video)), + _origin: origin, + driver, + requests, + }, + ) +} + +fn video_reorder(c: &mut Criterion) { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let mut group = c.benchmark_group("video-reorder"); + group.sample_size(10); + group.warm_up_time(Duration::from_millis(50)); + group.measurement_time(Duration::from_millis(200)); + for pattern in ["pyramid", "flat", "backward"] { + for count in [64, 256, 1024] { + for size in [256, 16_384] { + group.bench_function(BenchmarkId::new(pattern, format!("{count}x{size}")), |b| { + b.iter_batched( + || runtime.block_on(prepare_video(count, size, pattern)), + |(source, stream, fixture)| { + let refused = runtime.block_on(async { + let mut export = moq_mux::container::mkv::Export::new(source, stream) + .with_fragment_duration(Duration::ZERO); + loop { + match export.next().await { + Ok(Some(frame)) => { + std::hint::black_box(frame); + } + Ok(None) => break false, + Err(_) => break true, + } + } + }); + assert_eq!(refused, pattern != "pyramid"); + fixture + }, + BatchSize::PerIteration, + ); + }); + } + } + } + group.finish(); +} diff --git a/rs/moq-mux/benches/jitter.rs b/rs/moq-mux/benches/jitter.rs index f2b69d46272..e568cea0c65 100644 --- a/rs/moq-mux/benches/jitter.rs +++ b/rs/moq-mux/benches/jitter.rs @@ -23,7 +23,7 @@ async fn steer(tracks: u16, floors: u64) -> u64 { let arrival = jitter::Arrival { arrived: at, read: at, - decode: Timestamp::from_millis(second * 1_000).unwrap(), + decode: Timestamp::from_millis(second * 1_000).unwrap().into(), restart: 0, skip: 0, cut: 0, @@ -85,7 +85,7 @@ fn acquisition(c: &mut Criterion) { jitter::Arrival { arrived: start, read: start, - decode: Timestamp::from_millis(frame * 20).unwrap(), + decode: Timestamp::from_millis(frame * 20).unwrap().into(), restart: 0, skip: 0, cut: 0, diff --git a/rs/moq-mux/src/container/decode.rs b/rs/moq-mux/src/container/decode.rs index 68ad5d67e0d..26e025211e1 100644 --- a/rs/moq-mux/src/container/decode.rs +++ b/rs/moq-mux/src/container/decode.rs @@ -5,34 +5,21 @@ pub(crate) const MAX_REORDER: u64 = 2 * 90_000; /// A video rendition's decode clock, authored from presentation timestamps. /// -/// [`crate::container::Frame`] carries only a presentation timestamp and frames reach the muxer in decode order, -/// so a B-frame stream arrives with non-monotonic PTS and no decode time. An encoder decodes -/// the n-th frame in decode order at the n-th presentation time, a fixed reorder delay early, -/// so that is the DTS authored here: with decode-order PTS 0, 120, 40, 80 and a delay of one -/// 40 ms frame, the frames decode at -40, 0, 40, 80. It keeps the encoder's own spacing, a -/// frame's or a field's, through any switch between frame and field coding, and the delay is -/// a time, the least that keeps every DTS at or before its PTS (or the SPS's declared depth, -/// when that is more). On a source muxed that way this is its DTS, frame for frame. +/// Declared-depth clocks release the n-th group frame at its (n - depth)-th +/// presentation slot. The first depth slots precede the group's keyframe by one +/// 90 kHz tick each, so DTS may be negative. Draining after every push bounds the +/// held payloads by the depth, independently of frame spacing and group length. +/// Group completion flushes the remaining slots; a cut discards them and resets +/// the next group's origin. Invalid slots fail rather than move the timeline. /// -/// A frame's slot is settled once no frame still to come can be presented before it. A later -/// frame decodes after this one, and is presented at most the catalog `jitter` after it -/// decodes, so the slot is settled once the highest PTS read is that far past it. Frames wait -/// here until then, as long as the reordering spans and well inside the jitter buffer's delay. -/// Without a `jitter`, the SPS's depth bounds it, and so does how far a frame was presented -/// below the highest PTS read before it. A frame that still comes in below a settled slot -/// (reordering deeper than all of those) is nudged one tick past the last DTS, so the clock -/// stays strictly increasing. -/// -/// That settling, and a delay that only grows, depend on everything read before, so two -/// exporters that joined or lost frames at different points can clock the same frame apart. -/// [`Self::pop_counted`] depends on the frames held alone, for exporters that must agree. -/// -/// Ticks are continuous (unwrapped) 90 kHz, so the clock never wraps mid-stream; the 33-bit -/// wire wrap happens once at emission. +/// Undeclared legacy renditions still settle slots by timestamp lookahead, or +/// use a caller-supplied counted delay. Those paths retain their timing history. +/// Ticks are continuous 90 kHz; a container applies its wire representation later. pub(crate) struct DecodeClock { + fixed: Option, /// Frames read but not yet given a DTS, in decode order, with their PTS. held: VecDeque<(T, u64)>, - /// The held frames' PTS, ascending: the slots they decode in, in turn. + /// Presentation slots not yet assigned to a frame, in ascending order. slots: Vec, /// The highest PTS read. high: Option, @@ -47,6 +34,7 @@ pub(crate) struct DecodeClock { impl Default for DecodeClock { fn default() -> Self { Self { + fixed: None, held: VecDeque::new(), slots: Vec::new(), high: None, @@ -58,22 +46,85 @@ impl Default for DecodeClock { } impl DecodeClock { + /// Author group-local decode slots at this rendition's fixed reorder depth. + pub(crate) fn new(reorder: usize) -> crate::Result { + if reorder > crate::codec::video::Reorder::MAX_DEPTH as usize { + return Err(ClockError::Depth(reorder).into()); + } + Ok(Self { + fixed: Some(Fixed { + depth: reorder, + leading: reorder, + first: None, + last: None, + highest: Vec::new(), + }), + ..Self::default() + }) + } + + pub(crate) fn is_fixed(&self) -> bool { + self.fixed.is_some() + } + + #[cfg(test)] + pub(crate) fn len(&self) -> usize { + self.held.len() + } + /// Hold the next frame in decode order, presented at `pts`. - pub(crate) fn push(&mut self, item: T, pts: u64) { - if let Some(high) = self.high - && pts < high - { - self.reach = self.reach.max(high - pts + 1).min(MAX_REORDER); + pub(crate) fn push(&mut self, item: T, pts: u64) -> crate::Result<()> { + if let Some(fixed) = &mut self.fixed { + if self.held.len() > fixed.depth { + return Err(ClockError::Undrained.into()); + } + let at = fixed.highest.partition_point(|&prior| prior <= pts); + if fixed.highest.len() - at > fixed.depth { + return Err(ClockError::Reordered(fixed.depth).into()); + } + fixed.highest.insert(at, pts); + if fixed.highest.len() > fixed.depth + 1 { + fixed.highest.remove(0); + } + fixed.first.get_or_insert(pts); + } else { + if let Some(high) = self.high + && pts < high + { + self.reach = self.reach.max(high - pts + 1).min(MAX_REORDER); + } + self.high = Some(self.high.map_or(pts, |high| high.max(pts))); } - self.high = Some(self.high.map_or(pts, |high| high.max(pts))); let at = self.slots.partition_point(|&slot| slot <= pts); self.slots.insert(at, pts); self.held.push_back((item, pts)); + Ok(()) + } + + /// Release a frame once its declared lookahead is present, or the group is complete. + pub(crate) fn pop(&mut self, flush: bool) -> crate::Result> { + let fixed = self.fixed.as_mut().ok_or(ClockError::Undeclared)?; + if self.held.is_empty() || (!flush && self.held.len() <= fixed.depth) { + return Ok(None); + } + let dts = match fixed.leading { + 0 => i128::from(self.slots.remove(0)), + leading => { + fixed.leading -= 1; + i128::from(fixed.first.expect("a held frame supplies the origin")) - leading as i128 + } + }; + let (item, pts) = self.held.pop_front().expect("the held queue is nonempty"); + if dts > i128::from(pts) || fixed.last.is_some_and(|last| dts <= last) { + return Err(ClockError::Presentation.into()); + } + fixed.last = Some(dts); + Ok(Some((item, dts))) } /// The next frame and its DTS, once its slot is settled `lookahead` past (the catalog /// `jitter`), or at once when `flush`. `floor` is the least reorder delay, in ticks. - pub(crate) fn pop(&mut self, lookahead: u64, floor: u64, flush: bool) -> Option<(T, u64)> { + pub(crate) fn settled(&mut self, lookahead: u64, floor: u64, flush: bool) -> Option<(T, u64)> { let high = self.high?; let lookahead = lookahead.max(self.reach); let settled = |slot: u64| flush || high >= slot.saturating_add(lookahead); @@ -114,6 +165,12 @@ impl DecodeClock { pub(crate) fn discard(&mut self) { self.held.clear(); self.slots.clear(); + if let Some(fixed) = &mut self.fixed { + fixed.leading = fixed.depth; + fixed.first = None; + fixed.last = None; + fixed.highest.clear(); + } } fn after_last(&mut self, dts: u64) -> u64 { @@ -126,10 +183,125 @@ impl DecodeClock { } } +struct Fixed { + depth: usize, + leading: usize, + first: Option, + last: Option, + // Only the largest depth + 1 earlier PTS are needed to detect a deeper reorder. + highest: Vec, +} + +#[derive(Debug, thiserror::Error)] +enum ClockError { + #[error("reorder depth {0} exceeds the supported codec depth")] + Depth(usize), + #[error("video reorders past its declared depth {0}")] + Reordered(usize), + #[error("video presentation slots cannot author increasing DTS at or before PTS")] + Presentation, + #[error("the decode clock must be drained before reading another frame")] + Undrained, + #[error("the decode clock has no fixed reorder declaration")] + Undeclared, +} + +impl From for crate::Error { + fn from(err: ClockError) -> Self { + anyhow::Error::new(err).into() + } +} + #[cfg(test)] mod tests { use super::*; + fn fixed_slots(read: &[u64]) -> Vec { + let mut clock = DecodeClock::new(2).unwrap(); + let mut out = Vec::new(); + for &pts in read { + clock.push((), pts).unwrap(); + while let Some(((), dts)) = clock.pop(false).unwrap() { + out.push(dts); + } + assert!(clock.held.len() <= 2); + assert!(clock.slots.len() <= 4); + } + while let Some(((), dts)) = clock.pop(true).unwrap() { + out.push(dts); + } + out + } + + #[test] + fn fixed_slots_shift_by_count_and_start_before_zero() { + let read = PYRAMID.map(|i| i * FRAME); + let want: Vec = [-2, -1].into_iter().chain((0..7).map(|i| i * FRAME as i128)).collect(); + assert_eq!(fixed_slots(&read), want); + let far = [0, 6, 10, 1, 2, 3, 4, 5, 7, 8, 9].map(|i| i * FRAME); + let want: Vec = [-2, -1].into_iter().chain((0..9).map(|i| i * FRAME as i128)).collect(); + assert_eq!(fixed_slots(&far), want); + } + + #[test] + fn a_fixed_group_keeps_no_previous_groups_history() { + let next = PYRAMID.map(|i| (i + 20) * FRAME); + let want = fixed_slots(&next); + for count in 0..PYRAMID.len() { + let mut clock = DecodeClock::new(2).unwrap(); + for pts in PYRAMID[..count].iter().map(|i| (i + 2) * FRAME) { + clock.push((), pts).unwrap(); + while clock.pop(false).unwrap().is_some() {} + } + clock.discard(); + let mut out = Vec::new(); + for pts in next { + clock.push((), pts).unwrap(); + while let Some(((), dts)) = clock.pop(false).unwrap() { + out.push(dts); + } + } + while let Some(((), dts)) = clock.pop(true).unwrap() { + out.push(dts); + } + assert_eq!(out, want, "lost prefix of {count} frames"); + } + } + + #[test] + fn malformed_presentation_is_refused_without_retaining_its_payload() { + for backwards in [false, true] { + for count in [16, 128, 1024] { + for size in [256, 16_384] { + let mut clock = DecodeClock::new(2).unwrap(); + let mut refused = false; + for index in 0..count { + let pts = 90_000 - if backwards { index } else { 0 }; + if clock.push(vec![0u8; size], pts).is_err() || clock.pop(false).is_err() { + refused = true; + break; + } + assert!(clock.held.len() <= 2); + assert!(clock.slots.len() <= 4); + } + assert!(refused); + assert!(clock.held.len() <= 2); + } + } + } + } + + #[test] + fn a_zero_depth_has_no_lookahead_or_decode_delay() { + let mut clock = DecodeClock::new(0).unwrap(); + for pts in [0, 1, 90_000, u64::MAX] { + clock.push(pts, pts).unwrap(); + assert_eq!(clock.pop(false).unwrap(), Some((pts, i128::from(pts)))); + assert!(clock.held.is_empty()); + } + assert!(DecodeClock::<()>::new(17).is_err()); + } + /// A B-pyramid (`max_num_reorder_frames` 2) in decode order, in 40 ms frames. const PYRAMID: [u64; 9] = [0, 4, 2, 1, 3, 8, 6, 5, 7]; const FRAME: u64 = 3_600; @@ -141,10 +313,10 @@ mod tests { let mut out = Vec::new(); let pop = |clock: &mut DecodeClock, flush| match counted { true => clock.pop_counted(2, 2 * FRAME, flush), - false => clock.pop(4 * FRAME, 2 * FRAME, flush), + false => clock.settled(4 * FRAME, 2 * FRAME, flush), }; for &pts in read { - clock.push(pts, pts); + clock.push(pts, pts).unwrap(); out.extend(std::iter::from_fn(|| pop(&mut clock, false))); } out.extend(std::iter::from_fn(|| pop(&mut clock, true))); @@ -196,12 +368,12 @@ mod tests { let mut clock = DecodeClock::default(); let mut out = Vec::new(); for &pts in &first[..8] { - clock.push(pts, pts); + clock.push(pts, pts).unwrap(); out.extend(std::iter::from_fn(|| clock.pop_counted(2, 2 * FRAME, false))); } clock.discard(); for pts in next { - clock.push(pts, pts); + clock.push(pts, pts).unwrap(); out.extend(std::iter::from_fn(|| clock.pop_counted(2, 2 * FRAME, false))); } out.extend(std::iter::from_fn(|| clock.pop_counted(2, 2 * FRAME, true))); diff --git a/rs/moq-mux/src/container/flv/export.rs b/rs/moq-mux/src/container/flv/export.rs index ae203aa69fe..1f4d59dc44f 100644 --- a/rs/moq-mux/src/container/flv/export.rs +++ b/rs/moq-mux/src/container/flv/export.rs @@ -359,7 +359,7 @@ impl Export { crate::jitter::Arrival { arrived: track.empty, read: moq_clock::now(), - decode: frame.timestamp, + decode: frame.timestamp.into(), restart: track.source.markers(), skip: track.source.skips(), cut: track.source.cuts(), @@ -392,7 +392,7 @@ impl Export { crate::jitter::Arrival { arrived, read, - decode: moq_net::Timestamp::from_millis(u64::from(timestamp))?, + decode: moq_net::Timestamp::from_millis(u64::from(timestamp))?.into(), restart, skip, cut, diff --git a/rs/moq-mux/src/container/mkv/export.rs b/rs/moq-mux/src/container/mkv/export.rs index 2b129d434d1..26f5a87f307 100644 --- a/rs/moq-mux/src/container/mkv/export.rs +++ b/rs/moq-mux/src/container/mkv/export.rs @@ -83,34 +83,57 @@ struct MkvTrack { } impl MkvTrack { - fn release(&mut self, name: &str, jitter: &mut crate::jitter::Buffer, flush: bool) -> Result<()> { + fn configure_clock(&mut self) -> Result<()> { + if self.reorder.is_some() { + return Ok(()); + } let Some(config) = &self.config else { return Ok(()) }; - let (floor, lookahead) = *self.reorder.get_or_insert_with(|| { - let declared = self.source.description().and_then(|description| match config.codec { - VideoCodec::H264(_) => { - crate::codec::h264::sps_reorder(crate::codec::h264::Avcc::parse(description).ok()?.sps.first()?) - } - VideoCodec::H265(_) => { - crate::codec::h265::sps_reorder(crate::codec::h265::Hvcc::parse(description).ok()?.sps.first()?) - } - _ => None, - }); - let (floor, lookahead) = declared.map_or((0, 0), |reorder| reorder.timing(config.framerate)); - let limit = crate::container::decode::MAX_REORDER; - let lookahead = lookahead - .max( - config - .jitter - .map_or(0, |jitter| (jitter.as_nanos() * 90_000 / 1_000_000_000) as u64), - ) - .min(limit); - (floor.min(limit), lookahead) + let declared = self.source.description().and_then(|description| match config.codec { + VideoCodec::H264(_) => { + crate::codec::h264::sps_reorder(crate::codec::h264::Avcc::parse(description).ok()?.sps.first()?) + } + VideoCodec::H265(_) => { + crate::codec::h265::sps_reorder(crate::codec::h265::Hvcc::parse(description).ok()?.sps.first()?) + } + _ => None, }); - while let Some((mut arrival, dts)) = self.clock.pop(lookahead, floor, flush) { + // A progressive AVC SPS already declares the fixed frame-count contract. Legacy + // catalogs lacking that declaration keep their existing timing interpretation. + if matches!(config.codec, VideoCodec::H264(_)) + && let Some(declared) = declared.filter(|declared| !declared.fields) + { + self.clock = crate::container::decode::DecodeClock::new(declared.pictures())?; + } + let (floor, lookahead) = declared.map_or((0, 0), |reorder| reorder.timing(config.framerate)); + let limit = crate::container::decode::MAX_REORDER; + let lookahead = lookahead + .max( + config + .jitter + .map_or(0, |jitter| (jitter.as_nanos() * 90_000 / 1_000_000_000) as u64), + ) + .min(limit); + self.reorder = Some((floor.min(limit), lookahead)); + Ok(()) + } + + fn release(&mut self, name: &str, jitter: &mut crate::jitter::Buffer, flush: bool) -> Result<()> { + let Some((floor, lookahead)) = self.reorder else { + return Ok(()); + }; + loop { + let next = match self.clock.is_fixed() { + true => self.clock.pop(flush)?, + false => self + .clock + .settled(lookahead, floor, flush) + .map(|(frame, dts)| (frame, i128::from(dts))), + }; + let Some((mut arrival, dts)) = next else { break }; if !self.finished && !flush { arrival.arrived = arrival.arrived.max(self.latest); } - arrival.decode = moq_net::Timestamp::from_scale(dts, 90_000)?; + arrival.decode = crate::jitter::Decode::from_nanos((dts * 1_000_000_000).div_euclid(90_000)); jitter.push(name.to_owned(), arrival)?; } Ok(()) @@ -336,6 +359,11 @@ impl Export { #[cfg(test)] pub(super) fn buffered(&self) -> usize { self.jitter.len() + + self + .tracks + .values() + .map(|track| track.clock.len() + usize::from(track.pending.is_some())) + .sum::() } #[cfg(test)] @@ -362,7 +390,19 @@ impl Export { } let (frame, arrived) = match track.source.poll_event(waiter)? { Poll::Ready(Some(Event::Frame { frame, arrived })) => (frame, arrived.max(track.subscribed)), - Poll::Ready(Some(Event::GroupEnd | Event::FrameEnd { .. } | Event::Cut)) => continue, + Poll::Ready(Some(Event::GroupEnd)) => { + if track.clock.is_fixed() { + track.release(name, &mut self.jitter, true)?; + } + continue; + } + Poll::Ready(Some(Event::Cut)) => { + if track.clock.is_fixed() { + track.clock.discard(); + } + continue; + } + Poll::Ready(Some(Event::FrameEnd { .. })) => continue, Poll::Ready(None) => { track.finished = true; break; @@ -382,7 +422,7 @@ impl Export { crate::jitter::Arrival { arrived, read: moq_clock::now(), - decode: frame.timestamp, + decode: frame.timestamp.into(), restart: track.source.markers(), skip: track.source.skips(), cut: track.source.cuts(), @@ -395,14 +435,25 @@ impl Export { track.pending = Some(arrival); break; } - if track.kind == TrackKind::Video && !self.delay.is_zero() { + if track.kind == TrackKind::Video { + track.configure_clock()?; + } + if track.kind == TrackKind::Video && (!self.delay.is_zero() || track.clock.is_fixed()) { if arrival.restart != track.restart { track.release(name, &mut self.jitter, true)?; - track.clock = Default::default(); + if track.clock.is_fixed() { + track.clock.discard(); + } else { + track.clock = Default::default(); + } track.restart = arrival.restart; } + if arrival.item.keyframe && track.clock.is_fixed() { + track.release(name, &mut self.jitter, true)?; + track.clock.discard(); + } let pts = (arrival.item.timestamp.as_nanos() * 90_000 / 1_000_000_000) as u64; - track.clock.push(arrival, pts); + track.clock.push(arrival, pts)?; track.release(name, &mut self.jitter, false)?; } else { self.jitter.push(name.clone(), arrival)?; @@ -440,10 +491,11 @@ impl Export { } for (name, config) in &catalog.video.renditions { if let Some(track) = self.tracks.get_mut(name) { - if track - .config - .as_ref() - .is_some_and(|old| old.jitter != config.jitter || old.framerate != config.framerate) + if !track.clock.is_fixed() + && track + .config + .as_ref() + .is_some_and(|old| old.jitter != config.jitter || old.framerate != config.framerate) { track.reorder = None; } diff --git a/rs/moq-mux/src/container/mkv/export_test.rs b/rs/moq-mux/src/container/mkv/export_test.rs index e1d418735ac..779d1ed0176 100644 --- a/rs/moq-mux/src/container/mkv/export_test.rs +++ b/rs/moq-mux/src/container/mkv/export_test.rs @@ -22,6 +22,118 @@ use crate::container::test_util::{IDR, Live, PPS, SPS, raw_frame, video_frame}; /// completeness has to be asked for, exactly as a real recorder does. const RECORDING_MAX_DELAY: std::time::Duration = std::time::Duration::from_secs(30); +/// A progressive AVC rendition whose SPS declares a reorder depth of two. +fn declared_avc() -> Live { + Live::new(".avc1", |catalog, name| { + let mut config = hang::catalog::VideoConfig::new(hang::catalog::H264 { + profile: 100, + constraints: 0, + level: 10, + inline: false, + }); + config.coded_width = Some(64); + config.coded_height = Some(64); + config.container = hang::catalog::Container::Legacy; + config.description = Some( + crate::codec::h264::build_avcc( + &[Bytes::from_static(crate::codec::h264::fixtures::SPS_PYRAMID)], + &[Bytes::from_static(PPS)], + ) + .unwrap(), + ); + catalog.modify().unwrap().video.renditions.insert(name, config); + }) +} + +#[tokio::test(start_paused = true)] +async fn a_declared_sps_bounds_malformed_video_by_count_and_payload() { + for backwards in [false, true] { + for count in [16, 128, 1024] { + for size in [256, 16_384] { + let mut live = declared_avc(); + for index in 0..count { + let mut frame = raw_frame(1_000_000 - if backwards { index } else { 0 }, b"", index == 0); + let mut bytes = vec![0; size]; + bytes[..4].copy_from_slice(&((size - 4) as u32).to_be_bytes()); + bytes[4] = if index == 0 { 0x65 } else { 0x41 }; + frame.payload = bytes.into(); + live.track.write(frame).unwrap(); + } + live.track.finish().unwrap(); + live.catalog.finish().unwrap(); + let mut export = super::Export::new(live.source().await, live.catalog_stream().await) + .with_fragment_duration(std::time::Duration::ZERO) + .with_replay(); + export.next().await.unwrap().unwrap(); + let mut refused = false; + loop { + match export.next().await { + Err(err) => { + assert!(err.to_string().contains("video"), "{err}"); + refused = true; + break; + } + Ok(None) => break, + Ok(Some(_)) => {} + } + assert!(export.buffered() <= 3, "a depth-two clock and one released frame"); + } + assert!(refused, "backwards={backwards} count={count} size={size}"); + assert!(export.buffered() <= 3); + } + } + } +} + +/// MKV retains PTS while signed, group-local DTS orders video against audio. +#[tokio::test(start_paused = true)] +async fn declared_reorder_starts_before_zero_and_resets_each_group() { + let mut live = declared_avc(); + let mut audio = live.add_track("audio", |catalog, name| { + let mut config = hang::catalog::AudioConfig::new(hang::catalog::AAC { profile: 2 }, 44100, 2); + config.description = Some(Bytes::from_static(&[0x12, 0x10])); + config.container = hang::catalog::Container::Legacy; + catalog.modify().unwrap().audio.renditions.insert(name, config); + }); + for start in [0, 400_000] { + for (index, offset) in [0, 160_000, 80_000, 40_000, 120_000].into_iter().enumerate() { + let payload = if index == 0 { + &b"\x00\x00\x00\x01\x65"[..] + } else { + &b"\x00\x00\x00\x01\x41"[..] + }; + live.track + .write(raw_frame(start + offset, payload, index == 0)) + .unwrap(); + } + audio.write(raw_frame(start, b"audio", true)).unwrap(); + } + live.track.finish().unwrap(); + audio.finish().unwrap(); + live.catalog.finish().unwrap(); + let mut export = super::Export::new(live.source().await, live.catalog_stream().await) + .with_delay(std::time::Duration::from_secs(5)) + .with_fragment_duration(std::time::Duration::ZERO) + .with_replay(); + export.next().await.unwrap().unwrap(); + let mut timestamps = Vec::new(); + while let Some(chunk) = export.next().await.unwrap() { + let mut cursor = Cursor::new(chunk); + let mut base = 0; + for tag in WebmIterator::new(&mut cursor, &[]) { + match tag.unwrap() { + MatroskaSpec::Timestamp(timestamp) => base = timestamp as i64, + MatroskaSpec::SimpleBlock(bytes) => { + let block = SimpleBlock::try_from(bytes.as_slice()).unwrap(); + timestamps.push(base + i64::from(block.timestamp)); + } + _ => {} + } + } + } + assert_eq!(timestamps, [0, 160, 80, 0, 40, 120, 400, 560, 480, 400, 440, 520]); +} + #[tokio::test(start_paused = true)] async fn export_header_roundtrip_vp9_opus() { // Build a tiny synthetic WebM with one VP9 video track and one Opus audio track. diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index bb20a3944f0..24f70ae4619 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -415,7 +415,7 @@ impl Track { .describe(stream_type, self.source.description(), name, self.sync.is_some()); } let pts = to_ticks(pending.frame.timestamp); - self.clock.push((pending, self.source.description().cloned()), pts); + self.clock.push((pending, self.source.description().cloned()), pts)?; self.release(name, jitter, false) } @@ -812,7 +812,7 @@ impl Track { let counted = self.sync.as_ref().and(self.timing.depth()); while let Some(((mut pending, description), dts)) = match counted { Some(depth) => self.clock.pop_counted(depth, delay, flush), - None => self.clock.pop(lookahead, delay, flush), + None => self.clock.settled(lookahead, delay, flush), } { if !self.finished { pending.arrived = pending.arrived.max(self.latest); @@ -875,7 +875,7 @@ impl Track { let arrival = Arrival { arrived: arrived - ahead, read: read - ahead, - decode: key, + decode: key.into(), restart, skip, cut, @@ -4273,12 +4273,12 @@ mod tests { let mut clock = DecodeClock::default(); let mut dts = vec![0; pts.len()]; for (i, &p) in pts.iter().enumerate() { - clock.push(i, p); - while let Some((i, d)) = clock.pop(jitter, floor, false) { + clock.push(i, p).unwrap(); + while let Some((i, d)) = clock.settled(jitter, floor, false) { dts[i] = d; } } - while let Some((i, d)) = clock.pop(jitter, floor, true) { + while let Some((i, d)) = clock.settled(jitter, floor, true) { dts[i] = d; } dts diff --git a/rs/moq-mux/src/container/ts/release.rs b/rs/moq-mux/src/container/ts/release.rs index d3096e2b0f4..591cbdec939 100644 --- a/rs/moq-mux/src/container/ts/release.rs +++ b/rs/moq-mux/src/container/ts/release.rs @@ -159,7 +159,7 @@ impl Release { let arrival = Arrival { arrived: arrived.max(self.subscribed), read: moq_clock::now(), - decode: frame.timestamp, + decode: frame.timestamp.into(), restart: track.markers(), skip: track.skips(), cut: track.cuts(), diff --git a/rs/moq-mux/src/jitter.rs b/rs/moq-mux/src/jitter.rs index 7495047a5ef..d44f43c2d4b 100644 --- a/rs/moq-mux/src/jitter.rs +++ b/rs/moq-mux/src/jitter.rs @@ -9,6 +9,26 @@ use std::time::Duration; use moq_clock::{Instant, Sleep}; use moq_net::Timestamp; +/// An authored decode instant in signed nanoseconds, including times before PTS zero. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub(crate) struct Decode(i128); + +impl Decode { + pub(crate) fn from_nanos(nanos: i128) -> Self { + Self(nanos) + } + + fn as_nanos(self) -> i128 { + self.0 + } +} + +impl From for Decode { + fn from(timestamp: Timestamp) -> Self { + Self::from_nanos(timestamp.as_nanos() as i128) + } +} + /// A frame handed to [`Buffer::push`]. pub(crate) struct Arrival { /// When the frame reached the reader: when it landed in the cache, or when the reader @@ -19,7 +39,7 @@ pub(crate) struct Arrival { /// so frames read together do not look fresher than they are. pub read: Instant, /// When the frame decodes, nondecreasing within a track. - pub decode: Timestamp, + pub decode: Decode, /// How many times the source had restarted its timeline when the frame was read. pub restart: u64, /// How many times the source's playhead had jumped (a skipped group, or a restart) @@ -199,7 +219,7 @@ const FIT: f64 = 1e-9; struct Clock { generation: u64, anchor: Instant, - base: Timestamp, + base: Decode, /// As a fraction of the decode timeline. drift: f64, } @@ -225,7 +245,7 @@ impl Acquire { /// decode time) is that track's slack less its queueing, and of those, the one with the least /// slack, so the track sent latest against its decode time still has the delay. fn fresh(&self) -> (&K, &Arrival) { - let behind = |frame: &Arrival| since(self.since, frame.read) - frame.decode.as_nanos() as i128; + let behind = |frame: &Arrival| since(self.since, frame.read) - frame.decode.as_nanos(); let mut freshest: BTreeMap<&K, &Arrival> = BTreeMap::new(); for (key, frame) in &self.frames { let best = freshest.entry(key).or_insert(frame); @@ -245,7 +265,7 @@ impl Acquire { fn end(&self, delay: Duration) -> Instant { let (_, fresh) = self.fresh(); let due = |frame: &Arrival| { - let ahead = frame.decode.as_nanos() as i128 - fresh.decode.as_nanos() as i128; + let ahead = frame.decode.as_nanos() - fresh.decode.as_nanos(); offset(fresh.read + delay, ahead) }; self.frames @@ -342,7 +362,7 @@ struct Floor { struct Track { /// Frames in arrival order, each with its generation, decode time and deadline. - queue: VecDeque<(u64, Timestamp, Instant, T)>, + queue: VecDeque<(u64, Decode, Instant, T)>, /// The source's restart, skip and cut counters at the last frame. restart: u64, skip: u64, @@ -813,18 +833,18 @@ impl Buffer { /// When a frame of `generation` decoding at `decode` goes out: on the current clock, or on the /// previous one while that generation's tail is still going out. - pub fn at(&self, generation: u64, decode: Timestamp) -> Option { + pub fn at(&self, generation: u64, decode: impl Into) -> Option { let clock = [self.clock, self.previous] .into_iter() .flatten() .find(|clock| clock.generation == generation)?; - self.deadline(&clock, decode) + self.deadline(&clock, decode.into()) } /// When a frame decoding at `decode` goes out on `clock`, if any instant holds it. - fn deadline(&self, clock: &Clock, decode: Timestamp) -> Option { + fn deadline(&self, clock: &Clock, decode: Decode) -> Option { // Signed, since a frame can decode before the one that anchored the clock. - let since = decode.as_nanos() as i128 - clock.base.as_nanos() as i128; + let since = decode.as_nanos() - clock.base.as_nanos(); offset( clock.anchor, since + (since as f64 * clock.drift) as i128 + self.delay.as_nanos() as i128, @@ -1109,6 +1129,61 @@ mod tests { const DELAY: Duration = Duration::from_millis(100); + #[moq_clock::test] + async fn decode_before_zero_keeps_its_deadline_and_media_order() { + let start = moq_clock::now(); + let mut buffer = Buffer::new(DELAY).replay(); + buffer.push(2, arrival(start, 0, "audio")).unwrap(); + let before_zero = Decode::from_nanos(-22_223); + buffer + .push( + 1, + Arrival { + decode: before_zero, + ..arrival(start, 0, "video") + }, + ) + .unwrap(); + assert_eq!( + buffer.at(0, before_zero), + Some(start + DELAY - Duration::from_nanos(22_223)) + ); + assert_eq!(buffer.at(0, ms(0)), Some(start + DELAY)); + moq_clock::sleep(DELAY).await; + assert_eq!(due(&mut buffer), ["video", "audio"]); + } + + #[moq_clock::test] + async fn moving_media_zero_preserves_acquisition_and_deadlines() { + let start = moq_clock::now(); + let mut negative = Buffer::new(DELAY); + let mut positive = Buffer::new(DELAY); + for buffer in [&mut negative, &mut positive] { + buffer.expect([1, 2]); + } + for (key, nanos, item) in [(1, -22_223, "video"), (2, 0, "audio"), (1, -11_112, "next")] { + for (buffer, offset) in [(&mut negative, 0), (&mut positive, 1_000_000_000)] { + buffer + .push( + key, + Arrival { + decode: Decode::from_nanos(nanos + offset), + ..arrival(start, 0, item) + }, + ) + .unwrap(); + } + } + moq_clock::sleep(DELAY).await; + assert_eq!(due(&mut negative), due(&mut positive)); + for nanos in [-22_223, -11_112, 0, 11_111] { + assert_eq!( + negative.at(0, Decode::from_nanos(nanos)), + positive.at(0, Decode::from_nanos(nanos + 1_000_000_000)), + ); + } + } + /// Two floors from the recovery arm's exact audio clock and wandering video lead. /// Both envelopes interpolate their two samples, so rounding in their equal depths /// must not make the changing send-ahead look like source clock drift. @@ -1133,7 +1208,7 @@ mod tests { Arrival { arrived, read: arrived, - decode: ms(decode), + decode: ms(decode).into(), restart: 0, skip: 0, cut: 0, @@ -1747,7 +1822,7 @@ mod tests { margin: None, offset: None, }; - let mut last: Option<(f64, Timestamp)> = None; + let mut last: Option<(f64, Decode)> = None; let mut stepped = false; for k in 0..(hours * 3_600.0 * 2.0) as u64 { let decode = Duration::from_millis(k * 500); @@ -1764,7 +1839,7 @@ mod tests { let frame = Arrival { arrived, read: arrived, - decode: Timestamp::from_micros(decode.as_micros() as u64).unwrap(), + decode: Timestamp::from_micros(decode.as_micros() as u64).unwrap().into(), restart: 0, skip: 0, cut: 0, @@ -1950,7 +2025,7 @@ mod tests { let frame = Arrival { arrived, read: arrived, - decode: Timestamp::from_micros(decode.as_micros() as u64).unwrap(), + decode: Timestamp::from_micros(decode.as_micros() as u64).unwrap().into(), restart: 0, skip: 0, cut: 0, From 2f539bbece39ed620cd1b416651be35daa43e5cb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 10:47:39 -0700 Subject: [PATCH 03/11] fix(mux): clock declared catalog reorder per group in TS, MKV and FLV Absent reorder keeps the inferred clock; an explicit declaration, zero included, decodes each group from its own frames. Co-Authored-By: Claude Opus 5.5 --- doc/concept/hang.md | 1 + drafts/draft-lcurley-moq-hang.md | 13 +- js/hang/src/catalog/video.test.ts | 8 + js/hang/src/catalog/video.ts | 6 + rs/hang/src/catalog/root.rs | 1 + rs/hang/src/catalog/video/mod.rs | 23 + rs/moq-mux/benches/export.rs | 3 +- rs/moq-mux/src/codec/video.rs | 66 +++ rs/moq-mux/src/container/decode.rs | 442 ++++++++++-------- rs/moq-mux/src/container/flv/export.rs | 212 +++++++-- rs/moq-mux/src/container/flv/export_test.rs | 125 +++++ rs/moq-mux/src/container/mkv/export.rs | 173 +++---- rs/moq-mux/src/container/mkv/export_test.rs | 9 +- rs/moq-mux/src/container/ts/export.rs | 84 +++- rs/moq-mux/src/container/ts/export_test.rs | 109 +++++ .../src/container/ts/export_timing_test.rs | 48 ++ rs/moq-mux/src/jitter.rs | 5 + 17 files changed, 988 insertions(+), 340 deletions(-) diff --git a/doc/concept/hang.md b/doc/concept/hang.md index 841f74d2b0b..c780a908b67 100644 --- a/doc/concept/hang.md +++ b/doc/concept/hang.md @@ -67,6 +67,7 @@ A few things the catalog can express beyond decoder config: - **Track names.** The map key is a rendition's ID and, by default, its track name. A rendition sets `track` only when the two must differ, such as two renditions of same-named tracks in different broadcasts, or a rendition whose media changed and so needs a new name; readers subscribe to `track` when present. An ID can't name both a video and an audio rendition. - **Jitter.** A rendition can say how far its frames fell behind the media clock before the publisher flushed them, in whole milliseconds rounded up. Encoders report the spread of lateness above each rendition's own recent minimum, so a constant encoder delay is not jitter; container imports estimate batch spans without counting ingest delay. It describes the publisher, never the network, only grows over the life of a stream, and a player sizes its buffer to at least this much. A `0` is read as absent. - **Delay.** A rendition can also say how far its frames reach the transport behind the broadcast's earliest rendition, measured the same way from each rendition's minimum lateness, so a video encoder running 200 ms behind audio advertises `delay: 200` on video. It follows the same rules as jitter. A player holds the largest `delay + jitter` among the renditions it subscribes to, and never subtracts one rendition's `delay` from another's. +- **Reorder depth.** A video rendition may declare `reorder`, the most frames that can precede one in decode order and follow it in presentation, as H.264 `max_num_reorder_frames` counts them; `0` declares none, which is not the same as absent. The Rust H.264 and H.265 importers copy it from the SPS when the SPS declares one (progressive H.264 only), and `@moq/publish` declares `0`. The MPEG-TS, Matroska, and FLV exporters then derive each group's decode times from that group's frames alone, so exports that joined at different groups agree, and they end the export on a frame reordered deeper than declared or a depth the codec level cannot reach. Without it they infer decode times from the timestamps they have read, as before. A publisher never changes it once declared. - **Colour.** A video rendition may carry `color`: the H.273 primaries, transfer, matrix, and range, plus the HDR10 mastering display and content light levels, so a renderer can set up HDR without parsing the bitstream. The Rust importers fill it from the SPS VUI, AV1 sequence header, and VP9 header, then fill gaps from a container's `colr` box or Matroska `Colour` element; the stream wins a disagreement. The HDR10 values come from H.264 and H.265 SEI or AV1 metadata OBUs in MPEG-TS and raw elementary streams, an AV1 track's av1C, or Matroska, but not yet from H.264 or H.265 in fMP4 or FLV. Nothing in `@moq/watch` reads it yet. - **Spatial media.** A video rendition may carry `projection` (flat `rectilinear`, or `equirectangular` with the degrees of longitude it spans, 180 for VR180) and `stereo` (`mono`, `left-right`, or `top-bottom` frame packing), and an audio rendition may carry `layout: "ambisonic"` for AmbiX channels. Readers treat a value they don't recognize as absent. Nothing in `@moq/watch` renders them yet. - **Disabled renditions.** An audio or video rendition with `enabled: false` has no frames coming, and a player must not select it. Pausing a rendition in `@moq/publish` keeps it in the catalog this way rather than removing it, so a mute is a one-field update; a disabled video rendition encodes one black keyframe first, so a player that predates the field shows black instead of a frozen picture. The field is written only when false. The older video `stalled` field is no longer written, and readers ignore it. diff --git a/drafts/draft-lcurley-moq-hang.md b/drafts/draft-lcurley-moq-hang.md index fe79214a0f0..159dcf25d6c 100644 --- a/drafts/draft-lcurley-moq-hang.md +++ b/drafts/draft-lcurley-moq-hang.md @@ -194,7 +194,7 @@ This draft needs `version` 4 for: - a `track` ({{field-track}}) other than the entry's ID, which an older consumer subscribes to instead; - a video `warmup` ({{field-warmup}}) above zero, since an older consumer starts decoding a group at a picture that is not a keyframe. -Every other catalog change in this draft is safe to ignore: `enabled` ({{field-enabled}}) leaves an older consumer selecting a disabled rendition, which plays nothing; `archive` ({{archive-catalog}}) is ignored when undecodable; `stats` and `echo` ({{media-stats}}) only offer optional tracks; `jitter`, `delay`, audio `warmup`, and the data track estimates are hints. +Every other catalog change in this draft is safe to ignore: `enabled` ({{field-enabled}}) leaves an older consumer selecting a disabled rendition, which plays nothing; `archive` ({{archive-catalog}}) is ignored when undecodable; `stats` and `echo` ({{media-stats}}) only offer optional tracks; video `reorder` leaves an older consumer inferring decode times from timestamps; `jitter`, `delay`, audio `warmup`, and the data track estimates are hints. ## Clock {#clock} The catalog's root `clock` field is the broadcast's one continuous clock: @@ -258,6 +258,7 @@ type VideoDecoderConfigExtensions = { "displayAspectWidth": number | undefined, "displayAspectHeight": number | undefined, "preference": number | undefined, + "reorder": number | undefined, "color": Color | undefined, "projection": Projection | undefined, "stereo": string | undefined, @@ -275,6 +276,16 @@ A consumer selecting a rendition automatically MUST choose among the enabled ({{ A consumer MAY select another rendition at the viewer's explicit request. A consumer that carries every rendition, rather than selecting one, ignores the field. +`reorder` is the most frames that can precede a frame in decode order and follow it in presentation order, the unit of H.264 `max_num_reorder_frames` and HEVC `sps_max_num_reorder_pics`, as an integer from 0 to 255. +`0` declares that presentation order is decode order. +A publisher that does not know the depth MUST omit the field; absent declares nothing. +A publisher that declares it MUST NOT send a frame that more than `reorder` earlier frames of its group follow in presentation order, and MUST NOT change or remove it for the track once declared. + +A consumer that authors decode times, such as a transmuxer to a container that carries them, derives them from a declared `reorder` and the frames of the current group alone, so consumers that joined at different groups agree. +The n-th frame of a group in decode order decodes at the group's (n - `reorder`)-th earliest presentation time, counting from zero, and the group's first `reorder` frames decode at successive instants just before its earliest presentation time. +A consumer MUST refuse a frame reordered deeper than its rendition declares rather than author an invalid decode time, and SHOULD refuse a `reorder` the codec cannot reach: above the level's decoded picture buffer, or nonzero for a codec whose presentation order is its decode order. +Without `reorder`, a consumer infers decode times from the timestamps it has read. + `color` describes the decoded pixels so a renderer can configure its pipeline without parsing the bitstream: ~~~ diff --git a/js/hang/src/catalog/video.test.ts b/js/hang/src/catalog/video.test.ts index 8f80a4c0d68..ddb8ddff404 100644 --- a/js/hang/src/catalog/video.test.ts +++ b/js/hang/src/catalog/video.test.ts @@ -53,6 +53,14 @@ test("video config accepts a signed 32-bit preference", () => { expect(parse(2 ** 31).success).toBe(false); }); +test("video config keeps a zero reorder distinct from an absent one", () => { + const parse = (reorder?: number) => + VideoConfigSchema.safeParse({ codec: "vp8", container: { kind: "legacy" }, reorder }); + expect(parse().data?.reorder).toBeUndefined(); + expect(parse(0).data?.reorder).toBe(0); + expect(parse(256).success).toBe(false); +}); + test("ranked puts a known zero bitrate above an unknown one", () => { const renditions = { unknown: rendition({ codedWidth: 1280, codedHeight: 720 }), diff --git a/js/hang/src/catalog/video.ts b/js/hang/src/catalog/video.ts index 1d4ffdd5e86..17e28527e0a 100644 --- a/js/hang/src/catalog/video.ts +++ b/js/hang/src/catalog/video.ts @@ -136,6 +136,12 @@ export const VideoConfigSchema = z.object({ // The maximum frame rate of the video in frames per second. framerate: z.optional(z.number()), + // The most frames that can precede a frame in decode order and follow it in presentation order, + // as H.264 max_num_reorder_frames and HEVC sps_max_num_reorder_pics count them. 0 declares that + // presentation order is decode order. Absent declares nothing: a consumer that authors decode + // times infers them from the timestamps instead. Once declared, it holds for the track's life. + reorder: z.optional(u8Schema), + // The maximum bitrate of the video in bits per second. // TODO: Support up to Number.MAX_SAFE_INTEGER bitrate: z.optional(u53Schema), diff --git a/rs/hang/src/catalog/root.rs b/rs/hang/src/catalog/root.rs index 28f70e0ae97..a3cb63018c0 100644 --- a/rs/hang/src/catalog/root.rs +++ b/rs/hang/src/catalog/root.rs @@ -699,6 +699,7 @@ mod test { enabled: true, preference: 0, framerate: None, + reorder: None, optimize_for_latency: None, container: Container::Legacy, warmup: None, diff --git a/rs/hang/src/catalog/video/mod.rs b/rs/hang/src/catalog/video/mod.rs index b7d33daf7c8..103297c4dfd 100644 --- a/rs/hang/src/catalog/video/mod.rs +++ b/rs/hang/src/catalog/video/mod.rs @@ -290,6 +290,15 @@ pub struct VideoConfig { #[serde(default)] pub framerate: Option, + /// The most frames that can precede a frame in decode order and follow it in presentation + /// order, as H.264 `max_num_reorder_frames` and HEVC `sps_max_num_reorder_pics` count them. + /// + /// `0` declares that presentation order is decode order. Absent declares nothing: a consumer + /// that authors decode times infers them from the timestamps instead. A publisher that + /// declares it keeps it for the track's life, and never reorders deeper. + #[serde(default)] + pub reorder: Option, + /// If true, the decoder will optimize for latency. /// /// Default: true @@ -364,6 +373,7 @@ impl VideoConfig { enabled: true, preference: 0, framerate: None, + reorder: None, optimize_for_latency: None, container: Container::default(), jitter: None, @@ -396,6 +406,19 @@ mod test { } } + #[test] + fn reorder_keeps_zero_distinct_from_absent() { + let mut config = VideoConfig::new(VideoCodec::VP8); + assert!(serde_json::to_value(&config).unwrap().get("reorder").is_none()); + for depth in [0, 2] { + config.reorder = Some(depth); + let encoded = serde_json::to_value(&config).unwrap(); + assert_eq!(encoded["reorder"], depth); + let decoded: VideoConfig = serde_json::from_value(encoded).unwrap(); + assert_eq!(decoded.reorder, Some(depth)); + } + } + #[test] fn warmup_rounds_up() { let mut config = VideoConfig::new(VideoCodec::VP8); diff --git a/rs/moq-mux/benches/export.rs b/rs/moq-mux/benches/export.rs index 040636fd098..ac576fc5b1b 100644 --- a/rs/moq-mux/benches/export.rs +++ b/rs/moq-mux/benches/export.rs @@ -136,7 +136,7 @@ fn bench_export(c: &mut Criterion) { criterion_group!(benches, bench_export, video_reorder); criterion_main!(benches); -/// A progressive AVC SPS with a declared reorder depth of two (25 fps). +/// A progressive AVC SPS declaring a reorder depth of two (25 fps), as the catalog does. const REORDER_SPS: &[u8] = &[ 0x67, 0x64, 0x00, 0x0a, 0xac, 0xd9, 0x44, 0x26, 0xc0, 0x44, 0x00, 0x00, 0x03, 0x00, 0x04, 0x00, 0x00, 0x03, 0x00, 0xca, 0x3c, 0x48, 0x96, 0x58, @@ -162,6 +162,7 @@ async fn prepare_video( }); config.coded_width = Some(64); config.coded_height = Some(64); + config.reorder = Some(2); let mut description = vec![1, 100, 0, 10, 0xff, 0xe1]; description.extend_from_slice(&(REORDER_SPS.len() as u16).to_be_bytes()); description.extend_from_slice(REORDER_SPS); diff --git a/rs/moq-mux/src/codec/video.rs b/rs/moq-mux/src/codec/video.rs index 66cb1c1f192..3e688e1b92b 100644 --- a/rs/moq-mux/src/codec/video.rs +++ b/rs/moq-mux/src/codec/video.rs @@ -117,6 +117,72 @@ impl Reorder { } } +/// The deepest catalog `reorder` a rendition's codec and level can reach: what fits the level's +/// decoded picture buffer at the coded size (H.264 A.3.1 MaxDpbFrames, H.265 A.4.2 MaxDpbSize +/// less the current picture), or the codec's largest buffer when the level or size is unknown. +/// VP8, VP9 and AV1 present in decode order. +pub(crate) fn max_reorder(config: &hang::catalog::VideoConfig) -> u32 { + use hang::catalog::VideoCodec; + + match &config.codec { + VideoCodec::H264(h264) => { + // level_idc 11 with constraint_set3_flag is level 1b for Baseline, Main and Extended. + let level_1b = h264.level == 11 && h264.constraints & 0x10 != 0 && matches!(h264.profile, 66 | 77 | 88); + let max_dpb_mbs: Option = match h264.level { + _ if level_1b => Some(396), + 9 | 10 => Some(396), + 11 => Some(900), + 12 | 13 | 20 => Some(2_376), + 21 => Some(4_752), + 22 | 30 => Some(8_100), + 31 => Some(18_000), + 32 => Some(20_480), + 40 | 41 => Some(32_768), + 42 => Some(34_816), + 50 => Some(110_400), + 51 | 52 => Some(184_320), + 60..=62 => Some(696_320), + _ => None, + }; + let frame_mbs = config + .coded_width + .zip(config.coded_height) + .map(|(width, height)| u64::from(width.div_ceil(16)) * u64::from(height.div_ceil(16))); + match max_dpb_mbs.zip(frame_mbs.filter(|mbs| *mbs > 0)) { + Some((dpb, frame)) => (dpb / frame).min(u64::from(Reorder::MAX_DEPTH)) as u32, + None => Reorder::MAX_DEPTH, + } + } + VideoCodec::H265(h265) => { + let max_luma_ps: Option = match h265.level_idc { + 30 => Some(36_864), + 60 => Some(122_880), + 63 => Some(245_760), + 90 => Some(552_960), + 93 => Some(983_040), + 120 | 123 => Some(2_228_224), + 150 | 153 | 156 => Some(8_912_896), + 180 | 183 | 186 => Some(35_651_584), + _ => None, + }; + let samples = config + .coded_width + .zip(config.coded_height) + .map(|(width, height)| u64::from(width) * u64::from(height)); + let max_dpb_size = match max_luma_ps.zip(samples) { + Some((luma, size)) if size <= luma >> 2 => 16, + Some((luma, size)) if size <= luma >> 1 => 12, + Some((luma, size)) if size <= (3 * luma) >> 2 => 8, + Some(_) => 6, + None => 16, + }; + max_dpb_size - 1 + } + VideoCodec::VP8 | VideoCodec::VP9(_) | VideoCodec::AV1(_) => 0, + _ => Reorder::MAX_DEPTH, + } +} + /// The hypothetical reference decoder a video sequence header declares: its NAL HRD's last /// schedule, the one the transport stream's decoder buffers are sized from (ISO 13818-1 /// 2.14.3.1, 2.17.2). diff --git a/rs/moq-mux/src/container/decode.rs b/rs/moq-mux/src/container/decode.rs index 26e025211e1..15ad9523170 100644 --- a/rs/moq-mux/src/container/decode.rs +++ b/rs/moq-mux/src/container/decode.rs @@ -1,25 +1,41 @@ use std::collections::VecDeque; +use hang::catalog::VideoConfig; + /// The largest supported reorder span in 90 kHz ticks. pub(crate) const MAX_REORDER: u64 = 2 * 90_000; /// A video rendition's decode clock, authored from presentation timestamps. /// -/// Declared-depth clocks release the n-th group frame at its (n - depth)-th -/// presentation slot. The first depth slots precede the group's keyframe by one -/// 90 kHz tick each, so DTS may be negative. Draining after every push bounds the -/// held payloads by the depth, independently of frame spacing and group length. -/// Group completion flushes the remaining slots; a cut discards them and resets -/// the next group's origin. Invalid slots fail rather than move the timeline. +/// [`crate::container::Frame`] carries only a presentation timestamp and frames reach the muxer in decode order, +/// so a B-frame stream arrives with non-monotonic PTS and no decode time. An encoder decodes +/// the n-th frame in decode order at the n-th presentation time, a fixed reorder delay early, +/// so that is the DTS authored here: with decode-order PTS 0, 120, 40, 80 and a delay of one +/// 40 ms frame, the frames decode at -40, 0, 40, 80. It keeps the encoder's own spacing, a +/// frame's or a field's, through any switch between frame and field coding, and the delay is +/// a time, the least that keeps every DTS at or before its PTS (or the SPS's declared depth, +/// when that is more). On a source muxed that way this is its DTS, frame for frame. +/// +/// A frame's slot is settled once no frame still to come can be presented before it. A later +/// frame decodes after this one, and is presented at most the catalog `jitter` after it +/// decodes, so the slot is settled once the highest PTS read is that far past it. Frames wait +/// here until then, as long as the reordering spans and well inside the jitter buffer's delay. +/// Without a `jitter`, the SPS's depth bounds it, and so does how far a frame was presented +/// below the highest PTS read before it. A frame that still comes in below a settled slot +/// (reordering deeper than all of those) is nudged one tick past the last DTS, so the clock +/// stays strictly increasing. +/// +/// That settling, and a delay that only grows, depend on everything read before, so two +/// exporters that joined or lost frames at different points can clock the same frame apart. +/// [`Self::pop_counted`] depends on the frames held alone, for exporters that must agree. +/// A rendition that declares its catalog `reorder` gets a [`DeclaredClock`] instead. /// -/// Undeclared legacy renditions still settle slots by timestamp lookahead, or -/// use a caller-supplied counted delay. Those paths retain their timing history. -/// Ticks are continuous 90 kHz; a container applies its wire representation later. +/// Ticks are continuous (unwrapped) 90 kHz, so the clock never wraps mid-stream; the 33-bit +/// wire wrap happens once at emission. pub(crate) struct DecodeClock { - fixed: Option, /// Frames read but not yet given a DTS, in decode order, with their PTS. held: VecDeque<(T, u64)>, - /// Presentation slots not yet assigned to a frame, in ascending order. + /// The held frames' PTS, ascending: the slots they decode in, in turn. slots: Vec, /// The highest PTS read. high: Option, @@ -34,7 +50,6 @@ pub(crate) struct DecodeClock { impl Default for DecodeClock { fn default() -> Self { Self { - fixed: None, held: VecDeque::new(), slots: Vec::new(), high: None, @@ -46,85 +61,22 @@ impl Default for DecodeClock { } impl DecodeClock { - /// Author group-local decode slots at this rendition's fixed reorder depth. - pub(crate) fn new(reorder: usize) -> crate::Result { - if reorder > crate::codec::video::Reorder::MAX_DEPTH as usize { - return Err(ClockError::Depth(reorder).into()); - } - Ok(Self { - fixed: Some(Fixed { - depth: reorder, - leading: reorder, - first: None, - last: None, - highest: Vec::new(), - }), - ..Self::default() - }) - } - - pub(crate) fn is_fixed(&self) -> bool { - self.fixed.is_some() - } - - #[cfg(test)] - pub(crate) fn len(&self) -> usize { - self.held.len() - } - /// Hold the next frame in decode order, presented at `pts`. - pub(crate) fn push(&mut self, item: T, pts: u64) -> crate::Result<()> { - if let Some(fixed) = &mut self.fixed { - if self.held.len() > fixed.depth { - return Err(ClockError::Undrained.into()); - } - let at = fixed.highest.partition_point(|&prior| prior <= pts); - if fixed.highest.len() - at > fixed.depth { - return Err(ClockError::Reordered(fixed.depth).into()); - } - fixed.highest.insert(at, pts); - if fixed.highest.len() > fixed.depth + 1 { - fixed.highest.remove(0); - } - fixed.first.get_or_insert(pts); - } else { - if let Some(high) = self.high - && pts < high - { - self.reach = self.reach.max(high - pts + 1).min(MAX_REORDER); - } - self.high = Some(self.high.map_or(pts, |high| high.max(pts))); + pub(crate) fn push(&mut self, item: T, pts: u64) { + if let Some(high) = self.high + && pts < high + { + self.reach = self.reach.max(high - pts + 1).min(MAX_REORDER); } + self.high = Some(self.high.map_or(pts, |high| high.max(pts))); let at = self.slots.partition_point(|&slot| slot <= pts); self.slots.insert(at, pts); self.held.push_back((item, pts)); - Ok(()) - } - - /// Release a frame once its declared lookahead is present, or the group is complete. - pub(crate) fn pop(&mut self, flush: bool) -> crate::Result> { - let fixed = self.fixed.as_mut().ok_or(ClockError::Undeclared)?; - if self.held.is_empty() || (!flush && self.held.len() <= fixed.depth) { - return Ok(None); - } - let dts = match fixed.leading { - 0 => i128::from(self.slots.remove(0)), - leading => { - fixed.leading -= 1; - i128::from(fixed.first.expect("a held frame supplies the origin")) - leading as i128 - } - }; - let (item, pts) = self.held.pop_front().expect("the held queue is nonempty"); - if dts > i128::from(pts) || fixed.last.is_some_and(|last| dts <= last) { - return Err(ClockError::Presentation.into()); - } - fixed.last = Some(dts); - Ok(Some((item, dts))) } /// The next frame and its DTS, once its slot is settled `lookahead` past (the catalog /// `jitter`), or at once when `flush`. `floor` is the least reorder delay, in ticks. - pub(crate) fn settled(&mut self, lookahead: u64, floor: u64, flush: bool) -> Option<(T, u64)> { + pub(crate) fn pop(&mut self, lookahead: u64, floor: u64, flush: bool) -> Option<(T, u64)> { let high = self.high?; let lookahead = lookahead.max(self.reach); let settled = |slot: u64| flush || high >= slot.saturating_add(lookahead); @@ -165,12 +117,6 @@ impl DecodeClock { pub(crate) fn discard(&mut self) { self.held.clear(); self.slots.clear(); - if let Some(fixed) = &mut self.fixed { - fixed.leading = fixed.depth; - fixed.first = None; - fixed.last = None; - fixed.highest.clear(); - } } fn after_last(&mut self, dts: u64) -> u64 { @@ -183,27 +129,126 @@ impl DecodeClock { } } -struct Fixed { +/// The reorder depth `config` declares, or `None` when it declares none. +/// +/// Refuses a depth its codec and level cannot reach, so a declared clock never holds more frames +/// than a conforming decoder's picture buffer. +pub(crate) fn declared(config: &VideoConfig) -> crate::Result> { + let Some(reorder) = config.reorder else { + return Ok(None); + }; + let max = crate::codec::video::max_reorder(config); + if u32::from(reorder) > max { + return Err(ClockError::Depth { + reorder, + max, + codec: config.codec.to_string(), + } + .into()); + } + Ok(Some(usize::from(reorder))) +} + +/// One group's decode clock, at its rendition's declared reorder depth. +/// +/// The n-th frame of a group in decode order decodes at the group's (n - depth)-th presentation +/// slot. Once the group's first depth + 1 frames are read, no frame still to come can be presented +/// below the earliest of them, so from then on each frame is released as it is read: it takes the +/// least slot not yet taken, which a later frame could undercut only by following more than depth +/// frames in presentation order. The first depth frames decode one tick apart up to that earliest +/// slot, which needs no frame spacing, so a DTS can fall before the timeline's zero. A tick is the +/// caller's timescale, the finest its container writes: 90 kHz for MPEG-TS, 1 ms for FLV. +/// +/// A clock lives for one group, so its DTS depends on nothing read before the group, and a frame +/// reordered deeper than declared fails rather than move the timeline. +pub(crate) struct DeclaredClock { depth: usize, + /// The group's earliest PTS, once enough frames are read that none to come can precede it. + earliest: Option, + /// Frames still to be released that decode before [`Self::earliest`]. leading: usize, - first: Option, - last: Option, - // Only the largest depth + 1 earlier PTS are needed to detect a deeper reorder. + /// Frames read but not yet released, in decode order, with their PTS. + held: VecDeque<(T, u64)>, + /// Presentation slots not yet taken, ascending. + slots: Vec, + /// The largest depth + 1 PTS read, ascending: all a deeper reorder is checked against. highest: Vec, + /// The last DTS released. + last: Option, +} + +impl DeclaredClock { + /// A clock for one group of a rendition reordering at most `depth` frames, from [`declared`]. + pub(crate) fn new(depth: usize) -> Self { + Self { + depth, + earliest: None, + leading: depth, + held: VecDeque::new(), + slots: Vec::new(), + highest: Vec::new(), + last: None, + } + } + + /// How many frames are held. + #[cfg(test)] + pub(crate) fn len(&self) -> usize { + self.held.len() + } + + /// Hold the group's next frame in decode order, presented at `pts`. + pub(crate) fn push(&mut self, item: T, pts: u64) -> crate::Result<()> { + if self.held.len() > self.depth { + return Err(ClockError::Undrained.into()); + } + let at = self.highest.partition_point(|&prior| prior <= pts); + if self.highest.len() - at > self.depth { + return Err(ClockError::Reordered(self.depth).into()); + } + self.highest.insert(at, pts); + if self.highest.len() > self.depth + 1 { + self.highest.remove(0); + } + let at = self.slots.partition_point(|&slot| slot <= pts); + self.slots.insert(at, pts); + self.held.push_back((item, pts)); + Ok(()) + } + + /// The next frame and its DTS in signed ticks, once the group's earliest slot is known, + /// or at once when `flush` because the group ended. + pub(crate) fn pop(&mut self, flush: bool) -> crate::Result> { + if self.held.is_empty() || (self.earliest.is_none() && !flush && self.held.len() <= self.depth) { + return Ok(None); + } + let earliest = *self.earliest.get_or_insert(self.slots[0]); + let dts = match self.leading { + 0 => i128::from(self.slots.remove(0)), + leading => { + self.leading -= 1; + i128::from(earliest) - leading as i128 + } + }; + let (item, pts) = self.held.pop_front().expect("the held queue is nonempty"); + if dts > i128::from(pts) || self.last.is_some_and(|last| dts <= last) { + return Err(ClockError::Presentation.into()); + } + self.last = Some(dts); + Ok(Some((item, dts))) + } } #[derive(Debug, thiserror::Error)] enum ClockError { - #[error("reorder depth {0} exceeds the supported codec depth")] - Depth(usize), + #[error("video reorder depth {reorder} exceeds the {max} that {codec} can reach")] + Depth { reorder: u8, max: u32, codec: String }, #[error("video reorders past its declared depth {0}")] Reordered(usize), #[error("video presentation slots cannot author increasing DTS at or before PTS")] Presentation, #[error("the decode clock must be drained before reading another frame")] Undrained, - #[error("the decode clock has no fixed reorder declaration")] - Undeclared, } impl From for crate::Error { @@ -216,92 +261,6 @@ impl From for crate::Error { mod tests { use super::*; - fn fixed_slots(read: &[u64]) -> Vec { - let mut clock = DecodeClock::new(2).unwrap(); - let mut out = Vec::new(); - for &pts in read { - clock.push((), pts).unwrap(); - while let Some(((), dts)) = clock.pop(false).unwrap() { - out.push(dts); - } - assert!(clock.held.len() <= 2); - assert!(clock.slots.len() <= 4); - } - while let Some(((), dts)) = clock.pop(true).unwrap() { - out.push(dts); - } - out - } - - #[test] - fn fixed_slots_shift_by_count_and_start_before_zero() { - let read = PYRAMID.map(|i| i * FRAME); - let want: Vec = [-2, -1].into_iter().chain((0..7).map(|i| i * FRAME as i128)).collect(); - assert_eq!(fixed_slots(&read), want); - let far = [0, 6, 10, 1, 2, 3, 4, 5, 7, 8, 9].map(|i| i * FRAME); - let want: Vec = [-2, -1].into_iter().chain((0..9).map(|i| i * FRAME as i128)).collect(); - assert_eq!(fixed_slots(&far), want); - } - - #[test] - fn a_fixed_group_keeps_no_previous_groups_history() { - let next = PYRAMID.map(|i| (i + 20) * FRAME); - let want = fixed_slots(&next); - for count in 0..PYRAMID.len() { - let mut clock = DecodeClock::new(2).unwrap(); - for pts in PYRAMID[..count].iter().map(|i| (i + 2) * FRAME) { - clock.push((), pts).unwrap(); - while clock.pop(false).unwrap().is_some() {} - } - clock.discard(); - let mut out = Vec::new(); - for pts in next { - clock.push((), pts).unwrap(); - while let Some(((), dts)) = clock.pop(false).unwrap() { - out.push(dts); - } - } - while let Some(((), dts)) = clock.pop(true).unwrap() { - out.push(dts); - } - assert_eq!(out, want, "lost prefix of {count} frames"); - } - } - - #[test] - fn malformed_presentation_is_refused_without_retaining_its_payload() { - for backwards in [false, true] { - for count in [16, 128, 1024] { - for size in [256, 16_384] { - let mut clock = DecodeClock::new(2).unwrap(); - let mut refused = false; - for index in 0..count { - let pts = 90_000 - if backwards { index } else { 0 }; - if clock.push(vec![0u8; size], pts).is_err() || clock.pop(false).is_err() { - refused = true; - break; - } - assert!(clock.held.len() <= 2); - assert!(clock.slots.len() <= 4); - } - assert!(refused); - assert!(clock.held.len() <= 2); - } - } - } - } - - #[test] - fn a_zero_depth_has_no_lookahead_or_decode_delay() { - let mut clock = DecodeClock::new(0).unwrap(); - for pts in [0, 1, 90_000, u64::MAX] { - clock.push(pts, pts).unwrap(); - assert_eq!(clock.pop(false).unwrap(), Some((pts, i128::from(pts)))); - assert!(clock.held.is_empty()); - } - assert!(DecodeClock::<()>::new(17).is_err()); - } - /// A B-pyramid (`max_num_reorder_frames` 2) in decode order, in 40 ms frames. const PYRAMID: [u64; 9] = [0, 4, 2, 1, 3, 8, 6, 5, 7]; const FRAME: u64 = 3_600; @@ -313,10 +272,10 @@ mod tests { let mut out = Vec::new(); let pop = |clock: &mut DecodeClock, flush| match counted { true => clock.pop_counted(2, 2 * FRAME, flush), - false => clock.settled(4 * FRAME, 2 * FRAME, flush), + false => clock.pop(4 * FRAME, 2 * FRAME, flush), }; for &pts in read { - clock.push(pts, pts).unwrap(); + clock.push(pts, pts); out.extend(std::iter::from_fn(|| pop(&mut clock, false))); } out.extend(std::iter::from_fn(|| pop(&mut clock, true))); @@ -368,16 +327,127 @@ mod tests { let mut clock = DecodeClock::default(); let mut out = Vec::new(); for &pts in &first[..8] { - clock.push(pts, pts).unwrap(); + clock.push(pts, pts); out.extend(std::iter::from_fn(|| clock.pop_counted(2, 2 * FRAME, false))); } clock.discard(); for pts in next { - clock.push(pts, pts).unwrap(); + clock.push(pts, pts); out.extend(std::iter::from_fn(|| clock.pop_counted(2, 2 * FRAME, false))); } out.extend(std::iter::from_fn(|| clock.pop_counted(2, 2 * FRAME, true))); assert!(out.iter().all(|frame| full.contains(frame)), "{out:?}"); assert_eq!(out.len(), 6 + next.len()); } + + /// The DTS of one group read through a declared clock of `depth`, released as each frame is + /// read, checking it never holds more than `depth` frames between reads. + fn declared_dts(depth: usize, read: &[u64]) -> Vec { + let mut clock = DeclaredClock::new(depth); + let mut out = Vec::new(); + for &pts in read { + clock.push((), pts).unwrap(); + while let Some(((), dts)) = clock.pop(false).unwrap() { + out.push(dts); + } + assert!(clock.len() <= depth); + } + while let Some(((), dts)) = clock.pop(true).unwrap() { + out.push(dts); + } + out + } + + #[test] + fn declared_slots_shift_by_count_and_start_before_zero() { + let ticks = |frames: &[i128]| frames.iter().map(|i| i * FRAME as i128).collect::>(); + let leading = |dts: Vec| [vec![-2, -1], dts].concat(); + assert_eq!( + declared_dts(2, &PYRAMID.map(|i| i * FRAME)), + leading(ticks(&[0, 1, 2, 3, 4, 5, 6])) + ); + // References spaced wider than any frame period still take their slots by count. + let far = [0, 6, 10, 1, 2, 3, 4, 5, 7, 8, 9].map(|i| i * FRAME); + assert_eq!(declared_dts(2, &far), leading(ticks(&[0, 1, 2, 3, 4, 5, 6, 7, 8]))); + } + + /// Open-GOP leading pictures present before the keyframe that starts their group, so the + /// group's first frames decode below its earliest slot, not the keyframe's. + #[test] + fn leading_pictures_decode_before_the_keyframe() { + let read = [10, 8, 9, 14, 12, 11, 13]; + assert_eq!(declared_dts(2, &read), [6, 7, 8, 9, 10, 11, 12]); + // A group that ends before the lookahead fills still takes its earliest slot. + assert_eq!(declared_dts(2, &read[..2]), [6, 7]); + } + + #[test] + fn a_zero_depth_releases_each_frame_at_its_pts() { + let mut clock = DeclaredClock::new(0); + for pts in [0, 1, 90_000, u64::MAX] { + clock.push(pts, pts).unwrap(); + assert_eq!(clock.pop(false).unwrap(), Some((pts, i128::from(pts)))); + assert_eq!(clock.len(), 0); + } + clock.push(0, 0).unwrap_err(); + } + + /// Flat or backward PTS read past the declared depth fail within depth + 1 frames, however + /// many follow and however large they are, rather than accumulate. + #[test] + fn malformed_presentation_is_refused_without_retaining_its_payload() { + for backwards in [false, true] { + for count in [16, 128, 1024] { + for size in [256, 16_384] { + let mut clock = DeclaredClock::new(2); + let refused = (0..count).position(|index| { + let pts = 90_000 - if backwards { index } else { 0 }; + let refused = clock.push(vec![0u8; size], pts).is_err() + || std::iter::from_fn(|| clock.pop(false).transpose()).any(|popped| popped.is_err()); + assert!(clock.len() <= 2); + refused + }); + assert!(refused.is_some_and(|index| index <= 3), "backwards={backwards} count={count}"); + } + } + } + } + + #[test] + fn a_declared_depth_fits_the_codec_level() { + use hang::catalog::{H264, VideoCodec}; + + let config = |codec: VideoCodec, size: Option<(u32, u32)>, reorder: u8| { + let mut config = VideoConfig::new(codec); + config.coded_width = size.map(|(width, _)| width); + config.coded_height = size.map(|(_, height)| height); + config.reorder = Some(reorder); + config + }; + let h264 = |level| H264 { + profile: 100, + constraints: 0, + level, + inline: false, + }; + let h265: VideoCodec = "hvc1.1.6.L123.B0".parse().unwrap(); + let vp9: VideoCodec = "vp09.00.10.08".parse().unwrap(); + let cases = [ + // Level 4.0's 32768 macroblocks hold four 1920x1088 frames. + (config(h264(40).into(), Some((1920, 1080)), 4), true), + (config(h264(40).into(), Some((1920, 1080)), 5), false), + // Without a size, the deepest buffer any picture of the level gets. + (config(h264(40).into(), None, 16), true), + (config(h264(40).into(), None, 17), false), + // Level 4.1's MaxLumaPs holds six 1920x1080 pictures, one of them the current. + (config(h265.clone(), Some((1920, 1080)), 5), true), + (config(h265, Some((1920, 1080)), 6), false), + (config(vp9.clone(), None, 0), true), + (config(vp9, None, 1), false), + ]; + for (config, valid) in cases { + assert_eq!(declared(&config).is_ok(), valid, "{config:?}"); + } + assert_eq!(declared(&VideoConfig::new(VideoCodec::VP8)).unwrap(), None); + } } diff --git a/rs/moq-mux/src/container/flv/export.rs b/rs/moq-mux/src/container/flv/export.rs index 0f502960ff2..f31b03c4168 100644 --- a/rs/moq-mux/src/container/flv/export.rs +++ b/rs/moq-mux/src/container/flv/export.rs @@ -34,6 +34,7 @@ use super::{ VIDEO_EX_HEADER, VIDEO_PACKET_CODED_FRAMES, VIDEO_PACKET_MULTITRACK, VIDEO_PACKET_SEQUENCE_START, }; use crate::catalog::Stream; +use crate::container::decode::DeclaredClock; use crate::container::{ExportSource, Frame}; /// Before the header, unbind every track the catalog no longer lists (removed, or disabled) or @@ -154,6 +155,9 @@ pub struct Export { /// True once the file header and sequence headers have been emitted. header_emitted: bool, + /// Milliseconds added to every tag's timestamps, so the decode times a declared reorder + /// depth puts before a group's earliest presentation still fit FLV's unsigned clock. + shift: u32, } /// A frame waiting to go out as a tag. @@ -161,6 +165,8 @@ struct Tag { frame: Frame, /// The tag timestamp, the authored DTS for a codec with composition time. timestamp: u32, + /// The presentation time on the tag clock. + presentation: u32, /// The codec config record the frame was read under, which a sequence-header tag must /// have announced before it. record: Option, @@ -184,6 +190,11 @@ struct FlvTrack { /// tag, used when the catalog (and thus [`ExportSource::description`]) carries /// none. Only AV1 needs this today (its av1C is optional in the catalog). fallback_description: Option, + /// The video rendition's catalog entry as last listed. + config: Option, + /// The current group's clock, when its rendition declared `reorder` at the group's start, + /// holding each frame with the config record it was read under. + declared: Option, Option)>>, /// How far to run the authored decode clock behind the PTS, in FLV milliseconds. dts_reserve: u32, /// Last authored video DTS, in FLV milliseconds. @@ -200,8 +211,8 @@ impl FlvTrack { self.empty = moq_clock::now(); } - fn tag_timestamp(&self, frame: &Frame) -> anyhow::Result { - let pts = frame_timestamp_ms(frame)?; + /// The tag timestamp of a frame presented at `pts` on the tag clock, on the inferred clock. + fn tag_timestamp(&self, pts: u32) -> anyhow::Result { if self.flavor.has_composition_time() { author_dts(pts, self.dts_reserve, self.last_dts) } else { @@ -209,6 +220,107 @@ impl FlvTrack { } } + /// Start a group: queue the last one's frames, settled by `arrived`, the keyframe's arrival, + /// then clock this one at the depth its rendition declares now, or on the inferred clock. + fn start_group( + &mut self, + key: (u8, usize), + arrived: moq_clock::Instant, + shift: u32, + jitter: &mut crate::jitter::Buffer<(u8, usize), Tag>, + ) -> anyhow::Result<()> { + self.release(key, Some(arrived), shift, true, jitter)?; + let depth = match &self.config { + Some(config) if self.flavor.has_composition_time() => crate::container::decode::declared(config)?, + _ => None, + }; + self.declared = depth.map(DeclaredClock::new); + Ok(()) + } + + /// Queue each frame the declared clock releases, every one when `flush`. A frame held for a + /// later one is judged as arriving with it, at `settled`. + fn release( + &mut self, + key: (u8, usize), + settled: Option, + shift: u32, + flush: bool, + jitter: &mut crate::jitter::Buffer<(u8, usize), Tag>, + ) -> anyhow::Result<()> { + let Some(mut clock) = self.declared.take() else { + return Ok(()); + }; + let released = (|| { + while let Some(((mut arrival, record), dts)) = clock.pop(flush)? { + if let Some(settled) = settled { + arrival.arrived = arrival.arrived.max(settled); + } + let timestamp = u32::try_from(dts + i128::from(shift)) + .context("FLV decode time before the stream's origin or past 32 bits")?; + self.queue(key, arrival, record, timestamp, shift, jitter)?; + } + anyhow::Ok(()) + })(); + self.declared = Some(clock); + released + } + + /// Queue `arrival` as a tag decoding at `timestamp`, under the config `record` it was read with. + fn queue( + &mut self, + key: (u8, usize), + arrival: crate::jitter::Arrival, + record: Option, + timestamp: u32, + shift: u32, + jitter: &mut crate::jitter::Buffer<(u8, usize), Tag>, + ) -> anyhow::Result<()> { + if self.flavor.has_composition_time() { + self.last_dts = Some(timestamp); + } + let presentation = frame_timestamp_ms(&arrival.item, shift)?; + let crate::jitter::Arrival { + arrived, + read, + restart, + skip, + cut, + sync, + leading, + item, + .. + } = arrival; + let pushed = jitter.push( + key, + crate::jitter::Arrival { + arrived, + read, + decode: moq_net::Timestamp::from_millis(u64::from(timestamp))?.into(), + restart, + skip, + cut, + sync, + leading, + item: Tag { + frame: item, + timestamp, + presentation, + record, + }, + }, + )?; + if pushed == crate::jitter::Push::Late { + tracing::warn!( + kind = key.0, + index = key.1, + dropped = jitter.dropped(), + "frame missed its deadline; dropped" + ); + } + Ok(()) + } + /// The out-of-band config record for this track's sequence-header tag, or /// `None` for codecs that carry their config in band. fn config_record(&self) -> anyhow::Result> { @@ -241,6 +353,7 @@ impl Export { video: Vec::new(), audio: Vec::new(), header_emitted: false, + shift: 0, } } @@ -409,49 +522,34 @@ impl Export { track.pending = Some(arrival); break; } - let timestamp = track.tag_timestamp(&arrival.item)?; - if track.flavor.has_composition_time() { - track.last_dts = Some(timestamp); - } + let key = (kind, index); let record = track.config_record()?; - let crate::jitter::Arrival { - arrived, - read, - restart, - skip, - cut, - sync, - leading, - item, - .. - } = arrival; - if self.jitter.push( - (kind, index), - crate::jitter::Arrival { - arrived, - read, - decode: moq_net::Timestamp::from_millis(u64::from(timestamp))?.into(), - restart, - skip, - cut, - sync, - leading, - item: Tag { - frame: item, - timestamp, - record, - }, - }, - )? == crate::jitter::Push::Late - { - tracing::warn!( - kind, - index, - dropped = self.jitter.dropped(), - "frame missed its deadline; dropped" - ); + if kind == 0 && arrival.item.keyframe { + track.start_group(key, arrival.arrived, self.shift, &mut self.jitter)?; + } + match &mut track.declared { + Some(clock) => { + // Clocked in FLV's milliseconds, so the leading frames step a tag tick apart. + let pts = arrival.item.timestamp.as_millis() as u64; + let arrived = arrival.arrived; + clock.push((arrival, record), pts)?; + track.release(key, Some(arrived), self.shift, false, &mut self.jitter)?; + } + None => { + let timestamp = track.tag_timestamp(frame_timestamp_ms(&arrival.item, self.shift)?)?; + track.queue(key, arrival, record, timestamp, self.shift, &mut self.jitter)?; + } } } + let track = if kind == 0 { + &mut self.video[index] + } else { + &mut self.audio[index] + }; + // Nothing settles the frames held when a track ends, so they keep their own arrival. + if track.finished && self.header_emitted { + track.release((kind, index), None, self.shift, true, &mut self.jitter)?; + } } } Ok(()) @@ -468,6 +566,11 @@ impl Export { // single-track stream rebinds to a better-ranked rendition. A bound rendition the // catalog moves to another track follows it, under a new sequence-header tag when // its config record changes. + for track in &mut self.video { + if let Some(config) = catalog.video.renditions.get(&track.name) { + track.config = Some(config.clone()); + } + } if !self.header_emitted { self.bind_video(&catalog)?; self.bind_audio(&catalog)?; @@ -569,6 +672,8 @@ impl Export { finished: false, flavor, fallback_description, + config: Some(config.clone()), + declared: None, dts_reserve: dts_reserve(config), last_dts: None, announced: None, @@ -616,6 +721,8 @@ impl Export { finished: false, flavor, fallback_description: None, + config: None, + declared: None, dts_reserve: 0, last_dts: None, announced: None, @@ -659,6 +766,15 @@ impl Export { for track in self.video.iter_mut().chain(&mut self.audio) { track.announced = track.config_record()?; } + // A group decodes its first `reorder` frames a millisecond apart before its earliest + // presentation, so the deepest declaration bounds how far before zero a DTS can fall. + self.shift = self + .video + .iter() + .filter(|track| track.flavor.has_composition_time()) + .filter_map(|track| track.config.as_ref()?.reorder) + .max() + .map_or(0, u32::from); Ok(out.freeze()) } @@ -669,6 +785,7 @@ impl Export { let Tag { frame, timestamp: timestamp_ms, + presentation: pts_ms, record, } = tag; let mut out = BytesMut::with_capacity(TAG_HEADER_LEN + frame.payload.len() + 8); @@ -690,7 +807,6 @@ impl Export { let track = &self.video[index]; let flavor = track.flavor; let track_id = track.track_id; - let pts_ms = frame_timestamp_ms(&frame)?; let frame_type = if frame.keyframe { FRAME_TYPE_KEY } else { @@ -873,11 +989,11 @@ fn audio_flavor(config: &hang::catalog::AudioConfig) -> anyhow::Result { } } -fn frame_timestamp_ms(frame: &Frame) -> anyhow::Result { - frame - .timestamp - .as_millis() - .try_into() +/// The frame's presentation time on the tag clock, `shift` milliseconds past its own. +fn frame_timestamp_ms(frame: &Frame, shift: u32) -> anyhow::Result { + u32::try_from(frame.timestamp.as_millis()) + .ok() + .and_then(|pts| pts.checked_add(shift)) .context("FLV timestamp exceeds 32 bits") } diff --git a/rs/moq-mux/src/container/flv/export_test.rs b/rs/moq-mux/src/container/flv/export_test.rs index 00da99ecda0..168eb8befc8 100644 --- a/rs/moq-mux/src/container/flv/export_test.rs +++ b/rs/moq-mux/src/container/flv/export_test.rs @@ -1342,6 +1342,131 @@ async fn export_authors_dts_and_composition_time_for_reordered_avc() { ); } +/// The `(DTS, PTS)` of every AVC frame tag, and the audio tag timestamps, an export renders of +/// a broadcast that declares `reorder` and carries a group of `stamps` (ms, decode order) for +/// each entry, plus one audio frame at 20 ms. +async fn export_declared(reorder: u8, groups: &[&[u64]]) -> anyhow::Result<(Vec<(u32, i64)>, Vec)> { + use hang::catalog::{AAC, AudioConfig, Container, H264, VideoConfig}; + use moq_net::Timestamp; + + let mut producer = moq_net::broadcast::Info::new().produce(); + let consumer = producer.consume(); + let mut catalog = crate::catalog::Producer::new(&mut producer, crate::catalog::Config::default()).unwrap(); + let video_track = producer.create_track(producer.unique_name(".avc1"), timed()).unwrap(); + let audio_track = producer.create_track(producer.unique_name(".aac"), timed()).unwrap(); + let mut video_config = VideoConfig::new(H264 { + profile: 0x42, + constraints: 0xc0, + level: 0x1f, + inline: false, + }); + video_config.container = Container::Legacy; + video_config.description = Some(Bytes::from(avcc())); + video_config.reorder = Some(reorder); + let mut audio_config = AudioConfig::new(AAC { profile: 2 }, 44100, 2); + audio_config.container = Container::Legacy; + audio_config.description = Some(Bytes::from_static(&ASC)); + { + let mut catalog = catalog.modify().unwrap(); + catalog + .video + .renditions + .insert(video_track.name().to_string(), video_config); + catalog + .audio + .renditions + .insert(audio_track.name().to_string(), audio_config); + } + let frame = |ms: u64, payload: &'static [u8], keyframe| crate::container::Frame { + timestamp: Timestamp::from_millis(ms).unwrap(), + duration: None, + payload: Bytes::from_static(payload), + keyframe, + }; + let mut video = crate::container::Producer::new( + video_track, + crate::catalog::hang::Container::Legacy(crate::container::Kind::Data), + ); + for stamps in groups { + for (index, &ms) in stamps.iter().enumerate() { + let payload: &'static [u8] = if index == 0 { + &[0, 0, 0, 1, 0x65] + } else { + &[0, 0, 0, 1, 0x41] + }; + video.write(frame(ms, payload, index == 0)).unwrap(); + } + } + video.finish().unwrap(); + let mut audio = crate::container::Producer::new( + audio_track, + crate::catalog::hang::Container::Legacy(crate::container::Kind::Data), + ); + audio.write(frame(20, &[0xde, 0xad], true)).unwrap(); + audio.finish().unwrap(); + catalog.finish().unwrap(); + + let mut exporter = open_export(crate::source::announced(&consumer).await) + .await + .with_delay(RECORDING_MAX_DELAY) + .with_replay(); + let mut exported = Vec::new(); + if let Some(Err(err)) = drain_until_idle(&mut exporter, &mut exported).await { + return Err(err); + } + let tags = parse_tags(&exported); + let video = tags + .iter() + .filter(|tag| tag.tag_type == super::TAG_VIDEO && tag.body[1] == super::AVC_NALU) + .map(|tag| { + let pts = i64::from(tag.timestamp) + i64::from(super::read_i24(&tag.body[2..5])); + (tag.timestamp, pts) + }) + .collect(); + let audio = tags + .iter() + .filter(|tag| tag.tag_type == super::TAG_AUDIO && tag.body[1] == super::AAC_RAW) + .map(|tag| tag.timestamp) + .collect(); + Ok((video, audio)) +} + +/// A rendition that declares its reorder depth decodes each group from that group's frames: its +/// first frames a millisecond apart below its earliest presentation, and the rest at its +/// presentation slots. The whole stream shifts by the depth, so those fit FLV's unsigned clock. +#[tokio::test(start_paused = true)] +async fn export_declared_reorder_decodes_each_group_from_its_frames() { + let groups: [&[u64]; 2] = [&[0, 160, 80, 40, 120], &[400, 560, 480, 440, 520]]; + let (video, audio) = export_declared(2, &groups).await.unwrap(); + assert_eq!( + video, + [ + (0, 2), + (1, 162), + (2, 82), + (42, 42), + (82, 122), + (400, 402), + (401, 562), + (402, 482), + (442, 442), + (482, 522) + ] + ); + assert_eq!(audio, [22]); + + // A first group cut short leaves the next decoding where it would have. + let cut: [&[u64]; 2] = [&[0, 160, 80], &[400, 560, 480, 440, 520]]; + let (cut, _) = export_declared(2, &cut).await.unwrap(); + assert_eq!(cut[3..], video[5..]); + + // A depth of zero decodes at presentation, and refuses a frame that reorders. + let (video, _) = export_declared(0, &[&[0, 40, 80]]).await.unwrap(); + assert_eq!(video, [(0, 0), (40, 40), (80, 80)]); + let err = export_declared(0, &[&[0, 80, 40]]).await.unwrap_err(); + assert!(err.to_string().contains("declared depth"), "{err}"); +} + /// Poll `exporter` into `out` until it ends or goes idle; `None` if it went idle. async fn drain_until_idle( exporter: &mut Export, diff --git a/rs/moq-mux/src/container/mkv/export.rs b/rs/moq-mux/src/container/mkv/export.rs index 041dcd459eb..271929dcdeb 100644 --- a/rs/moq-mux/src/container/mkv/export.rs +++ b/rs/moq-mux/src/container/mkv/export.rs @@ -72,7 +72,10 @@ pub struct Export { } struct MkvTrack { + /// The inferred decode clock of a rendition that declares no `reorder`. clock: crate::container::decode::DecodeClock>, + /// The current group's clock, when its rendition declared `reorder` at the group's start. + declared: Option>>, config: Option, restart: u64, reorder: Option<(u64, u64)>, @@ -97,6 +100,7 @@ impl MkvTrack { fn new(source: ExportSource, kind: TrackKind, config: Option, track_number: u64) -> Self { Self { clock: Default::default(), + declared: None, config, restart: 0, reorder: None, @@ -122,6 +126,7 @@ impl MkvTrack { ) -> Result<()> { self.release(name, jitter, true)?; self.clock = Default::default(); + self.declared = None; self.reorder = None; self.source = source.continuing(&self.source); self.finished = false; @@ -153,57 +158,73 @@ impl MkvTrack { } } - fn configure_clock(&mut self) -> Result<()> { - if self.reorder.is_some() { - return Ok(()); - } - let Some(config) = &self.config else { return Ok(()) }; - let declared = self.source.description().and_then(|description| match config.codec { - VideoCodec::H264(_) => { - crate::codec::h264::sps_reorder(crate::codec::h264::Avcc::parse(description).ok()?.sps.first()?) - } - VideoCodec::H265(_) => { - crate::codec::h265::sps_reorder(crate::codec::h265::Hvcc::parse(description).ok()?.sps.first()?) - } - _ => None, - }); - // A progressive AVC SPS already declares the fixed frame-count contract. Legacy - // catalogs lacking that declaration keep their existing timing interpretation. - if matches!(config.codec, VideoCodec::H264(_)) - && let Some(declared) = declared.filter(|declared| !declared.fields) - { - self.clock = crate::container::decode::DecodeClock::new(declared.pictures())?; + /// Start a group: release the last one's frames, then clock this one at the depth its + /// rendition declares now, or on the inferred clock when it declares none. + fn start_group(&mut self, name: &str, jitter: &mut crate::jitter::Buffer) -> Result<()> { + let depth = match &self.config { + Some(config) => crate::container::decode::declared(config)?, + None => None, + }; + // The inferred clock spans groups, so it is only flushed when a declaration replaces it. + if self.declared.is_some() || depth.is_some() { + self.release(name, jitter, true)?; } - let (floor, lookahead) = declared.map_or((0, 0), |reorder| reorder.timing(config.framerate)); - let limit = crate::container::decode::MAX_REORDER; - let lookahead = lookahead - .max( - config - .jitter - .map_or(0, |jitter| (jitter.as_nanos() * 90_000 / 1_000_000_000) as u64), - ) - .min(limit); - self.reorder = Some((floor.min(limit), lookahead)); + self.declared = depth.map(crate::container::decode::DeclaredClock::new); Ok(()) } + /// Hold a video frame on its clock, then release what it settles. + fn push( + &mut self, + name: &str, + arrival: crate::jitter::Arrival, + jitter: &mut crate::jitter::Buffer, + ) -> Result<()> { + let pts = (arrival.item.timestamp.as_nanos() * 90_000 / 1_000_000_000) as u64; + match &mut self.declared { + Some(clock) => clock.push(arrival, pts)?, + None => self.clock.push(arrival, pts), + } + self.release(name, jitter, false) + } + fn release(&mut self, name: &str, jitter: &mut crate::jitter::Buffer, flush: bool) -> Result<()> { - let Some((floor, lookahead)) = self.reorder else { - return Ok(()); - }; - loop { - let next = match self.clock.is_fixed() { - true => self.clock.pop(flush)?, - false => self - .clock - .settled(lookahead, floor, flush) - .map(|(frame, dts)| (frame, i128::from(dts))), - }; - let Some((mut arrival, dts)) = next else { break }; + if let Some(clock) = &mut self.declared { + while let Some((mut arrival, dts)) = clock.pop(flush)? { + if !self.finished && !flush { + arrival.arrived = arrival.arrived.max(self.latest); + } + arrival.decode = crate::jitter::Decode::from_ticks(dts); + jitter.push(name.to_owned(), arrival)?; + } + } + let Some(config) = &self.config else { return Ok(()) }; + let (floor, lookahead) = *self.reorder.get_or_insert_with(|| { + let declared = self.source.description().and_then(|description| match config.codec { + VideoCodec::H264(_) => { + crate::codec::h264::sps_reorder(crate::codec::h264::Avcc::parse(description).ok()?.sps.first()?) + } + VideoCodec::H265(_) => { + crate::codec::h265::sps_reorder(crate::codec::h265::Hvcc::parse(description).ok()?.sps.first()?) + } + _ => None, + }); + let (floor, lookahead) = declared.map_or((0, 0), |reorder| reorder.timing(config.framerate)); + let limit = crate::container::decode::MAX_REORDER; + let lookahead = lookahead + .max( + config + .jitter + .map_or(0, |jitter| (jitter.as_nanos() * 90_000 / 1_000_000_000) as u64), + ) + .min(limit); + (floor.min(limit), lookahead) + }); + while let Some((mut arrival, dts)) = self.clock.pop(lookahead, floor, flush) { if !self.finished && !flush { arrival.arrived = arrival.arrived.max(self.latest); } - arrival.decode = crate::jitter::Decode::from_nanos((dts * 1_000_000_000).div_euclid(90_000)); + arrival.decode = moq_net::Timestamp::from_scale(dts, 90_000)?.into(); jitter.push(name.to_owned(), arrival)?; } Ok(()) @@ -428,11 +449,16 @@ impl Export { #[cfg(test)] pub(super) fn buffered(&self) -> usize { self.jitter.len() - + self - .tracks - .values() - .map(|track| track.clock.len() + usize::from(track.pending.is_some())) - .sum::() + } + + /// How many video frames wait on a declared clock for their DTS. + #[cfg(test)] + pub(super) fn held(&self) -> usize { + self.tracks + .values() + .filter_map(|track| track.declared.as_ref()) + .map(|clock| clock.len()) + .sum() } #[cfg(test)] @@ -459,18 +485,13 @@ impl Export { } let (frame, arrived) = match track.source.poll_event(waiter)? { Poll::Ready(Some(Event::Frame { frame, arrived })) => (frame, arrived.max(track.subscribed)), - Poll::Ready(Some(Event::GroupEnd)) => { - if track.clock.is_fixed() { + // A group that ended or was cut short holds frames no later one can precede. + Poll::Ready(Some(Event::GroupEnd | Event::Cut)) => { + if track.declared.is_some() { track.release(name, &mut self.jitter, true)?; } continue; } - Poll::Ready(Some(Event::Cut)) => { - if track.clock.is_fixed() { - track.clock.discard(); - } - continue; - } Poll::Ready(Some(Event::FrameEnd { .. })) => continue, Poll::Ready(None) => { track.finished = true; @@ -515,26 +536,19 @@ impl Export { track.pending = Some(arrival); break; } - if track.kind == TrackKind::Video { - track.configure_clock()?; + let video = track.kind == TrackKind::Video; + if video && arrival.restart != track.restart { + track.release(name, &mut self.jitter, true)?; + track.clock = Default::default(); + track.declared = None; + track.restart = arrival.restart; } - if track.kind == TrackKind::Video && (!self.delay.is_zero() || track.clock.is_fixed()) { - if arrival.restart != track.restart { - track.release(name, &mut self.jitter, true)?; - if track.clock.is_fixed() { - track.clock.discard(); - } else { - track.clock = Default::default(); - } - track.restart = arrival.restart; - } - if arrival.item.keyframe && track.clock.is_fixed() { - track.release(name, &mut self.jitter, true)?; - track.clock.discard(); - } - let pts = (arrival.item.timestamp.as_nanos() * 90_000 / 1_000_000_000) as u64; - track.clock.push(arrival, pts)?; - track.release(name, &mut self.jitter, false)?; + if video && arrival.item.keyframe { + track.start_group(name, &mut self.jitter)?; + } + // A declared depth is checked even when nothing is scheduled by decode time. + if video && (track.declared.is_some() || !self.delay.is_zero()) { + track.push(name, arrival, &mut self.jitter)?; } else { self.jitter.push(name.clone(), arrival)?; } @@ -587,11 +601,10 @@ impl Export { { track.rebind(name, source, &mut self.jitter)?; } - if !track.clock.is_fixed() - && track - .config - .as_ref() - .is_some_and(|old| old.jitter != config.jitter || old.framerate != config.framerate) + if track + .config + .as_ref() + .is_some_and(|old| old.jitter != config.jitter || old.framerate != config.framerate) { track.reorder = None; } diff --git a/rs/moq-mux/src/container/mkv/export_test.rs b/rs/moq-mux/src/container/mkv/export_test.rs index b78e90fb181..60ec8723775 100644 --- a/rs/moq-mux/src/container/mkv/export_test.rs +++ b/rs/moq-mux/src/container/mkv/export_test.rs @@ -22,7 +22,7 @@ use crate::container::test_util::{IDR, Live, PPS, SPS, raw_frame, video_frame}; /// completeness has to be asked for, exactly as a real recorder does. const RECORDING_MAX_DELAY: std::time::Duration = std::time::Duration::from_secs(30); -/// A progressive AVC rendition whose SPS declares a reorder depth of two. +/// An AVC rendition that declares a reorder depth of two. fn declared_avc() -> Live { Live::new(".avc1", |catalog, name| { let mut config = hang::catalog::VideoConfig::new(hang::catalog::H264 { @@ -33,6 +33,7 @@ fn declared_avc() -> Live { }); config.coded_width = Some(64); config.coded_height = Some(64); + config.reorder = Some(2); config.container = hang::catalog::Container::Legacy; config.description = Some( crate::codec::h264::build_avcc( @@ -46,7 +47,7 @@ fn declared_avc() -> Live { } #[tokio::test(start_paused = true)] -async fn a_declared_sps_bounds_malformed_video_by_count_and_payload() { +async fn a_declared_depth_bounds_malformed_video_by_count_and_payload() { for backwards in [false, true] { for count in [16, 128, 1024] { for size in [256, 16_384] { @@ -76,10 +77,10 @@ async fn a_declared_sps_bounds_malformed_video_by_count_and_payload() { Ok(None) => break, Ok(Some(_)) => {} } - assert!(export.buffered() <= 3, "a depth-two clock and one released frame"); + assert!(export.held() <= 2, "a depth-two clock"); } assert!(refused, "backwards={backwards} count={count} size={size}"); - assert!(export.buffered() <= 3); + assert!(export.held() <= 2 && export.buffered() <= 4); } } } diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index 30514952213..9d340fba86e 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -48,7 +48,7 @@ use crate::codec::annexb::InBandCodec; use crate::codec::video::{Hrd, Reorder}; use crate::codec::{aac, annexb, opus}; use crate::container::consumer::Event; -use crate::container::decode::{DecodeClock, MAX_REORDER}; +use crate::container::decode::{DeclaredClock, DecodeClock, MAX_REORDER}; use crate::container::{ExportSource, Frame}; use crate::jitter::{self, Arrival, Push}; @@ -317,8 +317,11 @@ struct Track { /// PMT ES-level descriptors to re-announce, captured verbatim on import (language, /// registration, ...). Empty for non-TS sources; AC-3/E-AC-3 then synthesize one. descriptors: Vec, - /// Authors each video frame's DTS, holding the frames until it can. + /// Authors each video frame's DTS, holding the frames until it can, for a rendition that + /// declares no `reorder`. clock: DecodeClock<(Pending, Option)>, + /// The current group's clock, when its rendition declared `reorder` at the group's start. + declared: Option)>>, /// Loss counter at the last read, excluding declared ends. recovery_gap: Option, /// Leading pictures below the recovery keyframe have unavailable references, and whether @@ -378,6 +381,7 @@ impl Track { kind, descriptors: Vec::new(), clock: DecodeClock::default(), + declared: None, recovery_gap: None, leading_floor: None, held: None, @@ -408,6 +412,7 @@ impl Track { self.release(name, jitter, true)?; self.restart = pending.restart; self.clock = DecodeClock::default(); + self.declared = None; if let Some(sync) = &mut self.sync { sync.lane.reset(); sync.released = None; @@ -438,14 +443,26 @@ impl Track { // sync they go, as on a leg that never read them. if self.sync.is_some() && self.held.is_some_and(|held| held != pending.cut) { self.clock.discard(); + self.declared = None; } self.held = Some(pending.cut); if pending.frame.keyframe { self.timing .describe(stream_type, self.source.description(), name, self.sync.is_some()); + // A group decodes on its own clock at the depth its rendition declares now. The + // inferred clock spans groups, so it is only flushed when a declaration replaces it. + let depth = self.timing.reorder; + if self.declared.is_some() || depth.is_some() { + self.release(name, jitter, true)?; + } + self.declared = depth.map(DeclaredClock::new); } let pts = to_ticks(pending.frame.timestamp); - self.clock.push((pending, self.source.description().cloned()), pts)?; + let item = (pending, self.source.description().cloned()); + match &mut self.declared { + Some(clock) => clock.push(item, pts)?, + None => self.clock.push(item, pts), + } self.release(name, jitter, false) } @@ -510,7 +527,10 @@ impl Track { self.settle = false; self.release(name, input.jitter, true)?; } - Some(_) => self.clock.discard(), + Some(_) => { + self.clock.discard(); + self.declared = None; + } } input.jitter.cut(&name.to_owned(), self.base.2 + self.source.cuts()); } @@ -838,27 +858,44 @@ impl Track { /// Under sync, a stream whose SPS declares its depth is settled by counting frames at the /// SPS's delay, so a frame's DTS never depends on what the leg read before it. fn release(&mut self, name: &str, jitter: &mut jitter::Buffer, flush: bool) -> anyhow::Result<()> { - let (delay, lookahead) = self.timing.reorder(); + if let Some(mut clock) = self.declared.take() { + let released = (|| { + while let Some(((mut pending, description), dts)) = clock.pop(flush)? { + if !self.finished { + pending.arrived = pending.arrived.max(self.latest); + } + self.push(name, pending, Some(dts), description, jitter)?; + } + anyhow::Ok(()) + })(); + self.declared = Some(clock); + released?; + } + let (delay, lookahead) = self.timing.inferred(); let lookahead = lookahead.max(self.timing.jitter); let counted = self.sync.as_ref().and(self.timing.depth()); while let Some(((mut pending, description), dts)) = match counted { Some(depth) => self.clock.pop_counted(depth, delay, flush), - None => self.clock.settled(lookahead, delay, flush), + None => self.clock.pop(lookahead, delay, flush), } { if !self.finished { pending.arrived = pending.arrived.max(self.latest); } - self.push(name, pending, Some(dts), description, jitter)?; + self.push(name, pending, Some(dts.into()), description, jitter)?; } Ok(()) } - /// Queue a frame decoding at `dts` (90 kHz ticks), else at its PTS. + /// Queue a frame decoding at `dts` (signed 90 kHz ticks), else at its PTS. + /// + /// A DTS before the timeline's zero, a declared group's leading frames at its start, wraps + /// into the 33-bit wire field like any other, and is scheduled from zero: the few ticks it + /// precedes zero by are below anything the T-STD resolves. fn push( &mut self, name: &str, pending: Pending, - dts: Option, + dts: Option, description: Option, jitter: &mut jitter::Buffer, ) -> anyhow::Result<()> { @@ -871,10 +908,12 @@ impl Track { read, group, } = pending; - let dts = dts.filter(|&dts| dts != to_ticks(frame.timestamp)); + let dts = dts.filter(|&dts| dts != i128::from(to_ticks(frame.timestamp))); let decode = dts - .and_then(|ticks| Timestamp::from_scale(ticks, 90_000).ok()) + .and_then(|ticks| Timestamp::from_scale(u64::try_from(ticks.max(0)).ok()?, 90_000).ok()) .unwrap_or(frame.timestamp); + // Two's complement, so the 33-bit mask wraps a DTS before zero as it would a later one. + let dts = dts.map(|ticks| ticks as u64); let (key, pes) = self.release_key(decode, jitter.delay(), &frame)?; // Planned as it is read, so a frame the jitter buffer drops still moves its group on. let planned = match self.sync.is_some() { @@ -946,6 +985,8 @@ struct Timing { declared: Option, /// The stream's level limits, from the catalog codec. level: Option, + /// The reorder depth the catalog declares, which clocks each group on its own. + reorder: Option, /// The HRD its SPS declares. hrd: Option, /// The NAL unit length and buffering period layout of an H.264 rendition with a NAL HRD. @@ -955,10 +996,12 @@ struct Timing { impl Timing { /// Take the rendition's timing from a catalog snapshot, before or after the program /// tables are written. - fn configure(&mut self, config: &VideoConfig, name: &str) { + fn configure(&mut self, config: &VideoConfig, name: &str) -> crate::Result<()> { self.jitter = config.jitter.map_or(0, |t| (t.as_micros() * 90_000 / 1_000_000) as u64); self.framerate = config.framerate.filter(|fps| fps.is_finite() && *fps > 0.0); self.level = Some(video_level(config, name)); + self.reorder = crate::container::decode::declared(config)?; + Ok(()) } /// Read the reorder depth from the codec config a keyframe is carried with. @@ -970,8 +1013,8 @@ impl Timing { self.declared = description.and_then(|d| declared_reorder(stream_type, d)); self.hrd = description.and_then(|d| declared_hrd(stream_type, d)); self.buffering = description.and_then(|d| declared_buffering(stream_type, d)); - tracing::debug!(track = %name, reorder = ?self.reorder(), "video reordering declared"); - if synced && self.declared.is_none() { + tracing::debug!(track = %name, reorder = ?self.inferred(), "video reordering declared"); + if synced && self.declared.is_none() && self.reorder.is_none() { tracing::warn!(track = %name, "no reorder depth declared: if it reorders, --sync legs may give it different DTS"); } } @@ -1026,7 +1069,7 @@ impl Timing { /// `framerate`'s. A frame can be presented up to `2^depth - 1` pictures below one decoded /// before it (a B-pyramid that deep), which bounds the lookahead the catalog `jitter` /// gives when it is not published. - fn reorder(&self) -> (u64, u64) { + fn inferred(&self) -> (u64, u64) { let Some(declared) = self.declared else { return (0, 0); }; @@ -1993,7 +2036,7 @@ where // `framerate` published after the tables still sizes the decode clock. for (name, config) in catalog.video.renditions.iter() { if let Some(track) = self.tracks.get_mut(name).filter(|track| track.listed) { - track.timing.configure(config, name); + track.timing.configure(config, name)?; } } // A track that leaves the catalog is read to its own end. A publisher retires a @@ -2093,7 +2136,7 @@ where }, }; track.kind = kind; - track.timing.configure(config, name); + track.timing.configure(config, name)?; self.refresh(name, track, pid, track_descriptors(&mpegts, name)); } for (name, config) in catalog.audio.renditions.iter() { @@ -2241,6 +2284,7 @@ where ); track.release(name, &mut self.jitter, true)?; track.clock = DecodeClock::default(); + track.declared = None; track.source = source.continuing(&track.source); track.finished = false; track.subscribed = moq_clock::now(); @@ -4719,12 +4763,12 @@ mod tests { let mut clock = DecodeClock::default(); let mut dts = vec![0; pts.len()]; for (i, &p) in pts.iter().enumerate() { - clock.push(i, p).unwrap(); - while let Some((i, d)) = clock.settled(jitter, floor, false) { + clock.push(i, p); + while let Some((i, d)) = clock.pop(jitter, floor, false) { dts[i] = d; } } - while let Some((i, d)) = clock.settled(jitter, floor, true) { + while let Some((i, d)) = clock.pop(jitter, floor, true) { dts[i] = d; } dts diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 46a0fa8bda9..71b1b5d9513 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -1149,6 +1149,115 @@ async fn export_decodes_where_the_source_did() { } } +/// A broadcast of one H.264 rendition declaring `reorder`, with a group of `stamps` (ms, in +/// decode order) for each entry, exported as a recording. +/// The producers stay alive with the export, so it reads the retained tracks. +async fn export_declared( + reorder: u8, + groups: &[&[u64]], +) -> (Export, impl Sized) { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let mut catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let track = broadcast + .create_track("video", hang::container::track_info(hang::catalog::PRIORITY.video)) + .unwrap(); + let mut config = VideoConfig::new(H264 { + profile: 0x64, + constraints: 0, + level: 0x1f, + inline: true, + }); + config.reorder = Some(reorder); + catalog.modify().unwrap().video.renditions.insert("video".into(), config); + let format = HangContainer::Legacy(Kind::Video); + for (sequence, stamps) in groups.iter().enumerate() { + let mut group = track + .create_group(moq_net::group::Info { + sequence: sequence as u64, + }) + .unwrap(); + let frames: Vec<_> = stamps + .iter() + .enumerate() + .map(|(index, &ms)| Frame { + timestamp: Timestamp::from_millis(ms).unwrap(), + duration: None, + keyframe: index == 0, + payload: if index == 0 { + annexb(&[SPS, PPS, &[0x65, 0xab, 0xcd]]) + } else { + annexb(&[&[0x41, 0xab, 0xcd]]) + }, + }) + .collect(); + format.write(&mut group, &frames).unwrap(); + group.finish().unwrap(); + } + track.finish().unwrap(); + let export = open_export(crate::source::announced(&consumer).await) + .await + .unwrap() + .with_delay(RECORDING_MAX_AGE) + .with_replay(); + (export, (broadcast, catalog, track)) +} + +/// A rendition that declares its reorder depth decodes each group from that group's frames: its +/// first frames a tick apart below its earliest presentation, wrapping the 33-bit DTS below the +/// timeline's zero, and the rest at its presentation slots. +#[tokio::test(start_paused = true)] +async fn export_declared_reorder_decodes_each_group_from_its_frames() { + let (mut export, _keep) = export_declared(2, &[&[0, 160, 80, 40, 120], &[400, 560, 480, 440, 520]]).await; + let bytes: Vec = drain_frames(&mut export) + .await + .iter() + .flat_map(|frame| frame.payload.iter().copied()) + .collect(); + let wrap = 1 << 33; + let ms = |ms: u64| ms * 90; + assert_eq!( + video_pes_timing(&bytes), + [ + (ms(0), wrap - 2), + (ms(160), wrap - 1), + (ms(80), 0), + (ms(40), ms(40)), + (ms(120), ms(80)), + (ms(400), ms(400) - 2), + (ms(560), ms(400) - 1), + (ms(480), ms(400)), + (ms(440), ms(440)), + (ms(520), ms(480)), + ] + ); +} + +/// A declared depth of zero decodes every frame at its presentation, and refuses a frame that +/// reorders, rather than inferring a decode delay. +#[tokio::test(start_paused = true)] +async fn export_declared_zero_reorder_decodes_at_presentation() { + let (mut export, _keep) = export_declared(0, &[&[0, 40, 80, 120]]).await; + let bytes: Vec = drain_frames(&mut export) + .await + .iter() + .flat_map(|frame| frame.payload.iter().copied()) + .collect(); + let timing = video_pes_timing(&bytes); + assert_eq!(timing.len(), 4); + assert!(timing.iter().all(|(pts, dts)| pts == dts), "{timing:?}"); + + let (mut export, _keep) = export_declared(0, &[&[0, 80, 40, 120]]).await; + let err = loop { + match tokio::time::timeout(DRAIN, export.next()).await { + Ok(Ok(Some(_))) => {} + Ok(Ok(None)) | Err(_) => panic!("a frame reordered past the declared depth was exported"), + Ok(Err(err)) => break err, + } + }; + assert!(err.to_string().contains("declared depth"), "{err}"); +} + /// #2937: the PCR must be a uniform bounded-interval ramp, not a sample of the /// per-unit decode clock. On a reordered (B-frame) capture the authored DTS is a /// saw (reference frames leap a reorder span, B-frames nudge one tick), so a PCR diff --git a/rs/moq-mux/src/container/ts/export_timing_test.rs b/rs/moq-mux/src/container/ts/export_timing_test.rs index 1a81554710e..a8675ffaf50 100644 --- a/rs/moq-mux/src/container/ts/export_timing_test.rs +++ b/rs/moq-mux/src/container/ts/export_timing_test.rs @@ -206,6 +206,14 @@ impl Live { self.reorder = true; } + /// Declare the B-pyramid's reorder depth in the catalog, so each group decodes on its own. + fn declare(&mut self) { + let mut catalog = self.catalog.modify().unwrap(); + for config in catalog.video.renditions.values_mut() { + config.reorder = Some(2); + } + } + /// Declare an 800 kb/s CBR NAL HRD with 0.4 s of buffer delay, and open each group with a /// buffering period SEI giving its first picture `delay` ticks of `initial_cpb_removal_delay`. fn buffering(&mut self, delay: u32) { @@ -1096,6 +1104,46 @@ async fn synced_legs_render_reordered_video_the_same_way() { } } +/// Each video frame's DTS (else PTS) by its PTS, in 90 kHz ticks, as a leg rendered it. +fn decode_times(leg: &Leg) -> std::collections::BTreeMap { + let bytes: Vec = leg.out.iter().flat_map(|(_, frame)| frame.payload.iter().copied()).collect(); + let mut reader = TsPacketReader::new(Cursor::new(bytes)); + let mut out = std::collections::BTreeMap::new(); + while let Some(packet) = reader.read_ts_packet().unwrap() { + if let Some(TsPayload::PesStart(pes)) = &packet.payload + && packet.header.pid.as_u16() == VIDEO_PID + && let Some(pts) = pes.header.pts + { + out.insert(pts.as_u64(), pes.header.dts.unwrap_or(pts).as_u64()); + } + } + out +} + +/// Legs that joined at different groups, one after reading a group cut short below a frame it +/// never received, author the same DTS for every frame both render once the rendition declares +/// its reorder depth: each group decodes from its own frames alone. +#[tokio::test(start_paused = true)] +async fn declared_legs_author_the_same_dts() { + let stall = |tick: u64| (2 * GOP + 8..3 * GOP).contains(&tick); + for join in [JOIN, 4 * GOP + 2] { + let mut live = Live::new(1.0, 0).await; + live.reorder(); + live.declare(); + let mut a = live.join(Duration::ZERO).await; + live.run(join, &mut [&mut a], stall).await; + let mut b = live.join(Duration::from_millis(100)).await; + live.run(12 * GOP, &mut [&mut a, &mut b], stall).await; + live.finish(&mut [&mut a, &mut b]).await; + let (a, b) = (decode_times(&a), decode_times(&b)); + let shared: Vec<_> = b.iter().filter(|(pts, _)| a.contains_key(pts)).collect(); + assert!(shared.len() > 6 * GOP as usize, "join {join}: {} shared frames", shared.len()); + for (pts, dts) in shared { + assert_eq!(a[pts], *dts, "join {join}: the frame presented at {pts}"); + } + } +} + /// Each leg loses groups the other receives. Each renders the other's packets wherever it /// holds them, so one leg's stuffing is the other's content, and the two merged are a clean /// leg's output. diff --git a/rs/moq-mux/src/jitter.rs b/rs/moq-mux/src/jitter.rs index 1fe8ded0d45..55ea0a3972d 100644 --- a/rs/moq-mux/src/jitter.rs +++ b/rs/moq-mux/src/jitter.rs @@ -19,6 +19,11 @@ impl Decode { Self(nanos) } + /// The instant `ticks` of a 90 kHz clock, rounded down. + pub(crate) fn from_ticks(ticks: i128) -> Self { + Self((ticks * 1_000_000_000).div_euclid(90_000)) + } + fn as_nanos(self) -> i128 { self.0 } From 8f14a6d4c1415ce5d62088a5a16d811f6c5da5c7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 11:05:16 -0700 Subject: [PATCH 04/11] feat(mux): declare reorder from the SPS and @moq/publish's encoder The H.264 and H.265 importers copy the SPS's reorder depth into the catalog, hold the first declaration, and refuse a deeper one. @moq/publish declares 0 and refuses reordered encoder output. The finished decode-order quest is replaced by a follow-up bounding the inferred clock. Co-Authored-By: Claude Opus 5.5 --- doc/bin/cli.md | 5 +- js/publish/src/video/encoder.ts | 12 ++ quest/m1/release-audit/README.md | 2 +- .../m1/release-audit/inferred-clock-bound.md | 25 +++ quest/m1/ts-reliability/README.md | 3 +- quest/m1/ts-reliability/decode-order.md | 155 ------------------ rs/moq-mux/MIGRATE.md | 16 +- rs/moq-mux/src/codec/h264/import.rs | 35 +++- rs/moq-mux/src/codec/h264/mod.rs | 11 +- rs/moq-mux/src/codec/h265/import.rs | 8 + rs/moq-mux/src/codec/h265/mod.rs | 11 +- rs/moq-mux/src/codec/video.rs | 20 +++ rs/moq-mux/src/container/flv/import.rs | 21 +-- rs/moq-mux/src/container/fmp4/import.rs | 1 + rs/moq-mux/src/container/ts/recovery_test.rs | 13 +- 15 files changed, 140 insertions(+), 198 deletions(-) create mode 100644 quest/m1/release-audit/inferred-clock-bound.md delete mode 100644 quest/m1/ts-reliability/decode-order.md diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 68a50ca3e77..bc8adaf3e7f 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -455,8 +455,9 @@ constant rate, the same on every leg since the lanes follow from it source's (a 10 Mb/s channel needs about 12), and takes neither `--stitch` nor `--passthrough`. A group that starts behind the one before it, on a lane too narrow to absorb that one, is logged and counted in `overruns`: the legs can -differ there. So can reordered video whose SPS declares no reorder depth -(`max_num_reorder_frames`), which is logged, and SI while one leg holds an older +differ there. So can reordered video whose catalog declares no +[`reorder`](/concept/hang), which the importers copy from the SPS and which is +logged, and SI while one leg holds an older revision than the other or around an SI table the catalog adds or drops, which each leg takes up when its catalog update arrives. diff --git a/js/publish/src/video/encoder.ts b/js/publish/src/video/encoder.ts index 605ab7c27de..c870d84a856 100644 --- a/js/publish/src/video/encoder.ts +++ b/js/publish/src/video/encoder.ts @@ -375,6 +375,8 @@ export class Encoder { let lastKeyframe: Micro | undefined; let lastEncoded: Micro | undefined; + // The catalog declares no reordering, so each chunk must present after the one before it. + let lastOutput: number | undefined; // The largest group demand asked for that a keyframe already answers. A join over the refresh // ratio asks for one past the largest, which the next frame opens. let answered: number | undefined; @@ -387,6 +389,14 @@ export class Encoder { effect.spawn(async () => { const encoder = new VideoEncoder({ output: (frame: EncodedVideoChunk) => { + if (lastOutput !== undefined && frame.timestamp <= lastOutput) { + this.#abort( + new Error(`encoder reordered a frame at ${frame.timestamp}us behind ${lastOutput}us`), + track, + ); + return; + } + lastOutput = frame.timestamp; const key = frame.type === "key"; if (key) { lastKeyframe = frame.timestamp as Micro; @@ -578,6 +588,8 @@ export class Encoder { codedWidth: Catalog.u53(config.width), codedHeight: Catalog.u53(config.height), optimizeForLatency: true, + // Realtime WebCodecs encoders emit no B-frames; the output callback refuses one that does. + reorder: Catalog.u8(0), container: { kind: "legacy" } as const, ...estimate, }; diff --git a/quest/m1/release-audit/README.md b/quest/m1/release-audit/README.md index 42a06c9fc67..95fbe218064 100644 --- a/quest/m1/release-audit/README.md +++ b/quest/m1/release-audit/README.md @@ -94,7 +94,7 @@ child records whether its behavior is new on main or inherited from release. - [Review HLS admission](/quest/m1/release-audit/hls-admission.md) - resolve the security-sensitive resource-boundary review before the cut - [Bound decoded datagram retention](/quest/m1/release-audit/datagram-memory.md) - shared ownership accounts for retained plain and compressed payloads across tracks -- [Decode order](/quest/m1/ts-reliability/decode-order.md) - the exporters' decode clock holds at most the declared reorder depth, so flat timestamps cannot bypass MKV/TS backpressure or cause quadratic draining +- [Bound the inferred decode clock](/quest/m1/release-audit/inferred-clock-bound.md) - a rendition without a declared `reorder` cannot hold unbounded frames on flat timestamps through MKV/TS export - [Refuse partial lite groups in JS](/quest/m1/release-audit/js-partial-group.md) - a nonzero `frameStart` closes the session as a protocol violation, not a renumbered complete group - [Parse QMux records without copying suffixes](/quest/m1/release-audit/qmux-copy.md) - many small frames do not cause quadratic copying - [Release AUTH wake listeners](/quest/m1/release-audit/auth-listeners.md) - session churn does not retain one terminal promise reaction per change diff --git a/quest/m1/release-audit/inferred-clock-bound.md b/quest/m1/release-audit/inferred-clock-bound.md new file mode 100644 index 00000000000..b0a9274782a --- /dev/null +++ b/quest/m1/release-audit/inferred-clock-bound.md @@ -0,0 +1,25 @@ +# [M] Bound the inferred decode clock + +## Goal + +An MKV or TS export of a video rendition that declares no catalog `reorder` +holds a bounded number of frames in its decode clock, whatever its timestamps +do: flat or backward PTS cannot bypass the export's backpressure or drain +quadratically. + +## Plan + +[moq-dev/moq#6453](https://github.com/moq-dev/moq/pull/6453) bounds a +rendition that declares `reorder`: its `DeclaredClock` holds at most that many +frames and refuses deeper reordering. The importers now declare it from the +SPS, so the remaining path is a catalog that omits it while the stream +reorders: a hostile or older publisher, or field-coded H.264. + +The maintainer kept that absence on today's inferred clock +(`DecodeClock::pop` and `pop_counted` in `rs/moq-mux/src/container/decode.rs`). +It settles a slot once the highest PTS read is the lookahead past it, so PTS +that stop advancing hold every frame read since. The bound must keep the DTS +a conforming source gets today; a held count tied to the inferred depth, or to +the lookahead in frames, are the candidates. Sweep held frames over frame +count and payload size through MKV and TS export, as the declared path's +tests do. diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index ccbb46d0361..895b526b674 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -62,7 +62,7 @@ refused means an error or a counted drop. | Mechanism | Contract | Owner | |:--|:--|:--| -| H.264 (0x1B), H.265 (0x24) | Preserved: video rendition, PID and ES descriptors in `mpegts.tracks`; DTS normalized from decode order. Today one access unit per PES | [access units](/quest/m1/ts-reliability/pes-access-units.md), [decode order](/quest/m1/ts-reliability/decode-order.md) | +| H.264 (0x1B), H.265 (0x24) | Preserved: video rendition, PID and ES descriptors in `mpegts.tracks`; DTS normalized from decode order. Today one access unit per PES | [access units](/quest/m1/ts-reliability/pes-access-units.md) | | AAC ADTS (0x0F), MP2, AC-3, E-AC-3, Opus (0x06 + `Opus`) | Preserved: audio rendition; Opus export labels mono and stereo, and 3 to 8 channels with an OpusHead in the Vorbis layout. Other mapping families (such as 255) refuse | landed | | Private and unknown PES types, LATM (0x11), metadata (0x15) | Preserved: `pesPacket`, PTS and DTS rebased | landed | | PES clock fields (ESCR, pack header, TREF), PES scrambling | Refused, counted in `refused` | landed | @@ -100,7 +100,6 @@ remain with their current quests. ## Required - [State bound](/quest/m1/ts-reliability/state-bound.md) - SI and revision state retire with what the source stops repeating -- [Decode order](/quest/m1/ts-reliability/decode-order.md) - TS, MKV and FLV author DTS from decode order and a declared `reorder` depth - [Shared parser](/quest/m1/ts-reliability/shared-parser.md) - `--program all` scans the multiplex once - [PMT export](/quest/m1/ts-reliability/pmt-export.md) - synced TS legs revise their PMT and grid identically - [Access units](/quest/m1/ts-reliability/pes-access-units.md) - a PES carrying several access units publishes each at its own time diff --git a/quest/m1/ts-reliability/decode-order.md b/quest/m1/ts-reliability/decode-order.md deleted file mode 100644 index cc55b839a39..00000000000 --- a/quest/m1/ts-reliability/decode-order.md +++ /dev/null @@ -1,155 +0,0 @@ -# [XL] Decode time comes from decode order and a declared reorder depth - -## Goal - -Every exporter that authors a decode time (TS, MKV, FLV) derives it only from -the frames of the current group, in the decode order they arrive, and from the -rendition's declared reorder depth. Two exporters that read the same group -author the same DTS for it, whatever they joined at or lost before, and an -exporter never holds more video frames than that depth. A rendition that -declares no reordering adds no decode delay. Media frames stay -presentation-time only: no decode time on the wire. - -## Plan - -### Checkpoint - -The Goal remains incomplete. This branch adds signed jitter admission and a -fixed-depth mode to the existing `DecodeClock`, consumed by MKV export when a -progressive AVC SPS declares the depth. It keeps at most that many payloads -after each drain, shifts by presentation slots, and resets at actual group -boundaries. Tests cover timeline zero, audio interleaving, independent groups, -a cut in an earlier group, distant references, zero depth, and malformed -flat/backward PTS swept over frame count and payload size. The exporter benchmark -sweeps those axes through the same MKV consumer. - -TS and FLV still use their existing clock authors, as do MKV's undeclared, -field-coded AVC and HEVC paths. Signed jitter admission reaches all three -exporters, but only MKV's declared progressive AVC path currently authors signed -slots. No new catalog field, producer declaration, JS behavior or wire format is -implemented. The remaining rollout includes codec-level depth validation, -TS/MKV/FLV parity, every producer/importer, Rust/JS catalogs, matching drafts and -migrations, and the full join/loss and T-STD controls described below. - -### Decision needed: published absence - -The decided `absent = 0` rule conflicts with the published-wire compatibility -rule. Published `hang-v0.21.16` has no `reorder` field, and the existing fMP4 -import control `unwrapped_samples_present_at_their_composition_time` publishes -decode-order PTS 1,041,666; 1,125,000; 1,083,333 microseconds without one. Treating -that absence as zero makes a new exporter refuse supported published media. -The current catalog version cannot distinguish that legacy absence from an -intentional zero declaration. - -Recommend a versioned or explicit-declaration contract that preserves published -catalog behavior, then complete the fixed-depth producer/export rollout. The -alternative is deliberate migration/refusal of legacy reordering catalogs, -which changes the published compatibility requirement and needs approval. -Do not silently choose either contract or remove this Goal while unresolved. - -### Settled design - -Why: a DTS that depends on what the exporter read before has been patched -six times ([moq-dev/moq#1843](https://github.com/moq-dev/moq/pull/1843), -[#2967](https://github.com/moq-dev/moq/pull/2967), -[#4500](https://github.com/moq-dev/moq/pull/4500), -[#4574](https://github.com/moq-dev/moq/pull/4574), -[#4645](https://github.com/moq-dev/moq/pull/4645), -[#5949](https://github.com/moq-dev/moq/pull/5949)). Today there are three -authors: timestamp-settled `DecodeClock::settled` (`rs/moq-mux/src/container/decode.rs`, TS without -`--sync` and MKV) settles a slot by the highest PTS read and a delay that only -grows; `pop_counted` (TS `--sync`) depends on the held frames alone; FLV -export authors `PTS - jitter` bumped past the last DTS, which passes the PTS -on B-frames. fMP4 export already anchors its decode timeline at each group's -first frame and needs no change. - -Decided 2026-10-22 by the maintainer, after the TS root-cause audit: - -- Frames carry PTS only, in decode order. A decoder needs order; an export - needs *a* DTS that is monotonic, never after its PTS, and fits the T-STD - buffer, not the source's own. Source DTS spacing (field coding, pulldown) - is normalized, not preserved. -- Rejected: carrying decode time in the frame (CMAF composition offsets, a - LOC property, Legacy inference). It adds a wire field to every container - to preserve spacing no consumer needs. -- One clock serves TS, MKV and FLV. The counted path becomes the only one, - and it is not called `pop_counted`: the depth is fixed for the - rendition's life, so it moves into the constructor - (`DecodeClock::new(reorder)`) and the release is a plain `pop(flush)`. - A new rendition gets a new clock. Delete the timestamp-settled `pop`, its - growing delay, `reach`, FLV's `author_dts`/`dts_reserve`, and the catalog - `jitter`'s role in DTS. -- The depth is a new optional `reorder` field on the Hang video rendition: - the most frames that can precede a frame in decode order and follow it in - presentation, the unit of H.264 `max_num_reorder_frames` and HEVC - `sps_max_num_reorder_pics`. -- DTS shifts by slots, not by a time: the nth frame of a group decodes at the - group's (n - `reorder`)th presentation slot, which no frame still to come - can precede. The group's first `reorder` frames step one 90 kHz tick apart - up to the keyframe's slot: the clock cannot know the group's spacing until - frames it may not hold have arrived (`references_far_apart_keep_their_slots`), - and one tick needs no spacing, keeps DTS increasing within a group and - across groups without history, holds no frame past `reorder`, and only - relaxes those frames' delivery deadlines (found in review). Rejected: a delay of - `reorder` times a frame period, since the catalog `framerate` may change in - place and two exporters would build different clocks (found in review), and - freezing the rate or requiring an SPS fixed rate, which both add a rule - the slot shift does not need. -- Those first frames can decode before the timeline's zero, so the clock - works in signed ticks and each exporter maps them to its wire: TS wraps - modulo 2^33 as its timestamps already do, MKV writes no DTS and only - schedules by it, and FLV, whose tag time is unsigned, rebases its whole - audio and video timeline at its first DTS. Never saturate or push a DTS - past its PTS. -- Absent means 0: no reordering and no decode delay, so a stream without - B-frames pays no latency. Rejected: absent meaning the level's - `MaxDpbFrames` (hundreds of ms on High profile streams that never reorder), - and inferring 0 from Baseline profile only (still charges High profile). -- Every producer that can reorder sets it before publishing the rendition: - encoders from their config; importers (TS, FLV, fMP4, MKV) from the SPS's - declared depth, else from the source's own decode offsets on its first - keyframe (TS PTS - DTS, FLV composition time, fMP4 `ctts`/`trun`): none - means 0, any means the level's `MaxDpbFrames`. MKV carries no decode - offsets, so it takes `MaxDpbFrames` when the SPS declares nothing. An - importer knows a frame reorders when its PTS is below an earlier frame's in - decode order; one that reorders deeper than declared publishes a new - rendition under a fresh name, as a PMT revision does. Rejected (from - review): sampling the whole first group before publishing, which delays - every start by a group to catch a source the new rendition already covers. - `js/publish` refuses encoder output that reorders, since WebCodecs exposes - no B-frame setting to declare. -- An exporter that reads a frame reordered deeper than its rendition's - `reorder` refuses the export, never authors an invalid DTS. So does a - `reorder` above the codec level's `MaxDpbFrames` (at most 16), which no - conformant stream can reach, and a nonzero `reorder` on a codec whose - presentation order is its decode order. This bounds the - clock's held frames by the declared depth whatever the timestamps do, which - replaces the release audit's export-reorder quest: flat or backward PTS can - no longer hold frames without bound or drain quadratically. - -Rust: `rs/hang` catalog; `rs/moq-mux` decode clock, TS/MKV/FLV export, and -the TS, FLV, fMP4 and MKV importers; `rs/moq-video` encoders. JS: `js/hang` -catalog, `js/publish`. Draft: `drafts/draft-lcurley-moq-hang.md` (the field -and its absent meaning), `drafts/draft-lcurley-moq-mpegts.md` (how export -authors DTS). Public API and wire: an optional catalog field whose absence -now means "never reorders", a breaking change for any producer that reorders -without it; add a `MIGRATE.md` entry for `hang`, `@moq/hang` and `moq-mux`, whose -direct users must now set it. - -Tests: two exporters joining one B-frame source at different groups, and one -losing a group, author identical DTS for every group both read, in TS, MKV -and FLV; a stream without `reorder` exports with DTS equal to PTS; a frame -reordered past the declared depth refuses the export, and so does a `reorder` -above the level's `MaxDpbFrames`; a framerate change mid-rendition leaves every -DTS unchanged; a group starting at timestamp zero with a nonzero `reorder` -exports valid TS, MKV and FLV; a keyframe without a decode offset followed by -a reordered frame publishes a new rendition; a source deepening its -reordering publishes a new rendition; flat and backward PTS through MKV and -TS export retain at most `reorder` frames, swept over item count and payload -size; each importer sets `reorder` from the SPS and from source offsets. -Keep the T-STD controls passing. - -## Related - -- [Deterministic TS legs for H.265](/quest/m2/ts-hitless-h265.md) - the `--sync` group anchor this path shares -- [Access units across PES](/quest/m1/ts-reliability/pes-access-units.md) - TS import's other video timing gap diff --git a/rs/moq-mux/MIGRATE.md b/rs/moq-mux/MIGRATE.md index fe8b55509aa..95664bbd45f 100644 --- a/rs/moq-mux/MIGRATE.md +++ b/rs/moq-mux/MIGRATE.md @@ -4,15 +4,13 @@ How to update your code for each breaking change, newest first. ## Unreleased -### MKV schedules declared progressive AVC by presentation slots - -A progressive AVC SPS that declares a reorder depth now fixes MKV export's decode -clock for each group. Its first decode slots can precede timestamp zero; block -presentation timestamps remain unchanged. Catalog `jitter` and `framerate` no -longer change that rendition's decode slots. A source that exceeds its declared -depth or cannot produce increasing decode slots now ends the export instead of -moving its timestamps. Correct the source timestamps or declare a new rendition -with the actual reorder depth. +### Exports clock a declared `reorder` per group (#6453) + +The H.264 and H.265 importers now declare the catalog's video `reorder` from the SPS (`max_num_reorder_frames` on a frame-only H.264 SPS, `sps_max_num_reorder_pics` on H.265), and keep the first declaration for the track: a later SPS that reorders deeper fails the import. +For a rendition that declares `reorder`, the MPEG-TS, Matroska, and FLV exports derive each group's decode times from that group's frames alone, ignoring `jitter` and `framerate`. +A group's first `reorder` frames decode just before its earliest presentation time, so a DTS can precede zero: MPEG-TS wraps it in the 33-bit field, and FLV shifts every timestamp by the deepest declared `reorder` in milliseconds. +A frame reordered deeper than declared, or a `reorder` the codec level cannot reach, now ends the export instead of moving its timestamps. +Correct the source, or leave `reorder` unset to keep inferring decode times from timestamps as before. ### Passthrough fMP4 refuses a rewound group as `TimestampRewind` (#6426) diff --git a/rs/moq-mux/src/codec/h264/import.rs b/rs/moq-mux/src/codec/h264/import.rs index 1d0a70db968..152f0321844 100644 --- a/rs/moq-mux/src/codec/h264/import.rs +++ b/rs/moq-mux/src/codec/h264/import.rs @@ -255,7 +255,7 @@ pub fn config(buf: &[u8]) -> Result { /// The config an `AVCDecoderConfigurationRecord` describes, stored verbatim as the `description` /// (`inline: false`, the avc1 shape). -fn config_from_avcc(avcc_bytes: &[u8]) -> Result { +pub(crate) fn config_from_avcc(avcc_bytes: &[u8]) -> Result { let avcc = super::Avcc::parse(avcc_bytes)?; let mut config = hang::catalog::VideoConfig::new(hang::catalog::H264 { @@ -394,6 +394,39 @@ mod tests { assert_eq!((color.mastering, color.content_light), (None, None)); } + /// A frame-only SPS declares its reorder depth, which holds for the track's life: a later SPS + /// declaring less or none keeps it, and one declaring more is refused. + #[tokio::test(start_paused = true)] + async fn a_declared_reorder_depth_holds_for_the_track() { + use crate::codec::h264::fixtures::{PPS, SPS_FIELD, SPS_IPB, SPS_PYRAMID}; + + let idr: &[u8] = &[0x65, 0x88, 0x84, 0x21]; + let (deep, catalog) = setup("deep"); + let (shallow, _) = setup("shallow"); + let mut deep = Import::new(deep, catalog.reserve(), Default::default()).unwrap(); + let mut shallow = Import::new(shallow, catalog.reserve(), Default::default()).unwrap(); + let mut split = Split::new(); + let mut keyframe = |import: &mut Import, name: &str, ms: u64, sps: &[u8]| { + let mut annexb = BytesMut::new(); + for nal in [sps, PPS, idr] { + annexb.extend_from_slice(&[0, 0, 0, 1]); + annexb.extend_from_slice(nal); + } + let pts = moq_net::Timestamp::from_millis(ms).unwrap(); + let mut frames = split.decode(&annexb, pts).expect("split keyframe"); + frames.extend(split.flush(pts).expect("flush keyframe")); + import + .decode(frames) + .map(|()| catalog.snapshot().video.renditions[name].reorder) + }; + + assert_eq!(keyframe(&mut deep, "deep", 0, SPS_PYRAMID).unwrap(), Some(2)); + assert_eq!(keyframe(&mut deep, "deep", 40, SPS_IPB).unwrap(), Some(2)); + assert_eq!(keyframe(&mut deep, "deep", 80, SPS_FIELD).unwrap(), Some(2), "fields declare none"); + assert_eq!(keyframe(&mut shallow, "shallow", 0, SPS_IPB).unwrap(), Some(1)); + assert!(keyframe(&mut shallow, "shallow", 40, SPS_PYRAMID).is_err()); + } + /// A new out-of-band avcC starts a new coded sequence, so in-band HDR metadata from the last one /// is dropped. #[tokio::test(start_paused = true)] diff --git a/rs/moq-mux/src/codec/h264/mod.rs b/rs/moq-mux/src/codec/h264/mod.rs index 75f83bcda23..2fdcad8bd22 100644 --- a/rs/moq-mux/src/codec/h264/mod.rs +++ b/rs/moq-mux/src/codec/h264/mod.rs @@ -196,11 +196,6 @@ pub(crate) fn sps_hrd(nal: &[u8]) -> Option { sps_vui(nal)?.hrd } -/// The colour an SPS NAL unit's VUI signals, when it carries a video signal type. -pub(crate) fn sps_color(nal: &[u8]) -> Option { - sps_vui(nal)?.color -} - /// How a buffering period SEI lays out the delays of the NAL HRD an SPS NAL unit declares. pub(crate) fn sps_buffering(nal: &[u8]) -> Option { sps_vui(nal)?.buffering @@ -288,6 +283,11 @@ impl Vui { config.display_aspect_width = display_aspect.map(|(width, _)| width); config.display_aspect_height = display_aspect.map(|(_, height)| height); config.color = self.color.clone(); + // A field picture counts as half a frame, so only a frame-only SPS declares in frames. + config.reorder = self + .depth + .filter(|depth| self.frame_only && *depth <= crate::codec::video::max_reorder(config)) + .and_then(|depth| u8::try_from(depth).ok()); } } @@ -958,6 +958,7 @@ mod tests { /// Only a VUI with a video signal type adds colour, and its colour description is read whole. #[test] fn sps_color_reads_the_video_signal_type() { + let sps_color = |nal: &[u8]| sps_vui(nal)?.color; let sdr = sps_color(fixtures::SPS_BT709).unwrap(); assert_eq!( (sdr.primaries, sdr.transfer, sdr.matrix, sdr.range), diff --git a/rs/moq-mux/src/codec/h265/import.rs b/rs/moq-mux/src/codec/h265/import.rs index 60f5256b18c..9b9abcc64e3 100644 --- a/rs/moq-mux/src/codec/h265/import.rs +++ b/rs/moq-mux/src/codec/h265/import.rs @@ -378,6 +378,14 @@ mod tests { assert_eq!(config(&ceiling).unwrap().framerate, None); } + /// The SPS's reorder depth at its highest sub-layer becomes the catalog's declaration. + #[test] + fn config_declares_the_sps_reorder_depth() { + let mut pyramid = vec![0, 0, 0, 1]; + pyramid.extend_from_slice(fixtures::SPS_PYRAMID); + assert_eq!(config(&pyramid).unwrap().reorder, Some(2)); + } + /// Square samples carry no display aspect, so the picture is shown at its own size. #[test] fn square_samples_carry_no_display_aspect() { diff --git a/rs/moq-mux/src/codec/h265/mod.rs b/rs/moq-mux/src/codec/h265/mod.rs index dd4fa27835e..edcf6e1b6ad 100644 --- a/rs/moq-mux/src/codec/h265/mod.rs +++ b/rs/moq-mux/src/codec/h265/mod.rs @@ -237,7 +237,7 @@ pub(crate) fn config_from_hvcc(hvcc: &[u8]) -> Result Result<()> { @@ -255,6 +255,7 @@ pub(crate) fn apply_hvcc_sps(hvcc: &mp4_atom::Hvcc, config: &mut hang::catalog:: config.display_aspect_width = metadata.display_aspect_width; config.display_aspect_height = metadata.display_aspect_height; config.color = metadata.color; + config.reorder = metadata.reorder; Ok(()) } @@ -292,6 +293,14 @@ pub(crate) fn config_from_sps(sps_nal: &[u8], in_band: bool) -> Result crate::Result<()> { self.resolved = Some(config.clone()); self.hdr.apply(&mut config.color); self.hint.apply(&mut config); + if let Some(declared) = self.last.as_ref().and_then(|last| last.reorder) { + match config.reorder { + Some(reorder) if reorder > declared => { + return Err(anyhow::Error::new(Deepened { declared, reorder }).into()); + } + _ => config.reorder = Some(declared), + } + } if self.last.as_ref() == Some(&config) { return Ok(()); } diff --git a/rs/moq-mux/src/container/flv/import.rs b/rs/moq-mux/src/container/flv/import.rs index 2d5e6df58e6..5aee6abe6b4 100644 --- a/rs/moq-mux/src/container/flv/import.rs +++ b/rs/moq-mux/src/container/flv/import.rs @@ -24,7 +24,7 @@ use std::collections::BTreeMap; use anyhow::Context; use bytes::{Buf, Bytes, BytesMut}; -use hang::catalog::{AAC, AudioCodec, AudioConfig, H264, VideoConfig}; +use hang::catalog::{AAC, AudioCodec, AudioConfig, VideoConfig}; use super::{ AAC_RAW, AAC_SEQUENCE_HEADER, AUDIO_FORMAT_AAC, AUDIO_FORMAT_EX, AUDIO_FORMAT_MP3, AUDIO_PACKET_CODED_FRAMES, @@ -221,7 +221,7 @@ impl Import { let data = &body[5..]; match avc_packet_type { - AVC_SEQUENCE_HEADER => self.init_video(DEFAULT_TRACK_ID, config_from_avcc(data)?), + AVC_SEQUENCE_HEADER => self.init_video(DEFAULT_TRACK_ID, crate::codec::h264::config_from_avcc(data)?), AVC_NALU => self.write_video( DEFAULT_TRACK_ID, data, @@ -291,7 +291,7 @@ impl Import { match packet_type { VIDEO_PACKET_SEQUENCE_START => { let config = match fourcc { - b"avc1" => config_from_avcc(payload)?, + b"avc1" => crate::codec::h264::config_from_avcc(payload)?, b"hvc1" => crate::codec::h265::config_from_hvcc(payload)?, b"av01" => crate::codec::av1::config_from_av1c(payload)?, // VP9 carries its config in band; the SequenceStart vpcC is @@ -724,21 +724,6 @@ fn take_fourcc(cursor: &mut &[u8]) -> Option<[u8; 4]> { } /// Build a video config for the `avc1` shape from an `AVCDecoderConfigurationRecord`. -fn config_from_avcc(avcc_bytes: &[u8]) -> anyhow::Result { - let avcc = crate::codec::h264::Avcc::parse(avcc_bytes)?; - let mut config = VideoConfig::new(H264 { - profile: avcc.profile, - constraints: avcc.constraints, - level: avcc.level, - inline: false, - }); - config.description = Some(Bytes::copy_from_slice(avcc_bytes)); - config.coded_width = avcc.coded_width; - config.coded_height = avcc.coded_height; - config.color = avcc.sps.first().and_then(|sps| crate::codec::h264::sps_color(sps)); - Ok(config) -} - /// Build an audio config for AAC from an `AudioSpecificConfig`. fn config_from_asc(asc_bytes: &[u8]) -> anyhow::Result { let mut cursor = asc_bytes; diff --git a/rs/moq-mux/src/container/fmp4/import.rs b/rs/moq-mux/src/container/fmp4/import.rs index af5787ba58c..3550bf854e4 100644 --- a/rs/moq-mux/src/container/fmp4/import.rs +++ b/rs/moq-mux/src/container/fmp4/import.rs @@ -713,6 +713,7 @@ impl Import { config.display_aspect_width = metadata.display_aspect_width; config.display_aspect_height = metadata.display_aspect_height; config.color = metadata.color; + config.reorder = metadata.reorder; config.description = Some(description.freeze()); config.container = container; Ok(config) diff --git a/rs/moq-mux/src/container/ts/recovery_test.rs b/rs/moq-mux/src/container/ts/recovery_test.rs index 063f1903fe7..706ce7c4dca 100644 --- a/rs/moq-mux/src/container/ts/recovery_test.rs +++ b/rs/moq-mux/src/container/ts/recovery_test.rs @@ -1667,11 +1667,16 @@ async fn a_congested_path_sheds_what_it_made_late() { "the queue made pictures late" ); let lags = lags(&run); + // A group's leading pictures decode just before its keyframe is presented, so they leave + // later than the rest: compare the least lag of a whole group at each end. + let group = 12; + let (first, last) = ( + lags[..group].iter().min().unwrap(), + lags[lags.len() - group..].iter().min().unwrap(), + ); assert!( - lags.last().unwrap().abs_diff(lags[0]) <= FRAME, - "the latency recovers: {:?} at first, {:?} at last", - lags[0], - lags.last().unwrap() + last.abs_diff(*first) <= FRAME, + "the latency recovers: {first:?} at first, {last:?} at last" ); } From 0c282dfbb80547946f61261f691ac7582eb41329 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 12:05:04 -0700 Subject: [PATCH 05/11] fix(mux): carry reorder through VideoHint and format moq-video advertises its probed rendition as a hint, so the first keyframe's SPS confirms the declared depth instead of adding it. Co-Authored-By: Claude Opus 5.5 --- js/hang/src/catalog/video.test.ts | 2 +- rs/moq-mux/src/catalog/tracks.rs | 4 ++++ rs/moq-mux/src/codec/h264/import.rs | 6 +++++- rs/moq-mux/src/container/decode.rs | 5 ++++- rs/moq-mux/src/container/ts/export_test.rs | 12 +++++++----- rs/moq-mux/src/container/ts/export_timing_test.rs | 12 ++++++++++-- rs/moq-video/src/encode/producer.rs | 1 + 7 files changed, 32 insertions(+), 10 deletions(-) diff --git a/js/hang/src/catalog/video.test.ts b/js/hang/src/catalog/video.test.ts index ddb8ddff404..975ba2d4b48 100644 --- a/js/hang/src/catalog/video.test.ts +++ b/js/hang/src/catalog/video.test.ts @@ -57,7 +57,7 @@ test("video config keeps a zero reorder distinct from an absent one", () => { const parse = (reorder?: number) => VideoConfigSchema.safeParse({ codec: "vp8", container: { kind: "legacy" }, reorder }); expect(parse().data?.reorder).toBeUndefined(); - expect(parse(0).data?.reorder).toBe(0); + expect(parse(0).data?.reorder).toBe(Catalog.u8(0)); expect(parse(256).success).toBe(false); }); diff --git a/rs/moq-mux/src/catalog/tracks.rs b/rs/moq-mux/src/catalog/tracks.rs index 4c1cb8fd793..49ff318575e 100644 --- a/rs/moq-mux/src/catalog/tracks.rs +++ b/rs/moq-mux/src/catalog/tracks.rs @@ -200,6 +200,8 @@ pub struct VideoHint { pub bitrate: Option, /// The frame rate in frames per second. pub framerate: Option, + /// The rendition's [`reorder`](hang::catalog::VideoConfig::reorder) depth. + pub reorder: Option, /// If true, the decoder optimizes for latency. pub optimize_for_latency: Option, /// The maximum jitter before the next frame is emitted. @@ -243,6 +245,7 @@ impl From for VideoHint { color: config.color, bitrate: config.bitrate, framerate: config.framerate, + reorder: config.reorder, optimize_for_latency: config.optimize_for_latency, jitter: config.jitter, delay: config.delay, @@ -269,6 +272,7 @@ impl VideoHint { crate::codec::color::fill(&mut config.color, self.color.as_ref(), "hint"); fill(&mut config.bitrate, self.bitrate); fill(&mut config.framerate, self.framerate); + fill(&mut config.reorder, self.reorder); fill(&mut config.optimize_for_latency, self.optimize_for_latency); fill(&mut config.jitter, self.jitter); fill(&mut config.delay, self.delay); diff --git a/rs/moq-mux/src/codec/h264/import.rs b/rs/moq-mux/src/codec/h264/import.rs index 152f0321844..5c4e1f4d616 100644 --- a/rs/moq-mux/src/codec/h264/import.rs +++ b/rs/moq-mux/src/codec/h264/import.rs @@ -422,7 +422,11 @@ mod tests { assert_eq!(keyframe(&mut deep, "deep", 0, SPS_PYRAMID).unwrap(), Some(2)); assert_eq!(keyframe(&mut deep, "deep", 40, SPS_IPB).unwrap(), Some(2)); - assert_eq!(keyframe(&mut deep, "deep", 80, SPS_FIELD).unwrap(), Some(2), "fields declare none"); + assert_eq!( + keyframe(&mut deep, "deep", 80, SPS_FIELD).unwrap(), + Some(2), + "fields declare none" + ); assert_eq!(keyframe(&mut shallow, "shallow", 0, SPS_IPB).unwrap(), Some(1)); assert!(keyframe(&mut shallow, "shallow", 40, SPS_PYRAMID).is_err()); } diff --git a/rs/moq-mux/src/container/decode.rs b/rs/moq-mux/src/container/decode.rs index 15ad9523170..a7839e03998 100644 --- a/rs/moq-mux/src/container/decode.rs +++ b/rs/moq-mux/src/container/decode.rs @@ -407,7 +407,10 @@ mod tests { assert!(clock.len() <= 2); refused }); - assert!(refused.is_some_and(|index| index <= 3), "backwards={backwards} count={count}"); + assert!( + refused.is_some_and(|index| index <= 3), + "backwards={backwards} count={count}" + ); } } } diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 71b1b5d9513..0b88ae46c45 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -1152,10 +1152,7 @@ async fn export_decodes_where_the_source_did() { /// A broadcast of one H.264 rendition declaring `reorder`, with a group of `stamps` (ms, in /// decode order) for each entry, exported as a recording. /// The producers stay alive with the export, so it reads the retained tracks. -async fn export_declared( - reorder: u8, - groups: &[&[u64]], -) -> (Export, impl Sized) { +async fn export_declared(reorder: u8, groups: &[&[u64]]) -> (Export, impl Sized) { let mut broadcast = moq_net::broadcast::Info::new().produce(); let consumer = broadcast.consume(); let mut catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); @@ -1169,7 +1166,12 @@ async fn export_declared( inline: true, }); config.reorder = Some(reorder); - catalog.modify().unwrap().video.renditions.insert("video".into(), config); + catalog + .modify() + .unwrap() + .video + .renditions + .insert("video".into(), config); let format = HangContainer::Legacy(Kind::Video); for (sequence, stamps) in groups.iter().enumerate() { let mut group = track diff --git a/rs/moq-mux/src/container/ts/export_timing_test.rs b/rs/moq-mux/src/container/ts/export_timing_test.rs index a8675ffaf50..62e021a2df3 100644 --- a/rs/moq-mux/src/container/ts/export_timing_test.rs +++ b/rs/moq-mux/src/container/ts/export_timing_test.rs @@ -1106,7 +1106,11 @@ async fn synced_legs_render_reordered_video_the_same_way() { /// Each video frame's DTS (else PTS) by its PTS, in 90 kHz ticks, as a leg rendered it. fn decode_times(leg: &Leg) -> std::collections::BTreeMap { - let bytes: Vec = leg.out.iter().flat_map(|(_, frame)| frame.payload.iter().copied()).collect(); + let bytes: Vec = leg + .out + .iter() + .flat_map(|(_, frame)| frame.payload.iter().copied()) + .collect(); let mut reader = TsPacketReader::new(Cursor::new(bytes)); let mut out = std::collections::BTreeMap::new(); while let Some(packet) = reader.read_ts_packet().unwrap() { @@ -1137,7 +1141,11 @@ async fn declared_legs_author_the_same_dts() { live.finish(&mut [&mut a, &mut b]).await; let (a, b) = (decode_times(&a), decode_times(&b)); let shared: Vec<_> = b.iter().filter(|(pts, _)| a.contains_key(pts)).collect(); - assert!(shared.len() > 6 * GOP as usize, "join {join}: {} shared frames", shared.len()); + assert!( + shared.len() > 6 * GOP as usize, + "join {join}: {} shared frames", + shared.len() + ); for (pts, dts) in shared { assert_eq!(a[pts], *dts, "join {join}: the frame presented at {pts}"); } diff --git a/rs/moq-video/src/encode/producer.rs b/rs/moq-video/src/encode/producer.rs index 81e789887d1..d2d336ebdcb 100644 --- a/rs/moq-video/src/encode/producer.rs +++ b/rs/moq-video/src/encode/producer.rs @@ -59,6 +59,7 @@ fn rendition_hint(rendition: hang::catalog::VideoConfig) -> moq_mux::catalog::Vi hint.display_aspect_height = rendition.display_aspect_height; hint.color = rendition.color; hint.framerate = rendition.framerate; + hint.reorder = rendition.reorder; hint.bitrate = rendition.bitrate; hint.optimize_for_latency = rendition.optimize_for_latency; // Authoritative for both the catalog entry and the wire, so dropping it would silently From 33923b1eb66d403527ccdac9fe6614d32d3dfab7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 17:49:58 -0700 Subject: [PATCH 06/11] fix(mux): flush FLV's declared clock when its group ends FLV read through `poll_read`, which swallows group ends and cuts, so a group shorter than its declared `reorder` stayed held until the next keyframe or the track's end. Read events, as MKV and TS do, and flush the clock on a group end or cut. Co-Authored-By: Claude Opus 5.5 --- rs/moq-mux/src/container/flv/export.rs | 11 ++++++++-- rs/moq-mux/src/container/flv/export_test.rs | 24 +++++++++++++++------ 2 files changed, 26 insertions(+), 9 deletions(-) diff --git a/rs/moq-mux/src/container/flv/export.rs b/rs/moq-mux/src/container/flv/export.rs index f31b03c4168..60f05171b09 100644 --- a/rs/moq-mux/src/container/flv/export.rs +++ b/rs/moq-mux/src/container/flv/export.rs @@ -34,6 +34,7 @@ use super::{ VIDEO_EX_HEADER, VIDEO_PACKET_CODED_FRAMES, VIDEO_PACKET_MULTITRACK, VIDEO_PACKET_SEQUENCE_START, }; use crate::catalog::Stream; +use crate::container::consumer::Event; use crate::container::decode::DeclaredClock; use crate::container::{ExportSource, Frame}; @@ -491,8 +492,14 @@ impl Export { if track.finished { break; } - let frame = match track.source.poll_read(waiter)? { - Poll::Ready(Some(frame)) => frame, + let frame = match track.source.poll_event(waiter)? { + Poll::Ready(Some(Event::Frame { frame, .. })) => frame, + // A group that ended or was cut short holds frames no later one can precede. + Poll::Ready(Some(Event::GroupEnd | Event::Cut)) => { + track.release((kind, index), None, self.shift, true, &mut self.jitter)?; + continue; + } + Poll::Ready(Some(Event::FrameEnd { .. })) => continue, Poll::Ready(None) => { track.finished = true; break; diff --git a/rs/moq-mux/src/container/flv/export_test.rs b/rs/moq-mux/src/container/flv/export_test.rs index 168eb8befc8..b585a9a0c67 100644 --- a/rs/moq-mux/src/container/flv/export_test.rs +++ b/rs/moq-mux/src/container/flv/export_test.rs @@ -1344,8 +1344,9 @@ async fn export_authors_dts_and_composition_time_for_reordered_avc() { /// The `(DTS, PTS)` of every AVC frame tag, and the audio tag timestamps, an export renders of /// a broadcast that declares `reorder` and carries a group of `stamps` (ms, decode order) for -/// each entry, plus one audio frame at 20 ms. -async fn export_declared(reorder: u8, groups: &[&[u64]]) -> anyhow::Result<(Vec<(u32, i64)>, Vec)> { +/// each entry, plus one audio frame at 20 ms. With `open`, the last group is cut but the video +/// track stays open. +async fn export_declared(reorder: u8, groups: &[&[u64]], open: bool) -> anyhow::Result<(Vec<(u32, i64)>, Vec)> { use hang::catalog::{AAC, AudioConfig, Container, H264, VideoConfig}; use moq_net::Timestamp; @@ -1397,7 +1398,10 @@ async fn export_declared(reorder: u8, groups: &[&[u64]]) -> anyhow::Result<(Vec< video.write(frame(ms, payload, index == 0)).unwrap(); } } - video.finish().unwrap(); + match open { + true => video.cut(None).unwrap(), + false => video.finish().unwrap(), + } let mut audio = crate::container::Producer::new( audio_track, crate::catalog::hang::Container::Legacy(crate::container::Kind::Data), @@ -1414,6 +1418,7 @@ async fn export_declared(reorder: u8, groups: &[&[u64]]) -> anyhow::Result<(Vec< if let Some(Err(err)) = drain_until_idle(&mut exporter, &mut exported).await { return Err(err); } + drop(video); let tags = parse_tags(&exported); let video = tags .iter() @@ -1437,7 +1442,7 @@ async fn export_declared(reorder: u8, groups: &[&[u64]]) -> anyhow::Result<(Vec< #[tokio::test(start_paused = true)] async fn export_declared_reorder_decodes_each_group_from_its_frames() { let groups: [&[u64]; 2] = [&[0, 160, 80, 40, 120], &[400, 560, 480, 440, 520]]; - let (video, audio) = export_declared(2, &groups).await.unwrap(); + let (video, audio) = export_declared(2, &groups, false).await.unwrap(); assert_eq!( video, [ @@ -1457,14 +1462,19 @@ async fn export_declared_reorder_decodes_each_group_from_its_frames() { // A first group cut short leaves the next decoding where it would have. let cut: [&[u64]; 2] = [&[0, 160, 80], &[400, 560, 480, 440, 520]]; - let (cut, _) = export_declared(2, &cut).await.unwrap(); + let (cut, _) = export_declared(2, &cut, false).await.unwrap(); assert_eq!(cut[3..], video[5..]); // A depth of zero decodes at presentation, and refuses a frame that reorders. - let (video, _) = export_declared(0, &[&[0, 40, 80]]).await.unwrap(); + let (video, _) = export_declared(0, &[&[0, 40, 80]], false).await.unwrap(); assert_eq!(video, [(0, 0), (40, 40), (80, 80)]); - let err = export_declared(0, &[&[0, 80, 40]]).await.unwrap_err(); + let err = export_declared(0, &[&[0, 80, 40]], false).await.unwrap_err(); assert!(err.to_string().contains("declared depth"), "{err}"); + + // A group that ends goes out whole while the track stays open, though fewer frames than the + // depth settle nothing. + let (video, _) = export_declared(2, &[&[0, 80]], true).await.unwrap(); + assert_eq!(video, [(0, 2), (1, 82)]); } /// Poll `exporter` into `out` until it ends or goes idle; `None` if it went idle. From 95e9d3967b81ef5558882fd106a0a1b551efb647 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 18:23:55 -0700 Subject: [PATCH 07/11] fix(mux): keep trimmed leading pictures' slots on TS's declared clock TS export trims open-GOP leading pictures a join or recovery orphans before they reach the decode clock, so on a declared rendition the rest of their group decoded at different slots than on an export that kept them. Hold a trimmed picture's slot on the declared clock and drop it at release. Co-Authored-By: Claude Opus 5.5 --- rs/moq-mux/src/container/ts/export.rs | 25 ++++++++++++--- rs/moq-mux/src/container/ts/export_test.rs | 37 ++++++++++++++++++++++ 2 files changed, 57 insertions(+), 5 deletions(-) diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index af59b1755a3..a7bb2948acb 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -221,6 +221,9 @@ struct Tally { tables: Option<(VersionNumber, psi::Pmt)>, } +/// A video frame waiting on a decode clock, with the codec config it was read under. +type Held = (Pending, Option); + /// A frame read from its source. struct Pending { frame: Frame, @@ -290,9 +293,10 @@ struct Track { descriptors: Vec, /// Authors each video frame's DTS, holding the frames until it can, for a rendition that /// declares no `reorder`. - clock: DecodeClock<(Pending, Option)>, + clock: DecodeClock, /// The current group's clock, when its rendition declared `reorder` at the group's start. - declared: Option)>>, + /// A trimmed leading picture holds its slot as `None`. + declared: Option>>, /// Loss counter at the last read, excluding declared ends. recovery_gap: Option, /// Leading pictures below the recovery keyframe have unavailable references, and whether @@ -394,7 +398,15 @@ impl Track { && !radl(stream_type, self.source.description(), &pending.frame) { jitter.lose(name, ahead); - return Ok(()); + // It still takes its slot, so the group's other frames decode where they do on an + // exporter that kept it. + return match &mut self.declared { + Some(clock) => { + clock.push(None, to_ticks(pending.frame.timestamp))?; + self.release(name, jitter, false) + } + None => Ok(()), + }; } self.held = Some(pending.cut); if pending.frame.keyframe { @@ -410,7 +422,7 @@ impl Track { let pts = to_ticks(pending.frame.timestamp); let item = (pending, self.source.description().cloned()); match &mut self.declared { - Some(clock) => clock.push(item, pts)?, + Some(clock) => clock.push(Some(item), pts)?, None => self.clock.push(item, pts), } self.release(name, jitter, false) @@ -623,7 +635,10 @@ impl Track { fn release(&mut self, name: &str, jitter: &mut jitter::Buffer, flush: bool) -> anyhow::Result<()> { if let Some(mut clock) = self.declared.take() { let released = (|| { - while let Some(((mut pending, description), dts)) = clock.pop(flush)? { + while let Some((item, dts)) = clock.pop(flush)? { + let Some((mut pending, description)) = item else { + continue; + }; if !self.finished { pending.arrived = pending.arrived.max(self.latest); } diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index bfbbf0ed7fc..5cfdf99d9bb 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -1235,6 +1235,43 @@ async fn export_declared_reorder_decodes_each_group_from_its_frames() { ); } +/// Open-GOP leading pictures an export trims, because it started at their group, still take +/// their slots: the group's other frames decode where they do on an export that read the group +/// before and kept them. +#[tokio::test(start_paused = true)] +async fn trimmed_leading_pictures_keep_their_slots() { + async fn timing(groups: &[&[u64]]) -> Vec<(u64, u64)> { + let (mut export, _keep) = export_declared(2, groups).await; + let bytes: Vec = drain_frames(&mut export) + .await + .iter() + .flat_map(|frame| frame.payload.iter().copied()) + .collect(); + video_pes_timing(&bytes) + } + let open: &[u64] = &[400, 320, 360, 560, 480, 440, 520]; + let kept = timing(&[&[0, 160, 80, 40, 120], open]).await; + let joined = timing(&[open]).await; + let ms = |ms: u64| ms * 90; + assert_eq!( + kept[5..], + [ + (ms(400), ms(320) - 2), + (ms(320), ms(320) - 1), + (ms(360), ms(320)), + (ms(560), ms(360)), + (ms(480), ms(400)), + (ms(440), ms(440)), + (ms(520), ms(480)), + ] + ); + let shared: Vec<_> = kept[5..] + .iter() + .filter(|(pts, _)| ![ms(320), ms(360)].contains(pts)) + .collect(); + assert_eq!(joined.iter().collect::>(), shared); +} + /// A declared depth of zero decodes every frame at its presentation, and refuses a frame that /// reorders, rather than inferring a decode delay. #[tokio::test(start_paused = true)] From 16e29eceaa9d7cbb30c65753bcdd6fc29a2905e3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 19:09:16 -0700 Subject: [PATCH 08/11] docs(drafts): log the video reorder field in the moq-hang-04 changelog Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-hang.md | 1 + 1 file changed, 1 insertion(+) diff --git a/drafts/draft-lcurley-moq-hang.md b/drafts/draft-lcurley-moq-hang.md index 159dcf25d6c..c47005e40b2 100644 --- a/drafts/draft-lcurley-moq-hang.md +++ b/drafts/draft-lcurley-moq-hang.md @@ -1209,6 +1209,7 @@ This document has no IANA actions. - Each archive timeline declares its own fixed nominal record duration, omitted for an untimed track; removed the broadcast-wide duration bound. - Removed the Recording section: the store layout is a moq-archive format, documented with that crate, not part of this draft. - Added the optional video `preference` field: a consumer selects automatically among the highest preference it can decode. +- Added the optional video `reorder` field: the most frames that can precede one in decode order and follow it in presentation, fixed for the track once declared. A consumer that authors decode times derives them from it and the current group alone, refusing deeper reordering; without it, a consumer infers them from timestamps. - Added an optional `epoch` beside every rendition and data track `broadcast` reference, and `replayEpoch` beside the archive's `replay` ({{field-epoch}}). A consumer reads each referenced path from the first broadcast it resolves for a catalog instance; a present `epoch` fails a rendition proven to resolve another instance, and is best effort over a route without an Epoch. - Derived HLS and DASH exports hand disabled references to enabled renditions at the first record numbered past the advertised ones, keeping the record `sequence` as the HLS media sequence and starting a DASH period. - Added optional `stats` and `echo` catalog sections, publisher/viewer snapshots keyed by rendition ID, and the `.echo` feedback broadcast convention. From 52780bf86c7f1e579cbf727e4f6a1bcafceb3a15 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 19:34:02 -0700 Subject: [PATCH 09/11] fix(mux): raise FLV's shift for a reorder declared after the header The shift was fixed at the header, so a catalog that declared `reorder` later put its first group's leading DTS before zero near the timeline's start and failed the export. Raise the shift to the deeper declaration; it never shrinks, so tag times keep increasing. Co-Authored-By: Claude Opus 5.5 --- rs/moq-mux/src/container/flv/export.rs | 5 ++ rs/moq-mux/src/container/flv/export_test.rs | 69 +++++++++++++++++++++ 2 files changed, 74 insertions(+) diff --git a/rs/moq-mux/src/container/flv/export.rs b/rs/moq-mux/src/container/flv/export.rs index 60f05171b09..703cb9e2081 100644 --- a/rs/moq-mux/src/container/flv/export.rs +++ b/rs/moq-mux/src/container/flv/export.rs @@ -575,6 +575,11 @@ impl Export { // its config record changes. for track in &mut self.video { if let Some(config) = catalog.video.renditions.get(&track.name) { + // A depth declared after the header deepens the shift, so its group's first DTS stays + // at or past zero. The shift never shrinks, so tag times keep increasing. + if self.header_emitted && track.flavor.has_composition_time() { + self.shift = self.shift.max(config.reorder.map_or(0, u32::from)); + } track.config = Some(config.clone()); } } diff --git a/rs/moq-mux/src/container/flv/export_test.rs b/rs/moq-mux/src/container/flv/export_test.rs index b585a9a0c67..4a6e495c0d3 100644 --- a/rs/moq-mux/src/container/flv/export_test.rs +++ b/rs/moq-mux/src/container/flv/export_test.rs @@ -1477,6 +1477,75 @@ async fn export_declared_reorder_decodes_each_group_from_its_frames() { assert_eq!(video, [(0, 2), (1, 82)]); } +/// A depth the catalog declares only after the header raises the shift for the groups it clocks, +/// rather than put their first DTS before zero. +#[tokio::test(start_paused = true)] +async fn a_reorder_declared_after_the_header_raises_the_shift() { + use hang::catalog::{Container, H264, VideoConfig}; + use moq_net::Timestamp; + + let mut producer = moq_net::broadcast::Info::new().produce(); + let consumer = producer.consume(); + let mut catalog = crate::catalog::Producer::new(&mut producer, crate::catalog::Config::default()).unwrap(); + let video_track = producer.create_track(producer.unique_name(".avc1"), timed()).unwrap(); + let name = video_track.name().to_string(); + let mut config = VideoConfig::new(H264 { + profile: 0x42, + constraints: 0xc0, + level: 0x1f, + inline: false, + }); + config.container = Container::Legacy; + config.description = Some(Bytes::from(avcc())); + catalog.modify().unwrap().video.renditions.insert(name.clone(), config); + let frame = |ms: u64, keyframe| crate::container::Frame { + timestamp: Timestamp::from_millis(ms).unwrap(), + duration: None, + payload: Bytes::from_static(if keyframe { + &[0, 0, 0, 1, 0x65] + } else { + &[0, 0, 0, 1, 0x41] + }), + keyframe, + }; + let mut video = crate::container::Producer::new( + video_track, + crate::catalog::hang::Container::Legacy(crate::container::Kind::Data), + ); + video.write(frame(0, true)).unwrap(); + + let mut exporter = open_export(crate::source::announced(&consumer).await) + .await + .with_delay(RECORDING_MAX_DELAY) + .with_replay(); + let mut exported = exporter.next().await.unwrap().expect("the header").to_vec(); + + catalog + .modify() + .unwrap() + .video + .renditions + .get_mut(&name) + .unwrap() + .reorder = Some(2); + for (ms, keyframe) in [(1, true), (5, false), (3, false)] { + video.write(frame(ms, keyframe)).unwrap(); + } + video.finish().unwrap(); + catalog.finish().unwrap(); + drain_until_idle(&mut exporter, &mut exported).await.unwrap().unwrap(); + + let video: Vec<_> = parse_tags(&exported) + .iter() + .filter(|tag| tag.tag_type == super::TAG_VIDEO && tag.body[1] == super::AVC_NALU) + .map(|tag| { + let pts = i64::from(tag.timestamp) + i64::from(super::read_i24(&tag.body[2..5])); + (tag.timestamp, pts) + }) + .collect(); + assert_eq!(video, [(0, 2), (1, 3), (2, 7), (3, 5)]); +} + /// Poll `exporter` into `out` until it ends or goes idle; `None` if it went idle. async fn drain_until_idle( exporter: &mut Export, From b68685e84321fc0ee3a3d9b285c89b9c7eada209 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 19:34:02 -0700 Subject: [PATCH 10/11] fix(publish): drop a repeated capture timestamp before encoding A source that repeats a frame's timestamp made the encoder emit a chunk at the same time, which the `reorder: 0` guard treated as reordering and aborted the track. Skip the repeat at input; a backward timestamp still fails the track. Co-Authored-By: Claude Opus 5.5 --- js/publish/src/video/encoder.test.ts | 13 +++++++++++++ js/publish/src/video/encoder.ts | 3 +++ 2 files changed, 16 insertions(+) diff --git a/js/publish/src/video/encoder.test.ts b/js/publish/src/video/encoder.test.ts index 67217c74fea..c3c6a9e4236 100644 --- a/js/publish/src/video/encoder.test.ts +++ b/js/publish/src/video/encoder.test.ts @@ -998,6 +998,19 @@ test("cut forces a keyframe on the next frame, coalescing requests without spaci expect(keys).toEqual([0, 200, 300]); }); +test("a repeated capture timestamp is dropped rather than failing the track", async () => { + let closed: unknown; + const keys = await recordKeyframes({}, async ({ encoder, track, send }) => { + await send(0); + await send(0); + encoder.cut(); + await send(100); + closed = track.closed.peek(); + }); + expect(keys).toEqual([0, 100]); + expect(closed).toBeUndefined(); +}); + test("a join over the refresh ratio forces a keyframe on the next frame", async () => { const keys = await recordKeyframes({ refresh: 2 }, async ({ track, send }) => { const watching = track.subscribe(); diff --git a/js/publish/src/video/encoder.ts b/js/publish/src/video/encoder.ts index c870d84a856..50e7c25305f 100644 --- a/js/publish/src/video/encoder.ts +++ b/js/publish/src/video/encoder.ts @@ -477,6 +477,9 @@ export class Encoder { // Allow half an interval of slack so jittery capture timestamps don't drop // a frame we meant to keep. const targetFrameRate = config?.frameRate; + // A repeated timestamp would present with the frame before it, which no reordering + // declared allows; a backward one still fails the track at output. + if (frame.timestamp === lastEncoded) continue; if (targetFrameRate && lastEncoded !== undefined) { const minGap = Micro.fromSecond((1 / targetFrameRate) as Second); if (frame.timestamp - lastEncoded < minGap - minGap / 2) continue; From 7cfcce4e39adac25827a1920c1f88067f8096968 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 09:12:08 -0700 Subject: [PATCH 11/11] quest: drop the release audit's emptied Required list Co-Authored-By: Claude Opus 5.5 --- quest/m1/release-audit/README.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/quest/m1/release-audit/README.md b/quest/m1/release-audit/README.md index eb01467349c..63434a8bf8d 100644 --- a/quest/m1/release-audit/README.md +++ b/quest/m1/release-audit/README.md @@ -85,12 +85,6 @@ The 2026-10-22 binding and OBS audit at main rate recovery child. Its two OBS findings went to m2 next to the other OBS work, because obs-moq ships on its own schedule. -## Required - -The entries comprise reproduced or source-confirmed findings -and contract/lifetime investigations. Each -child records whether its behavior is new on main or inherited from release. - ## Related - [JS subscription ranges](/quest/m1/subscribe-ranges/js.md) - range lists and frame offsets build on the bounds fixed here