Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions doc/bin/ts.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,8 +185,9 @@ what the network drops, so random packet loss alone loses no content.
hole longer than the export can pad over skips PCR slots, and then the first
PCR after them flags the clock jump. PAT/PMT versions do not change either
way. A group cut short keeps what arrived. Video resumes at its next
keyframe, dropping orphaned leading pictures; other tracks carry on. On a path slower than the media, whole groups
are shed and the latency holds. After an outage that exhausts the buffered
keyframe, dropping H.264 and H.265 leading pictures that reference the group
before; MPEG-2 video goes out whole. Other tracks carry on. On a path slower
than the media, whole groups are shed and the latency holds. After an outage that exhausts the buffered
media, overdue padding can arrive in a burst when media resumes.
- **Passthrough:** surviving packets go out unchanged, on their source schedule.
A missing interval stays missing: the export never repairs continuity counters,
Expand Down
21 changes: 0 additions & 21 deletions quest/m1/ts-open-gop-trim.md

This file was deleted.

36 changes: 0 additions & 36 deletions quest/m5/pes-hold-open-gop.md

This file was deleted.

24 changes: 24 additions & 0 deletions quest/m5/ts-open-gop-join.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# [XS] TS open-GOP recovery tests cover a join at an in-flight group

## Goal

The H.264 open-GOP and HEVC RASL recovery tests also cover a TS export
that starts at a group already in flight, so a regression in how a join
orphans that group's leading pictures fails the merge gate.

## Plan

Decided 2026-10-31 (user, /quest-plan of the follow-ups from the 2026-10-31 /quest-complete run), from [moq-dev/moq#7365](https://github.com/moq-dev/moq/pull/7365).
Rescoped 2026-11-17 on [moq-dev/moq#8830](https://github.com/moq-dev/moq/pull/8830), which deleted the MPEG-2 picture parser
and the MPEG-2 recovery arm: MPEG-2 video now goes out whole, so the
original goal of trimming zero-stuffed MPEG-2 leading B-pictures is dropped,
and that arm was the only one with a nonzero `join`.

In `rs/moq-mux/src/container/ts/recovery_test.rs`, give
`open_gop_trims_only_the_orphans` (`Profile::Open`) and
`hevc_trims_only_orphaned_rasl` (`Profile::Rasl`) an arm with a nonzero
`join` that lands inside a group, and assert what the export does with the
leading pictures of the group it starts on. Check the grader's join-group
accounting (`joined` in `recover`) models that case before relying on it.

Public API: none. Wire: none.
179 changes: 9 additions & 170 deletions rs/moq-mux/src/container/ts/export.rs
Original file line number Diff line number Diff line change
Expand Up @@ -379,13 +379,6 @@ struct Track {
settle: bool,
/// The PTS of the last keyframe queued, which leading pictures present before.
keyframe: Option<Timestamp>,
/// The `temporal_reference` of the I-picture the last MPEG-1/2 video group opened on, when
/// its GOP is open: the B-pictures presented before it reference the group before.
open_gop: Option<u16>,
/// The last MPEG-1/2 video PES read holds only pictures of the group before's: B-pictures
/// presented before the open GOP's I-picture, or the tail of one. The next PES's tail of
/// its last picture is one too.
orphan: bool,
/// What the catalog and the SPS declare about the video's reordering.
timing: Timing,
/// The receiver's buffers for an audio or verbatim track; video's come from [`Timing`].
Expand Down Expand Up @@ -446,8 +439,6 @@ impl Track {
held: None,
settle: false,
keyframe: None,
open_gop: None,
orphan: false,
timing: Timing::default(),
buffer: None,
pack: None,
Expand All @@ -473,13 +464,8 @@ impl Track {
self.keyframe = None;
self.leading_floor = None;
}
// MPEG-1/2 video is placed, so its pictures rather than its timestamps say what leads.
let stream_type = match self.kind {
Kind::Video(stream_type) => Some(stream_type),
Kind::PesPacket {
stream_type: 0x01 | 0x02,
} => None,
_ => return self.push(name, pending, None, None, jitter),
let Kind::Video(stream_type) = self.kind else {
return self.push(name, pending, None, None, jitter);
};
// A declared endpoint increments both counters; only delivery loss arms recovery.
let gap = pending.skip.saturating_sub(pending.restart);
Expand All @@ -488,30 +474,10 @@ impl Track {
let ahead = self.held == Some(pending.cut);
self.leading_floor = recover.then_some((pending.frame.timestamp, ahead));
}
let leading = match stream_type {
Some(stream_type) => {
self.leading_floor
.is_some_and(|(floor, _)| pending.frame.timestamp < floor)
&& !radl(stream_type, self.source.description(), &pending.frame)
}
None => {
let (payload, after) = (&pending.frame.payload, self.orphan);
let pictures = Mpeg2Picture::read(payload);
if pending.frame.keyframe {
self.open_gop = (pictures.first())
.filter(|picture| picture.gop == Some(false))
.map(|picture| picture.temporal_reference);
} else if pictures.iter().any(|picture| picture.gop.is_some()) {
// A GOP inside the group: its leading pictures reference only the group's own.
self.open_gop = None;
}
// A PES that also holds a picture others reference goes out whole, its orphans
// with it.
self.orphan = !pending.frame.keyframe
&& (self.open_gop).is_some_and(|opened| Mpeg2Picture::orphaned(payload, opened, after));
self.orphan
}
};
let leading = self
.leading_floor
.is_some_and(|(floor, _)| pending.frame.timestamp < floor)
&& !radl(stream_type, self.source.description(), &pending.frame);
if let Some((_, ahead)) = self.leading_floor
&& leading
{
Expand All @@ -527,9 +493,6 @@ impl Track {
};
}
self.held = Some(pending.cut);
let Some(stream_type) = stream_type else {
return self.push(name, pending, None, None, jitter);
};
if pending.frame.keyframe {
self.timing.describe(stream_type, self.source.description(), name);
// A group decodes on its own clock at the depth its rendition declares now. The
Expand Down Expand Up @@ -893,8 +856,7 @@ impl Track {
&& self.keyframe.is_some_and(|keyframe| frame.timestamp < keyframe)
&& !radl(stream_type, description.as_ref(), &frame)
}
// Read by [`Self::queue`] just before, for MPEG-1/2 video carried whole.
_ => self.orphan,
_ => false,
};
if video && frame.keyframe {
self.keyframe = Some(frame.timestamp);
Expand Down Expand Up @@ -4915,76 +4877,6 @@ fn stream_id(kind: &Kind) -> Option<u8> {
matches!(kind, Kind::Opus { .. }).then_some(StreamId::PRIVATE_STREAM_1)
}

/// Where a picture of an MPEG-1/2 video PES sits in its GOP (ISO 13818-2 6.2.2.6, 6.2.3).
#[derive(Clone, Copy)]
struct Mpeg2Picture {
/// Whether a GOP header leads the picture, and if so whether it sets `closed_gop`. A leading
/// B-picture of a closed GOP references only the pictures after it.
gop: Option<bool>,
/// Its place in display order within the GOP.
temporal_reference: u16,
/// A B-picture (`picture_coding_type` 3).
bidirectional: bool,
}

impl Mpeg2Picture {
/// Read every picture header of the whole PES `pes`, in order. `broken_link` is the
/// encoder's own mark of an edit, so an open GOP is judged by `closed_gop` alone.
fn read(pes: &[u8]) -> Vec<Self> {
psi::PesStart::parse(pes).map_or_else(|_| Vec::new(), |start| Self::scan(start.data))
}

/// Read every picture header of the elementary stream `es`, in order.
fn scan(es: &[u8]) -> Vec<Self> {
let (mut pictures, mut gop) = (Vec::new(), None);
for at in memchr::memmem::find_iter(es, &[0, 0, 1]) {
match &es[at + 3..] {
// time_code is 25 bits, then closed_gop.
[0xb8, _, _, _, flags, ..] => gop = Some(flags & 0x40 != 0),
[0x00, high, low, ..] => pictures.push(Self {
gop: gop.take(),
temporal_reference: u16::from(*high) << 2 | u16::from(low >> 6),
bidirectional: (low >> 3) & 0x07 == 3,
}),
_ => {}
}
}
pictures
}

/// Whether the PES `pes` holds only B-pictures presented before the I-picture at temporal
/// reference `opened`, or the tail of one.
///
/// A PES need not start a picture. One opening on bytes outside any start code, or on a
/// slice before its first picture header, carries the tail of the last picture of the PES
/// before, an orphan only when that PES was one (`after`). A PES that may end inside a start
/// code or the header after it is kept, since the next PES holds the rest of a picture it
/// cannot read. Trailing zero bytes are kept too: usually stuffing, but trimming half a
/// reference picture's start code would lose it, while an orphan kept is only skipped.
fn orphaned(pes: &[u8], opened: u16, after: bool) -> bool {
let Ok(start) = psi::PesStart::parse(pes) else {
return false;
};
let es = start.data;
let mut codes = memchr::memmem::find_iter(es, &[0, 0, 1]).peekable();
let first = codes.peek().copied().unwrap_or(es.len());
let opens = es[..first].iter().all(|&byte| byte == 0);
let continues = !opens
|| codes
.map(|at| es.get(at + 3).copied())
.take_while(|&code| code != Some(0x00))
.any(|code| matches!(code, Some(0x01..=0xaf)));
// A GOP header, the longest read, ends 8 bytes after its start code begins.
let cut = es.last() == Some(&0) || memchr::memmem::rfind(es, &[0, 0, 1]).is_some_and(|at| es.len() - at < 8);
let pictures = Self::scan(es);
let tail = match continues {
true => after,
false => !pictures.is_empty(),
};
tail && !cut && (pictures.iter()).all(|picture| picture.bidirectional && picture.temporal_reference < opened)
}
}

/// Whether a video frame is an HEVC RADL picture, which leads its keyframe but references only
/// it, so it goes out wherever that keyframe does rather than being trimmed as an orphan.
fn radl(stream_type: StreamType, description: Option<&Bytes>, frame: &Frame) -> bool {
Expand Down Expand Up @@ -5242,8 +5134,8 @@ mod tests {
use moq_net::Timestamp;

use super::{
Buffer, DecodeClock, Dolby, Kind, Laid, Mpeg2Picture, PSI_INTERVAL, PesPacket, Psi, Settle, dolby_removals,
due, is_complete_section, si_due, slot, unwrap_pts, verbatim_buffer,
Buffer, DecodeClock, Dolby, Kind, Laid, PSI_INTERVAL, PesPacket, Psi, Settle, dolby_removals, due,
is_complete_section, si_due, slot, unwrap_pts, verbatim_buffer,
};

/// The tables as they stand take the version after the last laid, whatever was set on the
Expand Down Expand Up @@ -6010,59 +5902,6 @@ mod tests {
pes
}

#[test]
fn mpeg2_orphans_are_whole_pes_of_leading_b_pictures() {
// A picture header: temporal_reference, then picture_coding_type (1 I, 2 P, 3 B).
let picture = |reference: u16, coding: u8| {
[
0,
0,
1,
0,
(reference >> 2) as u8,
((reference & 3) as u8) << 6 | coding << 3,
0xff,
0xf8,
]
};
let slice = [0, 0, 1, 1, 0x12, 0x34, 0x56, 0x78];
let video = |pictures: &[&[u8]]| pes(0xe0, 0x00, &[], &pictures.concat());
for after in [false, true] {
assert!(Mpeg2Picture::orphaned(
&video(&[&picture(0, 3), &picture(1, 3)]),
2,
after
));
// A B-picture presented after the I-picture references only its own GOP.
assert!(!Mpeg2Picture::orphaned(&video(&[&picture(3, 3)]), 2, after));
// The P-picture later pictures reference keeps its PES, orphans and all.
assert!(!Mpeg2Picture::orphaned(
&video(&[&picture(0, 3), &picture(1, 3), &picture(5, 2)]),
2,
after
));
assert!(!Mpeg2Picture::orphaned(&video(&[]), 2, after));
// Zero stuffing before the first start code opens no tail.
assert!(Mpeg2Picture::orphaned(&video(&[&[0], &picture(0, 3)]), 2, after));
// Half a start code or picture header: the next PES holds a picture this one cannot
// read.
assert!(!Mpeg2Picture::orphaned(&video(&[&picture(0, 3), &[0, 0]]), 2, after));
assert!(!Mpeg2Picture::orphaned(
&video(&[&picture(0, 3), &[0, 0, 1, 0, 0]]),
2,
after
));
}
// The tail of a picture the PES before began, as slices or mid-slice bytes, is an orphan
// only when that PES was: it may finish the I-picture its group opened on.
for tail in [&slice[..], &slice[4..]] {
assert!(!Mpeg2Picture::orphaned(&video(&[tail, &picture(0, 3)]), 2, false));
assert!(Mpeg2Picture::orphaned(&video(&[tail, &picture(0, 3)]), 2, true));
assert!(!Mpeg2Picture::orphaned(&video(&[tail]), 2, false));
assert!(Mpeg2Picture::orphaned(&video(&[tail]), 2, true));
}
}

#[test]
fn pes_packet_validation() {
// A PTS of 0x1_2345_6789 behind the '0010' prefix, read back as written.
Expand Down
Loading