diff --git a/drafts/draft-lcurley-moq-mpegts.md b/drafts/draft-lcurley-moq-mpegts.md index 6549bfcf5c2..884b2bdc088 100644 --- a/drafts/draft-lcurley-moq-mpegts.md +++ b/drafts/draft-lcurley-moq-mpegts.md @@ -93,7 +93,7 @@ What remains is media in a codec-neutral catalog, plus the signaling a demultipl | | mpeg2ts packaging | mpegts section | |:--------|:--------|:--------| | MoQ track | one per program or multiplex | one per elementary stream | -| Object payload | whole source packets | one media frame, or one verbatim PES payload or section | +| Object payload | whole source packets | one media frame, or one verbatim PES payload, PES packet, or section | | Packaging | `mpeg2ts` | none defined; media keeps its own | | Catalog | `mpeg2ts*` members on each track | one `mpegts` root member | | Relay | caches and drops whole objects | drops and prioritizes per elementary stream | @@ -126,6 +126,7 @@ type Track = { "pid": number, "descriptors": Descriptor[] | undefined, "verbatim": Verbatim | undefined, + "pesPacket": PesPacket | undefined, } ~~~ @@ -134,9 +135,10 @@ A publisher MUST NOT record 0x0000 (PAT) or 0x1FFF (null packets), which carry n `descriptors` are the track's ES-level descriptors from the PMT ({{descriptor}}), in PMT order. -`verbatim` marks a track carrying an undecoded elementary stream ({{verbatim}}). +`verbatim` and `pesPacket` mark a track carrying an undecoded elementary stream ({{verbatim}}). +An entry MUST NOT carry both, and a consumer MUST refuse a catalog with one that does. A decoded track MAY appear here too, to preserve its `pid` and `descriptors`. -A consumer MUST ignore an entry with neither a `verbatim` record nor a track the catalog describes. +A consumer MUST ignore an entry with no `verbatim` record, no `pesPacket` record, and no track the catalog describes, so one that predates `pesPacket` drops such a track from the stream it rebuilds. ## programDescriptors The PMT's program-level descriptors (`program_info`), in PMT order ({{descriptor}}). @@ -243,7 +245,22 @@ A publisher SHOULD record it, since strict demultiplexers reject a relabeled PES Each frame MUST be a keyframe in its own group: a verbatim payload is all-or-nothing. -A verbatim track has no codec, dimensions, or decoder configuration, so a publisher MUST NOT describe it as a media rendition and MUST describe it only by its `tracks` entry. +A track can instead carry each PES whole, header included: + +~~~ +type PesPacket = { + "streamType": number, +} +~~~ + +`streamType` is the PMT `stream_type` to re-announce, as for `Verbatim`. +Each frame MUST be exactly one PES packet, from its start code through its payload, with a `PES_packet_length` that matches the frame, or zero only where {{mpeg2}} lets a video stream leave it unbounded. +The `stream_id`, the PTS and DTS when present, and every other header field ride in the frame, with the PTS and DTS on the catalog clock; a PES without a PTS carries none. +The frame's timestamp is the packet's placement: the program clock at its first byte, on the same timeline. +A consumer MUST refuse a frame that is not one well-formed PES packet. +The record is a member beside `verbatim` rather than a `framing` value, so a consumer that predates it ignores the track instead of refusing the catalog. + +A verbatim or whole-PES track has no codec, dimensions, or decoder configuration, so a publisher MUST NOT describe it as a media rendition and MUST describe it only by its `tracks` entry, and a consumer MUST refuse a catalog that describes one as both. Nothing there declares a container, so its frames use hang's `legacy` container ({{hang}}): a varint microsecond timestamp followed by the payload. @@ -349,6 +366,7 @@ A consumer rebuilding a transport stream: - MUST re-emit each track's `descriptors` as its ES-level descriptors, and `programDescriptors` as the PMT's `program_info`. - MUST re-emit each `si` entry's sections byte-for-byte on that entry's PID, reading them from its track ({{si-track}}), at least as often as its `interval` when declared. - MUST repacketize each verbatim track per its `framing` and `streamType`, using `streamId` when recorded. +- MUST packetize each `pesPacket` frame unchanged on its track's PID, its first byte going out no earlier than the frame's timestamp, and SHOULD send it as close to that as the multiplex allows. - SHOULD pad the output with null packets to `muxRate` when present, so the rebuilt stream is constant-rate again; a source that exceeds the rate is passed through rather than delayed or dropped. With no `program` the consumer synthesizes an identity, and SHOULD then omit any carried `si`, which describes a program that no longer exists. @@ -423,6 +441,7 @@ A broadcast demultiplexed from a DVB transport stream: video and audio described {:numbered="false"} - Added the `m2ts` section, a transport stream carried whole, with its mapping onto MSFTS; the comparison uses MSFTS's `mpeg2ts` names. +- Added the `pesPacket` record: each frame one whole PES packet, timestamped with its placement. A consumer that predates it drops the track rather than refusing the catalog. ## draft-lcurley-moq-mpegts-00 {:numbered="false"} diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index f54585bfe89..30841498da5 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -101,7 +101,6 @@ remain with their current quests. - [Plan: semantic fidelity](/quest/m1/ts-reliability/fidelity.md) - explicit profiles and PMT lifecycle contracts produce implementation quests - [Raw stream types](/quest/m1/ts-reliability/stream-type.md) - unknown stream types survive export instead of failing the PMT -- [Whole-PES export](/quest/m1/ts-reliability/pes-export.md) - export writes `pesPacket` tracks byte for byte at their placement - [Whole-PES import](/quest/m1/ts-reliability/pes-import.md) - private PES travels whole at its source placement, a missing PTS stays missing, and the `pes` framing is deleted - [Section placement](/quest/m1/ts-reliability/section-placement.md) - sections share the PCR placement clock instead of the video PTS - [Loss and recovery tests](/quest/m1/ts-reliability/recovery.md) - actual MoQ omissions exercise reacquisition and unaffected tracks diff --git a/quest/m1/ts-reliability/pes-export.md b/quest/m1/ts-reliability/pes-export.md deleted file mode 100644 index 700d2600856..00000000000 --- a/quest/m1/ts-reliability/pes-export.md +++ /dev/null @@ -1,73 +0,0 @@ -# [M] TS export emits whole PES packets at their placement - -## Goal - -`moq export ts` reads tracks the catalog marks `pesPacket` and writes each -frame as the PES packet it holds, byte for byte, sent at the frame's -timestamp as its placement time. A PES without a PTS goes out without one; -nothing in its header is synthesized. Private data placed alongside or after -the tune-in keyframe survives the tune-in rule. - -## Plan - -Decided 2026-10-16 (see the [epic](/quest/m1/ts-reliability/README.md)): -a `pesPacket` frame is one complete PES packet, start code through payload, -whose PTS, DTS and every other header field are already on the catalog clock. -Its frame timestamp is the source placement time: the program clock at the -PES's first byte, on the media timeline. This lands before the importer -writes the member, so the round trip never meets a catalog export can't read. - -- **Catalog.** Add an optional `pesPacket` member to the `mpegts.tracks` - entry in `rs/moq-mux/src/container/ts/catalog.rs`, beside `verbatim`, - holding the PMT `streamType` (for example `catalog::PesPacket`). It is a - member rather than a `framing` value so old exporters ignore the entry - instead of refusing the catalog. The stream_id travels in each frame, so - it has no `streamId`. An entry with both `verbatim` and `pesPacket`, or - a `pesPacket` entry whose track is also a media rendition, is refused, - as the draft already forbids for `verbatim`. -- **Mux.** Packetize the frame's bytes unchanged on the track's PID: no - `PesHeader`, no AC-3 repacking. The timestamp is an arrival time, not - a decode deadline: the PES's first byte goes out at the placement, - never earlier, with its TS header just before it, so this needs its own path rather than `Track::push`'s - decode-time deadline and `Schedule`'s early slots. A regression test - asserts the first byte's output position. Where `verbatim_buffer` models - the PID (DVB AC-3, teletext), admission reads the PTS from the header - bytes. That PTS covers only the first AC-3 sync frame, so the model - walks the sync frames in the bytes and removes each at its own decode - time, as `ac3_frames` does today, without rewriting the PES. -- **Tune-in.** The rule that drops non-video frames before `video_start` - compares a `pesPacket` frame's placement with the tune-in keyframe's - departure in this export's schedule, not its PTS. Placement is the - frame's due time in that same schedule, so the rule keeps exactly what - can go out after the keyframe; comparing against the PTS drops data - muxed beside it. The source's keyframe placement is not carried, so a - source decoder lead that differs from `--delay` shifts private data - against video by the difference, as any remux with a new delay does. -- **Refusal.** A frame that is not exactly one well-formed PES packet - (start code, a `PES_packet_length` matching the frame, a complete - optional header where the stream_id has one) fails the track, like a - malformed section frame does. A length of zero is allowed only for a - video payload, as H.222.0 permits: a video stream_id (0xE0-0xEF), or an - extended one (0xFD) whose stream type is video, such as VC-1. - -Leave the legacy `pes` framing as it is here; the -[importer quest](/quest/m1/ts-reliability/pes-import.md) deletes it once -nothing writes it. - -Tests, in `export_test.rs` from hand-built catalogs and frames, wired into -the existing `cargo test` CI: PTS 0 versus no PTS, distinct PTS and DTS, -optional fields (ES rate, trick mode, copy info, CRC, extension, stuffing), -`data_alignment_indicator` clear, private_stream_2 and padding stream IDs, -private data before, beside and after the tune-in keyframe, and between -media, an entry naming both members, and one that is also a rendition. Assert the output PES bytes equal -the frame. - -Update `drafts/draft-lcurley-moq-mpegts.md` (the `Track` type, the "MUST -ignore an entry" rule, {{verbatim}}, {{rebuild}}, changelog -01) with the -member, its timestamp meaning, and the fact that a consumer that predates -it drops the track, plus the TS rows of `doc/lib/rs/moq-mux.md`. No JS -change: nothing in `js/` interprets the section. - -## Related - -- [Raw stream types](/quest/m1/ts-reliability/stream-type.md) - unknown stream types are the other way a verbatim track fails export diff --git a/quest/m1/ts-reliability/pes-import.md b/quest/m1/ts-reliability/pes-import.md index 39503a2e45e..0f9128f9f68 100644 --- a/quest/m1/ts-reliability/pes-import.md +++ b/quest/m1/ts-reliability/pes-import.md @@ -54,7 +54,7 @@ round-trip test first. header, so rebasing leaves it valid; assert that with a CRC fixture), extension data, stuffing, and the special stream-ID forms that have no optional header. Each PES keeps its own stream_id. - **Catalog.** The entry records `pid`, `descriptors` and the `pesPacket` - member that [export](/quest/m1/ts-reliability/pes-export.md) reads, and + member that export reads, and no `verbatim` record. A published exporter ignores such an entry and silently leaves the stream out of its TS; that is the accepted cost. - **Refusal.** A PES carrying a clock this does not rebase (ESCR, a @@ -96,7 +96,3 @@ Update `drafts/draft-lcurley-moq-mpegts.md` (the importer writes `pesPacket` and states what it refuses; `framing` is `"section"` only and required, `streamId` removed, changelog -01 noting the break with -00) and the TS lines of `doc/bin/cli.md` and `doc/lib/rs/moq-mux.md`. - -## Required - -- [Whole-PES export](/quest/m1/ts-reliability/pes-export.md) - export reads `pesPacket` before import writes it diff --git a/quest/m1/ts-reliability/section-placement.md b/quest/m1/ts-reliability/section-placement.md index 825cb641bdb..2a6f06cff1f 100644 --- a/quest/m1/ts-reliability/section-placement.md +++ b/quest/m1/ts-reliability/section-placement.md @@ -15,8 +15,7 @@ derives from the PCR in `SectionStream`, replacing `last_pts`/`start_pts` and the edge clamp in `SectionStream::emit`. SCTE-35 splice times live in the section bytes and keep their `pts_adjustment` rebasing. -Export treats the new timestamp the way -[whole-PES export](/quest/m1/ts-reliability/pes-export.md) treats +Export treats the new timestamp the way it treats `pesPacket`: an arrival time on the same placement path, so the first section byte goes out at the placement and never earlier, with the TS header and pointer field just before it (not `Track::push`'s decode diff --git a/rs/moq-mux/src/container/ts/catalog.rs b/rs/moq-mux/src/container/ts/catalog.rs index 78449bb0bc6..612171b3883 100644 --- a/rs/moq-mux/src/container/ts/catalog.rs +++ b/rs/moq-mux/src/container/ts/catalog.rs @@ -2,9 +2,9 @@ //! //! The `mpegts` section carries everything needed to faithfully re-mux a broadcast //! back to MPEG-TS that doesn't belong in the codec-neutral media configs: one -//! entry per track (its original PID and PMT descriptors), a `verbatim` carriage -//! record for every elementary stream we don't decode (SCTE-35, teletext, DVB -//! subtitles, private data, ...), the program-level PMT descriptors, the program +//! entry per track (its original PID and PMT descriptors), a `verbatim` or +//! `pesPacket` carriage record for every elementary stream we don't decode (SCTE-35, +//! teletext, DVB subtitles, private data, ...), the program-level PMT descriptors, the program //! identity ([`Program`]), and the standalone SI table map ([`SiEntry`]). //! Demuxed media tracks keep their codec config in the base `video`/`audio` //! sections; only their MPEG-TS identity lands here. A constant-rate source also @@ -86,7 +86,8 @@ pub(super) fn si_interval(table_id: u8) -> Option { #[non_exhaustive] pub struct Mpegts { /// Per-track MPEG-TS info, keyed by MoQ track name. Media tracks record their - /// PID and PMT descriptors; undecoded tracks add a [`Verbatim`] carriage record. + /// PID and PMT descriptors; undecoded tracks add a [`Verbatim`] or [`PesPacket`] + /// carriage record. #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] pub tracks: BTreeMap, @@ -306,6 +307,11 @@ pub struct Track { /// Present when the stream is carried verbatim (not decoded into `video`/`audio`). #[serde(default, skip_serializing_if = "Option::is_none")] pub verbatim: Option, + + /// Present when each frame is one whole PES packet, sent at its timestamp. An entry + /// carries this or [`Self::verbatim`], never both. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub pes_packet: Option, } impl Track { @@ -315,6 +321,7 @@ impl Track { pid, descriptors: Vec::new(), verbatim: None, + pes_packet: None, } } } @@ -352,6 +359,29 @@ impl Verbatim { } } +/// Carriage record for an undecoded elementary stream whose frames are whole PES packets. +/// +/// Each frame is one complete PES packet, start code through payload, with its PTS, DTS +/// and every other header field already on the catalog clock. Its timestamp is the +/// packet's placement: the program clock at its first byte, which export sends it at. +/// A member beside [`Verbatim`] rather than a [`Framing`], so a consumer that predates it +/// ignores the track instead of refusing the catalog. +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Default)] +#[serde(rename_all = "camelCase")] +#[non_exhaustive] +pub struct PesPacket { + /// PMT `stream_type` to re-announce (0x06 private PES, ...). The `stream_id` rides + /// in each packet. + pub stream_type: u8, +} + +impl PesPacket { + /// A new whole-PES carriage record of the given `stream_type`. + pub fn new(stream_type: u8) -> Self { + Self { stream_type } + } +} + /// How a verbatim stream's payload is framed on the wire, so export can repacketize it. #[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq, Default)] #[serde(rename_all = "camelCase")] @@ -464,6 +494,7 @@ mod test { data: Bytes::from_static(b"eng\x00"), }], verbatim: None, + pes_packet: None, }, ); // A verbatim SCTE-35 track. @@ -473,6 +504,17 @@ mod test { pid: 0x102, descriptors: Vec::new(), verbatim: Some(Verbatim::new(0x86, Framing::Section)), + pes_packet: None, + }, + ); + // A whole-PES private data track. + mpegts.tracks.insert( + "0.ts".to_string(), + Track { + pid: 0x103, + descriptors: Vec::new(), + verbatim: None, + pes_packet: Some(PesPacket::new(0x06)), }, ); mpegts.program_descriptors.push(Descriptor { @@ -488,6 +530,10 @@ mod test { json.contains("\"muxRate\":2500000"), "mux rate is a bare integer: {json}" ); + assert!( + json.contains("\"pesPacket\":{\"streamType\":6}"), + "the whole-PES member is camelCase: {json}" + ); let parsed: Ext = serde_json::from_str(&json).unwrap(); assert_eq!(parsed.mpegts, mpegts, "mpegts section round-trips"); @@ -654,6 +700,7 @@ mod test { data: Bytes::from_static(b"CUEI"), }], verbatim: Some(Verbatim::new(0x86, Framing::Section)), + pes_packet: None, }, ); let expected = guard.ext.mpegts.clone(); diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index b1c028da39e..ca3b0dc38ec 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -187,10 +187,11 @@ where /// up before it ever configures video. `None` until the tables are built, and for /// programs with no video track (nothing to align to). video_start: Option, - /// Whether a video frame has gone out since [`Self::video_start`] was set. Until one has, - /// what the other tracks release is dropped: a video track whose start group was cut - /// before it went out starts at its next keyframe instead ([`jitter::Buffer`]). - video_out: bool, + /// The jitter buffer key of the first video frame out since [`Self::video_start`] was set. + /// Until one has gone, what the other tracks release is dropped: a video track whose + /// start group was cut before it went out starts at its next keyframe instead + /// ([`jitter::Buffer`]). + video_out: Option, /// Frames dropped before the jitter buffer: ahead of the first video keyframe, or of their /// codec config. trimmed: u64, @@ -367,6 +368,8 @@ impl Track { return Ok(()); } let is_video = matches!(self.kind, Kind::Video(_)); + // A placed PES is judged at release, against the keyframe's departure ([`Export::leads_video`]). + let aligned = !is_video && !matches!(self.kind, Kind::PesPacket { .. }); loop { if !input.waiting_for_header && input.jitter.full(&self.pid) { break; @@ -387,7 +390,7 @@ impl Track { }; // A new timeline must reach the reset before tune-in alignment can drop it. if let Some(start) = input.video_start - && !is_video && pending.restart == self.restart + && aligned && pending.restart == self.restart && pending.frame.timestamp < start { *input.trimmed += 1; @@ -554,9 +557,11 @@ impl Track { } /// How far ahead of its decode time the track's units may go out under `delay`: as far - /// as [`schedule::ahead`] lets them, and nothing without a delay. + /// as [`schedule::ahead`] lets them, and nothing without a delay or for a placed PES, + /// whose timestamp is when it goes out. fn ahead(&self, delay: Duration) -> Duration { - if delay.is_zero() { + // A placed PES goes out at its placement, never ahead of it. + if delay.is_zero() || matches!(self.kind, Kind::PesPacket { .. }) { return Duration::ZERO; } let slots = schedule::ahead(self.buffer().as_ref(), schedule::slot(delay.as_nanos())); @@ -792,6 +797,9 @@ enum Kind { framing: catalog::Framing, stream_id: Option, }, + /// An undecoded elementary stream whose frames are whole PES packets, written unchanged + /// and sent at their timestamp, the source's placement ([`catalog::PesPacket`]). + PesPacket { stream_type: u8 }, } impl Kind { @@ -806,7 +814,7 @@ impl Kind { Kind::Mp2 { .. } => ".mp2", Kind::Ac3 => ".ac3", Kind::Eac3 => ".eac3", - Kind::Verbatim { .. } => ".ts", + Kind::Verbatim { .. } | Kind::PesPacket { .. } => ".ts", } } @@ -818,7 +826,7 @@ impl Kind { Kind::Aac { .. } | Kind::Opus { .. } | Kind::Mp2 { .. } | Kind::Ac3 | Kind::Eac3 => { super::stats::Class::Audio } - Kind::Verbatim { .. } => super::stats::Class::Data, + Kind::Verbatim { .. } | Kind::PesPacket { .. } => super::stats::Class::Data, } } } @@ -1104,7 +1112,7 @@ where queue: VecDeque::new(), liveness: Default::default(), video_start: None, - video_out: false, + video_out: None, trimmed: 0, mux_rate: None, mux_rate_override: None, @@ -1256,7 +1264,7 @@ where self.video_start = self.first_video_pts(); if let Some(start) = self.video_start { for track in self.tracks.values_mut() { - if !matches!(track.kind, Kind::Video(_)) + if !matches!(track.kind, Kind::Video(_) | Kind::PesPacket { .. }) && track.pending.as_ref().is_some_and(|p| p.frame.timestamp < start) { track.pending = None; @@ -1313,13 +1321,13 @@ where ready.generation >= self.generation, "the jitter buffer released an older generation after a newer one" ); - let (name, is_video) = self + let (name, kind) = self .tracks .iter() .find(|(_, t)| t.pid == ready.track) - .map(|(name, t)| (name.clone(), matches!(t.kind, Kind::Video(_)))) + .map(|(name, t)| (name.clone(), t.kind.clone())) .context("frame for an unknown PID")?; - if self.leads_video(is_video, ready.item.frame.timestamp) { + if self.leads_video(&kind, &ready.item) { self.jitter.discard(&ready); } else { self.last_timestamp = Some(ready.item.frame.timestamp); @@ -1470,12 +1478,13 @@ where } fn update_catalog(&mut self, mut catalog: Catalog) -> anyhow::Result<()> { - self.source.retain_selectable(&mut catalog); - // The MPEG-TS section lives in the extension. The trait only exposes // `mpegts_mut`, and this snapshot is owned, so clone it out (`()` yields the // empty default: no verbatim streams, no preserved PIDs/descriptors). let mpegts = catalog.ext.mpegts_mut().cloned().unwrap_or_default(); + reject_ambiguous_tracks(&catalog, &mpegts)?; + self.source.retain_selectable(&mut catalog); + self.program_descriptors = mpegts.program_descriptors.clone(); self.program = mpegts.program.clone(); // An explicit override wins even when it is refused (leaving the output @@ -1548,7 +1557,7 @@ where active.insert(name.clone(), ()); } for (name, track) in mpegts.tracks.iter() { - if track.verbatim.is_some() { + if track.verbatim.is_some() || track.pes_packet.is_some() { active.insert(name.clone(), ()); } } @@ -1644,13 +1653,16 @@ where self.refresh(name, track, pids[name], track_descriptors(&mpegts, name)); } for (name, entry) in mpegts.tracks.iter() { - let Some(verbatim) = &entry.verbatim else { - continue; - }; - let kind = Kind::Verbatim { - stream_type: verbatim.stream_type, - framing: verbatim.framing, - stream_id: verbatim.stream_id, + let kind = match (&entry.verbatim, &entry.pes_packet) { + (Some(verbatim), _) => Kind::Verbatim { + stream_type: verbatim.stream_type, + framing: verbatim.framing, + stream_id: verbatim.stream_id, + }, + (None, Some(pes)) => Kind::PesPacket { + stream_type: pes.stream_type, + }, + (None, None) => continue, }; let mut track = match old.remove(name) { Some(track) => track, @@ -1671,7 +1683,7 @@ where let continuous = self .tracks .values() - .filter(|track| !matches!(track.kind, Kind::Verbatim { .. }) || track.buffer.is_some()); + .filter(|track| track.kind.class() != super::stats::Class::Data || track.buffer.is_some()); self.jitter.expect(continuous.map(|track| track.pid)); Ok(()) } @@ -1699,7 +1711,11 @@ where ExportSource::for_video(&self.source, name, config, budget)?.map(|s| at_edge(s, self.replay)) } else if let Some(config) = catalog.audio.renditions.get(name) { ExportSource::for_audio(&self.source, name, config, budget)?.map(|s| at_edge(s, self.replay)) - } else if mpegts.tracks.get(name).is_some_and(|t| t.verbatim.is_some()) { + } else if mpegts + .tracks + .get(name) + .is_some_and(|t| t.verbatim.is_some() || t.pes_packet.is_some()) + { Some(ExportSource::for_stream(&self.source, name, budget)?) } else { None @@ -1879,35 +1895,51 @@ where Ok(()) } - /// Whether a frame released at `timestamp` presents ahead of the video keyframe the stream + /// Whether a frame of `kind` released now presents ahead of the video keyframe the stream /// begins on ([`Self::video_start`]), and so is dropped. A video track whose start group /// was cut or went stale before it went out starts at its next keyframe instead /// ([`jitter::Buffer`]), so until video goes out the start is the earliest frame a video /// track holds, and with none held, everything leads the keyframe yet to arrive. Video goes /// out further ahead of its decode time than the other tracks, so what they release before /// it presents ahead of it too. - fn leads_video(&mut self, is_video: bool, timestamp: Timestamp) -> bool { + /// + /// A placed PES has no presentation to compare: it goes out at its placement, so it is + /// dropped only when that comes before the keyframe departs. Both are release times a + /// delay before their jitter buffer keys, so it compares keys, and keeps whatever goes + /// out beside or after the keyframe even when its PTS is earlier. + fn leads_video(&mut self, kind: &Kind, queued: &Queued) -> bool { let Some(start) = self.video_start else { return false; }; - if is_video { - if !self.video_out { - self.video_out = true; - self.video_start = Some(start.max(timestamp)); + if matches!(kind, Kind::Video(_)) { + if self.video_out.is_none() { + self.video_out = Some(queued.key); + self.video_start = Some(start.max(queued.frame.timestamp)); } return false; } - if self.video_out { - return timestamp < start; + let placed = matches!(kind, Kind::PesPacket { .. }); + if let Some(departed) = self.video_out { + return match placed { + true => queued.key < departed, + false => queued.frame.timestamp < start, + }; } let next = self .tracks .values() .filter(|track| matches!(track.kind, Kind::Video(_))) .filter_map(|track| self.jitter.peek(&track.pid)) - .map(|queued| queued.frame.timestamp) + .map(|next| match placed { + true => next.key, + false => next.frame.timestamp.max(start), + }) .min(); - next.is_none_or(|next| timestamp < next.max(start)) + let at = match placed { + true => queued.key, + false => queued.frame.timestamp, + }; + next.is_none_or(|next| at < next) } /// Discard what has not gone out and restart the program clock. Every rendition @@ -2034,7 +2066,7 @@ where Kind::Mp2 { .. } => StreamType::Mpeg1Audio, Kind::Ac3 => StreamType::DolbyDigitalUpToSixChannelAudio, Kind::Eac3 => StreamType::DolbyDigitalPlusUpTo16ChannelAudioForAtsc, - Kind::Verbatim { stream_type, .. } => { + Kind::Verbatim { stream_type, .. } | Kind::PesPacket { stream_type } => { StreamType::from_u8(*stream_type).map_err(anyhow::Error::msg)? } }; @@ -2198,6 +2230,8 @@ where framing: catalog::Framing::Section, .. } => None, + // The frame is the PES already, written by [`Self::place`] below. + Kind::PesPacket { .. } => None, }; let mut out = Vec::with_capacity(TsPacket::SIZE); @@ -2256,6 +2290,9 @@ where } } + if let Kind::PesPacket { stream_type } = kind { + return self.place(name, stream_type, &frame, out); + } match es_payload { // Section-framed verbatim (SCTE-35, ...) rides in private sections, not PES; // carry the bytes verbatim. @@ -2290,6 +2327,37 @@ where self.push_unit(name, pid, decode, out, keyframe, Vec::new()) } + /// Queue `frame`, one whole PES packet, unchanged on `name`'s PID behind `tables`, its + /// first byte going out at the frame's timestamp, the source's placement, and no earlier. + /// + /// A PID the T-STD models holds the PES's bytes in its decoder buffer until they decode, + /// timed by the PTS in its header. For DVB AC-3 that times only the first sync frame + /// that begins in it, so each one after counts on from there and leaves at its own time. + /// The bytes ahead of the first end a frame an earlier PES began; they leave with it, + /// which only overstates what the buffer holds. A PES without a PTS holds nothing there. + fn place(&mut self, name: &str, stream_type: u8, frame: &Frame, tables: Vec) -> anyhow::Result<()> { + let pes = PesPacket::parse(&frame.payload, stream_type) + .with_context(|| format!("MPEG-TS track '{name}' carries a malformed PES packet"))?; + let track = self.tracks.get(name).context("missing track")?; + let pid = track.pid; + let place = frame.timestamp.as_nanos(); + let decode = pes.pts.map(|pts| unwrap_pts(pts, frame.timestamp)); + let ac3 = stream_type == 0x06 && track.descriptors.iter().any(|d| d.tag == AC3_DESCRIPTOR); + let frames = match decode { + Some(decode) if ac3 => ac3_removals(&frame.payload[pes.payload..], decode), + _ => Vec::new(), + }; + let buffer = track.buffer(); + let mut out = std::mem::take(&mut self.tables); + out.extend_from_slice(&tables); + self.write_pes_packet(&mut out, pid, &frame.payload)?; + if let Some(buffer) = buffer { + self.schedule.set_buffer(pid, buffer); + } + self.schedule + .push_placed(pid, (place, decode.unwrap_or(place)), out, frame.keyframe, frames) + } + /// Mux an audio frame into `name`'s PES, sending the PES on first if the frame cannot join /// it, and leave the tables muxed ahead of it to ride the next unit queued. A PES carries /// frames while they run on from its PTS without a gap and fit the PID's decoder buffer @@ -2661,6 +2729,37 @@ where Ok(()) } + /// Packetize `pes`, a whole PES packet, into TS packets on `pid` without touching a byte of + /// it, stuffing the last packet's adaptation field. The `mpeg2ts` writer would rebuild the + /// PES header instead, so the packets are laid out by hand. + fn write_pes_packet(&mut self, out: &mut Vec, pid: u16, pes: &[u8]) -> anyhow::Result<()> { + anyhow::ensure!(pid < 0x1FFF, "PID out of range: {pid}"); + let mut offset = 0; + while offset < pes.len() { + let start = offset == 0; + let flag = start && self.breaks(pid); + // The adaptation field: its length byte, then a flags byte for the break. + let field = if flag { 2 } else { 0 }; + let take = (TsBytes::MAX_SIZE - field).min(pes.len() - offset); + let field = TsBytes::MAX_SIZE - take; + out.extend_from_slice(&[0x47, u8::from(start) << 6 | (pid >> 8) as u8, pid as u8]); + match field { + 0 => out.push(0x10), + _ => { + out.push(0x30); + out.push((field - 1) as u8); + if field > 1 { + out.push(if flag { 0x80 } else { 0 }); + out.resize(out.len() + field - 2, 0xff); + } + } + } + out.extend_from_slice(&pes[offset..offset + take]); + offset += take; + } + Ok(()) + } + /// Packetize a complete PSI or private section verbatim. The first packet /// carries the pointer_field plus the section start as a `Section` payload (sets /// the unit-start bit so the receiver finds the pointer_field); continuations are @@ -2772,6 +2871,170 @@ const PES_OPTIONAL_LEN: usize = 3 + 5; /// Extra bytes when the optional region also carries a DTS (5 DTS bytes). const PES_DTS_LEN: usize = 5; +/// What the mux reads of a frame holding one whole PES packet ([`Kind::PesPacket`]). +struct PesPacket { + /// Its PTS in 90 kHz ticks, as the 33-bit field carries it. + pts: Option, + /// Where its payload begins, past the header. + payload: usize, +} + +impl PesPacket { + /// Read `bytes` as exactly one well-formed PES packet of a stream of `stream_type`: + /// the start code, a `PES_packet_length` that matches, and a complete optional header + /// where the `stream_id` has one (ISO 13818-1 2.4.3.6). Only video may leave the + /// length zero, unbounded, as 2.4.3.7 permits. + fn parse(bytes: &[u8], stream_type: u8) -> anyhow::Result { + anyhow::ensure!( + bytes.len() >= PES_PREFIX_LEN && bytes.starts_with(&[0, 0, 1]), + "no PES start code" + ); + let stream_id = bytes[3]; + let length = usize::from(u16::from_be_bytes([bytes[4], bytes[5]])); + if length == 0 { + let video = matches!(stream_id, 0xe0..=0xef) || (stream_id == 0xfd && is_video_stream_type(stream_type)); + anyhow::ensure!( + video, + "PES_packet_length 0 on stream_id {stream_id:#04x}, which only a video stream may leave unbounded" + ); + } else { + anyhow::ensure!( + length == bytes.len() - PES_PREFIX_LEN, + "PES_packet_length {length} where the frame holds {} bytes after it", + bytes.len() - PES_PREFIX_LEN + ); + } + // program_stream_map, padding, private_stream_2, ECM, EMM, directory, DSM-CC and + // H.222.1 type E carry their payload straight after the length. + if matches!(stream_id, 0xbc | 0xbe | 0xbf | 0xf0 | 0xf1 | 0xf2 | 0xf8 | 0xff) { + return Ok(Self { + pts: None, + payload: PES_PREFIX_LEN, + }); + } + anyhow::ensure!( + bytes.len() >= 9 && bytes[6] & 0xc0 == 0x80, + "PES optional header missing or without its '10' marker" + ); + let flags = bytes[7]; + let payload = 9 + usize::from(bytes[8]); + let fields = bytes + .get(9..payload) + .context("PES_header_data_length runs past the packet")?; + let pts_dts = flags >> 6; + anyhow::ensure!(pts_dts != 0b01, "PTS_DTS_flags '01' is forbidden"); + // Each flagged field in order (Table 2-21): PTS, DTS, ESCR, ES_rate, DSM trick mode, + // additional_copy_info, previous_PES_CRC; stuffing fills the rest. + let mut at = [ + (0x80, 5), + (0x40, 5), + (0x20, 6), + (0x10, 3), + (0x08, 1), + (0x04, 1), + (0x02, 2), + ] + .iter() + .filter(|&&(flag, _)| flags & flag != 0) + .map(|&(_, len)| len) + .sum::(); + if flags & 0x01 != 0 { + let read = |at: usize| fields.get(at).copied().context("PES extension runs past its header"); + let extension = read(at)?; + at += 1; + // PES_private_data, then pack_header_field behind its own length byte. + if extension & 0x80 != 0 { + at += 16; + } + if extension & 0x40 != 0 { + at += 1 + usize::from(read(at)?); + } + // program_packet_sequence_counter and P-STD_buffer. + at += 2 * usize::from(extension & 0x20 != 0) + 2 * usize::from(extension & 0x10 != 0); + if extension & 0x01 != 0 { + at += 1 + usize::from(read(at)? & 0x7f); + } + } + anyhow::ensure!( + at <= fields.len(), + "PES optional fields need {at} bytes where PES_header_data_length gives {}", + fields.len() + ); + // Each clock field opens with its 4-bit prefix, PTS '0010' alone or '0011' beside a DTS + // '0001', and sets the marker bit after each of its three parts. + let clock = |field: &[u8], prefix: u8| { + anyhow::ensure!( + field[0] >> 4 == prefix && field[0] & field[2] & field[4] & 1 == 1, + "PES clock field without its '{prefix:04b}' prefix or marker bits" + ); + Ok(u64::from((field[0] >> 1) & 7) << 30 + | u64::from(field[1]) << 22 + | u64::from(field[2] >> 1) << 15 + | u64::from(field[3]) << 7 + | u64::from(field[4] >> 1)) + }; + let pts = match pts_dts { + 0b10 => Some(clock(&fields[..5], 0b0010)?), + 0b11 => { + clock(&fields[5..10], 0b0001)?; + Some(clock(&fields[..5], 0b0011)?) + } + _ => None, + }; + Ok(Self { pts, payload }) + } +} + +/// Whether a PMT `stream_type` carries video (ISO 13818-1 Table 2-34, and VC-1's 0xEA), whose +/// PES alone may leave `PES_packet_length` unbounded. +fn is_video_stream_type(stream_type: u8) -> bool { + matches!( + stream_type, + 0x01 | 0x02 | 0x10 | 0x1b | 0x1e..=0x26 | 0x28..=0x2b | 0x31..=0x35 | 0xea + ) +} + +/// A 33-bit PTS read off a placed PES, on the media timeline in nanoseconds: the value +/// nearest its placement `at` that the wrapped field allows, since a PES decodes close to +/// where it is placed. +fn unwrap_pts(pts: u64, at: Timestamp) -> u128 { + const WRAP: i128 = 1 << 33; + let base = i128::from(to_ticks(at)); + let ahead = (i128::from(pts) - base).rem_euclid(WRAP); + let ticks = base + if ahead >= WRAP / 2 { ahead - WRAP } else { ahead }; + (ticks.max(0) as u128) * 1_000_000_000 / 90_000 +} + +/// When each AC-3 sync frame that begins in a PES's `payload` leaves the decoder buffer and +/// the bytes it takes, counting on from `decode`, the PES's PTS: [`Schedule::push_placed`]'s +/// frames. The bytes ahead of the first frame go with it, and a frame the PES leaves +/// unfinished keeps its time for the bytes it holds. Empty with fewer than two frames, +/// where the whole PES leaves at once. +fn ac3_removals(payload: &[u8], decode: u128) -> Vec<(u128, usize)> { + let codec = &crate::codec::ac3::DESCRIPTOR; + let Some(first) = ac3_sync(payload) else { + return Vec::new(); + }; + let mut frames: Vec<(u128, usize)> = Vec::new(); + let (mut at, mut offset) = (decode, first); + while offset < payload.len() { + let Ok(header) = (codec.parse)(&payload[offset..]) else { + break; + }; + let len = header.len.min(payload.len() - offset); + frames.push((at, len)); + at += 1_000_000_000 * u128::from(header.samples) / u128::from(header.sample_rate.max(1)); + offset += len.max(1); + } + if let Some(head) = frames.first_mut() { + head.1 += first; + } + if let Some(tail) = frames.last_mut() { + tail.1 += payload.len().saturating_sub(offset); + } + frames +} + /// Whether `timestamp` has crossed into a later repetition slot than `last`. /// /// Slots are absolute on the media timeline (`floor(timestamp / interval)`) rather than @@ -3040,11 +3303,12 @@ const TELETEXT_DESCRIPTOR: u8 = 0x56; /// data (ATSC A/52 Annex A 5.4), and teletext, whose 480-byte transport buffer drains at /// 6.75 Mb/s (ETSI EN 300 472 5). Any other stream has none the schedule knows. fn verbatim_buffer(kind: &Kind, descriptors: &[catalog::Descriptor]) -> Option { - let Kind::Verbatim { + let (Kind::Verbatim { stream_type: 0x06, framing: catalog::Framing::Pes, .. - } = kind + } + | Kind::PesPacket { stream_type: 0x06 }) = kind else { return None; }; @@ -3383,6 +3647,23 @@ fn reject_colliding_si_pids(mpegts: &catalog::Mpegts, pmt_pid: u16, es_pids: &[u Ok(()) } +/// Refuse an `mpegts.tracks` entry that names both carriages, or carries a track the +/// catalog also describes as a media rendition: either leaves two ways to write one PID. +fn reject_ambiguous_tracks(catalog: &Catalog, mpegts: &catalog::Mpegts) -> anyhow::Result<()> { + for (name, track) in &mpegts.tracks { + anyhow::ensure!( + track.verbatim.is_none() || track.pes_packet.is_none(), + "mpegts track '{name}' names both verbatim and pesPacket carriage" + ); + anyhow::ensure!( + (track.verbatim.is_none() && track.pes_packet.is_none()) + || !(catalog.video.renditions.contains_key(name) || catalog.audio.renditions.contains_key(name)), + "mpegts track '{name}' is carried undecoded but also described as a media rendition" + ); + } + Ok(()) +} + /// The PMT descriptors recorded for `name` in the `mpegts` section, if any. fn track_descriptors(mpegts: &catalog::Mpegts, name: &str) -> Vec { mpegts @@ -3454,7 +3735,10 @@ mod tests { use moq_net::Timestamp; - use super::{Buffer, DecodeClock, Kind, PSI_INTERVAL, due, is_complete_section, si_due, slot, verbatim_buffer}; + use super::{ + Buffer, DecodeClock, Kind, PSI_INTERVAL, PesPacket, ac3_removals, due, is_complete_section, si_due, slot, + unwrap_pts, verbatim_buffer, + }; fn ms(value: u64) -> Timestamp { Timestamp::from_millis(value).unwrap() @@ -3826,4 +4110,98 @@ mod tests { // declared section_length (27) does not match the actual length (3). assert!(!is_complete_section(&[0xfc, 0x30, 0x1b])); } + + /// A PES of `stream_id` with an optional header of `flags` and `fields`, its length right. + fn pes(stream_id: u8, flags: u8, fields: &[u8], payload: &[u8]) -> Vec { + let mut pes = vec![0, 0, 1, stream_id]; + pes.extend_from_slice(&((3 + fields.len() + payload.len()) as u16).to_be_bytes()); + pes.extend_from_slice(&[0x84, flags, fields.len() as u8]); + pes.extend_from_slice(fields); + pes.extend_from_slice(payload); + pes + } + + #[test] + fn pes_packet_validation() { + // A PTS of 0x1_2345_6789 behind the '0010' prefix, read back as written. + let pts = [0x29, 0x8d, 0x15, 0xcf, 0x13]; + let parsed = PesPacket::parse(&pes(0xbd, 0x80, &pts, b"data"), 0x06).unwrap(); + assert_eq!(parsed.pts, Some(0x1_2345_6789)); + assert_eq!(parsed.payload, 14); + assert_eq!( + PesPacket::parse(&pes(0xbd, 0x00, &[], b"data"), 0x06).unwrap().pts, + None + ); + // private_stream_2 and padding carry no optional header. + let bare = [0, 0, 1, 0xbf, 0, 2, 0xaa, 0xbb]; + assert_eq!(PesPacket::parse(&bare, 0x06).unwrap().payload, 6); + + // No start code, a length that disagrees, or a header that runs past the packet. + assert!(PesPacket::parse(&[0, 0, 2, 0xbd, 0, 0], 0x06).is_err()); + let mut long = pes(0xbd, 0x00, &[], b"data"); + long.push(0); + assert!(PesPacket::parse(&long, 0x06).is_err()); + let mut truncated = pes(0xbd, 0x00, &[], &[]); + truncated[8] = 4; + assert!(PesPacket::parse(&truncated, 0x06).is_err()); + // The marker bits, the forbidden PTS_DTS_flags, and fields past the header length. + let mut marker = pes(0xbd, 0x00, &[], b"data"); + marker[6] = 0x44; + assert!(PesPacket::parse(&marker, 0x06).is_err()); + assert!(PesPacket::parse(&pes(0xbd, 0x40, &pts, b""), 0x06).is_err()); + assert!(PesPacket::parse(&pes(0xbd, 0xc0, &pts, b""), 0x06).is_err()); + assert!(PesPacket::parse(&pes(0xbd, 0x01, &[0x80, 0, 0], b""), 0x06).is_err()); + // A clock field with the wrong prefix, or a marker bit cleared, PTS and DTS alike. + let mut prefix = pts; + prefix[0] = 0x39; + assert!(PesPacket::parse(&pes(0xbd, 0x80, &prefix, b""), 0x06).is_err()); + let mut marker = pts; + marker[2] &= !1; + assert!(PesPacket::parse(&pes(0xbd, 0x80, &marker, b""), 0x06).is_err()); + let mut both = [pts, pts].concat(); + both[0] = 0x39; + both[5] = 0x19; + assert!(PesPacket::parse(&pes(0xbd, 0xc0, &both, b""), 0x06).is_ok()); + both[9] &= !1; + assert!(PesPacket::parse(&pes(0xbd, 0xc0, &both, b""), 0x06).is_err()); + + // Only video may leave the length unbounded: its stream IDs, or 0xFD under a video type. + let unbounded = |stream_id: u8| { + let mut pes = pes(stream_id, 0x00, &[], b"picture"); + pes[4..6].fill(0); + pes + }; + assert!(PesPacket::parse(&unbounded(0xe0), 0x06).is_ok()); + assert!(PesPacket::parse(&unbounded(0xfd), 0xea).is_ok()); + assert!(PesPacket::parse(&unbounded(0xfd), 0x06).is_err()); + assert!(PesPacket::parse(&unbounded(0xbd), 0x06).is_err()); + } + + #[test] + fn placed_pts_unwraps_near_its_placement() { + // A PTS just past the 33-bit wrap belongs after a placement just before it. + let at = Timestamp::from_micros(((1u64 << 33) - 900) * 1_000_000 / 90_000).unwrap(); + assert_eq!(unwrap_pts(900, at), ((1u128 << 33) + 900) * 1_000_000_000 / 90_000); + // A PTS a little ahead, and one a little behind, stay where they are. + let at = ms(10_000); + assert_eq!(unwrap_pts(90 * 10_100, at), 10_100_000_000); + assert_eq!(unwrap_pts(90 * 9_900, at), 9_900_000_000); + } + + #[test] + fn ac3_removals_count_on_from_the_pts() { + // 32 kb/s 48 kHz sync frames: 128 bytes, 32 ms each. + let mut sync = vec![0x0b, 0x77, 0x00, 0x00, 0x00, 0x40, 0x40]; + sync.resize(128, 0); + let mut payload = vec![0xee; 10]; + payload.extend(sync.repeat(2)); + payload.extend_from_slice(&sync[..50]); + let removals = ac3_removals(&payload, 1_000_000_000); + // The tail of an earlier frame goes with the first, and the cut frame keeps its time. + assert_eq!( + removals, + vec![(1_000_000_000, 138), (1_032_000_000, 128), (1_064_000_000, 50)] + ); + assert!(ac3_removals(&[0xee; 20], 0).is_empty(), "no sync frame begins"); + } } diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 1d9d9c59b85..867f5046935 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -1816,6 +1816,7 @@ async fn export_scte35_roundtrip() { pid: 0x102, descriptors: Vec::new(), verbatim: Some(tscat::Verbatim::new(0x86, tscat::Framing::Section)), + pes_packet: None, }; catalog .modify() @@ -2062,6 +2063,7 @@ async fn scte35_without_video_export_is_rejected() { pid: 0x102, descriptors: Vec::new(), verbatim: Some(tscat::Verbatim::new(0x86, tscat::Framing::Section)), + pes_packet: None, }; catalog.modify().unwrap().ext.mpegts.tracks.insert(scte_name, track); } @@ -9709,3 +9711,439 @@ async fn a_backlog_stays_in_the_source_until_due() { assert_eq!(export.dropped(), 0, "the sink's stall does not make arrived media late"); } } + +/// The PID of the whole-PES private data track in the tests below. +const PLACED_PID: u16 = 0x104; + +/// A program carrying a `pesPacket` track, for [`export_placed`]. +struct Placed { + /// The PTS in ms of each video frame, the first a keyframe; none leaves AAC to clock it. + video: Vec, + /// The private data track's `mpegts.tracks` entry. + entry: tscat::Track, + /// Each frame's placement in ms, and its bytes: one PES packet. + pes: Vec<(u64, Vec)>, + /// Also describe the private data track as an AAC rendition. + rendition: bool, + mux_rate: Option, +} + +impl Placed { + fn new(pes: Vec<(u64, Vec)>) -> Self { + let mut entry = tscat::Track::new(PLACED_PID); + entry.pes_packet = Some(tscat::PesPacket::new(0x06)); + Self { + video: Vec::new(), + entry, + pes, + rendition: false, + mux_rate: None, + } + } +} + +/// Export `placed` beside an AAC rendition running past its last PES, returning every slice. +async fn export_placed(placed: Placed) -> crate::Result> { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let mut catalog = crate::catalog::Producer::new( + &mut broadcast, + crate::catalog::Config::default().with_catalog(crate::catalog::hang::Catalog::::default()), + ) + .unwrap(); + + let mut video = None; + if !placed.video.is_empty() { + let track = broadcast + .create_track( + broadcast.unique_name(".avc1"), + hang::container::track_info(hang::catalog::PRIORITY.video), + ) + .unwrap(); + let mut cfg = VideoConfig::new(H264 { + profile: 0x42, + constraints: 0xc0, + level: 0x1f, + inline: false, + }); + cfg.container = Container::Legacy; + cfg.description = + Some(crate::codec::h264::build_avcc(&[Bytes::from_static(SPS)], &[Bytes::from_static(PPS)]).unwrap()); + catalog + .modify() + .unwrap() + .video + .renditions + .insert(track.name().to_string(), cfg); + video = Some(Producer::new( + track, + HangContainer::Legacy(crate::container::Kind::Video), + )); + } + + let mut aac = AudioConfig::new(AAC { profile: 2 }, 48_000, 2); + aac.container = Container::Legacy; + let track = broadcast + .create_track( + broadcast.unique_name(".aac"), + hang::container::track_info(hang::catalog::PRIORITY.audio), + ) + .unwrap(); + catalog + .modify() + .unwrap() + .audio + .renditions + .insert(track.name().to_string(), aac.clone()); + let mut audio = Producer::new(track, HangContainer::Legacy(crate::container::Kind::Audio)); + + let track = broadcast + .create_track( + broadcast.unique_name(".ts"), + hang::container::track_info(hang::catalog::PRIORITY.video), + ) + .unwrap(); + { + let mut guard = catalog.modify().unwrap(); + guard.ext.mpegts.tracks.insert(track.name().to_string(), placed.entry); + if placed.rendition { + guard.audio.renditions.insert(track.name().to_string(), aac); + } + } + let mut data = Producer::new(track, HangContainer::Legacy(crate::container::Kind::Data)); + + for (k, &pts) in placed.video.iter().enumerate() { + let slice = vec![if k == 0 { 0x65u8 } else { 0x41 }; 400]; + video + .as_mut() + .unwrap() + .write(Frame { + timestamp: Timestamp::from_millis(pts).unwrap(), + duration: None, + payload: length_prefixed(&[&slice]), + keyframe: k == 0, + }) + .unwrap(); + } + let end = placed + .pes + .iter() + .map(|&(ms, _)| ms) + .chain(placed.video.iter().copied()) + .max(); + let end = (end.unwrap_or(0) + 500) * 1_000; + let mut index = 0; + while index * AUDIO_US < end { + audio + .write(Frame { + timestamp: Timestamp::from_micros(index * AUDIO_US).unwrap(), + duration: None, + payload: Bytes::from(vec![index as u8; 180]), + keyframe: index.is_multiple_of(AUDIO_GROUP), + }) + .unwrap(); + index += 1; + } + for (ms, pes) in placed.pes { + data.write(Frame { + timestamp: Timestamp::from_millis(ms).unwrap(), + duration: None, + payload: Bytes::from(pes), + keyframe: true, + }) + .unwrap(); + } + if let Some(video) = video.as_mut() { + video.finish().unwrap(); + } + audio.finish().unwrap(); + data.finish().unwrap(); + + let mut exporter = open_ts( + crate::source::announced(&consumer).await, + crate::catalog::CatalogFormat::Hang, + ) + .await? + .with_delay(RECORDING_MAX_AGE) + .with_replay(); + if let Some(rate) = placed.mux_rate { + exporter = exporter.with_mux_rate(rate); + } + let mut out = Vec::new(); + while let Ok(next) = tokio::time::timeout(DRAIN, exporter.next()).await { + match next? { + Some(frame) => out.push(frame), + None => break, + } + } + Ok(out) +} + +/// Where a PES went out: the slice it began in, its first byte's offset into the slice, and +/// the bytes in its packets' payloads. +#[derive(Debug)] +struct PlacedOut { + slice: usize, + offset: usize, + bytes: Vec, +} + +/// Every PES on [`PLACED_PID`], reassembled from its packets' payloads. +fn placed_out(slices: &[Frame]) -> Vec { + let mut out: Vec = Vec::new(); + for (slice, frame) in slices.iter().enumerate() { + for (position, packet) in frame.payload.as_chunks::<188>().0.iter().enumerate() { + let pid = (u16::from(packet[1] & 0x1f) << 8) | u16::from(packet[2]); + if pid != PLACED_PID { + continue; + } + let start = 4 + if packet[3] & 0x20 != 0 { + usize::from(packet[4]) + 1 + } else { + 0 + }; + if packet[1] & 0x40 != 0 { + out.push(PlacedOut { + slice, + offset: position * 188 + start, + bytes: Vec::new(), + }); + } + out.last_mut() + .expect("a continuation before any PES start") + .bytes + .extend_from_slice(&packet[start..]); + } + } + out +} + +/// A 33-bit PTS or DTS field behind its 4-bit `prefix`, as ISO 13818-1 2.4.3.7 lays it out. +fn pes_clock(prefix: u8, ticks: u64) -> [u8; 5] { + [ + prefix << 4 | ((ticks >> 29) & 0x0e) as u8 | 1, + (ticks >> 22) as u8, + ((ticks >> 14) & 0xfe) as u8 | 1, + (ticks >> 7) as u8, + ((ticks << 1) & 0xfe) as u8 | 1, + ] +} + +/// A PES with an optional header: `marker` is its first flags byte ('10' then scrambling, +/// priority, alignment, copyright, original), `flags` its second, `fields` the optional +/// fields those flags name, then `stuffing` bytes of 0xFF. +fn pes_with(stream_id: u8, marker: u8, flags: u8, fields: &[u8], stuffing: usize, payload: &[u8]) -> Vec { + let header = fields.len() + stuffing; + let mut pes = vec![0, 0, 1, stream_id]; + pes.extend_from_slice(&((3 + header + payload.len()) as u16).to_be_bytes()); + pes.extend_from_slice(&[marker, flags, header as u8]); + pes.extend_from_slice(fields); + pes.resize(pes.len() + stuffing, 0xff); + pes.extend_from_slice(payload); + pes +} + +/// A PES of a `stream_id` with no optional header (private_stream_2, padding, ...). +fn pes_bare(stream_id: u8, payload: &[u8]) -> Vec { + let mut pes = vec![0, 0, 1, stream_id]; + pes.extend_from_slice(&(payload.len() as u16).to_be_bytes()); + pes.extend_from_slice(payload); + pes +} + +/// Each whole PES goes out exactly as written, whatever its header holds: PTS 0 or none, a +/// DTS apart from its PTS, every optional field, `data_alignment_indicator` clear, the +/// stream IDs without an optional header, and sizes on either side of a packet's payload. +#[tokio::test(start_paused = true)] +async fn placed_pes_goes_out_byte_for_byte() { + let mut extension = vec![0x80 | 0x10 | 0x01]; + extension.extend_from_slice(&[0x5a; 16]); + extension.extend_from_slice(&[0x60, 0x20]); + extension.extend_from_slice(&[0x80 | 3, 1, 2, 3]); + let mut optional = pes_clock(0b0010, 90_000).to_vec(); + optional.extend_from_slice(&[0x80, 0x01, 0x01]); + optional.push(0x1f); + optional.push(0xa5); + optional.extend_from_slice(&[0x12, 0x34]); + optional.extend_from_slice(&extension); + let mut both = pes_clock(0b0011, 180_000).to_vec(); + both.extend_from_slice(&pes_clock(0b0001, 171_000)); + + let packets = vec![ + pes_with(0xbd, 0x84, 0x80, &pes_clock(0b0010, 0), 0, b"pts zero"), + pes_with(0xbd, 0x84, 0x00, &[], 0, b"no pts"), + pes_with(0xbd, 0x84, 0xc0, &both, 0, b"pts and dts"), + pes_with(0xbd, 0x84, 0x80 | 0x1f, &optional, 4, b"every optional field"), + pes_with(0xbd, 0x80, 0x80, &pes_clock(0b0010, 1), 0, b"unaligned"), + pes_bare(0xbf, b"private_stream_2"), + pes_bare(0xbe, &[0xff; 40]), + pes_with(0xbd, 0x84, 0x00, &[], 0, &[0x33; 500]), + // Exactly one packet's payload, and one byte short of it. + pes_with(0xbd, 0x84, 0x00, &[], 0, &[0x44; 184 - 9]), + pes_with(0xbd, 0x84, 0x00, &[], 0, &[0x55; 183 - 9]), + ]; + let pes = packets + .iter() + .enumerate() + .map(|(k, pes)| (1_000 + 100 * k as u64, pes.clone())) + .collect(); + let out = placed_out(&export_placed(Placed::new(pes)).await.unwrap()); + let bytes: Vec<&Vec> = out.iter().map(|pes| &pes.bytes).collect(); + assert_eq!(bytes, packets.iter().collect::>()); +} + +/// The PCR, in nanoseconds, that opens a slice: in its first packet's adaptation field. +fn slice_pcr(frame: &Frame) -> u128 { + let packet = &frame.payload[..188]; + assert!( + packet[3] & 0x20 != 0 && packet[5] & 0x10 != 0, + "the slice opens with its PCR" + ); + let b = &packet[6..12]; + let base = u64::from(b[0]) << 25 + | u64::from(b[1]) << 17 + | u64::from(b[2]) << 9 + | u64::from(b[3]) << 1 + | u64::from(b[4] >> 7); + let extension = u64::from(b[4] & 1) << 8 | u64::from(b[5]); + u128::from(base * 300 + extension) * 1_000 / 27 +} + +/// A PES goes out at its placement, never earlier: at a multiplex rate, whose byte +/// positions keep time, its first byte lands at the first position past it, with the TS +/// header just ahead; unpadded, in the first slot that opens at or after it. +#[tokio::test(start_paused = true)] +async fn placed_pes_goes_out_at_its_placement() { + const RATE: u64 = 2_000_000; + let pes = pes_with(0xbd, 0x84, 0x00, &[], 0, b"placed"); + // Off the 25 ms grid, so neither slot edge lines up with it. + let placements = [1_010, 1_533, 2_017]; + let written = || placements.iter().map(|&ms| (ms, pes.clone())).collect(); + + let slices = export_placed(Placed { + mux_rate: Some(RATE), + ..Placed::new(written()) + }) + .await + .unwrap(); + let out = placed_out(&slices); + assert_eq!(out.len(), placements.len()); + let packet = 188 * 8 * 1_000_000_000 / u128::from(RATE); + for (pes, &ms) in out.iter().zip(&placements) { + let first = slice_pcr(&slices[pes.slice]) + pes.offset as u128 * 8 * 1_000_000_000 / u128::from(RATE); + let place = u128::from(ms) * 1_000_000; + assert!( + first >= place, + "{ms} ms: first byte at {first} ns, ahead of its placement" + ); + assert!( + first < place + packet, + "{ms} ms: first byte at {first} ns, a packet past its placement" + ); + } + + let slices = export_placed(Placed::new(written())).await.unwrap(); + let out = placed_out(&slices); + assert_eq!(out.len(), placements.len()); + for (pes, &ms) in out.iter().zip(&placements) { + let opens = slice_pcr(&slices[pes.slice]); + let place = u128::from(ms) * 1_000_000; + assert!(opens >= place, "{ms} ms: the slot opening at {opens} ns is ahead of it"); + assert!( + opens < place + PCR_INTERVAL.as_nanos(), + "{ms} ms: the slot opening at {opens} ns is a slot late" + ); + } +} + +/// Tune-in drops a PES placed before the keyframe departs, and keeps one placed beside it, +/// after it departs but before its PTS, as well as one placed after it among the media, +/// each in the slot of its placement. Comparing placement with the keyframe's PTS would drop +/// the one beside it. +#[tokio::test(start_paused = true)] +async fn placed_pes_survives_tune_in_beside_the_keyframe() { + // Level 3.1 without an HRD reaches a second ahead, so the keyframe at 2 s departs at 1 s. + let video: Vec = (0..25).map(|k| 2_000 + 40 * k).collect(); + let before = pes_with(0xbd, 0x84, 0x00, &[], 0, b"before"); + let beside = pes_with(0xbd, 0x84, 0x00, &[], 0, b"beside"); + let after = pes_with(0xbd, 0x84, 0x00, &[], 0, b"after"); + let slices = export_placed(Placed { + video, + ..Placed::new(vec![(500, before), (1_500, beside.clone()), (2_500, after.clone())]) + }) + .await + .unwrap(); + let out = placed_out(&slices); + let bytes: Vec<&Vec> = out.iter().map(|pes| &pes.bytes).collect(); + assert_eq!( + bytes, + vec![&beside, &after], + "only the PES before the keyframe departs is dropped" + ); + for (pes, ms) in out.iter().zip([1_500u128, 2_500]) { + assert_eq!( + slices[pes.slice].timestamp.as_micros(), + (ms * 1_000).div_ceil(PCR_INTERVAL.as_micros()) * PCR_INTERVAL.as_micros() + PCR_INTERVAL.as_micros(), + "{ms} ms: rides the first slot opening after its placement" + ); + } + // The keyframe is the first video PES out. + let keyframe = slices + .iter() + .position(|slice| { + slice.payload.as_chunks::<188>().0.iter().any(|packet| { + let start = 4 + if packet[3] & 0x20 != 0 { + usize::from(packet[4]) + 1 + } else { + 0 + }; + packet[1] & 0x40 != 0 && packet[start..].starts_with(&[0, 0, 1, 0xe0]) + }) + }) + .expect("the keyframe went out"); + assert!(keyframe < out[0].slice, "the keyframe leads the PES beside it"); +} + +/// A frame that is not exactly one well-formed PES packet fails the export. +#[tokio::test(start_paused = true)] +async fn placed_pes_refuses_a_malformed_frame() { + let mut short = pes_with(0xbd, 0x84, 0x00, &[], 0, b"payload"); + short.pop(); + let err = export_placed(Placed::new(vec![(1_000, short)])).await.unwrap_err(); + assert!(err.to_string().contains("malformed PES packet"), "{err}"); +} + +/// An entry naming both carriages, or carrying a track that is also a media rendition, has +/// two ways to write one PID, so the export refuses it. +#[tokio::test(start_paused = true)] +async fn placed_pes_refuses_an_ambiguous_entry() { + let pes = || vec![(1_000, pes_with(0xbd, 0x84, 0x00, &[], 0, b"payload"))]; + let mut both = Placed::new(pes()); + both.entry.verbatim = Some(tscat::Verbatim::new(0x06, tscat::Framing::Pes)); + let err = export_placed(both).await.unwrap_err(); + assert!(err.to_string().contains("names both"), "{err}"); + + let err = export_placed(Placed { + rendition: true, + ..Placed::new(pes()) + }) + .await + .unwrap_err(); + assert!(err.to_string().contains("media rendition"), "{err}"); +} + +/// DVB AC-3 carried whole keeps every sync frame of a PES in it, unsplit, where the legacy +/// `pes` framing cuts the PES into its frames. +#[tokio::test(start_paused = true)] +async fn placed_ac3_pes_keeps_its_sync_frames_together() { + let payload: Vec = (0..3).flat_map(ac3_frame).collect(); + let pes = pes_with(0xbd, 0x84, 0x80, &pes_clock(0b0010, 90 * 1_100), 0, &payload); + let mut placed = Placed::new(vec![(1_000, pes.clone())]); + placed.entry.descriptors = vec![tscat::Descriptor { + tag: 0x6a, + data: Bytes::from_static(&[0x00]), + }]; + let out = placed_out(&export_placed(placed).await.unwrap()); + assert_eq!(out.len(), 1); + assert_eq!(out[0].bytes, pes); +} diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index 7ee91575e51..c0de1a90b8a 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -1380,6 +1380,7 @@ fn register_verbatim( pid, descriptors, verbatim: Some(catalog::Verbatim::new(stream_type, framing)), + pes_packet: None, }, ); drop(guard); diff --git a/rs/moq-mux/src/container/ts/mod.rs b/rs/moq-mux/src/container/ts/mod.rs index b342a3f513e..3141428d9e4 100644 --- a/rs/moq-mux/src/container/ts/mod.rs +++ b/rs/moq-mux/src/container/ts/mod.rs @@ -41,7 +41,7 @@ mod si; // they read as `ts::Catalog`, `ts::Mpegts`, ... instead of stuttering under `catalog`. mod catalog; -pub use catalog::{Catalog, Descriptor, Ext, Framing, Mpegts, Program, SiEntry, Track, Verbatim}; +pub use catalog::{Catalog, Descriptor, Ext, Framing, Mpegts, PesPacket, Program, SiEntry, Track, Verbatim}; pub use export::Export; pub use follower::Follower; pub use import::*; diff --git a/rs/moq-mux/src/container/ts/schedule.rs b/rs/moq-mux/src/container/ts/schedule.rs index e66993409e6..b5ce0daf017 100644 --- a/rs/moq-mux/src/container/ts/schedule.rs +++ b/rs/moq-mux/src/container/ts/schedule.rs @@ -165,6 +165,9 @@ struct Unit { keyframe: bool, /// It was pushed after its due slot went out, so it goes out as soon as it can. late: bool, + /// For a PES the source placed, the media time in nanoseconds its first byte goes out at, + /// no earlier. It keeps no deadline of ours: it goes out as soon after as the buffers let it. + place: Option, /// Its bytes in the PID's decoder buffer, until it decodes. held: usize, /// The decode time and bytes of each frame of a PES that packs several, which leave the @@ -675,6 +678,34 @@ impl Schedule { timing: Timing, packets: Vec, keyframe: bool, + frames: Vec<(u128, usize)>, + ) -> anyhow::Result<()> { + self.queue(pid, timing, None, packets, keyframe, frames) + } + + /// Queue a PES on `pid` the source placed at `place` nanoseconds on the media timeline: + /// its first byte goes out then, never earlier, with its TS header just before it. It + /// decodes at `decode`, or each of `frames` at its own time, as [`Self::push_frames`] + /// lists them, which only the PID's decoder buffer reads: the unit misses no deadline. + pub fn push_placed( + &mut self, + pid: u16, + (place, decode): (u128, u128), + packets: Vec, + keyframe: bool, + frames: Vec<(u128, usize)>, + ) -> anyhow::Result<()> { + let timing = Timing { by: place, decode }; + self.queue(pid, timing, Some(place), packets, keyframe, frames) + } + + fn queue( + &mut self, + pid: u16, + timing: Timing, + place: Option, + packets: Vec, + keyframe: bool, mut frames: Vec<(u128, usize)>, ) -> anyhow::Result<()> { if packets.is_empty() { @@ -694,7 +725,18 @@ impl Schedule { .filter(|packet| self::pid(*packet) == pid) }; let bytes: usize = on_pid().map(|packet| payload(packet).len()).sum(); - if bytes > size { + // The PES header leaves the decoder buffer with the first frame after it. + if let Some(first) = frames.first_mut() { + first.1 += bytes - on_pid().map(|packet| elementary(packet).len()).sum::(); + } + // A placed PES's frames leave the buffer one by one as it streams in, so only each + // has to fit; any other unit arrives whole by its first frame's decode time. + if let Some(largest) = frames.iter().map(|&(_, bytes)| bytes).max().filter(|_| place.is_some()) { + anyhow::ensure!( + largest <= size, + "MPEG-TS PES on PID {pid} holds a {largest}-byte frame, exceeding its {size}-byte decoder buffer" + ); + } else if bytes > size { // Only the error names the elementary size, so the PES headers are parsed here. let headers: usize = on_pid() .filter(|packet| packet[1] & 0x40 != 0) @@ -707,10 +749,6 @@ impl Schedule { "MPEG-TS access unit on PID {pid} has {elementary} payload bytes ({bytes} bytes with PES headers), exceeding its {size}-byte decoder buffer" ); } - // The PES header leaves the decoder buffer with the first frame after it. - if let Some(first) = frames.first_mut() { - first.1 += bytes - on_pid().map(|packet| elementary(packet).len()).sum::(); - } } // The deadline its last frame would have on its own. let last = timing.by @@ -730,6 +768,7 @@ impl Schedule { sent: 0, keyframe, late: self.next.is_some_and(|next| due < next), + place, held: 0, frames, runs, @@ -766,7 +805,19 @@ impl Schedule { /// The first slot `unit` may go out in: [`ahead`] of its due slot, and never before the /// slot after the unit pushed ahead of it needs. A packed PES is due by its first frame. + /// + /// A placed PES goes no earlier than the slot whose bytes reach its placement: at a + /// multiplex rate the one whose span holds it, where [`Self::padded`] times each packet, + /// and otherwise the first that opens at or after it, as nothing times the packets + /// within an unpadded slot. Slot `index` opens one slot behind its boundary ([`grid_pcr`]). fn release(&self, unit: &Unit) -> u128 { + if let Some(place) = unit.place { + let slot = PCR_INTERVAL.as_nanos(); + return match self.rate { + Some(_) => place / slot + 1, + None => place.div_ceil(slot) + 1, + }; + } if unit.late { return 0; } @@ -816,8 +867,10 @@ impl Schedule { } let mut take = self.admit(index, usize::MAX); - for (unit, take) in self.units.iter_mut().zip(take.iter_mut()) { - if unit.due <= index && *take < unit.remaining() { + // A placed PES is due only once released, which may come after its due slot. + let released: Vec = self.units.iter().map(|unit| self.release(unit) <= index).collect(); + for ((unit, take), released) in self.units.iter_mut().zip(take.iter_mut()).zip(released) { + if unit.due <= index && released && *take < unit.remaining() { if self.buffers.get(&unit.pid).is_some_and(|buffer| buffer.size.is_some()) { unit.held += unit.packets_on(unit.sent + *take..unit.count(), unit.pid) * PAYLOAD; } @@ -979,6 +1032,18 @@ impl Schedule { let packet = opening.as_deref().unwrap_or(rest); let packet = &packet[..TsPacket::SIZE]; let pid = self::pid(packet); + // A placed PES's first byte, after its packet's header, waits for its placement. + if let Some(place) = unit.place + && pid == unit.pid + && packet[1] & 0x40 != 0 + { + let header = (TsPacket::SIZE - payload(packet).len()) as u128; + let at = place as i128 - (header * 8 * 1_000_000_000 / u128::from(rate)) as i128; + if now < at { + waiting.push(Reverse((at, lane))); + continue; + } + } let buffer = self.buffers.get(&pid); let duration = buffer.map_or(0, |buffer| { (TsPacket::SIZE as u128 * 8 * 1_000_000_000).div_ceil(u128::from(buffer.rate)) @@ -1088,7 +1153,11 @@ impl Schedule { } continue; } - if !self.ended && !unit.late && pid == unit.pid && done > unit.timing.decode as i128 { + if !self.ended + && !unit.late && unit.place.is_none() + && pid == unit.pid + && done > unit.timing.decode as i128 + { return Err(self.missed(pid, rate)); } self.transport.insert(tb, (arrived, since)); @@ -1172,11 +1241,9 @@ impl Schedule { } let end = time(count); if !self.ended - && let Some(unit) = self - .units - .iter() - .find(|unit| !unit.late && unit.timing.by as i128 <= end && unit.remaining() > 0) - { + && let Some(unit) = self.units.iter().find(|unit| { + !unit.late && unit.place.is_none() && unit.timing.by as i128 <= end && unit.remaining() > 0 + }) { return Err(self.missed(unit.pid, rate)); } for unit in self.units.iter().filter(|unit| unit.remaining() == 0) {