diff --git a/doc/bin/srt.md b/doc/bin/srt.md index 710ca43fce..aaea4bb152 100644 --- a/doc/bin/srt.md +++ b/doc/bin/srt.md @@ -10,8 +10,9 @@ source (`--connect`); `moq export srt` serves SRT to players or pushes to a remote. The payload is MPEG-TS, so the same codecs and behavior as [`import ts`](/bin/ts) apply: H.264/H.265 video and AAC, MP2, AC-3, E-AC-3, or Opus audio. A lost or flagged-corrupt packet drops only its unit and the session -stays up; video freezes until the next keyframe that arrives intact. A unit -that arrives whole but does not decode ends the session with the cause. +stays up; video freezes until the next keyframe that arrives intact. A unit, +PES header, or adaptation field that arrives whole but does not parse ends the +session with the cause. ```bash # Accept a contribution feed and publish it diff --git a/doc/bin/ts.md b/doc/bin/ts.md index 7479821cf8..d90cf5b836 100644 --- a/doc/bin/ts.md +++ b/doc/bin/ts.md @@ -200,8 +200,11 @@ Damage in the source is the importer's concern, not MoQ's. In semantic mode a lost packet, or one the receiver flagged with the transport error indicator, is dropped on its own PID, so video freezes until its next keyframe while the other tracks carry on, and feed health (stalled streams, TR 101 290 counters) -is logged and reported without changing what publishes. A media unit that -arrives whole but does not decode ends the import with the cause. +is logged and reported without changing what publishes. A media unit, PES +header, or adaptation field that arrives whole but does not parse ends the +import with the cause: with neither loss nor the flag to explain it, the muxer +wrote it wrong. A packet or PES scrambled by conditional access is refused and +counted, and its stream resumes once it is clear, video at its next keyframe. Passthrough carries damaged packets as they came. ## Evidence @@ -225,7 +228,7 @@ details each grader and its negative controls. | Whole PES (KLV, DVB AC-3) come back byte-equal outside PTS/DTS, at their placement | `just test ts --klv --strict`; [`pes_packet_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/pes_packet_test.rs) | Nightly, PR | | MPEG-2 video decodes to the source's pictures | `just test ts --mpeg2`; [`routes_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/routes_test.rs) `mpeg2_video_rides_whole_and_survives_a_round_trip` | Nightly, PR | | SCTE-35 and DSM-CC sections round-trip | [`export_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/export_test.rs) `scte35_fixtures_survive_roundtrip`; [`routes_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/routes_test.rs) `dsmcc_types_ride_as_sections` | PR | -| Refusals: PES clock fields, scrambling, ambiguous `0x86`, unclaimed PIDs | [`pes_packet_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/pes_packet_test.rs) `a_pes_with_a_field_no_rebase_reaches_refuses_its_stream`; [`routes_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/routes_test.rs) `ambiguous_streams_are_refused_and_latm_rides_whole`, `unclaimed_pids_count_once_the_pmt_is_in` | PR | +| Refusals: PES clock fields, scrambling, ambiguous `0x86`, unclaimed PIDs | [`pes_packet_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/pes_packet_test.rs) `a_pes_with_a_field_no_rebase_reaches_refuses_its_stream`; [`routes_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/routes_test.rs) `ambiguous_streams_are_refused_and_latm_rides_whole`, `unclaimed_pids_count_once_the_pmt_is_in`; [`import.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/import.rs) `scrambled_pes_is_refused_until_clear` | PR | | DVB SI survives, on its own interval, pending versions dropped | [`export_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/export_test.rs) `service_layer_survives_roundtrip`, `si_pids_are_re_emitted_on_their_own_interval`; `just test ts-eit` | PR, Nightly | | PMT revisions: streams joining and leaving, retypes, PID reuse | `just test ts --revision --strict`; [`revision_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/revision_test.rs) | Nightly, PR | | PAT changes: moved PMT, programs added and dropped | [`revision_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/revision_test.rs) `a_moved_pmt_moves_behind_a_new_pat`; [`programs.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/programs.rs) `programs_follow_the_pat` | PR | @@ -261,7 +264,7 @@ details each grader and its negative controls. | Passthrough: a missing group leaves its time out and counts one gap; a cut group loses only its tail; a late run is dropped alone | [`release_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/release_test.rs) `a_group_never_delivered_leaves_its_time_out`, `a_group_cut_short_loses_only_its_tail`, `an_object_past_its_deadline_is_a_gap_in_that_release_only` | PR | | Passthrough: an overflowing path keeps surviving packets in place through the source's last | `just test ts --passthrough --rate 7000000 --queue 250ms` | Nightly only | | Source damage drops only the damaged PID's unit | [`import.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/import.rs) `damaged_tei_recovers`, `video_drops_a_partial_access_unit_across_a_continuity_break` | PR | -| A whole unit that does not decode ends the import | [`import.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/import.rs) `undecodable_nal_ends_the_import`, `malformed_opus_pes_ends_the_import_unpublished` | PR | +| A whole unit or header that does not parse ends the import | [`import.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/import.rs) `undecodable_nal_ends_the_import`, `malformed_opus_pes_ends_the_import_unpublished`, `malformed_headers_end_the_import`, `invalid_adaptation_fields_end_the_import_on_media_and_clock_pids` | PR | ### Not claimed diff --git a/quest/m1/ts-packet-abort.md b/quest/m1/ts-packet-abort.md deleted file mode 100644 index 950762484c..0000000000 --- a/quest/m1/ts-packet-abort.md +++ /dev/null @@ -1,40 +0,0 @@ -# [S] A TS import ends on a packet header it cannot parse - -## Goal - -moq-mux's TS import ends with the cause on a TS adaptation field or PES header -that arrives intact but does not parse, instead of counting it as `damaged` -and resuming at the next PES. Loss (a continuity break) and packets the -receiver flagged with the transport error indicator keep today's handling: -both are understood source behavior, detected where they happen. - -## Plan - -Found by [moq-dev/moq#9170](https://github.com/moq-dev/moq/pull/9170), which -made an undecodable media unit end the import and left these packet-level -paths alone because its quest's Goal named only `Fault` and `unit_error`. -The same rule applies (maintainer, 2026-11-18): "We should not try to support -random bit flips that we don't understand and limp along with broken -functionality." Over UDP and SRT a packet arrives whole or not at all, and RF -corruption sets the transport error indicator, so a malformed header with -neither is a muxer bug or an unexplained flip. - -In `rs/moq-mux/src/container/ts/import.rs`, `Import::damage` is reached for -`malformed TS adaptation field` (`route` and `pes_packet`), a `PesStart` -parse or length failure (`pes_packet`, `pes_header`), and `scrambled PES is -not supported` (`handle_pes_start`). Recommended: the first two end the -import; truncation by loss, a header cut short at a PMT revision or end of -input, and the transport error indicator stay as damage. Scrambled PES is a -known source (conditional access) and likely belongs with `refuse` rather -than `damage`; decide it here with a test that names the source. - -Tests: `damaged_video_recovers`' `pes`, `adaptation`, and `adaptation-clock` -cases, `invalid_adaptation_lengths_damage_media_and_dedicated_clock_pids`, -`damaged_adaptation_on_a_dedicated_pcr_pid_keeps_the_clock`, -`damaged_pes_start_flushes_the_unit_before_it`, and -`malformed_pes_prefix_is_refused_without_waiting_for_more_bytes` become -"ends with the cause". `doc/bin/ts.md` and `doc/bin/srt.md` already say a -whole unit that does not decode ends the import; extend them to headers. - -Public API: none. Behavior: a TS ingest with a malformed packet header ends -instead of continuing. Wire: none. diff --git a/rs/moq-mux/src/container/ts/error.rs b/rs/moq-mux/src/container/ts/error.rs index 2bc36e7d10..0f9a834b6c 100644 --- a/rs/moq-mux/src/container/ts/error.rs +++ b/rs/moq-mux/src/container/ts/error.rs @@ -1,7 +1,8 @@ /// Why a TS import or export stopped. /// -/// A damaged packet, PES, or access unit is not one: the importer counts it in its -/// [`stats`](super::stats) and carries on. +/// A packet lost or flagged by the transport error indicator is not one: the importer counts +/// the unit it breaks in its [`stats`](super::stats) and carries on. An intact packet, header, +/// or unit that does not parse is one: [`Error::Malformed`]. #[derive(Clone, Debug, thiserror::Error)] #[non_exhaustive] pub enum Error { diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index 5507336b7c..2693e93878 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -69,7 +69,7 @@ mod video; /// /// Every PID is accounted for. A listed stream that is not carried counts once in /// [`stats::Stream::refused`] with a warning, and so does each TS-scrambled packet on a carried -/// one. Once the PMT is in, a packet on a PID no table claims (the null PID aside) counts in its +/// one and each PES-scrambled PES on a decoded one. Once the PMT is in, a packet on a PID no table claims (the null PID aside) counts in its /// own row: see [`stats::Snapshot::streams`]. That holds when the PAT lists only the imported /// program; a [`with_program`](Self::with_program) import of a multiplex cannot tell another /// program's PIDs from unclaimed ones, so it leaves them to [`Programs`](super::Programs). @@ -606,18 +606,19 @@ impl Import { if matches!(continuation, Some(Continuation::Duplicate)) { return Ok(()); } - // A media or PCR packet whose adaptation field overruns itself is refused before - // its clock bits can reach the program clock. Its counter already joined the - // chain, so a gap in front of it still salvages what came before. + // A media or PCR packet whose adaptation field overruns itself ends the import before + // its clock bits can reach the program clock: it arrived whole and unflagged, so the + // muxer wrote it wrong. if let Ok(pid) = Pid::new(pid) - && (self.streams.contains_key(&pid) || self.pcr_pid == Some(pid)) + && (self + .streams + .get(&pid) + .is_some_and(|stream| !matches!(stream, Stream::Ignored)) + || self.pcr_pid == Some(pid)) && pkt[1] & 0x80 == 0 && !adaptation_valid(pkt) { - if matches!(continuation, Some(Continuation::Broken)) { - self.salvage(pid)?; - } - return self.damage(pid, &malformed!("malformed TS adaptation field")); + return Err(malformed!("malformed TS adaptation field on PID {}", pid.as_u16())); } // Read the clock's own flag before routing, so the break lands between the media // either side of it. The PCR PID can be a dedicated one nothing below routes, and @@ -917,30 +918,26 @@ impl Import { let payload = match payload(pkt) { Payload::None => return Ok(()), Payload::Bytes(payload) => payload, - // No payload can be found in it, so like a corrupt packet it breaks the PES. Payload::Malformed => { - return self.damage(pid, &malformed!("malformed TS adaptation field")); + return Err(malformed!("malformed TS adaptation field on PID {}", pid.as_u16())); } }; if pkt[1] & 0x40 != 0 { - // The new start ends the preceding PES even while its own header is incomplete. - self.flush(pid)?; - if self.headers.remove(&pid).is_some() { - self.damage(pid, &malformed!("PES header truncated at next start"))?; + // Loss, a flagged packet, and scrambling all drop a partial header, so one still here + // is a header the muxer cut short. + if self.headers.contains_key(&pid) { + return Err(malformed!( + "PES header on PID {} truncated at the next start", + pid.as_u16() + )); } + self.flush(pid)?; // The payload ends the packet, so this is where its start code sits in the input. let at = offset + (TsPacket::SIZE - payload.len()) as u64; // Taken here, not once the header is whole, so a PCR landing before the header's // continuation is the one after it. let span = self.placement.span(at); - return match PesStart::header_len(payload) { - Ok(len) if payload.len() >= len => match PesStart::parse(payload) { - Ok(pes) => self.handle_pes_start(pid, pes, span), - Err(err) => self.damage(pid, &err), - }, - Ok(_) => self.pes_header(pid, span, payload.to_vec()), - Err(err) => self.damage(pid, &err), - }; + return self.pes_header(pid, span, payload.to_vec()); } if let Some((span, mut header)) = self.headers.remove(&pid) { header.extend_from_slice(payload); @@ -950,20 +947,18 @@ impl Import { } } - /// Continue a PES header split across packets, placed by `span` from its start code. + /// Open the PES whose header `header` begins, placed by `span` from its start code, or hold + /// the header until the packets after it complete it. A header that arrived intact but does + /// not parse ends the import. fn pes_header(&mut self, pid: Pid, span: Option, header: Vec) -> Result<()> { - let len = match PesStart::header_len(&header) { - Ok(len) => len, - Err(err) => return self.damage(pid, &err), - }; + let malformed = |err| malformed!("PES header on PID {}: {err}", pid.as_u16()); + let len = PesStart::header_len(&header).map_err(malformed)?; if header.len() < len { self.headers.insert(pid, (span, header)); return Ok(()); } - match PesStart::parse(&header) { - Ok(pes) => self.handle_pes_start(pid, pes, span), - Err(err) => self.damage(pid, &err), - } + let pes = PesStart::parse(&header).map_err(malformed)?; + self.handle_pes_start(pid, pes, span) } fn ensure_stream(&mut self, pid: Pid, stream_type: u8, descriptors: &[catalog::Descriptor]) -> Result<()> { @@ -1307,6 +1302,10 @@ impl Import { return Ok(()); }; + // Conditional access, like TS-level scrambling: refused while it lasts, on any carriage. + if pes.scrambled() { + return self.scrambled(pid); + } let whole = match stream { Stream::PesPacket(stream) => Some(stream.stream_type), _ => None, @@ -1316,9 +1315,6 @@ impl Import { { return self.refuse(pid, &why); } - if pes.scrambled() { - return self.damage(pid, &malformed!("scrambled PES is not supported")); - } let stream = &self.streams[&pid]; let is_video = stream.video(); @@ -1424,8 +1420,9 @@ impl Import { Ok(()) } - /// Refuse the PES a damaged packet breaks (loss, the transport error indicator, or a - /// malformed TS or PES header) without changing any other PID or the program clock. + /// Refuse the PES a damaged packet breaks (loss, the transport error indicator, or a header + /// cut short by either, a PMT revision, or the end of input) without changing any other PID + /// or the program clock. fn damage(&mut self, pid: Pid, err: &Error) -> Result<()> { // Complete AUs ahead of the damaged packet remain valid, including those // in an unbounded PES. Its final NAL/AU is never flushed at a loss boundary. @@ -1588,8 +1585,8 @@ impl Import { Ok(()) } - /// A TS-scrambled packet on the carried `pid`: refused, and the unit it interrupts lost with - /// it. The route stays, so the stream resumes once its packets are clear again. + /// A TS-scrambled packet or PES-scrambled PES on the carried `pid`: refused, and the unit it + /// interrupts lost with it. The route stays, so the stream resumes once it is clear again. fn scrambled(&mut self, pid: Pid) -> Result<()> { self.pending.remove(&pid); self.headers.remove(&pid); @@ -1605,7 +1602,7 @@ impl Import { if self.scrambled.insert(pid.as_u16()) { tracing::warn!( pid = pid.as_u16(), - "refusing TS-scrambled packets; carry the multiplex whole to keep them" + "refusing scrambled payloads; carry the multiplex whole to keep them" ); } Ok(()) @@ -2657,13 +2654,11 @@ impl PesPacketStream { } /// Why a whole-PES track cannot carry `pes` unaltered, if it cannot: a header export refuses -/// ([`PesStart::whole`]), a clock field only the PTS and DTS of are rebased, or a scrambled -/// payload. +/// ([`PesStart::whole`]), or a clock field only the PTS and DTS of are rebased. fn refusal(pes: &PesStart, stream_type: u8) -> Option { match pes.whole(stream_type) { Err(err) => Some(format!("{err:#}")), - Ok(Some(clock)) => Some(clock.to_string()), - Ok(None) => pes.scrambled().then(|| "PES-level scrambling".to_string()), + Ok(clock) => clock.map(|clock| clock.to_string()), } } @@ -8844,8 +8839,10 @@ pub(super) mod test { import } + /// A header cut short by loss or the end of input is dropped; one the muxer cut short with + /// the next start ends the import. #[test] - fn pes_fragmented_header_resets_on_loss_new_start_and_eof() { + fn pes_fragmented_header_resets_on_loss_and_eof_and_ends_at_a_new_start() { let pid = Pid::new(0x50).unwrap(); let header = [0, 0, 1, 0xe0, 0, 0, 0x80, 0x80, 5]; for end in ["loss", "start", "eof"] { @@ -8856,27 +8853,27 @@ pub(super) mod test { assert_eq!(import.headers[&pid].1.len(), 9); match end { "loss" => import.broken(pid).unwrap(), - "start" => import - .pes_packet( - pid, - &pes_fragment(0x50, true, 1, &[0, 0, 1, 0xe0, 0, 0, 0x80, 0, 0, 0xaa]), - 0, - ) - .unwrap(), + "start" => { + let err = import + .pes_packet( + pid, + &pes_fragment(0x50, true, 1, &[0, 0, 1, 0xe0, 0, 0, 0x80, 0, 0, 0xaa]), + 0, + ) + .expect_err("an intact header cut short is the muxer's"); + assert!(err.to_string().contains("truncated at the next start"), "{err}"); + continue; + } _ => import.finish_in_place().unwrap(), } assert!(!import.headers.contains_key(&pid)); - if end == "start" { - assert_eq!(import.pending[&pid].data, [0xaa]); - } else { - import - .pes_packet(pid, &pes_fragment(0x50, false, 1, &[0xff; 5]), 0) - .unwrap(); - assert!( - !import.pending.contains_key(&pid), - "orphan bytes cannot complete the lost header" - ); - } + import + .pes_packet(pid, &pes_fragment(0x50, false, 1, &[0xff; 5]), 0) + .unwrap(); + assert!( + !import.pending.contains_key(&pid), + "orphan bytes cannot complete the lost header" + ); } } @@ -9041,13 +9038,15 @@ pub(super) mod test { } #[test] - fn malformed_pes_prefix_is_refused_without_waiting_for_more_bytes() { + fn malformed_pes_prefix_ends_the_import_without_waiting_for_more_bytes() { let pid = Pid::new(0x50).unwrap(); for bad in [&[1][..], &[0, 1][..], &[0, 0, 2][..]] { let mut import = ignored_pes_import(); - import.pes_packet(pid, &pes_fragment(0x50, true, 0, bad), 0).unwrap(); + let err = import + .pes_packet(pid, &pes_fragment(0x50, true, 0, bad), 0) + .expect_err("a malformed prefix"); + assert!(err.to_string().contains("start code prefix"), "{err}"); assert!(import.headers.is_empty()); - assert!(import.pending.is_empty()); } } @@ -9055,15 +9054,15 @@ pub(super) mod test { fn pes_length_overruns_fail_before_waiting_for_the_optional_header() { let pid = Pid::new(0x50).unwrap(); let mut import = ignored_pes_import(); - import + let err = import .pes_packet( pid, &pes_fragment(0x50, true, 0, &[0, 0, 1, 0xe0, 0, 4, 0x80, 0x80, 255]), 0, ) - .unwrap(); + .expect_err("a PES length shorter than its header"); + assert!(err.to_string().contains("shorter than its header"), "{err}"); assert!(import.headers.is_empty()); - assert!(import.pending.is_empty()); } #[tokio::test(start_paused = true)] @@ -9677,16 +9676,16 @@ pub(super) mod test { ); } - /// Damage on one PID refuses that unit and its dependent pictures while another PID - /// carries on. A later decode call resumes from the next keyframe. - async fn damaged_video_recovers(damage: &str) { + /// The fault `damage` names hits one packet of a later video PES on one PID while another PID + /// carries on, returning the mux and the offsets of the damaged packet and of the GOP after. + fn damaged_video(damage: &str) -> (Mux, usize, usize) { let peer = VIDEO + 1; let mut mux = Mux { out: synth_pmt(&[(StreamType::H264, VIDEO), (StreamType::H264, peer)], false), ..Default::default() }; mux.gop(VIDEO, 90_000); - if matches!(damage, "pes" | "adaptation" | "adaptation-clock" | "tei") { + if damage != "clean" { // These faults hide the next PES entirely. An AUD received ahead of // the fault proves the preceding picture complete; the AUD's own // picture remains unconfirmed and is discarded at the loss. @@ -9709,6 +9708,8 @@ pub(super) mod test { packet[5] = 0x90; } "tei" => packet[1] |= 0x80, + // PES_scrambling_control, as conditional access sets it. + "scrambled" => packet[payload_at + 6] |= 0x10, "clean" => {} _ => unreachable!(), } @@ -9720,7 +9721,16 @@ pub(super) mod test { let resume_at = mux.out.len(); mux.gop(VIDEO, 90_000 + 8 * FRAME); mux.gop(peer, 90_000 + 8 * FRAME); + (mux, damaged_at, resume_at) + } + /// A packet the receiver flagged corrupt (`tei`) loses its unit and the dependent pictures + /// while another PID carries on, and so does a PES conditional access scrambled + /// (`scrambled`), refused rather than damaged. A later decode call resumes from the next + /// keyframe. + async fn damaged_video_recovers(damage: &str) { + let peer = VIDEO + 1; + let (mux, _, resume_at) = damaged_video(damage); let mut broadcast = moq_net::broadcast::Info::new().produce(); let consumer = broadcast.consume(); let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); @@ -9737,7 +9747,11 @@ pub(super) mod test { .decode(&mux.out[resume_at..]) .expect("next keyframe must recover"); import.finish_in_place().unwrap(); - assert_eq!(import.stats().streams[&VIDEO].damaged, u64::from(damage != "clean")); + let row = &import.stats().streams[&VIDEO]; + assert_eq!( + (row.damaged, row.refused), + (u64::from(damage == "tei"), u64::from(damage == "scrambled")) + ); assert_eq!(import.stats().streams[&peer].damaged, 0); for (pid, expected) in [(VIDEO, if damage == "clean" { 12 } else { 8 }), (peer, 12)] { let Stream::H264 { import: video, .. } = &import.streams[&Pid::new(pid).unwrap()] else { @@ -9755,9 +9769,33 @@ pub(super) mod test { } } + /// An intact, unflagged packet whose PES header or adaptation field does not parse ends the + /// import with the cause, after the pictures ahead of it, and counts no damage. The clock + /// bits a malformed field announces never reach the program clock. #[tokio::test(start_paused = true)] - async fn damaged_pes_header_recovers() { - damaged_video_recovers("pes").await; + async fn malformed_headers_end_the_import() { + for (damage, cause) in [ + ("pes", "PES header"), + ("adaptation", "adaptation field"), + ("adaptation-clock", "adaptation field"), + ] { + let (mux, damaged_at, _) = damaged_video(damage); + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let mut import = super::Import::new(broadcast, catalog.reserve()); + let before = import.decode(&mux.out[..damaged_at]).map(|_| import.last_pts); + let err = import.decode(&mux.out[damaged_at..]).expect_err(damage); + assert!(err.to_string().contains(cause), "{damage}: {err}"); + assert_eq!(import.last_pts, before.unwrap(), "{damage} moved the clock"); + let stats = import.stats(); + assert!(stats.streams.values().all(|stream| stream.damaged == 0), "{damage}"); + let Stream::H264 { import: video, .. } = &import.streams[&Pid::new(VIDEO).unwrap()] else { + panic!("video route"); + }; + let frames = read_track(&consumer, video.name(), crate::container::Kind::Video).await; + assert_eq!(frames.len(), 4, "{damage}: the pictures an AUD proved whole"); + } } /// PES boundaries need not fall on NALs, so the first PES after a continuity break can open @@ -9841,18 +9879,13 @@ pub(super) mod test { } #[tokio::test(start_paused = true)] - async fn damaged_adaptation_recovers() { - damaged_video_recovers("adaptation").await; - } - - #[tokio::test(start_paused = true)] - async fn damaged_adaptation_cannot_reset_the_program_clock() { - damaged_video_recovers("adaptation-clock").await; + async fn damaged_tei_recovers() { + damaged_video_recovers("tei").await; } #[tokio::test(start_paused = true)] - async fn damaged_tei_recovers() { - damaged_video_recovers("tei").await; + async fn scrambled_pes_is_refused_until_clear() { + damaged_video_recovers("scrambled").await; } #[tokio::test(start_paused = true)] @@ -9935,11 +9968,17 @@ pub(super) mod test { } } + /// A malformed adaptation field on a carried PID or the clock's ends the import before its + /// clock bits are read; one on a PID the import does not carry is not read at all. #[test] - fn invalid_adaptation_lengths_damage_media_and_dedicated_clock_pids() { + fn invalid_adaptation_fields_end_the_import_on_media_and_clock_pids() { + // Clear of the PMT PID and every stream, so only the clock reads it. const PCR: u16 = 0x0200; - for pid in [VIDEO, PCR] { - for (control, length) in [(0b10, 0), (0b10, 182), (0b11, 183)] { + const IGNORED: u16 = 0x0300; + // Lengths the packet control forbids, and a PCR flag on a field too short to hold it. + let faults: [(u8, u8, &[u8]); 4] = [(0b10, 0, &[]), (0b10, 182, &[]), (0b11, 183, &[]), (0b10, 1, &[0x90])]; + for pid in [VIDEO, PCR, IGNORED] { + for (control, length, field) in faults { let mut mux = Mux { out: synth_pmt(&[(StreamType::H264, VIDEO)], false), ..Default::default() @@ -9950,55 +9989,34 @@ pub(super) mod test { let mut import = super::Import::new(broadcast, catalog.reserve()); import.decode(&mux.out).unwrap(); import.pcr_pid = Some(Pid::new(PCR).unwrap()); + import.streams.insert(Pid::new(IGNORED).unwrap(), Stream::Ignored); + let last_pts = import.last_pts; + assert!(last_pts.is_some()); let mut packet = clock_break_packet(pid); packet[3] = control << 4; packet[4] = length; - import.decode(&packet).unwrap(); - assert!(import.last_pts.is_some(), "invalid adaptation reset the clock"); - assert_eq!( - import.stats().streams[&pid].damaged, - 1, - "pid={pid} control={control:b} length={length}" + packet[5..5 + field.len()].copy_from_slice(field); + let case = format!("pid={pid} control={control:b} length={length}"); + if pid == IGNORED { + import.decode(&packet).expect(&case); + } else { + let err = import.decode(&packet).expect_err(&case); + assert!(err.to_string().contains("adaptation field"), "{case}: {err}"); + } + assert_eq!(import.last_pts, last_pts, "{case} moved the clock"); + assert!( + import.stats().streams.values().all(|stream| stream.damaged == 0), + "{case}" ); } } } - /// A dedicated PCR PID's malformed adaptation field cannot declare a timebase break. - #[test] - fn damaged_adaptation_on_a_dedicated_pcr_pid_keeps_the_clock() { - // Clear of the PMT PID and every stream, so only the clock-only row can carry it. - const PCR: u16 = 0x0200; - let mut mux = Mux { - out: synth_pmt(&[(StreamType::H264, VIDEO)], false), - ..Default::default() - }; - mux.gop(VIDEO, 90_000); - let mut broadcast = moq_net::broadcast::Info::new().produce(); - let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); - let mut import = super::Import::new(broadcast, catalog.reserve()); - import.decode(&mux.out).unwrap(); - import.pcr_pid = Some(Pid::new(PCR).unwrap()); - assert!(import.last_pts.is_some()); - - // The flag byte is the field's only byte, so the PCR it announces overruns it. - let mut packet = clock_break_packet(PCR); - packet[4] = 1; - packet[5] = 0x90; - import.decode(&packet).unwrap(); - assert!(import.last_pts.is_some(), "a malformed field reset the program clock"); - let stats = import.stats(); - assert_eq!( - (stats.streams[&PCR].track.as_str(), stats.streams[&PCR].damaged), - ("", 1) - ); - } - - /// A damaged PES start does not revoke the AU a codec boundary already - /// completed in the preceding unbounded PES. + /// A malformed PES start ends the import after the AU a codec boundary already completed in + /// the preceding unbounded PES published. #[test] - fn damaged_pes_start_flushes_the_unit_before_it() { + fn malformed_pes_start_ends_the_import_after_the_unit_before_it() { let mut broadcast = moq_net::broadcast::Info::new().produce(); let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); let mut import = super::Import::new(broadcast, catalog.reserve()); @@ -10018,13 +10036,15 @@ pub(super) mod test { let start = 5 + usize::from(damaged[4]); damaged[start + 6..start + 19].fill(0); data.extend(damaged); - import.decode(&data).unwrap(); - import.finish_in_place().unwrap(); - assert_eq!(import.stats().streams[&VIDEO].damaged, 1); + let err = import + .decode(&data) + .expect_err("a malformed PES header ends the import"); + assert!(err.to_string().contains("PES header on PID"), "{err}"); + assert_eq!(import.stats().streams[&VIDEO].damaged, 0); assert_eq!( import.stats().streams[&VIDEO].units, 1, - "the keyframe before the damage" + "the keyframe before the malformed header" ); } diff --git a/rs/moq-mux/src/container/ts/pes_packet_test.rs b/rs/moq-mux/src/container/ts/pes_packet_test.rs index 194bbc326e..874efd37ae 100644 --- a/rs/moq-mux/src/container/ts/pes_packet_test.rs +++ b/rs/moq-mux/src/container/ts/pes_packet_test.rs @@ -520,7 +520,7 @@ async fn dvb_ac3_keeps_its_sync_frames_in_one_pes() { /// and the refusal is counted. #[tokio::test(start_paused = true)] async fn a_pes_with_a_field_no_rebase_reaches_refuses_its_stream() { - let cases: [(&str, Vec); 8] = [ + let cases: [(&str, Vec); 7] = [ // PES_extension_flag set with no header byte left for the extension. ("truncated extension", pes_with(0xbd, 0x80, 0x01, &[], 0, b"ext")), // ES_rate flagged with one of its three bytes inside PES_header_data_length. @@ -546,7 +546,6 @@ async fn a_pes_with_a_field_no_rebase_reaches_refuses_its_stream() { b"tref", ), ), - ("scrambling", pes_with(0xbd, 0x90, 0x00, &[], 0, b"scrambled")), ("unbounded", { let mut pes = pes_with(0xbd, 0x80, 0x00, &[], 0, b"unbounded"); pes[4..6].fill(0); @@ -586,6 +585,29 @@ async fn a_pes_with_a_field_no_rebase_reaches_refuses_its_stream() { } } +/// A PES-scrambled PES is refused alone, as a TS-scrambled packet is: the track stays, and the +/// clear PES after it publishes. +#[tokio::test(start_paused = true)] +async fn a_scrambled_pes_is_refused_until_clear() { + let mut feed = Feed::new(0x06, &[]); + let start = 1_000 * MS; + feed.pcr(start).picture(start).data(klv(1, 8)).pcr(start + 40 * MS); + feed.picture(start + 40 * MS) + .data(pes_with(0xbd, 0x90, 0x00, &[], 0, b"scrambled")) + .pcr(start + 80 * MS) + .picture(start + 80 * MS) + .data(klv(2, 8)) + .pcr(start + 120 * MS); + + let imported = import(&feed.bytes(), false).unwrap(); + let (name, _) = imported.entry().expect("the track stays in the catalog"); + let (frames, err) = read(&imported.consumer, &name).await; + let got: Vec<_> = frames.iter().map(|frame| frame.payload.to_vec()).collect(); + assert_eq!(got, [feed.pes[0].1.clone(), feed.pes[2].1.clone()]); + assert!(err.is_none(), "the track ends cleanly"); + assert_eq!(imported.stats().refused, 1); +} + /// Begin importing `head` onto a clock the first PTS places. fn start_import(head: &[u8]) -> Imported { let mut imported = start(false); diff --git a/rs/moq-mux/src/container/ts/programs.rs b/rs/moq-mux/src/container/ts/programs.rs index 824848854b..b795d912e1 100644 --- a/rs/moq-mux/src/container/ts/programs.rs +++ b/rs/moq-mux/src/container/ts/programs.rs @@ -2231,7 +2231,8 @@ mod test { assert_eq!(programs.stats().crc_error, 1); // A waiting program's PMT PID that a started program reads only as media: no program - // reassembles PMTs there, so the scan counts the corrupt section. + // reassembles PMTs there, so the scan counts the corrupt section, before the started + // program ends on reading a section as a PES header. let origin = crate::source::produce_origin(); let mut programs = Programs::new( origin, @@ -2241,7 +2242,8 @@ mod test { let mut data = psi_packets(0, &mut 0, &[], &pat_section(&[(1, 0x0100), (2, 0x0061)])); data.extend(psi_packets(0x0100, &mut 0, &[], &pmt(1))); data.extend(psi_packets(0x0061, &mut 0, &[], &corrupt(pmt(2)))); - programs.decode(&data).unwrap(); + let err = programs.decode(&data).expect_err("a section is no PES"); + assert!(err.to_string().contains("PES header on PID 97"), "{err}"); assert_eq!(programs.programs.len(), 1); assert_eq!(programs.stats().crc_error, 1); } diff --git a/rs/moq-mux/src/container/ts/stats.rs b/rs/moq-mux/src/container/ts/stats.rs index 91921d8854..dc7df401ee 100644 --- a/rs/moq-mux/src/container/ts/stats.rs +++ b/rs/moq-mux/src/container/ts/stats.rs @@ -181,9 +181,10 @@ pub struct Stream { /// registration, an Opus configuration it cannot map, any undecoded stream without the /// `mpegts` catalog section or in a program with no PCR to place it by), or a whole-PES /// stream with a clock field only the PTS and DTS of are rebased (ESCR, a - /// `pack_header_field`, a TREF), PES-level scrambling, an unbounded non-video PES, or a - /// header export refuses, whose track ends and leaves the catalog. Once for each - /// TS-scrambled packet on a carried stream, which the raw lane could carry. And once for + /// `pack_header_field`, a TREF), an unbounded non-video PES, or a header export refuses, + /// whose track ends and leaves the catalog. Once for each TS-scrambled packet on a carried + /// stream, which the raw lane could carry, and each PES-scrambled PES on one; the stream + /// resumes once it is clear. And once for /// each packet on a PID no table claims ([`Snapshot::streams`]). Export stays zero. #[serde(skip_serializing_if = "is_zero")] pub refused: u64, diff --git a/rs/moq-mux/src/container/ts/step.rs b/rs/moq-mux/src/container/ts/step.rs index edb6236a5a..fe14196468 100644 --- a/rs/moq-mux/src/container/ts/step.rs +++ b/rs/moq-mux/src/container/ts/step.rs @@ -238,7 +238,7 @@ impl Headers { return (Some(chain), None); } } - // The import refuses a malformed adaptation field and cannot read a scrambled payload. + // The import ends on a malformed adaptation field and cannot read a scrambled payload. if !adaptation_valid(pkt) || pkt[3] & 0xc0 != 0 { chain.begun = None; return (Some(chain), None); @@ -246,7 +246,7 @@ impl Headers { let Payload::Bytes(payload) = payload(pkt) else { return (Some(chain), None); }; - // A PES start abandons whatever header was begun before it, as the import's does. + // A PES start abandons whatever header was begun before it, where the import ends. let begun = chain.begun.take(); let header = if pkt[1] & 0x40 != 0 { payload.to_vec()