diff --git a/doc/bin/srt.md b/doc/bin/srt.md index 1e75aa8b79f..710ca43fcec 100644 --- a/doc/bin/srt.md +++ b/doc/bin/srt.md @@ -9,8 +9,9 @@ description: SRT contribution and playback source (`--connect`); `moq export srt` serves SRT to players or pushes to a remote. The payload is MPEG-TS, so the same codecs and behavior as [`import ts`](/bin/ts) apply: H.264/H.265 video and AAC, MP2, AC-3, E-AC-3, or -Opus audio. A damaged unit is dropped and the session stays up; video resumes at its -next keyframe, freezing for up to one GOP. +Opus audio. A lost or flagged-corrupt packet drops only its unit and the session +stays up; video freezes until the next keyframe that arrives intact. A unit +that arrives whole but does not decode ends the session with the cause. ```bash # Accept a contribution feed and publish it diff --git a/doc/bin/ts.md b/doc/bin/ts.md index 1d0357d5e8a..7479821cf83 100644 --- a/doc/bin/ts.md +++ b/doc/bin/ts.md @@ -197,9 +197,11 @@ what the network drops, so random packet loss alone loses no content. the output decodes, or meets the T-STD, across the gap. Damage in the source is the importer's concern, not MoQ's. In semantic mode a -damaged packet is dropped on its own PID, so video freezes until its next -keyframe while the other tracks carry on, and feed health (stalled streams, -TR 101 290 counters) is logged and reported without changing what publishes. +lost packet, or one the receiver flagged with the transport error indicator, is +dropped on its own PID, so video freezes until its next keyframe while the +other tracks carry on, and feed health (stalled streams, TR 101 290 counters) +is logged and reported without changing what publishes. A media unit that +arrives whole but does not decode ends the import with the cause. Passthrough carries damaged packets as they came. ## Evidence @@ -258,7 +260,8 @@ details each grader and its negative controls. | Semantic: a congested and an overflowing path shed whole groups, stay T-STD compliant, keep their latency | `just test ts --headroom --rate 7000000`, with `--queue 250ms` | Nightly only | | Passthrough: a missing group leaves its time out and counts one gap; a cut group loses only its tail; a late run is dropped alone | [`release_test.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/release_test.rs) `a_group_never_delivered_leaves_its_time_out`, `a_group_cut_short_loses_only_its_tail`, `an_object_past_its_deadline_is_a_gap_in_that_release_only` | PR | | Passthrough: an overflowing path keeps surviving packets in place through the source's last | `just test ts --passthrough --rate 7000000 --queue 250ms` | Nightly only | -| Source damage drops only the damaged PID's unit | [`import.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/import.rs) `damaged_h264_nal_recovers`, `video_drops_a_partial_access_unit_across_a_continuity_break` | PR | +| Source damage drops only the damaged PID's unit | [`import.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/import.rs) `damaged_tei_recovers`, `video_drops_a_partial_access_unit_across_a_continuity_break` | PR | +| A whole unit that does not decode ends the import | [`import.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-mux/src/container/ts/import.rs) `undecodable_nal_ends_the_import`, `malformed_opus_pes_ends_the_import_unpublished` | PR | ### Not claimed diff --git a/quest/m1/ts-packet-abort.md b/quest/m1/ts-packet-abort.md new file mode 100644 index 00000000000..950762484c1 --- /dev/null +++ b/quest/m1/ts-packet-abort.md @@ -0,0 +1,40 @@ +# [S] A TS import ends on a packet header it cannot parse + +## Goal + +moq-mux's TS import ends with the cause on a TS adaptation field or PES header +that arrives intact but does not parse, instead of counting it as `damaged` +and resuming at the next PES. Loss (a continuity break) and packets the +receiver flagged with the transport error indicator keep today's handling: +both are understood source behavior, detected where they happen. + +## Plan + +Found by [moq-dev/moq#9170](https://github.com/moq-dev/moq/pull/9170), which +made an undecodable media unit end the import and left these packet-level +paths alone because its quest's Goal named only `Fault` and `unit_error`. +The same rule applies (maintainer, 2026-11-18): "We should not try to support +random bit flips that we don't understand and limp along with broken +functionality." Over UDP and SRT a packet arrives whole or not at all, and RF +corruption sets the transport error indicator, so a malformed header with +neither is a muxer bug or an unexplained flip. + +In `rs/moq-mux/src/container/ts/import.rs`, `Import::damage` is reached for +`malformed TS adaptation field` (`route` and `pes_packet`), a `PesStart` +parse or length failure (`pes_packet`, `pes_header`), and `scrambled PES is +not supported` (`handle_pes_start`). Recommended: the first two end the +import; truncation by loss, a header cut short at a PMT revision or end of +input, and the transport error indicator stay as damage. Scrambled PES is a +known source (conditional access) and likely belongs with `refuse` rather +than `damage`; decide it here with a test that names the source. + +Tests: `damaged_video_recovers`' `pes`, `adaptation`, and `adaptation-clock` +cases, `invalid_adaptation_lengths_damage_media_and_dedicated_clock_pids`, +`damaged_adaptation_on_a_dedicated_pcr_pid_keeps_the_clock`, +`damaged_pes_start_flushes_the_unit_before_it`, and +`malformed_pes_prefix_is_refused_without_waiting_for_more_bytes` become +"ends with the cause". `doc/bin/ts.md` and `doc/bin/srt.md` already say a +whole unit that does not decode ends the import; extend them to headers. + +Public API: none. Behavior: a TS ingest with a malformed packet header ends +instead of continuing. Wire: none. diff --git a/quest/m1/ts-step-damage.md b/quest/m1/ts-step-damage.md index cd6ad78999b..d2cbd998fb7 100644 --- a/quest/m1/ts-step-damage.md +++ b/quest/m1/ts-step-damage.md @@ -41,9 +41,9 @@ Superseded 2026-11-18 (maintainer, /quest-plan of the enum-return audit): flips that we don't understand and limp along with broken functionality." This replaces the 2026-11-17 design above that drops and counts a lone stream's stray packets as damage (rejected now: tolerating an unexplained -step). [TS unit abort](/quest/m1/ts-unit-abort.md) removes the import's -damage tolerance; this quest adds back only a step with a known source, as an -explicit case. If no real source turns up, delete this quest and record why. +step). [moq-dev/moq#9170](https://github.com/moq-dev/moq/pull/9170) removed +the import's unit damage tolerance; this quest adds back only a step with a +known source, as an explicit case. If no real source turns up, delete this quest and record why. Tests: a capture or generator reproducing the step's real source, and the import's behavior on it; for unexplained corruption, the import ends with the @@ -56,10 +56,6 @@ Caused by, for the fix PR's `## Caused by`: - [moq-dev/moq#7898](https://github.com/moq-dev/moq/pull/7898) - made a lone stream's rewind fatal to the import - [moq-dev/moq#5967](https://github.com/moq-dev/moq/pull/5967) - PCR unwrap that can go negative on a data-only program -## Required - -- [TS unit abort](/quest/m1/ts-unit-abort.md) - removes the import's damage tolerance, so a known step is added back as an explicit case - ## Related - [moq-dev/moq#9035](https://github.com/moq-dev/moq/pull/9035) - plans `quest/m2/ts-passthrough-splice.md`, which handles a whole-source PCR step on the passthrough lane; this quest is the lone-stream step on the import diff --git a/quest/m1/ts-unit-abort.md b/quest/m1/ts-unit-abort.md deleted file mode 100644 index 54bb1ab1d5c..00000000000 --- a/quest/m1/ts-unit-abort.md +++ /dev/null @@ -1,35 +0,0 @@ -# [M] A TS import ends on a unit it cannot decode - -## Goal - -moq-mux's TS import ends with the cause on any media unit it fails to decode, -instead of publishing around it. `Fault`, its `Damaged { cause, published }` -partial success, and `unit_error`, which decides damage by matching about 15 -error variants, are gone. Packet-level continuity handling (duplicate and -missing TS packets, `ts/health.rs`) is not a unit decode error and stays. - -## Plan - -Decided 2026-11-18 (maintainer, /quest-plan): ✅ abort on any error, as on -[moq-dev/moq#9041](https://github.com/moq-dev/moq/pull/9041) for HLS and -moq-stats (rejected: keeping tolerance with a `Written { units, damage }` -return). "We should not try to support random bit flips that we don't -understand and limp along with broken functionality." A case the import -tolerates must come from a source behavior that is understood and -reproduced, handled where it is detected, never by classifying an error -afterwards. - -`Fault` came with [moq-dev/moq#8939](https://github.com/moq-dev/moq/pull/8939) -("TS import reports typed outcomes"), which the PR lists under `## Caused by`; -it lives in -`rs/moq-mux/src/container/ts/import.rs` and `ts/video.rs`. Before deleting a -tolerated case, check whether a test names a real source for it (such as -joining mid-PES at startup); keep such a case as an explicit check where it -happens, and list each kept one in the PR. - -Public API: none (the TS import's error surface is unchanged). Behavior: a -TS ingest with a damaged unit ends instead of continuing. Wire: none. - -## Related - -- [TS timestamp step](/quest/m1/ts-step-damage.md) - investigates whether one stream's timestamp step has a real source diff --git a/quest/m3/error-matches.md b/quest/m3/error-matches.md index c865c7bf44c..7e0019cfdbc 100644 --- a/quest/m3/error-matches.md +++ b/quest/m3/error-matches.md @@ -55,7 +55,6 @@ Sites, with recommendations (re-grep each on the head): Not here: moq-lite's control streams ([control-stream-recv](/quest/m3/control-stream-recv.md)), the moq-json stream producer and moqsink ([error-outcomes-rest](/quest/m3/error-outcomes-rest.md)), -the TS import's `unit_error` ([TS unit abort](/quest/m1/ts-unit-abort.md)), moq-quic's stream errors (reshaped in [moq-dev/moq#9152](https://github.com/moq-dev/moq/pull/9152)), `Duplicate` ([small-public](/quest/m2/outcome-enums/small-public.md)), and moq-e2ee ([e2ee-format](/quest/m3/e2ee-format.md)). diff --git a/rs/moq-mux/src/codec/aac/mod.rs b/rs/moq-mux/src/codec/aac/mod.rs index 6f26edebdc1..21eed219a5c 100644 --- a/rs/moq-mux/src/codec/aac/mod.rs +++ b/rs/moq-mux/src/codec/aac/mod.rs @@ -372,6 +372,11 @@ fn read_general_audio(reader: &mut BitReader, core: u8) -> Result bool { + block.first().is_some_and(|&byte| u32::from(byte >> 5) == ID_PCE) +} + /// Build the AudioSpecificConfig for a stream that signals its fields per frame, as ADTS does. /// /// A `channel_config` of 0 means a program config element leads `block`, the stream's first raw diff --git a/rs/moq-mux/src/codec/h264/split.rs b/rs/moq-mux/src/codec/h264/split.rs index aa5c31f62de..694111abc00 100644 --- a/rs/moq-mux/src/codec/h264/split.rs +++ b/rs/moq-mux/src/codec/h264/split.rs @@ -268,7 +268,7 @@ impl Split { self.pending.clear(); } - /// The retained parameter sets, for [`restore`](Self::restore) to put back. + /// The retained parameter sets, which bootstrap a reset picture clock. pub(crate) fn params(&self) -> Params { Params { sps: self.sps.clone(), @@ -276,13 +276,6 @@ impl Split { } } - /// Put back parameter sets a refused access unit replaced, so a later bare keyframe - /// re-injects the last good ones rather than the refused unit's. - pub(crate) fn restore(&mut self, params: Params) { - self.sps = params.sps; - self.pps = params.pps; - } - fn pts(&mut self, hint: Option) -> Result { if let Some(pts) = hint { return Ok(pts); diff --git a/rs/moq-mux/src/codec/h265/split.rs b/rs/moq-mux/src/codec/h265/split.rs index e372f772999..d4714b28120 100644 --- a/rs/moq-mux/src/codec/h265/split.rs +++ b/rs/moq-mux/src/codec/h265/split.rs @@ -276,7 +276,7 @@ impl Split { self.pending.clear(); } - /// The retained parameter sets, for [`restore`](Self::restore) to put back. + /// The retained parameter sets, which bootstrap a reset picture clock. pub(crate) fn params(&self) -> Params { Params { vps: self.vps.clone(), @@ -285,14 +285,6 @@ impl Split { } } - /// Put back parameter sets a refused access unit replaced, so a later bare keyframe - /// re-injects the last good ones rather than the refused unit's. - pub(crate) fn restore(&mut self, params: Params) { - self.vps = params.vps; - self.sps = params.sps; - self.pps = params.pps; - } - fn pts(&mut self, hint: Option) -> Result { if let Some(pts) = hint { return Ok(pts); diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index 3cd3040fb46..68da62b9dc9 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -1424,31 +1424,15 @@ impl Import { Ok(()) } - fn video_damage(&mut self, pid: Pid) { - let damage = match self.streams.get_mut(&pid) { - Some(Stream::H264 { assembly, .. } | Stream::H265 { assembly, .. }) => assembly.take_damage(), - _ => None, - }; - if let Some((count, err)) = damage { - *self.damaged.entry(pid.as_u16()).or_default() += count; - self.damage_log.record(pid.as_u16(), &err); - } - } - - /// Refuse one damaged unit without changing any other PID or the program clock. + /// Refuse the PES a damaged packet breaks (loss, the transport error indicator, or a + /// malformed TS or PES header) without changing any other PID or the program clock. fn damage(&mut self, pid: Pid, err: &Error) -> Result<()> { // Complete AUs ahead of the damaged packet remain valid, including those // in an unbounded PES. Its final NAL/AU is never flushed at a loss boundary. if let Some(pending) = self.pending.remove(&pid) && let Some(stream @ (Stream::H264 { .. } | Stream::H265 { .. })) = self.streams.get_mut(&pid) { - let result = stream.write(pending, false); - self.video_damage(pid); - let units = match result { - Ok(units) => units, - Err(Fault::Damaged { published, .. }) => published, - Err(Fault::Fatal(err)) => return Err(err), - }; + let units = stream.write(pending, false)?; self.published |= units > 0; self.liveness.delivered(pid.as_u16(), units); self.record_media_track(pid)?; @@ -1483,16 +1467,8 @@ impl Import { return self.damage(pid, &malformed!("PES payload truncated before its declared length")); } - let result = stream.write(pending, batched); - self.video_damage(pid); - let (units, damage) = match result { - Ok(units) => (units, None), - Err(Fault::Damaged { cause, published }) => (published, Some(cause)), - Err(Fault::Fatal(err)) => return Err(err), - }; - if let Some(err) = damage { - self.damage(pid, &err)?; - } else if truncated { + let units = stream.write(pending, batched)?; + if truncated { if matches!(self.streams.get(&pid), Some(Stream::H264 { .. } | Stream::H265 { .. })) { self.damage(pid, &malformed!("PES payload truncated before its declared length"))?; } else { @@ -1837,26 +1813,9 @@ impl Import { } fn finish_stream(&mut self, pid: Pid) -> Result { - let Some(stream) = self.streams.get_mut(&pid) else { - return Ok(0); - }; - let result = stream.finish(); - self.video_damage(pid); - match result { - Ok(units) => Ok(units), - Err(Fault::Damaged { - cause, - published: units, - }) => { - self.damage(pid, &cause)?; - self.streams - .get_mut(&pid) - .expect("damage preserves the route") - .finish() - .map_err(Fault::fatal)?; - Ok(units) - } - Err(Fault::Fatal(err)) => Err(err), + match self.streams.get_mut(&pid) { + Some(stream) => stream.finish(), + None => Ok(0), } } @@ -3223,72 +3182,6 @@ impl SectionReassembler { } } -/// Why a stream stopped publishing what it was handed. -#[derive(Debug)] -enum Fault { - /// A parse failure confined to a unit, which the import drops and carries on past, after - /// `published` units of the same call went out. - Damaged { cause: Error, published: u64 }, - /// A publishing, clock, or stream failure, which ends the import. - Fatal(Error), -} - -impl Fault { - /// The failure this fault reports where nothing recovers from damage. - fn fatal(self) -> Error { - match self { - Self::Damaged { cause, .. } | Self::Fatal(cause) => cause, - } - } - - /// A damaged unit, with nothing published before it. - fn damaged(cause: impl Into) -> Self { - Self::Damaged { - cause: cause.into(), - published: 0, - } - } - - /// This fault after `published` units of the same call went out, so damage still counts them. - fn after(self, published: u64) -> Self { - match self { - Self::Damaged { cause, .. } => Self::Damaged { cause, published }, - fatal => fatal, - } - } -} - -impl> From for Fault { - fn from(err: E) -> Self { - Self::Fatal(err.into()) - } -} - -/// `err` as damage to the unit it came from, when it is a parse failure confined to that unit. -fn unit_error(err: crate::Error) -> Fault { - let damaged = matches!( - &err, - crate::Error::Annexb(_) - | crate::Error::H264( - h264::Error::NalTooShort - | h264::Error::ForbiddenZeroBit - | h264::Error::SpsTooShort - | h264::Error::SpsParse - | h264::Error::NotInitialized - ) | crate::Error::Aac( - aac::Error::ProgramConfigMissing | aac::Error::ProgramConfigTruncated | aac::Error::ProgramConfigEmpty - ) | crate::Error::H265( - h265::Error::NalTooShort - | h265::Error::ForbiddenZeroBit - | h265::Error::SpsParse - | h265::Error::MissingLevelIdc - | h265::Error::NotInitialized - | h265::Error::MissingSps - ) - ); - if damaged { Fault::damaged(err) } else { err.into() } -} - /// Whether `stream_type` is MPEG-1 or MPEG-2 video, which is carried whole but still paces the /// program like decoded video. fn mpeg_video(stream_type: u8) -> bool { @@ -3347,7 +3240,7 @@ enum Stream { impl Stream { /// Route one reassembled PES, returning how many access units it published. - fn write(&mut self, pending: Pending, batched: bool) -> Result { + fn write(&mut self, pending: Pending, batched: bool) -> Result { match self { Stream::H264 { split, @@ -3507,7 +3400,7 @@ impl Stream { } /// Finish the track, returning the access units a drained tail published. - fn finish(&mut self) -> Result { + fn finish(&mut self) -> Result { match self { // Only the self-describing audio holds a frame back for a successor to confirm. Stream::Aac(stream) => return stream.finish(), @@ -3891,13 +3784,13 @@ struct AacStream { } impl AacStream { - fn write(&mut self, pending: Pending, batched: bool) -> Result { + fn write(&mut self, pending: Pending, batched: bool) -> Result { // The frames a PES completes share one group, closed once the PES is spent, even by an // error. The transport sends a track's newest group first, so a group per frame would // deliver a multi-frame PES newest first, its oldest frame waiting behind the rest. let mut open = None; let result = self.split(pending, batched, &mut open); - // A failed cut is a publishing failure, which outranks a damaged unit the caller recovers from. + // A failed cut is a publishing failure, which outranks the split's own error. if let (Some(end), Some(import)) = (open, &mut self.import) { import.cut(Some(end))?; } @@ -3907,7 +3800,7 @@ impl AacStream { /// Publish the frames `pending` completes into one group, leaving `open` at where its last frame /// stops while it is open. A frame without a PTS is stamped by the producer, which checks only /// where groups start, so it keeps a group of its own, as every frame once did. - fn split(&mut self, pending: Pending, batched: bool, open: &mut Option) -> Result { + fn split(&mut self, pending: Pending, batched: bool, open: &mut Option) -> Result { let pes_base = on_clock(pending.pts, pending.offset)?; // Prepend the partial frame left by the previous PES, if any. @@ -4006,8 +3899,7 @@ impl AacStream { return Err(match err { Some(err) => malformed!("{context}: {err}"), None => Error::Malformed(context), - } - .into()); + }); } // Take the PES PTS only where the tail is actually abandoned: restoring a fallback // keeps the candidate, so it keeps the timestamp it was found with. @@ -4065,22 +3957,26 @@ impl AacStream { Some(import) => { // The description already carries the layout, so a repeated program config // element (the TS export writes one after each PAT/PMT) leaves the frame too. - if header.channel_config == 0 { + if header.channel_config == 0 && aac::leads_with_program_config(block) { let mut rest = block; - match aac::in_band_config(header.object_type, header.sample_rate, 0, &mut rest) { - Ok(asc) if asc == self.asc => block = rest, - Ok(_) => { - return Err(Fault::damaged(malformed!( - "AAC program config element changed mid-stream" - )) - .after(published)); - } - Err(aac::Error::ProgramConfigMissing) => {} - Err(err) => return Err(unit_error(err.into()).after(published)), - } + let asc = aac::in_band_config(header.object_type, header.sample_rate, 0, &mut rest)?; + ensure!(asc == self.asc, "AAC program config element changed mid-stream"); + block = rest; } import } + // ffmpeg writes the element in its first frame only, so a receiver that joined + // later may never see one. Skip the frame, and stop withholding the catalog for + // this track; it joins the catalog if an element arrives. + None if header.channel_config == 0 && !aac::leads_with_program_config(block) => { + self.reserved.take(); + pts = advance_pts(pts, 1024, header.sample_rate)?; + // The frame was confirmed, so the stream is in sync even though nothing published. + self.resync.published(unvouched); + fallback = None; + offset = end; + continue; + } None => { // Synthesize the AudioSpecificConfig `description` so out-of-band consumers // (fMP4/MKV export, WebCodecs) can configure the decoder. A channel_config of 0 @@ -4091,14 +3987,7 @@ impl AacStream { header.sample_rate, header.channel_config, &mut block, - ) - .map_err(|err| { - // ffmpeg writes the element in its first frame only, so a receiver that - // joined later may never see one. Stop withholding the catalog for this - // track; it joins the catalog if an element arrives. - self.reserved.take(); - unit_error(err.into()).after(published) - })?; + )?; let mut config = aac::config(&asc)?; config.container = self.container.clone(); // Consume the reservation held since the PMT: this resolves the gated rendition, @@ -4186,7 +4075,7 @@ impl AacStream { self.resync.desynced(); } - fn finish(&mut self) -> Result { + fn finish(&mut self) -> Result { // Drain a frame held only for want of a successor to confirm it: at end of stream // that successor is never coming. Only once this stream has published a frame, // though. Before that nothing has vouched for any boundary, so accepting one here @@ -4233,10 +4122,10 @@ struct OpusStream { } impl OpusStream { - fn write(&mut self, pending: Pending) -> Result { + fn write(&mut self, pending: Pending) -> Result { let base = on_clock(pending.pts, pending.offset)?; - let packets = opus_packets(&pending.data).map_err(Fault::damaged)?; + let packets = opus_packets(&pending.data)?; let mut published = 0; // 48 kHz samples elapsed since this PES's PTS, advancing each packet after the first. let mut elapsed: u64 = 0; @@ -4533,11 +4422,11 @@ struct LegacyStream { } impl LegacyStream { - fn write(&mut self, pending: Pending) -> Result { + fn write(&mut self, pending: Pending) -> Result { // One group per PES, closed even by an error: see `AacStream::write`. let mut open = None; let result = self.split(pending, &mut open); - // A failed cut is a publishing failure, which outranks a damaged unit the caller recovers from. + // A failed cut is a publishing failure, which outranks the split's own error. if let (Some(end), Some(import)) = (open, &mut self.import) { import.cut(Some(end))?; } @@ -4547,7 +4436,7 @@ impl LegacyStream { /// Publish the frames `pending` completes into one group, leaving `open` at where its last frame /// stops while it is open. A frame without a PTS is stamped by the producer, which checks only /// where groups start, so it keeps a group of its own, as every frame once did. - fn split(&mut self, pending: Pending, open: &mut Option) -> Result { + fn split(&mut self, pending: Pending, open: &mut Option) -> Result { let mut published = 0; let pes_base = on_clock(pending.pts, pending.offset)?; @@ -4653,8 +4542,7 @@ impl LegacyStream { return Err(match err { Some(err) => malformed!("{context}: {err}"), None => Error::Malformed(context), - } - .into()); + }); } // The tail's frame boundary is what we just lost, so its PTS no longer // describes anything. Re-anchor on the PES, whose PTS covers the first @@ -4793,7 +4681,7 @@ impl LegacyStream { self.resync.desynced(); } - fn finish(&mut self) -> Result { + fn finish(&mut self) -> Result { // Drain a frame held only for want of a successor to confirm it: at end of stream // that successor is never coming. Only once this stream has published a frame, // though. Before that nothing has vouched for any boundary, so accepting one here @@ -4837,11 +4725,11 @@ impl LegacyStream { /// Swallow a [`MissingKeyframe`](crate::container::MissingKeyframe) from a video /// decode: a TS capture can join mid-GOP, so the deltas before the first keyframe /// have no group to anchor and are simply dropped rather than aborting the demux. -fn skip_missing_keyframe(result: crate::Result<()>) -> Result { +fn skip_missing_keyframe(result: crate::Result<()>) -> Result { match result { Ok(()) => Ok(true), Err(crate::Error::MissingKeyframe(_)) => Ok(false), - Err(e) => Err(unit_error(e)), + Err(e) => Err(e.into()), } } @@ -7550,7 +7438,7 @@ pub(super) mod test { &[bare.as_slice(), &bare, &bare].concat(), )); import.decode(&mux.out).unwrap(); - assert_eq!(import.stats().streams[&AAC_PID].damaged, 1); + assert_eq!(import.stats().streams[&AAC_PID].damaged, 0, "a join is not damage"); let first = kio::clock::timeout(Duration::from_millis(10), updates.next()) .await @@ -7587,10 +7475,10 @@ pub(super) mod test { } } - // A program config element that changes the layout mid-stream refuses its unit as - // damaged, and the track carries on with the frames around it. - #[tokio::test] - async fn aac_program_config_changed_mid_stream_refuses_the_unit() { + // A program config element that changes the layout mid-stream ends the import with the + // cause, after the frames ahead of it published. + #[tokio::test(start_paused = true)] + async fn aac_program_config_changed_mid_stream_ends_the_import() { const AAC_PID: u16 = 0x0060; // ffmpeg's quad layout: one front and one back channel pair. let mut quad = vec![0x11, 0x80, 0x04, 0xC4, 0x04, 0x00, 0x21, 0x10, 0x0C]; @@ -7618,25 +7506,19 @@ pub(super) mod test { out: synth_pmt(&[(StreamType::AdtsAac, AAC_PID)], false), ..Default::default() }; - // The third PES publishes a frame before its changed element, which still counts. for (pts, unit) in [ (90_000, [with.as_slice(), &bare]), - (180_000, [changed.as_slice(), &bare]), - (270_000, [bare.as_slice(), &changed]), + (180_000, [bare.as_slice(), &changed]), ] { let cc = mux.cc(AAC_PID); mux.out.extend(audio_pes_packet(AAC_PID, cc, pts, &unit.concat())); } let cc = mux.cc(AAC_PID); - mux.out.extend(audio_pes_packet(AAC_PID, cc, 360_000, &bare)); - import.decode(&mux.out).unwrap(); - import.finish_in_place().unwrap(); - let stream = &import.stats().streams[&AAC_PID]; - assert_eq!(stream.damaged, 2); - + mux.out.extend(audio_pes_packet(AAC_PID, cc, 270_000, &bare)); + let err = import.decode(&mux.out).expect_err("a changed layout ends the import"); + assert!(err.to_string().contains("changed mid-stream"), "{err}"); let frames = read_audio_frames(&consumer, &catalog).await; - assert_eq!(frames.len(), 4, "the changed units are dropped, the rest publish"); - assert_eq!(stream.units, 4, "every published frame counts as delivered"); + assert_eq!(frames.len(), 3, "the frames ahead of it"); } // Resync is bounded: a PID carrying something other than the codec its PMT declares is a @@ -9795,46 +9677,6 @@ pub(super) mod test { ); } - #[tokio::test(start_paused = true)] - async fn video_parameter_sets_commit_per_au_instead_of_per_pes() { - use crate::container::test_util::{IDR, PPS, SPS}; - let extra_pps = [0x68, 0x4e, 0x3c, 0x80]; - let mut payload = Vec::new(); - for nal in [SPS, PPS, &extra_pps, IDR, &[0x67, 0x42], &[0x65, 0x80]] { - payload.extend([0, 0, 0, 1]); - payload.extend(nal); - } - let mut data = synth_pmt(&[(StreamType::H264, VIDEO)], false); - data.extend(video_pes(VIDEO, 0, 90_000, None, &payload)); - // A later keyframe proves the damaged AU ended and must recover with - // the last committed parameter sets, including the extra PPS above. - let mut bare = vec![0, 0, 0, 1]; - bare.extend(IDR); - data.extend(video_pes(VIDEO, 1, 180_000, None, &bare)); - let (consumer, catalog, import) = import_all(&data).unwrap(); - let name = catalog.snapshot().video.renditions.keys().next().unwrap().clone(); - let frames = read_track(&consumer, &name, crate::container::Kind::Video).await; - assert_eq!(frames.len(), 2); - assert_eq!(import.stats().streams[&VIDEO].damaged, 1); - assert_eq!(import.stats().streams[&VIDEO].units, 2); - for frame in &frames { - assert!(frame.payload.windows(extra_pps.len()).any(|nal| nal == extra_pps)); - } - } - - #[test] - fn video_damage_in_the_final_access_unit_is_counted() { - let mut data = synth_pmt(&[(StreamType::H264, VIDEO)], false); - data.extend(video_pes(VIDEO, 0, 90_000, None, &annexb_au(true))); - // Only end of input ends this AU, so finish() refuses it: its truncated - // SPS assembles but fails to parse. - let au = [0, 0, 0, 1, 0x67, 0x42, 0, 0, 0, 1, 0x65, 0x80]; - data.extend(video_pes(VIDEO, 1, 90_000 + FRAME, None, &au)); - let (_consumer, _catalog, import) = import_all(&data).unwrap(); - assert_eq!(import.stats().streams[&VIDEO].units, 1); - assert_eq!(import.stats().streams[&VIDEO].damaged, 1); - } - /// Damage on one PID refuses that unit and its dependent pictures while another PID /// carries on. A later decode call resumes from the next keyframe. async fn damaged_video_recovers(damage: &str) { @@ -9861,25 +9703,6 @@ pub(super) mod test { let payload_at = 5 + usize::from(packet[4]); match damage { "pes" => packet[payload_at + 6..payload_at + 19].fill(0), - "nal" => { - let nal = packet - .windows(5) - .position(|bytes| bytes[..4] == [0, 0, 0, 1] && bytes[4] & 0x1f == 5) - .unwrap(); - packet[nal + 4] |= 0x80; - } - "late-nal" => { - let mut au = annexb_au(true); - au.extend_from_slice(&[0, 0, 0, 1, 0xe1, 0x80]); - packet.copy_from_slice(&video_pes(VIDEO, 4, 90_000 + 5 * FRAME, Some(90_000 + 4 * FRAME), &au)); - } - "queued" => { - // The AUD closes a valid IDR before the forbidden NAL in the next - // AU fails. That already-admitted IDR must remain committed. - let mut au = annexb_au(true); - au.extend_from_slice(&[0, 0, 0, 1, 0x09, 0xf0, 0, 0, 0, 1, 0xe1, 0x80, 0, 0, 0, 1, 0x41, 0x9a]); - packet.copy_from_slice(&video_pes(VIDEO, 4, 90_000 + 5 * FRAME, Some(90_000 + 4 * FRAME), &au)); - } "adaptation" => packet[4] = 255, "adaptation-clock" => { packet[4] = 1; @@ -9916,35 +9739,16 @@ pub(super) mod test { import.finish_in_place().unwrap(); assert_eq!(import.stats().streams[&VIDEO].damaged, u64::from(damage != "clean")); assert_eq!(import.stats().streams[&peer].damaged, 0); - for (pid, expected) in [ - ( - VIDEO, - if damage == "clean" { - 12 - } else if damage == "queued" { - 9 - } else { - 8 - }, - ), - (peer, 12), - ] { + for (pid, expected) in [(VIDEO, if damage == "clean" { 12 } else { 8 }), (peer, 12)] { let Stream::H264 { import: video, .. } = &import.streams[&Pid::new(pid).unwrap()] else { panic!("video route"); }; let frames = read_track(&consumer, video.name(), crate::container::Kind::Video).await; assert_eq!(frames.len(), expected, "published pictures on PID {pid}"); if pid == VIDEO && damage != "clean" { - let recovery = if damage == "queued" { 5 } else { 4 }; - if damage == "queued" { - assert!( - frames[4].keyframe, - "the valid AU before the damaged tail remains committed" - ); - } - assert!(frames[recovery].keyframe, "recovery starts at a keyframe"); + assert!(frames[4].keyframe, "recovery starts at a keyframe"); assert_eq!( - frames[recovery].timestamp.as_micros().unwrap(), + frames[4].timestamp.as_micros().unwrap(), u128::from(90_000 + 9 * FRAME) * 1_000_000 / 90_000 ); } @@ -9956,19 +9760,84 @@ pub(super) mod test { damaged_video_recovers("pes").await; } + /// PES boundaries need not fall on NALs, so the first PES after a continuity break can open + /// inside a NAL that began in the lost one. Its bytes up to the next start code are dropped, + /// not decoded, and the stream recovers at its next keyframe. #[tokio::test(start_paused = true)] - async fn damaged_h264_nal_recovers() { - damaged_video_recovers("nal").await; - } + async fn a_pes_opening_mid_nal_after_a_break_is_dropped() { + let mut mux = Mux { + out: synth_pmt(&[(StreamType::H264, VIDEO)], false), + ..Default::default() + }; + mux.gop(VIDEO, 90_000); + // The PES holding the NAL's start is lost, and the next carries only its tail. + mux.cc(VIDEO); + let cc = mux.cc(VIDEO); + mux.out.extend(video_pes( + VIDEO, + cc, + 90_000 + 5 * FRAME, + None, + &[0x9a, 0xbc, 0xde, 0xf0], + )); + mux.gop(VIDEO, 90_000 + 8 * FRAME); - #[tokio::test(start_paused = true)] - async fn damaged_trailing_nal_refuses_the_whole_access_unit() { - damaged_video_recovers("late-nal").await; + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let mut import = super::Import::new(broadcast, catalog.reserve()); + import + .decode(&mux.out) + .expect("a NAL's lost start is loss, not a decode error"); + import.finish_in_place().unwrap(); + let name = catalog.snapshot().video.renditions.keys().next().unwrap().clone(); + let frames = read_track(&consumer, &name, crate::container::Kind::Video).await; + // Nothing proved the first GOP's last picture whole before the break, so it goes too. + assert_eq!(frames.len(), 7, "both GOPs, around the break"); } + /// A NAL that does not parse ends the import with the cause, wherever it sits in its + /// access unit, after the pictures ahead of it published. #[tokio::test(start_paused = true)] - async fn damaged_tail_preserves_an_access_unit_already_admitted() { - damaged_video_recovers("queued").await; + async fn undecodable_nal_ends_the_import() { + // A forbidden_zero_bit set on the IDR slice, on a trailing NAL, and on a NAL in the AU + // after a valid IDR that an AUD proved whole and so already published. + let mut forbidden = annexb_au(true); + let nal = forbidden + .windows(5) + .position(|bytes| bytes[..4] == [0, 0, 0, 1] && bytes[4] & 0x1f == 5) + .unwrap(); + forbidden[nal + 4] |= 0x80; + let mut trailing = annexb_au(true); + trailing.extend_from_slice(&[0, 0, 0, 1, 0xe1, 0x80]); + let mut queued = annexb_au(true); + queued.extend_from_slice(&[0, 0, 0, 1, 0x09, 0xf0, 0, 0, 0, 1, 0xe1, 0x80, 0, 0, 0, 1, 0x41, 0x9a]); + for (au, published) in [(forbidden, 4), (trailing, 4), (queued, 5)] { + let mut mux = Mux { + out: synth_pmt(&[(StreamType::H264, VIDEO)], false), + ..Default::default() + }; + mux.gop(VIDEO, 90_000); + let cc = mux.cc(VIDEO); + mux.out + .extend(video_pes(VIDEO, cc, 90_000 + 5 * FRAME, Some(90_000 + 4 * FRAME), &au)); + mux.gop(VIDEO, 90_000 + 8 * FRAME); + + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let mut import = super::Import::new(broadcast, catalog.reserve()); + let err = import.decode(&mux.out).expect_err("an undecodable NAL ends the import"); + assert!(err.to_string().contains("forbidden"), "{err}"); + let name = catalog.snapshot().video.renditions.keys().next().unwrap().clone(); + let frames = read_track(&consumer, &name, crate::container::Kind::Video).await; + assert_eq!(frames.len(), published, "the pictures ahead of it"); + assert_eq!( + import.stats().streams[&VIDEO].damaged, + 0, + "a decode error is not damage" + ); + } } #[tokio::test(start_paused = true)] @@ -9991,9 +9860,9 @@ pub(super) mod test { damaged_video_recovers("clean").await; } - /// A keyframe refused for a malformed inline SPS must not replace the splitter's last good - /// parameter sets: a later bare keyframe re-injects those and recovers. - async fn damaged_sps_keeps_the_last_good_parameter_sets(stream_type: StreamType) { + /// A keyframe whose inline SPS does not parse ends the import with the cause, after the + /// pictures ahead of it published. + fn malformed_sps_ends_the_import(stream_type: StreamType) { let (params, idr, delta): (&[&[u8]], &[u8], &[u8]) = match stream_type { StreamType::H264 => { use crate::container::test_util::{IDR, PPS, SPS}; @@ -10005,9 +9874,6 @@ pub(super) mod test { } _ => unreachable!(), }; - let good_sps = params[params.len() - 2]; - // The SPS keeps its NAL header and loses the rest. - let truncated = &good_sps[..2]; let au = |nals: &[&[u8]]| { let mut out = Vec::new(); for nal in nals { @@ -10018,17 +9884,10 @@ pub(super) mod test { }; let mut keyframe = params.to_vec(); keyframe.push(idr); - let mut damaged = params.to_vec(); - damaged[params.len() - 2] = truncated; - damaged.push(idr); - let units = [ - au(&keyframe), - au(&[delta]), - au(&damaged), - au(&[delta]), - au(&[idr]), - au(&[delta]), - ]; + // The SPS keeps its NAL header and loses the rest. + let mut damaged = keyframe.clone(); + damaged[params.len() - 2] = ¶ms[params.len() - 2][..2]; + let units = [au(&keyframe), au(&[delta]), au(&damaged), au(&[delta])]; let mut data = synth_pmt(&[(stream_type, VIDEO)], false); for (cc, unit) in units.iter().enumerate() { let pts = 90_000 + cc as u64 * FRAME; @@ -10036,29 +9895,20 @@ pub(super) mod test { } let mut broadcast = moq_net::broadcast::Info::new().produce(); - let consumer = broadcast.consume(); let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); let mut import = super::Import::new(broadcast, catalog.reserve()); - import.decode(&data).expect("a damaged SPS stays local to its unit"); - let _finished = import.finish().unwrap(); - let name = catalog.snapshot().video.renditions.keys().next().unwrap().clone(); - let frames = read_track(&consumer, &name, crate::container::Kind::Video).await; - assert_eq!(frames.len(), 4, "the bare keyframe after the damage recovers"); - assert!(frames[2].keyframe); - assert!( - frames[2].payload.windows(good_sps.len()).any(|bytes| bytes == good_sps), - "the recovery keyframe carries the last good SPS" - ); + import.decode(&data).expect_err("a malformed SPS ends the import"); + assert_eq!(import.stats().streams[&VIDEO].units, 2); } - #[tokio::test(start_paused = true)] - async fn damaged_h264_sps_keeps_the_last_good_parameter_sets() { - damaged_sps_keeps_the_last_good_parameter_sets(StreamType::H264).await; + #[test] + fn malformed_h264_sps_ends_the_import() { + malformed_sps_ends_the_import(StreamType::H264); } - #[tokio::test(start_paused = true)] - async fn damaged_h265_sps_keeps_the_last_good_parameter_sets() { - damaged_sps_keeps_the_last_good_parameter_sets(StreamType::H265).await; + #[test] + fn malformed_h265_sps_ends_the_import() { + malformed_sps_ends_the_import(StreamType::H265); } #[test] @@ -10211,39 +10061,33 @@ pub(super) mod test { ); } - /// A malformed unbounded PES drained at EOF is still refused and counted. + /// A malformed unbounded PES drained at EOF ends the finish with the cause. #[test] - fn damaged_final_access_unit_is_refused() { + fn malformed_final_access_unit_fails_the_finish() { let mut au = annexb_au(true); let nal = au .windows(5) .position(|bytes| bytes[..4] == [0, 0, 0, 1] && bytes[4] & 0x1f == 5) .unwrap(); au[nal + 4] |= 0x80; - for au in [au, vec![0], vec![0, 0], vec![0, 0, 0]] { - let mut broadcast = moq_net::broadcast::Info::new().produce(); - let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); - let mut import = super::Import::new(broadcast, catalog.reserve()); - let mut packet = video_pes(VIDEO, 0, 90_000, None, &au); - let start = 5 + usize::from(packet[4]); - packet[start + 4..start + 6].fill(0); - let mut data = synth_pmt(&[(StreamType::H264, VIDEO)], false); - data.extend(packet); - import.decode(&data).unwrap(); - assert_eq!(import.stats().streams[&VIDEO].damaged, 0, "PES has not ended yet"); - import.finish_in_place().unwrap(); - assert_eq!(import.stats().streams[&VIDEO].damaged, 1); - assert_eq!(import.stats().streams[&VIDEO].units, 0); - assert!(catalog.snapshot().video.renditions.is_empty()); - } + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let mut import = super::Import::new(broadcast, catalog.reserve()); + let mut packet = video_pes(VIDEO, 0, 90_000, None, &au); + let start = 5 + usize::from(packet[4]); + packet[start + 4..start + 6].fill(0); + let mut data = synth_pmt(&[(StreamType::H264, VIDEO)], false); + data.extend(packet); + import.decode(&data).expect("the PES has not ended yet"); + import.finish_in_place().expect_err("the final AU does not decode"); + assert!(catalog.snapshot().video.renditions.is_empty()); } - /// An invalid trailing Opus control header refuses the complete PES before its first - /// packet is published. Independently decodable packets in the next PES recover. - #[tokio::test(start_paused = true)] - async fn damaged_opus_pes_is_not_half_published() { + /// An invalid trailing Opus control header ends the import before any packet of its PES + /// is published. + #[test] + fn malformed_opus_pes_ends_the_import_unpublished() { let mut broadcast = moq_net::broadcast::Info::new().produce(); - let consumer = broadcast.consume(); let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); let mut import = super::Import::new(broadcast, catalog.reserve()); import @@ -10263,54 +10107,8 @@ pub(super) mod test { for (cc, payload) in [(0, good.as_slice()), (1, bad.as_slice()), (2, good.as_slice())] { data.extend(video_pes(VIDEO, cc, 90_000 + u64::from(cc) * FRAME, None, payload)); } - import.decode(&data).unwrap(); - import.finish_in_place().unwrap(); - assert_eq!(import.stats().streams[&VIDEO].damaged, 1); - assert_eq!(import.stats().streams[&VIDEO].units, 2); - let name = catalog.snapshot().audio.renditions.keys().next().unwrap().clone(); - let frames = read_track(&consumer, &name, crate::container::Kind::Audio).await; - assert_eq!(frames.len(), 2, "no packet from the damaged PES is published"); - } - - /// An aborted track is a feed-wide publishing failure, never a damaged media unit. - #[test] - fn damaged_recovery_keeps_publishing_errors_fatal() { - let mut broadcast = moq_net::broadcast::Info::new().produce(); - let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); - let track = broadcast - .create_track("video", hang::container::track_info(hang::catalog::PRIORITY.video)) - .unwrap(); - let video = crate::codec::h264::Import::new(track.clone(), catalog.reserve(), Default::default()).unwrap(); - let mut import = super::Import::new(broadcast, catalog.reserve()); - import.streams.insert( - Pid::new(VIDEO).unwrap(), - Stream::H264 { - split: crate::codec::h264::Split::new(), - assembly: super::video::Assembly::new(super::video::Codec::Avc), - import: Box::new(video), - }, - ); - track.abort(moq_net::Error::Closed).unwrap(); - let mut data = Vec::new(); - for cc in 0..3 { - data.extend(video_pes( - VIDEO, - cc, - 90_000 + u64::from(cc) * FRAME, - None, - &annexb_au(true), - )); - } - let err = import.decode(&data).expect_err("closed producer must end ingest"); - assert!( - matches!(&err, super::Error::Mux(err) if matches!(**err, crate::Error::Moq(_))), - "{err:?}" - ); - assert!(import.damaged.is_empty(), "publishing failure is not local damage"); - assert!(matches!( - super::unit_error(crate::Error::H264(crate::codec::h264::Error::MissingTimestamp)), - super::Fault::Fatal(_) - )); + import.decode(&data).expect_err("a malformed Opus PES ends the import"); + assert_eq!(import.stats().streams[&VIDEO].units, 1, "only the PES ahead of it"); } /// A break closes the video group where its content stops, not a GOP later at the next diff --git a/rs/moq-mux/src/container/ts/stats.rs b/rs/moq-mux/src/container/ts/stats.rs index 0256ec2eed3..91921d88540 100644 --- a/rs/moq-mux/src/container/ts/stats.rs +++ b/rs/moq-mux/src/container/ts/stats.rs @@ -167,7 +167,7 @@ pub struct Stream { /// counted separately from a resync. #[serde(skip_serializing_if = "is_zero")] pub unconfirmed: u64, - /// Damaged packets, PES, or access units refused whole on this PID. Export stays zero. + /// Damaged packets or PES refused whole on this PID. Export stays zero. #[serde(skip_serializing_if = "is_zero")] pub damaged: u64, /// Whole PES or sections, SI included, dropped because the program clock could not place diff --git a/rs/moq-mux/src/container/ts/video.rs b/rs/moq-mux/src/container/ts/video.rs index 093a14293ec..5651fea108c 100644 --- a/rs/moq-mux/src/container/ts/video.rs +++ b/rs/moq-mux/src/container/ts/video.rs @@ -8,9 +8,9 @@ use std::collections::VecDeque; use bytes::{Buf, Bytes, BytesMut}; use moq_net::Timestamp; -use super::{Fault, Pending, Retained, on_clock, reorder_delay, unit_error}; +use super::{Pending, Retained, on_clock, reorder_delay}; use crate::codec::annexb::{START_CODE, after_start_code, find_start_code}; -use crate::container::ts::error::{Error, Result, ensure, ensure_supported, malformed}; +use crate::container::ts::error::{Result, ensure, ensure_supported, malformed}; #[derive(Clone, Copy)] pub(super) enum Codec { @@ -28,24 +28,22 @@ impl Codec { /// Whether this NAL begins a picture or its leading non-VCL data /// (H.264 7.4.1.2.3, H.265 7.4.2.4.4), and whether it is a slice. - fn boundary(self, nal: &[u8]) -> Result<(bool, bool), Fault> { + fn boundary(self, nal: &[u8]) -> Result<(bool, bool)> { match self { Self::Avc => { let header = *nal .first() .ok_or(crate::codec::h264::Error::NalTooShort) - .map_err(crate::Error::from) - .map_err(unit_error)?; + .map_err(crate::Error::from)?; if header & 0x80 != 0 { - return Err(unit_error(crate::codec::h264::Error::ForbiddenZeroBit.into())); + return Err(crate::Error::from(crate::codec::h264::Error::ForbiddenZeroBit).into()); } let kind = header & 31; let slice = matches!(kind, 1..=5); let first = if slice { *nal.get(1) .ok_or(crate::codec::h264::Error::NalTooShort) - .map_err(crate::Error::from) - .map_err(unit_error)? + .map_err(crate::Error::from)? & 0x80 != 0 } else { matches!(kind, 6..=9 | 14..=18) @@ -56,18 +54,16 @@ impl Codec { let header = *nal .first() .ok_or(crate::codec::h265::Error::NalTooShort) - .map_err(crate::Error::from) - .map_err(unit_error)?; + .map_err(crate::Error::from)?; if header & 0x80 != 0 { - return Err(unit_error(crate::codec::h265::Error::ForbiddenZeroBit.into())); + return Err(crate::Error::from(crate::codec::h265::Error::ForbiddenZeroBit).into()); } let kind = (header >> 1) & 63; let slice = kind <= 31; let first = if slice { *nal.get(2) .ok_or(crate::codec::h265::Error::NalTooShort) - .map_err(crate::Error::from) - .map_err(unit_error)? + .map_err(crate::Error::from)? & 0x80 != 0 } else { matches!(kind, 32..=35 | 39 | 41..=44 | 48..=55) @@ -99,7 +95,6 @@ mod timing; pub(super) struct Assembly { pub clock: timing::Clock, - damage: Option<(u64, Error)>, codec: Codec, tail: BytesMut, searched: usize, @@ -110,13 +105,16 @@ pub(super) struct Assembly { ending: bool, parameters: bool, retained: Option, + /// No start code has been seen since a join or a break. ISO 13818-1 doesn't align PES to + /// NALs, so the first PES after either can open mid-NAL; its bytes before the first start + /// code belong to a NAL that began in a PES this stream never got. + acquiring: bool, } impl Assembly { pub fn new(codec: Codec) -> Self { Self { clock: timing::Clock::default(), - damage: None, codec, tail: BytesMut::new(), searched: 0, @@ -127,6 +125,7 @@ impl Assembly { ending: false, parameters: false, retained: None, + acquiring: true, } } @@ -172,17 +171,29 @@ impl Assembly { } /// Return one AU at a time so later parameter sets cannot precede its validation. - pub fn next(&mut self, eof: bool) -> Result, Fault> { + pub fn next(&mut self, eof: bool) -> Result> { loop { if self.tail.is_empty() { return if eof { self.emit() } else { Ok(None) }; } - let start = after_start_code(&self.tail) - .map_err(crate::Error::from) - .map_err(unit_error)?; + if self.acquiring { + match find_start_code(&self.tail) { + Some((at, _)) => { + self.discard(at)?; + self.acquiring = false; + } + None => { + // Keep only what could begin a start code the next PES completes. + let keep = if eof { 0 } else { 3 }; + self.discard(self.tail.len().saturating_sub(keep))?; + return Ok(None); + } + } + } + let start = after_start_code(&self.tail).map_err(crate::Error::from)?; let Some(start) = start else { if eof { - return Err(unit_error(crate::codec::annexb::Error::MissingStartCode.into())); + return Err(crate::Error::from(crate::codec::annexb::Error::MissingStartCode).into()); } return Ok(None); }; @@ -265,16 +276,7 @@ impl Assembly { }; self.current.extend_from_slice(&START_CODE); self.current.extend_from_slice(&nal); - let mut remaining = consumed; - while remaining > 0 { - let span = self.spans.front_mut().expect("consumed bytes have an owner"); - let take = remaining.min(span.len); - span.len -= take; - remaining -= take; - if span.len == 0 { - self.spans.pop_front(); - } - } + self.spend(consumed); self.account()?; if self.ending { if self.has_slice { @@ -289,7 +291,27 @@ impl Assembly { } } - fn emit(&mut self) -> Result, Fault> { + /// Release the ownership of `len` bytes taken from the front of the tail. + fn spend(&mut self, mut len: usize) { + while len > 0 { + let span = self.spans.front_mut().expect("consumed bytes have an owner"); + let take = len.min(span.len); + span.len -= take; + len -= take; + if span.len == 0 { + self.spans.pop_front(); + } + } + } + + /// Drop `len` bytes from the front of the tail, with their timestamps and charges. + fn discard(&mut self, len: usize) -> Result<()> { + self.tail.advance(len); + self.spend(len); + self.account() + } + + fn emit(&mut self) -> Result> { if !self.has_slice { return Ok(None); } @@ -311,15 +333,9 @@ impl Assembly { })) } - pub fn take_damage(&mut self) -> Option<(u64, Error)> { - self.damage.take() - } - - /// Release everything buffered at end of input, keeping damage the final drain refused. + /// Release everything buffered at end of input. pub fn finish(&mut self) { - let damage = self.damage.take(); *self = Self::new(self.codec); - self.damage = damage; } pub fn reset(&mut self) { @@ -333,6 +349,7 @@ impl Assembly { self.ending = false; self.parameters = false; self.retained = None; + self.acquiring = true; } } @@ -368,70 +385,39 @@ macro_rules! publish { import: &mut crate::codec::$codec::Import, assembly: &mut Assembly, eof: bool, - ) -> Result { + ) -> Result { let mut published = 0; - let result = (|| { - while let Some(unit) = assembly.next(eof)? { - let params = unit.parameters.then(|| split.params()); - let result = (|| { - // Structural damage is local even when it prevents timing - // inference. No provisional timestamp reaches the producer. - let provisional = unit.pts.unwrap_or(Timestamp::ZERO); - let mut frames = split.decode(&unit.data, provisional).map_err(unit_error)?; - frames.extend(split.flush(provisional).map_err(unit_error)?); - ensure_supported!(frames.len() == 1, "unsupported TS video access-unit framing"); - for frame in &frames { - if frame.keyframe { - crate::codec::$codec::config(&frame.payload) - .map_err(crate::Error::from) - .map_err(unit_error)?; - } - } - let pts = assembly.clock.$name( - &unit, - params.as_ref().into_iter().flat_map(|params| params.nals()), - )?; - let mut published = 0; - for mut frame in frames { - frame.timestamp = pts; - published += u64::from(super::skip_missing_keyframe(import.decode([frame]))?); - } - Ok::<_, Fault>(published) - })(); - match result { - Ok(units) => { - published += units; - assembly.clock.commit(); - } - Err(Fault::Damaged { cause, .. }) => { - // This AU has a known end. Keep the next AU's bytes, - // which may already be here in the same or following PES. - if let Some(params) = params { - split.restore(params); - } - split.reset(); - assembly.clock.reset(); - import.cut(None)?; - let count = assembly.damage.take().map_or(1, |(count, _)| count + 1); - assembly.damage = Some((count, cause)); - continue; - } - Err(err) => return Err(err), - } - if let Some(reorder) = unit.reorder { - import.observe_reorder(reorder)?; - } - if unit.end_sequence { - assembly.clock.reset(); + while let Some(unit) = assembly.next(eof)? { + let params = unit.parameters.then(|| split.params()); + // No provisional timestamp reaches the producer: the AU's own replaces it below. + let provisional = unit.pts.unwrap_or(Timestamp::ZERO); + let mut frames = split + .decode(&unit.data, provisional) + .map_err(crate::Error::from)?; + frames.extend(split.flush(provisional).map_err(crate::Error::from)?); + ensure_supported!(frames.len() == 1, "unsupported TS video access-unit framing"); + for frame in &frames { + if frame.keyframe { + crate::codec::$codec::config(&frame.payload).map_err(crate::Error::from)?; } } - Ok::<_, Fault>(()) - })(); - match result { - Ok(()) => Ok(published), - Err(Fault::Damaged { cause, .. }) => Err(Fault::Damaged { cause, published }), - Err(Fault::Fatal(err)) => Err(Fault::Fatal(err)), + let pts = assembly.clock.$name( + &unit, + params.as_ref().into_iter().flat_map(|params| params.nals()), + )?; + for mut frame in frames { + frame.timestamp = pts; + published += u64::from(super::skip_missing_keyframe(import.decode([frame]))?); + } + assembly.clock.commit(); + if let Some(reorder) = unit.reorder { + import.observe_reorder(reorder)?; + } + if unit.end_sequence { + assembly.clock.reset(); + } } + Ok(published) } }; } @@ -558,6 +544,43 @@ mod tests { assert_eq!(staging.units.load(Ordering::Relaxed), 0); } + #[test] + fn a_pes_opening_mid_nal_is_acquired_at_its_first_start_code() { + // The orphan bytes end in a partial start code the next PES completes, at a join and + // after a break alike. + for reset in [false, true] { + let staging = Arc::default(); + let mut assembly = Assembly::new(Codec::Avc); + if reset { + assembly + .push(pending(&staging, &[0, 0, 1, 0x65, 0x80, 0x11], Some(90_000))) + .unwrap(); + assert!(assembly.next(false).unwrap().is_none()); + assembly.reset(); + } + assembly + .push(pending(&staging, &[0x9a, 0xbc, 0xde, 0xf0, 0, 0, 0], Some(180_000))) + .unwrap(); + assert!(assembly.next(false).unwrap().is_none()); + assert_eq!( + staging.bytes.load(Ordering::Relaxed), + 3, + "only a possible start code is kept" + ); + assembly.push(pending(&staging, &[1, 0x65, 0x80, 0x22], None)).unwrap(); + let unit = assembly + .next(true) + .unwrap() + .expect("the picture after the orphan bytes"); + assert_eq!(unit.data.as_ref(), [START_CODE.as_ref(), &[0x65, 0x80, 0x22]].concat()); + assert_eq!(unit.pts.unwrap().as_secs().unwrap(), 2, "its PES's timestamp"); + drop(unit); + assembly.reset(); + assert_eq!(staging.bytes.load(Ordering::Relaxed), 0); + assert_eq!(staging.units.load(Ordering::Relaxed), 0); + } + } + #[test] fn lost_tail_releases_bytes_stamps_and_clock() { let staging = Arc::default();